Topic: Any explanation for this?

[Cienega32]

I don't know how I did but it's not the first time for me. I think it's something to do with importing WLink files but can't swear to it. Any ideas on how I clear this mess up?

[Weather Display]

what data recording interval was the .wlk files recorded at?
it looks like a long data interval
and so hence the squashed up data look

you need to make sure you set the recording interval correctly in the WD Davis log import setup
(e.g 10 for 10 minutes data recording interal (having the data logger set to 1 minutes is best (note that changing that clears the data in the data logger))

[Cienega32]

So what happens if I ran a month with more than one arcint? WD can't import it?

I run a 10 minute arcint in my wlk files. It seems to duplicate data - time and date - here and there. Prior to me uninstalling/reinstalling WD, I had 3 entries for Nov 16, amongst others after importing a 'download.txt' file more than once (my bad). The graph was similar to the one I attached. Days repeated here and there and full days displayed as hours on the 24 hour graph. That bottom graph is displaying at least 8 days of data, counting the solar sensor cycles.

My original install was on 11-07 (37Pb04) and a full reinstall on 11-17 (37Pb08). Reason for the reinstall was the "corrupted data" and my own learning process for the switches, etc. I wanted to start with a fresh copy. I jumped right in and imported all my wlk files from 04-07 to present. I did come across a few from summer 08 that need work but I can't even get last month's in without a problem even after the reinstall.

A lot of Oct data is displayed as Nov data with similar day numbers showing side by side, etc. If I do Oct before Nov, it runs the data right up to the starting date of the software install (11-17). If I do Nov before Oct, it gives me duped data. Either order can give me dates out of order and/or duped. There are a few other scenarios that I ran into but I can't remember them right now but in its current state, the "Advance graphing" function really shows it - "Last hour" showing 5 days, etc.

The sudden 3 day cold snap at the end of Oct (between much higher temps) makes it easy for me to detect the data problems I'm having.

I'm really enjoying this software but I keep running into problems importing my WLink history. I don't want to lose 2.5 years of data and I'm going to keep trying to get around whatever it is that crosses my path until I can get it in there cleanly. Spending a lot of time on searches here and on WD Forum so far with no solution.

I think the best advice I could get right now is in detailing how to remove every bit of data from WD usage for a clean slate without another uninstall/reinstall.

[Cienega32]

Another interesting anomaly is found in the NOAA reports and extremes. The Oct data that I imported shows for the month but any and all "Time" entries are to the hh:m9 and not the hh:m0 (as well as mismatches in data), as shown in my WLink NOAA reports. All my data is in 10 minute increments.

For example - Oct 2nd shows a HI of 81.6 @ 16:09 and LO of -0.00 @ 00:25 in WD but shows as 81.8 @ 4:20p and 48.2 @ 7:30a in WLink.

I have another -0.00 LO temp showing @ 00:25 in WD on another date as well. All other "Time" entries are hh:m9

The WLink numbers match all the way around from the browse data function to deleting and recreating that month's NOAA report. The wlk file itself shows no errors in the WLK Reader/Writer from "jruys".

Another notice is in rain totals. I had 3 events in Oct - .36, .28, .07 totaling .71 . WD shows the correct amounts for the three events but totals it as 00.03 .

[Weather Display]

you should be using a month stamped .wlk data file to import, instead of the download.txt file

you can also edit (clean up. add in missing data (WD needs every 1 minute data), removing other months data that should not be there) the WD logfiles, e.g 102009lg.txt
then use action, convert WD logfiles to data files, and select that logfile, to re convert it to WD's data file
then use the recreate last 31 days data file too

to start clean, simply delete  the WD logfiles and data files files

[Cienega32]

OK - clean slate and first imported Nov wlk then Oct. then Sept.

Under View/Avg-Extremes I find all my Oct under the month of Nov. but only up to "Nov 29". The Oct report shows Oct.

My Oct report shows 4 rain days - correct. Total rain only represents one of those events.

All my "Daily" rain totals appear correct but none of the totals are.

Forget all that - now I'm looking at the Oct data that was imported into WD.

OK - now I'm looking at the Oct 1st data using "Log/data file view" and the first line at time mark 00:00 holds the data that actually occurred at my 00:20 arcint. Minutes 1 thru 10 represent data found in my 00:30 arcint. That 20 minute difference continues on until 06:00 when both WLink and WD data match.

In WD, between 05:30 and 06:00 there is a void in the minutes - it goes from 05:30 to 06:00

The WD data stays in sync until 10-2-2009 @ 00:10. Then the temps zero out until 00:26. That I find odd as it seems that WD fills in the minutes between arcints with the data found at each WLink archive logged. The data found at 00:26 is WLink logged at the 04:40 interval.

All WD entries at that point update on a hh:m6 pattern. That mismatch in WD continues until 01:45. The entry immediately after 01:45 is not 01:46 but (coincidentally enough) 06:00 and there the data matches again.

Randomly skimming thru the Oct log in the WD viewer shows various problems at the 00:10 interval that usually correct at the 06:00 interval. For instance - Oct 23 @ 00:00 - all is good until 00:10 when the data displayed is from 01:10. That one hour skew in the data remains until WD would be showing 05:00. The next entry after 04:59 is 06:00 and all is well... until the next occurrence of a 00:10 time entry. At Oct 24 @ 00:10 the data jumps ahead 20 minutes (2 arcints) until 05:39. The next entry is 06:00 and data syncs once again.

The "jump ahead" intervals are never consistently the same but the time events at which it happens seem to have a common denominator of occurring at the 00:10 mark and correcting at the 06:00 mark. What ever the "jump ahead" interval is will be the difference between the 06:00 entry and the entry that precedes it. This appears to be on a daily basis thru-out the data file.

The fact that my WLink data logger arcint is set at 10 minutes has me looking at the first line of data available to WD. I've only imported SEP, OCT, and NOV wlk files into WD. Looking at SEP in the "Log data file viewer" of WD, I see my first arcint data at time 00:00. Minutes 1 thru 10 are the 00:20 archive in WLink. I have to assume that WD "backfills" the minutes between each arcint available, correct? If that is the case, then it seems there's a problem with the way the first arcint is imported and that anomaly carries itself over to the next month's data as well.

Now I delete all log and data files in WD and start with a clean slate.

No data to be found. OK - good. Now I import the NOV wlk file and open the WD data viewer. There it is. 00:00 shows my arcint data from 00:20 under WLink. My WLink archive data at the 00:30 mark now backfills the WD log file from 00:10 back to 00:01. That 30 minute discrepancy is adjusted at 06:00 with the 06:00 Wlink data showing at WD's 06:00 but from backfilling at the 06:09 mark. The 06:10 entry is now backfilled from the 06:19 entry which is actually the WLink 06:10 entry. That seems to explain the oddly reported times of hh:m9 that I mentioned earlier in the thread.

So (after the import of WLink NOV wlk only) my 00:10 WLink archive data is stripped, the 00:20 is placed into WD's 00:00 entry and my WLink 00:30 is the first true data reported for NOV1 by WD from 00:01 to 00:10. From there it cascades into other anomalies after the 06:00 "adjustments".

[Weather Display]

i think you are refering to problems with WD padding out the data to get 1 minute updates, from the less than 1 minute data logging interval you had in use (1 minute works best)

[Cienega32]

Perhaps I am. If by 'padding' you mean the same thing as I do when I say 'backfilling' then I guess so. 

However, it would seem that your suggestion of using 1 minute intervals would be hiding the error by eliminating any backfilling between events and minimizing the error to being almost negligible; a 2 minute skew of data at best. The first record of the wlk is stripped, the 2nd recond fills the 00:00 time slot and the 3rd record fills the 00:01 time slot - the first true daily record of Time in a Daily Summary Record.

The way it appears to me is "00:01" is the first true data point of any given day in a wlk file - regardless of logging intervals. As found in the Davis "Readme" file under the structure of the archive record: "The time value represents the number of minutes after midnight that the specified event took place (actually the time of the archive record)". That definition, to me, excludes 00:00 as the start of a complete monthly wlk file (or of any given day) at any logging interval. The first being 00:01 in every Daily Summary Record inside a wlk file.

Any and all archive records in the wlk contain a BYTE holding the number of minutes that the record represents. A short holds the value for the minutes past midnite of the end of the archive period; the time of the archive record. With that configuration, how could any complete month's wlk file or even any Daily Summary Record hold a value for 00:00 to start with? A 1 minute archive record would show 1 minute for the total number of minutes in that record that ended and was archived at 00:01. 00:00 would actually contain the data for the last minute of the day prior - 23:59:01 to 24:00:00. The first record of any given day at a 1 minute logging interval would represent 00:00:01 to 00:01:00 and show as an "event" at "00:01".

(I think that) My 10 minute interval shouldn't be a problem if the WD import read the value of archiveInterval, and backfilled from the value of packedTime as contained in each wlk file's archive record and eliminated the use of 00:00.

In other words, it finds packedtime and writes the data at that value of minutes after midnite and every minute prior to equal the value of archiveInterval. The wlk import function doesn't seem to accomplish this cleanly/accurately because of the inclusion of 00:00 as the first time point of each day and even more so with the first Daily Summary Record of any wlk file.

My 10 minute archive interval should show as an event occurring at 00:10 and backfilled with that exact weather data 9 times to 00:01 - 10 1 minute records (for WD) in all. Likewise a 5 minute interval would be at 00:05 and backfilled 4 minutes - 5 1 minute records in all.

Unless there was a sudden and extreme 1 minute change in weather data, it would go virtually unnoticed within each 'Daily Summary Record' that should start at 00:01 (not 00:00) for each day.

Bear in mind that I'm speaking only of the wlk importation not of any live data logging.


By the way GBW - are you Brian (WD author)?

[Cienega32]

GBW - has our dialogue in this thread come to an end???