Power failure & loss of system clock corrupts data

[RSMAC]

Greetings,

I had a power failure last night. My PC was on a UPS and I shut it down gracefully. However, the power was off so long that the PC lost its clock. When the power returned, I restarted the PC but didn't check it. I noticed this morning that no data was being displayed and vws was complaining about an incorrect data point and to check the clock. I corrected the clock without quitting out of vws, probably a mistake, and it started plotting data after day 50 (fifty) of the month. Hmm.

I restarted vws but now all the charts that show data by day of the month don't seem to graph far enough into the day. For example, this snapshot was taken at 10:45 AM so the plot should be almost half way through the day, yet it looks like it's in the first hour or so.


How do I correct this plot? I confess, I've looked though the data directory at the myriad files, and it's not obvious how to correct it. I have a Davis Vantage Pro 2 so it might be possible to fool vws into thinking it needs to retrieve the data logger data again from the time after the power failure.

Thanks.

[Garth Bock]

You said you lost the time and date on your PC ? That would indicate that the CMOS battery is bad or dead. Easy to change in most computers. If this is an very old box your battery is rectangular rather than the hearing aid style you can get at Walgreens. 

Garth
In Illinois....Our Governors Make Our License Plates
In Illinois....You're Innocent Until Elected Governor

[Bushman]

Let's hope the RTC battery is not one of those soldered in.    PITA

As for editing data, try http://www.ruysfamily.com/WLKReaderWriter/WLKAppDownload.html

[RSMAC]

You said you lost the time and date on your PC ? That would indicate that the CMOS battery is bad or dead

Yes, it is an ancient computer, circa 2000. It's only running vws. I'm hoping to replace it, and several other dedicated old fossils with one computer multipurpose computer soon.

[RSMAC]

As for editing data, try http://www.ruysfamily.com/WLKReaderWriter/WLKAppDownload.html

Is that where vws keeps its files? All I see in my C:\\vws\data\ folder are .csv files. I did try to edit those by deleting all the data after the power failure hoping vws would reread everything after that from the data logger, but that didn't work so I restored from my backup.

I suspect that vws is confused by the new data it received after I reset the clock, but may have put that in the wrong place or with the wrong timestamp. I also noticed I have a bogus high temp recorded in one of the panels.

Where does vws keep track of the data so that it knows what to download from the data logger? Does it keep some other record of what it has seen?

Thanks for your help.

[IngemarS]

Well, if you use VWS and not Weatherlink, you have better trying to edit the dbase.csv and 2009dbase.csv (or whatever new files have been created because of the old date - mostly BOISes fall back to the date of the BIOS produced or somthing similar.

I you want to edit CSV, I suggest you have a look at CSVed by Sam Francke.  http://csved.sjfrancke.nl/index.html

I have had some similar problems, though not BIOS-clock related and have found the above mentioned utility very useful.

At one time I also used some other large editors to save portions of a corrupted file into severl others, then I fixed the bad part and joined them together again with another utility.

At one time in the beginning of having the weatherstation I by mistake started using wrong type of measurements. (mmHg instead of hPa, and not Sea Level corrected ... I had to write a program in VB to convert that data - when the small utility had been made it took some 10 minutes to deal with 3.5 years of csv:s).

If you mant more and better suggestions, tell us more about how the data was corrupted?

Cheers,

IngemarS

 

[Bushman]

Sorry - Ingemar is right.   When I read  WL, I assumed Davis files, not VWS .csv files. 

Any way, you can edit .csv files in Excel.  If there are binary or hex numbers though things could get interesting.

[RSMAC]

Any way, you can edit .csv files in Excel.

I did try that by removing all the data that was entered since updating the clock. That didn't work as the graphs showed about two points, I assume from the two minutes it took to edit the file and delete the entries. All the previous history was not graphed. That's why I suspect that there is some other file recording what was last seen from the data logger.

I don't know how many points are stored in the logger, but I think I'll restore the whole vws directory from the early morning backup and see if it roles forward all the way. If not, then I'm no worse off than now.

[RSMAC]

... I think I'll restore the whole vws directory from the early morning backup and see if it roles forward all the way.

No joy. All that happened is that the graphs showed data up until the power failure last night and then started graphing new data thinking it was still yesterday. Now I'm really confused.

I don't know how vws communicates with the Vantage Pro 2 so I don't know if something told the data logger not to send all records from last night. What I'm afraid of now is that this thing will remain out of sync permanently. I guess it's time to try the publisher's tech support.

[Garth Bock]

I would look into changing the CMOS\RTC battery before the next outage because I can see the moment you fix all your data and sit back .....the power goes out again.

Garth
In Illinois....Our Governors Make Our License Plates
In Illinois....You're Innocent Until Elected Governor

[RSMAC]

I would look into changing the CMOS\RTC battery before the next outage because I can see the moment you fix all your data and sit back .....the power goes out again.

Yeah, just my luck. I've been holding off replacing this fossil in the hope that I could replace it with a new computer. Not sure when I can do that though. So, after all this hassle, it's surely worth the price of a battery.

Ed replied very quickly to my tech support request and suggested deleting all the "bad" records. I've replied saying that I had already done that but vws did not seem to request the missing data from the data logger. Nor did it do it when I restored the entire vws folder from backup.

I wish I knew how vws requested data from the logger so maybe I could trick it into requesting the missing data.

[IngemarS]

Well, RSMAC...

I think that once the data is out of the logger it is not there any more - can't be downloaded twice.

Make sure VWS is NOT started while you edit the files. (If you f ex have Davis VPxx, the console, if set up for it, it stores data in the console until the console is in contact with the computer and VWS).

The files to edit are dbase.csv and 2009dbase.csv, most urgent is dbase.csv. (and I prefer CSVed to Excel even though I have been working for MS for 16+ years. For huge csv-files Excel will be really slow.)

When data is correct in the file and VWS and console are connected it will take a while until it has updated the screen meters correctly.

And if you want to correct things while running VWS on the computer with new CMOS battery, you can do that also. Just make sure you divide the corrupt files into different parts before you do it - VWS will create a new dbase.csv if none is found - and later on you can merge the corrected file with the dbase.csv that VWS created instead.

Perhaps this sounds tricky, but it isn't.

Cheers,

IngemarS

[RSMAC]

I think that once the data is out of the logger it is not there any more - can't be downloaded twice.

That's what I'm afraid of, and it does make sense given what I'm seeing. I suppose when vws was downloading from the logger when the time was bad, it may have just been throwing everything away. Speculation!

I haven't heard back from Ed again since I asked how to force a redownload from the logger so that may be another sign the missing data is gone forever.

I monkeyed around with the plot duration, changing it from the last 48 hours to the last 24, and sure enough, the x-axis now shows a segment of missing hours from around 9 PM last night until around 8 AM this morning, which is the missing entries corresponding to the PC clock time screwup.

The graph will probably correct itself if I let a couple of days go by. For now, I'm leaving the graph set for the last 24 hours. It probably just looked worse than it really was with the 48 hour duration, probably because there's a bug in the plotting routine.

[George Richardson]

The Datalogger is a Davis product so they might me the people to ask how to access. I have a lot of questions concerning the operation of the Datalogger. I THINK there is an option of wiping the Datalogger after each download or allowing it to fill and roll old info out the back. However, if that is an option, why would anyone wish to wipe with each download? I KNOW there are several timing options for storing data in the logger which store from a very detailed day to an overview of a couple of years. I would like to know how to change that storage frequency and I would like to know if that setting is reflected in and/or affects either WeatherLink or Virtual Weather Station.

While the answers to these questions may not answer what to do about your corrupted data, it would perhaps give a better overall knowledge of the data flow from the ISS to the various weather programs.

George

[SLOweather]

Check out the Vantage Pro Serial Communications Reference for just about everything you want to know about how to access the logger and what controls it.

[George Richardson]

Thank You.