Author Topic: basic steps to move from VWS to weather-data  (Read 1100 times)

0 Members and 1 Guest are viewing this topic.

Offline marklyn

  • Member
  • *
  • Posts: 43
basic steps to move from VWS to weather-data
« on: July 31, 2020, 08:54:55 PM »
Forgiveness if there is a post already, I looked and didn't see an obvious answers to questions below.
I've had VWS since 2006 and am ready to give it up to move to WD based on what appears to be a pretty robust software and community.
Right now my VWS is set up to upload to WU only and I have a web interface created that I use.
I also have the VWS datalogger in use.
I'd like to repeat this setup in WD and see that I can import archive yearly files into WD.
What would be the basic steps to move over to WD from VWS?
Can I run them both until I'm comfortable enough to discontinue VWS?
Does WD have csv export ability (yes/no will do, just want to know for sure since I didn't see it in trial)
In general is WD a solid product to move to from VWS?
Thanks all.

Offline waiukuweather

  • Forecaster
  • *****
  • Posts: 1072
Re: basic steps to move from VWS to weather-data
« Reply #1 on: July 31, 2020, 09:09:16 PM »
you can use your wunderground data as a way to get data into WD
see under action, import...then select WU and then enter your WU ID

yes WD can be set to create csv logfiles (see in the logfile setup) or even custom logfiles (using the custom tags)

Offline marklyn

  • Member
  • *
  • Posts: 43
Re: basic steps to move from VWS to weather-data
« Reply #2 on: July 31, 2020, 10:42:37 PM »
you can use your wunderground data as a way to get data into WD
see under action, import...then select WU and then enter your WU ID

yes WD can be set to create csv logfiles (see in the logfile setup) or even custom logfiles (using the custom tags)
It sounds like I have two ways to get my data into WD, WU and import from WVS... is one better over the other (easier/more accurate)?
I notice from WU import you choose from day to day 31 then each month, but from VWS you just choose an entire database file for the year...
Wouldn't that be faster from VWS?
« Last Edit: July 31, 2020, 10:52:01 PM by marklyn »

Offline waiukuweather

  • Forecaster
  • *****
  • Posts: 1072
Re: basic steps to move from VWS to weather-data
« Reply #3 on: August 01, 2020, 01:49:04 AM »
I would test the VWS CSV import for 1 file
and if that works then you can use the convert all found for the year selected

Offline marklyn

  • Member
  • *
  • Posts: 43
Re: basic steps to move from VWS to weather-data
« Reply #4 on: August 01, 2020, 08:46:06 AM »
I would test the VWS CSV import for 1 file
and if that works then you can use the convert all found for the year selected

I just read somewhere that v15 of VWS only holds about 40k records and if you have VWS set to log every 5 minutes you won't have a full year of data in the database... in this case it would seem I should consider importing from WU.  Anyone confirm this?

Offline galfert

  • Global Moderator
  • Forecaster
  • *****
  • Posts: 6822
Re: basic steps to move from VWS to weather-data
« Reply #5 on: August 01, 2020, 09:03:47 AM »
60 min / 5 min = 12 records per hour
12 x 24 = 300 records per day
40,000 / 300 = 133.33333 days worth of records

EDIT: This math does not apply. See next post.
« Last Edit: August 01, 2020, 09:28:51 AM by galfert »
Ecowitt GW1000 | Meteobridge on Raspberry Pi
WU: KFLWINTE111  |  PWSweather: KFLWINTE111
CWOP: FW3708  |  AWEKAS: 14814
Windy: pws-f075acbe
Weather Underground Issue Tracking
Tele-Pole

Offline galfert

  • Global Moderator
  • Forecaster
  • *****
  • Posts: 6822
Re: basic steps to move from VWS to weather-data
« Reply #6 on: August 01, 2020, 09:16:17 AM »
But the it seems it creates yearly backups and the limit is 20,000 records according to the manual. Unless the manual that is available is not current. A bit odd wording below from the manual. 20,000 records becomes 15,000 records and then backed up it seems.

Quote
A database  timer  allows  you  to  customize how often data  is written  to  the  database.   The  database is  limited to  20,000 records,  so  the  recommended  minimum  database  timer is 10 minutes. Once the database  reaches  20,000  records,  is  automatically resized  to  15,000 records.   A backup record  of  the database is  stored in the following  file:

\VWS\data\yyyydbase.csv where yyyy is the current year. If the database is damaged or deleted, simply rename the backup database.

« Last Edit: August 01, 2020, 09:21:56 AM by galfert »
Ecowitt GW1000 | Meteobridge on Raspberry Pi
WU: KFLWINTE111  |  PWSweather: KFLWINTE111
CWOP: FW3708  |  AWEKAS: 14814
Windy: pws-f075acbe
Weather Underground Issue Tracking
Tele-Pole

Offline galfert

  • Global Moderator
  • Forecaster
  • *****
  • Posts: 6822
Re: basic steps to move from VWS to weather-data
« Reply #7 on: August 01, 2020, 09:24:14 AM »
What is the contents of your \VWS\data\  directory folder?
What are the names of the .csv files?
Ecowitt GW1000 | Meteobridge on Raspberry Pi
WU: KFLWINTE111  |  PWSweather: KFLWINTE111
CWOP: FW3708  |  AWEKAS: 14814
Windy: pws-f075acbe
Weather Underground Issue Tracking
Tele-Pole

Offline marklyn

  • Member
  • *
  • Posts: 43
Re: basic steps to move from VWS to weather-data
« Reply #8 on: August 01, 2020, 10:04:12 AM »
What is the contents of your \VWS\data\  directory folder?
What are the names of the .csv files?
I'm not sure why there are old 2006-2009 database files, if I created them I don't remember.
 [ You are not allowed to view attachments ]

Offline galfert

  • Global Moderator
  • Forecaster
  • *****
  • Posts: 6822
Re: basic steps to move from VWS to weather-data
« Reply #9 on: August 01, 2020, 02:44:10 PM »
Looks like some of those were created on Dec 31 of each given year during the last few minutes of that year, so they were created automatically. I'm not sure why you have a gap and then the current database shows today's date as last modified date. There is CSV folder that may contain more files. Look there.
Ecowitt GW1000 | Meteobridge on Raspberry Pi
WU: KFLWINTE111  |  PWSweather: KFLWINTE111
CWOP: FW3708  |  AWEKAS: 14814
Windy: pws-f075acbe
Weather Underground Issue Tracking
Tele-Pole

Offline marklyn

  • Member
  • *
  • Posts: 43
Re: basic steps to move from VWS to weather-data
« Reply #10 on: August 01, 2020, 03:02:42 PM »
Looks like some of those were created on Dec 31 of each given year during the last few minutes of that year, so they were created automatically. I'm not sure why you have a gap and then the current database shows today's date as last modified date. There is CSV folder that may contain more files. Look there.

We saw the gap too and don't understand. There are no more database files in any of the other folders. We suspect that there was a change in the program at one point during an update which no longer created the yearly database files.
I guess my main question about WD is whether or not I can import my weather data from WU from previous years and run reports from WD that includes that data.  I wasn't able to find a clear answer on the WD website but I'm still googling around. If anyone knows this, would appreciate hearing the answer.

Offline droiddk

  • Forecaster
  • *****
  • Posts: 334
Re: basic steps to move from VWS to weather-data
« Reply #11 on: August 01, 2020, 03:10:36 PM »
There is CSV folder that may contain more files. Look there.

Yes...its been a while since I used VWS but as far as I remember the good data-files were located in /csv

Regards

Offline marklyn

  • Member
  • *
  • Posts: 43
Re: basic steps to move from VWS to weather-data
« Reply #12 on: August 01, 2020, 03:15:50 PM »
There is CSV folder that may contain more files. Look there.

Yes...its been a while since I used VWS but as far as I remember the good data-files were located in /csv

Regards
My /csv folder was empty. I don't recall ever putting any files in that folder and I never remembered deleting any files from any folder.
The archive folder has a daily folder which seems to have .txt files for each day of the year under a year named folder (through 2016, not sure why 2017-2019 aren't there).
It appears the 2017-2019 files are under the root of the daily folder instead of yearly named folders. Again, probably a change in the program?
So it looks like I've got several years data in .txt files but not sure if WD imports text files or still, if getting the data from WU is easier.  Still investigating.
« Last Edit: August 01, 2020, 03:17:31 PM by marklyn »

Offline droiddk

  • Forecaster
  • *****
  • Posts: 334
Re: basic steps to move from VWS to weather-data
« Reply #13 on: August 01, 2020, 03:18:15 PM »

My /csv folder was empty. I don't recall ever putting any files in that folder and I never remembered deleting any files from any folder.
The archive folder has a daily folder which seems to have .txt files for each day of the year under a year named folder (through 2016, not sure why 2017-2019 aren't there).
So it looks like I've got data through 2016 stored locally, but I still wonder if it's just easier and more complete getting it from WU.  Still investigating.

Yes...it may have been a optional feature saving to /csv but I recall that folder being full of subfolders with .csv files with daily, monthly and yearly data.

I know, it wont bring you any longer :/

Regards

Offline 92merc

  • BismarckWeather.net
  • Forecaster
  • *****
  • Posts: 1308
  • BismarckWeather.net
    • BismarckWeather.net
Re: basic steps to move from VWS to weather-data
« Reply #14 on: August 01, 2020, 03:24:31 PM »
I ran into the same thing when I left VWS.  I ran Cumulus for a little while, then added WD to the mix so I run both.  But I had data going back to 2004 in the same CSV files.  Then I had about a 2 year gap.  I remembered that was about the time I moved to a new PC with VWS.  I think a setting didn't get tripped up on the move.  Those CSV files are small, and I wanted a copy for each year.  But after the move, I'm guessing that setting didn't stick.

But I was able to use the WD import from WU to get the data.  With WU sucking a$$, I'm not sure that import will still work.  But my VWS CSV files imported perfectly.
https://www.BismarckWeather.net
Davis VP2, Cumulus, WeatherDisplay, Blitzortung, Saratoga Scripts, NOAA Stream via PI

Offline marklyn

  • Member
  • *
  • Posts: 43
Re: basic steps to move from VWS to weather-data
« Reply #15 on: August 01, 2020, 03:28:05 PM »
I ran into the same thing when I left VWS.  I ran Cumulus for a little while, then added WD to the mix so I run both.  But I had data going back to 2004 in the same CSV files.  Then I had about a 2 year gap.  I remembered that was about the time I moved to a new PC with VWS.  I think a setting didn't get tripped up on the move.  Those CSV files are small, and I wanted a copy for each year.  But after the move, I'm guessing that setting didn't stick.

But I was able to use the WD import from WU to get the data.  With WU sucking a$$, I'm not sure that import will still work.  But my VWS CSV files imported perfectly.
I've done a few months test from WU -> WD and it seems to be pulling data but I'm still learning how to use WD to run reports to see if the data is complete (or vs my WVS data I have on disk). Still playing around with it.  Bit of a learning/research curve.