Author Topic: Weather Data Collection Strategies  (Read 1644 times)

0 Members and 1 Guest are viewing this topic.

Offline sillyfreak

  • Member
  • *
  • Posts: 5
    • Buda, Texas- Leisurewoods
Weather Data Collection Strategies
« on: February 26, 2017, 08:17:45 PM »
I recently purchased a Davis Vantage Pro2 system and WeatherLinkIP data logger/software.  I have my "free" account on www.weatherlink.com setup and have been playing around with the provided WeatherLink software.  I understand from other comments on the subject that it might not be the best available, but the strip chart/plotting functionality seems to me to be adequate for my purposes.  I have chosen a 5-minute archive interval, which seems to be a common choice of uses on the forum.

I would like to be able to analyze long-term weather data to get things like heating and cooling degree days and rainfall amounts on a monthly and yearly basis.  I would also like to be able to extract extreme events such as rainfall rates and wind speeds.

As I begin my endeavours, I was looking for (and have been unable to locate) a thread for general schemes/strategies for long term collection of weather data.  With this in mind, I have a few questions I was hoping to get some replies to from the experts out there.

1) At 5 minute archive intervals, I see "HELP" file entries that report that I will have about five days worth of storage on my data logger.  How can I determine the current "percent full" for my data logger?

2) What is a good strategy for downloading and analyzing weather data from the data logger?  Under "BROWSE" I see an "EXPORT RECORDS" option were I can create tab delimited files of the logger data.  I can open these in Excel, but I believe I would prefer to view them in the WeatherLink software.  Is it possible (or desirable) to append this data to the current data from the data logger?  I see that I can view plots for an entire year, but It is unclear how to append data to the existing data downloaded from the data logger.

3) Should I consider other data collection/analysis programs/applications?

4) Is CWOP something I should consider?

I have a hundred other questions, but I will leave it with these four for now.  Any insight for a "newbie" will be appreciated.

-Mark

Offline Bushman

  • Forecaster
  • *****
  • Posts: 6593
    • Eagle Bay Weather
Re: Weather Data Collection Strategies
« Reply #1 on: February 26, 2017, 09:05:51 PM »
1) According to the fancy manual, at 5 min intervals it is 8 days.  AFAIK there is no way to interrogate the logger; you'd have to approximate from the last date of download to know what is left.  You should set it up to download regularly if you want current data.

2) While WLK (Weatherlink) files are great  they are hard to work with so I'd use an SQL database to capture the data.  There's a post on the forum here about some guy who did that by sniffing the TCP stream.  Or get a Meteobridge box and put it into the loop and then you get all sorts of stuff.

3) Sure - why not?  Lots of options from Cumulus (donationware) to Weatherdisplay (paid) to roll yer own,

4) What would that accomplish?  Or do you mean publish to Wunderground or other PWS sites?

Offline johnd

  • Forecaster
  • *****
  • Posts: 2288
    • www.weatherstations.co.uk
Re: Weather Data Collection Strategies
« Reply #2 on: February 27, 2017, 03:03:00 AM »
First, have a really good read of the Weatherlink Help pages (ie via the Help menu) - there's a huge amount of description and explanation there of what the software can do.

Do the downloads from weatherlink.com and not direct from the logger: (a) it's usually faster and more robust; (b) you get 10,240 records' worth of storage there, rather than 2560 in the logger itself, ie about 20 days @5min interval, rather than just 5 days. That said, it's good policy to download as often as is convenient for you, eg maybe daily if not more often - this way you get early warning if three are any data anomalies (assuming that you actually check the data post download).
Prodata Weather Systems
Prodata's dedicated Davis EnviroMonitor website
UK Davis Premier Dealer - All Davis stations, accessories and spares
Littleport, Ely, Cambs UK

Offline Scalphunter

  • Forecaster
  • *****
  • Posts: 2314
Re: Weather Data Collection Strategies
« Reply #3 on: February 27, 2017, 05:50:56 AM »
I would got with CWOP. Your data is then sent to MADIS and also MesoWest in Utah. At the present WU  so messed up  they not sure if their head is stuck in the sand or in their butt.  There an forum section here on WU , take some time to read all the problems folks are having with them.  Also CWOP   you can watch your data  and  get  some what of an clue if something is going out of whack with Madis checking it.

John

Offline dalecoy

  • Forecaster
  • *****
  • Posts: 5898
    • Lee's Summit, MO
Re: Weather Data Collection Strategies
« Reply #4 on: February 27, 2017, 10:22:36 AM »
I would like to be able to analyze long-term weather data to get things like heating and cooling degree days and rainfall amounts on a monthly and yearly basis.  I would also like to be able to extract extreme events such as rainfall rates and wind speeds.

If you are really serious about this, consider getting an Envoy (Davis model 6316) to add to your setup.  But otherwise, just make sure that your computer (with WeatherLink software) gets the data.

Offline SnowHiker

  • Contributor
  • ***
  • Posts: 142
Re: Weather Data Collection Strategies
« Reply #5 on: February 28, 2017, 03:35:36 AM »
2) While WLK (Weatherlink) files are great  they are hard to work with so I'd use an SQL database to capture the data.  There's a post on the forum here about some guy who did that by sniffing the TCP stream.  Or get a Meteobridge box and put it into the loop and then you get all sorts of stuff.
Is an SQL database easier to work with than a .csv type spreadsheet?

I export the records from WeatherLink to text, then paste into a spreadsheet, then I can work with the data how I want.  I don't know why WL doesn't export directly to spreadsheet, but the conversion to SQL looked more involved to me, but I've only had very little experience with SQLite.

Offline johnd

  • Forecaster
  • *****
  • Posts: 2288
    • www.weatherstations.co.uk
Re: Weather Data Collection Strategies
« Reply #6 on: February 28, 2017, 03:58:10 AM »
I don't know why WL doesn't export directly to spreadsheet...

Not quite sure what you mean by 'directly', but the tab-delimited text files from WL can be imported directly into eg Excel. No need for cut and paste.
Prodata Weather Systems
Prodata's dedicated Davis EnviroMonitor website
UK Davis Premier Dealer - All Davis stations, accessories and spares
Littleport, Ely, Cambs UK

Offline Bushman

  • Forecaster
  • *****
  • Posts: 6593
    • Eagle Bay Weather
Re: Weather Data Collection Strategies
« Reply #7 on: February 28, 2017, 09:53:56 AM »
2) While WLK (Weatherlink) files are great  they are hard to work with so I'd use an SQL database to capture the data.  There's a post on the forum here about some guy who did that by sniffing the TCP stream.  Or get a Meteobridge box and put it into the loop and then you get all sorts of stuff.
Is an SQL database easier to work with than a .csv type spreadsheet?

I export the records from WeatherLink to text, then paste into a spreadsheet, then I can work with the data how I want.  I don't know why WL doesn't export directly to spreadsheet, but the conversion to SQL looked more involved to me, but I've only had very little experience with SQLite.


SQL has far more power and storage etc.  A bit more complicated but if you stuff your data into an SQL DB you could run free analytics SW over it (visualization SW like Tableau).  Excel is easier but you run into all sorts of constraints.  Depends on your goals.

Offline Jáchym

  • Meteotemplate Developer
  • Forecaster
  • *****
  • Posts: 8483
    • Meteotemplate
Re: Weather Data Collection Strategies
« Reply #8 on: February 28, 2017, 10:00:33 AM »
Excel is obviously more user-friendly, but SQL can do much more if you know how to use it. Also remember that Excel has limitations, eg. max number of rows, which might seem like you will never reach that limit, but with regular weather data it actually doesnt take that long...

Offline SnowHiker

  • Contributor
  • ***
  • Posts: 142
Re: Weather Data Collection Strategies
« Reply #9 on: February 28, 2017, 02:41:32 PM »
Thanks for the replies on SQL.  I may have to learn something new.  :shock:

As for as the size of the data I only log 30 minute intervals and work with a year at a time to get the data I'm looking for.  I also extract some data from the NOAA monthly reports (that's where the paste comes in).  I know I'm probably doing it all wrong, but I still have fun, might have to see how fun (or not) SQL is now.

This is what I think we're talking about: http://www.wxforum.net/index.php?topic=29552 "WeatherLink Observations in a MYSQL database"

Hopefully this may give the OP some ideas, I didn't intend to hijack his thread.


Offline Bushman

  • Forecaster
  • *****
  • Posts: 6593
    • Eagle Bay Weather
Re: Weather Data Collection Strategies
« Reply #10 on: February 28, 2017, 04:34:05 PM »
Yeah, that SQL reader/DB is exactly what we are talking about.  You could overlay a free visualization tool on the DB and have tremendous graphics/analytics.

Offline sillyfreak

  • Member
  • *
  • Posts: 5
    • Buda, Texas- Leisurewoods
Re: Weather Data Collection Strategies
« Reply #11 on: March 04, 2017, 09:03:49 AM »
Thanks to everyone for the great responses.  To summarize my thoughts/findings/understandings...

1) At five minute archive intervals, it looks like (approximately) 8 days worth of data is what the logger will store.  My strategy will be to download (a minimum) of once a week.  I understand that when the the logger gets full, it just writes the current data set over the first data set.  Therefore, after the first eight days, the data logger always contains the latest (approximately) eight days worth of data.

2) As I download current data, the WeatherLink software seems to "remember" the previously downloaded data.  When I select BROWSE and CHOOSE DATE, and select the date in which I created the station, it gives me all the data back to this creation date.  Looking in the Weatherlink folders, I see a 2017-02.wlk and 2017-03.wlk file that seem to get created automatically every time I exit the program.

With this understanding, my strategy is to just make sure to DOWNLOAD the data from the logger at least once a week and I will be able to view any/all data ranges using my Weatherlink software.

If I have a hard drive crash, I would just need to go to my Carbonite backup (within eight days of the last station download) and restore all the YYYY-MM.wlk files.  Then after launching Weatherlink, it would automatically read them back in and I would be back in business.

If I want to view/analyze in a database or excel, I just do an EXPORT to export all the data.

3) I plan to investigate suggestions such as Cumulus and Weathersiaplay in the future as I develop my long range plan.  I believe my strategies described above will allow me to gather and import all the historical data from my weather station and upload into any weather station software.

4) I registered with CWOP and am currently working to get the automatic uploads setup.  I currently have some internet/communication errors that I am working through.

Thanks again for all the ideas/suggestions!


Offline PaulMy

  • Forecaster
  • *****
  • Posts: 3591
    • KomokaWeather
Re: Weather Data Collection Strategies
« Reply #12 on: March 04, 2017, 09:58:31 AM »
Your thoughts/findings/understandings... are well thought out and right on!   There are many options but you've got a pretty simple one planned, for now!!

Enjoy,
Paul

Offline darkpenguin

  • Senior Member
  • **
  • Posts: 51
    • HoosierWeather
Re: Weather Data Collection Strategies
« Reply #13 on: March 04, 2017, 11:19:57 AM »
Good Morning!

I will try to keep this as brief as possible, but you mention a couple of points here that I would like to chime in on.

For item #2, these of course are your data files.  When new observations are recorded, data is always recorded to the latest YYYY-MM.wlk file.  You mention something that I have always find peculiar, the first data file always gets modified in this process too.  In my situation, my first data file is 2008-04.  I've never been clear as to why this file would get continuously touched as well.  When April gets here, you should then notice that your 2017-02 and 2017-04 files will get modified.  Like I said, I'm still curious as to why this happens, so if anyone else has a theory, I'd love to hear it.

Personally, I am paranoid about data loss.  At home, on my importance scale, my weather data finishes a close 2nd next to my family pictures.  It's all about how much resources you throw at the problem, but how much of the data are you prepared to lose?  I try to take a multi-tier approach to protecting my weather data.

I will keep the shameless plugging to a minimum, but I wrote a powershell script to help with my weather data backup strategy.  My strategy looks like this:

- On my home server, my live weather data exists on two mirrored SSDs. 
- At the top of every hour, my script will copy the entire weatherlink folder to a temporary, single SSD.  Once copied to this temporary SSD, the script will call 7-zip to compress it.  This takes the file size from ~300Mb to ~30Mb.  The script will then copy the data to large, slower drives in a RAID 5 configuration for archiving.  I have it configured to keep backups every hour, for 30 days.
- Every month I then make a complete backup of the entire server to an external drive that I keep off-site.
- My data does exist in a cloud-based SQL server.  While it is not complete, I am working on a new feature of the program I use for storing the data in a database that would allow me to recreate WLK files from the observations in the database.  When it is done, this will be another avenue for recovery. (capability to repair/edit files is a goal as well)

In the event of drive failures, I feel that I am at risk of losing only up to an hour of data.  The chances are small that I would lose both the mirrored set and the RAID 5 at the same time.

What I am not well protected against now is the event that my home is destroyed, or even if my house is broken into and my equipment is stolen.  I haven't got around to adding this to the backup script yet, but to protect myself against those scenarios, I am going to add a step that would allow you to copy and archive the data to a cloud-based storage solution like dropbox or something.  Since the compressed backup is only ~30Mb, this should not take up a lot of bandwidth or cloud storage.

Sorry for the rambling, this is something I unfortunately think about all the time.
Jeeps don't get stuck, they just stop to pose for pictures.

Offline Jáchym

  • Meteotemplate Developer
  • Forecaster
  • *****
  • Posts: 8483
    • Meteotemplate
Re: Weather Data Collection Strategies
« Reply #14 on: March 04, 2017, 11:24:52 AM »
Wow, great backup system!

I think that after you make sure you have a back up at two physically separated places (eg. using cloud), the only thing that could lead to complete loss of data is if the Earth was hit by an asteroid :D

Offline sillyfreak

  • Member
  • *
  • Posts: 5
    • Buda, Texas- Leisurewoods
Re: Weather Data Collection Strategies
« Reply #15 on: March 04, 2017, 11:40:44 AM »
You mention something that I have always find peculiar, the first data file always gets modified in this process too.  In my situation, my first data file is 2008-04.  I've never been clear as to why this file would get continuously touched as well.  When April gets here, you should then notice that your 2017-02 and 2017-04 files will get modified.  Like I said, I'm still curious as to why this happens, so if anyone else has a theory, I'd love to hear it.

That is interesting.  Since I only have two WLK files, I assumed they all got saved/modified.  I will look for the change in April when I will have three WLK files.  As far as a theory, I am not sure I have one that makes sense.  You would think every dataset for a particular interval would be self contained and not require a look back to the first WLK file.

Your backup strategy is rock solid.  My Carbonite solution is not free, but I believe it is well worth the $60 a year I spend to backup these and other files.

Offline Jáchym

  • Meteotemplate Developer
  • Forecaster
  • *****
  • Posts: 8483
    • Meteotemplate
Re: Weather Data Collection Strategies
« Reply #16 on: March 04, 2017, 11:45:26 AM »
The question is if that file really is modified.

It could be that the SW is just for some reason checking it, opens it and closes it and this would show as "modified" if it is opened and resaved. Best way to check this would be creating a backup, then deleting the "modified" file, trying to put the original one back and see if it works.

Offline Old Tele man

  • Singing in the rain...
  • Forecaster
  • *****
  • Posts: 997
Re: Weather Data Collection Strategies
« Reply #17 on: March 04, 2017, 11:47:51 AM »
Doesn't the Davis® WeatherLink website LIMIT your retrieval access to something like once per hour?
• SYS: Davis VP2/WL-IP & Envoy8X/WL-USB
• CWOP: DW6988 - 2 miles NNE of Cortaro, AZ
• WU - KAZTUCSO202, Countryside

Offline sillyfreak

  • Member
  • *
  • Posts: 5
    • Buda, Texas- Leisurewoods
Re: Weather Data Collection Strategies
« Reply #18 on: March 04, 2017, 11:53:25 AM »
It has a new file save date, but it does not seem to get modified.  I copied the first WLK file to a dummy file, exited Weatherlink, and did a DOS file compare (FC /B) on the two files.  It did not detect any differences.  Therefore, I believe it does not get modified, it just gets opened and saved.
« Last Edit: March 04, 2017, 12:07:22 PM by sillyfreak »

Offline sillyfreak

  • Member
  • *
  • Posts: 5
    • Buda, Texas- Leisurewoods
Re: Weather Data Collection Strategies
« Reply #19 on: March 04, 2017, 11:59:24 AM »
Doesn't the Davis® WeatherLink website LIMIT your retrieval access to something like once per hour?

I am not sure.  I am retrieving the data from my console using the WeatherLinkIP, not from weatherlink.com.

I am poking around on my weatherlink.com account and I can't seem to find how to download the data...  All I see is an "UPLOAD" link at the top.  Where is the option to download data?

Offline PaulMy

  • Forecaster
  • *****
  • Posts: 3591
    • KomokaWeather
Re: Weather Data Collection Strategies
« Reply #20 on: March 04, 2017, 12:07:31 PM »
My .wkl files going back to 2008-07 all have a time of 3/01/17 8:10 PM which is when I last started WL to run for a few minutes, which I do twice a month to maintain my data, with the exception of 2017-02 and 2017-03.wlk which have a time of 8:13 PM and that is when I exited from WL.  The only action I took on 3/01/17 was to download the logger data and then browse the log, graphs, NOAA and Rain reports, then exit and restart Cumulus.

Paul

Offline Old Tele man

  • Singing in the rain...
  • Forecaster
  • *****
  • Posts: 997
Re: Weather Data Collection Strategies
« Reply #21 on: March 04, 2017, 04:30:15 PM »
Doesn't the Davis® WeatherLink website LIMIT your retrieval access to something like once per hour?

I am not sure.  I am retrieving the data from my console using the WeatherLinkIP, not from weatherlink.com.

I am poking around on my weatherlink.com account and I can't seem to find how to download the data...  All I see is an "UPLOAD" link at the top.  Where is the option to download data?
When you click on the 2nd from left icon-button in the WeatherLink 6.0.3 control bar, it is actually "querying" back to the Davis WL website for it to download the wx data that was transmitted to it from the WL-IP dongle over the internet. So, in effect, you're "downloading" what was sent in the past...thus, you're "retrieving" the data BACK into your computer, the data that was automatically uploaded/sent.
« Last Edit: March 04, 2017, 04:55:31 PM by Old Tele man »
• SYS: Davis VP2/WL-IP & Envoy8X/WL-USB
• CWOP: DW6988 - 2 miles NNE of Cortaro, AZ
• WU - KAZTUCSO202, Countryside

Offline Mattk

  • Forecaster
  • *****
  • Posts: 408
Re: Weather Data Collection Strategies
« Reply #22 on: March 04, 2017, 05:00:22 PM »
Doesn't the Davis® WeatherLink website LIMIT your retrieval access to something like once per hour?

I am not sure.  I am retrieving the data from my console using the WeatherLinkIP, not from weatherlink.com.

I am poking around on my weatherlink.com account and I can't seem to find how to download the data...  All I see is an "UPLOAD" link at the top.  Where is the option to download data?

Data will be downloaded into WeatherLink (the software) depending on the settings in Setup>Communication Port. Local Device ID selected then data will come directly from the WLIP and each new record is immediately available, Web Download selected (requires user ID & Password) then data will come from WeatherLink.com, data which has been previously uploaded as part of WeatherLink.com

As somebody mentioned data is downloaded from WeatherLink.com in 1 hour blocks, mainly because that appears to be the upload setting time frame. Something I have never tested with WeatherLink.com is the 2 hour archive interval? 

Offline Bushman

  • Forecaster
  • *****
  • Posts: 6593
    • Eagle Bay Weather
Re: Weather Data Collection Strategies
« Reply #23 on: March 04, 2017, 05:34:10 PM »
Two hour archive does not make sense to me.  Doesn't match weather services and too much happens in 120 min

Offline Jáchym

  • Meteotemplate Developer
  • Forecaster
  • *****
  • Posts: 8483
    • Meteotemplate
Re: Weather Data Collection Strategies
« Reply #24 on: March 04, 2017, 06:00:08 PM »
If you want to show your data online on a website I would be aiming for max delays of 5-10 mins. I know some users of MT update the data every 2.5s! Which is probably bit of an overkill, but in Meteotemplate I recommend 1 minute for the real-time data - an ideal compromise between how current the data is and server load.