WXforum.net
Weather Software => WeatherLink/weatherlink.com by Davis Instruments => Topic started by: sillyfreak on February 26, 2017, 08:17:45 PM
-
I recently purchased a Davis Vantage Pro2 system and WeatherLinkIP data logger/software. I have my "free" account on www.weatherlink.com setup and have been playing around with the provided WeatherLink software. I understand from other comments on the subject that it might not be the best available, but the strip chart/plotting functionality seems to me to be adequate for my purposes. I have chosen a 5-minute archive interval, which seems to be a common choice of uses on the forum.
I would like to be able to analyze long-term weather data to get things like heating and cooling degree days and rainfall amounts on a monthly and yearly basis. I would also like to be able to extract extreme events such as rainfall rates and wind speeds.
As I begin my endeavours, I was looking for (and have been unable to locate) a thread for general schemes/strategies for long term collection of weather data. With this in mind, I have a few questions I was hoping to get some replies to from the experts out there.
1) At 5 minute archive intervals, I see "HELP" file entries that report that I will have about five days worth of storage on my data logger. How can I determine the current "percent full" for my data logger?
2) What is a good strategy for downloading and analyzing weather data from the data logger? Under "BROWSE" I see an "EXPORT RECORDS" option were I can create tab delimited files of the logger data. I can open these in Excel, but I believe I would prefer to view them in the WeatherLink software. Is it possible (or desirable) to append this data to the current data from the data logger? I see that I can view plots for an entire year, but It is unclear how to append data to the existing data downloaded from the data logger.
3) Should I consider other data collection/analysis programs/applications?
4) Is CWOP something I should consider?
I have a hundred other questions, but I will leave it with these four for now. Any insight for a "newbie" will be appreciated.
-Mark
-
1) According to the fancy manual, at 5 min intervals it is 8 days. AFAIK there is no way to interrogate the logger; you'd have to approximate from the last date of download to know what is left. You should set it up to download regularly if you want current data.
2) While WLK (Weatherlink) files are great they are hard to work with so I'd use an SQL database to capture the data. There's a post on the forum here about some guy who did that by sniffing the TCP stream. Or get a Meteobridge box and put it into the loop and then you get all sorts of stuff.
3) Sure - why not? Lots of options from Cumulus (donationware) to Weatherdisplay (paid) to roll yer own,
4) What would that accomplish? Or do you mean publish to Wunderground or other PWS sites?
-
First, have a really good read of the Weatherlink Help pages (ie via the Help menu) - there's a huge amount of description and explanation there of what the software can do.
Do the downloads from weatherlink.com and not direct from the logger: (a) it's usually faster and more robust; (b) you get 10,240 records' worth of storage there, rather than 2560 in the logger itself, ie about 20 days @5min interval, rather than just 5 days. That said, it's good policy to download as often as is convenient for you, eg maybe daily if not more often - this way you get early warning if three are any data anomalies (assuming that you actually check the data post download).
-
I would got with CWOP. Your data is then sent to MADIS and also MesoWest in Utah. At the present WU so messed up they not sure if their head is stuck in the sand or in their butt. There an forum section here on WU , take some time to read all the problems folks are having with them. Also CWOP you can watch your data and get some what of an clue if something is going out of whack with Madis checking it.
John
-
I would like to be able to analyze long-term weather data to get things like heating and cooling degree days and rainfall amounts on a monthly and yearly basis. I would also like to be able to extract extreme events such as rainfall rates and wind speeds.
If you are really serious about this, consider getting an Envoy (Davis model 6316) to add to your setup. But otherwise, just make sure that your computer (with WeatherLink software) gets the data.
-
2) While WLK (Weatherlink) files are great they are hard to work with so I'd use an SQL database to capture the data. There's a post on the forum here about some guy who did that by sniffing the TCP stream. Or get a Meteobridge box and put it into the loop and then you get all sorts of stuff.
Is an SQL database easier to work with than a .csv type spreadsheet?
I export the records from WeatherLink to text, then paste into a spreadsheet, then I can work with the data how I want. I don't know why WL doesn't export directly to spreadsheet, but the conversion to SQL looked more involved to me, but I've only had very little experience with SQLite.
-
I don't know why WL doesn't export directly to spreadsheet...
Not quite sure what you mean by 'directly', but the tab-delimited text files from WL can be imported directly into eg Excel. No need for cut and paste.
-
2) While WLK (Weatherlink) files are great they are hard to work with so I'd use an SQL database to capture the data. There's a post on the forum here about some guy who did that by sniffing the TCP stream. Or get a Meteobridge box and put it into the loop and then you get all sorts of stuff.
Is an SQL database easier to work with than a .csv type spreadsheet?
I export the records from WeatherLink to text, then paste into a spreadsheet, then I can work with the data how I want. I don't know why WL doesn't export directly to spreadsheet, but the conversion to SQL looked more involved to me, but I've only had very little experience with SQLite.
SQL has far more power and storage etc. A bit more complicated but if you stuff your data into an SQL DB you could run free analytics SW over it (visualization SW like Tableau). Excel is easier but you run into all sorts of constraints. Depends on your goals.
-
Excel is obviously more user-friendly, but SQL can do much more if you know how to use it. Also remember that Excel has limitations, eg. max number of rows, which might seem like you will never reach that limit, but with regular weather data it actually doesnt take that long...
-
Thanks for the replies on SQL. I may have to learn something new. :shock:
As for as the size of the data I only log 30 minute intervals and work with a year at a time to get the data I'm looking for. I also extract some data from the NOAA monthly reports (that's where the paste comes in). I know I'm probably doing it all wrong, but I still have fun, might have to see how fun (or not) SQL is now.
This is what I think we're talking about: http://www.wxforum.net/index.php?topic=29552 "WeatherLink Observations in a MYSQL database"
Hopefully this may give the OP some ideas, I didn't intend to hijack his thread.
-
Yeah, that SQL reader/DB is exactly what we are talking about. You could overlay a free visualization tool on the DB and have tremendous graphics/analytics.
-
Thanks to everyone for the great responses. To summarize my thoughts/findings/understandings...
1) At five minute archive intervals, it looks like (approximately) 8 days worth of data is what the logger will store. My strategy will be to download (a minimum) of once a week. I understand that when the the logger gets full, it just writes the current data set over the first data set. Therefore, after the first eight days, the data logger always contains the latest (approximately) eight days worth of data.
2) As I download current data, the WeatherLink software seems to "remember" the previously downloaded data. When I select BROWSE and CHOOSE DATE, and select the date in which I created the station, it gives me all the data back to this creation date. Looking in the Weatherlink folders, I see a 2017-02.wlk and 2017-03.wlk file that seem to get created automatically every time I exit the program.
With this understanding, my strategy is to just make sure to DOWNLOAD the data from the logger at least once a week and I will be able to view any/all data ranges using my Weatherlink software.
If I have a hard drive crash, I would just need to go to my Carbonite backup (within eight days of the last station download) and restore all the YYYY-MM.wlk files. Then after launching Weatherlink, it would automatically read them back in and I would be back in business.
If I want to view/analyze in a database or excel, I just do an EXPORT to export all the data.
3) I plan to investigate suggestions such as Cumulus and Weathersiaplay in the future as I develop my long range plan. I believe my strategies described above will allow me to gather and import all the historical data from my weather station and upload into any weather station software.
4) I registered with CWOP and am currently working to get the automatic uploads setup. I currently have some internet/communication errors that I am working through.
Thanks again for all the ideas/suggestions!
-
Your thoughts/findings/understandings... are well thought out and right on! There are many options but you've got a pretty simple one planned, for now!!
Enjoy,
Paul
-
Good Morning!
I will try to keep this as brief as possible, but you mention a couple of points here that I would like to chime in on.
For item #2, these of course are your data files. When new observations are recorded, data is always recorded to the latest YYYY-MM.wlk file. You mention something that I have always find peculiar, the first data file always gets modified in this process too. In my situation, my first data file is 2008-04. I've never been clear as to why this file would get continuously touched as well. When April gets here, you should then notice that your 2017-02 and 2017-04 files will get modified. Like I said, I'm still curious as to why this happens, so if anyone else has a theory, I'd love to hear it.
Personally, I am paranoid about data loss. At home, on my importance scale, my weather data finishes a close 2nd next to my family pictures. It's all about how much resources you throw at the problem, but how much of the data are you prepared to lose? I try to take a multi-tier approach to protecting my weather data.
I will keep the shameless plugging to a minimum, but I wrote a powershell script to help with my weather data backup strategy. My strategy looks like this:
- On my home server, my live weather data exists on two mirrored SSDs.
- At the top of every hour, my script will copy the entire weatherlink folder to a temporary, single SSD. Once copied to this temporary SSD, the script will call 7-zip to compress it. This takes the file size from ~300Mb to ~30Mb. The script will then copy the data to large, slower drives in a RAID 5 configuration for archiving. I have it configured to keep backups every hour, for 30 days.
- Every month I then make a complete backup of the entire server to an external drive that I keep off-site.
- My data does exist in a cloud-based SQL server. While it is not complete, I am working on a new feature of the program I use for storing the data in a database that would allow me to recreate WLK files from the observations in the database. When it is done, this will be another avenue for recovery. (capability to repair/edit files is a goal as well)
In the event of drive failures, I feel that I am at risk of losing only up to an hour of data. The chances are small that I would lose both the mirrored set and the RAID 5 at the same time.
What I am not well protected against now is the event that my home is destroyed, or even if my house is broken into and my equipment is stolen. I haven't got around to adding this to the backup script yet, but to protect myself against those scenarios, I am going to add a step that would allow you to copy and archive the data to a cloud-based storage solution like dropbox or something. Since the compressed backup is only ~30Mb, this should not take up a lot of bandwidth or cloud storage.
Sorry for the rambling, this is something I unfortunately think about all the time.
-
Wow, great backup system!
I think that after you make sure you have a back up at two physically separated places (eg. using cloud), the only thing that could lead to complete loss of data is if the Earth was hit by an asteroid :D
-
You mention something that I have always find peculiar, the first data file always gets modified in this process too. In my situation, my first data file is 2008-04. I've never been clear as to why this file would get continuously touched as well. When April gets here, you should then notice that your 2017-02 and 2017-04 files will get modified. Like I said, I'm still curious as to why this happens, so if anyone else has a theory, I'd love to hear it.
That is interesting. Since I only have two WLK files, I assumed they all got saved/modified. I will look for the change in April when I will have three WLK files. As far as a theory, I am not sure I have one that makes sense. You would think every dataset for a particular interval would be self contained and not require a look back to the first WLK file.
Your backup strategy is rock solid. My Carbonite solution is not free, but I believe it is well worth the $60 a year I spend to backup these and other files.
-
The question is if that file really is modified.
It could be that the SW is just for some reason checking it, opens it and closes it and this would show as "modified" if it is opened and resaved. Best way to check this would be creating a backup, then deleting the "modified" file, trying to put the original one back and see if it works.
-
Doesn't the Davis® WeatherLink website LIMIT your retrieval access to something like once per hour?
-
It has a new file save date, but it does not seem to get modified. I copied the first WLK file to a dummy file, exited Weatherlink, and did a DOS file compare (FC /B) on the two files. It did not detect any differences. Therefore, I believe it does not get modified, it just gets opened and saved.
-
Doesn't the Davis® WeatherLink website LIMIT your retrieval access to something like once per hour?
I am not sure. I am retrieving the data from my console using the WeatherLinkIP, not from weatherlink.com.
I am poking around on my weatherlink.com account and I can't seem to find how to download the data... All I see is an "UPLOAD" link at the top. Where is the option to download data?
-
My .wkl files going back to 2008-07 all have a time of 3/01/17 8:10 PM which is when I last started WL to run for a few minutes, which I do twice a month to maintain my data, with the exception of 2017-02 and 2017-03.wlk which have a time of 8:13 PM and that is when I exited from WL. The only action I took on 3/01/17 was to download the logger data and then browse the log, graphs, NOAA and Rain reports, then exit and restart Cumulus.
Paul
-
Doesn't the Davis® WeatherLink website LIMIT your retrieval access to something like once per hour?
I am not sure. I am retrieving the data from my console using the WeatherLinkIP, not from weatherlink.com.
I am poking around on my weatherlink.com account and I can't seem to find how to download the data... All I see is an "UPLOAD" link at the top. Where is the option to download data?
When you click on the 2nd from left icon-button in the WeatherLink 6.0.3 control bar, it is actually "querying" back to the Davis WL website for it to download the wx data that was transmitted to it from the WL-IP dongle over the internet. So, in effect, you're "downloading" what was sent in the past...thus, you're "retrieving" the data BACK into your computer, the data that was automatically uploaded/sent.
-
Doesn't the Davis® WeatherLink website LIMIT your retrieval access to something like once per hour?
I am not sure. I am retrieving the data from my console using the WeatherLinkIP, not from weatherlink.com.
I am poking around on my weatherlink.com account and I can't seem to find how to download the data... All I see is an "UPLOAD" link at the top. Where is the option to download data?
Data will be downloaded into WeatherLink (the software) depending on the settings in Setup>Communication Port. Local Device ID selected then data will come directly from the WLIP and each new record is immediately available, Web Download selected (requires user ID & Password) then data will come from WeatherLink.com, data which has been previously uploaded as part of WeatherLink.com
As somebody mentioned data is downloaded from WeatherLink.com in 1 hour blocks, mainly because that appears to be the upload setting time frame. Something I have never tested with WeatherLink.com is the 2 hour archive interval?
-
Two hour archive does not make sense to me. Doesn't match weather services and too much happens in 120 min
-
If you want to show your data online on a website I would be aiming for max delays of 5-10 mins. I know some users of MT update the data every 2.5s! Which is probably bit of an overkill, but in Meteotemplate I recommend 1 minute for the real-time data - an ideal compromise between how current the data is and server load.
-
Two hour archive does not make sense to me. Doesn't match weather services and too much happens in 120 min
No 120 minutes doesn't match typical weather services reporting (as such) but was/is useful in some situations and mainly pre web based comms or where there are no comms as a data archive and/or backup data archive as some data is better than no data at all and 120 minutes provided around 130 days without download.
-
That was then - this is now. I used to use 60 minute archive but with the advent of UPS, Weaved/RPi etc. I am at 5 minutes. :)
-
That was then - this is now. I used to use 60 minute archive but with the advent of UPS, Weaved/RPi etc. I am at 5 minutes. :)
Even now 120 minutes is still used even with all the modern comms, mainly for backup, multiple loggers supporting normal data requirements and real time voice systems will run the primary data logger at say 10 minutes and the serial logger feeding the voice system at 120 minutes, mainly because with voice there is no way to access the logged data but if there is no live access to the primary data logger for some reason then after 17 days (@ 10 minutes) you start loosing data, at least the other logger at 120 minutes will retain data for 130 days before rolling it off which at least gives one a chance and some timely grace with fixing the primary problem and still having a continuous data record even if it's by manual download means.
-
Not. Like Environment Canada. Or most forecast models. Two hours is essentially USELESS.
-
Not. Like Environment Canada. Or most forecast models. Two hours is essentially USELESS.
I think people sometimes forget that Davis stations are bought by a whole spectrum of users. IME hobbyists, as largely represented in forums like this, are only a minority category of user.
Users overall do sometimes have applications where certainly 1-hour and occasionally 2-hour intervals are the only option that can be used. Say you're setting up weather logging as background information for some long-term ecological or environmental project in a relatively inaccessible location. You may be planning to visit the site only eg 2 or 3 times a year (especially if the site cannot easily be reached over the winter months) to do a data download to a laptop so 1-hour (100 days logging) and 2-hour (200 days) are often the only practical options.
And even if you're running a Connect for data connectivity then if you're using a 1-hour data plan (which is fairly common) then that's the granularity of your data, nothing more.
Remember that min/max temperatures, max wind gust etc are all recorded individually within an interval, so even 2-hour logging does generate a reasonably detailed picture of weather conditions over each 24-hour period.
-
CLARIFICATION: that 1-hour Davis update interval is for "manual" downloading from the Davis WL site to the Davis WL-software 6.0.3 software program in your computer, it has NOTHING to do with either the upload/send interval nor how fast "other" (outside) programs can fetch or query the Davis WL site for data...I believe.
-
CLARIFICATION: ...
Not 100% sure we're talking about exactly the same thing, but AIUI the context is:
There are three ways of uploading to wl.com: WL v6 subscription; 6555 WLIP logger; Connect;
WL v6 and 6555 logger only upload archive data once per hour. So it's not that you can't try to download archive data more often than hourly, it's more that there's no point in doing so because there won't be any new data to download.
Connect archive data does seem to be available at the plan interval, ie if you have a 5 or 15min plan then new data is there after every upload. (The hourly plan obviously behaves as per WL v6 and 6555 logger.)
-
CLARIFICATION: ...
Not 100% sure we're talking about exactly the same thing, but AIUI the context is:
There are three ways of uploading to wl.com: WL v6 subscription; 6555 WLIP logger; Connect;
WL v6 and 6555 logger only upload archive data once per hour. So it's not that you can't try to download archive data more often than hourly, it's more that there's no point in doing so because there won't be any new data to download.
Connect archive data does seem to be available at the plan interval, ie if you have a 5 or 15min plan then new data is there after every upload. (The hourly plan obviously behaves as per WL v6 and 6555 logger.)
So, you're telling me (us) that the WL-IP #6555 logger does NOT send data immediately, but rather "holds" it until it receives a 'control' HOUR-command (from where cometh?) before it sends anything onto the internet?
-
So, you're telling me (us) that the WL-IP #6555 logger does NOT sent data immediately, but rather "holds" it until it receives a 'control' HOUR-command (from where cometh?) before it sends anything onto the internet?
Not exactly: I'm making a distinction between the current conditions packets (ie what you see on the wl.com web pages) which are uploaded every minute but are not - AFAIK - stored anywhere and the archive packets which are uploaded only once per hour and are stored (to a current max of 10240 records) at wl.com. The number of archive packets uploaded each hour will depend on the archive interval set, eg a 10-min interval will result in 6 packets being uploaded each hour.
The trigger for the upload is presumably just the local console time. AFAIK wl.com never talks to the IP logger in the sense of issuing commands (though maybe the individual data packets are acknowledged as part of whichever upload protocol is being used), it's one-way upload traffic only
-
Normally the 6555 is uploading current conditions within a 60 second window (unless of course other hardware is controlling the WLIP upload interval) to the My Weather & Summary pages. What may not be exactly understood is how the archive packets are handled as it makes little sense to be uploading data on a 60 second basis and then turn around an upload basically the same data (mostly) again for archive records but according to the WLIP summary page it does show a specific time when the archive upload occurred and it would make sense for the user device to crunch the upload numbers instead of wasting server capacity? BTW the WLIP upload data usage is extremely frugal ~ 2mb/day generally less for the standard VP2.
What is well known is the uploaded record data can only be downloaded on an hourly (approx.) basis regardless of the archive interval (be it 1 minute or 10 minutes) and it perhaps is made to do this so as to reduce server load especially if users tried to continually poll 1 minute data.
-
What is well known is the uploaded record data can only be downloaded on an hourly (approx.) basis regardless of the archive interval (be it 1 minute or 10 minutes) and it perhaps is made to do this so as to reduce server load especially if users tried to continually poll 1 minute data.
This isn't the right way of looking at it - see my posts above for the Davis explanation. Since _archive_ data is only uploaded once per hour, there is no point in trying to download more than once per hour - any attempts to do so are bound to fail because there is no new data there!
-
What is well known is the uploaded record data can only be downloaded on an hourly (approx.) basis regardless of the archive interval (be it 1 minute or 10 minutes) and it perhaps is made to do this so as to reduce server load especially if users tried to continually poll 1 minute data.
This isn't the right way of looking at it - see my posts above for the Davis explanation. Since _archive_ data is only uploaded once per hour, there is no point in trying to download more than once per hour - any attempts to do so are bound to fail because there is no new data there!
And when you think about it that upload interval is part and parcel of the WLIP firmware and Davis would do that for some reason and one reason is to smooth out server activity and minimize CPU activity.
And yes there is no new data under the hour to download as it hasn't been uploaded yet, so the primary controlling device would appear to be the WLIP.
The other thing to think about is any email data which has to be up to date to the nearest 30 minutes regardless but this is probably just a function of the Summary page data?
-
Just to add a bit more to this and being a little curious I run a WLIP test rig at various archive intervals 1 min, 5 min, 10min etc which as expected conform to the hourly upload sequence but then ran a 120 min archive in which the WLIP did an upload cycle (as shown by WeatherLink.com) an hour latter but there was no data, the next hourly interval produced 1 page / 1 interval (2 hourly) at which time the WLIP must have determined the interval was 120 minutes so from this point on only fired an upload every second hour so there a little bit of intelligence going on.
The other thing a 120 minute interval highlighted is that since the archive interval fall on even time scales the upload and subsequently the download can be up to nearly an hour behind the actual interval recorded.