Author Topic: Weather Data Collection Strategies  (Read 1643 times)

0 Members and 1 Guest are viewing this topic.

Offline Mattk

  • Forecaster
  • *****
  • Posts: 408
Re: Weather Data Collection Strategies
« Reply #25 on: March 04, 2017, 06:33:45 PM »
Two hour archive does not make sense to me.  Doesn't match weather services and too much happens in 120 min

No 120 minutes doesn't match typical weather services reporting (as such) but was/is useful in some situations and mainly pre web based comms or where there are no comms as a data archive and/or backup data archive as some data is better than no data at all and 120 minutes provided around 130 days without download. 


Offline Bushman

  • Forecaster
  • *****
  • Posts: 6593
    • Eagle Bay Weather
Re: Weather Data Collection Strategies
« Reply #26 on: March 04, 2017, 06:36:02 PM »
That was then - this is now.  I used to use 60 minute archive but with the advent of UPS, Weaved/RPi etc. I am at 5 minutes.  :)

Offline Mattk

  • Forecaster
  • *****
  • Posts: 408
Re: Weather Data Collection Strategies
« Reply #27 on: March 04, 2017, 07:18:18 PM »
That was then - this is now.  I used to use 60 minute archive but with the advent of UPS, Weaved/RPi etc. I am at 5 minutes.  :)

Even now 120 minutes is still used even with all the modern comms, mainly for backup, multiple loggers supporting normal data requirements and real time voice systems will run the primary data logger at say 10 minutes and the serial logger feeding the voice system at 120 minutes, mainly because with voice there is no way to access the logged data but if there is no live access to the primary data logger for some reason then after 17 days (@ 10 minutes) you start loosing data, at least the other logger at 120 minutes will retain data for 130 days before rolling it off which at least gives one a chance and some timely grace with fixing the primary problem and still having a continuous data record even if it's by manual download means.
« Last Edit: March 04, 2017, 07:21:04 PM by Mattk »

Offline Bushman

  • Forecaster
  • *****
  • Posts: 6593
    • Eagle Bay Weather
Re: Weather Data Collection Strategies
« Reply #28 on: March 04, 2017, 09:54:21 PM »
Not.  Like Environment Canada.  Or most forecast models.  Two hours is essentially USELESS.

Offline johnd

  • Forecaster
  • *****
  • Posts: 2288
    • www.weatherstations.co.uk
Re: Weather Data Collection Strategies
« Reply #29 on: March 05, 2017, 04:38:45 AM »
Not.  Like Environment Canada.  Or most forecast models.  Two hours is essentially USELESS.

I think people sometimes forget that Davis stations are bought by a whole spectrum of users. IME hobbyists, as largely represented in forums like this, are only a minority category of user.

Users overall do sometimes have applications where certainly 1-hour and occasionally 2-hour intervals are the only option that can be used. Say you're setting up weather logging as background information for some long-term ecological or environmental project in a relatively inaccessible location. You may be planning to visit the site only eg 2 or 3 times a year (especially if the site cannot easily be reached over the winter months) to do a data download to a laptop so 1-hour (100 days logging) and 2-hour (200 days) are often the only practical options.

And even if you're running a Connect for data connectivity then if you're using a 1-hour data plan (which is fairly common) then that's the granularity of your data, nothing more.

Remember that min/max temperatures, max wind gust etc are all recorded individually within an interval, so even 2-hour logging does generate a reasonably detailed picture of weather conditions over each 24-hour period.
« Last Edit: March 05, 2017, 04:42:10 AM by johnd »
Prodata Weather Systems
Prodata's dedicated Davis EnviroMonitor website
UK Davis Premier Dealer - All Davis stations, accessories and spares
Littleport, Ely, Cambs UK

Offline Old Tele man

  • Singing in the rain...
  • Forecaster
  • *****
  • Posts: 997
Re: Weather Data Collection Strategies
« Reply #30 on: March 05, 2017, 01:05:10 PM »
CLARIFICATION: that 1-hour Davis update interval is for "manual" downloading from the Davis WL site to the Davis WL-software 6.0.3 software program in your computer, it has NOTHING to do with either the upload/send interval nor how fast "other" (outside) programs can fetch or query the Davis WL site for data...I believe.
SYS: Davis VP2/WL-IP & Envoy8X/WL-USB
CWOP: DW6988 - 2 miles NNE of Cortaro, AZ
WU - KAZTUCSO202, Countryside

Offline johnd

  • Forecaster
  • *****
  • Posts: 2288
    • www.weatherstations.co.uk
Re: Weather Data Collection Strategies
« Reply #31 on: March 05, 2017, 01:38:59 PM »
CLARIFICATION: ...

Not 100% sure we're talking about exactly the same thing, but AIUI the context is:

There are three ways of uploading to wl.com: WL v6 subscription; 6555 WLIP logger; Connect;

WL v6 and 6555 logger only upload archive data once per hour. So it's not that you can't try to download archive data more often than hourly, it's more that there's no point in doing so because there won't be any new data to download.

Connect archive data does seem to be available at the plan interval, ie if you have a 5 or 15min plan then new data is there after every upload. (The hourly plan obviously behaves as per WL v6 and 6555 logger.)
Prodata Weather Systems
Prodata's dedicated Davis EnviroMonitor website
UK Davis Premier Dealer - All Davis stations, accessories and spares
Littleport, Ely, Cambs UK

Offline Old Tele man

  • Singing in the rain...
  • Forecaster
  • *****
  • Posts: 997
Re: Weather Data Collection Strategies
« Reply #32 on: March 05, 2017, 04:01:25 PM »
CLARIFICATION: ...

Not 100% sure we're talking about exactly the same thing, but AIUI the context is:

There are three ways of uploading to wl.com: WL v6 subscription; 6555 WLIP logger; Connect;

WL v6 and 6555 logger only upload archive data once per hour. So it's not that you can't try to download archive data more often than hourly, it's more that there's no point in doing so because there won't be any new data to download.

Connect archive data does seem to be available at the plan interval, ie if you have a 5 or 15min plan then new data is there after every upload. (The hourly plan obviously behaves as per WL v6 and 6555 logger.)
So, you're telling me (us) that the WL-IP #6555 logger does NOT send data immediately, but rather "holds" it until it receives a 'control' HOUR-command (from where cometh?) before it sends anything onto the internet?
« Last Edit: March 06, 2017, 09:24:36 AM by Old Tele man »
SYS: Davis VP2/WL-IP & Envoy8X/WL-USB
CWOP: DW6988 - 2 miles NNE of Cortaro, AZ
WU - KAZTUCSO202, Countryside

Offline johnd

  • Forecaster
  • *****
  • Posts: 2288
    • www.weatherstations.co.uk
Re: Weather Data Collection Strategies
« Reply #33 on: March 05, 2017, 04:44:42 PM »
So, you're telling me (us) that the WL-IP #6555 logger does NOT sent data immediately, but rather "holds" it until it receives a 'control' HOUR-command (from where cometh?) before it sends anything onto the internet?

Not exactly: I'm making a distinction between the current conditions packets (ie what you see on the wl.com web pages) which are uploaded every minute but are not - AFAIK - stored anywhere and the archive packets which are uploaded only once per hour and are stored (to a current max of 10240 records) at wl.com. The number of archive packets uploaded each hour will depend on the archive interval set, eg a 10-min interval will result in 6 packets being uploaded each hour.

The trigger for the upload is presumably just the local console time. AFAIK wl.com never talks to the IP logger in the sense of issuing commands (though maybe the individual data packets are acknowledged as part of whichever upload protocol is being used), it's one-way upload traffic only
« Last Edit: March 05, 2017, 04:47:22 PM by johnd »
Prodata Weather Systems
Prodata's dedicated Davis EnviroMonitor website
UK Davis Premier Dealer - All Davis stations, accessories and spares
Littleport, Ely, Cambs UK

Offline Mattk

  • Forecaster
  • *****
  • Posts: 408
Re: Weather Data Collection Strategies
« Reply #34 on: March 06, 2017, 12:58:12 AM »
Normally the 6555 is uploading current conditions within a 60 second window (unless of course other hardware is controlling the WLIP upload interval) to the My Weather & Summary pages. What may not be exactly understood is how the archive packets are handled as it makes little sense to be uploading data on a 60 second basis and then turn around an upload basically the same data (mostly) again for archive records but according to the WLIP summary page it does show a specific time when the archive upload occurred and it would make sense for the user device to crunch the upload numbers instead of wasting server capacity? BTW the WLIP upload data usage is extremely frugal ~ 2mb/day generally less for the standard VP2.

What is well known is the uploaded record data can only be downloaded on an hourly (approx.) basis regardless of the archive interval (be it 1 minute or 10 minutes) and it perhaps is made to do this so as to reduce server load especially if users tried to continually poll 1 minute data.     
« Last Edit: March 06, 2017, 01:06:11 AM by Mattk »

Offline johnd

  • Forecaster
  • *****
  • Posts: 2288
    • www.weatherstations.co.uk
Re: Weather Data Collection Strategies
« Reply #35 on: March 06, 2017, 03:01:04 AM »
What is well known is the uploaded record data can only be downloaded on an hourly (approx.) basis regardless of the archive interval (be it 1 minute or 10 minutes) and it perhaps is made to do this so as to reduce server load especially if users tried to continually poll 1 minute data.     

This isn't the right way of looking at it - see my posts above for the Davis explanation. Since _archive_ data is only uploaded once per hour, there is no point in trying to download more than once per hour - any attempts to do so are bound to fail because there is no new data there!
Prodata Weather Systems
Prodata's dedicated Davis EnviroMonitor website
UK Davis Premier Dealer - All Davis stations, accessories and spares
Littleport, Ely, Cambs UK

Offline Mattk

  • Forecaster
  • *****
  • Posts: 408
Re: Weather Data Collection Strategies
« Reply #36 on: March 06, 2017, 04:09:50 AM »
What is well known is the uploaded record data can only be downloaded on an hourly (approx.) basis regardless of the archive interval (be it 1 minute or 10 minutes) and it perhaps is made to do this so as to reduce server load especially if users tried to continually poll 1 minute data.     

This isn't the right way of looking at it - see my posts above for the Davis explanation. Since _archive_ data is only uploaded once per hour, there is no point in trying to download more than once per hour - any attempts to do so are bound to fail because there is no new data there!

And when you think about it that upload interval is part and parcel of the WLIP firmware and Davis would do that for some reason and one reason is to smooth out server activity and minimize CPU activity.

And yes there is no new data under the hour to download as it hasn't been uploaded yet, so the primary controlling device would appear to be the WLIP.

The other thing to think about is any email data which has to be up to date to the nearest 30 minutes regardless but this is probably just a function of the Summary page data? 

Offline Mattk

  • Forecaster
  • *****
  • Posts: 408
Re: Weather Data Collection Strategies
« Reply #37 on: March 08, 2017, 04:58:43 AM »
Just to add a bit more to this and being a little curious I run a WLIP test rig at various archive intervals 1 min, 5 min, 10min etc which as expected conform to the hourly upload sequence but then ran a 120 min archive in which the WLIP did an upload cycle (as shown by WeatherLink.com) an hour latter but there was no data, the next hourly interval produced 1 page / 1 interval (2 hourly) at which time the WLIP must have determined the interval was 120 minutes so from this point on only fired an upload every second hour so there a little bit of intelligence going on.

The other thing a 120 minute interval highlighted is that since the archive interval fall on even time scales the upload and subsequently the download can be up to nearly an hour behind the actual interval recorded.

       

 

anything