Author Topic: Reading data locally from WLL  (Read 2367 times)

0 Members and 1 Guest are viewing this topic.

Offline johnd

  • Forecaster
  • *****
  • Posts: 3143
    • www.weatherstations.co.uk
Re: Reading data locally from WLL
« Reply #75 on: May 16, 2019, 03:29:37 AM »
...and locally the device doesn't store historical data at all...

Yes, we had been speculating about that. One possibility is that the per-minute current conditions messages are simply queued in the event of a network outage and that effectively becomes the local archive data. (But that's just one speculation and might easily be wrong.)

Could be checked I guess by eg connecting WLL via cabled Ethernet, pulling the network plug for a while and then reestablishing contact and examining the data.
Prodata Weather Systems
Prodata's dedicated Davis EnviroMonitor website
UK Davis Premier Dealer - All Davis stations, accessories and spares
Littleport, Ely, Cambs UK

Offline Brientim

  • Senior Member
  • **
  • Posts: 65
Re: Reading data locally from WLL
« Reply #76 on: May 16, 2019, 03:58:22 AM »
As per the specifications the defaults archiving is 15 minutes but this can be modified to 2 hours (120 minutes) but it does not provide any explanation on storage capabilities.
« Last Edit: May 16, 2019, 05:37:22 AM by Brientim »

Offline johnd

  • Forecaster
  • *****
  • Posts: 3143
    • www.weatherstations.co.uk
Re: Reading data locally from WLL
« Reply #77 on: May 16, 2019, 06:39:41 AM »
As per the specifications the defaults archiving is 15 minutes but this can be modified to 2 hours (120 minutes) but it does not provide any explanation on storage capabilities.

And of course down to 5 mins or 1 min if you have a Pro or Pro Plus plan.

But just to be clear, I'm not suggesting that the WLL unit may not store archive data as we know it (albeit presumably in JSON rather than binary form) and indeed I really hope this is the case (and also that Davis allow local access to it even in the absence of a Pro plan). I'm simply speculating that this is not the only conceivable solution - archiving the data, ie chunking into eg 15min summary records could in theory be done solely at weatherlink.com.
Prodata Weather Systems
Prodata's dedicated Davis EnviroMonitor website
UK Davis Premier Dealer - All Davis stations, accessories and spares
Littleport, Ely, Cambs UK

Offline Brientim

  • Senior Member
  • **
  • Posts: 65
Reading data locally from WLL
« Reply #78 on: May 16, 2019, 04:41:29 PM »
Here is a response from Davis

“WeatherLink Live can store about 180 days of data, assuming is is listening to just one sensor suite and has a 15-minute archive interval. More transmitters and faster archive intervals would decrease this.”

This doesn’t answer the specifics of the questions especially access to the stored data
« Last Edit: May 16, 2019, 04:44:12 PM by Brientim »

Offline kobuki

  • Forecaster
  • *****
  • Posts: 726
Re: Reading data locally from WLL
« Reply #79 on: May 16, 2019, 04:51:41 PM »
This doesn’t answer the specifics of the questions especially access to the stored data
This might indeed indicate that its internal storage is just a buffer for cases of internet connection problems. It doesn't necessarily have a facility to actually access that buffered data (it would be nice, though). I think it would be in the docs linked earlier if there was. But to be honest, local access of real-time data and current conditions is likely only there to support 3rd-party applications that do have their database for storing historical data.

Offline docbee

  • Forecaster
  • *****
  • Posts: 711
    • smartbedded
Re: Reading data locally from WLL
« Reply #80 on: May 18, 2019, 04:10:16 AM »
I am still looking for data snippets that contain leaf/soil data. Does anybody out there have that connected to a WLL and is willing to post a data set here? I can help to have it generated.
founder of smartbedded.com - home of meteohub, meteoplug, meteobridge, meteostick

Offline johnd

  • Forecaster
  • *****
  • Posts: 3143
    • www.weatherstations.co.uk
Re: Reading data locally from WLL
« Reply #81 on: May 18, 2019, 04:19:04 AM »
I am still looking for data snippets that contain leaf/soil data.

Doesn't message Type 2 in https://weatherlink.github.io/weatherlink-live-local-api/ show you what you need? I guess all the sensor values shown are null - is that the problem?
« Last Edit: May 18, 2019, 04:22:31 AM by johnd »
Prodata Weather Systems
Prodata's dedicated Davis EnviroMonitor website
UK Davis Premier Dealer - All Davis stations, accessories and spares
Littleport, Ely, Cambs UK

Offline docbee

  • Forecaster
  • *****
  • Posts: 711
    • smartbedded
Re: Reading data locally from WLL
« Reply #82 on: May 18, 2019, 04:46:58 AM »
To have some samples would be nice to test the code.
founder of smartbedded.com - home of meteohub, meteoplug, meteobridge, meteostick

Offline Brientim

  • Senior Member
  • **
  • Posts: 65
Re: Reading data locally from WLL
« Reply #83 on: May 18, 2019, 04:50:10 AM »
To have some samples would be nice to test the code.

Code: [Select]
{"data":{"did":"001D00000000","ts":1558169339,"conditions":[{"lsid":227174,"data_structure_type":1,"txid":1,"temp": 56.1,"hum":80.4,"dew_point": 50.1,"wet_bulb": 52.4,"heat_index": 56.1,"wind_chill": 56.1,"thw_index": 56.1,"thsw_index": 54.1,"wind_speed_last":0.00,"wind_dir_last":0,"wind_speed_avg_last_1_min":0.00,"wind_dir_scalar_avg_last_1_min":null,"wind_speed_avg_last_2_min":0.00,"wind_dir_scalar_avg_last_2_min":null,"wind_speed_hi_last_2_min":0.00,"wind_dir_at_hi_speed_last_2_min":0,"wind_speed_avg_last_10_min":0.00,"wind_dir_scalar_avg_last_10_min":null,"wind_speed_hi_last_10_min":0.00,"wind_dir_at_hi_speed_last_10_min":0,"rain_size":2,"rain_rate_last":0,"rain_rate_hi":0,"rainfall_last_15_min":0,"rain_rate_hi_last_15_min":0,"rainfall_last_60_min":0,"rainfall_last_24_hr":0,"rain_storm":null,"rain_storm_start_at":null,"solar_rad":0,"uv_index":0.0,"rx_state":0,"trans_battery_flag":0,"rainfall_daily":0,"rainfall_monthly":0,"rainfall_year":0,"rain_storm_last":null,"rain_storm_last_start_at":null,"rain_storm_last_end_at":null},{"lsid":227177,"data_structure_type":2,"txid":4,"temp_1": 52.8,"temp_2": 51.2,"temp_3":null,"temp_4":null,"moist_soil_1":13.8,"moist_soil_2":32.8,"moist_soil_3":null,"moist_soil_4":null,"wet_leaf_1":0.6,"wet_leaf_2":null,"rx_state":0,"trans_battery_flag":0},{"lsid":227173,"data_structure_type":4,"temp_in": 75.6,"hum_in":38.7,"dew_point_in": 48.7,"heat_index_in": 74.5},{"lsid":227172,"data_structure_type":3,"bar_sea_level":30.392,"bar_trend": 0.043,"bar_absolute":30.289}]},"error":null}

Offline kobuki

  • Forecaster
  • *****
  • Posts: 726
Re: Reading data locally from WLL
« Reply #84 on: May 18, 2019, 06:11:55 AM »
Brientim's sample in a readable format:
Code: [Select]
{
    "data":{
        "did":"001D00000000",
        "ts":1558169339,
        "conditions":[
            {
                "lsid":227174,
                "data_structure_type":1,
                "txid":1,
                "temp":56.1,
                "hum":80.4,
                "dew_point":50.1,
                "wet_bulb":52.4,
                "heat_index":56.1,
                "wind_chill":56.1,
                "thw_index":56.1,
                "thsw_index":54.1,
                "wind_speed_last":0.00,
                "wind_dir_last":0,
                "wind_speed_avg_last_1_min":0.00,
                "wind_dir_scalar_avg_last_1_min":null,
                "wind_speed_avg_last_2_min":0.00,
                "wind_dir_scalar_avg_last_2_min":null,
                "wind_speed_hi_last_2_min":0.00,
                "wind_dir_at_hi_speed_last_2_min":0,
                "wind_speed_avg_last_10_min":0.00,
                "wind_dir_scalar_avg_last_10_min":null,
                "wind_speed_hi_last_10_min":0.00,
                "wind_dir_at_hi_speed_last_10_min":0,
                "rain_size":2,
                "rain_rate_last":0,
                "rain_rate_hi":0,
                "rainfall_last_15_min":0,
                "rain_rate_hi_last_15_min":0,
                "rainfall_last_60_min":0,
                "rainfall_last_24_hr":0,
                "rain_storm":null,
                "rain_storm_start_at":null,
                "solar_rad":0,
                "uv_index":0.0,
                "rx_state":0,
                "trans_battery_flag":0,
                "rainfall_daily":0,
                "rainfall_monthly":0,
                "rainfall_year":0,
                "rain_storm_last":null,
                "rain_storm_last_start_at":null,
                "rain_storm_last_end_at":null
            },
            {
                "lsid":227177,
                "data_structure_type":2,
                "txid":4,
                "temp_1":52.8,
                "temp_2":51.2,
                "temp_3":null,
                "temp_4":null,
                "moist_soil_1":13.8,
                "moist_soil_2":32.8,
                "moist_soil_3":null,
                "moist_soil_4":null,
                "wet_leaf_1":0.6,
                "wet_leaf_2":null,
                "rx_state":0,
                "trans_battery_flag":0
            },
            {
                "lsid":227173,
                "data_structure_type":4,
                "temp_in":75.6,
                "hum_in":38.7,
                "dew_point_in":48.7,
                "heat_index_in":74.5
            },
            {
                "lsid":227172,
                "data_structure_type":3,
                "bar_sea_level":30.392,
                "bar_trend":0.043,
                "bar_absolute":30.289
            }
        ]
    },
    "error":null
}

Offline docbee

  • Forecaster
  • *****
  • Posts: 711
    • smartbedded
Re: Reading data locally from WLL
« Reply #85 on: May 18, 2019, 08:23:17 AM »
Thanks, this helped to test the code. Support for record type 1 is now also included to the Meteobridge WLL driver.
founder of smartbedded.com - home of meteohub, meteoplug, meteobridge, meteostick

Offline mcrossley

  • Forecaster
  • *****
  • Posts: 423
    • Wilmslow Astro
Re: Reading data locally from WLL
« Reply #86 on: May 21, 2019, 03:36:32 PM »
I'm having a few issues with this access. I whipped up a small .Net 4.7 app to test out the data acquisition.

I can receive the broadcast data fine, but I'm having problems with the Realtime and Current data HTTP fetches.

The .Net httpClient aborts with an exception...
Message :System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.WebException: The server committed a protocol violation. Section=ResponseStatusLine

Now if it put a "normal" web site in the URL instead of the WLL the code works fine, even fetching JSON data from a web server.

Any ideas?
Mark

Offline kobuki

  • Forecaster
  • *****
  • Posts: 726
Re: Reading data locally from WLL
« Reply #87 on: May 21, 2019, 03:51:57 PM »
Please dump a request/response pair using curl, or even better, a wireshark capture. It's impossible to say what kind of violation the .net library thinks happened on the status line.

Offline waiukuweather

  • Forecaster
  • *****
  • Posts: 688
Re: Reading data locally from WLL
« Reply #88 on: May 21, 2019, 03:52:37 PM »
I had the same problem with .NET HTTPClient and so had to use another HTTP library etc

Offline mcrossley

  • Forecaster
  • *****
  • Posts: 423
    • Wilmslow Astro
Re: Reading data locally from WLL
« Reply #89 on: May 21, 2019, 05:09:53 PM »
Please dump a request/response pair using curl, or even better, a wireshark capture. It's impossible to say what kind of violation the .net library thinks happened on the status line.
I have a WS capture using Chrome browser, one using my util is on the way (I don't have a WLL so debugging via email at the mo). The capture does not show anything obviously out of sorts to me, but the WLL is must be cutting a corner somewhere? [ You are not allowed to view attachments ]

I had the same problem with .NET HTTPClient and so had to use another HTTP library etc
Thanks, I have tried both HttpClient and the older WebClient, both give a protocol error with the WLL, but work with every other URL I try.
Mark

Offline kobuki

  • Forecaster
  • *****
  • Posts: 726
Re: Reading data locally from WLL
« Reply #90 on: May 21, 2019, 05:53:07 PM »
The only obvious thing I can see is the line terminating characters are all over the place in the response. It's LFCR after the status line, then LFLF after the header part, but it must always be one CRLF after the status line, one CRLF after each header line, and after the last header line, CRLF 2 times (right before the response body). The .net lib could be a little more forgiving, but it errs out absolutely rightfully. I suggest parsing the response body using your own function...

Offline mcrossley

  • Forecaster
  • *****
  • Posts: 423
    • Wilmslow Astro
Re: Reading data locally from WLL
« Reply #91 on: May 21, 2019, 06:12:18 PM »
Thanks, I was looking for TCP type errors, and didn't look too closely at the http content!
Mark

Offline mcrossley

  • Forecaster
  • *****
  • Posts: 423
    • Wilmslow Astro
Re: Reading data locally from WLL
« Reply #92 on: May 22, 2019, 05:07:10 AM »
So my interpretation of this is as follows...

The WLL is actually sending:
Code: [Select]
HTTP/1.1 200 OK\0a\0d
content-type: application/json\0a\0a

{"data":{"broadcast_port":22222,"duration":1200},"error":null}\0a\0a\0d

When according to the HTTP standards it should be sending:
Code: [Select]
HTTP/1.1 200 OK\0d\0a
content-type: application/json\0d\0a\0d\0a

{"data":{"broadcast_port":22222,"duration":1200},"error":null}\0d\0a

Which means the interpreter thinks the body is still in the header - or some other mess!
« Last Edit: May 22, 2019, 05:12:10 AM by mcrossley »
Mark

Offline kobuki

  • Forecaster
  • *****
  • Posts: 726
Re: Reading data locally from WLL
« Reply #93 on: May 22, 2019, 05:26:59 AM »
So my interpretation of this is as follows...

The WLL is actually sending:
Code: [Select]
HTTP/1.1 200 OK\0a\0d
content-type: application/json\0a\0a

{"data":{"broadcast_port":22222,"duration":1200},"error":null}\0a\0a\0d

When according to the HTTP standards it should be sending:
Code: [Select]
HTTP/1.1 200 OK\0d\0a
content-type: application/json\0d\0a\0d\0a

{"data":{"broadcast_port":22222,"duration":1200},"error":null}\0d\0a

Which means the interpreter thinks the body is still in the header - or some other mess!
Isn't this the same thing I said above in other words? The RFC refers to \0d as CR and \0a as LF, and \0d\0a as CRLF.

Offline mcrossley

  • Forecaster
  • *****
  • Posts: 423
    • Wilmslow Astro
Re: Reading data locally from WLL
« Reply #94 on: May 22, 2019, 05:38:49 AM »
I hope so, I looked at the WS trace and codified what you had said in words. I was looking for confirmation it is correct before trying to get Davis involved in fixing it.
Mark

Offline kobuki

  • Forecaster
  • *****
  • Posts: 726
Re: Reading data locally from WLL
« Reply #95 on: May 22, 2019, 05:40:47 AM »
I hope so, I looked at the WS trace and codified what you had said in words. I was looking for confirmation it is correct before trying to get Davis involved in fixing it.
Please refer them to https://www.ietf.org/rfc/rfc2616.txt

BTW my question was a rhetorical one...

Offline havtrail

  • Contributor
  • ***
  • Posts: 110
    • Haverford Weather Station
Re: Reading data locally from WLL
« Reply #96 on: May 22, 2019, 11:12:29 AM »
We need a rhetorical smiley!  :-)

Rich K.
Onset HOBO RX3003 Cellular
https://weather.havtrail.com
WU KPAHAVER17

 

anything