WXforum.net

Weather Station Hardware => Davis Instruments Weather Stations => Topic started by: docbee on May 04, 2019, 05:17:03 AM

Title: Reading data locally from WLL
Post by: docbee on May 04, 2019, 05:17:03 AM
While monitoring the TCP traffic of the WLL it is quite obvious how to get data from it. Please lets share information about that in this thread. As there is no formal API definition from Davis yet, we might do one preliminary for ourselves instead of waiting.

It is plain HTTP GET requests, so no magic tricks from Davis at this point.

As I don't have sensors connected yet it returns an error message when doing a request on "/v1/real_time":

Code: [Select]
mars:~ # curl 192.168.123.232/v1/real_time

{"data":null,"error":{"code":409,"message":"No ISS Transmitters.  Real Time broadcast not enabled"}}

More interesting is asking for current conditions:

Code: [Select]
mars:~ # curl 192.168.123.232/v1/current_conditions

{"data":{"did":"XXXXXXXXXXXX","ts":1556961293,"conditions":[{"lsid":225156,"data_structure_type":4,"temp_in": 77.2,"hum_in":34.9,"dew_point_in": 47.4,"heat_index_in": 75.7},{"lsid":225155,"data_structure_type":3,"bar_sea_level":29.596,"bar_trend":null,"bar_absolute":29.585}]},"error":null}

All rather straight forward JSON.  :grin:

Can you please also post example replies here, preferably with some larger arrays connected?
Title: Re: Reading data locally from WLL
Post by: waiukuweather on May 04, 2019, 05:34:33 AM
interesting
good work
I would presume its not a static IP address?
so how does the Davis app get around the problem of when the IP address is changed? (e.g if the WLL is reset?)
Title: Re: Reading data locally from WLL
Post by: kobuki on May 04, 2019, 06:42:59 AM
It is plain HTTP GET requests, so no magic tricks from Davis at this point.
That's exactly what I thought after someone mentioned in the other general thread that probing the WLL with a browser, a JSON error string is returned :)  I was 100% sure someone will take the effort and sniff that traffic. Nice, Boris. I guess you're eager to support this device, too.
Title: Re: Reading data locally from WLL
Post by: kobuki on May 04, 2019, 06:45:34 AM
so how does the Davis app get around the problem of when the IP address is changed? (e.g if the WLL is reset?)
There are standard ways for that like STUN, some kind of broadcast, client app probing the local subnet, mDNS, or simply the WLL reports its local IP to the cloud and that passes on the info to the app. I'm sure I could quickly invent some more.
Title: Re: Reading data locally from WLL
Post by: docbee on May 04, 2019, 07:52:50 AM
I guess you're eager to support this device, too.

Yes, of course as the ones not wanting to be locked-in to Davis and like to do with their weather data what, when and how they like are my target market.

As mentioned before, it will speed up things, if users of WLL with a larger sensor set will post their data here. So, please gimme and the other developers listening here data to work with.  :grin:
Title: Re: Reading data locally from WLL
Post by: kobuki on May 04, 2019, 08:01:17 AM
So, please gimme and the other developers listening here data to work with.  :grin:
I would of course but my little paradox is I don't want to buy one until I see I can freely access the data it's spitting out. I wonder how the real time update of eg. the wind data is transmitted.
Title: Re: Reading data locally from WLL
Post by: johnd on May 04, 2019, 08:25:17 AM
Might be useful to understand/document the trigger for the UDP broadcasts.
Title: Re: Reading data locally from WLL
Post by: miraculon on May 04, 2019, 08:30:02 AM
I just pasted the URL for my WLL into the Edge browser (Firefox shows it briefly and then gives a "connection was reset" error). I found the IP address using AngryIP. There is a Hostname of WeatherLinkLIVE.localdomain listed by AngryIP. It works in the browser with v1/current_conditions appended. This responds with the same result as the IP address: http://weatherlinklive.localdomain/v1/current_conditions (http://weatherlinklive.localdomain/v1/current_conditions)

My WLL shows this for "current_conditions":

Code: [Select]
{"data":{"did":"001D0_REDACTED","ts":1556972192,"conditions":[{"lsid":224114,"data_structure_type":1,"txid":2,"temp":-99.0,"hum":null,"dew_point":null,"wet_bulb":null,"heat_index":null,"wind_chill":-107.8,"thw_index":null,"thsw_index":null,"wind_speed_last":1.00,"wind_dir_last":288,"wind_speed_avg_last_1_min":0.06,"wind_dir_scalar_avg_last_1_min":288,"wind_speed_avg_last_2_min":0.50,"wind_dir_scalar_avg_last_2_min":288,"wind_speed_hi_last_2_min":1.00,"wind_dir_at_hi_speed_last_2_min":288,"wind_speed_avg_last_10_min":1.37,"wind_dir_scalar_avg_last_10_min":286,"wind_speed_hi_last_10_min":4.00,"wind_dir_at_hi_speed_last_10_min":287,"rain_size":1,"rain_rate_last":0,"rain_rate_hi":0,"rainfall_last_15_min":0,"rain_rate_hi_last_15_min":0,"rainfall_last_60_min":0,"rainfall_last_24_hr":0,"rain_storm":null,"rain_storm_start_at":null,"solar_rad":1782,"uv_index":null,"rx_state":0,"trans_battery_flag":0,"rainfall_daily":0,"rainfall_monthly":0,"rainfall_year":0,"rain_storm_last":null,"rain_storm_last_start_at":null,"rain_storm_last_end_at":null},{"lsid":224115,"data_structure_type":1,"txid":3,"temp": 44.1,"hum":72.2,"dew_point": 35.7,"wet_bulb": 39.7,"heat_index": 44.1,"wind_chill":null,"thw_index":null,"thsw_index":null,"wind_speed_last":null,"wind_dir_last":null,"wind_speed_avg_last_1_min":null,"wind_dir_scalar_avg_last_1_min":null,"wind_speed_avg_last_2_min":null,"wind_dir_scalar_avg_last_2_min":null,"wind_speed_hi_last_2_min":null,"wind_dir_at_hi_speed_last_2_min":null,"wind_speed_avg_last_10_min":null,"wind_dir_scalar_avg_last_10_min":null,"wind_speed_hi_last_10_min":null,"wind_dir_at_hi_speed_last_10_min":null,"rain_size":1,"rain_rate_last":0,"rain_rate_hi":0,"rainfall_last_15_min":0,"rain_rate_hi_last_15_min":0,"rainfall_last_60_min":0,"rainfall_last_24_hr":0,"rain_storm":0,"rain_storm_start_at":null,"solar_rad":null,"uv_index":null,"rx_state":0,"trans_battery_flag":0,"rainfall_daily":0,"rainfall_monthly":847,"rainfall_year":1037,"rain_storm_last":847,"rain_storm_last_start_at":1556706240,"rain_storm_last_end_at":1556924461},{"lsid":224116,"data_structure_type":1,"txid":4,"temp": 46.7,"hum":null,"dew_point":null,"wet_bulb":null,"heat_index":null,"wind_chill": 46.7,"thw_index":null,"thsw_index":null,"wind_speed_last":1.00,"wind_dir_last":24,"wind_speed_avg_last_1_min":0.25,"wind_dir_scalar_avg_last_1_min":180,"wind_speed_avg_last_2_min":0.62,"wind_dir_scalar_avg_last_2_min":217,"wind_speed_hi_last_2_min":2.00,"wind_dir_at_hi_speed_last_2_min":272,"wind_speed_avg_last_10_min":1.37,"wind_dir_scalar_avg_last_10_min":265,"wind_speed_hi_last_10_min":4.00,"wind_dir_at_hi_speed_last_10_min":270,"rain_size":1,"rain_rate_last":0,"rain_rate_hi":0,"rainfall_last_15_min":0,"rain_rate_hi_last_15_min":0,"rainfall_last_60_min":0,"rainfall_last_24_hr":0,"rain_storm":null,"rain_storm_start_at":null,"solar_rad":1731,"uv_index":16.0,"rx_state":0,"trans_battery_flag":0,"rainfall_daily":0,"rainfall_monthly":0,"rainfall_year":0,"rain_storm_last":null,"rain_storm_last_start_at":null,"rain_storm_last_end_at":null},{"lsid":224117,"data_structure_type":1,"txid":5,"temp": 42.3,"hum":80.6,"dew_point": 36.8,"wet_bulb": 39.5,"heat_index": 42.3,"wind_chill": 42.3,"thw_index": 42.3,"thsw_index": 40.3,"wind_speed_last":0.00,"wind_dir_last":0,"wind_speed_avg_last_1_min":0.00,"wind_dir_scalar_avg_last_1_min":null,"wind_speed_avg_last_2_min":0.00,"wind_dir_scalar_avg_last_2_min":null,"wind_speed_hi_last_2_min":null,"wind_dir_at_hi_speed_last_2_min":null,"wind_speed_avg_last_10_min":0.06,"wind_dir_scalar_avg_last_10_min":null,"wind_speed_hi_last_10_min":1.00,"wind_dir_at_hi_speed_last_10_min":351,"rain_size":1,"rain_rate_last":0,"rain_rate_hi":0,"rainfall_last_15_min":0,"rain_rate_hi_last_15_min":0,"rainfall_last_60_min":0,"rainfall_last_24_hr":0,"rain_storm":0,"rain_storm_start_at":null,"solar_rad":309,"uv_index":0.8,"rx_state":0,"trans_battery_flag":0,"rainfall_daily":0,"rainfall_monthly":90,"rainfall_year":95,"rain_storm_last":3,"rain_storm_last_start_at":1556834760,"rain_storm_last_end_at":1556924461},{"lsid":224118,"data_structure_type":1,"txid":6,"temp": 44.8,"hum":null,"dew_point":null,"wet_bulb":null,"heat_index":null,"wind_chill":null,"thw_index":null,"thsw_index":null,"wind_speed_last":null,"wind_dir_last":null,"wind_speed_avg_last_1_min":null,"wind_dir_scalar_avg_last_1_min":null,"wind_speed_avg_last_2_min":null,"wind_dir_scalar_avg_last_2_min":null,"wind_speed_hi_last_2_min":null,"wind_dir_at_hi_speed_last_2_min":null,"wind_speed_avg_last_10_min":null,"wind_dir_scalar_avg_last_10_min":null,"wind_speed_hi_last_10_min":null,"wind_dir_at_hi_speed_last_10_min":null,"rain_size":1,"rain_rate_last":0,"rain_rate_hi":0,"rainfall_last_15_min":0,"rain_rate_hi_last_15_min":0,"rainfall_last_60_min":0,"rainfall_last_24_hr":0,"rain_storm":0,"rain_storm_start_at":null,"solar_rad":null,"uv_index":null,"rx_state":0,"trans_battery_flag":0,"rainfall_daily":0,"rainfall_monthly":106,"rainfall_year":113,"rain_storm_last":106,"rain_storm_last_start_at":1556709480,"rain_storm_last_end_at":1556924461},{"lsid":224119,"data_structure_type":1,"txid":7,"temp": 39.9,"hum":null,"dew_point":null,"wet_bulb":null,"heat_index":null,"wind_chill":null,"thw_index":null,"thsw_index":null,"wind_speed_last":null,"wind_dir_last":null,"wind_speed_avg_last_1_min":null,"wind_dir_scalar_avg_last_1_min":null,"wind_speed_avg_last_2_min":null,"wind_dir_scalar_avg_last_2_min":null,"wind_speed_hi_last_2_min":null,"wind_dir_at_hi_speed_last_2_min":null,"wind_speed_avg_last_10_min":null,"wind_dir_scalar_avg_last_10_min":null,"wind_speed_hi_last_10_min":null,"wind_dir_at_hi_speed_last_10_min":null,"rain_size":1,"rain_rate_last":0,"rain_rate_hi":0,"rainfall_last_15_min":0,"rain_rate_hi_last_15_min":0,"rainfall_last_60_min":0,"rainfall_last_24_hr":0,"rain_storm":null,"rain_storm_start_at":null,"solar_rad":null,"uv_index":null,"rx_state":0,"trans_battery_flag":0,"rainfall_daily":0,"rainfall_monthly":0,"rainfall_year":0,"rain_storm_last":null,"rain_storm_last_start_at":null,"rain_storm_last_end_at":null},{"lsid":224112,"data_structure_type":4,"temp_in": 65.1,"hum_in":44.9,"dew_point_in": 43.2,"heat_index_in": 62.9},{"lsid":224111,"data_structure_type":3,"bar_sea_level":30.125,"bar_trend": 0.033,"bar_absolute":29.472}]},"error":null}
and real_time:

Code: [Select]
{"data":{"broadcast_port":22222,"duration":1200},"error":null}
Greg H.
Title: Re: Reading data locally from WLL
Post by: docbee on May 04, 2019, 09:00:18 AM
@miraculon: Looks like you have
on TX #2 a wind plus solar station
on TX #3 a temp/hum plus rain station
on TX #4 a wind plus UV and solar station
on TX #5 a ISS with UV and solar
on TX #6 a temp/hum plus rain station
on TX #7 a temp station

Is that right? Existing data does not seem to match available Davis stations types perfectly.
Station identifier (telling what kind of station it is) seems not to be in the data. May be the user has to define that within the SW and SW just grabs the data pieces from JSON that make sense to the selected sensor type... still guessing
Title: Re: Reading data locally from WLL
Post by: kobuki on May 04, 2019, 09:00:49 AM
Might be useful to understand/document the trigger for the UDP broadcasts.
Are you sure there's such a thing? That UDP broadcast type thing is for a different company's unique custom solution.
Title: Re: Reading data locally from WLL
Post by: johnd on May 04, 2019, 09:04:47 AM
Are you sure there's such a thing?

Yes.
Title: Re: Reading data locally from WLL
Post by: kobuki on May 04, 2019, 09:05:21 AM
...
My WLL shows this for "current_conditions":
...
That's a lot of data you got there. Hint: I pasted your data to http://json.parser.online.fr/ (http://json.parser.online.fr/) to have a better look.
Title: Re: Reading data locally from WLL
Post by: kobuki on May 04, 2019, 09:07:09 AM
Are you sure there's such a thing?
Yes.
And what do those broadcast messages carry? All we're aware of are the json messages, no mention of any UDP thing. Could you please point me to the source of that information? Maybe I have missed an important bit here.

@docbee, what are you seeing? Have you encountered those UDP messages?
Title: Re: Reading data locally from WLL
Post by: docbee on May 04, 2019, 09:08:15 AM
Data from Greg in a better format...

Code: [Select]
{

    "data":{
        "did":"XXXXXXXXXX",
        "ts":1556972192,
        "conditions":[
            {
                "lsid":224114,
                "data_structure_type":1,
                "txid":2,
                "temp":-99.0,
                "hum":null,
                "dew_point":null,
                "wet_bulb":null,
                "heat_index":null,
                "wind_chill":-107.8,
                "thw_index":null,
                "thsw_index":null,
                "wind_speed_last":1.00,
                "wind_dir_last":288,
                "wind_speed_avg_last_1_min":0.06,
                "wind_dir_scalar_avg_last_1_min":288,
                "wind_speed_avg_last_2_min":0.50,
                "wind_dir_scalar_avg_last_2_min":288,
                "wind_speed_hi_last_2_min":1.00,
                "wind_dir_at_hi_speed_last_2_min":288,
                "wind_speed_avg_last_10_min":1.37,
                "wind_dir_scalar_avg_last_10_min":286,
                "wind_speed_hi_last_10_min":4.00,
                "wind_dir_at_hi_speed_last_10_min":287,
                "rain_size":1,
                "rain_rate_last":0,
                "rain_rate_hi":0,
                "rainfall_last_15_min":0,
                "rain_rate_hi_last_15_min":0,
                "rainfall_last_60_min":0,
                "rainfall_last_24_hr":0,
                "rain_storm":null,
                "rain_storm_start_at":null,
                "solar_rad":1782,
                "uv_index":null,
                "rx_state":0,
                "trans_battery_flag":0,
                "rainfall_daily":0,
                "rainfall_monthly":0,
                "rainfall_year":0,
                "rain_storm_last":null,
                "rain_storm_last_start_at":null,
                "rain_storm_last_end_at":null
            },
            {
                "lsid":224115,
                "data_structure_type":1,
                "txid":3,
                "temp":44.1,
                "hum":72.2,
                "dew_point":35.7,
                "wet_bulb":39.7,
                "heat_index":44.1,
                "wind_chill":null,
                "thw_index":null,
                "thsw_index":null,
                "wind_speed_last":null,
                "wind_dir_last":null,
                "wind_speed_avg_last_1_min":null,
                "wind_dir_scalar_avg_last_1_min":null,
                "wind_speed_avg_last_2_min":null,
                "wind_dir_scalar_avg_last_2_min":null,
                "wind_speed_hi_last_2_min":null,
                "wind_dir_at_hi_speed_last_2_min":null,
                "wind_speed_avg_last_10_min":null,
                "wind_dir_scalar_avg_last_10_min":null,
                "wind_speed_hi_last_10_min":null,
                "wind_dir_at_hi_speed_last_10_min":null,
                "rain_size":1,
                "rain_rate_last":0,
                "rain_rate_hi":0,
                "rainfall_last_15_min":0,
                "rain_rate_hi_last_15_min":0,
                "rainfall_last_60_min":0,
                "rainfall_last_24_hr":0,
                "rain_storm":0,
                "rain_storm_start_at":null,
                "solar_rad":null,
                "uv_index":null,
                "rx_state":0,
                "trans_battery_flag":0,
                "rainfall_daily":0,
                "rainfall_monthly":847,
                "rainfall_year":1037,
                "rain_storm_last":847,
                "rain_storm_last_start_at":1556706240,
                "rain_storm_last_end_at":1556924461
            },
            {
                "lsid":224116,
                "data_structure_type":1,
                "txid":4,
                "temp":46.7,
                "hum":null,
                "dew_point":null,
                "wet_bulb":null,
                "heat_index":null,
                "wind_chill":46.7,
                "thw_index":null,
                "thsw_index":null,
                "wind_speed_last":1.00,
                "wind_dir_last":24,
                "wind_speed_avg_last_1_min":0.25,
                "wind_dir_scalar_avg_last_1_min":180,
                "wind_speed_avg_last_2_min":0.62,
                "wind_dir_scalar_avg_last_2_min":217,
                "wind_speed_hi_last_2_min":2.00,
                "wind_dir_at_hi_speed_last_2_min":272,
                "wind_speed_avg_last_10_min":1.37,
                "wind_dir_scalar_avg_last_10_min":265,
                "wind_speed_hi_last_10_min":4.00,
                "wind_dir_at_hi_speed_last_10_min":270,
                "rain_size":1,
                "rain_rate_last":0,
                "rain_rate_hi":0,
                "rainfall_last_15_min":0,
                "rain_rate_hi_last_15_min":0,
                "rainfall_last_60_min":0,
                "rainfall_last_24_hr":0,
                "rain_storm":null,
                "rain_storm_start_at":null,
                "solar_rad":1731,
                "uv_index":16.0,
                "rx_state":0,
                "trans_battery_flag":0,
                "rainfall_daily":0,
                "rainfall_monthly":0,
                "rainfall_year":0,
                "rain_storm_last":null,
                "rain_storm_last_start_at":null,
                "rain_storm_last_end_at":null
            },
            {
                "lsid":224117,
                "data_structure_type":1,
                "txid":5,
                "temp":42.3,
                "hum":80.6,
                "dew_point":36.8,
                "wet_bulb":39.5,
                "heat_index":42.3,
                "wind_chill":42.3,
                "thw_index":42.3,
                "thsw_index":40.3,
                "wind_speed_last":0.00,
                "wind_dir_last":0,
                "wind_speed_avg_last_1_min":0.00,
                "wind_dir_scalar_avg_last_1_min":null,
                "wind_speed_avg_last_2_min":0.00,
                "wind_dir_scalar_avg_last_2_min":null,
                "wind_speed_hi_last_2_min":null,
                "wind_dir_at_hi_speed_last_2_min":null,
                "wind_speed_avg_last_10_min":0.06,
                "wind_dir_scalar_avg_last_10_min":null,
                "wind_speed_hi_last_10_min":1.00,
                "wind_dir_at_hi_speed_last_10_min":351,
                "rain_size":1,
                "rain_rate_last":0,
                "rain_rate_hi":0,
                "rainfall_last_15_min":0,
                "rain_rate_hi_last_15_min":0,
                "rainfall_last_60_min":0,
                "rainfall_last_24_hr":0,
                "rain_storm":0,
                "rain_storm_start_at":null,
                "solar_rad":309,
                "uv_index":0.8,
                "rx_state":0,
                "trans_battery_flag":0,
                "rainfall_daily":0,
                "rainfall_monthly":90,
                "rainfall_year":95,
                "rain_storm_last":3,
                "rain_storm_last_start_at":1556834760,
                "rain_storm_last_end_at":1556924461
            },
            {
                "lsid":224118,
                "data_structure_type":1,
                "txid":6,
                "temp":44.8,
                "hum":null,
                "dew_point":null,
                "wet_bulb":null,
                "heat_index":null,
                "wind_chill":null,
                "thw_index":null,
                "thsw_index":null,
                "wind_speed_last":null,
                "wind_dir_last":null,
                "wind_speed_avg_last_1_min":null,
                "wind_dir_scalar_avg_last_1_min":null,
                "wind_speed_avg_last_2_min":null,
                "wind_dir_scalar_avg_last_2_min":null,
                "wind_speed_hi_last_2_min":null,
                "wind_dir_at_hi_speed_last_2_min":null,
                "wind_speed_avg_last_10_min":null,
                "wind_dir_scalar_avg_last_10_min":null,
                "wind_speed_hi_last_10_min":null,
                "wind_dir_at_hi_speed_last_10_min":null,
                "rain_size":1,
                "rain_rate_last":0,
                "rain_rate_hi":0,
                "rainfall_last_15_min":0,
                "rain_rate_hi_last_15_min":0,
                "rainfall_last_60_min":0,
                "rainfall_last_24_hr":0,
                "rain_storm":0,
                "rain_storm_start_at":null,
                "solar_rad":null,
                "uv_index":null,
                "rx_state":0,
                "trans_battery_flag":0,
                "rainfall_daily":0,
                "rainfall_monthly":106,
                "rainfall_year":113,
                "rain_storm_last":106,
                "rain_storm_last_start_at":1556709480,
                "rain_storm_last_end_at":1556924461
            },
            {
                "lsid":224119,
                "data_structure_type":1,
                "txid":7,
                "temp":39.9,
                "hum":null,
                "dew_point":null,
                "wet_bulb":null,
                "heat_index":null,
                "wind_chill":null,
                "thw_index":null,
                "thsw_index":null,
                "wind_speed_last":null,
                "wind_dir_last":null,
                "wind_speed_avg_last_1_min":null,
                "wind_dir_scalar_avg_last_1_min":null,
                "wind_speed_avg_last_2_min":null,
                "wind_dir_scalar_avg_last_2_min":null,
                "wind_speed_hi_last_2_min":null,
                "wind_dir_at_hi_speed_last_2_min":null,
                "wind_speed_avg_last_10_min":null,
                "wind_dir_scalar_avg_last_10_min":null,
                "wind_speed_hi_last_10_min":null,
                "wind_dir_at_hi_speed_last_10_min":null,
                "rain_size":1,
                "rain_rate_last":0,
                "rain_rate_hi":0,
                "rainfall_last_15_min":0,
                "rain_rate_hi_last_15_min":0,
                "rainfall_last_60_min":0,
                "rainfall_last_24_hr":0,
                "rain_storm":null,
                "rain_storm_start_at":null,
                "solar_rad":null,
                "uv_index":null,
                "rx_state":0,
                "trans_battery_flag":0,
                "rainfall_daily":0,
                "rainfall_monthly":0,
                "rainfall_year":0,
                "rain_storm_last":null,
                "rain_storm_last_start_at":null,
                "rain_storm_last_end_at":null
            },
            {
                "lsid":224112,
                "data_structure_type":4,
                "temp_in":65.1,
                "hum_in":44.9,
                "dew_point_in":43.2,
                "heat_index_in":62.9
            },
            {
                "lsid":224111,
                "data_structure_type":3,
                "bar_sea_level":30.125,
                "bar_trend":0.033,
                "bar_absolute":29.472
            }
        ]
    },
    "error":null
Title: Re: Reading data locally from WLL
Post by: docbee on May 04, 2019, 09:16:05 AM
@docbee, what are you seeing? Have you encountered those UDP messages?

Nope, I am happy to work on the HTTP GET responses. If they update in good frequency this is all we need ;-)
Title: Re: Reading data locally from WLL
Post by: kobuki on May 04, 2019, 09:53:00 AM
@docbee, what are you seeing? Have you encountered those UDP messages?

Nope, I am happy to work on the HTTP GET responses. If they update in good frequency this is all we need ;-)
I see, not even if you start the WLL app in an Android emulator like Nox (if you've tried, that is)?
Title: Re: Reading data locally from WLL
Post by: johnd on May 04, 2019, 09:57:25 AM
@kobuki: Just in case you don't spot it, you should have a PM.
Title: Re: Reading data locally from WLL
Post by: kobuki on May 04, 2019, 10:08:21 AM
@kobuki: Just in case you don't spot it, you should have a PM.
Thanks, just read it. Thanks for the heads-up, as I rely on the email notifications and they tend to be sent out slowly nowadays from wxforum. Anyway, looks interesting.
Title: Re: Reading data locally from WLL
Post by: galfert on May 04, 2019, 10:40:37 AM
Very nice Boris. You know your stuff.
Title: Re: Reading data locally from WLL
Post by: kobuki on May 04, 2019, 11:22:36 AM
With consent from John, I'm pasting a single UDP packet worth of json data from his packet capture:

Code: [Select]
{
    "did":"mac-address-of-WLL",
    "ts":1556813312,
    "conditions":[
        {
            "lsid":220585,
            "data_structure_type":1,
            "txid":1,
            "wind_speed_last":0.75,
            "wind_dir_last":310,
            "rain_size":2,
            "rain_rate_last":0,
            "rain_15_min":0,
            "rain_60_min":0,
            "rain_24_hr":5,
            "rain_storm":5,
            "rain_storm_start_at":1556740140,
            "rainfall_daily":1,
            "rainfall_monthly":5,
            "rainfall_year":17,
            "wind_speed_hi_last_10_min":2.31,
            "wind_dir_at_hi_speed_last_10_min":337
        },
        {
            "lsid":220586,
            "data_structure_type":1,
            "txid":2,
            "wind_speed_last":0.00,
            "wind_dir_last":0,
            "rain_size":2,
            "rain_rate_last":0,
            "rain_15_min":0,
            "rain_60_min":0,
            "rain_24_hr":0,
            "rain_storm":null,
            "rain_storm_start_at":null,
            "rainfall_daily":0,
            "rainfall_monthly":0,
            "rainfall_year":0,
            "wind_speed_hi_last_10_min":0.00,
            "wind_dir_at_hi_speed_last_10_min":0
        }
    ]
}

They are sent at standard LOOP packet intervals, apparently (it could be radio message interval, I haven't investigated thoroughly). Unfortunately it is currently not known how to trigger these broadcast messages. The flow is possibly triggered by the startup of the mobile app, a login on the WL website, or some similar action. More captures and analysis is needed, unless we somehow acquire an API docs from Davis in the meantime. However, with the info in this thread, it's possible to start making an independent driver for eg. WeeWx...

EDIT: maybe it's worth a try calling http://weatherlinklive.localdomain/v1/real_time with sensors connected as mentioned in previous posts...
Title: Re: Reading data locally from WLL
Post by: docbee on May 04, 2019, 11:31:22 AM
Payload looks pretty much the same. Always good to have 2 ways to get data. While the HTTP GET request can be routed the UDP broadcast does not need constant polling (but we currently don't know how to start it). My fear is that Davis will need you to have a certain subscription that tells the WLL unit to support UDP broadcasts. But at current state this is just guessing.

Do the HTTP Get requests also show updated data (especially wind data) every 2.5 seconds when polled with that frequency?
Title: Re: Reading data locally from WLL
Post by: kobuki on May 04, 2019, 11:41:13 AM
Yeah. There are a few things to try:

1. Get (pull) data using a standard free subscription via http from the device each minute, for example. It should just work, I think.
2. Pull the real_time url to try starting the udp stream.

1. would be good for normal archive records, 2. would initiate real time updates. If 2. can work independently from 1, it would be enough in itself to build the contents of the archive records.
Title: Re: Reading data locally from WLL
Post by: johnd on May 04, 2019, 11:58:56 AM
The flow is possibly triggered by the startup of the mobile app, a login on the WL website, or some similar action.

The UDP flow is absent until the WL app starts up on the local network - that appears to be the (or at least one) trigger in principle. But the details of the message exchange that starts (and then maintains) the flow aren't clear as yet. Mark, not sure whether you might be able to add any extra snippetts?
Title: Re: Reading data locally from WLL
Post by: kobuki on May 04, 2019, 12:05:34 PM
Code: [Select]
mars:~ # curl 192.168.123.232/v1/real_time

{"data":null,"error":{"code":409,"message":"No ISS Transmitters.  Real Time broadcast not enabled"}}
What about this, with transmitters actually connected?
Title: Re: Reading data locally from WLL
Post by: waiukuweather on May 04, 2019, 02:35:23 PM
so the real time http get returns the port number of the UDP broadcast
to get the real time windspeed and direction and rain data
but its not known what triggers that UDP data to start up sending

but for now using the current conditions http get call you can get all the other data (I wonder how fast WLL can process those calls?)
Title: Re: Reading data locally from WLL
Post by: kobuki on May 04, 2019, 03:15:43 PM
but its not known what triggers that UDP data to start up sending
At the expense of embarrassment repeating myself, did someone try the following with an ISS connected to the WLL? So far we only see an attempt without. The wording of the error message in the first post would imply that with at least an ISS connected, it would start the UDP stream.

Try pulling the URL: http://ip-of-wll/v1/real_time via browser or any other tool, with app not running but transmitters (ISS, ATK, etc) running.

Title: Re: Reading data locally from WLL
Post by: johnd on May 04, 2019, 03:20:13 PM
Mine's in the office - won't be back there until Tuesday now. Mayday holiday on Monday :?
Title: Re: Reading data locally from WLL
Post by: miraculon on May 04, 2019, 06:56:13 PM
@miraculon: Looks like you have
on TX #2 a wind plus solar station
on TX #3 a temp/hum plus rain station
on TX #4 a wind plus UV and solar station
on TX #5 a ISS with UV and solar
on TX #6 a temp/hum plus rain station
on TX #7 a temp station

Is that right? Existing data does not seem to match available Davis stations types perfectly.
Station identifier (telling what kind of station it is) seems not to be in the data. May be the user has to define that within the SW and SW just grabs the data pieces from JSON that make sense to the selected sensor type... still guessing

on TX #2 a wind plus solar station ACTUAL: wind station only
on TX #3 a temp/hum plus rain station CORRECT
on TX #4 a wind plus UV and solar station ACTUAL: wind w/ temp
on TX #5 a ISS with UV and solar CORRECT
on TX #6 a temp/hum plus rain station ACTUAL: temp and rain (no hum) (this is primarily a rain station with a buried temp probe for soil temp)
on TX #7 a temp station CORRECT

Greg H.
Title: Re: Reading data locally from WLL
Post by: johnd on May 05, 2019, 03:54:11 AM
Almost certainly an obvious point, but there wouldn't necessarily be a reading for every sensor connected to a transmitter in every message (at least not unless Davis have changed their general approach). The update rate from transmitters can't have changed, so eg a new solar/UV reading might be expected only every 50-60 seconds.
Title: Re: Reading data locally from WLL
Post by: docbee on May 05, 2019, 04:16:17 AM
I think the WLL logs the data from sensors coming in an internal generic structure that is the same for all sensors, but just some of the fields are filled/relevant depending on the data the sensor provides. Request for current conditions just returns these internal structures for all transmitters in JSON format. Therefore, when you know it is a UV sensor just updating every 50 seconds, then asking for the record every 10 seconds is not wrong, but you simply should expect to get unchanged data reported a lot. When looking for wind speed, things are different. But we still have to prove, if this assumption of mine is right. 

I also think that the records for itself do not tell which type of sensor it is. Some evidence can be taken from which sensor data is returned as "null", but as Greg replied my guesses are not completely right and I doubt there is a perfect guessing schema. Therefore, I will implement a feature in Meteobridge, where the user has to define per transmitter, which kind of sensor is connected. I think it is a severe design flaw with Davis sensors from the beginning, that those don't identify which kind they are. May be it makes life more easy for production (can use the ISS modules with all the other sensor types) but it adds a lot of confusion and manual hassle to get things sorted out. In general it is simply stupid to let the user tell the system that this is a pure wind sensor instead of the wind sensor telling this within its payload data. But I stop here listing what Davis engineers are not doing right on a conceptual level... would be a rather long list ;-)
Title: Re: Reading data locally from WLL
Post by: johnd on May 05, 2019, 04:26:29 AM
I think the WLL logs the data from sensors coming in an internal generic structure that is the same for all sensors, but just some of the fields are filled/relevant depending on the data the sensor provides. Request for current conditions just returns these internal structures for all transmitters in JSON format.

Agreed that would be logical (Captain!) but it's not what seems to be happening, at least as I read Greg's post.

So his Tx #2 message did not contain solar data apparently. So either solar is not included in every #2 message or maybe it is only included if there is a different value from the previous reading? But maybe I've misunderstood Greg's post.
Title: Re: Reading data locally from WLL
Post by: kobuki on May 05, 2019, 06:12:36 AM
The inconsistency can be a bug in the current JSON API. This might be a reason the docs aren't published yet. I think it's worth waiting for the yet undocumented API to stabilize before jumping to conclusions, though knowing it to an extent can help developing support for beta drivers where only these issues need to  be addressed later.
Title: Re: Reading data locally from WLL
Post by: johnd on May 05, 2019, 06:47:05 AM
This might be a reason the docs aren't published yet.

That's my reading of the situation too FWIW.

If I had to guess, I suspect that Davis is no different to any other company with a new product about to launch, ie there's a little tension between the commercial side of the business who want it launched just as soon as it's a stable, credible product (and initial signs are that WLL is behaving at wl.com fully as designed) and the development team who'd prefer to see all the i's dotted and t's crossed before launch.

What this probably comes down to in practice is that stable performance of the product in its primary role is given maximum development priority while add-on features like a solid and properly documented API suitable for use by 3rd parties, repeater compatibility etc need to wait a few months before development is signed off and a firmware update is released. (Incidentally, one WLL feature I've not been able to spot so far anywhere is an indication of what the current firmware revision might be. Maybe there is an as yet hidden web interface that reveals such details?)
Title: Re: Reading data locally from WLL
Post by: miraculon on May 05, 2019, 08:17:22 AM
I thought that some screenshots might help. The sensor configuration "Edit" screens have checkboxes for each type of sensor.

I am attaching the main config and the 3 individual edit screens for #2, #4 and #6.

EDIT: I have a temp sensor attached to #4, but didn't select it for WLL. (I use it for meteohub)

Greg H.
Title: Re: Reading data locally from WLL
Post by: johnd on May 05, 2019, 08:24:10 AM
I thought that some screenshots might help.

Ah, OK, sorry that's probably my fault for confusing things. I was interpreting your 'ACTUAL' as the data content in the message, whereas you were referring to actual sensor configuration. That makes more sense. So the #5 message might be of more interest.
Title: Re: Reading data locally from WLL
Post by: docbee on May 05, 2019, 02:01:43 PM
I couldn't resist to make a first experimental port to Meteobridge. So Meteobridge users can give it a try and report bugs back to me. As I only have an early US WLL to test with and no sensors sending on US frequencies here in Germany, I need support from the Meteobridge user base.

I tested with Gregs data.

This is how I did setup the transmitters in Meteobridge:

 [ You are not allowed to view attachments ]

And this is the reported data:

 [ You are not allowed to view attachments ]

Test version is released, so any Meteobridge user with a WLL can give it a test drive.
 
Title: Re: Reading data locally from WLL
Post by: kobuki on May 05, 2019, 02:04:57 PM
The WLL allows arbitrary (almost) conbimations of sensors, will you implement that later?
Title: Re: Reading data locally from WLL
Post by: docbee on May 05, 2019, 02:08:28 PM
currently it is restricted to these combinations:

 [ You are not allowed to view attachments ]
Title: Re: Reading data locally from WLL
Post by: kobuki on May 05, 2019, 02:19:25 PM
currently it is restricted to these combinations:

 [ You are not allowed to view attachments ]
I'm aware, that's why I asked "later"...
Title: Re: Reading data locally from WLL
Post by: docbee on May 05, 2019, 03:58:01 PM
yes, if a real need occurs. I try to keep it simple to address 95% of the situations first. Does Davis allow to configure the transmitters apart from their standard setups and do they make use of that in weather link? Anyway, I am happy that we got that far over a single weekend.  :grin: 

BTW: Leaf/Soil stuff is not supported yet. I will need a user to send me example data how this looks like in the JSON. I guess that will be in the so far not spotted "data_structure_type" #2. Anybody out there using this with an WLL? If so, could you please sniff some data for us?
Title: Re: Reading data locally from WLL
Post by: miraculon on May 05, 2019, 04:27:58 PM
docbee,

I decided to give it a try. The indoor temperature is too low. (is it using the offset for the internal PRO sensors?)

Indoor   29 sec   -4.1°C 46% 1011.2hPa (989.3hPa)   24.6°F 46% 29.86inHg (29.21inHg)

Baro and Outdoor sensors look OK. I am not sure which wind set is in use, but for a brief test it will be OK.

I'll run it as a test for a couple of hours, but then I am going to switch back to the normal configuration with the internal RED receiver.

Greg H.
Title: Re: Reading data locally from WLL
Post by: johnd on May 05, 2019, 05:38:39 PM
Does Davis allow to configure the transmitters apart from their standard setups and do they make use of that in weather link?

Not 100% sure what the question is, but if you mean: Can WLL receive the same fully arbitrary mix of transmitters as an Envoy8X then there's every indication that the answer is yes. Until someone sets up 8 separate ISS units simultaneously then I guess we can't be absolutely sure  that the maximum can be reached but it can certainly receive all data from two separate ISS units, which a standard VP2 obviously cannot do.

And weatherlink.com can show all of that data. This isn't surprising since it's effectively what an EM station does routinely at wl.com.

Title: Re: Reading data locally from WLL
Post by: docbee on May 05, 2019, 05:43:34 PM
Indoor   29 sec   -4.1°C 46% 1011.2hPa (989.3hPa)   24.6°F 46% 29.86inHg (29.21inHg)
Baro and Outdoor sensors look OK. I am not sure which wind set is in use, but for a brief test it will be OK.

Yes, indoor temp correction gets applied, so you will need to remove that if using the WLL.
Title: Re: Reading data locally from WLL
Post by: waiukuweather on May 06, 2019, 02:50:39 PM
just to say great work docbee and thanks for sharing :)
Title: Re: Reading data locally from WLL
Post by: johnd on May 07, 2019, 12:26:14 PM
Try pulling the URL: http://ip-of-wll/v1/real_time via browser or any other tool, with app not running but transmitters (ISS, ATK, etc) running.

Just seems to return the default JSON error string and nothing more I'm afraid.
Title: Re: Reading data locally from WLL
Post by: waiukuweather on May 07, 2019, 02:11:32 PM
calling the real_time
does return info about the broadcast port

but has anyone then got an UDP data captured on that port?
Title: Re: Reading data locally from WLL
Post by: kobuki on May 07, 2019, 02:14:10 PM
calling the real_time
does return info about the broadcast port
Could you please paste the full output? (Please use the # icon to use code tags.)
Title: Re: Reading data locally from WLL
Post by: waiukuweather on May 07, 2019, 02:26:12 PM
Quote
{"data":{"broadcast_port":22222,"duration":1200},"error":null}

is what I got with the help from a tester
which has already been reported earlier in this thread
that is the same port that the IP data logger normally uses
listening in for UDP packets on that port..nothing
Title: Re: Reading data locally from WLL
Post by: kobuki on May 07, 2019, 03:38:41 PM
Quote
{"data":{"broadcast_port":22222,"duration":1200},"error":null}

listening in for UDP packets on that port..nothing
Hmm, it's actually quite promising. This is not an error message (see "error":null), but looks like a confirmation message about enabling the broadcast message stream for 1200 seconds (I assume seconds). You need to listen to broadcast messages, not messages directed to your IP. Usually you listen on the wildcard address (0.0.0.0) and a specific port (in this case port 22222) to receive UDP broadcasts. When you receive one you can check the source IP address to see if the packet is one you're interested in.
Title: Re: Reading data locally from WLL
Post by: waiukuweather on May 07, 2019, 03:59:44 PM
oh, I just realised have done my code wrong...will try again
Title: Re: Reading data locally from WLL
Post by: johnd on May 08, 2019, 07:08:15 AM
but looks like a confirmation message about enabling the broadcast message stream for 1200 seconds (I assume seconds).

Give that man a coconut! Yes it seems to last exactly 480 messages at what appears to be 2.5sec intervals (only integer seconds reported as 'ts' so interval tends to alternate 2 and 3 secs) or approx 1200 secs.

But no sign of data_structure_type 2 or 3 in the UDP data, though both are there in current_conditions. Guess there are 3 possibilities: Leaf/soil & Bar not implemented yet; Not planned to be implemented (HTTP GET may be considered sufficient for these long-refresh parameters); Different trigger needed?
Title: Re: Reading data locally from WLL
Post by: kobuki on May 08, 2019, 07:12:15 AM
Give that man a coconut! Yes it seems to last exactly 480 messages at what appears to be 2.5sec intervals (only integer seconds reported as 'ts' so interval tends to alternate 2 and 3 secs) or approx 1200 secs.
You can give a like instead here. So we've discovered almost all the dirty little details. I wonder whose driver will be released first...
Title: Re: Reading data locally from WLL
Post by: waiukuweather on May 08, 2019, 03:04:05 PM
Yes, I have got wind/ rain UDP Data coming in now
Once I fixed my parsing out of the port number :)
Title: Re: Reading data locally from WLL
Post by: kobuki on May 08, 2019, 03:19:13 PM
Yes, I have got wind/ rain UDP Data coming in now
Once I fixed my parsing our of the port number :)
What system or software is it for?
Title: Re: Reading data locally from WLL
Post by: waiukuweather on May 08, 2019, 04:34:16 PM
not really wanting to say, as am not wanting people to think I am just trying to promote my software here
Title: Re: Reading data locally from WLL
Post by: kobuki on May 08, 2019, 05:12:03 PM
not really wanting to say, as am not wanting people to think I am just trying to promote my software here
I don't think people here would think that, but do as you feel.
Title: Re: Reading data locally from WLL
Post by: mcrossley on May 08, 2019, 05:13:50 PM
The username gives you a clue  ;)
Title: Re: Reading data locally from WLL
Post by: waiukuweather on May 08, 2019, 05:19:15 PM
I didn't even want the username to give me away
I am though an original wxforum member (which came about from another forum that Gary Oldham set up,weatherforum.net ) :) and I have been working on my weather software for nearly 20 years so I am no newbie to all this :)
Title: Re: Reading data locally from WLL
Post by: galfert on May 08, 2019, 05:20:06 PM
Can someone explain in nonprogramming terms why UDP sensor data broadcast would be better than the HTTP GET (curl).

I'm techie enough as I'm a network systems manager. So I get that UDP traffic being best effort blind send versus TCP traffic having delivery confirmation and retransmission. So that isn't the question I'm asking.

I want to know from the perspective of getting data from the WLL why one would favor one method over the other. Does one method deliver quicker sensor updates? Does one method require less overhead for the device requesting the data? Does one method provide more sensors data (meaning maybe one method is missing sensors and only sends a subset of info and is not complete?)

What if you wanted to run multiple weather software. In the past we had things like Virtual VP and needing to balance logger polling. Does the WLL allow through either method the ability to run multiple weather software? I would think at at least the HTTP GET would work in this sense. Not sure about the UDP if the WLL gets set to broadcast UDP mode and then it can only do so to one IP destination as from what I've read then maybe it doesn't support multicast?
Title: Re: Reading data locally from WLL
Post by: waiukuweather on May 08, 2019, 05:46:22 PM
UDP is less over head
because the data is sent out to any listener on that local port
i.e so more than one listener can use that data
and that listener does not have to keep asking for the data and the sender does not have to then process that request and send it to who requested it
and so the wind and current rain is sent out at 2.5 seconds via UDP
and then the other data that does not change as fast (temperature/humidity/barometer etc) can be requested say every minute via a HTTP get
Title: Re: Reading data locally from WLL
Post by: kobuki on May 08, 2019, 05:52:49 PM
Can someone explain in nonprogramming terms why UDP sensor data broadcast would be better than the HTTP GET (curl).
In short, a message oriented protocol such as UDP is quicker and uses less resources. There's no such thing as TCP broadcast, either. There are very few values that change quickly enough to warrant a stream oriented standard protocol with all the bells and whistles. Broadcasts are an easy way to publish quickly changing conditions in short messages in the broadcast domain of your device. Also, you would need to poll with http gets, where the frequency and http overhead is getting stressful with a small device when many devices are polling. Yes, there are modern methods like websockets, but I think UPD broadcasts are an adequate solution to the problem at hand, for instance no need to maintain a heap of connection states in memory, etc.
Title: Re: Reading data locally from WLL
Post by: galfert on May 08, 2019, 06:57:26 PM
Okay thank you both.

So UDP for the WLL only sends wind and rain I'm hearing. So the HTTP GET is still required for other sensors.

Regarding running multiple weather software simultaneously, any issues?
Title: Re: Reading data locally from WLL
Post by: waiukuweather on May 08, 2019, 11:35:59 PM
only issue with more than 1 software running would be if there is a clash with the http get request for the other data and well the WLL can handle that?
Title: Re: Reading data locally from WLL
Post by: johnd on May 09, 2019, 03:42:43 AM
So UDP for the WLL only sends wind and rain I'm hearing.

No, it sends more than that. AFAICR it sends Type 1 messages via UDP, but not Type 2 and Type 3. My picture is:

Type 1 = ISS-type transmitter (basically any active Tx that uses the ISS SIM board, which includes 6332 etc)

Type 2 = 6345 Tx data (ie leaf soil/moisture etc)

Type 3 = Pressure data
Title: Re: Reading data locally from WLL
Post by: waiukuweather on May 09, 2019, 03:50:32 AM
here is the UDP data
{"did":"001D0A710197","ts":1557314117,"conditions":[{"lsid":223656,"data_structure_type":1,"txid":1,"wind_speed_last":0.00,"wind_dir_last":0,"rain_size":2,"rain_rate_last":0,"rain_15_min":0,"rain_60_min":0,"rain_24_hr":0,"rain_storm":null,"rain_storm_start_at":null,"rainfall_daily":0,"rainfall_monthly":0,"rainfall_year":0,"wind_speed_hi_last_10_min":0.00,"wind_dir_at_hi_speed_last_10_min":0}]}


so not sure where there is more than wind and rain data?
Title: Re: Reading data locally from WLL
Post by: johnd on May 09, 2019, 03:56:04 AM
Hmm, you may be right - don't have my data in front of me right now. Just assumed that a Type 1 message meant a complete Type 1 message, but maybe it doesn't? So potentially there are two Type 1 messages - complete (HTTP) and partial (UDP).
Title: Re: Reading data locally from WLL
Post by: waiukuweather on May 09, 2019, 03:57:03 AM
 [tup]
Title: Re: Reading data locally from WLL
Post by: kobuki on May 09, 2019, 05:33:37 AM
Type X might refer to a type of structure X in json that Davis made up for the device, not necessitating a complete one but one that has certain values in certain structure elements. After all, it's really only wind and rain that changes rapidly enough.
Title: Re: Reading data locally from WLL
Post by: galfert on May 09, 2019, 07:30:10 AM
Well this WLL API protocol is looking really really nice. Seems to me like the WLL will be key in the next console from Davis. All Davis needs to do now is come out with a reasonable cost color display that gets its data from the WLL.

But we don't need to wait for Davis to do that. Now someone can write an Android or iOS app with a nice console like display graphics and you've got a better console. It would even be neat to see for nostalgia a virtual VP2 console and a Vue one also. Pretty much like the web version I've seen some people have on their personal website. The real killer would be something that looks like the Ambient WS-2000 display where all the data is visible without having to press any buttons.

Davis can even come out with the app themselves instead of a real console.

Then the only thing left for Davis to do is come out with a new ISS that uses I2C bus sensors. And that would be good enough for me to consider it a VP3.
Title: Re: Reading data locally from WLL
Post by: kobuki on May 09, 2019, 07:37:56 AM
I think the wisdom has been established about Davis' stance already. Starting with the WLL they're distancing themselves from the idea of modern colorful consoles and seem to encourage interested parties make apps instead, using their (yet undocumented) API.
Title: Re: Reading data locally from WLL
Post by: waiukuweather on May 09, 2019, 02:38:29 PM
here is a curve ball
with the Davis IP data logger, it acted like a usb data logger re commands and so software could request a history data dump
I wonder if a data history data dump can be still done with the WLL
maybe by setting a start and end date/time in the http Get call via json commands or similar (probably will need to wait for the api release to now?)
Title: Re: Reading data locally from WLL
Post by: Simon on May 15, 2019, 07:31:56 PM
Here's the solution:

https://weatherlink.github.io/weatherlink-live-local-api/
https://weatherlink.github.io/v2-api/
https://app.swaggerhub.com/apis/DavisInstruments/WeatherLink/v2
Title: Re: Reading data locally from WLL
Post by: kobuki on May 15, 2019, 07:44:16 PM
Thanks. The material on the first link appears to originate from Jeremy Wall, System Architect and Principal Web Developer at Davis Instruments and weatherlink (according to GitHub), so I guess it's semi-official, or something like that. Stuff seems detailed enough for implementation in drivers, even some Python examples are provided for the real-time and the pull/REST API. Nice one, Davis!  [tup]
Title: Re: Reading data locally from WLL
Post by: waiukuweather on May 15, 2019, 08:37:32 PM
only a few things new others had not worked out
does not look to be any way to get history data
(although you can get that from weatherlink.com)
Title: Re: Reading data locally from WLL
Post by: kobuki on May 16, 2019, 03:17:31 AM
I'd assume accessing the history is tied to a cloud account where that's stored and locally the device doesn't store historical data at all, or only for catching up in cases of loss or disturbance of the internet connection or upload. I'd say it's fine as long as local data querying is not hindered in any way.
Title: Re: Reading data locally from WLL
Post by: johnd on May 16, 2019, 03:29:37 AM
...and locally the device doesn't store historical data at all...

Yes, we had been speculating about that. One possibility is that the per-minute current conditions messages are simply queued in the event of a network outage and that effectively becomes the local archive data. (But that's just one speculation and might easily be wrong.)

Could be checked I guess by eg connecting WLL via cabled Ethernet, pulling the network plug for a while and then reestablishing contact and examining the data.
Title: Re: Reading data locally from WLL
Post by: Brientim on May 16, 2019, 03:58:22 AM
As per the specifications the defaults archiving is 15 minutes but this can be modified to 2 hours (120 minutes) but it does not provide any explanation on storage capabilities.
Title: Re: Reading data locally from WLL
Post by: johnd on May 16, 2019, 06:39:41 AM
As per the specifications the defaults archiving is 15 minutes but this can be modified to 2 hours (120 minutes) but it does not provide any explanation on storage capabilities.

And of course down to 5 mins or 1 min if you have a Pro or Pro Plus plan.

But just to be clear, I'm not suggesting that the WLL unit may not store archive data as we know it (albeit presumably in JSON rather than binary form) and indeed I really hope this is the case (and also that Davis allow local access to it even in the absence of a Pro plan). I'm simply speculating that this is not the only conceivable solution - archiving the data, ie chunking into eg 15min summary records could in theory be done solely at weatherlink.com.
Title: Reading data locally from WLL
Post by: Brientim on May 16, 2019, 04:41:29 PM
Here is a response from Davis

“WeatherLink Live can store about 180 days of data, assuming is is listening to just one sensor suite and has a 15-minute archive interval. More transmitters and faster archive intervals would decrease this.”

This doesn’t answer the specifics of the questions especially access to the stored data
Title: Re: Reading data locally from WLL
Post by: kobuki on May 16, 2019, 04:51:41 PM
This doesn’t answer the specifics of the questions especially access to the stored data
This might indeed indicate that its internal storage is just a buffer for cases of internet connection problems. It doesn't necessarily have a facility to actually access that buffered data (it would be nice, though). I think it would be in the docs linked earlier if there was. But to be honest, local access of real-time data and current conditions is likely only there to support 3rd-party applications that do have their database for storing historical data.
Title: Re: Reading data locally from WLL
Post by: docbee on May 18, 2019, 04:10:16 AM
I am still looking for data snippets that contain leaf/soil data. Does anybody out there have that connected to a WLL and is willing to post a data set here? I can help to have it generated.
Title: Re: Reading data locally from WLL
Post by: johnd on May 18, 2019, 04:19:04 AM
I am still looking for data snippets that contain leaf/soil data.

Doesn't message Type 2 in https://weatherlink.github.io/weatherlink-live-local-api/ (https://weatherlink.github.io/weatherlink-live-local-api/) show you what you need? I guess all the sensor values shown are null - is that the problem?
Title: Re: Reading data locally from WLL
Post by: docbee on May 18, 2019, 04:46:58 AM
To have some samples would be nice to test the code.
Title: Re: Reading data locally from WLL
Post by: Brientim on May 18, 2019, 04:50:10 AM
To have some samples would be nice to test the code.

Code: [Select]
{"data":{"did":"001D00000000","ts":1558169339,"conditions":[{"lsid":227174,"data_structure_type":1,"txid":1,"temp": 56.1,"hum":80.4,"dew_point": 50.1,"wet_bulb": 52.4,"heat_index": 56.1,"wind_chill": 56.1,"thw_index": 56.1,"thsw_index": 54.1,"wind_speed_last":0.00,"wind_dir_last":0,"wind_speed_avg_last_1_min":0.00,"wind_dir_scalar_avg_last_1_min":null,"wind_speed_avg_last_2_min":0.00,"wind_dir_scalar_avg_last_2_min":null,"wind_speed_hi_last_2_min":0.00,"wind_dir_at_hi_speed_last_2_min":0,"wind_speed_avg_last_10_min":0.00,"wind_dir_scalar_avg_last_10_min":null,"wind_speed_hi_last_10_min":0.00,"wind_dir_at_hi_speed_last_10_min":0,"rain_size":2,"rain_rate_last":0,"rain_rate_hi":0,"rainfall_last_15_min":0,"rain_rate_hi_last_15_min":0,"rainfall_last_60_min":0,"rainfall_last_24_hr":0,"rain_storm":null,"rain_storm_start_at":null,"solar_rad":0,"uv_index":0.0,"rx_state":0,"trans_battery_flag":0,"rainfall_daily":0,"rainfall_monthly":0,"rainfall_year":0,"rain_storm_last":null,"rain_storm_last_start_at":null,"rain_storm_last_end_at":null},{"lsid":227177,"data_structure_type":2,"txid":4,"temp_1": 52.8,"temp_2": 51.2,"temp_3":null,"temp_4":null,"moist_soil_1":13.8,"moist_soil_2":32.8,"moist_soil_3":null,"moist_soil_4":null,"wet_leaf_1":0.6,"wet_leaf_2":null,"rx_state":0,"trans_battery_flag":0},{"lsid":227173,"data_structure_type":4,"temp_in": 75.6,"hum_in":38.7,"dew_point_in": 48.7,"heat_index_in": 74.5},{"lsid":227172,"data_structure_type":3,"bar_sea_level":30.392,"bar_trend": 0.043,"bar_absolute":30.289}]},"error":null}
Title: Re: Reading data locally from WLL
Post by: kobuki on May 18, 2019, 06:11:55 AM
Brientim's sample in a readable format:
Code: [Select]
{
    "data":{
        "did":"001D00000000",
        "ts":1558169339,
        "conditions":[
            {
                "lsid":227174,
                "data_structure_type":1,
                "txid":1,
                "temp":56.1,
                "hum":80.4,
                "dew_point":50.1,
                "wet_bulb":52.4,
                "heat_index":56.1,
                "wind_chill":56.1,
                "thw_index":56.1,
                "thsw_index":54.1,
                "wind_speed_last":0.00,
                "wind_dir_last":0,
                "wind_speed_avg_last_1_min":0.00,
                "wind_dir_scalar_avg_last_1_min":null,
                "wind_speed_avg_last_2_min":0.00,
                "wind_dir_scalar_avg_last_2_min":null,
                "wind_speed_hi_last_2_min":0.00,
                "wind_dir_at_hi_speed_last_2_min":0,
                "wind_speed_avg_last_10_min":0.00,
                "wind_dir_scalar_avg_last_10_min":null,
                "wind_speed_hi_last_10_min":0.00,
                "wind_dir_at_hi_speed_last_10_min":0,
                "rain_size":2,
                "rain_rate_last":0,
                "rain_rate_hi":0,
                "rainfall_last_15_min":0,
                "rain_rate_hi_last_15_min":0,
                "rainfall_last_60_min":0,
                "rainfall_last_24_hr":0,
                "rain_storm":null,
                "rain_storm_start_at":null,
                "solar_rad":0,
                "uv_index":0.0,
                "rx_state":0,
                "trans_battery_flag":0,
                "rainfall_daily":0,
                "rainfall_monthly":0,
                "rainfall_year":0,
                "rain_storm_last":null,
                "rain_storm_last_start_at":null,
                "rain_storm_last_end_at":null
            },
            {
                "lsid":227177,
                "data_structure_type":2,
                "txid":4,
                "temp_1":52.8,
                "temp_2":51.2,
                "temp_3":null,
                "temp_4":null,
                "moist_soil_1":13.8,
                "moist_soil_2":32.8,
                "moist_soil_3":null,
                "moist_soil_4":null,
                "wet_leaf_1":0.6,
                "wet_leaf_2":null,
                "rx_state":0,
                "trans_battery_flag":0
            },
            {
                "lsid":227173,
                "data_structure_type":4,
                "temp_in":75.6,
                "hum_in":38.7,
                "dew_point_in":48.7,
                "heat_index_in":74.5
            },
            {
                "lsid":227172,
                "data_structure_type":3,
                "bar_sea_level":30.392,
                "bar_trend":0.043,
                "bar_absolute":30.289
            }
        ]
    },
    "error":null
}
Title: Re: Reading data locally from WLL
Post by: docbee on May 18, 2019, 08:23:17 AM
Thanks, this helped to test the code. Support for record type 1 is now also included to the Meteobridge WLL driver.
Title: Re: Reading data locally from WLL
Post by: mcrossley on May 21, 2019, 03:36:32 PM
I'm having a few issues with this access. I whipped up a small .Net 4.7 app to test out the data acquisition.

I can receive the broadcast data fine, but I'm having problems with the Realtime and Current data HTTP fetches.

The .Net httpClient aborts with an exception...
Message :System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.WebException: The server committed a protocol violation. Section=ResponseStatusLine

Now if it put a "normal" web site in the URL instead of the WLL the code works fine, even fetching JSON data from a web server.

Any ideas?
Title: Re: Reading data locally from WLL
Post by: kobuki on May 21, 2019, 03:51:57 PM
Please dump a request/response pair using curl, or even better, a wireshark capture. It's impossible to say what kind of violation the .net library thinks happened on the status line.
Title: Re: Reading data locally from WLL
Post by: waiukuweather on May 21, 2019, 03:52:37 PM
I had the same problem with .NET HTTPClient and so had to use another HTTP library etc
Title: Re: Reading data locally from WLL
Post by: mcrossley on May 21, 2019, 05:09:53 PM
Please dump a request/response pair using curl, or even better, a wireshark capture. It's impossible to say what kind of violation the .net library thinks happened on the status line.
I have a WS capture using Chrome browser, one using my util is on the way (I don't have a WLL so debugging via email at the mo). The capture does not show anything obviously out of sorts to me, but the WLL is must be cutting a corner somewhere? [ You are not allowed to view attachments ]

I had the same problem with .NET HTTPClient and so had to use another HTTP library etc
Thanks, I have tried both HttpClient and the older WebClient, both give a protocol error with the WLL, but work with every other URL I try.
Title: Re: Reading data locally from WLL
Post by: kobuki on May 21, 2019, 05:53:07 PM
The only obvious thing I can see is the line terminating characters are all over the place in the response. It's LFCR after the status line, then LFLF after the header part, but it must always be one CRLF after the status line, one CRLF after each header line, and after the last header line, CRLF 2 times (right before the response body). The .net lib could be a little more forgiving, but it errs out absolutely rightfully. I suggest parsing the response body using your own function...
Title: Re: Reading data locally from WLL
Post by: mcrossley on May 21, 2019, 06:12:18 PM
Thanks, I was looking for TCP type errors, and didn't look too closely at the http content!
Title: Re: Reading data locally from WLL
Post by: mcrossley on May 22, 2019, 05:07:10 AM
So my interpretation of this is as follows...

The WLL is actually sending:
Code: [Select]
HTTP/1.1 200 OK\0a\0d
content-type: application/json\0a\0a

{"data":{"broadcast_port":22222,"duration":1200},"error":null}\0a\0a\0d

When according to the HTTP standards it should be sending:
Code: [Select]
HTTP/1.1 200 OK\0d\0a
content-type: application/json\0d\0a\0d\0a

{"data":{"broadcast_port":22222,"duration":1200},"error":null}\0d\0a

Which means the interpreter thinks the body is still in the header - or some other mess!
Title: Re: Reading data locally from WLL
Post by: kobuki on May 22, 2019, 05:26:59 AM
So my interpretation of this is as follows...

The WLL is actually sending:
Code: [Select]
HTTP/1.1 200 OK\0a\0d
content-type: application/json\0a\0a

{"data":{"broadcast_port":22222,"duration":1200},"error":null}\0a\0a\0d

When according to the HTTP standards it should be sending:
Code: [Select]
HTTP/1.1 200 OK\0d\0a
content-type: application/json\0d\0a\0d\0a

{"data":{"broadcast_port":22222,"duration":1200},"error":null}\0d\0a

Which means the interpreter thinks the body is still in the header - or some other mess!
Isn't this the same thing I said above in other words? The RFC refers to \0d as CR and \0a as LF, and \0d\0a as CRLF.
Title: Re: Reading data locally from WLL
Post by: mcrossley on May 22, 2019, 05:38:49 AM
I hope so, I looked at the WS trace and codified what you had said in words. I was looking for confirmation it is correct before trying to get Davis involved in fixing it.
Title: Re: Reading data locally from WLL
Post by: kobuki on May 22, 2019, 05:40:47 AM
I hope so, I looked at the WS trace and codified what you had said in words. I was looking for confirmation it is correct before trying to get Davis involved in fixing it.
Please refer them to https://www.ietf.org/rfc/rfc2616.txt

BTW my question was a rhetorical one...
Title: Re: Reading data locally from WLL
Post by: havtrail on May 22, 2019, 11:12:29 AM
We need a rhetorical smiley!  :-)

Rich K.