Topic: HP2553 custom weather server report interval changes

[hiljo]

Hello everyone,

I've set up my HP2553 weather console already since january this year.
Some time ago i created some scripts to let the weather station report to a custom weather server. This works great, and all the values are saved in my database. One thing i notice however is that the time of reporting is not consequent. If I set the report interval to 300 (seconds), the reporting times are the following:
2021-08-21 11:56:35
2021-08-21 11:51:19
2021-08-21 11:46:04
2021-08-21 11:41:04
2021-08-21 11:35:36
2021-08-21 11:30:20
2021-08-21 11:25:05

As you can see, it isn't 300 seconds as entered.
I played a bit with the value (more and less then 300 seconds) and finally changed it to 295 seconds. Then the report times are:
2021-08-21 12:34:18
2021-08-21 12:29:18
2021-08-21 12:24:18
2021-08-21 12:19:18
2021-08-21 12:14:19
2021-08-21 12:09:19
2021-08-21 12:04:19

This is more close to 300 seconds, but over time this slightly changes offcourse.
Is this behaviour normal? Anyone having the same issues and what interval do we need to set to have it 'exactly' at 300 seconds interval?

Curious about your experiences!

[mahi]

I see similar issues with a WH2650 and I assume it is the same with the GW1000/1100 and other consoles. I have the interval set to 60 seconds, yet the actual interval often fluctuates between 45 and 75 seconds. It's not just the time in the messages that is off. The device really sends at irregular intervals.

Code: [Select]

2021-08-19+22:00:41
2021-08-19+22:01:25
2021-08-19+22:02:26
2021-08-19+22:03:27
2021-08-19+22:04:38
2021-08-19+22:05:29
2021-08-19+22:06:30
2021-08-19+22:07:31
2021-08-19+22:08:35
2021-08-19+22:09:33
2021-08-19+22:10:34
2021-08-19+22:11:35
2021-08-19+22:12:36
2021-08-19+22:13:52
2021-08-19+22:14:38
2021-08-19+22:15:39
2021-08-19+22:16:40
2021-08-19+22:17:49

[hiljo]

Thats what i also see, it is not even the observation time that is off, but also the report interval. You also haven't found out a better value or solution to this right?
(I sometimes even see a double report, so 2 reports within 2 minutes, where 5 minutes interval is set.. 

[mahi]

Sorry but no.

After your message I experimented a bit with different intervals and found 59 seconds more stable than 60 seconds:

Code: [Select]

2021-08-22+08:58:36
2021-08-22+08:59:36
2021-08-22+09:00:36
2021-08-22+09:01:36
2021-08-22+09:02:54 <---
2021-08-22+09:03:36
2021-08-22+09:04:36
2021-08-22+09:05:36
2021-08-22+09:06:51 <---
2021-08-22+09:07:36
2021-08-22+09:08:36
2021-08-22+09:09:36
2021-08-22+09:10:49 <---
2021-08-22+09:11:36
2021-08-22+09:12:36
2021-08-22+09:13:36
2021-08-22+09:14:46 <---
2021-08-22+09:15:36
The average interval (every :36 in my example) still drifts over time but previously it would drift a second per minute. So it definitely got a lot better.

Every fourth (sometimes fifth) minute the timing is way off.

[davidefa]

I think this is due to the uploading to various weather services.
I removed the upload to wu and wow ( enabled only ecowitt.net and a custom upload to my meteotemplate server, now set at 59 seconds ), and now I have an upload exactly every 60 seconds

[Rover1822]

I would be interested in more data from others. I do believe you are running into a limitation on the MPU or CPU and the code that is in use or applicable.
These units (consoles) are pretty much single threaded and that involves loop time on sensor input from the radio receiver and also from outbound calculations and connections.

I would expect also, that if you also had a low number of sensors, the relative accuracy on time would also go up.

Other words, hey, the little console does not have the computing power to handle everything on point, and that is expected.

For those of us that have coded on these or similar devices, there is this thing we call "missing the boat"  and it has to do with loop timing and other factors.

If I were coding this, my primary timing emphasis would be on the receipt of signals from the sensors. Everything else is secondary, and will take a back seat as far as timing. We are not talking high end multi core processors , so time interrupts should be expected, depending on factors. Outbound TCPIP connection take time , more or less depending on how one gets to the target site



Just my thoughts

[mahi]

I figured out what causes the irregular sending intervals of my WH2650 (firmware 1.6.8)...

First I experimented by removing the batteries of all sensors but that did not seem to have any effect on the timing.

I then looked at the network traffic. All weather services are disabled in WS View (apart from custom upload) but the console still tries to phone home to http://cdnrtpdate.ecowitt.net/data/report/. Because I use the console for internal use only all communication with the outside world is blocked by my firewall (*). That means the console can't phone home and probably hangs the thread until a timeout passes. My guess is that every few minutes the send time of the custom upload overlaps with that timeout period and pushes it back until the timeout passes.

This theory is confirmed by redirecting cdnrtpdate.ecowitt.net to my own internal webserver via a DNS override. I put an empty page at /data/report/. That seems to satisfy the phone home service as the sending irregularity of the custom upload was now completely gone. I still had to use 59 seconds to get the interval to 1 minute and over a day it still drifts 4-5 seconds, but that's acceptable.

(*) I know the Ecowitt consoles use online NTP servers for time synchronization (pool.ntp.org and time.nist.gov?) so blocking all Internet communication is problematic for accurate timestamping. I solved that by redirecting all traffic on port 123 (NTP) to my own local NTP server.


The phone home occurs precisely every 61 seconds (really, there must be a +1 bug somewhere in the Ecowitt interval code) and the message format is the typical Ecowitt message that starts with the PASSKEY. There's no "confidential" information in there but the ecowitt.net upload should be off according to the settings. This proves you can't trust IOT devices. Hence why I block all Internet access by default unless required.

I'm a bit puzzled by the ecowitt.net settings in WS View. As an experiment I enabled the ecowitt.net upload with a 2 minute interval. However, this did not seem to have any effect. The console still uploaded every 61 seconds. It's as if the screen in WS View does not work at all (it does work: the settings persist when I close and reopen WS View). Is this a bug in the console firmware or is this because I redirected cdnrtpdate.ecowitt.net/data/report/ to an empty page? Does the console expect a reply from ecowitt.net?


I also discovered a bug in the old WS View app for iOS (fixed in WS View Plus): Changing the settings of the ecowitt.net upload erases the path of the custom upload.


Anyway, the conclusion is that to get accurate intervals for a custom upload you must

• subtract 1 second from the interval (thus a 1 minute interval must be specified as 59 seconds)
• make sure http://cdnrtpdate.ecowitt.net/data/report/ can be reached - either the real site or a dummy substitute

Note that when you have other weather services enabled, downtime or slowness of these services may impact the timing. I only tested the custom server.

This information applies to the WH2650 console with firmware 1.6.8 and very likely also the GW1000/1100. The behavior may be different with other consoles and firmwares. I can't test that - I only have this WH2650.

[WA4OPQ]

Yes, we found the same problems with the GW1000 last year.
I have a solar powered weather station out in a cornfield, it needed a webserver for /data/report and a timeserver added.
All the data was going through WeeWx so I didn't observer the 61 second issue.
Glad it's now working well for you.

[hiljo]

Good to see all your replies! I'm on holiday currently, so not checking the forum everyday as you can imagine  . Although it is very nice that I can view the weather at home with my Ecowitt devices

First of all I'd like to say that the drifting in reporting time is not a problem for me. I was just wondering if this is normal behaviour or not when I looked at database records a while ago. But according to all your replies, it is 'normal' in Ecowitt terms 

I then looked at the network traffic. All weather services are disabled in WS View (apart from custom upload) but the console still tries to phone home to http://cdnrtpdate.ecowitt.net/data/report/. Because I use the console for internal use only all communication with the outside world is blocked by my firewall (*). That means the console can't phone home and probably hangs the thread until a timeout passes. My guess is that every few minutes the send time of the custom upload overlaps with that timeout period and pushes it back until the timeout passes.

Interesting findings from your side! Although you are talking about a WH2650, I think this also applies to my device some sort of way. I didn't know the device 'phones home' every minute or so. I didn't check it with my device, but I can imagine mine would do the same (I guess Ecowitt 'reuses' code for different devices for development and maintenance purposes).

I would be interested in more data from others. I do believe you are running into a limitation on the MPU or CPU and the code that is in use or applicable.
These units (consoles) are pretty much single threaded and that involves loop time on sensor input from the radio receiver and also from outbound calculations and connections.

Summing up your suggestion and the findings of Mahi, it does seem that at least the upload function is single threaded. It first phones home, and also uploads to other weather services and finally (at the end of the loop?) it uploads to the custom weather server.

What I see in my records is (I'm also uploading to WU and WoW UK), that it drifts more than a few seconds per day. In two hours, it drifts about 7 seconds, and also jumps like 5 times from e.g. hh:mm:40(ss) to hh:mm:56(ss). I'm guessing that if one of the other reports to e.g., Ecowitt or WU goes wrong or takes longer, the other processes are also delayed.

From a technical point of view, I'd suggest that the reporting now works based on a 'waiting time' between the loops, but it would be better to work like a 'cron job' way, activating when the clock in the device is at a certain time.

But would Ecowitt be open to work on this? It is not a real issue offcourse, but would make the upload (and also errors when occurring) more stable.

[hiljo]

Today I received an answer from Ecowitt support.
At first they replied that the data is sent to internet, and therefore its timing is not precise. They can’t guarantee an exact 5 minutes interval.

I wrote back that the entered number just isn't very accurate. It's not just the time in the messages that is off. The device really sends at irregular intervals. So when we enter eg. 60 seconds, yet the actual interval often fluctuates between 45 and 75 seconds (as we already discussed here).

Currently I'm also beta testing the Wittboy including the GW2000 gateway. The interval I set on this device is perfectly used (600 seconds). So it seems to be possible, although the GW2000 is a different device then the HP2550 offcourse.

Today Ecowitt answered:

Our software team are looking at this time interval issue on HP2550 console. We should be able to find a solution for this. Please hang on until we fix this problem.

So fingers crossed!

[Gyvate]

I made a similar comment in another thread  (FOSHKplugin) regarding this situation.
I made a series of tests myself - all inside the local network so no bandwidth or receiving server overload etc. influencing the traffic processing at the receiver end.
The test was done with 4 consoles:
WS2320E
HP2551
GW1000
GW2000
measuring instruments:
a. FOSHKplugin log entries
b. weewx interceptor driver
result:
1. the GWx000 consoles keep the timings set up in WS View (with one second more systematically 16-->17, 32-->33, 60-->61 ...)
2. the WS2320E takes about double the time than set up in WS View but constantly
3. the HP2551 takes not only more than double the time but posts in a strange rhythm: e.g. 32, 32, 44, 32, 32, 44 ...
or 32, 28, 32, 28, ... etc.

The reply from Ecowitt was, that the GWx000 console have their own WiFi SoC with their own extra processor whereas the other consoles have a WiFi modem. The modem activities are (also) handled by the same processor which does all other work in the console. So the other consoles seem to have an internal performance issue (not to exclude that the modem has in addition a bandwidth issue).

I conclude that the fact that the WS2320E was faster than the HP2551 comes from the payload size. WHile the WS2320E handles only the basic sensors rain, wind, T/H [in/out], solar, the HP2551 has to handlemy whole other fleet of sensors - 30-40 readings more including the sensors' battery status.
An explanation for the x, x, y, x, x, y ...transmission pattern I haven't found yet.

[Nekota]

I am using a Ambient WS-5000 and exploring the custom server settings.  My display device looks like the same as the HP2553 display.  I have setup the custom weather server to use the Ambient setting and used a raspberry pi with a python script to capture the data stream.  I used the 16 second interval and find the timing to be erratic.  To better understand if I was doing something wrong with my custom socket server script I used tcpdump to monitor the WS-5000.  First I was surprised to find the WS-5000 sends a UDP packet around 45 times per minute always 45 bytes long.  Second I found the TCP Get request is output 16 to 32 seconds with about half at 16 and the other half close to 32.  When I increased the interval time to 20 seconds the output requests were mostly at 32 seconds.  Further testing with intervals greater than 32 seconds shows the WS-5000 likes to use intervals spaced at 16 seconds resulting in 32, 48, 64 intervals rather than the 1 second settings. Setting the interval to say 50 seconds results in an average 64 second  interval. 

To capture data I used a raspberry pi with tcpdump.  I set the WS-5000 to send to the raspberry pi IP address at port 23456.  To capture the UDP broadcasts a monitor shell with command sudo tcpdump src 192.168.xxx.xxx and udp > udpws5000.txt will capture the udp summary information with microsecond timing.  The udp packets are separated in time by 1.84 and 2.04 seconds.  The data in the UDP packet does not appear to change. It does have AMBWeather-WIFIECD3.V4.3.2 embedded in it.
sample udp packet
22:11:49.041499 IP esp-13ecd3.lan.2525 > 255.255.255.255.59387: UDP, length 45
   0x0000:  4500 0049 45fc 0000 8011 ddd3 c0a8 562c  E..IE.........V,
   0x0010:  ffff ffff 09dd e7fb 0035 8313 ffff 1200  .........5......
   0x0020:  2bd8 f15b 13ec d3c0 a856 2caf c81a 414d  +..[.....V,...AM
   0x0030:  4257 6561 7468 6572 2d57 4946 4945 4344  BWeather-WIFIECD
   0x0040:  3320 5634 2e33 2e32 14                   3.V4.3.2.

To capture the TCP data stream I use sudo tcpdump -X -vv port 23456 > tcpws5000.txt which shows the hex and ASCII (-X) and verbose packet description.  I use python scripts to extract timing data from the udp and tcp capture files.

What I have discovered is  the TCP packet is a GET request with 3 headers and a message. The headers are Host: 192.168.xxx.xxx (raspberry ip address), Connection: close, and User-Agent: esp8266.  The message part of the TCP packet has GET followed by the path setting then a series of & delineated fields with data delineated with = .  The data matches what is displayed on the WS-5000 console as where // is the path setting and MAC address as the PASSKEY.  The barometer units are inches while the WS-5000 is in hpa.  The hourlyrainin shows the WS-5000 Rate variable rather than the Hourly accumulation value.  The last value reports batt_co2 which I don't have but I suspect is the rain gauge battery level which is otherwise missing.



sample decoded TCP packet
//
stationtype AMBWeatherV4.3.2
PASSKEY D8:F1:5B:00:00:00
dateutc 2022-02-26+19:50:52
tempinf 68.5
battin 1
humidityin 46
baromrelin 30.343
baromabsin 29.758
tempf 46.6
battout 1
humidity 65
winddir 63
winddir_avg10m 86
windspeedmph 3.1
windspdmph_avg10m 3.4
windgustmph 5.8
maxdailygust 16.6
hourlyrainin 0.000
eventrainin 0.000
dailyrainin 0.000
weeklyrainin 0.122
monthlyrainin 1.602
yearlyrainin 5.350
solarradiation 174.90
uv 1
batt_co2 1

I hope the information can be helpful and I welcome any suggestions to help understand the custom server settings.

[olicat]

Hi!

I welcome any suggestions to help understand the custom server settings.

You should keep an eye on my EAR.

And of course you could also look at FOSHKplugin, which receives and processes/forwards the data sent via custom server.

Oliver