Topic: Operation of the GW1000 without internet - time server and WIFI stability

[olicat]

Hi!

I just sent an email to Lucy regarding some points I figured out yesterday.
The behavior of the GW1000 should also be interesting for others. If only to better explain strange times or frequent WIFI interruptions.

Do you see it that way or am I more of an isolated case?
Maybe someone has even better solutions/ideas?

Oliver

--------------------
Hi Lucy,

is there a chance to make the NTP-server configurable for GW1000 (and probably HP2551C) someday?
Currently, cn.pool.ntp.org is hard coded in the firmware without the possibility of changing the NTP server to another NTP server - e.g. a local one.
This is particularly problematic for installations in which Internet access is only sporadically or generally not available or not desired.
When restarting, the GW1000 fetches the current time from this server via NTP. However, if it is not available, counting continues at the time 08/22/2019 10:00:00 that was automatically set when the device was started.
This leads to wrong times in the messages that are sent to local systems via the custom server.
Here it should definitely be possible to configure an NTP server yourself!
Optionally, the GW1000's DHCP client could also use the correct time server transmitted from the DHCP server using option 042 of the DHCP protocol.

During the tests with the self-sufficient operation of a GW1000, I noticed another unsightly oddity:
Apparently the GW1000 is testing the availability of the WIFI-connection by checking every 10 minutes whether a connection to cdnrtpdate.ecowitt.net was successful.
If not, the WIFI connection will be re-established as a precaution.
Because this happens every 10 minutes, the WIFI connection appears to be very unstable - although everything is actually ok. Datagrams that would have to be sent during the period of the DHCP negotiation are not sent.
Loss of data does not improve the reputation!
But even if the Internet is available, the WIFI connection is re-established every 10 minutes if sending to ecowitt.net is not enabled.
This compulsion to send to ecowitt.net is certainly not planned and an error in the firmware (tested here with v1.6.1).
Here it should be checked whether any successful connection (http return code 200..203) has been made within the time interval - regardless of the destination.
This should also include the transmission via the custom server!

I am happy to hear from your engineers about the possibility and period of time for these firmware revisions.

Sincerely, Oliver

[galfert]

I already have brought this issue up with Ecowitt. Here is their response.

This is designed in the firmware. It will restart after 10 minutes without network. This can solve the problem of network abnormality when connecting to some routers.

It's waiting for the upload of rtpdate.ecowitt.net to successfully reply 200 ok.
You can simulate this reply, and if successful, the wifi LED will not flash.

The reboot is not NTP related. I thought it was NTP related when I asked them too. I also thought that there was some sort of heart beat ping to sense connectivity but also not exactly. According to their engineers it is purely a check with Ecowitt.net. Even if you turn off Ecowitt.net upload it seems. In a way this 200 OK is the same as my predicted ping sentinel.

[WA4OPQ]

Thank you for your assistance in this. It's very much appreciated.

Oliver,
When testing, did you observe what port was used to connect to cdnrtpdate.ecowitt.net?

Thanks again.

[WA4OPQ]

I don't think the time and NTP is an important issue. The only time that having the wrong time on the GW1000 would be critical is if it was directly sending to online services as WU. It definitely is not. It's able to send the data to WeeWx and WeeWx has the correct time.
I think the reset issue is a much higher priority.

[olicat]

Hi!

When testing, did you observe what port was used to connect to cdnrtpdate.ecowitt.net?

It is port 80/http as POST.

send the data to WeeWx and WeeWx has the correct time.

Yes, for weewx this should not be a problem but what if you want to feed another program than weewx like PWT or PWSDashboard?
I think the NTP server should generally be configurable.

Regards, Oliver

[WA4OPQ]

what if you want to feed another program than weewx like PWT or PWSDashboard?

I'm not familiar with PWT or PWSDashboard, but wouldn't they derive their time from the computer they are operating on? Or do they get the timing from the GW1000?

[olicat]

Hi!

wouldn't they derive their time from the computer they are operating on?

Of course!
But it's about how the software handles the time stamp of every message from the weather station.
With FOSHKplugin, I have so far also relied on the incoming time stamp and adopted it for further processing of the messages.
Only with the upcoming version can FOSHKplugin be configured in such a way that it uses the reception time instead of the transmission time of the weather station.

Or do they get the timing from the GW1000?

In fact, I haven't taken a closer look at how PWT and PWSDashboard handle this. So that was really meant as a question.
But there are two completely different approaches to using the "real" time of the measurement or the time of arrival of these measured values.
I would always prefer the time of the actual acquisition on the GW1000.
But if you can't rely on a correct time there, you have to find other ways and adapt the software accordingly.

Regards, Oliver

[WA4OPQ]

I've spent a couple of days working on this. I added Apache2 and modified Dnsmasq.
I can bring up cdnrtpdate.ecowitt.net on my hotspot and I see the default index page, using the browser on my cell phone.
So it looks like I've successfully spoofed the hotspot.
However, the GW-1000 is not fooled and continues to reboot every 10 minutes.
Back to the drawing board. I'll keep you informed.

[olicat]

Hi!

However, the GW-1000 is not fooled and continues to reboot every 10 minutes.

Have you also activated the upload to ecowitt.net? Only then the watchdog apparently is satisfied (if it gets a 200 from remote).
As far as I could see here, the GW1000 does not boot completely but re-establishes the WIFI connection.

Oliver

[Old Salt]

Have folks thought of redirecting NTP and the heartbeat to a local server?  We may be able to do this with the hosts file. (windows/system32/drivers/etc/hosts on Windows systems)

The ability to change this at the Ecowitt console or hub would be ideal, but the above may be a work-around until Ecowitt changes things.

[WA4OPQ]

That was my plan of action.  For now I ignored the NTP problem and was more concerned about the WiFi rebooting.
My raspberry pi is running as a WiFi hotspot, I set up Apache2 for a html server and used dnsmasq to spoof the host address. It worked for a browser but not the GW1000 heartbeat.

I'll devote another rainy day soon to try again. I'm going to dig into the NTP protocol and see if I can have Apache send it something it likes.

[WA4OPQ]

I can report success with the first part of the problem. I've solved the problem of the ten minute WiFi reboot.
I set Dnsmasq to spoof cdnrtpdate.ecowitt.net and 0.cn.pool.ntp.org to my pi server
I set up Apache2 with two virtual domains
I found that cdnrtpdate.ecowitt.net needed /data/report/index.html  Once I stopped the 404 it was happy and didn't reboot.
Now I want to go back and use a more lightweight server like NGINX or Lighttpd

I'm learning Chrony, it looks like it will work as a time server.
I originally thought that it would matter little if the GW1000 time was off. I could not have been more wrong.
I'm using the data out of WeeWx for CWOP, two of the rainfall measurements use the GW1000 time and are totally wrong. The rainfall after midnight never resets.

I'll let you know what I find and how it is progressing.

[galfert]

Nice work.

Does your Pi have an RTC module?

[olicat]

Hi!

I am annoyed that I have not carried out my suggestion for the local web server. Please forgive that!
Of course you can just put an index.html under /report/ to get an http return of 200:

(if it gets a 200 from remote)

Sorry, I just assumed that.
But you don't necessarily need a complete web server like Apache or nginx or ...
A small tool that listens on port 80 and acknowledges all requests with 200 would probably be sufficient:
https://gist.github.com/bradmontgomery/2219997
https://gist.github.com/mdonkers/63e115cc0c79b4f6b8b3a6b797e485c7

Ideally, however, you have a web server running on this machine anyway, then this one page doesn't bother you either.
Depending on what is possible with dnsmasq, incoming requests on port 80 could also be redirected to another port on the server.
So you could run a dummy web server on a different port.

The ntp story shouldn't be that difficult to realize either.
You install a client so that the Raspi actually receives the correct time over the network and distributes this to requesting clients via the server.
This looks suitable for your purposes: http://raspberrypi.tomasgreno.cz/ntp-client-and-server.html.

If your time server is also not conncted to any time source just use a RTC and pray for a long battery life.

Oliver

[WA4OPQ]

Good Morning!

Yes, Galfert, I have a RTC. I chose a high precision model as I needed one with temperature compensation for outdoor use.

Oliver, thanks for the suggestions for the small webserver tools. I wasn't aware of those and that is what I actually need for a Pi Zero. I'll be trying those out right away. I started my test with Apache only because I'm more familiar with it.  I'll also check out your NTP suggestion.

I'm still new to Wireshark, so it's probably due to my inexperience, but I'm still not seeing a 200 sent even though I know it's supposed to be there. I saw a 404 returned until I created the proper directory, I expected a 200 to replace it but it didn't.

While I am indoors this fall my hardware is outdoors testing to see if my solar panel is adequate for the shorter days. It is close enough to my house that I can connect to it's WiFi from my desk.

 [ You are not allowed to view attachments ]

[olicat]

Hi!

You could also use FOSHKplugin as the webserver on Port 80. Nice side effect: you'll get all the data actually intended for Ecowitt and don't necessarily have to configure the custom server.

Oliver

[WA4OPQ]

Oliver,
I'm going to install FOSHKplugin tonight.
I see how it responds to different URLs
I've studied the configuration file and can't see a way to add my " /data/report/index.html"
Any suggestions?

[olicat]

Hi!

I've studied the configuration file and can't see a way to add my " /data/report/index.html"

You don't need that with FOSHKplugin, because it waits internally for POST requests that contain "report" and responds accordingly.
It takes the request, responds with a 200 and processes the incoming data; generates corresponding forwards or saves them as CSV.

At the moment there is unfortunately still an Exec function missing in FOSHKplugin. But I already have this planned. The aim is that the incoming line can be used as a parameter for any program or shell script to be started. I think this function would probably be extremely helpful for your application scenario.
Until then, however, you could set up a cron job and start a shell script in it, which fetches the data in a format that you can process. Have a look to Saving or further processing of incoming data sets (set-wise) in the recipes section.

Oliver

[WA4OPQ]

Oliver
Thanks for your help.
I installed FOSHKplugin tonight and got it working as I was able to see the data on the URLS. I set it for 80 rather than 8080. Unfortunately it did not stop the WiFi reboot. Wireshark did not show a 404 response (but I've never been able to see the 200).
Any suggestions?  I confirmed that dnsmasq is spoofing the ecowitt.net address.
Bill

[olicat]

Hi Bill,

Wireshark did not show a 404 response (but I've never been able to see the 200).

a Wireshark-filter to see all POST and their 200-responses from remote could be:

Code: [Select]

http.request.method == "POST" or http.response.code == 200If you want to see all responses you just have to modify the filter a bit:

Code: [Select]

http.request.method == "POST" or http.responseHave a look to the attachment.

I see a difference between the response from FOSHKplugin and from Ecowitt (a line "{"errcode":"0","errmsg":"ok","UTC_offset":"7200"}" is returned there) but I can do not imagine that this matters.
After all, your Apache installation probably only sends back a 200 and not this string. And there - as you told - the WIFI-reboot does not appear.
Strange ...
Perhaps your Wireshark analysis will help us.

Oliver

[zoomx]

The ntp story shouldn't be that difficult to realize either.
You install a client so that the Raspi actually receives the correct time over the network and distributes this to requesting clients via the server.
This looks suitable for your purposes: http://raspberrypi.tomasgreno.cz/ntp-client-and-server.html.

If your time server is also not conncted to any time source just use a RTC and pray for a long battery life.

I believe that a GPS NTP can do the trick. Maybe also a old Android with a GPS NTP server app.

[WA4OPQ]

Hello Oliver,
I will try your filter tonight. I now see that my problem in seeing the 200 is my inexperience with Wireshark. I was just filtering for packets including "report". Thank you again.

Hello Zoomx  My application is a solar powered remote weather station. It currently uses 3.5 watts while using a battery operated RTC. I don't think I can afford the power consumption of a GPS device.

[vinceskahan]

I originally thought that it would matter little if the GW1000 time was off. I could not have been more wrong.
I'm using the data out of WeeWx for CWOP, two of the rainfall measurements use the GW1000 time and are totally wrong. The rainfall after midnight never resets.

You might ask on the weewx-users google group about this one.  There might be a setting in weewx.conf or on the gw1000 driver that is there (or can be added) to help you out there.

I just got my gateway and outside temp/hum sensor this week and am pretty impressed with them so far, but the need to phone home bothers me quite a bit.  Fortunately it seems like it can be faked with a pihole (which many of us have) and a small ntp+web server (to add to the pihole pi) but it's still distressing.

It would be great if there were 'advanced settings' where we could turn the ntp and web needs off for standalone cases.  Seems to me that it would be an easy feature switch to add to their firmware. But I 'did' like the auto firmware update on first boot right out of the box.

My concern is more the IoT typical thing - if they go out of business or their website breaks badly did that brick my devices that were working just fine before 'their' side had a service issue.   I'm fine with handling 'my' weewx side, but that isn't going to be affected by EcoWitt service interruptions or Internet connectivity issues between here and there...

[zoomx]

Hello Zoomx  My application is a solar powered remote weather station. It currently uses 3.5 watts while using a battery operated RTC. I don't think I can afford the power consumption of a GPS device.

I am talking about these GPS
https://www.aliexpress.com/item/1872389528.html

Another option is a WWVB (or DCF77 in Europe) receiver.

Maybe you can use an cheap USB-SDR receiver and RTL_433 and intercept the data from sensors. But I believe that they use a lot of power.

[olicat]

Hi!

the need to phone home bothers me quite a bit

Please you and everyone else who do not want this forced connection to Chinese servers: please complain to Ecowitt!  Technically there is no need for it.  The more users point this out to Ecowitt, the more likely there is a change. Ecowitt is known for responding to user requests.
The configurability of the NTP server is somewhat more complex to implement.  But here, too, Ecowitt should not only be informed of this request by a few but by as many users as possible.
So, please write a request to Ecowitt.
Thanks!

Oliver