Topic: Operation of the GW1000 without internet - time server and WIFI stability

[vinceskahan]

Hi!

the need to phone home bothers me quite a bit

So, please write a request to Ecowitt.

Would love to...what's the preferred way to send a nice request and explanation so it gets to somebody who cares there ?

[WA4OPQ]

Would love to...what's the preferred way to send a nice request and explanation so it gets to somebody who cares there ?

Send a note to Lucy at support@ecowitt.com

[WA4OPQ]

Oliver,

I used your Wireshark filters to great success.
I tried a small server, micro-httpd today, it served up pages but I didn't see 200 and it didn't stop the reboot.
I returned to FOSHKplugin and saw the 200 response but it still rebooted the WiFi  (see attached)
Now I'm going to go back to Apache to see why it worked.
What are your thoughts or suggestions?
Bill

[WA4OPQ]

Oliver,

I set up Apache again, no virtual domains this time. It works as before, no WiFi reboots.
There must be something magical about it's 200 message, and it's 810 bytes. See attached.

Eventually I hope to find a smaller program than Apache to run on the Pi Zero; but for now I'm testing on a Pi 4B+.
I'm going to use Apache for now and move on to the NTP problem.

Bill

[WA4OPQ]

You might ask on the weewx-users google group about this one.  There might be a setting in weewx.conf or on the gw1000 driver that is there (or can be added) to help you out there.

Good advice, Thanks. 
I've been in contact with the author of the GW1000 driver, he helped me define what the rainfall problem was. A total rewrite of the driver might help, but as I'm the only one affected I wouldn't consider asking.

[galfert]

Sorry to nit pick... Raspberry Pi 4B+ doesn't exist [yet]. It is just 4B. There is a 3B and a 3B+ though.   

[WA4OPQ]

Ha! That's what happens late at night when your fleet of pi are all 3B+ and just one 4B.

I made some good progress last night. It was easier than I expected to solve the NTP problem.
I added two lines to the Chrony config file and the NTP problem disappeared.
The GW1000 jumped from 2019 to 2020 and it stopped the hammering dns requests.
I need some further testing to make sure Chrony plays nice with the RTC. By default it wants to reset the RTC.
I also found a promising webserver called Hiawatha. It's specifically designed for low power devices.
I got Hiawatha complied for the pi late last night, I'll test it this morning.

If all goes well I'll configure a new SD card and then test it in the solar station. Then wait for rain to check for any rainfall errors.
Hopefully I can wrap this up, let Ecowitt's engineers know and I'll post a pdf of the procedure here.

[vinceskahan]

Thanks for the updates...

Definitely 'please' post exactly how somebody can recreate your success.  Not much is uglier than finding a post with the exact question you wanted to ask, and the original poster follows up with "never mind I solved it".  Grrrrrr....

[...suggestion mode on...]

You don't need to go chrony if you want to do old-school ntpd on your pi.   Just do:

• sudo apt-get install -y ntp
• sudo systemctl enable ntp
• sudo systemctl start ntp

There's a feature in systemd where it will not run its internal (lame) time server if ntpd is present and enabled.

For a pi, just run nginx and get it over with.  It's easily light enough for even an old model-B pi, and there are 'many' howtos for just about everything available.  If you stick with a mainstream webserver it'll be easier to get help if you need some special configuration....

I'll do that email to Lucy tomorrow to add to the request list for the stuff you found so far.....

[WA4OPQ]

More success tonight.
I gave up on Hiawatha, my port to raspbian wasn't perfect and it operated intermittently.
So at the moment I'm using Chrony and Lighttpd. I installed it on a pi zero and it's running just fine. A lot going on for a low horsepower board.

The syslog is quite boring compared to before.

Vince, I've just about finished the pdf so it'll be documented. It was a coin toss between lighttpd and nginx, and I'd already installed chrony.

I have other things I want to try, but this is complete enough that I can install in in my remote solar powered station for testing.
And about the time I finalize everything then Ecowitt will respond to our email requests with a new firmware update that will change everything. 

[olicat]

Hi!

Apparently the GW1000 is testing the availability of the WIFI-connection by checking every 10 minutes whether a connection to cdnrtpdate.ecowitt.net was successful.
If not, the WIFI connection will be re-established as a precaution.

The behavior of the internal watchdog of the GW1000 now finally seems to be clarified.
The GW1000 expects a positive response (return code 200) and a string as a reply from http://cdnrtpdate.ecowitt.net/data/report/ within 10 minutes. The Ecowitt server itself returns the string

Code: [Select]

{"errcode": "0", "errmsg": "ok", "UTC_offset": "3600"} together with the return code 200, but any string will be accepted.
If this does not happen within 10 minutes, the GW1000 restarts the WIFI connection.

It would be interesting to see whether this behavior also occurs with the HP2551C. Has anyone noticed constant WIFI reconnects there when not sending data to Ecowitt?

Oliver

[vinceskahan]

It would be interesting to see whether this behavior also occurs with the HP2551C. Has anyone noticed constant WIFI reconnects there when not sending data to Ecowitt?

Somebody buy me one and I'll put a sniffer online in two minutes afterward

You might try running pihole and at least looking at the DNS requests, which might help a little in figuring out which FQDN it will likely try to access.   You'd need to point your gear at that for DNS of course.

(sent email to Lucy adding my voice FWIW to the requests for a watchdog on/off feature)

[WA4OPQ]

I am pleased to report that my system is now running perfectly. No WiFi reboots and the GW1000 has the correct time and date.
Many thanks to Oliver for his assistance and his modifications of FOSHKplugin.
My configuration uses WeeWx, the new GW1000 API extension, Hostapd, DNSmasq, Chrony and FOSHKplugin.
I am attaching a PDF with more information and installation procedures.

The new version of FOSHKplugin will be available soon.

[WA4OPQ]

The last remaining item to test was rainfall. When the GW1000 did not have a time server the CWOP readings of 24 hour rain and Rain since midnight were corrupted.
We've now had a rainstorm and all is working well. I'm pleased this project has went well and I'd like to thank again all who have helped me complete this.

[galfert]

Excellent work and nice documentation.

[Gyvate]

......
My configuration uses WeeWx, the new GW1000 API extension, Hostapd, DNSmasq, Chrony and FOSHKplugin.
I am attaching a PDF with more information and installation procedures.

The new version of FOSHKplugin will be available soon.

Hi Bill
nice work 
One question, what exactly are you referring to with the "new GW1000 API extension" - a version of the ecowitt-client/fineoffset-bridge interceptor ? (Which one ?)
Please advise. 

[WA4OPQ]

Hello,

This is the extension that I am referring to:
https://github.com/gjr80/weewx-gw1000

[Gyvate]

Hello,

This is the extension that I am referring to:
https://github.com/gjr80/weewx-gw1000

Thanks - great stuff !

So if I want to use that driver and am already using the Interceptor driver which I want to replace, I have to comment-out (= #) the active entries in the Interceptor section ([stanza]) or even the complete [stanza], correct ?
And add the respective section(s)/[stanza] mentioned in the above github weewx linked readme.md ?

Otherwise (with both entries active) what would happen ? Weewx would be confused (or would just take the first (?) or last (?) entry/[stanza] it encounters and just use that one) ? Just to know out of curiosity.

[WA4OPQ]

Actually, I'm not the right person to answer this. I've never converted from one extension to another. I usually do a clean install and start over.

I'm sure others here can answer.

[Vetti52]

In the GW1000 explanation there is one action not mentioned, when you want to install manually:
Change an entry in the stanza

Code: [Select]

[Station]
    # Set to type of station hardware. There must be a corresponding stanza
    # in this file with a 'driver' parameter indicating the driver to be used.
    station_type =  Interceptor
into

Code: [Select]

    station_type =  GW1000
You do not need to delete the other entries. But, be sure, you have updated the GW1000 part in weewx.conf, and tested the functionality, according to the readme.txt provided with GW1000 driver

[Gyvate]

In the GW1000 explanation there is one action not mentioned, when you want to install manually:
Change an entry in the stanza

Code: [Select]

[Station]
    # Set to type of station hardware. There must be a corresponding stanza
    # in this file with a 'driver' parameter indicating the driver to be used.
    station_type =  Interceptor
into

Code: [Select]

    station_type =  GW1000
You do not need to delete the other entries. But, be sure, you have updated the GW1000 part in weewx.conf, and tested the functionality, according to the readme.txt provided with GW1000 driver

That was a good hint - thanks ! in "GW1000" the "GW" has to be written in capital letters. I had it in lowercase first (like the driver name) and it wouldn't work.
This I figured out rather fast.
So far so good - and ALMOST perfect.
What has happened after the switching from Interceptor to API is that my solar radiation data disappeared. Everything else is there. (And the data still appear in Ecowitt.net, in my Meteobridge etc. - so it must have to do with the new setup). But I haven't figured out where. I have compared the number and names of the Accumulator entries - they are the same in the weewx.conf as in the readme file. That was my first suspicion.
Now I'm stuck.

 [ You are not allowed to view attachments ]

The data collected from the GW1000 (I also switched between two to make sure that one of them isn't "lazy" in providing data).
 [ You are not allowed to view attachments ]
I assume that that's the full list of sensors and values - and I assume that the radiation data is "sitting" in 'uvradiation' or/and in 'luminosity" (as Lux are to be converted into W/m²).

The timing when the radiation data stop showing in the graph is exactly the switching time between Interceptor and API.

Any suggestions, hints, .... how and where "my" radiation got lost ?

[WA4OPQ]

Any suggestions, hints, .... how and where "my" radiation got lost ?

I recently created a small weewx report. I tried several different ways to get solar radiation with no success. I was able to get luminosity so I did the conversion to watts/m².

The author of the extension, gjr80, is active on https://groups.google.com/g/weewx-user. He could best answer your questions

[Gyvate]

Thanks for the hint - I found him while digging in Google - and posted the issue with the logs etc. there.
Let me see what it gives. Will share the results.

By the way - afaik luminosity is the field - provides data in Lux, need to be divided by 126.something to get the W/m².
The exact number used by the firmware is mentioned somewhere in the manual. It refers to visible light of a certain frequency.
That's what the console firmware does.

What is strange that it worked well with the interceptor and not with the API driver (but I have also seen that there is a newer version 0.2.1b under construction, and the officially available is still 0.1.012b. Obviously there is still some bug fixing going on).

Probably it's easier with the interceptor as the console firmware does already the work and posts it in Ecowitt protocol while the API driver still has to manage this itself.

[navzptc]

By the way - afaik luminosity is the field - provides data in Lux, need to be divided by 126.something to get the W/m².
The exact number used by the firmware is mentioned somewhere in the manual. It refers to visible light of a certain frequency.
That's what the console firmware does.

@Gyvate is correct - you just need to add the following line under [[Corrections]] in the [StdCalibrate] area in weewx.conf file.

Code: [Select]

#   This section can adjust data using calibration expressions.

[StdCalibrate]

    [[Corrections]]
        # For each type, an arbitrary calibration expression can be given.
        # It should be in the units defined in the StdConvert section.
        # Example:
        radiation = luminosity/126.7 if luminosity is not None else None

I came across it a while back in the Weewx google group when I had the same issue! 

[Gyvate]

By the way - afaik luminosity is the field - provides data in Lux, need to be divided by 126.something to get the W/m².
The exact number used by the firmware is mentioned somewhere in the manual. It refers to visible light of a certain frequency.
That's what the console firmware does.

@Gyvate is correct - you just need to add the following line under [[Corrections]] in the [StdCalibrate] area in weewx.conf file.

Code: [Select]

#   This section can adjust data using calibration expressions.

[StdCalibrate]

    [[Corrections]]
        # For each type, an arbitrary calibration expression can be given.
        # It should be in the units defined in the StdConvert section.
        # Example:
        radiation = luminosity/126.7 if luminosity is not None else None

I came across it a while back in the Weewx google group when I had the same issue! 

@ navzptc
ooops - when I first read your reply, I thought it was meant for WA4OPQ and his issue. Especially as you quoted that part of my post regarding how the consoles create radiation values out of the luminosity they measure.

So, even though I thought that your remark was not for me, just confirming my statement about luminosity,  I still sort of "just for fun" added that line in the [corrections] stanza of the weewx.conf file - and, surprise, surprise, my solar radiation is back. 

 [ You are not allowed to view attachments ]

Thanks for that hint - that really worked !! 

But then, I have no clue what that is supposed to have to do with calibration (but it's probably the if-clause that does the job  ) ....
(weird looking logics - we call this in my language "shooting or stabbing someone from behind through the chest in the eye" if you see the picture/metaphor  )
Or is it just a possible place where to correct a missing feature that should have been done somewhere else in the coding earlier ???
[update, after receiving the mail from the weewx GW1000 API driver developer (see next post) - it's a feature the way weewx works, not a missing one, but a correction like the above mentioned one is needed]

So far I had only an entry there to correct my relative pressure - as somehow the console calibration didn't change the value in weewx. I have the same situation in my Meteobridge - correction/calibration needed after the data logger has read the values from the GW1000. So the console calibration seems to become effective only for sending the data to Ecowitt.net (or to a custom server).
But that's just a side remark - otherwise I'm moving rather off topic I'm afraid.

[Gyvate]

For those interested in the "official" reply of the developer of the weewx GW1000 API (Gary) - below the reply to my enquiry in the weewx forum:
(the details may be enlightening for some of our GW1000 API "deep divers" - and others too  )
-----
gjr80 <gjroderick < at > gmail.com>
   
8:34 AM (2 hours ago)
   
to weewx-user
Hi,

The short answer is that radiation (aka solar insulation) or an equivalent is not available via the GW1000 API so the GW1000 driver does not emit it. The reason you see it on Ecowitt pages and WU is that the GW1000 takes what the GW1000 API refers to as 'light' in lux and approximates solar insolation by dividing by 126.7. The GW1000 then uploads this derived value to Ecowitt and WU (and to anywhere else the GW1000 is configured to upload). A similar result can be achieved with WeeWX by having the WeeWX StdCalculate service calculate WeeWX field radiation.
This is done by adding an entry for radiation under [StdCalculate] [[Corrections]] in weewx.conf:

[StdCalibrate]
     ....
    [[Corrections]]
        ....
        radiation = luminosity/126.7 if luminosity is not None else None

The reason the interceptor driver picked up radiation is that the interceptor driver intercepts the the same data the GW1000 uploads to Ecowitt or WU, so the interceptor driver sees the radiation field that was calculated by the GW1000. WeeWX drivers do not calculate/derive observations; rather they decode/unit translate data provided buy the station/sensors only. Calculation of derived observations is taken care of by other WeeWX services, typically the StdCalculate service.

I can't speak for what Cumulus does but I know Cumulus obtains the exact same data from the API that the GW1000 driver does. Cumulus must be set to automatically derive solar insolation from luminosity.

As further background, the GW1000 API emits three 'radiation' related observations; what the API refers to as 'light' in lux, 'UV' in microwatts/square metre and 'UVI' an index from 0-15. Of these observations 'light' corresponds to the WeeWX extended schema field luminosity and 'UVI' corresponds to WeeWX field UV. Consequently, (and by default) these observations are mapped to the equivalent WeeWX field. There is no equivalent to the GW1000 API field 'UV' so the GW1000 driver passes this field through as uvradiation (uvradiation was chosen to distinguish it from WeeWX field UV). Note that uvradiation is not equivalent to WeeWX field radiation.

I am in the process of producing a wiki page for the driver repo to explain this.

Gary
------