Author Topic: Belchertown skin - MQTT  (Read 520 times)

0 Members and 1 Guest are viewing this topic.

Offline gszlag

  • Senior Contributor
  • ****
  • Posts: 249
  • ..have you calibrated your barometer today?
    • Michael's Bay - Manitoulin Island weather
Belchertown skin - MQTT
« on: January 05, 2022, 03:40:19 PM »
After some T&E, I've got this skin running on my linux test bed machine with MQTT.

Equipment is Ecowitt WS68, WH40, WH32.

- archive interval = 60 seconds,
-  GW1100 poll interval =20 seconds.
- web sockets used but not TLS.
- MQTT uses user:password.
- eventually figured out you need to install a local web server to see graphs..used nginx.

Everything is connected and updating in real time on my local ip with a local installed MQTT broker.

Now that I know this set up is running satisfactorily, I would like to upload this to my commercial web host. Nothing fancy, just want to add the broker and the skin to my existing WeeWX/GW1100 set up on my new rpi zero 2W, and away we go...or not...
 
After doing a lot of searching, user groups, etc. I still could not find how to configure the skin/MQTT for a remote web host.

And the more I read..the less I know.

Questions.

Ok. I have a local MQTT broker running. How is a local MQTT broker going to update the skin in realtime on the remote host when it is sitting on my local machine? I am guessing with this configuration, it would update only at the archive interval. Can't get there from here, so to speak.

Maybe use a public MQTT broker to update the skin at the commercial web host? Looked at HiveMQ but TLS mandatory? Would rather avoid TLS if I can for now.
Other public MQTT brokers.?.reliability?

Any cues and clues would be greatly appreciated.
Ecowitt GW1000/GW1100
Ecowitt WS68: Anemometer, UV/solar
Ecowitt WH40: Rain gauge
Ecowitt WH32E: Outside T & H sensor
Ambient Weather WS-2000
Stratus Rain Gauge (manual)
Raspberry Pi 3B+ (WeeWX/CumulusMX)
Raspberry Pi Zero 2W WeeWX/MQTT/Belchertown)
---
http://weather.glenns.ca (pwsdashboard - live)
http://weewx.glenns.ca
http://glenns.ca/cumulusmx2/index.htm
---
WU: ITEHKUMM2
PWSweather.com: MBAYWX1
AWEKAS:id=15920
WINDY: Michael's Bay, Manitoulin Island
WOW: Michael's Bay

Offline davidefa

  • Forecaster
  • *****
  • Posts: 313
Re: Belchertown skin - MQTT
« Reply #1 on: January 06, 2022, 09:49:07 AM »
Maybe use a public MQTT broker to update the skin at the commercial web host? Looked at HiveMQ but TLS mandatory? Would rather avoid TLS if I can for now.
Other public MQTT brokers.?.reliability?

Any cues and clues would be greatly appreciated.

As free public broker you can use:
broker.hivemq.com
broker.emqx.io
test.mosquitto.org

Why TLS mandatory on hivemq ( don't think so )? On the free broker you don't have user/password authentication, but you can always simulate it using a topic prefix like /user/password ( well sorta )

I had some T&E too with mqtt ( but I call it HitYourHeadOnTheWall ): everything worked as expected on my local machine but it didn't on my server on the net.
The answer is simply: firewall, my hosting provider had a firewall preventing my script contacting the broker ( I found it only asking the hosting support center ).
Had to find a new hosting provider.

Offline gszlag

  • Senior Contributor
  • ****
  • Posts: 249
  • ..have you calibrated your barometer today?
    • Michael's Bay - Manitoulin Island weather
Re: Belchertown skin - MQTT
« Reply #2 on: January 07, 2022, 10:53:47 AM »
Maybe use a public MQTT broker to update the skin at the commercial web host? Looked at HiveMQ but TLS mandatory? Would rather avoid TLS if I can for now.
Other public MQTT brokers.?.reliability?

Any cues and clues would be greatly appreciated.

As free public broker you can use:
broker.hivemq.com
broker.emqx.io
test.mosquitto.org

Why TLS mandatory on hivemq ( don't think so )? On the free broker you don't have user/password authentication, but you can always simulate it using a topic prefix like /user/password ( well sorta )

I had some T&E too with mqtt ( but I call it HitYourHeadOnTheWall ): everything worked as expected on my local machine but it didn't on my server on the net.
The answer is simply: firewall, my hosting provider had a firewall preventing my script contacting the broker ( I found it only asking the hosting support center ).
Had to find a new hosting provider.

Thanks for the tips!... and the heads up regarding possible issues with a web host. I'll give them a shout just in case.
Ecowitt GW1000/GW1100
Ecowitt WS68: Anemometer, UV/solar
Ecowitt WH40: Rain gauge
Ecowitt WH32E: Outside T & H sensor
Ambient Weather WS-2000
Stratus Rain Gauge (manual)
Raspberry Pi 3B+ (WeeWX/CumulusMX)
Raspberry Pi Zero 2W WeeWX/MQTT/Belchertown)
---
http://weather.glenns.ca (pwsdashboard - live)
http://weewx.glenns.ca
http://glenns.ca/cumulusmx2/index.htm
---
WU: ITEHKUMM2
PWSweather.com: MBAYWX1
AWEKAS:id=15920
WINDY: Michael's Bay, Manitoulin Island
WOW: Michael's Bay

Offline vinceskahan

  • Senior Member
  • **
  • Posts: 85
    • home site
Re: Belchertown skin - MQTT
« Reply #3 on: January 07, 2022, 07:25:48 PM »
Getting your computed weewx html+images to your 'commercial web host' (whatever that is) is as simple as setting up some ssh keys and using the rsync uploader.  Super easy.

Getting the realtime websockets stuff to work there is a problem though.  You 'could' add a broker to your internet site and have your LAN weewx instance feed the internet host, but then you'd have websockets working on Internet only I guess.

(and be careful with security - don't let anybody other than your host write to it, and you 'will' see bot probes basically instantly there.  If you have a stable ip address at home from your ISP this isn't too hard to lock down if you're using AWS EC2 or Lightsail.)

Your question probably should go to the weewx-user google group which is the primary location for support.   I'd bet somebody has solved this one already.
« Last Edit: January 08, 2022, 05:25:41 PM by vinceskahan »
WeeWX sites:
  Davis VP2+DFARS to a Seagate Dockstar (pogoplug)
  EcoWitt GW1000, WH32 outdoor T+H, multiple WH31 indoor T+H, WH51 soilMoisture (docker)
Home site:        https://www.skahan.net/
Wunderground: KWAFEDER15
PWS:                KWFEDER15
CWOP:             CW6881

Offline gszlag

  • Senior Contributor
  • ****
  • Posts: 249
  • ..have you calibrated your barometer today?
    • Michael's Bay - Manitoulin Island weather
Re: Belchertown skin - MQTT
« Reply #4 on: January 09, 2022, 02:50:10 PM »
Getting your computed weewx html+images to your 'commercial web host' (whatever that is) is as simple as setting up some ssh keys and using the rsync uploader.  Super easy.

Getting the realtime websockets stuff to work there is a problem though.  You 'could' add a broker to your internet site and have your LAN weewx instance feed the internet host, but then you'd have websockets working on Internet only I guess.

(and be careful with security - don't let anybody other than your host write to it, and you 'will' see bot probes basically instantly there.  If you have a stable ip address at home from your ISP this isn't too hard to lock down if you're using AWS EC2 or Lightsail.)

Your question probably should go to the weewx-user google group which is the primary location for support.   I'd bet somebody has solved this one already.
Thanks vinceskahan. I would have used weewx-user but noticed there are some users here using the Belchertown skin. I have spent some days reading several dozen Belchertown posts on weewx-user but info is somewhat piecemeal and I haven't found a complete solution yet.

With my limited computing skill set, I would need to learn more of the basics. However, if I am  stuck, I would not hesitate to go to weewx-user. I've posted there in the past and the help there was more than excellent.

Not sure if got the terminology correct but what I meant about a "commercial web host" refers to companies that provide web(site) hosting  - companies like godaddy, blue host, digital ocean, etc.
I wanted to distinguish a “web host “ versus some sort of hosting with a home server.  On this forum, I see that the term  "hoster" is also  used - don't think there is a wikipedia reference for that! I use a plain jane hosting company called fullhost.com.

Your recommendation to use rsync is interesting. I've never tried it. I just use plain old ftp but it is a somewhat slow process. For the default Seasons skin, it typically takes 22 seconds on my rpi for each ftp transfer. It would be interesting to see how if rsync can speed things up.

For the Ecowitt equipment that I have, realtime for me would be about 17 seconds or a few seconds more than that. I would be happy to see the Belchertown skin updating every 17 -20 seconds as the “fastest” sensor is the Ecowitt WS68 anemometer which I think puts out data every 16.5 seconds.

Thanks for your help on this.
Ecowitt GW1000/GW1100
Ecowitt WS68: Anemometer, UV/solar
Ecowitt WH40: Rain gauge
Ecowitt WH32E: Outside T & H sensor
Ambient Weather WS-2000
Stratus Rain Gauge (manual)
Raspberry Pi 3B+ (WeeWX/CumulusMX)
Raspberry Pi Zero 2W WeeWX/MQTT/Belchertown)
---
http://weather.glenns.ca (pwsdashboard - live)
http://weewx.glenns.ca
http://glenns.ca/cumulusmx2/index.htm
---
WU: ITEHKUMM2
PWSweather.com: MBAYWX1
AWEKAS:id=15920
WINDY: Michael's Bay, Manitoulin Island
WOW: Michael's Bay

Offline vinceskahan

  • Senior Member
  • **
  • Posts: 85
    • home site
Re: Belchertown skin - MQTT
« Reply #5 on: January 09, 2022, 03:38:31 PM »

My logs show my rsync of a couple custom skins and belchertown taking about 3 seconds from a computer much slower than a raspi.

Jan  9 11:56:48 debian weewx[24655] INFO weeutil.rsyncupload: rsync'd 70 files (2342853 bytes) in 2.10 seconds

If you are running a pi at home and have weewx+belchertown realtime updates working on your LAN, you already have almost all of the computing skillset required to host a minimal site on Internet and rsync to there automatically, so I wouldn't worry much there.

Make sure your MQTT setup at least uses a username and password for that.  I wouldn't worry too much about the ssl stuff probably.  Just make sure you have allow_anonymous=false on your broker.

Setting up rsync in weewx is just editing your weewx.conf to put in the [[RSYNC]] settings for the host you want to rsync to:

    [[RSYNC]]
        skin = Rsync
        enable = true
        server = x.x.x.x                                   # ip address or fully qualified host.domain here
        user = remote_user_here
        path = /remote/path/to/rsync/to/here
        log_success = false
        log_failure = true
        HTML_ROOT = /local/path/to/rsync/from
        delete = 0


There is some text in the default weewx.conf for how to do this, but if you've used ssh with keypairs you'll be ok.  If not, there are lots of howto docs for how to set up password-less ssh using keys.  Very straightforward once you do it the first time.

Creating a tiny host on Amazon AWS Lightsail would get you set up in just a few minutes and the free tier is good for 30 days if you're new to AWS.  Cost afterward is under $5/month.





WeeWX sites:
  Davis VP2+DFARS to a Seagate Dockstar (pogoplug)
  EcoWitt GW1000, WH32 outdoor T+H, multiple WH31 indoor T+H, WH51 soilMoisture (docker)
Home site:        https://www.skahan.net/
Wunderground: KWAFEDER15
PWS:                KWFEDER15
CWOP:             CW6881

Offline gszlag

  • Senior Contributor
  • ****
  • Posts: 249
  • ..have you calibrated your barometer today?
    • Michael's Bay - Manitoulin Island weather
Re: Belchertown skin - MQTT
« Reply #6 on: January 09, 2022, 04:26:42 PM »

My logs show my rsync of a couple custom skins and belchertown taking about 3 seconds from a computer much slower than a raspi.

Jan  9 11:56:48 debian weewx[24655] INFO weeutil.rsyncupload: rsync'd 70 files (2342853 bytes) in 2.10 seconds

If you are running a pi at home and have weewx+belchertown realtime updates working on your LAN, you already have almost all of the computing skillset required to host a minimal site on Internet and rsync to there automatically, so I wouldn't worry much there.

Make sure your MQTT setup at least uses a username and password for that.  I wouldn't worry too much about the ssl stuff probably.  Just make sure you have allow_anonymous=false on your broker.

Setting up rsync in weewx is just editing your weewx.conf to put in the [[RSYNC]] settings for the host you want to rsync to:

    [[RSYNC]]
        skin = Rsync
        enable = true
        server = x.x.x.x                                   # ip address or fully qualified host.domain here
        user = remote_user_here
        path = /remote/path/to/rsync/to/here
        log_success = false
        log_failure = true
        HTML_ROOT = /local/path/to/rsync/from
        delete = 0


There is some text in the default weewx.conf for how to do this, but if you've used ssh with keypairs you'll be ok.  If not, there are lots of howto docs for how to set up password-less ssh using keys.  Very straightforward once you do it the first time.

Creating a tiny host on Amazon AWS Lightsail would get you set up in just a few minutes and the free tier is good for 30 days if you're new to AWS.  Cost afterward is under $5/month.

Fantastic instructions! I haven't used passwordless SSH yet but I just learned a few weeks ago to remote in to the pi(s) using SSH and how to launch a GUI app. Pretty cool - to do that from the comand line.

Looks like I have some more reading to do.
Thanks once again!
Ecowitt GW1000/GW1100
Ecowitt WS68: Anemometer, UV/solar
Ecowitt WH40: Rain gauge
Ecowitt WH32E: Outside T & H sensor
Ambient Weather WS-2000
Stratus Rain Gauge (manual)
Raspberry Pi 3B+ (WeeWX/CumulusMX)
Raspberry Pi Zero 2W WeeWX/MQTT/Belchertown)
---
http://weather.glenns.ca (pwsdashboard - live)
http://weewx.glenns.ca
http://glenns.ca/cumulusmx2/index.htm
---
WU: ITEHKUMM2
PWSweather.com: MBAYWX1
AWEKAS:id=15920
WINDY: Michael's Bay, Manitoulin Island
WOW: Michael's Bay

Offline gszlag

  • Senior Contributor
  • ****
  • Posts: 249
  • ..have you calibrated your barometer today?
    • Michael's Bay - Manitoulin Island weather
Re: Belchertown skin - MQTT
« Reply #7 on: January 11, 2022, 06:02:03 PM »
@vinceskahan @davidefa

..getting to be quite an adventure!

Finally found out from the webhost that MQTT is strictly verboten. Nope - not gonna open ports on a shared server..they said. But if you only pay 3x more we should be able to accomodate...

passwordless SSH took a while to set up. Gave up. Had to contact support and found out the port is not  22. That helps  :-)
Note to self: contact support before you spend hours/days on trying out all sorts of things.

Decided to test rsync on the command line first, but it fails: can not find file or directory plus IO error.. Tried single quotes or double quotes for ssh -p 1234 ... both fail.
Note: remote target path is /public_html/some_folder/belchertown is pre-existing..why mkdir?

$ rsync -avz -e "ssh -p 1234" /home/user/test/ remote_user@mydomain.ca:/public_html/some_folder/belchertown/
sending incremental file list
rsync: mkdir "/public_html/some_folder/belchertown" failed: No such file or directory (2)
rsync error: error in file IO (code 11) at main.c(657) [Receiver=3.1.2]
---
OK. Tech support solved it. There was error in the path to the remote server. ..had to remove the "/" before public_html

the correct command is:

$ rsync -avz -e "ssh -p 1234" /home/user/test/ remote_user@mydomain.ca:public_html/some_folder/belchertown/


...will rsync /var/www/html/weewx/belchertown. I guess I will have to change the HTML_ROOT in weewx.conf (rsync section) to match.

the adventure continues...
Ecowitt GW1000/GW1100
Ecowitt WS68: Anemometer, UV/solar
Ecowitt WH40: Rain gauge
Ecowitt WH32E: Outside T & H sensor
Ambient Weather WS-2000
Stratus Rain Gauge (manual)
Raspberry Pi 3B+ (WeeWX/CumulusMX)
Raspberry Pi Zero 2W WeeWX/MQTT/Belchertown)
---
http://weather.glenns.ca (pwsdashboard - live)
http://weewx.glenns.ca
http://glenns.ca/cumulusmx2/index.htm
---
WU: ITEHKUMM2
PWSweather.com: MBAYWX1
AWEKAS:id=15920
WINDY: Michael's Bay, Manitoulin Island
WOW: Michael's Bay

Offline davidefa

  • Forecaster
  • *****
  • Posts: 313
Re: Belchertown skin - MQTT
« Reply #8 on: January 12, 2022, 02:47:57 AM »
Finally found out from the webhost that MQTT is strictly verboten. Nope - not gonna open ports on a shared server..they said. But if you only pay 3x more we should be able to accomodate...

Can't understand why they fear so much contacting an mqtt server ( I think this is due to prevent some use/abuse of their servers ).

P.S.
But if you show the colour of your dollars...
By the way, what's the colour of Canadian dollars?

Offline gszlag

  • Senior Contributor
  • ****
  • Posts: 249
  • ..have you calibrated your barometer today?
    • Michael's Bay - Manitoulin Island weather
Re: Belchertown skin - MQTT
« Reply #9 on: January 12, 2022, 07:57:20 AM »
Finally found out from the webhost that MQTT is strictly verboten. Nope - not gonna open ports on a shared server..they said. But if you only pay 3x more we should be able to accomodate...

Can't understand why they fear so much contacting an mqtt server ( I think this is due to prevent some use/abuse of their servers ).

P.S.
But if you show the colour of your dollars...
By the way, what's the colour of Canadian dollars?
Despite that, the web host has excellent support. I have too much invested there to move. Plus it is only $6CAD (an old legacy account) where the price is fixed.

I guess it depends how much I really like the skin. I am looking at AWS lightsail(as vinceskahan suggested) to add to the web host collection but those are USD. CAD would cost about 30% more at current exchange rates.

Colour of Canadian dollars?...we have all the colours!!!
Ecowitt GW1000/GW1100
Ecowitt WS68: Anemometer, UV/solar
Ecowitt WH40: Rain gauge
Ecowitt WH32E: Outside T & H sensor
Ambient Weather WS-2000
Stratus Rain Gauge (manual)
Raspberry Pi 3B+ (WeeWX/CumulusMX)
Raspberry Pi Zero 2W WeeWX/MQTT/Belchertown)
---
http://weather.glenns.ca (pwsdashboard - live)
http://weewx.glenns.ca
http://glenns.ca/cumulusmx2/index.htm
---
WU: ITEHKUMM2
PWSweather.com: MBAYWX1
AWEKAS:id=15920
WINDY: Michael's Bay, Manitoulin Island
WOW: Michael's Bay

Offline gszlag

  • Senior Contributor
  • ****
  • Posts: 249
  • ..have you calibrated your barometer today?
    • Michael's Bay - Manitoulin Island weather
Re: Belchertown skin - MQTT
« Reply #10 on: January 26, 2022, 01:32:33 PM »
Quote
the adventure continues...
When I could not proceed any further I went to the weewx-user groups (whatever happened to alt.?)  Is vince and vinceskahan one and the same?

Vince's recommendation to use rsync and a hint that weewx runs under root turned on the lightbulb but not before my IP was banned for 24 hours...multiple SSH failures...

In the end it got done. Rsync basically syncs the whole local web site to the web host at amazingly fast speeds.

See my vanity web site at wx.glenns.ca to see it working (doesn't work under https though..)
Ecowitt GW1000/GW1100
Ecowitt WS68: Anemometer, UV/solar
Ecowitt WH40: Rain gauge
Ecowitt WH32E: Outside T & H sensor
Ambient Weather WS-2000
Stratus Rain Gauge (manual)
Raspberry Pi 3B+ (WeeWX/CumulusMX)
Raspberry Pi Zero 2W WeeWX/MQTT/Belchertown)
---
http://weather.glenns.ca (pwsdashboard - live)
http://weewx.glenns.ca
http://glenns.ca/cumulusmx2/index.htm
---
WU: ITEHKUMM2
PWSweather.com: MBAYWX1
AWEKAS:id=15920
WINDY: Michael's Bay, Manitoulin Island
WOW: Michael's Bay

 

anything