Author Topic: CumulusMX raspberry pi out-of-memory  (Read 2017 times)

0 Members and 1 Guest are viewing this topic.

Offline gszlag

  • Forecaster
  • *****
  • Posts: 480
  • "barometers always need correcting"
    • Michael's Bay - Manitoulin Island weather
CumulusMX raspberry pi out-of-memory
« on: February 03, 2023, 10:40:59 AM »
Unfortunately the rpi zero 2W with its 512MB memory wasn't up to task running CumulusMX and WeeWX at the same time. Mark mentioned mono is the likely culprit causing a memory leak.

Although it looked promising at the beginning, this tiny microcomputer would eventually pack it in under the load so I moved CumulusMX to my other rpi - the 3B+

The bigger pi 3B+ is definitely betterand buys me uptime  but if I forget to restart MX or reboot the computer on a regular basis, the 3B+ eventually caves in too.

Here are the symptoms:

- VNC fails to connect or slow to connect and if I do connect, the desktop takes a long time to load ( a sure sign I am running out of memory and better do something soon).
- SSH fails to connect or is slow to connect so I can't  send a shutdown -r on the terminal.

My computing skills are basic at best  and I could look up how to do a cron job and reboot on a fixed schedule but is there a linux way to restart a failing CumulusMX without rebooting?  CMX is installed as a service.

Any linux experts out there? I am running raspberrypiOS (Debian 10).
Ambient Weather WS-2000
Ecowitt WS3900 console
Ecowitt GW1000/GW1100
Ecowitt WS68: Anemometer, UV/solar
Ecowitt WH40: Rain gauge
Ecowitt WH57 Lightning sensor
Ecowitt WH32E: Outside T & H sensor
Ecowitt WN30 temperature sensor
Stratus Rain Gauge (manual)
---
Raspberry Pi 3B+ (WeeWX/CumulusMX)
Raspberry Pi Zero 2W (WeeWX/MQTT/Belchertown)
---
Barometer wiki: http://meshka.eu/Ecowitt/dokuwiki/doku.php?id=barometer#barometer
---
Uploading to: AWN, ecowitt.net, Weather Underground, PWSweather.com, AWEKAS, Windy.com, WOW

Offline Gyvate

  • Forecaster
  • *****
  • Posts: 3512
Re: CumulusMX raspberry pi out-of-memory
« Reply #1 on: February 19, 2023, 11:53:17 AM »
you could create a script stopping the service and restarting it
e.g. a file named cmx-restart.sh in the e.g. /pi/CumulusMX directory - or whatever your path to CMX is
inside it would look something like

Code: [Select]
#!/bin/sh
#
### restart CMX
#
systemctl stop cumulusmx
sleep 10
systemctl restart cumulusmx

run manually with e.g. sudo sh cmx-restart.sh (from the directory the script resides in)
run periodically with an entry in /etc/crontab
something like below would run the job/script every Sunday

Code: [Select]
*   *   * * 7   root    sh /pi/CumulusMX/cmx-restart.sh
this is assuming that the cumulusmx script called to start the service contains the start of mono - and that the stop section also stops mono
(not knowing how you run CMX as a service and what the service script looks like)
WS2350 1.6.7, GW1000(3) 1.7.7,WH2650 WiFi (2) 1.7.7 (test/backup), GW1100 2.3.2, GW1200 1.3.1, GW2000(3) 3.1.3, HP2551 1.9.6,5.1.6;HP3500 1.7.2,WS3800 1.3.0, WN1910 1.2.3,WN1980 1.2.3;
Ecowitt WS90(2)1.4.3/1.4.0, WS80(2)1.2.8, WS68, WS69, WH40,WH31,WH31-EP,WN30,WN34L,WN35,WH32,WH32-EP, WH32B, WH57 [Lightning], WH41[PM2.5], WH51, WH45, WH55
MeteobridgePro(2)[test,prod] 5.8 Mar 01 2024, 15185 - Blake-Larsen Sun Recorder - RPi4/weewx 4.8.0/4.10.2/CumulusMX 3283/Meteobridge RPi4B-2GB,
Barani Meteoshield Pro, MetSpecRad02
weather landing page: http://meshka.eu
WIKI http://meshka.eu/Ecowitt/dokuwiki

Offline gszlag

  • Forecaster
  • *****
  • Posts: 480
  • "barometers always need correcting"
    • Michael's Bay - Manitoulin Island weather
Re: CumulusMX raspberry pi out-of-memory
« Reply #2 on: February 19, 2023, 03:16:52 PM »
you could create a script stopping the service and restarting it
e.g. a file named cmx-restart.sh in the e.g. /pi/CumulusMX directory - or whatever your path to CMX is
inside it would look something like

Code: [Select]
#!/bin/sh
#
### restart CMX
#
systemctl stop cumulusmx
sleep 10
systemctl restart cumulusmx

run manually with e.g. sudo sh cmx-restart.sh (from the directory the script resides in)
run periodically with an entry in /etc/crontab
something like below would run the job/script every Sunday

Code: [Select]
*   *   * * 7   root    sh /pi/CumulusMX/cmx-restart.sh
this is assuming that the cumulusmx script called to start the service contains the start of mono - and that the stop section also stops mono
(not knowing how you run CMX as a service and what the service script looks like)
Thanks Gyvate! Appreciate the help.
I think the CMX service was set up  the official way following the instructions :Running  MX as a Linux systemd"service
When I stop cumulusmx i appears that mono-service.exe stops as well (according to htop).
Your post is timely as I was looking up cron, crontab cron job from a beginner's perspective.
What i"ve learned so far:
Definitely an unusual way to edit a config file. You don't use a text editor like kate or geany, you launch the editor thusly: contab -e and will be asked to choose an editor.
I wimped out and chose nano.
You will also have to decide if you want to use crontab as user or as root. I gather each has their own onfig file somewhere.
If you are using crontab to restart a service then using a crontab as user will fail ( not enough permissions) .
Crontab also seems fussy about specifying a full path to commands or scripts. Sometimes you have to specify root as user on the command line but sometimes not.
Definitely will try your script. Thanks once again.
Ambient Weather WS-2000
Ecowitt WS3900 console
Ecowitt GW1000/GW1100
Ecowitt WS68: Anemometer, UV/solar
Ecowitt WH40: Rain gauge
Ecowitt WH57 Lightning sensor
Ecowitt WH32E: Outside T & H sensor
Ecowitt WN30 temperature sensor
Stratus Rain Gauge (manual)
---
Raspberry Pi 3B+ (WeeWX/CumulusMX)
Raspberry Pi Zero 2W (WeeWX/MQTT/Belchertown)
---
Barometer wiki: http://meshka.eu/Ecowitt/dokuwiki/doku.php?id=barometer#barometer
---
Uploading to: AWN, ecowitt.net, Weather Underground, PWSweather.com, AWEKAS, Windy.com, WOW

Offline mcrossley

  • Forecaster
  • *****
  • Posts: 1153
    • Wilmslow Astro
Re: CumulusMX raspberry pi out-of-memory
« Reply #3 on: February 19, 2023, 03:24:27 PM »
@gszlag remind me, which station are you using with MX?
Mark

Offline gszlag

  • Forecaster
  • *****
  • Posts: 480
  • "barometers always need correcting"
    • Michael's Bay - Manitoulin Island weather
Re: CumulusMX raspberry pi out-of-memory
« Reply #4 on: February 19, 2023, 03:33:36 PM »
@gszlag remind me, which station are you using with MX?
I am running the ecowitt equipment on CMX with Gary's GW1000 driver.
Oops..sorry. I am running and uploading both CMX and WeeWX from the same pi3.
« Last Edit: February 19, 2023, 03:35:43 PM by gszlag »
Ambient Weather WS-2000
Ecowitt WS3900 console
Ecowitt GW1000/GW1100
Ecowitt WS68: Anemometer, UV/solar
Ecowitt WH40: Rain gauge
Ecowitt WH57 Lightning sensor
Ecowitt WH32E: Outside T & H sensor
Ecowitt WN30 temperature sensor
Stratus Rain Gauge (manual)
---
Raspberry Pi 3B+ (WeeWX/CumulusMX)
Raspberry Pi Zero 2W (WeeWX/MQTT/Belchertown)
---
Barometer wiki: http://meshka.eu/Ecowitt/dokuwiki/doku.php?id=barometer#barometer
---
Uploading to: AWN, ecowitt.net, Weather Underground, PWSweather.com, AWEKAS, Windy.com, WOW

Offline mcrossley

  • Forecaster
  • *****
  • Posts: 1153
    • Wilmslow Astro
Re: CumulusMX raspberry pi out-of-memory
« Reply #5 on: February 19, 2023, 04:29:25 PM »
Ok, yep still has a memory leak then. I fixed the leak in the Davis WLL code in a recent update.
Mark

Offline Gyvate

  • Forecaster
  • *****
  • Posts: 3512
Re: CumulusMX raspberry pi out-of-memory
« Reply #6 on: February 20, 2023, 02:38:38 AM »
I know that mono usually stops when you stop CMX ....
just to be on the safe side not to have multiple mono instances running ...

- permissions is something one has to understand under Linux
- whether you run a script/program via crontab as user root and just provide the script with path or as user (e.g.) pi and call the script via sudo doesn't make a difference for practical purposes
- as you usually log into your RPi as user pi, you will have to run the editor (Geany, nano) with sudo
e.g. sudo nano /etc/crontab
- in order to start Geany for editing (and saving) crontab as root (user pi will not be allowed to save files user pi doesn't own) from the file manager, you can open a file manager window using "sudo pcmanfm" in an extra console window (but don't close the console window afterwards  ;) - closing the console window closes the Geany instance) - this file manager instance runs with root permissions
right-click the crontab file in the /etc directory and choose "Geany Programmer's editor" from the pop-up menu
- my mini-script assumes that you have a script/config file cumulusmx.service which uses systemd
(--> https://cumulus.hosiene.co.uk/viewtopic.php?t=18766)
WS2350 1.6.7, GW1000(3) 1.7.7,WH2650 WiFi (2) 1.7.7 (test/backup), GW1100 2.3.2, GW1200 1.3.1, GW2000(3) 3.1.3, HP2551 1.9.6,5.1.6;HP3500 1.7.2,WS3800 1.3.0, WN1910 1.2.3,WN1980 1.2.3;
Ecowitt WS90(2)1.4.3/1.4.0, WS80(2)1.2.8, WS68, WS69, WH40,WH31,WH31-EP,WN30,WN34L,WN35,WH32,WH32-EP, WH32B, WH57 [Lightning], WH41[PM2.5], WH51, WH45, WH55
MeteobridgePro(2)[test,prod] 5.8 Mar 01 2024, 15185 - Blake-Larsen Sun Recorder - RPi4/weewx 4.8.0/4.10.2/CumulusMX 3283/Meteobridge RPi4B-2GB,
Barani Meteoshield Pro, MetSpecRad02
weather landing page: http://meshka.eu
WIKI http://meshka.eu/Ecowitt/dokuwiki

Offline gszlag

  • Forecaster
  • *****
  • Posts: 480
  • "barometers always need correcting"
    • Michael's Bay - Manitoulin Island weather
Re: CumulusMX raspberry pi out-of-memory
« Reply #7 on: February 20, 2023, 08:24:32 AM »
I know that mono usually stops when you stop CMX ....
just to be on the safe side not to have multiple mono instances running ...
..I have cmx running and it shows many instances of mono running. See screengrab. However in top it shows the one process running.
Normal?
Oddly, mono is running from a directory 4.5 (version #?) but the raspi package manager shows mono 6.12. I am still running rpiOS buster
I already have been playing around with crontab with shutdown -r and it works!
Thanks for the instructions!
Ambient Weather WS-2000
Ecowitt WS3900 console
Ecowitt GW1000/GW1100
Ecowitt WS68: Anemometer, UV/solar
Ecowitt WH40: Rain gauge
Ecowitt WH57 Lightning sensor
Ecowitt WH32E: Outside T & H sensor
Ecowitt WN30 temperature sensor
Stratus Rain Gauge (manual)
---
Raspberry Pi 3B+ (WeeWX/CumulusMX)
Raspberry Pi Zero 2W (WeeWX/MQTT/Belchertown)
---
Barometer wiki: http://meshka.eu/Ecowitt/dokuwiki/doku.php?id=barometer#barometer
---
Uploading to: AWN, ecowitt.net, Weather Underground, PWSweather.com, AWEKAS, Windy.com, WOW

Offline Gyvate

  • Forecaster
  • *****
  • Posts: 3512
Re: CumulusMX raspberry pi out-of-memory
« Reply #8 on: February 20, 2023, 10:37:09 AM »
you beat me on that - I made it max to two mono instances some time ago,
but so many mono instances look weird to me ...
do you have a script running that uses the
Code: [Select]
systemctl start cumulusmx
command ???
if so, I would change it into
Code: [Select]
systemctl restart cumulusmx as each start would also start a new mono instance ....
whereas in my understanding restart first stops the instance before starting it anew
maybe that helps
have you tagged inside the CMX interface in the settings the "stop second instance" option ?
(Settings --> Program settings --> General options)
something looks strange if not fishy in your system
WS2350 1.6.7, GW1000(3) 1.7.7,WH2650 WiFi (2) 1.7.7 (test/backup), GW1100 2.3.2, GW1200 1.3.1, GW2000(3) 3.1.3, HP2551 1.9.6,5.1.6;HP3500 1.7.2,WS3800 1.3.0, WN1910 1.2.3,WN1980 1.2.3;
Ecowitt WS90(2)1.4.3/1.4.0, WS80(2)1.2.8, WS68, WS69, WH40,WH31,WH31-EP,WN30,WN34L,WN35,WH32,WH32-EP, WH32B, WH57 [Lightning], WH41[PM2.5], WH51, WH45, WH55
MeteobridgePro(2)[test,prod] 5.8 Mar 01 2024, 15185 - Blake-Larsen Sun Recorder - RPi4/weewx 4.8.0/4.10.2/CumulusMX 3283/Meteobridge RPi4B-2GB,
Barani Meteoshield Pro, MetSpecRad02
weather landing page: http://meshka.eu
WIKI http://meshka.eu/Ecowitt/dokuwiki

Offline mcrossley

  • Forecaster
  • *****
  • Posts: 1153
    • Wilmslow Astro
Re: CumulusMX raspberry pi out-of-memory
« Reply #9 on: February 20, 2023, 03:58:07 PM »
Switch your top (or htop) into tree mode by pressing F5. They you will see how all the processes are related. You should see a primary mono process for CMX with a number of child processes.
Mark

Offline gszlag

  • Forecaster
  • *****
  • Posts: 480
  • "barometers always need correcting"
    • Michael's Bay - Manitoulin Island weather
Re: CumulusMX raspberry pi out-of-memory
« Reply #10 on: February 20, 2023, 04:39:03 PM »
- do you have a script running that uses the [systemctl start cumulusmx command ???
- have you tagged inside the CMX interface in the settings the "stop second instance" option

- no scripts by me
and
- the "stop second instance" check box has always been checked off.

My cumulumx.service file contains:

Code: [Select]
[Unit]
Description=CumulusMX service
Documentation=https://cumuluswiki.org/a/Main_Page
Wants=network-online.target
After=network-online.target

[Service]
User=root
Group=root
ExecStart=/usr/bin/mono-service -d:/home/install/CumulusMX CumulusMX.exe -service
Type=forking
ExecStopPost=/bin/rm -f /tmp/CumulusMX.exe.lock

[Install]
WantedBy=multi-user.target
Ambient Weather WS-2000
Ecowitt WS3900 console
Ecowitt GW1000/GW1100
Ecowitt WS68: Anemometer, UV/solar
Ecowitt WH40: Rain gauge
Ecowitt WH57 Lightning sensor
Ecowitt WH32E: Outside T & H sensor
Ecowitt WN30 temperature sensor
Stratus Rain Gauge (manual)
---
Raspberry Pi 3B+ (WeeWX/CumulusMX)
Raspberry Pi Zero 2W (WeeWX/MQTT/Belchertown)
---
Barometer wiki: http://meshka.eu/Ecowitt/dokuwiki/doku.php?id=barometer#barometer
---
Uploading to: AWN, ecowitt.net, Weather Underground, PWSweather.com, AWEKAS, Windy.com, WOW

Offline gszlag

  • Forecaster
  • *****
  • Posts: 480
  • "barometers always need correcting"
    • Michael's Bay - Manitoulin Island weather
Re: CumulusMX raspberry pi out-of-memory
« Reply #11 on: February 20, 2023, 04:40:30 PM »
Switch your top (or htop) into tree mode by pressing F5. They you will see how all the processes are related. You should see a primary mono process for CMX with a number of child processes.
[ You are not allowed to view attachments ]
Ambient Weather WS-2000
Ecowitt WS3900 console
Ecowitt GW1000/GW1100
Ecowitt WS68: Anemometer, UV/solar
Ecowitt WH40: Rain gauge
Ecowitt WH57 Lightning sensor
Ecowitt WH32E: Outside T & H sensor
Ecowitt WN30 temperature sensor
Stratus Rain Gauge (manual)
---
Raspberry Pi 3B+ (WeeWX/CumulusMX)
Raspberry Pi Zero 2W (WeeWX/MQTT/Belchertown)
---
Barometer wiki: http://meshka.eu/Ecowitt/dokuwiki/doku.php?id=barometer#barometer
---
Uploading to: AWN, ecowitt.net, Weather Underground, PWSweather.com, AWEKAS, Windy.com, WOW

Offline mcrossley

  • Forecaster
  • *****
  • Posts: 1153
    • Wilmslow Astro
Re: CumulusMX raspberry pi out-of-memory
« Reply #12 on: February 21, 2023, 04:49:59 AM »
I would uninstall mono-xsp4 unless you have a need for it and the Apache web service it installs. CMX does not need it.
Mark

Offline gszlag

  • Forecaster
  • *****
  • Posts: 480
  • "barometers always need correcting"
    • Michael's Bay - Manitoulin Island weather
Re: CumulusMX raspberry pi out-of-memory
« Reply #13 on: February 21, 2023, 11:28:47 AM »
I would uninstall mono-xsp4 unless you have a need for it and the Apache web service it installs. CMX does not need it.
If you remove the simple web server for ASPNET, the rpiOS package manager will replace it with the apache web server version.
if you remove the xsp4 base libraries then mono-complete is removed.
..some sort of dependency hell..can't get there from here:)
Ambient Weather WS-2000
Ecowitt WS3900 console
Ecowitt GW1000/GW1100
Ecowitt WS68: Anemometer, UV/solar
Ecowitt WH40: Rain gauge
Ecowitt WH57 Lightning sensor
Ecowitt WH32E: Outside T & H sensor
Ecowitt WN30 temperature sensor
Stratus Rain Gauge (manual)
---
Raspberry Pi 3B+ (WeeWX/CumulusMX)
Raspberry Pi Zero 2W (WeeWX/MQTT/Belchertown)
---
Barometer wiki: http://meshka.eu/Ecowitt/dokuwiki/doku.php?id=barometer#barometer
---
Uploading to: AWN, ecowitt.net, Weather Underground, PWSweather.com, AWEKAS, Windy.com, WOW

Offline mcrossley

  • Forecaster
  • *****
  • Posts: 1153
    • Wilmslow Astro
Re: CumulusMX raspberry pi out-of-memory
« Reply #14 on: February 21, 2023, 12:10:37 PM »
Ah - ok, just stop and disable the xsp4 service then with systemctl.
Mark