Topic: weewx CPU usage on Raspberry Pi Zero W

[lits]

Hi all,

I recently moved my weewx setup over from an old RPi3B to an RPi0W - mostly to hive it off to a machine that isn't getting restarted so frequently.

I've noticed that the CPU on the Zero is at or near 100% utilisation most of the time.

The system is running on Debian 11.0 Lite RPi OS, weewx is running the interceptor driver in listen mode and the mqtt extension, and updating the web output every 60 seconds. Everything else about the setup is vanilla.

htop shows that it's python3 using 97%~ of the CPU at its peak.

Is it reasonable for the CPU on a Pi Zero to max out under this load?

[gszlag]

Edit: I just realized you have a zero W (not a 2W?). If so, you have a zero W then comparing it to a zero 2W may not be a fair comparison. Still though, I would think a zero W would run weewx fairly easily...

Here is what I have on my rpi zero 2W using the "top" command(see below). You may want to run top on your rpi and compare.

It is currently running both WeeWX and Cumulus MX and the GUI desktop.

To free up more memory, I can disable the desktop. I will be moving a local MQTT broker to the rpi zero 2w as well.

I found Debian Bullseye problematic and reverted back to Buster. Tom's hardware has been particularly critical of Bullseye.

It might be worthwhile to run an experiment: get another SD micro card and flash it with Buster..install WeeWX and run the simulator or something and see if things improve.
--
top - 11:55:29 up 20:29,  2 users,  load average: 0.72, 0.70, 0.68
Tasks: 146 total,   1 running, 145 sleeping,   0 stopped,   0 zombie

%Cpu(s):  6.2 us,  2.4 sy,  0.0 ni, 90.6 id,  0.1 wa,  0.0 hi,  0.7 si,  0.0 st

MiB Mem :    428.0 total,     54.0 free,    180.4 used,    193.6 buff/cache
MiB Swap:    100.0 total,     14.5 free,     85.5 used.    177.1 avail Mem
--

[lits]

It looks like this was mostly to do with my update_interval under StdArchive. I had this set to 60 seconds, which the CPU on my Pi3B could handle, but apparently the CPU on the Pi0W could not.

For what it's worth, I did spin up an instance of Buster on another SD card, and ended up with the same high CPU utilisation as under Bullseye.

What ended up fixing it was increasing the update_interval to 300 seconds. Now I can see my CPU utilisation is spending most of its time around 10%~ and only spiking to 100% every 5 minutes when the archive process kicks off. It's not ideal, as now I'm only getting updates via MQTT every 5 minutes, instead of every 1 minute. But I guess I'll have to learn to live with it. Until I can get my hands on a Pi02W.

[vreihen]

It's not ideal, as now I'm only getting updates via MQTT every 5 minutes, instead of every 1 minute.

In weewx.conf under [StdRESTful], do you have MQTT binding set to include loop packets?  With these settings and my Davis VP2, WeeWX sends loop packets every two seconds (Davis' loop packet frequency) to MQTT with my archive interval set to five minutes:

Code: [Select]

    [[MQTT]]
        server_url = mqtt://user:pass@127.0.0.1:1883/
        topic = weather
        unit_system = US
        binding = archive, loop
        aggregation = aggregate
        log_success = false


[vinceskahan]

What ended up fixing it was increasing the update_interval to 300 seconds. Now I can see my CPU utilisation is spending most of its time around 10%~ and only spiking to 100% every 5 minutes when the archive process kicks off.

A 60-sec archive period is pretty quick.

You might look at your logs for how long it takes your reports to run, and also minimize how many skins and extensions that you have enabled.  If you have Saratoga or Belchertown enabled those take a long time to process on a weak system like a zero.   Running just Seasons or Standard should be pretty quick.

FWIW my whole cycle is about 92 seconds typically on a Seagate Dockstar (a 128MB RAM original pogoplug) which is likely ballpark similar to a zero, but half of that is Belchertown, so it might fit in a minute archive theoretically, but it would be close.

If you're simply running the zero as an engine to feed MQTT then I *think* you should be able to turn off all the StdReport skins, or substitute in your own. trivial one that just writes a one-liner html file out.

Regardless, your logs will tell you how long each piece is taking.  That'll help you figure out where the limits are.

[vinceskahan]

ok - did a quick test on an original zeroW with weewx using the simulator driver and only Seasons active as a skin.

The system idles at 2% cpu and pegs to 100% while the reports run.  No difference in behavior with the archive interval set to 60 seconds rather than the default 300.  Basically it takes however long it takes.   On a zerow that seems to be about 11 seconds running just Seasons as the only enabled skin.

[lits]

So I'm just running Seasons, nothing fancy. No other skins. I publish the weewx output externally, so I need to keep the skin running.

My report runs must take longer than 60 seconds. Increasing the archive_interval to 300 at least gave the CPU a chance to return to a reasonable idling usage between report runs. My report runs were probably overlapping at a 60s interval. So I'll run it at 300s for a while, and decrease it later if I need a more granular display of data.

I'll try that MQTT binding setting vreihen suggested to get more frequent data into Home Assistant. That might be all I need.

Other notes:

- On a clean build of Debian Buster (Lite), avahi-daemon was taking up around 40% of the CPU while the system was idling. Disabling that gave weewx a lot more headroom when it came to generating reports.

- I'm using a python script to push system stats to Home Assistant via MQTT. This is where I was seeing the CPU cranking to >100% the whole time. Watching CPU util in top/htop I can see it return to idling at 5% after the report run completes. In Home Assistant I'm now seeing the CPU utilisation at much more reasonable levels.

[vinceskahan]

My report runs must take longer than 60 seconds. Increasing the archive_interval to 300 at least gave the CPU a chance to return to a reasonable idling usage between report runs. My report runs were probably overlapping at a 60s interval.

It's pretty trivial to find out whether this is the case.  Just look at the timings in /var/log/syslog.

My Dockstar runs a 'lot' of stuff so it takes over two minutes to complete a run...

Code: [Select]

Jan 18 17:56:51 debian weewx[24655] INFO weeutil.rsyncupload: rsync'd 70 files (2574247 bytes) in 1.93 seconds
Jan 18 18:00:19 debian weewx[24655] INFO weewx.manager: Added record 2022-01-18 18:00:20 PST (1642557620) to database 'mem.sdb'
Jan 18 18:00:19 debian weewx[24655] INFO weewx.manager: Added record 2022-01-18 18:00:20 PST (1642557620) to daily summary in 'mem.sdb'
Jan 18 18:00:20 debian weewx[24655] INFO weewx.manager: Added record 2022-01-18 18:00:20 PST (1642557620) to database 'purpleair.sdb'
Jan 18 18:00:20 debian weewx[24655] INFO weewx.manager: Added record 2022-01-18 18:00:20 PST (1642557620) to daily summary in 'purpleair.sdb'
Jan 18 18:00:21 debian weewx[24655] INFO weewx.manager: Added record 2022-01-18 18:00:00 PST (1642557600) to database 'weewx.sdb'
Jan 18 18:00:21 debian weewx[24655] INFO weewx.manager: Added record 2022-01-18 18:00:00 PST (1642557600) to daily summary in 'weewx.sdb'
Jan 18 18:00:21 debian weewx[24655] INFO weewx.restx: MQTT: Published record 2022-01-18 18:00:00 PST (1642557600)
Jan 18 18:00:22 debian weewx[24655] INFO weewx.restx: PWSWeather: Published record 2022-01-18 18:00:00 PST (1642557600)
Jan 18 18:00:22 debian weewx[24655] INFO weewx.restx: Wunderground-PWS: Published record 2022-01-18 18:00:00 PST (1642557600)
Jan 18 18:00:22 debian weewx[24655] INFO weewx.cheetahgenerator: Generated 6 files for report SmartphoneReport in 0.45 seconds
Jan 18 18:00:34 debian weewx[24655] INFO weewx.imagegenerator: Generated 18 images for report SmartphoneReport in 12.15 seconds
Jan 18 18:00:34 debian weewx[24655] INFO weewx.reportengine: Copied 0 files to /home/weewx/public_html/smartphone
Jan 18 18:00:35 debian weewx[24655] INFO weewx.cheetahgenerator: Generated 1 files for report mem in 0.15 seconds
Jan 18 18:00:35 debian weewx[24655] INFO weewx.imagegenerator: Generated 1 images for report mem in 0.35 seconds
Jan 18 18:00:35 debian weewx[24655] INFO weewx.reportengine: Copied 0 files to /home/weewx/public_html/mem
Jan 18 18:00:45 debian weewx[24655] INFO weewx.cheetahgenerator: Generated 11 files for report vds-local in 9.13 seconds
Jan 18 18:00:45 debian weewx[24655] INFO weewx.reportengine: Copied 0 files to /home/weewx/public_html
Jan 18 18:01:24 debian weewx[24655] INFO weewx.imagegenerator: Generated 51 images for report vds-local in 39.38 seconds
Jan 18 18:01:25 debian weewx[24655] INFO weewx.cheetahgenerator: Generated 6 files for report vds-bootstrap in 0.91 seconds
Jan 18 18:01:25 debian weewx[24655] INFO weewx.reportengine: Copied 0 files to /home/weewx/public_html/bootstrap
Jan 18 18:01:25 debian weewx[24655] INFO weewx.cheetahgenerator: Generated 1 files for report lastrain in 0.12 seconds
Jan 18 18:01:32 debian weewx[24655] INFO user.belchertown: New forecast file downloaded to /home/weewx/public_html/weewx_belchertown/json/forecast.json
Jan 18 18:02:18 debian weewx[24655] INFO weewx.cheetahgenerator: Generated 11 files for report Belchertown in 52.09 seconds
Jan 18 18:02:18 debian weewx[24655] INFO weewx.reportengine: Copied 2 files to /home/weewx/public_html/weewx_belchertown
Jan 18 18:02:36 debian weewx[24655] INFO weewx.cheetahgenerator: Generated 1 files for report forecast in 0.72 seconds
Jan 18 18:02:36 debian weewx[24655] INFO weewx.reportengine: Copied 0 files to /home/weewx/public_html/forecast
Jan 18 18:02:39 debian weewx[24655] INFO weeutil.rsyncupload: rsync'd 115 files (2906185 bytes) in 2.35 seconds

In my case, I can tell by the time period between the rsyncuploads....it also tends to start about 15 seconds after every 5-minutes which is also built into weewx.  Very obvious timing.   I'm also running MQTT here but just archive, not loop for me here.

- I'm using a python script to push system stats to Home Assistant via MQTT. This is where I was seeing the CPU cranking to >100% the whole time. Watching CPU util in top/htop I can see it return to idling at 5% after the report run completes. In Home Assistant I'm now seeing the CPU utilisation at much more reasonable levels.

Yup - you might want to check your code.  A simple MQTT publish is pretty light in load to do.