Topic: MB data logger stuck - attempt to write a readonly database **solved**

[Gyvate]

Hi Boris,
since update 14074 my MB is stuck.
It keeps on connecting, disconnecting but is stuck with this database error. (also posted in the meteohub forum:https://forum.meteohub.de/viewtopic.php?f=61&t=15292)
Reboot doesn't change anything.
the (8) is a (8 ) !
logger (18.11.2020 14:18:40): DB cannot prepare select on 'wind0avgwind_year' (after 100 tries in 9217ms): attempt to write a readonly database (8)
logger (18.11.2020 14:18:50): DB cannot prepare select on 'wind0avgwind_year' (after 100 tries in 9213ms): attempt to write a readonly database (8)
logger (18.11.2020 14:18:57): child process received timeout signal (14).
logger (18.11.2020 14:18:57): disconnect station 0 (GW1000): exit
logger (18.11.2020 14:18:57): disconnect station 1 (GW1000): exit
logger (18.11.2020 14:18:57): disconnect station 9 (System Data): exit
logger (18.11.2020 14:18:58): data logger stopped.
logger (18.11.2020 14:18:58): databases disconnected.
logger (18.11.2020 14:18:58): parent process to be terminated.
logger (18.11.2020 14:18:58): databases connected. Update interval 3600 seconds.
logger (18.11.2020 14:18:58): data logger (version 5.1d, build 14074) started.
logger (18.11.2020 14:18:59): connect station 0 (GW1000 via TCP/IP).
logger (18.11.2020 14:18:59): connect station 1 (GW1000 via TCP/IP).
logger (18.11.2020 14:18:59): connect station 9 (System Data via Plug-in).
logger (18.11.2020 14:19:01): waiting for sensors (wind0, rain0, th0, thb0) to provide data.
logger (18.11.2020 14:19:01): services paused.
logger (18.11.2020 14:19:09): sensors are now providing data.
logger (18.11.2020 14:19:09): services started.
logger (18.11.2020 14:19:18): DB cannot prepare select on 'th0temp_day' (after 100 tries in 9213ms): attempt to write a readonly database (8)
logger (18.11.2020 14:19:27): DB cannot prepare select on 'th0temp_day' (after 100 tries in 9222ms): attempt to write a readonly database (8)
logger (18.11.2020 14:19:36): DB cannot prepare select on 'th0temp_day' (after 100 tries in 9200ms): attempt to write a readonly database (8)
logger (18.11.2020 14:19:45): DB cannot prepare select on 'th0temp_month' (after 100 tries in 9197ms): attempt to write a readonly database (8)
logger (18.11.2020 14:19:55): DB cannot prepare select on 'th0temp_month' (after 100 tries in 9280ms): attempt to write a readonly database (8)
logger (18.11.2020 14:20:04): DB cannot prepare select on 'th0temp_month' (after 100 tries in 9267ms): attempt to write a readonly database (8)
logger (18.11.2020 14:20:13): DB cannot prepare select on 'th0temp_year' (after 100 tries in 9202ms): attempt to write a readonly database (8)
logger (18.11.2020 14:20:23): DB cannot prepare select on 'th0temp_year' (after 100 tries in 9271ms): attempt to write a readonly database (8)
logger (18.11.2020 14:20:32): DB cannot prepare select on 'th0temp_year' (after 100 tries in 9206ms): attempt to write a readonly database (8)
logger (18.11.2020 14:20:41): DB cannot prepare select on 'th0hum_day' (after 100 tries in 9234ms): attempt to write a readonly database (8)
logger (18.11.2020 14:20:50): DB cannot prepare select on 'th0hum_day' (after 100 tries in 9263ms): attempt to write a readonly database (8)
logger (18.11.2020 14:20:59): DB cannot prepare select on 'th0hum_day' (after 100 tries in 9194ms): attempt to write a readonly database (8)
logger (18.11.2020 14:21:09): DB cannot prepare select on 'th0hum_month' (after 100 tries in 9224ms): attempt to write a readonly database (8)
logger (18.11.2020 14:21:18): DB cannot prepare select on 'th0hum_month' (after 100 tries in 9215ms): attempt to write a readonly database (8)
logger (18.11.2020 14:21:27): DB cannot prepare select on 'th0hum_month' (after 100 tries in 9232ms): attempt to write a readonly database (8)
logger (18.11.2020 14:21:36): DB cannot prepare select on 'th0hum_year' (after 100 tries in 9201ms): attempt to write a readonly database (8)
logger (18.11.2020 14:21:46): DB cannot prepare select on 'th0hum_year' (after 100 tries in 9199ms): attempt to write a readonly database (8)
logger (18.11.2020 14:21:55): DB cannot prepare select on 'th0hum_year' (after 100 tries in 9212ms): attempt to write a readonly database (8)
logger (18.11.2020 14:22:04): DB cannot prepare select on 'thb0seapress_day' (after 100 tries in 9205ms): attempt to write a readonly database (8)
logger (18.11.2020 14:22:13): DB cannot prepare select on 'thb0seapress_day' (after 100 tries in 9265ms): attempt to write a readonly database (8)
logger (18.11.2020 14:22:22): DB cannot prepare select on 'thb0seapress_day' (after 100 tries in 9195ms): attempt to write a readonly database (8)
logger (18.11.2020 14:22:32): DB cannot prepare select on 'thb0seapress_month' (after 100 tries in 9243ms): attempt to write a readonly database (8)
logger (18.11.2020 14:22:41): DB cannot prepare select on 'thb0seapress_month' (after 100 tries in 9262ms): attempt to write a readonly database (8)
logger (18.11.2020 14:22:50): DB cannot prepare select on 'thb0seapress_month' (after 100 tries in 9242ms): attempt to write a readonly database (8)
logger (18.11.2020 14:23:00): DB cannot prepare select on 'thb0seapress_year' (after 100 tries in 9630ms): attempt to write a readonly database (8)

[Gyvate]

"Progress" report:
I did a cold start (power cycle) - no change
tried to restore several backups - no change
tried to restore the database from my test/backup machine which is an identical clone of my prod system - no change
(the backup machine still runs on 14072 and the newest version on reboot is now deactivated).
So either the hardware is broken (unlikely) - or the 14074 introduced a bug which keeps my system down.

[galfert]

The Nov 16, 2020 release notes says:

• some behind the scenes changes in regard to storing data on NANO SD, MB PRO and Raspberry Pi.

source: https://www.meteobridge.com/wiki/index.php/Forum

I think that might not have gone over well and introduced a bug.

[Gyvate]

the Nov 16 release (14072) was still working ok - but the Nov 17 release (14074) seems to be the culprit.

[Gyvate]

Got it working again with "Rebuild on reboot".

Looks like the RAM DB was corrupted and messing up everything.
I had similar cases before, but then the system would just purge the RAM DB and start again from scratch (RAM DB).
Somehow this step didn't happen anymore.
Now I have to manually enter the missing data (6 hours) from the "mirrored" database. 12 entries per sensor and station.
Could have been worse.

But it's still not fully clear to me why a simple restore wouldn't work. 
The only explanation left to me that in such a case I have to connect to the MB via SSH,
delete the DB and the RAM DB and then only start the restore.

[docbee]

It is imho a system stretched to limit with database operations. Restoring backups in such a crowdy setup always is at risk to fail. However the rebuild on reboot option did fix it.

I don't think this very special usage pattern gives much hints for standard users ;.)

[Gyvate]

It is imho a system stretched to limit with database operations. Restoring backups in such a crowdy setup always is at risk to fail. However the rebuild on reboot option did fix it.

I don't think this very special usage pattern gives much hints for standard users ;.)

Come on, Boris. With all due respect. 
Having two stations with 18 sensors each ([Temp/Hum in, Temp/Hum out, solar, UV, wind, rain,] 5x soil moisture, 3 x extra Temp/Hum, Lightning, 2x PM2.5) is not too unusual (given that MB supports now up to five (!) stations).

Some simple mathematics:
a "classical" station has 7 sensors (WH2320E or HP2551 w/ WH65). 7 x 5 = 35, and I have 36 for 2 stations. So it's inside the warranted range, isn't it so ? And as for the http requests, there are 5 weather networks and 2 websites ...

The top model of the MB family, the MB Pro, should be able to manage that. And I have already spread the http load over two MB Pro machines.
By the way, I have already observed occasionally that strange behaviour earlier with only one station.
After some time the system then purges the RAM DB and starts from scratch again.
As I'm saving (exporting) my minute based data hourly, I'm not losing much when that purge occurs.
So I'm not sure it is only due to "extreme" loads on the database that this database corruption happens.

Right now it's working again and thanks for the swift support.

Keeping my fingers crossed that there was/is no deeper issue behind.
At least I know now how to handle such a situation in the future - and am happy that I invested in a backup machine (which interestingly didn't go down even though it mirrors the database  i.e. also runs on two stations and even took over all the additional load from the one with the DB error for the repair period). 

[docbee]

You are free to have a different opinion, what things should all be done in parallel on this units. What you have is all I can provide.

[galfert]

I'm not going against docbee by making the following recommendation. I'm simply just offering information that may be useful...

Meteobridge runs on top of OpenWRT. That said you are able to get into the OpenWRT settings and you can modify OpenWRT but you need to know what you are doing or you might mess up something that Meteobridge software requires. OpenWRT can be modified to have SNMP support:
https://openwrt.org/docs/guide-user/services/snmp/server

Do so at your own risk.