WXforum.net
Weather Software => RaspberryPI Weather Software => Topic started by: Wayne Orwig on April 17, 2020, 10:43:51 AM
-
I have a Raspberry PI 4, that pulls the data from my weather station off of ambientweather.net once a minute. I make a copy of that data, once every fifteen minutes to look for trends. The Raspberry PI is a web server that also randomly displays a new photo from a directory once per minute. So in the end, I have a web page that is a 'digital picture frame', that also displays my weather. The scripts that do this, mostly rename files, manipulating directories, not actually copying or moving a lot of data.
The question I have is related to the life of the SDXC card. I have a 64GB card, that is mostly empty. I am hoping the card is smart enough to spread the wear out.
Does anyone have a clue about how long those cards last?
Should I start working on putting the file system onto an external hard disk?
I keep it backed up well, so data loss isn't the issue, just the annoyance of setting everything back up.
-
WeeWX has this page about using RAM disks for scratch files:
https://github.com/weewx/weewx/wiki/Minimize-writes-on-SD-cards (https://github.com/weewx/weewx/wiki/Minimize-writes-on-SD-cards)
I have been using these strategies along with clone/rotate my weather Pi's onto fresh SD cards every year, and have not had a data loss so far. The last round of SD cards that I bought were dashcam-grade, which will hopefully provide a little bit of extra insurance.....
-
I save my data on a USB drive instead. This includes CRONTAB log files, automatic timelapse video creation. That way, I won't wear the SD Card where the OS resides.
-
I got some USB memory sticks. Now to move everything that I can that is written to to the USB stick.
-
Install your OS to USB and use a copy/copy to another USB for backup.
-
And this: http://www.pidramble.com/wiki/benchmarks/microsd-cards
-
And check the section on SD cards here: https://www.meteobridge.com/wiki/index.php/Meteobridge_NANO_SD
-
I use a small USB 3.0 SSD disk, you can pick up 120GB SSDs pretty cheap. The SSD is more durable, and the extra size means more wear levelling as well. And yes redirect all your temporary/transient files to a memory disk.
Only slight issue is the pi 4 unlike the 3 cannot currently boot from a USB disk, so you still need a boot partition on an SD card.
-
Well poo.
I moved most of my scripts and working files to the external memory stick. I have a script that runs in the middle of the night that backs everything up to a NAS drive. Now, that hangs up randomly.
Always something.
-
you can also use, instead or in addition to strategies like temporary file systems in RAM as proposed for Weewx, SLC SD cards (industrial standard, for high usage).
A 16 GB SLC from WD/SanDisk costs around 30 USD from Arrow or Mouser. For Meteobridge on the RPi the use of SLC storage is even enforced to keep the data(base) safe.
And the other strategies like re-directing your database to an external drive (hard disk, SSD, USB drive .... - but consumer USB drives (sticks etc., not USB connected hard drives) have similar weaknesses like SD cards) also work. Or, at the end of the day, starting your OS from an external disk (HDD, SSD, ...).
Let's say, in my opinion, the use of SLC storage is the simplest way to do - you just use another type of SD card.
They are availble from 2GB thru 64GB at still reasonable prices, but then, how many years does it take for a 16 GB SD card to fill with weather data created by a data logger.
My Meteobridge on RPi4 uses a 16GB SLC SD card, logs data from two stations with many sensors, and won't be full before 9 years.
And backups are just copied to a connected external disk drive (via USB).
For weewx and CumulusMX the more sophisticated strategies are needed, but then no SLC SD card needed (as if external SSD drives came for free).
If you (or the users) are not an IT adept, the SLC approach is the easiest and most efficient thing to do.
The other above mentioned strategies need some up to rather deep(er) knowledge of the operating system and the functioning of a computer
like mounting drives, permanently, temporarily. The function of the fstab, ownership and permission considerations .... you name it.
-
Well over the past year I have had to replace the USB memory sticks a couple of times.
In the mean time they have developed firmware for the PI 4 that makes it simple to move the OS to an external USB drive.
So I got a cheap USB SSD drive, and moved everything to that external drive. We'll see how long that lasts.
-
I'm also using old and now re-purposed 2.5" HDDs in an external case via USB for the database and for hourly database backups.
The speed is, even with the USB3 interface of the RPi4, not an issue - in the sense that the difference between external HDD and an external SSD is not noticeable.
And the "smaller" disks from my earlier laptops (250, 500 GB, 1 TB) are repurposed. I wouldn't recommend a HDD for the OS though.
-
I thought initially that all of my problems were related to power glitches. Now I'm thinking that it was mostly due to poor SD cards and memory sticks.
I keep my data well backed up, but I need to find a way to back up the external OS. Maybe a crappy memory stick. :shock:
-
I thought initially that all of my problems were related to power glitches. Now I'm thinking that it was mostly due to poor SD cards and memory sticks.
I keep my data well backed up, but I need to find a way to back up the external OS. Maybe a crappy memory stick. :shock:
You could create once in a while an image of your OS device (e.g. Win32 Disk Imager, dd on Linux, ...).
My Windows PCs are regularly (every 5 days) automatically backed-up by creating an OS drive disk image.
Has already turned out to be very useful ... 8-)
-
I have a number of pi's at remote locations so I am concerned about reliability. I have not had any problems so far, but I don't want any in the future.
By far, the best answer would be a SLC SD card (and I use SLC cards for Meteobridge). But for my usage I've found that the Sandisk Industrial card works well. It's about three dollars more than consumer grade and it's money well spent.
https://www.amazon.com/dp/B07CV344WJ (https://www.amazon.com/dp/B07CV344WJ)
-
I've had one remote site running on the same SD card since April 2015. Only configuration change I've made there to try and extend SD card lifespan is transmitting live data via RabbitMQ rather than the database. Logs, etc, still go to the SD card. The Pi is also on a proper UPS so it rarely if ever experiences unexpected power outages - it shuts down automatically when the UPS battery starts getting low.
Not sure what model SD card it is - IIRC nothing special, just some raspberry pi branded one preloaded with raspbian I got from element14 at the same time as the Pi.
Just in case it eventually fails databases are automatically backed up to a flash drive every night and every 6-12 months or so I image the SD card so in theory I should be able to get it up and running again fairly quickly should it fail. I'm still kind of tempted to replace it sometime rather than waiting for it to fail - 6 years seems like quite a long time for one SD card.
-
I've got a Pi responsible for receiving and forwarding data to an external server. Had some bad experiences with cards failing. Might have been due to the quality or power failures. But I try to write a little data as possible on the sd card. Hardly any logging activity. All incoming data is only stored in RAM.
Have had the idea to use an Intel NUC in stead of a Pi as primary processor of sensor data, but I guess running it on battery backup might be more of an issue.
-
How about using a small SSD drive via a USB to SATA cable. Small SSD's are cheap and the cable is cheap. Also lots of info on the Internet regarding booting the pi right from the SSD.
That's my plan just to have too many projects going on to get it done right now.
Cheers
MikeyM