Author Topic: Raspberry PI 4 SDXC life concern  (Read 1444 times)

0 Members and 1 Guest are viewing this topic.

Offline Wayne Orwig

  • Member
  • *
  • Posts: 20
Raspberry PI 4 SDXC life concern
« on: April 17, 2020, 10:43:51 AM »
I have a Raspberry PI 4, that pulls the data from my weather station off of ambientweather.net once a minute. I make a copy of that data, once every fifteen minutes to look for trends. The Raspberry PI is a web server that also randomly displays a new photo from a directory once per minute. So in the end, I have a web page that is a 'digital picture frame', that also displays my weather. The scripts that do this, mostly rename files, manipulating directories, not actually copying or moving a lot of data.

The question I have is related to the life of the SDXC card. I have a 64GB card, that is mostly empty. I am hoping the card is smart enough to spread the wear out.
Does anyone have a clue about how long those cards last?
Should I start working on putting the file system onto an external hard disk?

I keep it backed up well, so data loss isn't the issue, just the annoyance of setting everything back up.

Offline vreihen

  • El Niņo chaser
  • Forecaster
  • *****
  • Posts: 1144
  • K2BIG
Re: Raspberry PI 4 SDXC life concern
« Reply #1 on: April 17, 2020, 11:31:47 AM »
WeeWX has this page about using RAM disks for scratch files:

https://github.com/weewx/weewx/wiki/Minimize-writes-on-SD-cards

I have been using these strategies along with clone/rotate my weather Pi's onto fresh SD cards every year, and have not had a data loss so far.  The last round of SD cards that I bought were dashcam-grade, which will hopefully provide a little bit of extra insurance.....
WU Gold Stars for everyone! :lol:

Offline rosegeek

  • Member
  • *
  • Posts: 32
    • Canton MI USA PWS
Re: Raspberry PI 4 SDXC life concern
« Reply #2 on: April 25, 2020, 08:37:29 PM »
I save my data on a USB drive instead.  This includes CRONTAB log files, automatic timelapse video creation.  That way, I won't wear the SD Card where the OS resides.
Davis VP2, Meteobridge TPLink, PWS linked to Rachio Sprinkler System.
http://weather.gloryhomemi.com
http://weatherconsole.gloryhomemi.com

Offline Wayne Orwig

  • Member
  • *
  • Posts: 20
Re: Raspberry PI 4 SDXC life concern
« Reply #3 on: April 29, 2020, 06:07:08 PM »
I got some USB memory sticks. Now to move everything that I can that is written to to the USB stick.

Offline Bushman

  • Forecaster
  • *****
  • Posts: 7552
    • Eagle Bay Weather
Re: Raspberry PI 4 SDXC life concern
« Reply #4 on: April 29, 2020, 06:26:37 PM »
Install your OS to USB and use a copy/copy to another USB for backup.
Need low cost IP monitoring?  http://wirelesstag.net/wta.aspx?link=NisJxz6FhUa4V67/cwCRWA or PM me for 50% off Wirelesstags!!

Offline Bushman

  • Forecaster
  • *****
  • Posts: 7552
    • Eagle Bay Weather
Re: Raspberry PI 4 SDXC life concern
« Reply #5 on: April 29, 2020, 06:27:33 PM »
Need low cost IP monitoring?  http://wirelesstag.net/wta.aspx?link=NisJxz6FhUa4V67/cwCRWA or PM me for 50% off Wirelesstags!!

Offline Bushman

  • Forecaster
  • *****
  • Posts: 7552
    • Eagle Bay Weather
Re: Raspberry PI 4 SDXC life concern
« Reply #6 on: April 29, 2020, 06:28:32 PM »
Need low cost IP monitoring?  http://wirelesstag.net/wta.aspx?link=NisJxz6FhUa4V67/cwCRWA or PM me for 50% off Wirelesstags!!

Offline mcrossley

  • Forecaster
  • *****
  • Posts: 755
    • Wilmslow Astro
Re: Raspberry PI 4 SDXC life concern
« Reply #7 on: April 30, 2020, 06:54:49 AM »
I use a small USB 3.0 SSD disk, you can pick up 120GB SSDs pretty cheap. The SSD is more durable, and the extra size means more wear levelling as well. And yes redirect all your temporary/transient files to a memory disk.

Only slight issue is the pi 4 unlike the 3 cannot currently boot from a USB disk, so you still need a boot partition on an SD card.
Mark

Offline Wayne Orwig

  • Member
  • *
  • Posts: 20
Re: Raspberry PI 4 SDXC life concern
« Reply #8 on: May 01, 2020, 03:50:21 PM »
Well poo.

I moved most of my scripts and working files to the external memory stick. I have a script that runs in the middle of the night that backs everything up to a NAS drive. Now, that hangs up randomly.

Always something.

Offline Gyvate

  • Forecaster
  • *****
  • Posts: 913
Re: Raspberry PI 4 SDXC life concern
« Reply #9 on: January 22, 2021, 05:00:07 AM »
you can also use, instead or in addition to strategies like temporary file systems in RAM as proposed for Weewx, SLC SD cards (industrial standard, for high usage).
A 16 GB SLC from WD/SanDisk costs around 30 USD from Arrow or Mouser. For Meteobridge on the RPi the use of SLC storage is even enforced to keep the data(base) safe.

And the other strategies like re-directing your database to an external drive (hard disk, SSD, USB drive .... - but consumer USB drives (sticks etc., not USB connected hard drives) have similar weaknesses like SD cards) also work. Or, at the end of the day, starting your OS from an external disk (HDD, SSD, ...).

Let's say, in my opinion, the use of SLC storage is the simplest way to do - you just use another type of SD card.
They are availble from 2GB thru 64GB at still reasonable prices, but then, how many years does it take for a 16 GB SD card to fill with weather data created by a data logger.
My Meteobridge on RPi4 uses a 16GB SLC SD card, logs data from two stations with many sensors, and won't be full before 9 years.
And backups are just copied to a connected external disk drive (via USB).
For weewx and CumulusMX the more sophisticated strategies are needed, but then no SLC SD card needed (as if external SSD drives came for free).

If you (or the users) are not an IT adept, the SLC approach is the easiest and most efficient thing to do.
The other above mentioned strategies need some up to rather deep(er) knowledge of the operating system and the functioning of a computer
like mounting drives, permanently, temporarily. The function of the fstab, ownership and permission considerations .... you name it.
« Last Edit: January 22, 2021, 05:01:50 AM by Gyvate »
Froggit WH4000SE (=EW 2320E) 1.6.0/GW1100/Froggit DP1500 1.6.8 (+EW GW1000 (4) 1.6.8 test/backup)/HP1000SE Pro (HP2551) 1.6.0/1.7.4, WH2650 WiFi (2) 1.6.8
Ecowitt WS80,WH40,WH65,WH31(5),WH31-EP(1),WN30/DP30 (1),WH32(1),WH32-EP(1),WH32B [T/H/P] (1), WH57 [Lightning] (1), WH41 [PM2.5] (2), WH51 [SoilMoisture] (5), WH45, WH55 (2)
Meteobridge Pro (2) [test,prod] 5.2 Aug 16 2021, 14529 - Blake-Larsen Sun Recorder - RPi4/weewx 4.5.1/CumulusMX 3143/MB RPi4 (2546)
Meteoshield Pro, MetSpec Rad02
Ecowitt 5763,34418;WU ISAARB3(WH4000SE), ISAARB22(HP2553);Weathercloud 3011399141, http://meshka.eu/

Offline Wayne Orwig

  • Member
  • *
  • Posts: 20
Re: Raspberry PI 4 SDXC life concern
« Reply #10 on: July 30, 2021, 09:22:42 AM »
Well over the past year I have had to replace the USB memory sticks a couple of times.
In the mean time they have developed firmware for the PI 4 that makes it simple to move the OS to an external USB drive.
So I got a cheap USB SSD drive, and moved everything to that external drive. We'll see how long that lasts.

Offline Gyvate

  • Forecaster
  • *****
  • Posts: 913
Re: Raspberry PI 4 SDXC life concern
« Reply #11 on: July 30, 2021, 10:10:01 AM »
I'm also using old and now re-purposed 2.5" HDDs in an external case via USB for the database and for hourly database backups.
The speed is, even with the USB3 interface of the RPi4, not an issue - in the sense that the difference between external HDD and an external SSD is not noticeable.
And the "smaller" disks from my earlier laptops (250, 500 GB, 1 TB) are repurposed. I wouldn't recommend a HDD for the OS though.
Froggit WH4000SE (=EW 2320E) 1.6.0/GW1100/Froggit DP1500 1.6.8 (+EW GW1000 (4) 1.6.8 test/backup)/HP1000SE Pro (HP2551) 1.6.0/1.7.4, WH2650 WiFi (2) 1.6.8
Ecowitt WS80,WH40,WH65,WH31(5),WH31-EP(1),WN30/DP30 (1),WH32(1),WH32-EP(1),WH32B [T/H/P] (1), WH57 [Lightning] (1), WH41 [PM2.5] (2), WH51 [SoilMoisture] (5), WH45, WH55 (2)
Meteobridge Pro (2) [test,prod] 5.2 Aug 16 2021, 14529 - Blake-Larsen Sun Recorder - RPi4/weewx 4.5.1/CumulusMX 3143/MB RPi4 (2546)
Meteoshield Pro, MetSpec Rad02
Ecowitt 5763,34418;WU ISAARB3(WH4000SE), ISAARB22(HP2553);Weathercloud 3011399141, http://meshka.eu/

Offline Wayne Orwig

  • Member
  • *
  • Posts: 20
Re: Raspberry PI 4 SDXC life concern
« Reply #12 on: July 30, 2021, 11:24:43 AM »
I thought initially that all of my problems were related to power glitches. Now I'm thinking that it was mostly due to poor SD cards and memory sticks.

I keep my data well backed up, but I need to find a way to back up the external OS. Maybe a crappy memory stick.  :shock:


Offline Gyvate

  • Forecaster
  • *****
  • Posts: 913
Re: Raspberry PI 4 SDXC life concern
« Reply #13 on: July 30, 2021, 02:00:15 PM »
I thought initially that all of my problems were related to power glitches. Now I'm thinking that it was mostly due to poor SD cards and memory sticks.

I keep my data well backed up, but I need to find a way to back up the external OS. Maybe a crappy memory stick.  :shock:
You could create once in a while an image of your OS device (e.g. Win32 Disk Imager, dd on Linux, ...).
My Windows PCs are regularly (every 5 days) automatically backed-up by creating an OS drive disk image.
Has already turned out to be very useful ...  8-)
Froggit WH4000SE (=EW 2320E) 1.6.0/GW1100/Froggit DP1500 1.6.8 (+EW GW1000 (4) 1.6.8 test/backup)/HP1000SE Pro (HP2551) 1.6.0/1.7.4, WH2650 WiFi (2) 1.6.8
Ecowitt WS80,WH40,WH65,WH31(5),WH31-EP(1),WN30/DP30 (1),WH32(1),WH32-EP(1),WH32B [T/H/P] (1), WH57 [Lightning] (1), WH41 [PM2.5] (2), WH51 [SoilMoisture] (5), WH45, WH55 (2)
Meteobridge Pro (2) [test,prod] 5.2 Aug 16 2021, 14529 - Blake-Larsen Sun Recorder - RPi4/weewx 4.5.1/CumulusMX 3143/MB RPi4 (2546)
Meteoshield Pro, MetSpec Rad02
Ecowitt 5763,34418;WU ISAARB3(WH4000SE), ISAARB22(HP2553);Weathercloud 3011399141, http://meshka.eu/

Offline WA4OPQ

  • Senior Contributor
  • ****
  • Posts: 295
  • 4 stations: 2902 array, GW1000, 3 on Meteobridge
Re: Raspberry PI 4 SDXC life concern
« Reply #14 on: August 12, 2021, 10:41:49 PM »
I have a number of pi's at remote locations so I am concerned about reliability. I have not had any problems so far, but I don't want any in the future.
By far, the best answer would be a SLC SD card (and I use SLC cards for Meteobridge).  But for my usage I've found that the Sandisk Industrial card works well. It's about three dollars more than consumer grade and it's money well spent.

https://www.amazon.com/dp/B07CV344WJ

Offline davidg_nz

  • Member
  • *
  • Posts: 27
    • Sandy Bay Weather
Re: Raspberry PI 4 SDXC life concern
« Reply #15 on: August 13, 2021, 12:05:20 AM »
I've had one remote site running on the same SD card since April 2015. Only configuration change I've made there to try and extend SD card lifespan is transmitting live data via RabbitMQ rather than the database. Logs, etc, still go to the SD card. The Pi is also on a proper UPS so it rarely if ever experiences unexpected power outages - it shuts down automatically when the UPS battery starts getting low.

Not sure what model SD card it is - IIRC nothing special, just some raspberry pi branded one preloaded with raspbian I got from element14 at the same time as the Pi.

Just in case it eventually fails databases are automatically backed up to a flash drive every night and every 6-12 months or so I image the SD card so in theory I should be able to get it up and running again fairly quickly should it fail. I'm still kind of tempted to replace it sometime rather than waiting for it to fail - 6 years seems like quite a long time for one SD card.
Port Charles, Coromandel, New Zealand | Wired Vantage Pro2+, Ubiquiti AirCam
Hamilton, Waikato, New Zealand | Wireless Vantage Pro2+, 24h FARS, Leaf+Soil Station, Envoy

Offline hmderek

  • Contributor
  • ***
  • Posts: 103
    • Meteodrenthe
Re: Raspberry PI 4 SDXC life concern
« Reply #16 on: August 13, 2021, 08:52:06 AM »
I've got a Pi responsible for receiving and forwarding data to an external server. Had some bad experiences with cards failing. Might have been due to the quality or power failures. But I try to write a little data as possible on the sd card. Hardly any logging activity. All incoming data is only stored in RAM.

Have had the idea to use an Intel NUC in stead of a Pi as primary processor of sensor data, but I guess running it on battery backup might be more of an issue.
Davis VP2
Davis WeatherLink
Sensirion SHT35
Apogee Instruments TS-100
Barani Meteoshield Pro
Davis 7714
Davis AirLink
Wemos D1 Mini micro controllers
Voltaic solar panels
https://blog.meteodrenthe.nl
https://meteodrenthe.nl

Offline mikeym2m

  • Contributor
  • ***
  • Posts: 142
    • MikeyM's Thornhurst PA Weather
Re: Raspberry PI 4 SDXC life concern
« Reply #17 on: August 13, 2021, 02:12:07 PM »
How about using a small SSD drive via a USB to SATA cable. Small SSD's are cheap and the cable is cheap. Also lots of info on the Internet regarding booting the pi right from the SSD.

That's my plan just to have too many projects going on to get it done right now.

Cheers

MikeyM