Author Topic: Backup and Export Data  (Read 1730 times)

0 Members and 1 Guest are viewing this topic.

Offline Robbie936

  • Member
  • *
  • Posts: 40
Backup and Export Data
« on: August 11, 2022, 11:43:56 PM »
I am trying to get my meteobridge pro to back up my data and export it to a usb stick plugged in to the front of the meteobridge.
If i set my database to the usb it starts a whole new database and i lose my old data.
I've set up a script to export data but it goes to the internal storage so i am not understanding something somewhere. I've looked at the official wiki and it isn't easy to follow, youtube isn't any help either.
I've spend hours playing with this thing and with it being so slow it's bloody tedious.
Does anyone have a guide or can they explain the steps i need to carry out in order to export my data to the usb stick.
Thank you



Offline weatherdoc

  • Senior Contributor
  • ****
  • Posts: 180
    • Willowsford Farm Weather
Re: Backup and Export Data
« Reply #1 on: August 12, 2022, 07:17:43 AM »
Not sure if this will be helpful since I don't use Meteobridge - I use Weather Display (WD). But I backup my WD "database" (C:\wdisplay folder) to a cloud (I use OneDrive) every night with commercial backup software (I use Aomei Backupper Pro). I have a Synology NAS server that I also backup C:\wdisplay to every night using Synology's server/client backup. I do a differential backup to save drive space. Once per month it does a full backup. That way I always have three copies of all my data - one on my PC where WD resides, one in the cloud for offsite storage, and on my server for onsite backup. Before I had a server, I backed up to an external USB drive.

Offline Robbie936

  • Member
  • *
  • Posts: 40
Re: Backup and Export Data
« Reply #2 on: August 12, 2022, 01:44:28 PM »
Thank you for your reply weatherdoc.
The menu and tab arrangement for meteobridge i find is difficult to follow and i do struggle with it.
I'll get there in the end i guess.

Offline Gyvate

  • Forecaster
  • *****
  • Posts: 3298
Re: Backup and Export Data
« Reply #3 on: August 15, 2022, 04:36:38 AM »
Meteobridge behaves as it's supposed to  8-) - but you'll need to understand what means what ;)
On the System --> Administration page you have different options
1. version control - determines if you want to download the latest update at the next reboot or not (or even downgrade to an earlier version)
2. database media - determines where your database is located - by preference on the internal SLC storage, but you can also choose an external USB device plugged to the front or rear USB port. however then you may risk the device (especially USB sticks/thumb drives) to get corrupted by the frequent I/Os (unless it is a SLC type of USB stick or a USB-HDD or USB-SSD). When you change the location, MB starts the database from scratch. If you have made a backup before, you can restore that to whereever your MB database is located (see next point)
3. Restore database  select media for restore - MB looks for a so-called "special device" (Linux terminology) at one of the USB ports. From there you can choose the backup to restore
4. Backup you have two options - use the button "backup now" or plan a regular database backup in the Service --> events page
if you press "backup now", MB start the script backup.sh located in the \\METEOBRIDGE\data\scripts folder - this is also the script to select when creating a regular backup event (task)
the script looks for a device connected to one of the USB ports formatted in FAT32. If it finds one, it writes a copy of the database with the name dbase-yyyymmddhhmm to the device (e.g. dbase-202208151800 for 18:00 today)

Suggestion:
- you choose "internal storage" for your database (database media)
- you plug in a FAT32 formatted USB stick into the front USB port
- you press "backup now" from the System --> Administration page

to see your backup, you have two possibilities:
1. unplug the USB stick and insert it to your PC - you should see the backup file(s)
2. (if you are familiar with linux and know how to use a SSH connection)
connect via SSH (e.g. using the PuTTY program) to your MB Pro
(on your computer start PuTTY, enter the IP address of your MB Pro, select port 22 and connect - username and password are the same which you use in your browser)
with the USB stick (device) plugged in, you have to figure out under which name it is known to MB; usually sda1 or sdb1
to verify enter: ("#" is the command prompt for the root user as which you are logged in to MB)
#ls /dev/sd* -l
then you have to mount the USB stick into the file system which is visible to you from your computer (now assuming your USB device is sda1)
we first have to create a mount point in the file system (a directory/folder)
#mkdir /tmp/mnt/data/USB
now you can see the folder/directory USB as \\METEOBRIDGE\data\USB on your computer (may need to refresh your Windows File Explorer if you use Windows on your PC)
now make the content of your USB device visible in the \\METEOBRIDGE\data\USB folder = mounting the device into the file system
#mount /dev/sda1 /tmp/mnt/data/USB
When you refresh your Windows Explorer, you can now see the content of the USB device in the \\METEOBRIDGE\data\USB folder.
You can check the successful mounting by entering
#df -h
this command shows all mounted file systems - and should now also show /dev/sda1, your USB device

Beware: the mount will be undone by the backup script whenever it runs - your backup files will seem to disappear
(they only disappear from your view, not from the USB device - you want to see them again - redo the mounting)

by the way - it makes sense to run a backup on your MB at least once a day ! (of course a FAT32 formatted USB device needs to be plugged in  8-))
If the USB device is full, MB will overwrite the oldest backup on the device
WS2350 1.6.7, GW1000(3) 1.7.7,WH2650 WiFi (2) 1.7.7 (test/backup), GW1100 2.3.1, GW2000(3) 3.1.1, HP2551 1.9.5,5.1.5;HP3500 1.7.2,WS3800 1.2.8, WN1910 1.2.3,WN1980 1.2.3;
Ecowitt WS90(2)1.3.5/1.4.0, WS80(2)1.2.5, WS68, WS69, WH40, WH31, WH31-EP, WN30, WN34L, WN35, WH32, WH32-EP, WH32B, WH57 [Lightning], WH41 [PM2.5], WH51, WH45, WH55
MeteobridgePro(2)[test,prod] 5.8 Mar 01 2024, 15185 - Blake-Larsen Sun Recorder - RPi4/weewx 4.8.0/4.10.2/CumulusMX 3283/Meteobridge RPi4B-2GB(3169)
Barani Meteoshield Pro, MetSpec Rad02 - Ecowitt 5763,34418;WU ISAARB3(WH4000SE),ISAARB22(HP2553), http://meshka.eu

Offline Robbie936

  • Member
  • *
  • Posts: 40
Re: Backup and Export Data
« Reply #4 on: August 15, 2022, 01:18:03 PM »
Gyvate,
Thank you for your comprehensive explanation, I'll digest it and try your suggestions.
Hopefully I'll gain a better understanding and some success!

Offline Robbie936

  • Member
  • *
  • Posts: 40
Re: Backup and Export Data
« Reply #5 on: August 15, 2022, 03:26:33 PM »
Gyvate,
I've managed to back up the database to a usb on the front panel manually using the backup now button on the administration page.
Where i am having trouble is trying to set up a backup to run once a day.  I've set up a script to export the current year but it isn't going to the usb. What am i missing?
Thanks

Offline Gyvate

  • Forecaster
  • *****
  • Posts: 3298
Re: Backup and Export Data
« Reply #6 on: August 16, 2022, 09:27:26 AM »
backup and export are two different kettle of fish.
for export you have to use an existing export template or modify an existing one to your convenience (and save it under a different name). The file extension has to be .exp
You can use e.g. standard-iso.exp
Services --> events
create new event: "new" (buttom of page), select service "Data export", selecty event type "periodical"  --> Add service event
then the event will be in the event list - there you will have to choose
Data export - when it's supposed to happen (never mind when it says periodical, you can deactivated the event by choosing "off")
time frame: current year
resolution: this depends on your settings in System/Administration - if you haven't chosen minute based archuiving at the end of the page, you can only export two days of minute based data which is kept in the RAM database
otherwise it is archived hour based
template: standard-iso
when this is done "save" first !!
then press "Test" and the export will be initiated in the background
the export file will be saved under \\METEOBRIDGE\data\export under the name of the template with the timestamp and a reference to the time frame
e.g standard-iso-2022081614-min.txt if the export timeframe was hourly and the resolution was minutes and the export happened at the 16-Aug-2022 for 14:00-14:59
the file will be a comma (or whatever separator is chosen inside the template from which the export was generated)  separated value (CSV) file

Look up the standard-iso.exp file in \\METEOBRIDGE\data\templates
you may need to make yourself familiar with the template language MB uses - see WiKi www.meteobridge.com --> Expert Corner --> Templates

to setup a daily backup - let's say at midnight - do the following
create a new event: service "Script execution", event type "periodical"
once the event is inside the event list, choose
"after every new day" (or any other periodicity you want)
at the combo box "select predefined script" select "script:backup.sh"
save
you can test it by pressing the test button of the event --> the backup will be started right away
WS2350 1.6.7, GW1000(3) 1.7.7,WH2650 WiFi (2) 1.7.7 (test/backup), GW1100 2.3.1, GW2000(3) 3.1.1, HP2551 1.9.5,5.1.5;HP3500 1.7.2,WS3800 1.2.8, WN1910 1.2.3,WN1980 1.2.3;
Ecowitt WS90(2)1.3.5/1.4.0, WS80(2)1.2.5, WS68, WS69, WH40, WH31, WH31-EP, WN30, WN34L, WN35, WH32, WH32-EP, WH32B, WH57 [Lightning], WH41 [PM2.5], WH51, WH45, WH55
MeteobridgePro(2)[test,prod] 5.8 Mar 01 2024, 15185 - Blake-Larsen Sun Recorder - RPi4/weewx 4.8.0/4.10.2/CumulusMX 3283/Meteobridge RPi4B-2GB(3169)
Barani Meteoshield Pro, MetSpec Rad02 - Ecowitt 5763,34418;WU ISAARB3(WH4000SE),ISAARB22(HP2553), http://meshka.eu

Offline Robbie936

  • Member
  • *
  • Posts: 40
Re: Backup and Export Data
« Reply #7 on: August 16, 2022, 03:53:56 PM »
Cheers Gyvate, I'll follow your instructions and see how i go.
Thanks

Offline Robbie936

  • Member
  • *
  • Posts: 40
Re: Backup and Export Data
« Reply #8 on: August 19, 2022, 03:21:04 PM »
Just an update, I've managed to see through the fog of meteobridge (pun intended) and now got my backups working.
Thank you Gyvate.
« Last Edit: August 19, 2022, 03:23:35 PM by Robbie936 »

Offline wsNoordbergum

  • Contributor
  • ***
  • Posts: 134
    • wsNoordbergum
Re: Backup and Export Data
« Reply #9 on: October 12, 2022, 11:15:44 AM »
Thanks for the instructions on how to make the USB visible in explorer
It worked
I have the database of my Nano SD and want to use it on my Raspberry Pi version of meteobridge, see if it works...
When I click on restore database and select the 186Mb sized database, I get the error of not enough space
I have a 2 Gb Swissbit SLC SD card and a 8Gb USB storage also the Pi has 2 Gb of RAM

Strange
Davis Vantage Pro 2 UV/Solar
Meteobridge nano SD
Ecowitt WS90 WH40 WH31 WH57 GW2000 HP2550_c
CumulusMX/raspberry Pi 4b

Offline Gyvate

  • Forecaster
  • *****
  • Posts: 3298
Re: Backup and Export Data
« Reply #10 on: October 12, 2022, 11:32:17 AM »
Thanks for the instructions on how to make the USB visible in explorer
It worked
I have the database of my Nano SD and want to use it on my Raspberry Pi version of meteobridge, see if it works...
When I click on restore database and select the 186Mb sized database, I get the error of not enough space
I have a 2 Gb Swissbit SLC SD card and a 8Gb USB storage also the Pi has 2 Gb of RAM

Strange
is the database in the format e.g. dbase-202210121400 ? unpacked ? or still as a .gz archive ?

The usual place of the database is the internal storage (see System --> Administration). If so, the internal available storage is
SDcard capacity - operating system - data in the exported \\METEOBRIDGE\data directory - actual database   ("-" = minus)
df -h should show you the available capacity of your internal storage - the RAM and the external USB don't play a role here
(connected via SSH to the RPi - console window e.g. PuTTY)

root@MeteoBridge:~# df -h
Filesystem                Size      Used Available Use% Mounted on
/dev/root               102.4M     73.1M     27.2M  73% /
tmpfs                   929.5M     48.7M    880.8M   5% /tmp
/dev/mmcblk0p1           63.9M     16.7M     47.1M  26% /boot       Storage used by OS
tmpfs                   512.0K      4.0K    508.0K   1% /dev
/dev/mmcblk0p3           14.4G      6.1G      8.1G  43% /tmp/mnt   internal user/application storage
root@MeteoBridge:~#

in my example my microSD card has a 16 GB capacity (metric) = 15 GB (binary) out of which 64 MB are for the OS and 14.4 GB (binary) for database and other user data
mmcblk0px are the two partitions on the internal SDcard.

there you have to look what is really available.

ls /tmp/mnt/dbase* -l
should show how much space the actual database takes
ls /tmp/mnt/data -l
should show the size of the exported file system \\METEOBRIDGE\data
(you will have to add the sizes of the single files inside the directory  8-))

« Last Edit: October 13, 2022, 04:00:06 AM by Gyvate »
WS2350 1.6.7, GW1000(3) 1.7.7,WH2650 WiFi (2) 1.7.7 (test/backup), GW1100 2.3.1, GW2000(3) 3.1.1, HP2551 1.9.5,5.1.5;HP3500 1.7.2,WS3800 1.2.8, WN1910 1.2.3,WN1980 1.2.3;
Ecowitt WS90(2)1.3.5/1.4.0, WS80(2)1.2.5, WS68, WS69, WH40, WH31, WH31-EP, WN30, WN34L, WN35, WH32, WH32-EP, WH32B, WH57 [Lightning], WH41 [PM2.5], WH51, WH45, WH55
MeteobridgePro(2)[test,prod] 5.8 Mar 01 2024, 15185 - Blake-Larsen Sun Recorder - RPi4/weewx 4.8.0/4.10.2/CumulusMX 3283/Meteobridge RPi4B-2GB(3169)
Barani Meteoshield Pro, MetSpec Rad02 - Ecowitt 5763,34418;WU ISAARB3(WH4000SE),ISAARB22(HP2553), http://meshka.eu

Offline Gyvate

  • Forecaster
  • *****
  • Posts: 3298
Re: Backup and Export Data
« Reply #11 on: October 13, 2022, 04:01:19 AM »
to add to the before post (new post for @wsNoordbergum not to miss reading)

so you will first have to see if there is enough space for your database - if not, what files are/have become too big and might need to be removed - once you have enough space, just delete the existing database files, copy your (unpacked !) database over and reboot

# rm /tmp/mnt/dbase*
# cp /tmp/mnt/data/USB/dbase-202210121400 /tmp/mnt/dbase    (copy over and rename to "dbase")
# reboot                                                                                     (the "reboot" command will reboot the MB OS !)

your USB stick needs to be mounted into the USB directory which needs to exist in your \\METEOBRIDGE\data directory tree before you can execute this copy command - or into any other directory/mount point inside that tree e.g.
# mkdir /tmp/mnt/data/USB
# mount /dev/sda1 /tmp/mnt/data/USB

the backup database name in the copy command is an example following the file name convention MB uses when creating backups

at reboot MB will take over the "new" (restored) database, create a new RAM database and all other files needed
WS2350 1.6.7, GW1000(3) 1.7.7,WH2650 WiFi (2) 1.7.7 (test/backup), GW1100 2.3.1, GW2000(3) 3.1.1, HP2551 1.9.5,5.1.5;HP3500 1.7.2,WS3800 1.2.8, WN1910 1.2.3,WN1980 1.2.3;
Ecowitt WS90(2)1.3.5/1.4.0, WS80(2)1.2.5, WS68, WS69, WH40, WH31, WH31-EP, WN30, WN34L, WN35, WH32, WH32-EP, WH32B, WH57 [Lightning], WH41 [PM2.5], WH51, WH45, WH55
MeteobridgePro(2)[test,prod] 5.8 Mar 01 2024, 15185 - Blake-Larsen Sun Recorder - RPi4/weewx 4.8.0/4.10.2/CumulusMX 3283/Meteobridge RPi4B-2GB(3169)
Barani Meteoshield Pro, MetSpec Rad02 - Ecowitt 5763,34418;WU ISAARB3(WH4000SE),ISAARB22(HP2553), http://meshka.eu

Offline wsNoordbergum

  • Contributor
  • ***
  • Posts: 134
    • wsNoordbergum
Re: Backup and Export Data
« Reply #12 on: October 13, 2022, 06:57:18 AM »
One step at a time....
My newly created database on the RPi is about 2Mb and not a gz file
I am in the trial period so I can try some things and see if a RPi is a good substitute for my Nano SD in the future
The other database file of my Nano SD is gz type, so it is compressed, after decompression it is over 600Mb
I've done the putty thing and here are some screenshots attached
If I perform a backup on my RPi it is writing the backup onto the USB attached storage and not onto the SD card
I cannot choose in meteobridge where I want the backup to be stored
So as seen on the screenshot, OS is about 64Mb and free space is over 1Gb
I do not have a problem with the backup being wrote onto the USB storage but on my Nano SD it is stored in the backup folder of the SD card
On the screenshot is visible hat there is no dbase file present on the SD card
Or there more users of RPi meteobridge who have the dbase files written onto the USB flash drive and not onto the SD card
I will try some troubleshooting and swap the USB drive with another type
See what it does when I perform a backup then....


Davis Vantage Pro 2 UV/Solar
Meteobridge nano SD
Ecowitt WS90 WH40 WH31 WH57 GW2000 HP2550_c
CumulusMX/raspberry Pi 4b

Offline Gyvate

  • Forecaster
  • *****
  • Posts: 3298
Re: Backup and Export Data
« Reply #13 on: October 13, 2022, 07:54:37 AM »
One step at a time....
My newly created database on the RPi is about 2Mb and not a gz file
I am in the trial period so I can try some things and see if a RPi is a good substitute for my Nano SD in the future
The other database file of my Nano SD is gz type, so it is compressed, after decompression it is over 600Mb
I've done the putty thing and here are some screenshots attached
If I perform a backup on my RPi it is writing the backup onto the USB attached storage and not onto the SD card
I cannot choose in meteobridge where I want the backup to be stored
So as seen on the screenshot, OS is about 64Mb and free space is over 1Gb
I do not have a problem with the backup being wrote onto the USB storage but on my Nano SD it is stored in the backup folder of the SD card
On the screenshot is visible hat there is no dbase file present on the SD card
Or there more users of RPi meteobridge who have the dbase files written onto the USB flash drive and not onto the SD card
I will try some troubleshooting and swap the USB drive with another type
See what it does when I perform a backup then....
I think you have mixed up two commands ...
the command you show in your screenshot is ls /tmp/mnt/data/dbase* -l - and the OS reply is correct - not found
what you probably wanted to do is:
ls /tmp/mnt/dbase* -l   ;)
the database files are located in /tmp/mnt - not in /tmp/mnt/data

my question regarding the .gz file was in reference to your (NANO) backup
given the df -h info, you should be able to replace the actual database in the /tmp/mnt directory by using the commands mentioned earlier - after having unpacked the .gz file from the NANO and copied it onto your USB device

and - on a RPi - you cannot choose the location of the database neither the backup device - you can only attach different USB devices - the SLC (!) SDcard has been chosen for the database for a purpose !
(so you can't change the database location except with a few "dirty" tricks which however wouldn't usually make much sense, unless you wanted to use a SSD as your boot and database medium. The latter would usually be a waste of space as openWRT would reserve the whole disk for MB.)
And - from a data protection point of view it makes a lot of sense writing the backup to an external device

You may have noticed that I wrote USB device, i.e. any storage device connected to a USB port - the operating system is oblivious of what USB storage device you have - it only needs to be formatted in FAT32 for the backup to be written there which allows for a maximum size of 2 Terabytes.
« Last Edit: October 13, 2022, 08:07:07 AM by Gyvate »
WS2350 1.6.7, GW1000(3) 1.7.7,WH2650 WiFi (2) 1.7.7 (test/backup), GW1100 2.3.1, GW2000(3) 3.1.1, HP2551 1.9.5,5.1.5;HP3500 1.7.2,WS3800 1.2.8, WN1910 1.2.3,WN1980 1.2.3;
Ecowitt WS90(2)1.3.5/1.4.0, WS80(2)1.2.5, WS68, WS69, WH40, WH31, WH31-EP, WN30, WN34L, WN35, WH32, WH32-EP, WH32B, WH57 [Lightning], WH41 [PM2.5], WH51, WH45, WH55
MeteobridgePro(2)[test,prod] 5.8 Mar 01 2024, 15185 - Blake-Larsen Sun Recorder - RPi4/weewx 4.8.0/4.10.2/CumulusMX 3283/Meteobridge RPi4B-2GB(3169)
Barani Meteoshield Pro, MetSpec Rad02 - Ecowitt 5763,34418;WU ISAARB3(WH4000SE),ISAARB22(HP2553), http://meshka.eu

Offline wsNoordbergum

  • Contributor
  • ***
  • Posts: 134
    • wsNoordbergum
Re: Backup and Export Data
« Reply #14 on: October 13, 2022, 08:06:59 AM »
Gyvate, thank you for your support

It makes things for me clear and since I'm a new by in RPi stuff the location of the backup's are not that important
It is off course important that I can use the backup in meteobridge if necessary
I will try to import data from my nano SD export files
Give that a try

Again thank you for your quick response and support

Ps the RPi works great and is a lot more quicker than the Nano SD
It will replace my Nano SD in the future
Davis Vantage Pro 2 UV/Solar
Meteobridge nano SD
Ecowitt WS90 WH40 WH31 WH57 GW2000 HP2550_c
CumulusMX/raspberry Pi 4b

Offline Gyvate

  • Forecaster
  • *****
  • Posts: 3298
Re: Backup and Export Data
« Reply #15 on: October 13, 2022, 08:11:45 AM »
yep, MB on a RPi, especially on a RPi4B, gives a high performance experience.
Especially when you develop special charts (metacharts) over longer periods which need a lot of data reads
Even with - in my case - three stations and about 60 different minute-based observation objects to be saved in the database per station.
Neither a NANO SD nor a Meteobridge Pro would cope with this. Tested that myself. But the RPi4B just works like a charm.

Quote
I will try to import data from my nano SD export files
I think you read my suggestions for that in the MB forum at forum.meteohub.de
especially how to get the export files into the proper format to be easily used for import in the direct format.
« Last Edit: October 13, 2022, 08:16:33 AM by Gyvate »
WS2350 1.6.7, GW1000(3) 1.7.7,WH2650 WiFi (2) 1.7.7 (test/backup), GW1100 2.3.1, GW2000(3) 3.1.1, HP2551 1.9.5,5.1.5;HP3500 1.7.2,WS3800 1.2.8, WN1910 1.2.3,WN1980 1.2.3;
Ecowitt WS90(2)1.3.5/1.4.0, WS80(2)1.2.5, WS68, WS69, WH40, WH31, WH31-EP, WN30, WN34L, WN35, WH32, WH32-EP, WH32B, WH57 [Lightning], WH41 [PM2.5], WH51, WH45, WH55
MeteobridgePro(2)[test,prod] 5.8 Mar 01 2024, 15185 - Blake-Larsen Sun Recorder - RPi4/weewx 4.8.0/4.10.2/CumulusMX 3283/Meteobridge RPi4B-2GB(3169)
Barani Meteoshield Pro, MetSpec Rad02 - Ecowitt 5763,34418;WU ISAARB3(WH4000SE),ISAARB22(HP2553), http://meshka.eu

 

anything