Author Topic: Suggestions with local storage / USB  (Read 2010 times)

0 Members and 1 Guest are viewing this topic.

Offline whitling2k

  • Member
  • *
  • Posts: 9
Suggestions with local storage / USB
« on: June 21, 2021, 12:19:16 PM »
I have had my trusty old WH1080 running since c.2013, and it is slowly dying.   I started looking into a replacement earlier this year; and after a little amount of research ended up with a Bresser Explore Scientific 5-in-1 Complete WiFi Weather Station.  I was wooed in by the ability to upload my data to the cloud, then download it for local archiving, analysis and visualising.

Sadly, the 5-in-1 WiFi has been nothing but a let down.  The sensors seem OK (good enough for me), and more accurate when compared to other sensors, than my WH1080, but it loses so much data.  I'm losing 10-20% of the data that it should be pushing up per day.  Random observations, full rows, you name it.  It has been like it since the week after I set it up.  It's slowly getting worse.


Are there any stations (up to ~£200) that will save records (@~15 min) for a few (3+) months at a time, and allow me to download them (CSV or equivalent) to my PC via USB?

I'm mostly interested in logging internal temp & humidity, external temp & humidity, rainfall accumulations, wind speed & direction and barometric pressure.  Anything else is a bonus.

It would be nice if it could auto-upload to Weathercloud (or equivalent free service), but this is a nice-to-have and not a must.  The local storage and analysis is more important than viewing online for me.

I do have a personal cloud on a NAS server, but I don't have the time or energy right now to start doing clever things with those - but it is a direction of travel I would consider when the kids grow up more - so only tempted by such options if it's out box & minimum config.

Thanks in advance :-D


Offline doubleohwhatever

  • Senior Contributor
  • ****
  • Posts: 252
Re: Suggestions with local storage / USB
« Reply #1 on: June 21, 2021, 01:25:27 PM »
Have you checked out WeeWx for logging?

Offline whitling2k

  • Member
  • *
  • Posts: 9
Re: Suggestions with local storage / USB
« Reply #2 on: June 21, 2021, 03:29:05 PM »
No, I've not come across that.  It looks like something for the future though really.  For now, I just want a reliable station I can use to keep my record growing.

Offline box

  • Forecaster
  • *****
  • Posts: 384
Re: Suggestions with local storage / USB
« Reply #3 on: June 26, 2021, 05:41:25 AM »
I have had my trusty old WH1080 running since c.2013, and it is slowly dying.   I started looking into a replacement earlier this year; and after a little amount of research ended up with a Bresser Explore Scientific 5-in-1 Complete WiFi Weather Station.  I was wooed in by the ability to upload my data to the cloud, then download it for local archiving, analysis and visualising.

Sadly, the 5-in-1 WiFi has been nothing but a let down.  The sensors seem OK (good enough for me), and more accurate when compared to other sensors, than my WH1080, but it loses so much data.  I'm losing 10-20% of the data that it should be pushing up per day.  Random observations, full rows, you name it.  It has been like it since the week after I set it up.  It's slowly getting worse.


Are there any stations (up to ~£200) that will save records (@~15 min) for a few (3+) months at a time, and allow me to download them (CSV or equivalent) to my PC via USB?

I'm mostly interested in logging internal temp & humidity, external temp & humidity, rainfall accumulations, wind speed & direction and barometric pressure.  Anything else is a bonus.

It would be nice if it could auto-upload to Weathercloud (or equivalent free service), but this is a nice-to-have and not a must.  The local storage and analysis is more important than viewing online for me.

I do have a personal cloud on a NAS server, but I don't have the time or energy right now to start doing clever things with those - but it is a direction of travel I would consider when the kids grow up more - so only tempted by such options if it's out box & minimum config.

Thanks in advance :-D

My set up does these things, see my sig

I use the gw1000 to gateway across to a pc running cumulus Mx and in turn backing up to a cloud service (I just use Google drive) but you could use your NAS. Cumulus isn't very onerous and just runs happily, the pc is barely ticking over. I have a ups for the pc and a separate small one for the gateway

Edit-my WiFi router is on the same ups as the pc

It all worked straight out of the box, all I needed was to configure my pc with a local account and to autostart cumulus after the inevitable Windows update 

I think cumulus runs on Linux too so there are other options

Other solutions use raspberry pi for ultimate low power consumption. I already had a spare pc so didnt need to take that route

The key is the gateway unit which links your fine offset  (ambient) sensors to your logging computer

Offline whitling2k

  • Member
  • *
  • Posts: 9
Re: Suggestions with local storage / USB
« Reply #4 on: July 05, 2021, 10:05:06 AM »
Sorry for the late reply! 

That sounds interesting.  My 5-in-1 is back with the shop for them to 'test'.  But I'm not holding out much hope!

I have just bought a raspberry pi, I've wanted one for a while, but your post and the WeeWx recommendation both caught my attention.  I'm completely new to Linux - well, I've tried it in the past but never got very far.  I am happy with the hardware, OS installs, am OK with scripting in python etc - but it's the reliance on command line that throws me.  I grew up with DOS and Windows - Linux terminal feels so alien.

So I'm playing about with the Pi now and plan to see which system works for me.  A friend tells me that that WeeWX isn't very good at showing year-on-year and month-on-month comparative trends.  Or exporting (for example) monthly csv so I can automate the production of my own comparison charts and tables.

How does Cumulus work in that sense?  Is it easy to visualise - for example - a monthly comparison of max, min & mean temperatures over the 10+ years worth of data I have saved?  Or allow me to automatically export monthly 15-minute CSV files I can throw through R or Excel to produce summary tables?

Online PaulMy

  • Forecaster
  • *****
  • Posts: 5509
    • KomokaWeather
Re: Suggestions with local storage / USB
« Reply #5 on: July 05, 2021, 11:43:10 AM »
CumulusMX works with Ecowitt GW1000.  I am not aware CumulusMX works with that 5-in-1.

Quote
How does Cumulus work in that sense?  Is it easy to visualise - for example - a monthly comparison of max, min & mean temperatures over the 10+ years worth of data I have saved?  Or allow me to automatically export monthly 15-minute CSV files I can throw through R or Excel to produce summary tables?
The default CumulusMX local dashboard looks much like this website template http://komokaweather.com/cumulusmx/index.php
However the Annual Data tab is an extra in the website template http://komokaweather.com/cumulusmx/betel_datasummary.php?lang=en  and not in the local dashboard.  Annual Data uses the daily data from dayile.txt http://komokaweather.com/cumulusmx/dayfile.txt

Monthly logs are in the form komokaweather.com/cumulusmx/monthLog/Jul21log.txt and can be easily copied, FTP, or opened.

Enjoy,
Paul

Offline box

  • Forecaster
  • *****
  • Posts: 384
Re: Suggestions with local storage / USB
« Reply #6 on: July 05, 2021, 11:43:44 AM »
Sorry for the late reply! 

That sounds interesting.  My 5-in-1 is back with the shop for them to 'test'.  But I'm not holding out much hope!

I have just bought a raspberry pi, I've wanted one for a while, but your post and the WeeWx recommendation both caught my attention.  I'm completely new to Linux - well, I've tried it in the past but never got very far.  I am happy with the hardware, OS installs, am OK with scripting in python etc - but it's the reliance on command line that throws me.  I grew up with DOS and Windows - Linux terminal feels so alien.

So I'm playing about with the Pi now and plan to see which system works for me.  A friend tells me that that WeeWX isn't very good at showing year-on-year and month-on-month comparative trends.  Or exporting (for example) monthly csv so I can automate the production of my own comparison charts and tables.

How does Cumulus work in that sense?  Is it easy to visualise - for example - a monthly comparison of max, min & mean temperatures over the 10+ years worth of data I have saved?  Or allow me to automatically export monthly 15-minute CSV files I can throw through R or Excel to produce summary tables?
I am not familar with Wee Wx but the developer is often on these forums and there is a lot of expertise out there, no doubt he/they will comment

Cumulus MX generates a lot of data analysis as well as logging the raw data and has some pre-formatted reports, a summary here

https://cumuluswiki.org/a/Reports_folder

Outputs to csv are possible for other analysis

Again the developer and some key players frequent these boards and will no doubt comment

The cumulus website is https://cumulus.hosiene.co.uk/index.php and there is a pretty good wiki at https://cumuluswiki.org/a/Category:Cumulus_MX

Ooops PaulMy beat me to the post!



Offline whitling2k

  • Member
  • *
  • Posts: 9
Re: Suggestions with local storage / USB
« Reply #7 on: July 05, 2021, 11:59:14 AM »
I do need to start experimenting - my WH1080 has had a bit of a stay of execution, but I fear that the next time it wobbles it's game over.  Hopefully I can have a play and see how I get on with these two

Offline box

  • Forecaster
  • *****
  • Posts: 384
Re: Suggestions with local storage / USB
« Reply #8 on: July 05, 2021, 12:48:55 PM »
I do need to start experimenting - my WH1080 has had a bit of a stay of execution, but I fear that the next time it wobbles it's game over.  Hopefully I can have a play and see how I get on with these two

https://cumuluswiki.org/a/Supported_Devices

WH1080 appears on this but via USB interface rather than radio

i am not sure if WH1080 is compatible with GW1000 as its an earlier generation of technology

Offline whitling2k

  • Member
  • *
  • Posts: 9
Re: Suggestions with local storage / USB
« Reply #9 on: July 08, 2021, 03:35:29 PM »
So happy days, I got my refund for the dodgy Bresser.  Which means I’m in the market for a new model!

So disregarding the local storage, instead using some form of software on my raspberry pi - what all-in-1,  out the box setups for £150-£200 would be best suited to collecting and archiving the data on cumulus or weewx via raspberry pi/USB?

I don’t think I want to be playing with separate sensors, gateways, etc just yet.

I am thinking a Vantage Vue might be just a bit above my budget - does anyone know if I can plug the Pi directly into the console without a USB logger or WiFi thing?

Offline box

  • Forecaster
  • *****
  • Posts: 384
Re: Suggestions with local storage / USB
« Reply #10 on: July 08, 2021, 04:15:29 PM »


I don’t think I want to be playing with separate sensors, gateways, etc just yet.


You may think that a single sensor arrangement with USB is simpler but actually its very inflexible and your sensors will be positioned in a compromise location and you will need to transfer the data somehow

I started from scratch with separate anemometer, rain collector and thermometer. Joined them altogether with a gateway unit (really easy) and then connected it all to the internet. The whole process only took 30 minutes tops. The longest bits were putting up the anemometer on its pole and screwing stuff together, the networking was easy!

Now I have a rain collector in the right place and at the right height, an anemometer at the correct location/height and a well positioned thermometer. They are all in different parts of our garden and linked together by the gateway

Cumulus MX just needed to be installed on my PC and it found the gateway unit by itself - logging sorted in less than  5 minutes!

my system (GW1002) is available in the UK for £199 from weatherspares, but if you can wait you can get it from China for less

There are other options but wireless is definitely the way ahead


Offline galfert

  • Global Moderator
  • Forecaster
  • *****
  • Posts: 6822
Re: Suggestions with local storage / USB
« Reply #11 on: July 08, 2021, 06:32:20 PM »
Your budget and requirements are basically the same as what was recently discussed in this other thread. Granted this other thread discusses Alexa integration which is extra. You can just ignore that extra bit and only consider what stations are available to you in the UK.

https://www.wxforum.net/index.php?topic=42265.0

I will add that perhaps you should consider either the GW1001 or GW1002 kit as that doesn't have a display and is therefore less money. But in terms of getting data to your Pi software it will work just as well with the GW1000 device.

... does anyone know if I can plug the Pi directly into the console without a USB logger or WiFi thing?

With respect to FO clones with gateway device (GW1000) or new display consoles it is not possible to connect via USB. This is not a hindrance. All connectivity happens over the local network. Connecting over USB is considered antiquated. Take for example printers. It used to be that personal printers all connected via USB. It used to be that only more expensive office printers had cabled or wireless network connectivity. But network connectivity is now ubiquitous in the lowest of printers. The same is true for weather stations, and just about anything electronic.
« Last Edit: July 08, 2021, 06:43:17 PM by galfert »
Ecowitt GW1000 | Meteobridge on Raspberry Pi
WU: KFLWINTE111  |  PWSweather: KFLWINTE111
CWOP: FW3708  |  AWEKAS: 14814
Windy: pws-f075acbe
Weather Underground Issue Tracking
Tele-Pole

Offline whitling2k

  • Member
  • *
  • Posts: 9
Re: Suggestions with local storage / USB
« Reply #12 on: July 09, 2021, 07:22:26 AM »
OK - so the Ecowitt looks good.  I accept your points about using wired/wireless networking - I think I was burnt by the Bresser, and wanted to take control back.  I would be happy interfacing over a network as long as I can see the data on my own LAN devices - without having to watch it go up to the internet, partially disappear, then mostly come back down to my PC.

Can I check I'm getting this correctly - some of them are stupid, and I think I know the answer - but a non technical yes or no would help me filter out all the uber-techy stuff out there:
the GW1000 is the bridge that links the sensors to the network
The GW1002 is the GW1000 bridge + some sensors
The GW1003 is the GW1002 package, but a slightly (better? Different?) anemometer.  Is the ultra sonic an improvement?

The GW1002/3 both look good.  However, I do like the idea of display unit - just for real-time data - keeps the kids interested in what the weather is doing - are there any available for this?

If - I can get CumulusMX running on my Raspberry Pi, the GW1002/3 will work quite flawlessly with it out the box with minimum config..? i.e. without needing to be a Linux Commandline Pro.

Can I upload to Weathercloud *AND* use cumulus?  Or, is it one *OR* the other?

Anyone have any experience of running the cumulus' Rapsberry Pi Lite image, with a desktop installed?

Cumulus, will allow me to connect to a webpage on my LAN, which will display the current conditions, some trends etc without digging into complex configs?

If I buy a Raspberry Pi touch-screen display, it is possible to have a web browser displaying the cumulus page, on the device hosting it, acting as a console?

Does Cumulus keep a long-term record of all the observations stored?  Can I change the recording interval?  Where does it keep the data?  Can I export it to text/CSV for archival purposes?  Can I do this automatically?

Sorry if the above are stupid questions, I'm trying to get my head around a totally new approach that I don't fully understand - and can't afford to throw cash away on part-understood assumptions

Whitling2k

Offline galfert

  • Global Moderator
  • Forecaster
  • *****
  • Posts: 6822
Re: Suggestions with local storage / USB
« Reply #13 on: July 09, 2021, 07:48:10 AM »
OK - so the Ecowitt looks good.  I accept your points about using wired/wireless networking - I think I was burnt by the Bresser, and wanted to take control back.  I would be happy interfacing over a network as long as I can see the data on my own LAN devices - without having to watch it go up to the internet, partially disappear, then mostly come back down to my PC.

Yes, the data is accessible on the local LAN without it needing to be uploaded to the Internet.

Quote

Can I check I'm getting this correctly - some of them are stupid, and I think I know the answer - but a non technical yes or no would help me filter out all the uber-techy stuff out there:
the GW1000 is the bridge that links the sensors to the network

Yes

Quote

The GW1002 is the GW1000 bridge + some sensors

Yes

Quote

The GW1003 is the GW1002 package, but a slightly (better? Different?) anemometer.  Is the ultra sonic an improvement?

Different, not better. Some prefer one, some the other. There are pros and cons to each anemometer type.

Quote

The GW1002/3 both look good.  However, I do like the idea of display unit - just for real-time data - keeps the kids interested in what the weather is doing - are there any available for this?

You can get a GW100X kit and later add the HP2551-C display console. Or you can do it the other way around and just start off with the HP2551 based station and then add the GW1000. You'll need the GW1000 for Cumulus MX.

Quote

If - I can get CumulusMX running on my Raspberry Pi, the GW1002/3 will work quite flawlessly with it out the box with minimum config..? i.e. without needing to be a Linux Commandline Pro.

I think it will be pretty straightforward. There is plenty of good documentation and help on this forum and also in the official Cumulus forum. We won't let you fail. This is  a strong community.

Quote

Can I upload to Weathercloud *AND* use cumulus?  Or, is it one *OR* the other?

Yes, you can upload simultaneously to many online services without Cumulus MX. With Cumulus MX you'll just have even more that you can do simultaneously.

Quote

Anyone have any experience of running the cumulus' Rapsberry Pi Lite image, with a desktop installed?

Yes, many many many users.

Quote

Cumulus, will allow me to connect to a webpage on my LAN, which will display the current conditions, some trends etc without digging into complex configs?

Yes. The configs are web-based for most basic usage.

Quote

If I buy a Raspberry Pi touch-screen display, it is possible to have a web browser displaying the cumulus page, on the device hosting it, acting as a console?

Yes

Quote

Does Cumulus keep a long-term record of all the observations stored?  Can I change the recording interval?  Where does it keep the data?  Can I export it to text/CSV for archival purposes?  Can I do this automatically?

Yes, yes. Data. Is kept in local database on the Pi. Yes it can be exported to various formats including CSV. It will do so automatically also.

Quote

Sorry if the above are stupid questions, I'm trying to get my head around a totally new approach that I don't fully understand - and can't afford to throw cash away on part-understood assumptions

Whitling2k
Sounds good. You've done good research.
Ecowitt GW1000 | Meteobridge on Raspberry Pi
WU: KFLWINTE111  |  PWSweather: KFLWINTE111
CWOP: FW3708  |  AWEKAS: 14814
Windy: pws-f075acbe
Weather Underground Issue Tracking
Tele-Pole

Offline box

  • Forecaster
  • *****
  • Posts: 384
Re: Suggestions with local storage / USB
« Reply #14 on: July 09, 2021, 08:38:23 AM »
Just to add to galfert's comprehensive response there are some other display options

The WS View app (free) gives you one option, but of course it's designed for phones

There is also an android app that you could use on a spare tablet, there are descriptions on this forum

Finally if you do have a spare pc or laptop you can use the ecowitt web page or Cumulus Mx's own dashboard display

Offline whitling2k

  • Member
  • *
  • Posts: 9
Re: Suggestions with local storage / USB
« Reply #15 on: July 09, 2021, 08:40:34 AM »
Wow, thank you! Great reply!

Two quick follow-ons:
- any links to the HP2551-C on its own? I can only find it as a package with the full station. 
- I assume the config of Cumulus, and stop/start etc can be done locally via the Pi desktop’s terminal when installed, rather than still needing SSH?

I think the gw1002 looks best for me - just need to buy it and learn Linux now!

I’m sure I’ll be back as I get going with it :-)

Offline whitling2k

  • Member
  • *
  • Posts: 9
Re: Suggestions with local storage / USB
« Reply #16 on: July 09, 2021, 08:47:28 AM »
Just to add to galfert's comprehensive response there are some other display options….

Nice one, thanks for that.  An Uber-cheap android tablet might be cheaper and easier than the pi display actually.  Thinking about it, I have a free Amazon tablet on its way from a promotion - this will be my HUD for now!

Longer term, I want to replace the Chromecast backdrop of my TV with the Pi - randomised 4K slideshow from my NAS photo archive, combined with some weather obs.  Some sort of html page with some python grabbing random images seems my best bet for that.


Offline galfert

  • Global Moderator
  • Forecaster
  • *****
  • Posts: 6822
Re: Suggestions with local storage / USB
« Reply #17 on: July 09, 2021, 10:16:18 AM »
- any links to the HP2551-C on its own? I can only find it as a package with the full station. 
I think you can only get the HP2551-C separately from Ecowitt for UK residents.
https://www.ecowitt.com/shop/goodsDetail/79

I'm not sure if Froggit in Germany has resolved their shipping to UK after Brexit.
https://www.froggit.de/product_info.php?info=p443_expansion--replacement--display-unit-hp1000se-pro-incl--indoor-wireless-sensor--temperture-humidity-pressure-.html

Quote
- I assume the config of Cumulus, and stop/start etc can be done locally via the Pi desktop’s terminal when installed, rather than still needing SSH?
Yes it can all be managed from the Pi itself while in the desktop UI but with use of the Linux terminal prompt to start/stop. I would recommend setting Cumulus MX up to run as a service.

Quote
I think the gw1002 looks best for me - just need to buy it and learn Linux now!

I’m sure I’ll be back as I get going with it :-)
I recommend for UK users to go with 868 MHz.

« Last Edit: July 09, 2021, 10:17:49 AM by galfert »
Ecowitt GW1000 | Meteobridge on Raspberry Pi
WU: KFLWINTE111  |  PWSweather: KFLWINTE111
CWOP: FW3708  |  AWEKAS: 14814
Windy: pws-f075acbe
Weather Underground Issue Tracking
Tele-Pole

Offline whitling2k

  • Member
  • *
  • Posts: 9
Re: Suggestions with local storage / USB
« Reply #18 on: September 21, 2021, 06:20:16 AM »
Sorry for resurrecting a old thread - but success. I managed to eek a few months extra life out of my almost-dead WH1080 and learn more about how CumulusMX works by running it through a Pi.  Long story short, complete success.  Been logging away and using MX to share data to WeatherCloud.  Then my WH1080 died altogether :-(.  Learnt a lot about Pi and PiOS/Linux. 

And now my ECOWITT GW1003 arrived :-)

I'm awaiting a few parts to to finalise the physical install, my anemometer is currently 90 degrees out until I can source some tubing, but everything is linked up nicely.

The only things I need to do now are:
  • Work out how to reincorporate my historic data to MX, which I'm progressing nicely with
  • Work out how to use different MX website templates (and the annual data pages as shown here:http://komokaweather.com/cumulusmx/betel_datasummary.php?lang=en) 
  • And work out how to display the readings on a display - partly to keep the kids engaged, but also I don't want to pick up a phone/web browser every time I want to see the weather conditions

I have been tying myself in a bit of a knot with that last one - LCDs for the Pi look a minefield.  I don't want anything fancy, I don't want touch-screen but I want to backlight to be capable of being completely off when not in use.  And this seems to throw most Pi LCD options out the window.

I'm just not sure about the HP2551-C, I can't quite fathom out how it works.  Does it display WU data that's been uploaded, or live data from the sensors. Would it work with or against a GW1000 on the same network?  Could it replace the GW1000? 

Thanks for all your help - I would share photos, but the GW1000 isn't exactly photogenic, and the sensors are scattered around the garden, in their optimal locations!

Offline Mandrake

  • Forecaster
  • *****
  • Posts: 1102
Re: Suggestions with local storage / USB
« Reply #19 on: October 11, 2021, 10:43:57 AM »
The HP2551 console is just a receiver like the GW1000 but the difference is that it can display the data directly.
They can work together completely happy

The GW1000 is perhaps slightly more capable but in general the HP2551 can do nearly everything that the GW1000 can other than be remotely interrogated for sensor status/data
They can both send data externally to WU for instance
The HP2551 stores data locally and on a SDcard
The GW1000 stores no data but is network addressable for data
G1ZFO

Ecowitt HP2551A + WH65 Tri-Wing (Wunderground: IGUILDFO67)
Ecowitt GW1000 (Wunderground: IGUILDFO68)
Ecowitt GW1000 (Mk2) test environment driving CumulusMX on a RPi 3b
Ecowitt GW2000 (Test)
Ecowitt WS90 Wittboy - Test
Ecowitt WH51 (x6) Soil Moisture Sensor
Ecowitt WH41 PM2.5 AQM Sensor
Ecowitt WH31 (x8) Thermo/Hygro Sensor
Ecowitt WS80 Ultrasonic Anemometer (pre-prod test)
Ecowitt WH57 Lightning Sensor -test
Ecowitt WH32-EP (SHT35) + Davis 7714 Screen
Ecowitt WH45 CO2/PM2.5/PM10 -Test
Ecowitt WN34 Soil Temp Sensor -Test
Ecowitt WN34 Water Temp Sensor -Test
Ecowitt WN35 Leaf Moisture