Topic: GW2000 - temporarily store data locally

[wxfan]

Hi All,

I am sorry if this has been asked before, I have tried to search and have seen some messages but I am not clear on how to proceed.

From my understanding the data logged by the GW2000 need to be uploaded to Ecowitt server otherwise are lost. But it seems you can use an API to save the data locally.

I would like to log data 24/7 but would like to turn off completely my modem at certain times, like when I am not at home or at night. Would it be possible to log the data locally on something like a USB stick and then upload the data automatically to Ecowitt once the GW is connected again to the internet? Or will I have to run something like a raspberry pi arduino type of thing?

[Gyvate]

I suggest you read our MUST READ https://www.wxforum.net/index.php?topic=40730.0
chapter 8:  data flow between sensors, consoles, application software and internet weather services

Thre you will find plenty of examples for data logger software.
A few examples:
Simplest but not to be underestimated would be CumulusMX. https://cumuluswiki.org/a/Software  (Linux, WIndows/MacOS)
Weewx is the allrounder but also highly complex. http://www.weewx.com/docs (Linux)
Meteobridge on a RaspberryPi could be an option. https://meteobridge.com/wiki/index.php/Home (openWRT, comes with own OS)

[olicat]

Hi!

would like to turn off completely my modem

What exactly do you mean by modem in this context?
A Raspi would certainly be able to query the GW2000 data only in the local network without the Internet. You can therefore record this data locally 24/7 - assuming a network connection to the GW2000 via WIFI or LAN. You would then have the data available locally in any form (depending on the software used).

But:
Ecowitt offers no way to send data to Ecowitt.net retrospectively. This means that all data that cannot be sent to Ecowitt.net in real time is irretrievably lost with this service.

Oliver

[wxfan]

I suggest you read our MUST READ https://www.wxforum.net/index.php?topic=40730.0
chapter 8:  data flow between sensors, consoles, application software and internet weather services

Thre you will find plenty of examples for data logger software.
A few examples:
Simplest but not to be underestimated would be CumulusMX. https://cumuluswiki.org/a/Software  (Linux, WIndows/MacOS)
Weewx is the allrounder but also highly complex. http://www.weewx.com/docs (Linux)
Meteobridge on a RaspberryPi could be an option. https://meteobridge.com/wiki/index.php/Home (openWRT, comes with own OS)

I read the CH 8 but it's a bit overwhelming, even setting up CumulusMX on my PC is not so straightforward as in the Wizard there are many parameters to set.

I appreciate the information in CH8 but the figure with all those arrows is not so easy to understand.

[wxfan]

Hi!

would like to turn off completely my modem

What exactly do you mean by modem in this context?
A Raspi would certainly be able to query the GW2000 data only in the local network without the Internet. You can therefore record this data locally 24/7 - assuming a network connection to the GW2000 via WIFI or LAN. You would then have the data available locally in any form (depending on the software used).

But:
Ecowitt offers no way to send data to Ecowitt.net retrospectively. This means that all data that cannot be sent to Ecowitt.net in real time is irretrievably lost with this service.

Oliver

My GW2000 is connected via ethernet to my modem/router. At the moment I turn off the wifi but I need to leave the modem on to upload the data to the cloud.

It's a shame that we cannot send the data to Ecowitt retroactively, I don't want my data to be scattered around, some in the cloud and some locally. It would seem trivial to integrate local data in the cloud. I don't understand why they don't allow to do this.

Do any of the other cloud services allow for data to be uploaded retroactively? I imagine some of them are paid services?

[Gyvate]

I suggest you read our MUST READ https://www.wxforum.net/index.php?topic=40730.0
chapter 8:  data flow between sensors, consoles, application software and internet weather services

Thre you will find plenty of examples for data logger software.
A few examples:
Simplest but not to be underestimated would be CumulusMX. https://cumuluswiki.org/a/Software  (Linux, WIndows/MacOS)
Weewx is the allrounder but also highly complex. http://www.weewx.com/docs (Linux)
Meteobridge on a RaspberryPi could be an option. https://meteobridge.com/wiki/index.php/Home (openWRT, comes with own OS)

Many people have both, the Ecowitt cloud and local - don't forget, your 5-minute based data records will be kept at ecowitt.net only for 90 days - later on it will be 30 minute or even 4 hours records (--> MUST READ).

With your data locally you can have everything in your selected resolution and can be independent of the internet.
When you post regularly to ecowitt.net, CMX can even backfill its data from ecowitt.net if it happened to crash for some reason. You can also make CMX start with downloading your Ecowitt.net data first and then continue running on its own - a feature so far others don't offer (afaik).

I read the CH 8 but it's a bit overwhelming, even setting up CumulusMX on my PC is not so straightforward as in the Wizard there are many parameters to set.

guess what others like weewx would still be - CumulusMX so far almost everybody managed (after all you have to know what you want  ) - but almost fully out of the box would be rather Meteobridge.

But for CMX, just enter your station name, longitude, latitude and choose local Ecowitt API along with the IP address of your GW2000 and you can already go. Then look for your logging times, set a rain amount start and that's +/- it for a simple station.
When you want to do more things than seeing you realtime data, min/max of day, month, year etc., then you will have to configure a bit more.
But this requested Wizard info you will need to enter into every data logger program - the program needs to know certain things it cannot know by itself.

EDIT: some text jumped into the quote ....


Many people have both, the Ecowitt cloud and local - don't forget, your 5-minute based data records will be kept at ecowitt.net only for 90 days - later on it will be 30 minute or even 4 hours records (--> MUST READ).

With your data locally you can have everything in your selected resolution and can be independent of the internet.
When you post regularly to ecowitt.net, CMX can even backfill its data from ecowitt.net if it happened to crash for some reason. You can also make CMX start with downloading your Ecowitt.net data first and then continue running on its own - a feature so far others don't offer (afaik).

[olicat]

Hi!

Do any of the other cloud services allow for data to be uploaded retroactively? I imagine some of them are paid services?

Unfortunately - despite intensive research - I am not aware of any service that enables the automatic subsequent upload of data.
With Awekas, this works at least semi-automatically (you have to upload an Excel file and import it with a few clicks.

I also don't understand why nobody offers this option. But if not even the software authors provide this option for their own programmes, it is obviously not clear to the cloud providers that this is desired by the users.

So:
Write to Ecowitt and ask for this feature!

BTW:
The only system I know of that currently supports subsequent sending of data automatically and without any restrictions is InfluxDB.
FOSHKplugin temporarily stores data if InfluxDB is unavailable (whether for reasons of server maintenance or network unavailability) and automatically resends the data once the connection is restored.
This works wonderfully - even on an InfluxDB server installed on my notebook, which is only switched on from time to time.
The data on the server is always complete (after some time). Even if there is a weekend in between.

And:
The problem with all providers and software solutions known to me is that the time supplied with the data is simply ignored and the server time at the time of submission is used instead.
So it could easily be changed ...

So:
Write to Ecowitt and ask for this feature!


Oliver

[Rover1822]

From the service providers standpoint, I can see why they don't allow the backfill, imagine supporting that, with the number of people that upload and varying degrees of technical knowledge and the amount of user errors that could occur, and ensuing black eye the service gets.

Now if it was a pay system, I might see it, or the gateways themselves had the upload capability and locked down to backfill to reduce errors, but few of the consoles store data to start with.

[mcrossley]

Do any of the other cloud services allow for data to be uploaded retroactively? I imagine some of them are paid services?

Unfortunately - despite intensive research - I am not aware of any service that enables the automatic subsequent upload of data.

Davis does, their WLL device will buffer up to several months of data locally waiting for the link to come back - as will the old WeatherLink PC software, albeit limited to the VP2 logger capacity.

OpenWeatherMap supports backfill.

[wxfan]

Many people have both, the Ecowitt cloud and local - don't forget, your 5-minute based data records will be kept at ecowitt.net only for 90 days - later on it will be 30 minute or even 4 hours records (--> MUST READ).

With your data locally you can have everything in your selected resolution and can be independent of the internet.
When you post regularly to ecowitt.net, CMX can even backfill its data from ecowitt.net if it happened to crash for some reason. You can also make CMX start with downloading your Ecowitt.net data first and then continue running on its own - a feature so far others don't offer (afaik).

[/quote]

Thanks for the info! I have it running now, for the moment it looks like a dashboard from the 90s! Is there a guide somewhere, there are still a few things I don't understand. What are Upload and Local Copy mean? And Interval Settings and Real time Settings ? These are different from logging intervals?

I didn't really see the options to download Ecowitt data though.


I didn't realize Ecowitt would throw away some data but it makes sense. I will definitely log locally. So I can open MX whenever I want and it will get the data from Ecowitt for when it was not running? That's great as I will not keep my PC always on.

[wxfan]

From the service providers standpoint, I can see why they don't allow the backfill, imagine supporting that, with the number of people that upload and varying degrees of technical knowledge and the amount of user errors that could occur, and ensuing black eye the service gets.

Now if it was a pay system, I might see it, or the gateways themselves had the upload capability and locked down to backfill to reduce errors, but few of the consoles store data to start with.

We are not talking about terabytes of data, weather data for a single users is probably a few mb. I don't see any big technical challenge, do you realize that most data servers deal with huge amounts of data?!

such simple data can also be easily compressed. Climate data files containing 1tb can be compressed to 20% of their original size

[wxfan]

Do any of the other cloud services allow for data to be uploaded retroactively? I imagine some of them are paid services?

Unfortunately - despite intensive research - I am not aware of any service that enables the automatic subsequent upload of data.

Davis does, their WLL device will buffer up to several months of data locally waiting for the link to come back - as will the old WeatherLink PC software, albeit limited to the VP2 logger capacity.

OpenWeatherMap supports backfill.

Great to know, can I use OpenWeatherMap with my Ecowitt station for storing data? So much to learn!

[Rover1822]

From the service providers standpoint, I can see why they don't allow the backfill, imagine supporting that, with the number of people that upload and varying degrees of technical knowledge and the amount of user errors that could occur, and ensuing black eye the service gets.

Now if it was a pay system, I might see it, or the gateways themselves had the upload capability and locked down to backfill to reduce errors, but few of the consoles store data to start with.

We are not talking about terabytes of data, weather data for a single users is probably a few mb. I don't see any big technical challenge, do you realize that most data servers deal with huge amounts of data?!

such simple data can also be easily compressed. Climate data files containing 1tb can be compressed to 20% of their original size

It is not so much the amount, but the format, I mean this for situations where users store their own data by whatever method, and try to upload it. If it was directly from the device without user intervention on format , etc. then yeah. Problem is , most Ecowitt consoles, do not store data, so if you implemented a method to store it on your own, it would have to be perfect, to be accepted by the web weather services.

Is it something I would like to see, of course, I was just pointing out the difficulties on the owners of these web services to implement this.

Edit: I only mention this as I work as programmer on very large web sites with terabytes of data behind them . I agree it all sounds simple, but in reality...different story. And sadly, if the user doesn't do stuff right, they usually blame the service, or request support. Note , I have no affiliation with EcoWitt or anything related LOL, just offering a perspective

[Gyvate]

Many people have both, the Ecowitt cloud and local - don't forget, your 5-minute based data records will be kept at ecowitt.net only for 90 days - later on it will be 30 minute or even 4 hours records (--> MUST READ).

With your data locally you can have everything in your selected resolution and can be independent of the internet.
When you post regularly to ecowitt.net, CMX can even backfill its data from ecowitt.net if it happened to crash for some reason. You can also make CMX start with downloading your Ecowitt.net data first and then continue running on its own - a feature so far others don't offer (afaik).

Thanks for the info! I have it running now, for the moment it looks like a dashboard from the 90s! Is there a guide somewhere, there are still a few things I don't understand. What are Upload and Local Copy mean? And Interval Settings and Real time Settings ? These are different from logging intervals?

I didn't really see the options to download Ecowitt data though.


I didn't realize Ecowitt would throw away some data but it makes sense. I will definitely log locally. So I can open MX whenever I want and it will get the data from Ecowitt for when it was not running? That's great as I will not keep my PC always on.

there is a CMX WiKi available at https://cumuluswiki.org/a/Software
sometimes a bit challenging to read (my personal opinion), but there's also a forum where you can ask questions if the WiKi doesn't get you any further. https://cumulus.hosiene.co.uk/viewforum.php?f=40
upload means sending data (via FTP or PHP) to a website and copy do this locally (in your local network or on the same computer) if you run a website (your own or the CMX default website) locally e.g. on a RaspberryPi or on a NAS. Then copying the realtime data updates makes more sense than using services like (S)FTP

Logging is what CMX reads from your station and saves every logging interval (default setting one minute - I think).

Upload/copying means sending data to a remote site (e.g. your custom website, the CMX default website hosted at your local or remote server)

The backfill options are there - but first you need to create an APP key and an API key in your ecowitt.net account (--> profile).

In CMX you have to go to Settings --> Station Settings --> Ecowitt Cloud API access.

There are a few more things you need to know when you want to first download the last 90 days from ecowitt.net (that's a one time story unless you shut don CMX for 90 days). In the CumulusMX/data (or CumulusMX\data under Windows) directory is a file named today.ini

CMX updates this file every time it writes a log entry into its database (MMMYYlog.txt - e.g. Feb24log.txt) with a time stamp date/hour/minute.
The entry at the top of this file (in fact 2 entries) have to be set to now - 90 days e.g. 01.12.2023 00:00:00 - you may have slashes instead of points depending on your locale).

Shut CMX down.
Rename your existing Feb2024log.txt file (CMX should not find it under its name anymore).
Change the time stamp in today.ini as mentioned above.
Save the changes.
Use Notepad under Windows or nano under Linux as a text editor.
Make sure you APP and API keys are properly entered.
Then start CMX again.

It will now backfill the data from the Ecowitt cloud until it reaches the current time - and then continue logging normally.
When you e.g. shut down CMX in the evening and restart it again next morning, the same procedure will happen at startup from when CMX last wrote data to the logging file.

[kheller2]

I'll chime in here and state that it would be nice if the unit buffered data to be uploaded for a certain amount of time, say 24 hours -- to deal with network issues etc..  But I agree that allowing end users to upload legacy data is in practice horrible. It is an extremely complex thing to do (that is why I suggested the buffer above).

I also work at a company with large amounts of data, customer facing etc.  And the amount of times that data is uploaded incorrect, wrong format, duplicate, "oh I want to change this one bit of data".. "oh I uploaded a ton of wrong stuff, please use this instead.."  or the guy in the basement that wants to upload 10 years worth of data.  You can only go so far in providing guide rails, and the simplest solution is to not allow it at all.

Also, the data that is uploaded is also rolled, averaged, etc.  You can't go back 2 years and dig down to a specific time sample.  So, now with manual uploads you have to process all the data in one shot.  For samples taken every 3 seconds, you have 28,800 samples to process just for one day.

Sure, it isn't THAT much and not many people would do it.  But I can see some very evil ways to make things fall over.

[Wooks61]

I have read some posts from wxfan before and I believe he wants to switch off his PC and router when he is not at home and at night..
1st post...
"I would like to log data 24/7 but would like to turn off completely my modem at certain times, like when I am not at home or at night."
I'm not sure what you mean by "modem". Is your modem separate on integral to your router?

I know that you have the GW2000 connected via LAN cable.
But if connected by LAN cabled to the router, then switching it off will mean that the data from your GW200A will be going nowhere-


Later...
"So I can open MX whenever I want and it will get the data from Ecowitt for when it was not running? That's great as I will not keep my PC always on."

When your router is off, no data will get to Ecowitt. So running CMX later won't retrieve data that was not sent when the PC or router was switched off.

To log the data locally, you would have to have whatever is running CMX to be on and connected to the network 24/7.

My apologies if I've misunderstood ...

[Gyvate]

he can't have both ... the famous cake - you either have it or eat it

he has two scenarios which could fulfill his wishes:

- leave his router on and WiFi on (and the GW2000 connected via WLAN) - then data will be posted to ecowitt.net and CMX can backfill them when he restarts his PC (and CMX) again after being shutdown (or in sleeping mode) over night - may be too much electro-smog

- switch the WLAN portion ("WiFi") of his router (in case it's a combined router) or the WLAN access point off over night but have the GW2000 connected to the router via LAN - no electro smog - then again it posts over night and CMX can backfill once restarted

the WLAN portion of a combined LAN/WLAN router can usually be switched off while the routing capability for LAN connected devices remains active

router or modem off (not everyone has these often cheap combined devices - router plus internet interface): no chance

[Wooks61]

@Gyvate:
Precisely. I was trying to understand what exactly wxfan wants to turn off and how his devices are connected. For most people, the Router/Modem is one unit. It may be possible to switch off the WIFI only part.

For data to get to Ecowitt without gaps, the GW2000 must be connected to the internet. when any of the devices between GW2000 and Ecowitt are switched off no data can be stored at Ecowitt and the data is lost. The data (at Ecowitt) cannot be filled in afterwards.


GW2000 -> LAN cable -> Router -> Modem  -> Internet -> Ecowitt
Any part of the above off/disconnected: No Data at Ecowitt. Cannot be filled in later.

GW2000 -> LAN cable -> Router -> device running CMX(as data logger)
Data stored locally on device. No data at Ecowitt. if any of the above disconnected or switched off: No Data stored. Not transmitted to Ecowitt. On reconnection/ switching on CMX will recover data FROM Ecowitt only if it has been sent in the first place.

GW2000 -> LAN cable -> Router -> device running CMX(as data logger) ->CMX to Ecowitt (via Router/Modem)
Data stored locally on device running CMX and at Ecowitt.
If CMX stops it can retrieve data FROM Ecowitt on restarting. It will not however fill in the gaps at Ecowitt when the connection to Ecowitt is switched off/disconnected.

[Gyvate]

the only modification to the before post here is that the GW2000 posts directly to ecowitt.net and not via CumulusMX (CMX)
=> as long as the GW2000 and the router (GW2000 LAN connected or WLAN connected) are up and running, data will go to ecowitt.net
CMX retrieves its data via the local network API from the GW2000 i.e. independently.
When CMX is down, the GW2000 still posts to ecowitt.net and at restart CMX can backfill from ecowitt.net

[Wooks61]

Thanks Gyvate.
I was trying to clarify to wxfan that data lost by CMX (locally) can be retrieved from Ecowitt.net, but only if the the data has been sent in the first place.
i.e.  switching off the router/modem won't allow the GW2000 to post to Ecowitt.net. and there would be no data to retrieve.

Also the lost data at Ecowitt.net cannot be sent retrospectively even if it has been stored locally by CMX.

I think this clarifies his first question
"I would like to log data 24/7 but would like to turn off completely my modem at certain times, like when I am not at home or at night. Would it be possible to log the data locally on something like a USB stick and then upload the data automatically to Ecowitt once the GW is connected again to the internet?"
 

[wxfan]

It is not so much the amount, but the format, I mean this for situations where users store their own data by whatever method, and try to upload it. If it was directly from the device without user intervention on format , etc. then yeah. Problem is , most Ecowitt consoles, do not store data, so if you implemented a method to store it on your own, it would have to be perfect, to be accepted by the web weather services.

Is it something I would like to see, of course, I was just pointing out the difficulties on the owners of these web services to implement this.

Edit: I only mention this as I work as programmer on very large web sites with terabytes of data behind them . I agree it all sounds simple, but in reality...different story. And sadly, if the user doesn't do stuff right, they usually blame the service, or request support. Note , I have no affiliation with EcoWitt or anything related LOL, just offering a perspective

No problem I understand your point of view, and I agree, but we are just talking on a very generic level, I don't think the users should modify any of the data for this to work. I work as a climate scientists so I deal with huge amounts of climate data, there are specific data format that are international standards. But we also deal with multiple format and are able to merge everything, data merging has become a lot more fluid than in the past.

[wxfan]

Sorry guys I am catching up now with all messages. Thanks again for the discussion and advice.

My router/modem are one unit, I have a 4G modem. So far I have been turning off the 4G modem wifi at night, but the modem still generates 4G waves and also consumes electricity. It seems trivial but electric costs are going up and having a bunch of small devices makes a difference.

In any case there's no real solution to my problem. Either I leave the modem on or I will have to run CMX on a small independent device, but I understand that in this case I will have data gaps in my ecowitt data. I like the 24h buffer idea.

In any case, my solution to upload data retroactively would mean that data cannot be modified, they would be stored in the device and uploaded once internet connection is back up.