Author Topic: 1 min vs 5 min feed from Ambient Weather 1002 wifi  (Read 2100 times)

0 Members and 1 Guest are viewing this topic.

Offline ptsawyer

  • Member
  • *
  • Posts: 3
1 min vs 5 min feed from Ambient Weather 1002 wifi
« on: September 04, 2018, 01:10:31 PM »
Hello!

I am a new owner of a Ambient Weather 1002 wifi weatherstation.  I finally got it set up and running!

Right now I have it set to store a data point every minute.  However, when I log into AmbientWeather.net I only see five minute intervals.

Does the unit send the full data feed anywhere?  Or are the 1 minute intervals only stored within the device?  Can I retrieve the 1 minute data points from an API or some other source?

Offline galfert

  • Global Moderator
  • Forecaster
  • *****
  • Posts: 6822
Re: 1 min vs 5 min feed from Ambient Weather 1002 wifi
« Reply #1 on: September 04, 2018, 01:49:34 PM »
Ambient Weather accepts 1 minute uploads but it only keeps 5 minute data charts. The purpose of the 1 minute upload is to see your weather station somewhat live on ambientweather.net.

I have a different model weather station, but I save my data every 10 minutes to an SQL database. I may increase that to 5 minutes...I haven't decided yet. But to save a data set for every minute for data analysis is a bit much I think. Unless you have a specific reason or need to see that kind of resolution in your analysis.

Even Weather Underground which lets you upload every 5 seconds with their rapid fire system only keeps data charts for every 5 minutes. Again the same reason they allow 5 second uploads is so that you can see live data, and then they throw it out too. After that live moment is over then this level of detail is pretty much useless information (or more like too much information) and what then matters for historical purposes is probably good enough to have just data every 5 minutes.

If you ever decide to upload to PWS (WeatherForYou) you'll see that they let you upload every 1 minute, 5 minutes or 10 minutes...etc). In my testing since you never really see live data on their system the 1 minute resolution upload is too much data. Their charts are too small to see that level of detail and it all just gets to looking like a jumbled mess of scribble scrabble. I've played with the upload interval and I settled on 10 minutes as the best interval to see the best charts on their system.

For CWOP I send data every 5 minutes (fastest they allow). That matches MADIS that picks it up every 5 minutes so the most out of date my data is with MADIS is 9 min 59 seconds, but averages around 7.5 minutes.

I don't know about the internal data logging capabilities of your particular model. Maybe someone else can answer that question.

« Last Edit: June 12, 2020, 05:58:05 PM by galfert »
Ecowitt GW1000 | Meteobridge on Raspberry Pi
WU: KFLWINTE111  |  PWSweather: KFLWINTE111
CWOP: FW3708  |  AWEKAS: 14814
Windy: pws-f075acbe
Weather Underground Issue Tracking
Tele-Pole

Offline ptsawyer

  • Member
  • *
  • Posts: 3
Re: 1 min vs 5 min feed from Ambient Weather 1002 wifi
« Reply #2 on: September 04, 2018, 03:53:17 PM »
First, thank you very much for the very detailed reply.

I am a data and analytics professional.  So for me, more data is pretty much always better than less.  I can always filter out records, but I cannot add more that do not exist.

I am looking to build a large data set (hundreds of thousands of rows) that we can use to demo and test various data analysis, analytics, and BI platforms.

So just so I am sure I understand correctly:

1) Ambient Weather does not store the 1 minute feed, only every 5 minutes?  Same with Weather Underground?

2) Am I understanding correctly that WeatherForYou will allow me to upload every minute?

What I am really looking for is a free and easy solution to store my data.  I would like every minute, but I could also settle for every 5 minutes.  What I am trying to avoid is the process of setting up my own database, although I could certainly do that.

Internally, my particular model does allow for data loads every minute.  When I go in and look at the internal history on the screen unit that came with the station, you can see the minute by minute records.  What I am also trying to avoid is a manual export of the records via SD card.  Ideally I would like it as automated as possible.

Offline galfert

  • Global Moderator
  • Forecaster
  • *****
  • Posts: 6822
Re: 1 min vs 5 min feed from Ambient Weather 1002 wifi
« Reply #3 on: September 04, 2018, 04:53:54 PM »
Yes ambientweather.net only store a maximum of every 5 minutes. If you wan this data you have two ways of getting it.
- You can select a time period and then click the Download CSV icon. So its a manual process
- You can apply for an API key, requires approval. Then you can tie in your database program to pull the data.

PWS (WeatherForYou) does store data every 1 minutes. But I don't know of a way to get the data other than to see it on their website. I don't see a download button. I don't know if they have API support. I don't see any mention of any of this in their FAQ. You could reach out and ask them but I suspect if it is available they will charge you.

I think based on what you are looking to do I think your best bet is to gather the data before it goes out to the Internet. If you want free then set up your own server locally where your weather station is located. You can do it one of two ways and you can do both of these at the same time:
- Setup an SQL database. I use php and MariaDB (SQL compatible) running on my Synology NAS. I'm collecting the data but I haven't used it yet. I figured I'd collect it in case I have a need or want some day. (My Meteobridge is capturing and sending to SQL).
- Run a weather application and capture the data with it, and it will store the data in its own internal database format. I recommend looking at WeeWx or Weather Display.

With your weather station in order to capture the data locally you'll have to invest in some added hardware and software. There are several ways to do it. You can do WiFi redirection of Weather Underground traffic and capture it with WeeWx or Weather Display. You can also incorporate the use of Ambient's ObserverIP device and then capture the data using a Meteobridge. The Meteobridge can then send to SQL database. There are also what I call hybrid solutions where you use a combination of the above methods. The Weather Display program can also connect to the Meteobridge to get data.

Many ways to do it. Just for example: (these aren't the only solutions....just showing some data flow possibilities)
Wx Outdoor sensor --> console --> network redirection --> WeeWx or Weather Display --> SQL
Wx Outdoor sensor --> ObserverIP --> Meteobridge --> Weather Display --> SQL
Wx Outdoor sensor --> ObserverIP --> Meteobridge --> Weather Display (no SQL..just use Wx Software)
Wx Outdoor sensor --> ObserverIP --> Meteobridge --> SQL
Wx Outdoor sensor --> ObserverIP --> network redirection --> WeeWx or Weather Display --> SQL

* Having the ObserverIP still lets you use the console separately.
* Adding the Meteobridge is nice as you don't have to deal with Network Redirection techniques
* Meteobridge will also allow you to send the data to your own APRS server

I've got the ObserverIP and the Meteobridge. The Meteobridge is a really neat box that does a lot and is small.

What I mean by network redirection is that you need to send Weather Underground data to a local server device. This network redirection can be done with the ObserverIP running a downgraded firmware that allows this. Or you can sometimes change the console to do this depending on which console you have. Or you can employ the use of a clever router running custom firmware like WRT or Tomato ..etc to redirect that traffic to your server running weather software. This requires more technical skill, Linux and networking expertise.

I've also read a post here where someone combined WeeWx and network redirection into one box by using a Raspberry Pi and using a supported WiFi adapter to create an access point that only the weather station console connects to. Lots of possibilities.

I agree that the manual export via SD card would not be ideal. You would miss data during your pull out and copy. Not to mention that it is not automated.

« Last Edit: September 04, 2018, 05:13:31 PM by galfert »
Ecowitt GW1000 | Meteobridge on Raspberry Pi
WU: KFLWINTE111  |  PWSweather: KFLWINTE111
CWOP: FW3708  |  AWEKAS: 14814
Windy: pws-f075acbe
Weather Underground Issue Tracking
Tele-Pole

Offline ptsawyer

  • Member
  • *
  • Posts: 3
Re: 1 min vs 5 min feed from Ambient Weather 1002 wifi
« Reply #4 on: September 04, 2018, 05:11:06 PM »
Thanks again for another great reply.  Really appreciate the response.  A lot to think about here...

If I am willing to reduce my data resolution to every 5 minutes, could I simplify the process?  Then can I simply pull my history from Ambient Weather or WUnderground?  How far back do they store history?

Offline galfert

  • Global Moderator
  • Forecaster
  • *****
  • Posts: 6822
Re: 1 min vs 5 min feed from Ambient Weather 1002 wifi
« Reply #5 on: September 04, 2018, 05:24:00 PM »
Even if you are willing to reduce the frequency to every 5 minutes, you still are better off capturing the data on your network. But it kind of depends what you want to do with the data and how you will use it. And in what format you need it for whatever is going to use it.

I think you are looking for a simple solution to have some online service store the data and then you think you are going to go retrieve it from them. That is either going to cost you or it will be a manual and limited process.

I may be wrong and there may be this awesome service that gives you free storage of data and free API tie in support. If that exists I don't know about it. Maybe someone else can chime in.

All I know is that if I were you, I'd just collect it locally on your network, which is what most people do that want the data or want to do software analysis. That is free and reliable and doesn't even require your Internet to stay up and running. Even on a power outage you can run on battery backup or generator and not lose data. If I'm going through a hurricane I can bet that I'll lose Internet and power...but I have backup power and I'll continue to capture weather data locally to examine later (unless the hurricane takes my outdoor sensor for ride down the block). I'm collecting data now and I don't even have a use for it yet. I got the Meteobridge because it offered so many services to upload my data to in one simple solution. Since it could send to SQL, I did that too.


TO MODERATOR: I think this thread belongs in the Ambient Weather sub. ...not in the pictures sub.
« Last Edit: December 06, 2019, 01:57:51 PM by galfert »
Ecowitt GW1000 | Meteobridge on Raspberry Pi
WU: KFLWINTE111  |  PWSweather: KFLWINTE111
CWOP: FW3708  |  AWEKAS: 14814
Windy: pws-f075acbe
Weather Underground Issue Tracking
Tele-Pole

 

anything