Topic: Is there any other way to download data?

[kbellis]

 [ You are not allowed to view attachments ]
RE: Downloading data from AmbientWeather.net - Is there any other way to download data than what's outlined here?
https://ambientweather.net/help/how-do-i-download-data-ambientweather-net/

There are a number of issues related to the desktop PC web browser-based interface that makes downloading a desired .csv file cumbersome most of the time, or will give the browser fits and freezes.

Impotent Filtering
 [ You are not allowed to view attachments ]

Filtering selected data fields has zero impact on the tabular view of data.
Likewise, filtered choices are ignored in the downloaded .csv
 [ You are not allowed to view attachments ]

Consider implementing filtering choices for the tabular view of data; likewise, in the filter controlling the contents of the downloaded data.


Fetching Dates a Frig

There are no provided means to target a specific date or a range of dates. What is provided instead are left and right arrow buttons which increment according to the Day/ Week/ Month/ Year that the user has chosen; e.g., one click on the left arrow button = one day, week, month, year respectively. The user is not allowed to combine, say for example, three clicks to go back 3 months, then three clicks to go back an additional 3 weeks, because as soon as the user changes temporal increment setting, the interface either resets to displayed date_time to the present, or the browser (Firefox v66.0.3) throws a fit and the page essentially freezes. After several minutes, the display returns to the present moment in time.
 [ You are not allowed to view attachments ]

Consider instead, implementing pulldown calendars to target specific begin and end dates.


Download's Data Granularity

The Date Range is limited to (Day, Week, Month or Year) and with each choice is the implicit decimation of data

Temporal Unit	Decimated Interval ±
Day	5-minute
Week	30-minute
Month	4-hour
Year	1-day

   

This makes downloading historical data for a specific date range and at a specified data interval nearly impossible; e.g., 7 days worth of data at 5-minute intervals beginning 8 months ago.

Consider in addition to the implementation of the pulldown calendars for defining the date range, include another pulldown menu item to select desired data granularity independent of date range; e.g., 5-minue, 10-minute, 15-minute, 20-minute, 30-minute, 60-minute, etc.


Why does the AmbientWeather.net download arrow point up?

Here are a few download icons to consider, all with their arrows pointing down
 [ You are not allowed to view attachments ]


Using an API for downloading data

This then leads back to the subject line of this post: Is there any other way to download data?
Like most Ambient Weather customers, I'm no programmer, but am interested in learning if there are any programmers that have made a program to download our weather station's data.

Thank you for any reply.

Kind regards,

Kelly

--
V. Kelly Bellis, PLS
17 Union Street
Ellsworth, ME 04605

[galfert]

Those are all very good points. I've made those change recommendations to Ambient and then some others too. No word back other than "Thanks for the feedback." That was a while ago (months).

The only thing you mentioned that I didn't notice is the download icon. That is a rather curious use for download. On mouse over it does say "Download .csv" file. But you are right that it is counter intuitive for it to be an up arrow. I figure that from the website programmers perspective the website is doing an upload and your computer is doing the download. Still it is wrong as you pointed out.

Those are the tools the Ambient provides. If you want your data then you can look WeatherCloud but that only provides an Export of every 10 minutes for the data points, and you download a whole month at a time. The free account only lets you go back 12 months.

I would say if you really want your data then you should capture it locally. That is what I do with my Meteobridge. I save my data to my local SQL database every 5 minutes. This has the added benefit that it doesn't require the Internet. It is all local and I can specify just about every characteristic that many online services may not track. The Meteobridge is not the only solution. You can do this too with WeeWx if you use the Interceptor driver....or WeeWx with the ObserverIP driver if you have an ObserverIP.

With the WS-2000 there is always the SD card option too.

WU used to offer a download function. But it doesn't exist in the new WU website. Hopefully they will bring it back. For now the old site is still accessible and it works.

[kbellis]

Using an API for downloading data
As I mentioned, I'm interested in learning if there are any programmers that have made a program to download our weather station's data. Ambient Weather makes this API method sound so sensational, yet provide zero examples of it for their non-programming customers to even look at, much less use.

 [ You are not allowed to view attachments ]

[kbellis]

And this isn't what I meant by an example of it to look at:
 [ You are not allowed to view attachments ]

[galfert]

Ken has made use of the AmbientWeather.net API for his Saratoga template.
https://www.wxforum.net/index.php?topic=33944.0

If you are messing with APIs then that is pretty much always in the realm of programmers. The programmer then is the one that writes the user instructions for you to use their tool/app/program...etc.

If you want to directly use an API then you need to write a program. If you know how to write a program then you should be able to understand the API examples.

[kbellis]

Using an API for downloading data

This then leads back to the subject line of this post: Is there any other way to download data?
Like most Ambient Weather customers, I'm no programmer, but am interested in learning if there are any programmers that have made a program to download our weather station's data.

I'm not messing with APIs George, I'm asking if you, or anybody else, knows of any program that uses the Ambient Weather API for folks who would like to download there their data. And I understand, you've already answered the question.

[galfert]

Okay cool. 
It is just this line you said that seemed to me like you were looking to understand the API yourself directly which doesn't make sense.

Using an API for downloading data
Ambient Weather makes this API method sound so sensational, yet provide zero examples of it for their non-programming customers to even look at, much less use.

Since you are looking for other applications that talk to AmbientWeather.net using their API, here is another one that is a WordPress plug-in.
https://wordpress.org/plugins/live-weather-station/

Other useful links for this Wordpress plug-in:
https://weather.station.software/
https://weather.station.software/handbook/technical-specifications/

But I don't know if they provide export for the data. They make use of it but I don't know if they make the data directly available is what I'm saying.

[galfert]

Have you seen this? Looks like pretty good documentation... for programmers.

https://github.com/ambient-weather

[kbellis]

Yes, I've seen the docs. But like I said, I'm not a programmer. I also don't have a website, nor do I have anything to do with WordPress.

I asked Ed about this matter and he replied much the same as you, that he knows of no application using the ambient weather API in a program that customers can use to download their historic data.

[dunc1ca]

I was wondering something similar.  Is it possible to store and/or download data for each 16-second reading on the WS-2000.  Currently, I can only store once per minute on the unit's SD card and download real-time once per 5 minutes using either the AW app or WU.  I have a VBA program I use in Excel that parses data from my WU station but this is limited to 5-minute granularity.

[galfert]

The fastest update from an Ambient (Fine Offset) station is every 16 seconds as updated to WU Rapidfire. Using an ObserverIP is slower...that is only every 30 seconds so don't go that route. I've read that internal sensors only update every 14 seconds I think which is why WU Rapidfire is limited to every 16 seconds...if not it would be duplicating data. That guarantees that the WU update is always a new reading from the sensors.

So in order to get these 16 second updates you can do one of two things:

• Use WeeWx and the Interceptor driver. Then configure WeeWx to commit to log to database at the interval you desire. You can export this database or send to SQL.
• Use a SDR (software defined radio) to pull the data as it gets transmitted right out of the RF spectrum of your sensors. Then use the SDR software to save to log file or database.

In most cases logging data down to every 5 minutes is pretty normal. I save to a local SQL database every 5 minutes. Please share why you need this level of detail. That is going to create a lot more data and be that much more difficult to manage and analyze.

Stations that report updates within a few seconds are good for Live data viewing. But after the moment passes....eh.

[dunc1ca]

It's just to parse out peak wind gusts and determine when they occurred.  In it's current format, the 5-minute update is likely to skip over peak gusts even though the standard for gusts is supposed to be the highest speed in the past 10 minutes.

[comes]

Hej,

this is an old thread, i know, but today I had the same problem to fetch data from ecowitt.net.

I wrote a little dirty bash script, which did the job for me. All you need is your Session ID and your Device ID. Works fine for me

Code: [Select]

#!/bin/bash
SESSION_ID="SESSION ID"
DEVICE_ID="<MYID>" # Fetch this ID from ecowitt URL. https://www.ecowitt.net/home/index?id=<MYID>

function week2date () {
  local year=$1
  local week=$2
  local dayofweek=$3
  gdate -d "$year-01-01 +$(( $week * 7 + 1 - $(gdate -d "$year-01-04" +%u ) - 3 )) days -2 days + $dayofweek days" +"%Y-%m-%d"
}

function getDateOfDay() {
local day=$1
local year=$2
gdate -d "$year-1-1 +$day days" +%F
}

function loadByDate() {
local date=$1
curl -s 'https://webapi.www.ecowitt.net/index/get_data' \
-X 'POST' \
-H 'Accept: application/json, text/plain, */*' \
-H 'Content-Type: application/x-www-form-urlencoded' \
-H 'Origin: https://www.ecowitt.net' \
-H "Cookie: ousaite_session=${SESSION_ID}; ousaite_language=english; ousaite_loginstatus=1" \
-H 'Content-Length: 93' \
-H 'Accept-Language: en-us' \
-H 'Host: webapi.www.ecowitt.net' \
-H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0.2 Safari/605.1.15' \
-H "Referer: https://www.ecowitt.net/home/index?id=${DEVICE_ID}" \
-H 'Accept-Encoding: gzip, deflate, br' \
-H 'Connection: keep-alive' \
--data "device_id=${DEVICE_ID}&is_list=0&mode=0&sdate=${date}%2000%3A00&edate=${date}%2023%3A59&page=1" | gunzip > $date.json
}


function loadByWeek() {
local start=$1;
local end=$2;
local week=$3;
curl 'https://webapi.www.ecowitt.net/index/get_data' \
-X 'POST' \
-H 'Accept: application/json, text/plain, */*' \
-H 'Content-Type: application/x-www-form-urlencoded' \
-H 'Origin: https://www.ecowitt.net' \
-H "Cookie: ousaite_session=${SESSION_ID}; ousaite_language=english; ousaite_loginstatus=1" \
-H 'Content-Length: 93' \
-H 'Accept-Language: en-us' \
-H 'Host: webapi.www.ecowitt.net' \
-H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0.2 Safari/605.1.15' \
-H "Referer: https://www.ecowitt.net/home/index?id=${DEVICE_ID}" \
-H 'Accept-Encoding: gzip, deflate, br' \
-H 'Connection: keep-alive' \
--data "device_id=${DEVICE_ID}&is_list=0&mode=0&sdate=${start}%2001%3A00&edate=${end}%2023%3A59&page=1" | gunzip > $week.json
}

for day in {0..365}; do
DATE=$(getDateOfDay $day 2020);
echo "Fetching Date: $DATE";
loadByDate $DATE;
sleep 1;
done

for week in {1..53}; do
STARTDATE=$(week2date 2020 $week 1);
ENDDATE=$(week2date 2020 $week 7);
echo "Fetching Week $week ..."

loadByWeek $STARTDATE $ENDDATE $week
done


greetings,

comes

[Tallguy]

comes,

I've got exactly the same problem - started using an Ecowitt system last year, but have been a bit slow in getting Weewx running.  Now I want to export all the archive data @ 5 minute intervals from Ecowitt into the Weewx database.

Could you help me understand a bit more how I configure your bash script?   Where do I get the session ID?  How do I specify the date range I want the script to run over?

Thanks for your help


Julian

[comes]

Hej @Tallguy,

I rebuilt the process last year. I think you should use this PHP script: https://github.com/comes/ecowitt2weewx.

It extracts all your data from ecowitt and transforms them to be compatible with weewx.

If you have further questions, don't hesitate to contact me again.

cheers, J

[Eric971]

in practice, how do you execute this kind of application ?

[comes]

Which kind?

If you're pointing to my app on github, just follow the instructions of the README.md. You have to have PHP available.
I assume, you know how to clone or download something from github.

If you're reaching out to the bash script above, copy and paste in into a file and run it from the a UNIX based OS terminal via

Code: [Select]

bash ./name_of_the_script
cheeers

[Eric971]

Could you explain step to  step, or advise a tutorial ? I don't Know anything about php scripts...