Author Topic: Parsing Weather Data From The Dashboard (Read 5524 times)

Jack Bowman · « **on:** November 04, 2013, 08:16:00 AM »

Has anyone ever tried parsing the weather data from the Backyard Weather Dashboard web page? It then could be transmitted in turn to WUG, or whatever weather site you want. It looks like it updates every 5 minutes. That would be an improvement over the current 15 minute interval that Acurite has control over. If you right click on your dashboard, and look at the source, all the data is there with time stamps. This might be an interesting programming project. Just a thought.

Jack

nincehelser · « **Reply #1 on:** November 04, 2013, 06:33:09 PM »

Interesting idea.

One possible gotcha...I think some of the values like wind speed are averaged out in some way. I'm not sure about rainfall. Other values are probably OK.

I guess one would have to compare graphs at different reporting levels to try to figure out what they might be doing to the data.

Jack Bowman · « **Reply #2 on:** November 04, 2013, 08:49:45 PM »

Yes, I have noticed the wind is a little more intense to harvest.

Jack Bowman · « **Reply #3 on:** November 07, 2013, 09:06:57 AM »

It looks like there is someone developing this application right now. He goes by the name "Kevin Key" on the Acurite web site. I have sent him a message to get involved with beta testing. You can read his post at the end of this thread...

http://support.acurite.com/acurite/topics/how_about_a_premium_service_that_allows_rapid_fire_transfer_of_data_to_wunderground

nincehelser · « **Reply #4 on:** November 07, 2013, 11:18:00 AM »

I hope he's not refreshing the site every 15 seconds and then "screen scraping". Yuck.

If too many people do that, it could really be a burden on Acurite's resources. The way it looks, I doubt they designed their systems to refresh clients that frequently.

Jack Bowman · « **Reply #5 on:** November 07, 2013, 01:30:34 PM »

By looking at his WUG station he refreshing every 5 minutes now. I think he realized the acu-link site only updated every 5 minutes.

nincehelser · « **Reply #6 on:** November 07, 2013, 02:17:43 PM »

5 minutes would be reasonable. Acurite should be expecting that.

If he's wanting to refresh every 15 seconds 24/7, though, I'm sure the Acurite folks are having a cow.

I have to wonder if this morning's outage was a response to introduce counter-measures, but I guess they could just shut-off individual accounts if the load gets to be a problem.

Jack Bowman · « **Reply #7 on:** November 08, 2013, 08:42:42 AM »

Acu-link.com is down again today. Looks like it stopped functioning last night between 10:45pm, and 11:pm Eastern time. This is a great example of why having weather hardware that relies on web services is not a good idea. Direct access to your own data, such as your setup George is the only way to go.

nincehelser · « **Reply #8 on:** November 08, 2013, 09:29:57 AM »

Yep. I also notice that Kevin Keys opened up his screen scraper for everyone to download last night.

Coincidence?

Too many people trying to refresh their pages every 15 seconds could very well crash their proxy server. I'm just guessing, but nginx is a system that tries to balance the network load, and that's what's reporting back that there's a problem.

Jack Bowman · « **Reply #9 on:** November 08, 2013, 09:54:34 AM »

Where is the download link?

nincehelser · « **Reply #10 on:** November 08, 2013, 09:58:08 AM »

He posted it one of the threads on Acurite's support board. Now he also has it listed in the status of his wunderground station:

http://www.wunderground.com/weatherstation/WXDailyHistory.asp?ID=KCASANTE15&day=8&year=2013&month=11

Station Status: Using my custom-built "Kevin's Acu-Link Rapid Fire" app. Download it here: http://www.filedropper.com/aculinkscraper

Jack Bowman · « **Reply #11 on:** November 08, 2013, 10:01:37 AM »

Thanks George.

Jack Bowman · « **Reply #12 on:** November 08, 2013, 01:08:24 PM »

I have this App up, and running now. I just put it on my home media server, which runs 24/7 anyhow. Very easy to setup, and configure. I have it set at 5 minute intervals, and it is working great. System resources on the computer running the software are stable, and consistent. As long as Acu-Link.com can keep up, and running it looks like this is a great solution to the 15 minute WUG Blues. lol

nincehelser · « **Reply #13 on:** November 08, 2013, 02:17:03 PM »

Like I said, 5 minutes might work. That's the normal refresh rate.

However, I still feel it is highly unethical to screen scrape at 15 seconds. It's sucking resources that they don't have.

nincehelser · « **Reply #14 on:** November 08, 2013, 04:03:24 PM »

Quote from: ColoradoWeather on November 08, 2013, 01:08:24 PM

I have this App up, and running now. I just put it on my home media server, which runs 24/7 anyhow. Very easy to setup, and configure. I have it set at 5 minute intervals, and it is working great. System resources on the computer running the software are stable, and consistent. As long as Acu-Link.com can keep up, and running it looks like this is a great solution to the 15 minute WUG Blues. lol

All the other stuff aside... it doesn't look like you're graphing wind gust data. I noticed Kevin Key's site isn't either.

I'm not sure why that would be. I don't specifically report wind-gust data, but I get it graphed.

It would be interesting to see if rain registers properly.

Jack Bowman · « **Reply #15 on:** November 08, 2013, 04:21:21 PM »

What is your update frequency to WUG George?

nincehelser · « **Reply #16 on:** November 08, 2013, 05:02:19 PM »

Quote from: ColoradoWeather on November 08, 2013, 04:21:21 PM

What is your update frequency to WUG George?

One minute.

Jack Bowman · « **Reply #17 on:** November 08, 2013, 09:23:39 PM »

Same reason Acu-Link does not report Wind Gust.

windgustmph - [mph current wind gust, using software specific time period]

windgustmph_10m - [mph past 10 minutes wind gust mph ]

nincehelser · « **Reply #18 on:** November 08, 2013, 11:19:20 PM »

There's a default method wunderground uses for wind gust. I can't remember the details, but it's described on the site somewhere.

Essentially they assume the fasts readings are gusts...I think...I've no idea how technically accurate that is.

It probably has something to do with reporting frequency, but I didn't any wind gusts on Kevin Key's site, either.

Check my station KNEPERU3 and you'll see the green gust markings.

Jack Bowman · « **Reply #19 on:** November 09, 2013, 09:24:12 AM »

I Guess I thought you were calculating your own wind gust based on a time increment George. What does your URL string look like that you send to WUG?

nincehelser · « **Reply #20 on:** November 09, 2013, 12:07:58 PM »

I'd have to dig that out of the weewx code. Here's what I was referred to on wunderground:

http://wiki.wunderground.com/index.php/Educational_-_Wind_gusts_vs._gust_speed

Jack Bowman · « **Reply #21 on:** November 09, 2013, 09:01:19 PM »

Acurite 5-in-1 sensor is now reliably reporting Wind Gust data to Weather underground. Thanks for your help George!

Here is a link to download the Application for those who want their Acurite 5-in-1 sensor to ROCK!

http://www.filedropper.com/aculinkscraperdotnet4

Jack Bowman · « **Reply #22 on:** November 12, 2013, 09:19:55 AM »

Kevin has a blog going on his Windows software development...

http://kevin-key.blogspot.com/2013/11/weather-and-coding-geek-project-acu.html

Here is a link to the latest build version of the software...

http://www.filedropper.com/aculinkscraperdotnet4_4

He has hinted at the release of a new software that will pull readings directly from the bridge. Great stuff!

haroldashe · « **Reply #23 on:** November 12, 2013, 03:50:30 PM »

Every time I go to Kevin's site for this, I get a page asking me to download something called "file dropper." Is that supposed to happen?

Beaudog · « **Reply #24 on:** November 12, 2013, 04:28:09 PM »

File dropper is the web site. You are not downloading file dropper. You are downloading the file
http://www.filedropper.com/aculinkscraperdotnet4_4

All you see in the window is the first part of the file.

News:

Author Topic: Parsing Weather Data From The Dashboard (Read 5524 times)