Author Topic: Wunderground Data Volumes  (Read 2472 times)

0 Members and 1 Guest are viewing this topic.

Offline johnmn

  • Member
  • *
  • Posts: 29
Wunderground Data Volumes
« on: July 28, 2017, 04:54:54 AM »
Hi. I'm trying to find out how I can minimise the data traffic with Wunderground. I have my updates happening every 15 minutes, so that means nearly 3000 GET posts to wunderground a month.

This (plus responses and other overheads, whatever they might be) seems to be using up to 130MB of data a month, which works out to over 33KB per update!

This seems like a lot, and I'd like to cut it down if I could. Can anyone offer suggestions or observations?
Wunderground do not seem to be very helpful in this area (they don't seem to have an open forum like this...).

Thanks for any assistance.

Offline Jáchym

  • Meteotemplate Developer
  • Forecaster
  • *****
  • Posts: 8605
    • Meteotemplate
Re: Wunderground Data Volumes
« Reply #1 on: July 28, 2017, 05:31:08 AM »
Hi John,

please do not re-post stuff in different threads, especially when someone has already taken the time and replied to you.

http://www.wxforum.net/index.php?topic=32433.0

Thanks

Offline johnmn

  • Member
  • *
  • Posts: 29
Re: Wunderground Data Volumes
« Reply #2 on: July 28, 2017, 09:13:05 AM »
Given that the last post was in the Meteobridge area, and did not help with my problem, it seems reasonable to ask the question where there might be folk who knew more about Weather Underground and issues related to working with them.

Your authoritarian response is no help whatever, and can only discourage people trying to use this forum to get assistance.

Or are you are telling me that the same half a dozen people are all I'm ever going to encounter here?

Offline ValentineWeather

  • Forecaster
  • *****
  • Posts: 6362
    • Valentine Nebraska's Real-Time Weather
Re: Wunderground Data Volumes
« Reply #3 on: July 28, 2017, 10:33:00 AM »
130MB of data monthly isn't much. I send data every 4 seconds with rapid fire. I really wouldn't be concerned WU wants the data even more often than 15 minutes, plus they continue to urge the weather community to add stations cameras etc.

Now it's different if you have other issues like limited bandwidth and trying to reduce uploads you can always extend time or stop sending.
 
BTW I never saw the other post because I have certain forum topics blocked.
Randy

Offline Jáchym

  • Meteotemplate Developer
  • Forecaster
  • *****
  • Posts: 8605
    • Meteotemplate
Re: Wunderground Data Volumes
« Reply #4 on: July 28, 2017, 10:38:47 AM »
Given that the last post was in the Meteobridge area, and did not help with my problem, it seems reasonable to ask the question where there might be folk who knew more about Weather Underground and issues related to working with them.

Your authoritarian response is no help whatever, and can only discourage people trying to use this forum to get assistance.

Or are you are telling me that the same half a dozen people are all I'm ever going to encounter here?

Thats fine, but you can always ask the moderator to move your post  to a different topic - simply because others might see it and also see the answers which have already been provided.

Offline waiukuweather

  • Forecaster
  • *****
  • Posts: 1072
Re: Wunderground Data Volumes
« Reply #5 on: July 28, 2017, 04:11:59 PM »
Quote
Or are you are telling me that the same half a dozen people are all I'm ever going to encounter here?
basicly, yes as not everyone is willing to put their necks on the line and make suggestions or ideas on topics where the answer is not always straight forward or might be outside their area of expertise

Offline ocala

  • Forecaster
  • *****
  • Posts: 4383
Re: Wunderground Data Volumes
« Reply #6 on: July 28, 2017, 05:17:09 PM »
What software are you using to send your data to WU?
Also I agree with Randy in that I have certain  topics blocked so I didn't see it either.
« Last Edit: July 28, 2017, 05:19:15 PM by ocala »
The blues had a baby and they named it Rock & Roll

Offline johnmn

  • Member
  • *
  • Posts: 29
Re: Wunderground Data Volumes
« Reply #7 on: July 28, 2017, 08:39:10 PM »
Thanks for the replies.

The remote site is on a plan which triples in cost at 100MB, so it will be worth it if I can get it down below that. The folk who pay for it (model aircraft club) would be happier!

Certainly the traffic I'm aware of is less than 1/10 of my limit.

It's a shame that folk have to block to reduce their messages, perhaps the structure of the forum has become out of date?

I am using a Meteobridge to send the data, and am confident it is not behaving badly. I really can't see what the problem can be and have only been hoping for some "left field" viewpoint or inspiration. Originally I wondered if the remote modem was doing something to retain it's connection, for example. (No luck there so far).

I'm certainly not looking for any gold-plated answer or guarantee! Well meaning suggestions are fine.

Thanks again.

Offline ValentineWeather

  • Forecaster
  • *****
  • Posts: 6362
    • Valentine Nebraska's Real-Time Weather
Re: Wunderground Data Volumes
« Reply #8 on: July 28, 2017, 09:04:55 PM »
The reason for blocking sections of forum is when multiple messages get posted on subject matter (Not interested in like Metrobridge) it fills up (Show unread posts since last visit ) and you end up missing pertinent post you actually may have interest in.

I agree if this wouldn't get full people won't need to block anything.
Randy

Offline SLOweather

  • Global Moderator
  • Forecaster
  • *****
  • Posts: 3456
    • Weatherelement Moline IL
Re: Wunderground Data Volumes
« Reply #9 on: July 29, 2017, 09:55:44 AM »
OK, johnmn, here you go.

Wunderground uses HTTP GET to send weather data to their servers from any software or hardware device. I believe they do this because if you can web browse over a particular connection, you can send data, rather than using a special port or other protocol.

A typical upload looks like this:

Quote
https://weatherstation.wunderground.com/weatherstation/updateweatherstation.php?ID=KCASANFR5&PASSWORD=XXXXXX&dateutc=2000-01-01+10%3A32%3A35&winddir=230&windspeedmph=12&windgustmph=12&tempf=70&rainin=0&baromin=29.1&dewptf=68.2&humidity=90&weather=&clouds=&softwaretype=vws%20versionxx&action=updateraw

The first problem you may notice is that it's fairly verbose. The upload itself could be reduced by more than half if they would just allow abbreviations, instead of "windspeedmph" etc. It also includes stuff in every upload that probably doesn't change, like softwaretype and version.

The second problem is that, generally unseen by end users, web server software, such as Apache, returns its own verbose header stuff to the browser or other software/hardware as a part of every web transaction. Some of the information can be controlled in server settings, as I recall. Some can only be removed by recompiling the server software.

Sorry, I can't locate a good example of the return header right this minute.

We went through this using a very similar upload process for a cellular weather station project, and reduced our upload with abbreviations and another trick or 2, but never really handled the server end of it.

More on the WeatherUnderground upload is at http://wiki.wunderground.com/index.php/PWS_-_Upload_Protocol

Offline johnmn

  • Member
  • *
  • Posts: 29
Re: Wunderground Data Volumes
« Reply #10 on: July 29, 2017, 10:32:25 AM »
Thank you for this response.

So, if I am uploading only every 15min that is about 30K of upload a month, even including the 'success<CR,LF>' that comes back.

So of the 130MB used each month, 100MB is "verbose header stuff", and unless I monitor the packets I will never know what it is, and can't do anything about it anyway.

I was afraid that was the fact, and only wanted to know if there was some feasible way to change this.

Thanks again for your input.

Offline SLOweather

  • Global Moderator
  • Forecaster
  • *****
  • Posts: 3456
    • Weatherelement Moline IL
Re: Wunderground Data Volumes
« Reply #11 on: July 29, 2017, 11:16:35 AM »
You are welcome. If you are really interested , you can do a little work and actually see what comes back from the server.

Install a packet sniffer like Wireshark on your computer. Then create an upload packet by hand in a browser as shown in the wiki article, and see what is returned by the server in Wireshark.

Offline SLOweather

  • Global Moderator
  • Forecaster
  • *****
  • Posts: 3456
    • Weatherelement Moline IL
Re: Wunderground Data Volumes
« Reply #12 on: July 29, 2017, 01:39:02 PM »
I think this is a typical response from my server. Note that the only thing my hardware is interested in is the end... *OK

Quote
Server Response (193 bytes): HTTP/1.1 200 OK
Date: Tue, 15 Sep 2009 13:51:14 GMT
Server: Apache/2.2.8 (Fedora)
X-Powered-By: PHP/5.2.9
Content-Length: 3
Connection: close
Content-Type: text/html; charset=UTF-8

*OK

There is a lot of other stuff a server can send back as well. It would be interesting to see what Wunderground returns...


Offline SLOweather

  • Global Moderator
  • Forecaster
  • *****
  • Posts: 3456
    • Weatherelement Moline IL
Re: Wunderground Data Volumes
« Reply #13 on: July 29, 2017, 01:50:20 PM »
As long as we're talking about it, there's is also extra, unseen stuff sent to the server with the data upload, too. NOTE, this is NOT a WU upload, although parts of it look like it. Parts of it have been obfuscated or changed for security reasons.

Quote
GET /datasave.php?unit=xxx&password=abcdef&datelocal=2009-09-15&timelocal=06:48:
06&winddir=303&wind=0&gust=0&avgwind=2&tempout=60.3&tempin=68.9&dayrain=0.00&rai
nrate=0.00&uv=0.0&stormrain=0.00&stormstart=2127-15-31&solar=7&baro=29.994&bartr
end=0&rhout=75&rhin=54&etday=0.00&xmitbattstat=0&consolebatvolts=4.37&forecastic
on=6&forecastrule=45&ip=192.168.1.170&ver=4.4&xt=146,185,,,,,&xst=160,150,150,&x
lt=160,150,,&xh=90,,,,,,&xsm=177,,,&xlw=0,,,0&status=OK HTTP/1.1
Accept: text/*
Accept-Language: en-us
UA-CPU: x86
Accept-Encoding: gzip' deflate
Connection: Keep-Alive
Host: www.XXXYYYZZZ.com

So there is more data burden that affects your monthly limit. And, there may be even more stuff that can sent up that doesn't appear here.


Offline johnmn

  • Member
  • *
  • Posts: 29
Re: Wunderground Data Volumes
« Reply #14 on: July 30, 2017, 12:18:39 AM »
Thanks SLOweather.

Looking at the Wireshark data here at my office (the site in the field will probably be different and has no PC) it is clear there is a lot going on. Lot's of which means nothing to me.

It seems Weather Underground use Amazon Web Services, and so additional lookups and redirections happen.
I'm beginning to think it amazing I only use 130MB/month!

This also helps explain why Wunderground are so unresponsive to requests for explanation or assistance.

It may be that I will have to get my Meteobridge to send the data to another website with overheads we can manage.

Thanks very much for pointing me in the right direction.

Offline SLOweather

  • Global Moderator
  • Forecaster
  • *****
  • Posts: 3456
    • Weatherelement Moline IL
Re: Wunderground Data Volumes
« Reply #15 on: July 30, 2017, 03:11:45 PM »
You're welcome.

We spent a lot of time and effort to minimize our data burden for our cellular weather station. If I'm not mistaken, we got it down to about a megabyte a month at 10 minute updates. That's still almost 7kB per update.

Offline rods55555

  • Senior Member
  • **
  • Posts: 56
Re: Wunderground Data Volumes
« Reply #16 on: August 05, 2017, 08:33:28 PM »
I bought a new router, and now I have a better "view" on how much data is used/send to Weather Underground. My pws is an Ambient Weather ws-1001 clone. Just for curiosity since my isp have unlimited data.

 [ You are not allowed to view attachments ]

 [ You are not allowed to view attachments ]

 [ You are not allowed to view attachments ]
« Last Edit: August 05, 2017, 08:35:19 PM by rods55555 »

Offline johnmn

  • Member
  • *
  • Posts: 29
Re: Wunderground Data Volumes
« Reply #17 on: August 05, 2017, 09:18:21 PM »
I bought a new router, and now I have a better "view" on how much data is used/send to Weather Underground. My pws is an Ambient Weather ws-1001 clone. Just for curiosity since my isp have unlimited data.

Wow. The amount of data is amazing, especially the download, just to get 'success<CR/LF>' from WU!
There is a colossal amount of protocol chatter.

Thanks.

Offline johnmn

  • Member
  • *
  • Posts: 29
Re: Wunderground Data Volumes
« Reply #18 on: August 05, 2017, 09:21:00 PM »
We spent a lot of time and effort to minimize our data burden for our cellular weather station. If I'm not mistaken, we got it down to about a megabyte a month at 10 minute updates. That's still almost 7kB per update.

Actually, looking at these figures, 10min updates over a month is 4464 updates, so if a megabyte is 1024*1024 it comes to 234bytes per update.

That's great!