WXforum.net

Weather Software => Virtual Weather Station => Topic started by: powerline on February 23, 2012, 08:33:46 PM

Title: Wunderground Data Retrieval for Mesonet
Post by: powerline on February 23, 2012, 08:33:46 PM
For years I have retrieved data from Wunderground for my web page's Mesonet map.  For the past several months that process is intermittent at best.  Usually the process of retrieval times out.  I used to retrieve 17 sites.  I reduced that to 9 for a while and that worked.  Now I'm having a problem getting 3.  I've read similar problems here but I don't know if this problem has cleared for everyone else.  Please help if you have any knowledge of this.

http://www.oakridgewx.net/wx.htm
Title: Re: Wunderground Data Retrieval for Mesonet
Post by: jodec on February 24, 2012, 05:04:18 PM
Same thing has been happening to me on and off for a few weeks now.

I use VWS and never had issues with it.  Problems started with the VWSaprs not updating on the normal schedule of every 10 minutes, so I tried every 20 minutes, then 30 minutes.  No matter what options, it would update automatically for a few hours, then stop sending data.  I even went back to the older version and same thing, so just did a reset of my computer, and so far the newer version is still working (for how long, I do not know).  Seems like at this point, that issue has been resolved.

My main problem though is similar to yours.  I can no longer get forecasts like I have been able to get for many years.

I checked the mesonet map, and I noticed in the text information, that the last update was dated 02/18/12 for all mesonet stations!  There is definitely something wrong here.  I am no longer able to get current updates, and nothing has been changed on my end.

I cannot even get into the wunderground website.  Interestingly, I can get into the   classic.wunderground.com website though, and my data is showing up there up to date.. Go Figure.

I must also add here, that I got into the ambient website, and strangely, I clicked on a link to enter the forum, and ended up being redirected to some other website.  My Norton 360 immediately blocked that website, and notified me of a "Web Attack: Blackhole Toolkit Website 12" against my computer.  The Attacking Computer's IP Address was 129.121.93.254,80.  The Attacker URL was: 129.121.93.254/Home/index.php.   :roll:  My computer is safe, and so is my website.

I don't seem to have a way to contact them to report this, but the Norton360, has sent this information to Symantic, so I'm sure they will handle it from there.

As for wunderground, I'm wondering if they too could be infected, or may have possibly been hit with a denial of service attack.  

Jodec

Title: Re: Wunderground Data Retrieval for Mesonet
Post by: NDSWeather on February 25, 2012, 03:13:40 PM
I have the same problem for my web page's Mesonet map and with Wunderground.
I built a new Window XP computer and STILL Have the problem. No data or have the data for my map.
http://weather.mfc-cs.com/map.htm
NDS Weather


Update 8 March 2012:
Installed 1401p48 beta and still have problems.
Sometime half the map is showing and sometime all the map and sometimes no data on the map.
But put your mouse over the map and the links are still working!!

Update 9 March 2012:
Most of today it worked fine.
Update 11 March 2012:

(http://weather.mfc-cs.com/mesomap1.jpg)

Update 13 March 2012:
Looks like it is working now at least on my map above.
Title: Re: Wunderground Data Retrieval for Mesonet
Post by: TupeloWeather on February 26, 2012, 01:27:06 AM
Jodec - can you receive weather alerts and forecast from Wunderground.com?  I have an issue that started a couple of weeks back where neither of those are working correctly.  I even have an alert for a small town in SC (at least that's the town, state and zip that it's retrieving for) right now that is actually for a blizzard warning somewhere out west in the mountains.  Only occasionally will it update either the forecast or alerts.  Is that your issue and I'm wondering if others are experiencing the same exact thing?
Your other issue sounds as if you may have a form of spyware/malware on your computer that redirects you when you click a link.  If you point your cursor to the text or image that links, you can check in the lower left of your browser and it'll display where that link actually goes.  If you click it and it goes somewhere, your probable cause is spyware.  If you aren't already doing so, it would be a good idea to visit download.com and download MalwareBytes or click http://download.cnet.com/Malwarebytes-Anti-Malware/3000-8022_4-10804572.html?tag=mncol;1 (http://download.cnet.com/Malwarebytes-Anti-Malware/3000-8022_4-10804572.html?tag=mncol;1) to go directly to that page. 
Title: Re: Wunderground Data Retrieval for Mesonet
Post by: capeweather on February 26, 2012, 07:06:03 AM
I have been having the same problem as well. Meso will all of a sudden stop updating. I know the data is getting to WU but its failing during the meso fetch. I haven't changed these settings in several years.

http://www.capeweather.com/current-map.html
Title: Re: Wunderground Data Retrieval for Mesonet
Post by: Carson Weather on February 26, 2012, 11:09:54 PM
Me too. For the last several weeks I've had trouble with forecasts and EXPECIALLY warnings. I upgraded to current build and still have the same problems (14.01 P46)
Title: Re: Wunderground Data Retrieval for Mesonet
Post by: daybreak on February 27, 2012, 12:04:45 AM
I have also had the same problem with the mesonet and warnings from Wunderground in VWS.  I quit using my mesonet map because of the lack of readings.  Anyone have any ideas on this?

Richard
Title: Re: Wunderground Data Retrieval for Mesonet
Post by: gwwilk on February 27, 2012, 11:29:31 AM
I just checked the mesonet page on my old VWS driven site that's hosted on my DSL connection, and therefore bandwidth constrained:  http://www.gwwilk.com/mesonet.htm (http://www.gwwilk.com/mesonet.htm)

I haven't updated it in several years, so there are a number of dead '0' sites shown.  Otherwise it looks like it's just been chugging along, ugly as the devil!
Title: Re: Wunderground Data Retrieval for Mesonet
Post by: WeatherShip on February 29, 2012, 12:12:20 PM
My VWS is not downloading any WU data now....
Title: Re: Wunderground Data Retrieval for Mesonet
Post by: capeweather on February 29, 2012, 12:17:44 PM
Has anyone tried contacting WU tech support? My last update was 5 days ago at 4am.
Title: Re: Wunderground Data Retrieval for Mesonet
Post by: capeweather on February 29, 2012, 12:24:28 PM
This probably isn't helpful but found this...

http://wuhelp.wunderground.com/index.php?_m=knowledgebase&_a=viewarticle&kbarticleid=36
Title: Re: Wunderground Data Retrieval for Mesonet
Post by: WeatherShip on February 29, 2012, 05:17:39 PM
I opened up a ticket 4 days ago with WU.....no reply as of yet
Title: Re: Wunderground Data Retrieval for Mesonet
Post by: capeweather on February 29, 2012, 05:41:00 PM
Thanks, I opened one up too. Will let everybody know if they respond.

I opened up a ticket 4 days ago with WU.....no reply as of yet
Title: WU Data
Post by: wolfmarksden on March 03, 2012, 11:20:00 AM
Just wanted to add my 2 cents here and say that my station's mesonet, climate data and forecast downloads from WU have been very intermittant too.  I do seem to be getting weather warnings however, but they are often delayed by several minutes, although still active.  My uploads to them seem to be up-to-date and the flash data current. 
Title: Re: Wunderground Data Retrieval for Mesonet
Post by: daybreak on March 03, 2012, 11:23:18 AM
Everyone,

This is what Ed sent me yesterday, he's definitely aware of the problem:

Yes, I have been bombarded with emails, chats and calls. WU is timing out and VWS does not properly handle the time out. I am going to add something to the tag that states there was an error getting the data, and I have written Wunderground.com about it.
 
It will fix the bug at least but ultimately, they will need to fix the root problem.
 
Regards
,
Ed


So will see what happens.  The main problem is WU.

Richard
Title: Re: Wunderground Data Retrieval for Mesonet
Post by: wunderground-PWS on March 06, 2012, 12:35:28 PM
Hi Folks,

I'm a developer at wunderground. 
If you can send me any additional info about this, like URLs VWS is trying to hit to retrieve data, and exactly what data is missing, I can get this resolved.


Tim
tim@wunderground.com
Title: Re: Wunderground Data Retrieval for Mesonet
Post by: kray1000 on March 06, 2012, 11:35:28 PM
Tim,
Thank you for taking a few seconds to look into the problem.    #-o

The URLs are hard-coded into VWS by the programmer and are not displayed to the user, so you'll need to contact him directly.  In the meantime, do you have any ideas about where on the WU site to start looking, or whom to ask?  How many download servers do you have?  Do you have server logs you can review?  Do you have performance logs or a way of monitoring your servers for response times or CPU usage?  If you are being hit by DoS attacks, could you tell?

Regarding what data is missing... pretty much anything that VWS tries to download from WU.  For the mesonet data, it's random (doesn't matter which zip codes are being queried).  It's different with each download.  The warning data has also been a big problem.  There are issues downloading forecasts from WU.  My dad believes the file is being downloaded but the data in the file is stale.  As of Sunday evening, the 5-day forecast was still starting with Friday.  Not sure if people are having issues downloading climate data.

The download issues have been getting progressively worse for several months.  The timing might even coincide with the WU site change, when the classic version of wunderground.com was replaced with the high-fructose version.  Maybe there are redirects that are slowing things down.  Or perhaps you have a hardware issue such as a bad switch.  Or it might simply mean that you need more server capacity... with the steadily-increasing numbers of home weather stations being deployed, it's reasonable to expect that this might eventually be a problem.

A small part of the problem is that VWS polls for mesonet data right after it is restarted, even if it's only been a short time since the last download, rather than waiting for the scheduled time.  But that's been a bug in VWS for as long as I can remember, not just recently.

Another solution might be for WU to limit mesonet queries to no more often than every 30 minutes, or no more than 150-200 stations per hour.  Hopefully the true cause will be found and we won't need to use limitations like these.  But if it will make the downloads reliable again, I'd be OK with it.

Keep us posted...

(also sent via email)
Title: Re: Wunderground Data Retrieval for Mesonet
Post by: dkinsc on March 07, 2012, 11:05:43 AM
My experience is similar to Kevin's, except that it has sometimes appeared that the Mesonet data was current but the timestamp was old, resulting is the appearance of a bad update.  My problems are chiefly with Mesonet and 5-day forecast.  I have turned my net logger on and send Tim some connection logs for these updates.  Let's hope he can find the problem.

Del
Title: Re: Wunderground Data Retrieval for Mesonet
Post by: Weather Display on March 07, 2012, 12:40:40 PM
wundround setup a special area of formated data to download just specificly for VWS only (i.e known only to VWS and hard coded into VWS, under a partnership promotion thing)
that was done like nearly 10 years ago now, right at the start
I am surprised a developer at wunderground does not know about it..maybe it has been forgoten about (and so is not working very well now after changes) ?
Title: Re: Wunderground Data Retrieval for Mesonet
Post by: wunderground-PWS on March 07, 2012, 01:17:18 PM
Hi Guys,

This issue has been resolved.  We have a group of servers for weather exchange that handles all of this traffic, and it needed to be rebooted.   I just didn't make the connection from VWS to Weather Exchange.

Sorry for the downtime.

-tim
Title: Re: Wunderground Data Retrieval for Mesonet
Post by: capeweather on March 07, 2012, 01:31:09 PM
Thanks Tim,
Do you think it would be a good idea to have that server perform an automatic reboot once a week?

Hi Guys,

This issue has been resolved.  We have a group of servers for weather exchange that handles all of this traffic, and it needed to be rebooted.   I just didn't make the connection from VWS to Weather Exchange.

Sorry for the downtime.

-tim
Title: Re: Wunderground Data Retrieval for Mesonet
Post by: NDSWeather on March 08, 2012, 09:50:07 AM
Hi Guys,

This issue has been resolved.  We have a group of servers for weather exchange that handles all of this traffic, and it needed to be rebooted.   I just didn't make the connection from VWS to Weather Exchange.

Sorry for the downtime.

-tim

Not really!
NDS Weather
Title: Re: Wunderground Data Retrieval for Mesonet
Post by: kray1000 on March 08, 2012, 02:28:28 PM
I haven't seen any improvement either.
Title: Re: Wunderground Data Retrieval for Mesonet
Post by: capeweather on March 08, 2012, 02:50:51 PM
I don't use the forecast but the Mesonet seems to be updating ok at the moment.
Title: Re: Wunderground Data Retrieval for Mesonet
Post by: kray1000 on March 08, 2012, 03:10:10 PM
I don't use the forecast but the Mesonet seems to be updating ok at the moment.

Mine's definitely not.  One current observation and most of the rest are early this morning.
Title: Re: Wunderground Data Retrieval for Mesonet
Post by: dkinsc on March 08, 2012, 04:16:23 PM

Mine's definitely not.  One current observation and most of the rest are early this morning.
My experience, too.  I check sites based on two zipcodes.  One has data from shortly after noon, the other (all near a single zipcode) have not updated since about 8:30am.  All of these sites appear to have current data in the WU Wundermap for the region.
Title: Re: Wunderground Data Retrieval for Mesonet
Post by: capeweather on March 08, 2012, 04:19:38 PM
It just stopped updating for me too. Not working.
Title: Re: Wunderground Data Retrieval for Mesonet
Post by: powerline on March 09, 2012, 10:41:59 AM
Thanks to all of you that have pursued this topic.  Mine now works for a few sites only.  So it's still not working as it should.  If one of you has written Tim at Wunderground recently, I will not pile on.  If not, I'll send him a message.

Ron
Title: Re: Wunderground Data Retrieval for Mesonet
Post by: capeweather on March 09, 2012, 11:06:28 AM
I informed him of this thread so he should be getting the responses via email.

Tim,
Do you have any update on this? Have you spoken with Ed at Ambient at all? Thanks.


Thanks to all of you that have pursued this topic.  Mine now works for a few sites only.  So it's still not working as it should.  If one of you has written Tim at Wunderground recently, I will not pile on.  If not, I'll send him a message.

Ron
Title: Re: Wunderground Data Retrieval for Mesonet
Post by: kray1000 on March 09, 2012, 09:33:33 PM
Thanks to all of you that have pursued this topic.  Mine now works for a few sites only.  So it's still not working as it should.  If one of you has written Tim at Wunderground recently, I will not pile on.  If not, I'll send him a message.

Feel free to pile on.  They'll never know you're having an issue unless you tell them.  It's not like anyone at WU is actually reading this thread.
Title: Re: Wunderground Data Retrieval for Mesonet
Post by: capeweather on March 09, 2012, 09:57:17 PM
Here is the direct email for Tim, who I have been in contact with.

tim@wunderground.com
Title: Re: Wunderground Data Retrieval for Mesonet
Post by: powerline on March 13, 2012, 09:39:16 AM
Has anyone gotten an answer from Tim at Wunderground yet? 
Title: Re: Wunderground Data Retrieval for Mesonet
Post by: kray1000 on March 13, 2012, 05:16:01 PM
Hey, my mesonet is working.
Title: Re: Wunderground Data Retrieval for Mesonet
Post by: dasman on March 13, 2012, 07:32:13 PM
I wanted to add to this discussion.  4 weeks ago I made a meso map which obviously is updated with the mesonet info from wunderground through VWS.  I didn't notice right away but after some time I saw that my rapid fire at wunderground was not working, and was only updating every 30 minutes. If I did a VWS shut down and re-start, the rapid fire would work again for 12 hours or so. I did not see a connection between the 2 untill one day on the meso map I saw some of the data missing.  I shut the meonet data retrieval off in VWS and my rapid fire worked correctly again.  After a couple days I started the mesonet data retrieval again and the rapid fire started failing again.  So I have removed the meso map and stopped the mesonet retrieval.  I wonder if this is tied in to all of the issues you guys are having with the mesonet data retrieval.  If it is I hope its fixed soon so I can put the meso map back on my page, it was liked by the people who visit my site.
Title: Re: Wunderground Data Retrieval for Mesonet
Post by: kray1000 on March 13, 2012, 07:56:31 PM
I wanted to add to this discussion.  4 weeks ago I made a meso map which obviously is updated with the mesonet info from wunderground through VWS.  I didn't notice right away but after some time I saw that my rapid fire at wunderground was not working, and was only updating every 30 minutes. If I did a VWS shut down and re-start, the rapid fire would work again for 12 hours or so. I did not see a connection between the 2 untill one day on the meso map I saw some of the data missing.  I shut the meonet data retrieval off in VWS and my rapid fire worked correctly again.  After a couple days I started the mesonet data retrieval again and the rapid fire started failing again.  So I have removed the meso map and stopped the mesonet retrieval.  I wonder if this is tied in to all of the issues you guys are having with the mesonet data retrieval.  If it is I hope its fixed soon so I can put the meso map back on my page, it was liked by the people who visit my site.

I haven't noticed any issues with my Rapid Fire, but my mesonet and Rapid Fire are both working well at the moment.  I think this is the best the WU mesonet has worked for many months, and I'm cautiously optomistic that they've found the problem.  Perhaps you could turn on the mesonet updates and monitor it for a day or so before you put it back on your site.
Title: Re: Wunderground Data Retrieval for Mesonet
Post by: dasman on March 13, 2012, 08:53:51 PM
I think I will try that on Saturday.  I hope that they have solved the issue.

Thanks
Title: Re: Wunderground Data Retrieval for Mesonet
Post by: powerline on March 16, 2012, 11:02:35 PM
Mine has been working the past few days.  How's everyone else's?  Maybe thanks to Tim at Wunderground.


Ron
Title: Re: Wunderground Data Retrieval for Mesonet
Post by: capeweather on March 16, 2012, 11:08:38 PM
Been running good! Haven't noticed any problems.

Mine has been working the past few days.  How's everyone else's?  Maybe thanks to Tim at Wunderground.


Ron
Title: Re: Wunderground Data Retrieval for Mesonet
Post by: kray1000 on March 17, 2012, 07:00:30 PM
Been running good! Haven't noticed any problems.

Kinda curious what the issue was.