Author Topic: Wunderground Data Retrieval for Mesonet  (Read 3592 times)

0 Members and 1 Guest are viewing this topic.

Offline powerline

  • Member
  • *
  • Posts: 8
    • Oak Ridge North Weather
Wunderground Data Retrieval for Mesonet
« on: February 23, 2012, 08:33:46 PM »
For years I have retrieved data from Wunderground for my web page's Mesonet map.  For the past several months that process is intermittent at best.  Usually the process of retrieval times out.  I used to retrieve 17 sites.  I reduced that to 9 for a while and that worked.  Now I'm having a problem getting 3.  I've read similar problems here but I don't know if this problem has cleared for everyone else.  Please help if you have any knowledge of this.

http://www.oakridgewx.net/wx.htm

Offline jodec

  • Member
  • *
  • Posts: 2
    • JFDHOBBIES.COM - Hobbies are great
Re: Wunderground Data Retrieval for Mesonet
« Reply #1 on: February 24, 2012, 05:04:18 PM »
Same thing has been happening to me on and off for a few weeks now.

I use VWS and never had issues with it.  Problems started with the VWSaprs not updating on the normal schedule of every 10 minutes, so I tried every 20 minutes, then 30 minutes.  No matter what options, it would update automatically for a few hours, then stop sending data.  I even went back to the older version and same thing, so just did a reset of my computer, and so far the newer version is still working (for how long, I do not know).  Seems like at this point, that issue has been resolved.

My main problem though is similar to yours.  I can no longer get forecasts like I have been able to get for many years.

I checked the mesonet map, and I noticed in the text information, that the last update was dated 02/18/12 for all mesonet stations!  There is definitely something wrong here.  I am no longer able to get current updates, and nothing has been changed on my end.

I cannot even get into the wunderground website.  Interestingly, I can get into the   classic.wunderground.com website though, and my data is showing up there up to date.. Go Figure.

I must also add here, that I got into the ambient website, and strangely, I clicked on a link to enter the forum, and ended up being redirected to some other website.  My Norton 360 immediately blocked that website, and notified me of a "Web Attack: Blackhole Toolkit Website 12" against my computer.  The Attacking Computer's IP Address was 129.121.93.254,80.  The Attacker URL was: 129.121.93.254/Home/index.php.   :roll:  My computer is safe, and so is my website.

I don't seem to have a way to contact them to report this, but the Norton360, has sent this information to Symantic, so I'm sure they will handle it from there.

As for wunderground, I'm wondering if they too could be infected, or may have possibly been hit with a denial of service attack.  

Jodec

« Last Edit: February 24, 2012, 05:07:45 PM by jodec »

Offline bcavnaugh

  • NDS Weather - Ken Caryl Colorado
  • Contributor
  • ***
  • Posts: 115
  • NDS Weather - Ken Caryl Colorado
    • NDS Weather - Ken Caryl Colorado
Re: Wunderground Data Retrieval for Mesonet
« Reply #2 on: February 25, 2012, 03:13:40 PM »
I have the same problem for my web page's Mesonet map and with Wunderground.
I built a new Window XP computer and STILL Have the problem. No data or have the data for my map.
http://weather.mfc-cs.com/map.htm
NDS Weather


Update 8 March 2012:
Installed 1401p48 beta and still have problems.
Sometime half the map is showing and sometime all the map and sometimes no data on the map.
But put your mouse over the map and the links are still working!!

Update 9 March 2012:
Most of today it worked fine.
Update 11 March 2012:



Update 13 March 2012:
Looks like it is working now at least on my map above.
« Last Edit: March 13, 2012, 11:10:24 AM by bcavnaugh »
6153 Davis Wireless Vantage Pro2 FW 1.90 Serial Port
VWS 14.01p48b / ImageSalsa 2.0.13b Weather Edition / WeatherLink 6.0.0
VWSaprs 2.0.1.0 KCOLITTL9 CW4784

Offline TupeloWeather

  • Member
  • *
  • Posts: 26
    • Up-To-The-Minute Weather for Tupelo, MS
Re: Wunderground Data Retrieval for Mesonet
« Reply #3 on: February 26, 2012, 01:27:06 AM »
Jodec - can you receive weather alerts and forecast from Wunderground.com?  I have an issue that started a couple of weeks back where neither of those are working correctly.  I even have an alert for a small town in SC (at least that's the town, state and zip that it's retrieving for) right now that is actually for a blizzard warning somewhere out west in the mountains.  Only occasionally will it update either the forecast or alerts.  Is that your issue and I'm wondering if others are experiencing the same exact thing?
Your other issue sounds as if you may have a form of spyware/malware on your computer that redirects you when you click a link.  If you point your cursor to the text or image that links, you can check in the lower left of your browser and it'll display where that link actually goes.  If you click it and it goes somewhere, your probable cause is spyware.  If you aren't already doing so, it would be a good idea to visit download.com and download MalwareBytes or click http://download.cnet.com/Malwarebytes-Anti-Malware/3000-8022_4-10804572.html?tag=mncol;1 to go directly to that page. 

Offline capeweather

  • Administrator
  • Forecaster
  • *****
  • Posts: 998
    • http://www.capeweather.com
Re: Wunderground Data Retrieval for Mesonet
« Reply #4 on: February 26, 2012, 07:06:03 AM »
I have been having the same problem as well. Meso will all of a sudden stop updating. I know the data is getting to WU but its failing during the meso fetch. I haven't changed these settings in several years.

http://www.capeweather.com/current-map.html

Chris
Cape Coral, Florida
Website: http://www.capeweather.com
Website: http://www.fortmyersweather.net

Offline Carson Weather

  • Senior Member
  • **
  • Posts: 98
    • http://www.carsonweather.com
Re: Wunderground Data Retrieval for Mesonet
« Reply #5 on: February 26, 2012, 11:09:54 PM »
Me too. For the last several weeks I've had trouble with forecasts and EXPECIALLY warnings. I upgraded to current build and still have the same problems (14.01 P46)

Offline daybreak

  • Senior Member
  • **
  • Posts: 50
    • Alma Weather
Re: Wunderground Data Retrieval for Mesonet
« Reply #6 on: February 27, 2012, 12:04:45 AM »
I have also had the same problem with the mesonet and warnings from Wunderground in VWS.  I quit using my mesonet map because of the lack of readings.  Anyone have any ideas on this?

Richard

Online gwwilk

  • Southeast Lincoln Weather
  • Forecaster
  • *****
  • Posts: 826
    • SouthEast Lincoln, NE Weather
Re: Wunderground Data Retrieval for Mesonet
« Reply #7 on: February 27, 2012, 11:29:31 AM »
I just checked the mesonet page on my old VWS driven site that's hosted on my DSL connection, and therefore bandwidth constrained:  http://www.gwwilk.com/mesonet.htm

I haven't updated it in several years, so there are a number of dead '0' sites shown.  Otherwise it looks like it's just been chugging along, ugly as the devil!
Regards, Jerry Wilkins
gwwilk@inebraska.com

Offline weathership

  • WeatherShip
  • Member
  • *
  • Posts: 2
    • Weathership
Re: Wunderground Data Retrieval for Mesonet
« Reply #8 on: February 29, 2012, 12:12:20 PM »
My VWS is not downloading any WU data now....

Offline capeweather

  • Administrator
  • Forecaster
  • *****
  • Posts: 998
    • http://www.capeweather.com
Re: Wunderground Data Retrieval for Mesonet
« Reply #9 on: February 29, 2012, 12:17:44 PM »
Has anyone tried contacting WU tech support? My last update was 5 days ago at 4am.

Chris
Cape Coral, Florida
Website: http://www.capeweather.com
Website: http://www.fortmyersweather.net

Offline capeweather

  • Administrator
  • Forecaster
  • *****
  • Posts: 998
    • http://www.capeweather.com
Re: Wunderground Data Retrieval for Mesonet
« Reply #10 on: February 29, 2012, 12:24:28 PM »

Chris
Cape Coral, Florida
Website: http://www.capeweather.com
Website: http://www.fortmyersweather.net

Offline weathership

  • WeatherShip
  • Member
  • *
  • Posts: 2
    • Weathership
Re: Wunderground Data Retrieval for Mesonet
« Reply #11 on: February 29, 2012, 05:17:39 PM »
I opened up a ticket 4 days ago with WU.....no reply as of yet

Offline capeweather

  • Administrator
  • Forecaster
  • *****
  • Posts: 998
    • http://www.capeweather.com
Re: Wunderground Data Retrieval for Mesonet
« Reply #12 on: February 29, 2012, 05:41:00 PM »
Thanks, I opened one up too. Will let everybody know if they respond.

I opened up a ticket 4 days ago with WU.....no reply as of yet

Chris
Cape Coral, Florida
Website: http://www.capeweather.com
Website: http://www.fortmyersweather.net

Offline wolfmarksden

  • Member
  • *
  • Posts: 1
    • Wolfmarksden.com
WU Data
« Reply #13 on: March 03, 2012, 11:20:00 AM »
Just wanted to add my 2 cents here and say that my station's mesonet, climate data and forecast downloads from WU have been very intermittant too.  I do seem to be getting weather warnings however, but they are often delayed by several minutes, although still active.  My uploads to them seem to be up-to-date and the flash data current. 

Offline daybreak

  • Senior Member
  • **
  • Posts: 50
    • Alma Weather
Re: Wunderground Data Retrieval for Mesonet
« Reply #14 on: March 03, 2012, 11:23:18 AM »
Everyone,

This is what Ed sent me yesterday, he's definitely aware of the problem:

Yes, I have been bombarded with emails, chats and calls. WU is timing out and VWS does not properly handle the time out. I am going to add something to the tag that states there was an error getting the data, and I have written Wunderground.com about it.
 
It will fix the bug at least but ultimately, they will need to fix the root problem.
 
Regards
,
Ed


So will see what happens.  The main problem is WU.

Richard

Offline redtim

  • Member
  • *
  • Posts: 2
Re: Wunderground Data Retrieval for Mesonet
« Reply #15 on: March 06, 2012, 12:35:28 PM »
Hi Folks,

I'm a developer at wunderground. 
If you can send me any additional info about this, like URLs VWS is trying to hit to retrieve data, and exactly what data is missing, I can get this resolved.


Tim
tim@wunderground.com

Offline kray1000

  • Purveyor of wry
  • Forecaster
  • *****
  • Posts: 1326
    • http://www.roanokevalleyweather.com
Re: Wunderground Data Retrieval for Mesonet
« Reply #16 on: March 06, 2012, 11:35:28 PM »
Tim,
Thank you for taking a few seconds to look into the problem.    #-o

The URLs are hard-coded into VWS by the programmer and are not displayed to the user, so you'll need to contact him directly.  In the meantime, do you have any ideas about where on the WU site to start looking, or whom to ask?  How many download servers do you have?  Do you have server logs you can review?  Do you have performance logs or a way of monitoring your servers for response times or CPU usage?  If you are being hit by DoS attacks, could you tell?

Regarding what data is missing... pretty much anything that VWS tries to download from WU.  For the mesonet data, it's random (doesn't matter which zip codes are being queried).  It's different with each download.  The warning data has also been a big problem.  There are issues downloading forecasts from WU.  My dad believes the file is being downloaded but the data in the file is stale.  As of Sunday evening, the 5-day forecast was still starting with Friday.  Not sure if people are having issues downloading climate data.

The download issues have been getting progressively worse for several months.  The timing might even coincide with the WU site change, when the classic version of wunderground.com was replaced with the high-fructose version.  Maybe there are redirects that are slowing things down.  Or perhaps you have a hardware issue such as a bad switch.  Or it might simply mean that you need more server capacity... with the steadily-increasing numbers of home weather stations being deployed, it's reasonable to expect that this might eventually be a problem.

A small part of the problem is that VWS polls for mesonet data right after it is restarted, even if it's only been a short time since the last download, rather than waiting for the scheduled time.  But that's been a bug in VWS for as long as I can remember, not just recently.

Another solution might be for WU to limit mesonet queries to no more often than every 30 minutes, or no more than 150-200 stations per hour.  Hopefully the true cause will be found and we won't need to use limitations like these.  But if it will make the downloads reliable again, I'd be OK with it.

Keep us posted...

(also sent via email)
« Last Edit: March 06, 2012, 11:38:49 PM by kray1000 »
Kevin Ray
Roanoke, VA

Wireless VP2, Stardot NetCam, ImageSalsa 2.0.12, MovieSalsa, Nexstorm, WASP2, Skymet, VWS14.01p33, WD 10.37N build 03

Offline dkinsc

  • Contributor
  • ***
  • Posts: 146
    • ClemsonKFAP Weather
Re: Wunderground Data Retrieval for Mesonet
« Reply #17 on: March 07, 2012, 11:05:43 AM »
My experience is similar to Kevin's, except that it has sometimes appeared that the Mesonet data was current but the timestamp was old, resulting is the appearance of a bad update.  My problems are chiefly with Mesonet and 5-day forecast.  I have turned my net logger on and send Tim some connection logs for these updates.  Let's hope he can find the problem.

Del
Del
KSCCLEMS2 and DW6419

Offline Weather Display

  • Forecaster
  • *****
  • Posts: 2611
    • West Coast Road Weather Data
Re: Wunderground Data Retrieval for Mesonet
« Reply #18 on: March 07, 2012, 12:40:40 PM »
wundround setup a special area of formated data to download just specificly for VWS only (i.e known only to VWS and hard coded into VWS, under a partnership promotion thing)
that was done like nearly 10 years ago now, right at the start
I am surprised a developer at wunderground does not know about it..maybe it has been forgoten about (and so is not working very well now after changes) ?

Offline redtim

  • Member
  • *
  • Posts: 2
Re: Wunderground Data Retrieval for Mesonet
« Reply #19 on: March 07, 2012, 01:17:18 PM »
Hi Guys,

This issue has been resolved.  We have a group of servers for weather exchange that handles all of this traffic, and it needed to be rebooted.   I just didn't make the connection from VWS to Weather Exchange.

Sorry for the downtime.

-tim

Offline capeweather

  • Administrator
  • Forecaster
  • *****
  • Posts: 998
    • http://www.capeweather.com
Re: Wunderground Data Retrieval for Mesonet
« Reply #20 on: March 07, 2012, 01:31:09 PM »
Thanks Tim,
Do you think it would be a good idea to have that server perform an automatic reboot once a week?

Hi Guys,

This issue has been resolved.  We have a group of servers for weather exchange that handles all of this traffic, and it needed to be rebooted.   I just didn't make the connection from VWS to Weather Exchange.

Sorry for the downtime.

-tim

Chris
Cape Coral, Florida
Website: http://www.capeweather.com
Website: http://www.fortmyersweather.net

Offline bcavnaugh

  • NDS Weather - Ken Caryl Colorado
  • Contributor
  • ***
  • Posts: 115
  • NDS Weather - Ken Caryl Colorado
    • NDS Weather - Ken Caryl Colorado
Re: Wunderground Data Retrieval for Mesonet
« Reply #21 on: March 08, 2012, 09:50:07 AM »
Hi Guys,

This issue has been resolved.  We have a group of servers for weather exchange that handles all of this traffic, and it needed to be rebooted.   I just didn't make the connection from VWS to Weather Exchange.

Sorry for the downtime.

-tim

Not really!
NDS Weather
6153 Davis Wireless Vantage Pro2 FW 1.90 Serial Port
VWS 14.01p48b / ImageSalsa 2.0.13b Weather Edition / WeatherLink 6.0.0
VWSaprs 2.0.1.0 KCOLITTL9 CW4784

Offline kray1000

  • Purveyor of wry
  • Forecaster
  • *****
  • Posts: 1326
    • http://www.roanokevalleyweather.com
Re: Wunderground Data Retrieval for Mesonet
« Reply #22 on: March 08, 2012, 02:28:28 PM »
I haven't seen any improvement either.
Kevin Ray
Roanoke, VA

Wireless VP2, Stardot NetCam, ImageSalsa 2.0.12, MovieSalsa, Nexstorm, WASP2, Skymet, VWS14.01p33, WD 10.37N build 03

Offline capeweather

  • Administrator
  • Forecaster
  • *****
  • Posts: 998
    • http://www.capeweather.com
Re: Wunderground Data Retrieval for Mesonet
« Reply #23 on: March 08, 2012, 02:50:51 PM »
I don't use the forecast but the Mesonet seems to be updating ok at the moment.

Chris
Cape Coral, Florida
Website: http://www.capeweather.com
Website: http://www.fortmyersweather.net

Offline kray1000

  • Purveyor of wry
  • Forecaster
  • *****
  • Posts: 1326
    • http://www.roanokevalleyweather.com
Re: Wunderground Data Retrieval for Mesonet
« Reply #24 on: March 08, 2012, 03:10:10 PM »
I don't use the forecast but the Mesonet seems to be updating ok at the moment.

Mine's definitely not.  One current observation and most of the rest are early this morning.
Kevin Ray
Roanoke, VA

Wireless VP2, Stardot NetCam, ImageSalsa 2.0.12, MovieSalsa, Nexstorm, WASP2, Skymet, VWS14.01p33, WD 10.37N build 03