Author Topic: WeatherUnderground changed again.. WU-forecast.php broken (again)  (Read 7510 times)

0 Members and 1 Guest are viewing this topic.

Offline saratogaWX

  • Administrator
  • Forecaster
  • *****
  • Posts: 9279
  • Saratoga, CA, USA Weather - free PHP scripts
    • Saratoga-Weather.org
This time, it may be for keeps that the WU-forecast.php script is broken.  WU stopped publishing the forecast JSON in the public page and switched to using a jquery AJAX fetch of the data using their API instead.

So.. the current script (V2.08) is likely not-fixable as the data it used is no longer included in the public HTML of the page.

I'm working on a replacement script that uses the WU API directly, but-- in order to use it, you would have to acquire a WU API key for use on your site -- I hate that ... but they do not offer a 'public' API.  It will also mean respecifying ALL the links to the forecasts needed as the API uses a different (incompatible) addressing scheme.  Grrrr... ](*,)  ](*,)
Ken True/Saratoga, CA, USA main site: saratoga-weather.org
Davis VP1+ FARS, Blitzortung RED, GRLevel3, WD, WL, VWS, Cumulus, Meteobridge
Free weather PHP scripts/website templates - update notifications on Twitter saratogaWXPHP

Offline DrouinWeather

  • Member
  • *
  • Posts: 25
    • DrouinWeather
Re: WeatherUnderground changed again.. WU-forecast.php broken (again)
« Reply #1 on: October 30, 2014, 02:26:03 AM »

Oh I'm starting to really hate WU ...

Nigel Bryer (VK3GY) Drouin, Victoria, Australia www.drouinweather.com
Davis Vantage Pro2 Plus (Wireless) - Weather Display
Twitter Drouinweather

Offline Feilding Weather

  • Member
  • *
  • Posts: 10
    • Feilding Weather
Re: WeatherUnderground changed again.. WU-forecast.php broken (again)
« Reply #2 on: October 30, 2014, 02:34:10 AM »
Yup, just when everything's working well again.. Feel sorry for Ken!


Feilding Weather, Manawatu, New Zealand.
Visit our website: www.feildingweather.com
Find us on Facebook: www.facebook.com/Feildingweather

Offline marfanuk

  • Member
  • *
  • Posts: 28
  • Davis Vantage Vue/Meteobridge/Weather34
    • Nantwich Weather
Re: WeatherUnderground changed again.. WU-forecast.php broken (again)
« Reply #3 on: October 30, 2014, 06:20:43 AM »
Thats not really too fair of them, perhaps its now time to withhold to data from my weather station in revenge.

Offline George Richardson

  • WxElement panel
  • Forecaster
  • *****
  • Posts: 1391
    • Smith Mountain Lake Weather
Re: WeatherUnderground changed again.. WU-forecast.php broken (again)
« Reply #4 on: October 30, 2014, 10:24:23 AM »
"Thats not really too fair of them, perhaps its now time to withhold to data from my weather station in revenge"

Why don't you just stop paying them.

Offline wvdkuil

  • Wim van der kuil
  • Forecaster
  • *****
  • Posts: 1986
    • My PWS at Leuven Belgium Europe
Re: WeatherUnderground changed again.. WU-forecast.php broken (again)
« Reply #5 on: October 30, 2014, 11:19:40 AM »
Before this becomes a "withold my data from WU" we have to ask ourselves: Why are websites changing?
Not to anger the visitors but to give them a better "experience". So the changes at the WU sites are meant to let the visotrs to the WU sites have a quicker / better / more clear understanding of the data presented.

And we adapt our websites also to make it more worthwile to the visitors to come back to the site.

WU changed all of their sites and next to a modern look they also introduced far more javascript /HTML5 code. Result faster loading pages and easier maintenance.

As a side effect the weatherdata moved from html => embedded json => javascript loaded json. Which as a developer seems a quite natural grow-path for me.

They probably do not know that we are for years reading their "html" and taking the weatherdata from those pages and discarding the rest. For them our scripts seems to be a normal human visitor looking at their WU pages.
Instead we have to admit: We are behaving like a bot by doing this.. We get their forecast data and present is ourselves, thereby not giving them a change to earn money with the advertisements.

Atlernative:
In contrast to the html pages which are now obsolete due to improving the websites, WU has not changed the API in the past years.

So lets decide to modernize our sites and our scripts.
Get rid of the old: load html => cache the html page => find the data => use that data to display the information as we want it.
Change to: get data in machine readable form (json or xml)  =>  CACHE it => use that data to display the information as we want it.

The different Met.No and Yr.no scripts  are already doing that for years.

WXSIM scripts also process most of the data from CSV or processable txt files.
I myself skipped using the html for WU and   EC forcasts / Noaa and EC warnings  - more than a year ago.

The UV-forecast and meteoalarm.eu scripts can not be modernized, they need deciphering html as there is no xml or json feed for that data.

Wim

Offline saratogaWX

  • Administrator
  • Forecaster
  • *****
  • Posts: 9279
  • Saratoga, CA, USA Weather - free PHP scripts
    • Saratoga-Weather.org
Re: WeatherUnderground changed again.. WU-forecast.php broken (again)
« Reply #6 on: October 30, 2014, 03:02:43 PM »
I heartly agree with your summation and where we're headed.  WU likely did the change for the reason you cited (modern HTML5/AJAX just-in-time fetch of data) and not some malicious intent on their part.

I'd resisted using the API outright simply because it was a complication for folks to register to use the API and get an API Key to be able to access the data.  Kinda like the hassle of the old Google MAP key that was previously required before the V3 Google API was released.  WU has every right to offer and meter the data access, and the JSON method makes for a quick/predictable source for the info and no more dependency on scraping the data from an (ever changing) HTML page.

So, I'm about half-way through a replacement script using the API JSON data .. just have to figure out how much (or little) can be changed in the URLs folks had used for the customization to make the equivalent API calls and make sure the language functions work correctly.  I plan on setting the default cache life-time to 3 hours, which should be fast-enough for the general forecast changes by WU without tweaking an individual API user over the 500/day, 50/hr limits for the 'Developer' free API option.

Best regards,
Ken
Ken True/Saratoga, CA, USA main site: saratoga-weather.org
Davis VP1+ FARS, Blitzortung RED, GRLevel3, WD, WL, VWS, Cumulus, Meteobridge
Free weather PHP scripts/website templates - update notifications on Twitter saratogaWXPHP

Offline DW7240

  • Senior Contributor
  • ****
  • Posts: 225
    • The Vicarage Weather Feed
Re: WeatherUnderground changed again.. WU-forecast.php broken (again)
« Reply #7 on: October 30, 2014, 05:36:56 PM »
WOW,

Talk about "changes a foot", I agree with everything said so far........My gripe is along with the previous posting, if WU wants are data for free, then surely uping the limit on their API downloads seems to me a fair compromise.  After all, we are using their data for a purpose, we are not hackers, we don't exploit the data, either from WU or any other source, we (and I hope I speak for everyone) all use the correct acknowledgments as far as where the data has come from......namely using and displaying direct links back to the original source of the data.

I'm thinking down the road, as things progress, we will all eventually have to pay to get the data we require, this would be a shame in some respects and could result in killing off our interest.  Maybe we should all start to put our heads together on this one and come to a compromise which would leave everyone happy - before the decision is made for us and we regret not doing something about it now.  After all, WU has got the ball rolling, maybe now is the time to get one voice together, it's clear that WU knows that there are thousands of us literally sending data to them, yet they have steam-rolled us into a corner without so much as a "well we are proposing this, that  and the other.........you station owners can do this, that or whatever to obtain our data, etc, new guidelines will be implemented in the new year, whatever. 

Like always, big company's have big clout, but without our data, it leaves companies like WU with a hole.  NOAA is a prime example of how an organization uses us to get what they want, they do all the upgrade to their site, they let you know well in advance what they plan to do, also they keep the old parts of their sites running so as to give us the users the chance to upgrade and change over.......COME ON WU, lets be fair on this, we just need a little heads up, and more importantly..........A VOICE !!
« Last Edit: October 30, 2014, 05:39:43 PM by DW7240 »


Offline R_o_B

  • WxElement panel
  • Senior Member
  • *****
  • Posts: 85
Re: WeatherUnderground changed again.. WU-forecast.php broken (again)
« Reply #8 on: October 30, 2014, 10:52:30 PM »
Wim is right, many suppliers of data have moved to the API system.

As for most of you complaining that you freely give (upload) your data to Weather underground ... well, other than having to freely register an API account/plan with Weather Underground, the API service is free. Indeed, the 'Developer' plan (which I have been testing/playing around with for some time) is free - with that free 'Developer' plan, I am allowed 10 calls/clicks (HTTP connection to the API) per minute for a total of 500 calls/clicks (HTTP connection to the API) per day (24 hours based on US EST) - that is way more calls/click (connections) then the setting I have in the 'WU-forecast.php' script!

In the 'WU-forecast.php' script, the default setting of the refetch/refresh action/cycle/operation is set to 1800 seconds (or 30 minutes) - on a 24-hour period, that totals 48 contacts to download and hack the Weather Underground web page.

Quote
I'd resisted using the API outright simply because it was a complication for folks to register to use the API and get an API Key to be able to access the data.
Just curious, Ken... Are the weather station operators not required to create a free account with Weather Underground before being able to upload the data from their weather station and freely enjoy the many services provided by Weather Underground... So how different is it now... The weather station operators need to create a free account in order to access the API and download the data for their area!
R_o_B
---
eMail: weather@herray.net

Offline saratogaWX

  • Administrator
  • Forecaster
  • *****
  • Posts: 9279
  • Saratoga, CA, USA Weather - free PHP scripts
    • Saratoga-Weather.org
Re: WeatherUnderground changed again.. WU-forecast.php broken (again)
« Reply #9 on: October 31, 2014, 01:28:45 AM »
Hi Ray,

You're correct .. to upload data to WeatherUnderground, you have to create an account, and that allows ad-free access to the website, and the ability to upload weather data, webcam images, photos, and NOAA radio streams.  The API access just requires one additional step -- apply for a 'Developer' API key and that allows you access to the API with some limits (which should not be exceeded for hobbyist weather websites using caching scripts).  So, one extra step and it's done.

In general, as APIs are offered from sites, I tend to use them (over the prior page-scraping option) for two reasons:
1) some relief from the page tweaking requiring script changes
2) overall easier programming since the data is already in a useable format.

Most recently, the USGS had offered JSON feeds, so my quake-json.php script replaced several page-scraper scripts with
one world-functional script.  The WU-forecast.php V3.00 will do the same for WU, and be variable compatible, so the script should just 'plug-in' to existing sites, and only require the addition of the WU API Key in Settings.php for full operation.

As Wim said, not all the sites offer fully-functional APIs .. the EC offers an RSS feed, but it's sadly lacking icon information, so the ec-forecast.php script remains a page-scraper.  Likewise for the ec-radar.php, and the MeteoAlarm script.

I will be converting page-scrapers to API user scripts as the APIs become available.. just makes my (future) life less hectic as APIs are slow to change once established, while website look/feel is ever in flux.

Best regards,
Ken
« Last Edit: October 31, 2014, 01:31:09 AM by saratogaWX »
Ken True/Saratoga, CA, USA main site: saratoga-weather.org
Davis VP1+ FARS, Blitzortung RED, GRLevel3, WD, WL, VWS, Cumulus, Meteobridge
Free weather PHP scripts/website templates - update notifications on Twitter saratogaWXPHP

Offline Andy G

  • Forecaster
  • *****
  • Posts: 306
    • SunPrairieWeather.com
Re: WeatherUnderground changed again.. WU-forecast.php broken (again)
« Reply #10 on: October 31, 2014, 07:56:38 AM »
So I am guessing I can get my API key now so I am ready when Ver 3 is released? Thank you for all you efforts.

Andy
Once you can accept the universe as being something expanding into an infinite nothing which is something, wearing stripes with plaid is easy.
~Einstein


SunPrairieWeather.com
Davis VP2 24 FARS

Offline CNYWeather

  • Forecaster
  • *****
  • Posts: 2297
    • CNYWeather
Re: WeatherUnderground changed again.. WU-forecast.php broken (again)
« Reply #11 on: October 31, 2014, 12:32:06 PM »
The version 3 of the WU forecast is for the international users of the script I believe Andy. Ours is still coming from the NWS.
Tony




Offline Andy G

  • Forecaster
  • *****
  • Posts: 306
    • SunPrairieWeather.com
Re: WeatherUnderground changed again.. WU-forecast.php broken (again)
« Reply #12 on: October 31, 2014, 12:37:23 PM »
The version 3 of the WU forecast is for the international users of the script I believe Andy. Ours is still coming from the NWS.

Thanks
Once you can accept the universe as being something expanding into an infinite nothing which is something, wearing stripes with plaid is easy.
~Einstein


SunPrairieWeather.com
Davis VP2 24 FARS

Offline Feilding Weather

  • Member
  • *
  • Posts: 10
    • Feilding Weather
Re: WeatherUnderground changed again.. WU-forecast.php broken (again)
« Reply #13 on: October 31, 2014, 11:30:59 PM »
Thanks for the updated script Ken, just one problem since installing the script it replaces all the cunulus $temp webtags with ARRAY. ie: Array °C. can you suggest a fix for this please?


I heartly agree with your summation and where we're headed.  WU likely did the change for the reason you cited (modern HTML5/AJAX just-in-time fetch of data) and not some malicious intent on their part.

I'd resisted using the API outright simply because it was a complication for folks to register to use the API and get an API Key to be able to access the data.  Kinda like the hassle of the old Google MAP key that was previously required before the V3 Google API was released.  WU has every right to offer and meter the data access, and the JSON method makes for a quick/predictable source for the info and no more dependency on scraping the data from an (ever changing) HTML page.

So, I'm about half-way through a replacement script using the API JSON data .. just have to figure out how much (or little) can be changed in the URLs folks had used for the customization to make the equivalent API calls and make sure the language functions work correctly.  I plan on setting the default cache life-time to 3 hours, which should be fast-enough for the general forecast changes by WU without tweaking an individual API user over the 500/day, 50/hr limits for the 'Developer' free API option.

Best regards,
Ken


Feilding Weather, Manawatu, New Zealand.
Visit our website: www.feildingweather.com
Find us on Facebook: www.facebook.com/Feildingweather

Offline saratogaWX

  • Administrator
  • Forecaster
  • *****
  • Posts: 9279
  • Saratoga, CA, USA Weather - free PHP scripts
    • Saratoga-Weather.org
Re: WeatherUnderground changed again.. WU-forecast.php broken (again)
« Reply #14 on: October 31, 2014, 11:56:24 PM »
I'm not sure I understand your concern.  The $WUforecasttemp[] array always contained both the value and <span> elements to make red/blue text.  All that was added was '&deg;C' (or '&deg;F' for imperial).

The $temp variables were used as 'temporary' items as part of the old page-scraper parser and were not intended to be 'published'.. they're just work variables and were removed as unneeded code when the JSON API code was used.

If all you want is a number (for the temperature), then something like
Code: [Select]
   $thetemp = preg_replace('|[^\d]|i','',strip_tags($WUforecasttemp[$n]));
would strip off the HTML and extra unit designator and just leave the number itself.

Is that what you wanted?

Best regards,
Ken
Ken True/Saratoga, CA, USA main site: saratoga-weather.org
Davis VP1+ FARS, Blitzortung RED, GRLevel3, WD, WL, VWS, Cumulus, Meteobridge
Free weather PHP scripts/website templates - update notifications on Twitter saratogaWXPHP

Offline Feilding Weather

  • Member
  • *
  • Posts: 10
    • Feilding Weather
Re: WeatherUnderground changed again.. WU-forecast.php broken (again)
« Reply #15 on: November 01, 2014, 12:07:06 AM »
Thanks Ken, Have fixed the problem by changing the $temp to $temp1 on these 2 lines

    $temp1 = explode('.',$WUforecasttext[$n]); // split as sentences (sort of).
   
    $WUforecastcond[$n] = trim($temp1[0]); // take first one as summary.

I'm not sure I understand your concern.  The $WUforecasttemp[] array always contained both the value and <span> elements to make red/blue text.  All that was added was '&deg;C' (or '&deg;F' for imperial).

The $temp variables were used as 'temporary' items as part of the old page-scraper parser and were not intended to be 'published'.. they're just work variables and were removed as unneeded code when the JSON API code was used.

If all you want is a number (for the temperature), then something like
Code: [Select]
   $thetemp = preg_replace('|[^\d]|i','',strip_tags($WUforecasttemp[$n]));
would strip off the HTML and extra unit designator and just leave the number itself.

Is that what you wanted?

Best regards,
Ken


Feilding Weather, Manawatu, New Zealand.
Visit our website: www.feildingweather.com
Find us on Facebook: www.facebook.com/Feildingweather

Offline iceman62

  • Member
  • *
  • Posts: 10
    • Météo - Achiet le Grand
Re: WeatherUnderground changed again.. WU-forecast.php broken (again)
« Reply #16 on: November 01, 2014, 02:11:21 AM »
Hello Ken
Once again, thank you for this beautiful work.
However, I have an issue with the new version that will not be translated into French.

In setting.php, I have put $ SITE ['lang'] = 'fr';
In WU-forecast.php $ lang = 'fr'; // Default language

Nothing works. My page is still in English.

http://meteo-achiet.shost.ca/wxforecast.php?z=0

What I have not configured properly?

Offline Kylo

  • Member
  • *
  • Posts: 32
    • Private Wetterstation Braunschweig
Re: WeatherUnderground changed again.. WU-forecast.php broken (again)
« Reply #17 on: November 01, 2014, 07:27:44 AM »
Thanks for your work , Ken !!  =D&gt; =D&gt;

Quote
However, I have an issue with the new version that will not be translated into French.

In setting.php, I have put $ SITE ['lang'] = 'fr';
In WU-forecast.php $ lang = 'fr'; // Default language

Edit line 312 in : $WULANG = 'FR';
 
VP2 Pro Plus Aktiv, TOA Blitzortung System RED, NOAA WX-Sat, Weather Display


Offline Guillermo Villan

  • Ing. Eléctrical
  • Senior Member
  • **
  • Posts: 53
    • WXElortondo - Estación Meteorológica Automática de Elortondo
Re: WeatherUnderground changed again.. WU-forecast.php broken (again)
« Reply #18 on: November 01, 2014, 08:17:18 AM »
Ken, good day, where I set the language?
thank you very much again !!
wxelortondo.com.ar

Offline iceman62

  • Member
  • *
  • Posts: 10
    • Météo - Achiet le Grand
Re: WeatherUnderground changed again.. WU-forecast.php broken (again)
« Reply #19 on: November 01, 2014, 09:07:06 AM »
Thanks for your work , Ken !!  =D&gt; =D&gt;

Quote
However, I have an issue with the new version that will not be translated into French.

In setting.php, I have put $ SITE ['lang'] = 'fr';
In WU-forecast.php $ lang = 'fr'; // Default language

Edit line 312 in : $WULANG = 'FR';

Thank you Kylo, but it does not work. :p

Offline wvdkuil

  • Wim van der kuil
  • Forecaster
  • *****
  • Posts: 1986
    • My PWS at Leuven Belgium Europe
Re: WeatherUnderground changed again.. WU-forecast.php broken (again)
« Reply #20 on: November 01, 2014, 10:02:28 AM »
Thanks for your work , Ken !!  =D&gt; =D&gt;

Quote
However, I have an issue with the new version that will not be translated into French.

In setting.php, I have put $ SITE ['lang'] = 'fr';
In WU-forecast.php $ lang = 'fr'; // Default language

Edit line 312 in : $WULANG = 'FR';

Thank you Kylo, but it does not work. :p
http://meteo-achiet.shost.ca/wxforecast.php?z=0 is in french now.

You have to wait until the cache expires. Probably the information was read from the cache so  it did not change immidiatly.

Wim

Offline saratogaWX

  • Administrator
  • Forecaster
  • *****
  • Posts: 9279
  • Saratoga, CA, USA Weather - free PHP scripts
    • Saratoga-Weather.org
Re: WeatherUnderground changed again.. WU-forecast.php broken (again)
« Reply #21 on: November 01, 2014, 10:07:33 AM »
I found that I had omitted the logic for standalone use non-English language default selection ..

The V3.01 script is now available, and if $lang= is set in the script, that will be the default language for all forecasts.

As Wim says, after the update of the script, the cache may still hold the old English .. add ?force=1 to the URL to cause the cache to refresh with the new default language.

Best regards,
Ken
Ken True/Saratoga, CA, USA main site: saratoga-weather.org
Davis VP1+ FARS, Blitzortung RED, GRLevel3, WD, WL, VWS, Cumulus, Meteobridge
Free weather PHP scripts/website templates - update notifications on Twitter saratogaWXPHP

Offline iceman62

  • Member
  • *
  • Posts: 10
    • Météo - Achiet le Grand
Re: WeatherUnderground changed again.. WU-forecast.php broken (again)
« Reply #22 on: November 01, 2014, 10:25:05 AM »
Thank you ken. Congratulations for all this work!

Offline iceman62

  • Member
  • *
  • Posts: 10
    • Météo - Achiet le Grand
Re: WeatherUnderground changed again.. WU-forecast.php broken (again)
« Reply #23 on: November 01, 2014, 10:50:10 AM »
Another question Ken, What should I change to have "WeatherUnderground 7-day Forecast: Achiet-le-Grand" in French?

Offline R_o_B

  • WxElement panel
  • Senior Member
  • *****
  • Posts: 85
Re: WeatherUnderground changed again.. WU-forecast.php broken (again)
« Reply #24 on: November 01, 2014, 05:23:17 PM »
The version 3 of the WU forecast is for the international users of the script I believe Andy. Ours is still coming from the NWS.
Wrong!  ;)
The 'WU-forecast.php' script can be used for (almost) all locations, including the U.S.A. and Canada.  8-)
R_o_B
---
eMail: weather@herray.net