Topic: ec-forecast.php script -- no icons -- due to major EC website changes

[saratogaWX]

I'm not sure why the programmers for Environment Canada always seem to pick a time for major site changes just while I'm on vacation, but they've done it again.

So, sorry the script is broken.. I'll work on it over the next couple of days as time permits.

[weathergirl]

Thanks, Ken.

[WpgWeather]

Given that it's a public service it would be nice if they would post their release schedule in advance.
Greatly appreciate the work on the scripts.

[Jáchym]

I have to say that EC and NWS are experts on this.

I have been working on Meteotemplate since Jul 2015. Since then I had to update some EC and NWS scripts more than 3 times. Never happened with any other script. In some cases the changes just make no sense, when it is a change from http to https or similar I can understand it, but this?
It almost seems to me they are doing it for this very reason, waiting for people like me and other devs to simply give up and people having to visit their sites. With NWS it is probably even worse. And when they at least announce the changes it is the case that the final date is post-poned several times and the final changes dont always reflect what was announced...

Oh well...

[Jáchym]

version history of this particular script...

         #      v2.0 - Oct 21, 2016
   #         - rewritten code due to changed EC forecast webpage
   #      v2.1 - Nov 28, 2016
   #         - changes due to changes in source format
   #       v2.2 - Aug 30, 2017
   #          - changes due to major changes at EC website

and still no https

[WpgWeather]

I wish I was a programmer and I might tackle making a script off their XML. Apparently it hasn't changed.  This guy made a Android app off it with similiar results: https://github.com/paleolimbot/CanadaWeather/blob/master/canadaWeather/src/main/java/ca/fwe/caweather/backend/CityPageForecastParser.java

[saratogaWX]

Alas, I'd use the XML feed too, but..

Code: [Select]

<entry>
 <title>Tuesday night: A few clouds. Low 14.</title>
 <link type="text/html" href="http://www.weather.gc.ca/city/pages/on-107_metric_e.html"/>
 <updated>2017-08-29T19:30:00Z</updated>
 <published>2017-08-29T19:30:00Z</published>
 <category term="Weather Forecasts"/>
 <summary type="html">A few clouds. Wind northeast 20 km/h becoming light this evening. Low 14. Forecast issued 3:30 PM EDT Tuesday 29 August 2017</summary>
 <id>tag:weather.gc.ca,2013-04-16:on-107_fc1:20170829193000</id>
 </entry> they don't include Icon data in the feed

[saratogaWX]

BTW, the major change is that they now provide icons in 12hr intervals instead of just by day as in the past.  They now do the formatting using nested <div> with class= attributes to provide 'cells' in virtual tables.

Grrrr...

I'm working on an update as time permits -- I'm using VNC to access the development system at home while the wife and I are away visiting the grandkids.

[Bashy]

I'm working on an update as time permits -- I'm using VNC to access the development system at home while the wife and I are away visiting the grandkids.

Surely it can wait, everyone needs some downtime....

[wvdkuil]

Alas, I'd use the XML feed too, but..

Code: [Select]

<entry>
 <title>Tuesday night: A few clouds. Low 14.</title>
 <link type="text/html" href="http://www.weather.gc.ca/city/pages/on-107_metric_e.html"/>
 <updated>2017-08-29T19:30:00Z</updated>
 <published>2017-08-29T19:30:00Z</published>
 <category term="Weather Forecasts"/>
 <summary type="html">A few clouds. Wind northeast 20 km/h becoming light this evening. Low 14. Forecast issued 3:30 PM EDT Tuesday 29 August 2017</summary>
 <id>tag:weather.gc.ca,2013-04-16:on-107_fc1:20170829193000</id>
 </entry> they don't include Icon data in the feed

I am not familiair with the xml you are referring to. But there are multiple script writers using different files (html or xml) to get the forecast. Maybe we should standardise and make 1 (one) retrieval script.

I myself use  the "official" data-source for all EC city forecasts:  http://dd.meteo.gc.ca/citypage_weather/docs/README_citypage_weather.txt

The http://dd.weather.gc.ca/citypage_weather/xml/ directory contains the same XML information used to generate each city and 24h forecasts page on the Environment and Climate Change Canada web site http://www.weather.gc.ca/

Retrieving the data for Pickering (Ontario) the URL (either http or https) would be:
https://dd.meteo.gc.ca/citypage_weather/xml/ON/s0000710_e.xml

That xml contains  hourly forecasts and  12 hour forecasts including icon-number

Code: [Select]

<forecast>
<period textForecastName="Tonight">Tuesday night</period>
<textSummary>Partly cloudy. Clearing this evening. Low 12.</textSummary>
<cloudPrecip><textSummary>Partly cloudy. Clearing this evening.</textSummary></cloudPrecip>
<abbreviatedForecast>
<iconCode format="gif">31</iconCode>
<pop units="%"/>
<textSummary>A few clouds</textSummary>
</abbreviatedForecast>
<temperatures>
<textSummary>Low 12.</textSummary><temperature unitType="metric" units="C" class="low">12</temperature>
</temperatures>
<winds/>
<precipitation>
<textSummary/>
<precipType start="" end=""/>
</precipitation>
<relativeHumidity units="%">75</relativeHumidity>
</forecast>warnings

Code: [Select]

<event type="warning" priority="low" description="WIND WARNING  IN EFFECT">
 <dateTime name="eventIssue" zone="UTC" UTCOffset="0">
 <year>2013</year>
 <month name="September">09</month>
 <day name="Sunday">29</day>
 <hour>17</hour>
 <minute>55</minute>
 <timeStamp>20130929175500</timeStamp>
 <textSummary>Sunday September 29, 2013 at 17:55 UTC</textSummary>
 </dateTime>
 <dateTime name="eventIssue" zone="PDT" UTCOffset="-7">
 <year>2013</year>
 <month name="September">09</month>
 <day name="Sunday">29</day>
 <hour>10</hour>
 <minute>55</minute>
 <timeStamp>20130929105500</timeStamp>
 <textSummary>Sunday September 29, 2013 at 10:55 PDT</textSummary>
 </dateTime>
 </event> and multiple sets of  almanac data, regional normals and also current conditions.

The only minor problem:  the user of the script has to select the area, such as S000710 code from a long list of codes. As those files are in CSV format, I had to write a small load/display utility to let the user select a correct area-code.

Lists with location names and and SiteNameCodes, regrouped according to
various factors, can be found at the following addresses:
http://dd.weather.gc.ca/citypage_weather/docs/site_list_en.csv
http://dd.weather.gc.ca/citypage_weather/docs/site_list_provinces_en.csv
http://dd.weather.gc.ca/citypage_weather/docs/site_list_towns_en.csv

Although EC changed the city-html pages multiple times since 2013 when I wrote my version of the  EC-forecast script, the data it uses did not change.
An example of its use can be found at http://komokaweather.com/weather28/index.php?p=ec_print_fct&lang=en#data-area

Wim

[Jáchym]

I already implemented the changes yesterday, I need to release the update today. They changed the table to div elements. It seems to me a bit like they are doing it for the very reason to make us (the devs) to give up.

Not that NWS would be different though, but at least they announce it beforehand (and then postpone it 5 times )

[saratogaWX]

Thanks for the details Wim!  the XML link I'd tried was the subset available on the EC's forecast page itself-- and it was lacking the icon info.

I think I'll bite the bullet and do a conversion to the XML feed too.. now to use the current forecast page URLs with lookups to the XML file needed.  I wouldn't want for folks to have to convert all their current URLs without some automation behind that..

[saratogaWX]

ec-forecast.php V4.03 is now available to fix the icon display issues with the EC website changes on 29-Aug-2017.

Please note that this is a quick fix job (since I'm on vacation), so some functionality is not implemented:
1) the '*' for Abnormal temperature trends is not working
2) only the long Day is displayed for the icons (short day or day Mon is not implemented)

Saratoga template users: download using the update tool with a query for Base-Canada, Plugin-*, 31-Aug-2017

Standalone users: download from script page

After the vacation, I'll do a rewrite of the script to use the XML feeds from the EC which (as Wim notes) is MUCH more stable than the existing page-scraper.

Best regards,
Ken

[WpgWeather]

Working perfectly, thanks Ken!

- Dale

[Simnm]

Thanks

[weathergirl]

Looks great, Ken, thanks!