Author Topic: Anyone know of a php script that can capture a WU page?  (Read 13047 times)

0 Members and 1 Guest are viewing this topic.

Offline anchorageweather

  • Forecaster
  • *****
  • Posts: 445
    • http://eetee.us/station/station.php
Anyone know of a php script that can capture a WU page?
« on: September 24, 2006, 02:30:48 PM »
I was thinking it might be valuable to add the almanac data that Weather Underground keeps about my site at:

http://www.wunderground.com/weatherstation/WXDailyHistory.asp?ID=KKYLOUIS20

to my site.  However, their page is a little cluttered, and I was hoping to just use the weather data by itself.

Does anyone know of a php script that could be used to capture the daily, weekly, monthy, and yearly data of this URL?
South of the Tracks, Anchorage, KY

Offline saratogaWX

  • Administrator
  • Forecaster
  • *****
  • Posts: 9257
  • Saratoga, CA, USA Weather - free PHP scripts
    • Saratoga-Weather.org
Anyone know of a php script that can capture a WU page?
« Reply #1 on: September 24, 2006, 03:26:13 PM »
I did a little code to get just today's graph for my rapid-fire page ( http://saratoga-weather.org/rapid-fire.php ).  It only adjusts the WU call for the graphs, but you might find it useful:

Code: [Select]
<?php 
   
//Set the Timezone
 
putenv&#40;"TZ=US/Pacific"&#41;;
   
$now getdate&#40;&#41;;
   
$mon $now['mon'&#93;;
   
$year $now['year'&#93;;
   
$day  $now['mday'&#93;;
?>

<p><img src="http://www.weatherunderground.com/cgi-bin/wxStationGraphAll?day=<?php echo $day?>&amp;year=<?php echo $year?>&amp;month=<?php echo $mon?>&amp;ID=KCASARAT1&amp;type=3" width="600" height="645" />
</p>
Ken True/Saratoga, CA, USA main site: saratoga-weather.org
Davis VP1+ FARS, Blitzortung RED, GRLevel3, WD, WL, VWS, Cumulus, Meteobridge
Free weather PHP scripts/website templates - update notifications on Twitter saratogaWXPHP

Offline anchorageweather

  • Forecaster
  • *****
  • Posts: 445
    • http://eetee.us/station/station.php
Anyone know of a php script that can capture a WU page?
« Reply #2 on: September 24, 2006, 05:21:21 PM »
Quote from: "kenmtrue"
I did a little code to get just today's graph for my rapid-fire page ( http://saratoga-weather.org/rapid-fire.php ).  It only adjusts the WU call for the graphs, but you might find it useful:

Code: [Select]
<?php 
   
//Set the Timezone
 
putenv&#40;"TZ=US/Pacific"&#41;;
   
$now getdate&#40;&#41;;
   
$mon $now['mon'&#93;;
   
$year $now['year'&#93;;
   
$day  $now['mday'&#93;;
?>

<p><img src="http://www.weatherunderground.com/cgi-bin/wxStationGraphAll?day=<?php echo $day?>&amp;year=<?php echo $year?>&amp;month=<?php echo $mon?>&amp;ID=KCASARAT1&amp;type=3" width="600" height="645" />
</p>


Wow, that is very handy!  How did you ever find the cgi file to call?  I wonder if there is a file that will call the "Yearly Tabular Data"?

Heres what I have so far:

http://eetee.us/station/wugraphs.php
South of the Tracks, Anchorage, KY

Offline saratogaWX

  • Administrator
  • Forecaster
  • *****
  • Posts: 9257
  • Saratoga, CA, USA Weather - free PHP scripts
    • Saratoga-Weather.org
Anyone know of a php script that can capture a WU page?
« Reply #3 on: September 24, 2006, 06:51:54 PM »
Quote
How did you ever find the cgi file to call?

For that, 'view-source' is your friend.. <img> tags have to refer to the source of a graphic, and they used a CGI to generate it, so it was fairly easy to see what parameters they used on the call.  I see you've modified the call to do the last 6 months for your page.. way to go! :-)

It doesn't look obvious how they build the daily/monthly summary in text form however, except by calling the /weatherstation/WXDailyHistory.asp?... page with parms to generate the whole thing.

Ken
Ken True/Saratoga, CA, USA main site: saratoga-weather.org
Davis VP1+ FARS, Blitzortung RED, GRLevel3, WD, WL, VWS, Cumulus, Meteobridge
Free weather PHP scripts/website templates - update notifications on Twitter saratogaWXPHP

Offline anchorageweather

  • Forecaster
  • *****
  • Posts: 445
    • http://eetee.us/station/station.php
Anyone know of a php script that can capture a WU page?
« Reply #4 on: September 24, 2006, 10:19:01 PM »
Quote from: "kenmtrue"
For that, 'view-source' is your friend.. <img> tags have to refer to the source of a graphic, and they used a CGI to generate it, so it was fairly easy to see what parameters they used on the call.  I see you've modified the call to do the last 6 months for your page.. way to go! :-)

It doesn't look obvious how they build the daily/monthly summary in text form however, except by calling the /weatherstation/WXDailyHistory.asp?... page with parms to generate the whole thing.

Ken


Ahh, yes - I see it now.  You are correct, no file to call for the tables.  I think it might be possible to cache the file and cut it up (like Carter Lake's NOAA forecast script).  I guess I can try to modify that script; however, I must admit - I am a PHP NOVICE
South of the Tracks, Anchorage, KY

Offline SLOweather

  • Global Moderator
  • Forecaster
  • *****
  • Posts: 3456
    • Weatherelement Moline IL
Anyone know of a php script that can capture a WU page?
« Reply #5 on: September 24, 2006, 10:54:07 PM »
Quote from: "anchorageweather"
I guess I can try to modify that script; however, I must admit - I am a PHP NOVICE


That's how we learn... :)

Offline anchorageweather

  • Forecaster
  • *****
  • Posts: 445
    • http://eetee.us/station/station.php
Anyone know of a php script that can capture a WU page?
« Reply #6 on: September 25, 2006, 11:17:03 AM »
Quote from: "SLOweather"
Quote from: "anchorageweather"
I guess I can try to modify that script; however, I must admit - I am a PHP NOVICE


That's how we learn... :)


Hello All:

Well I must say, I'm kind of embarrassed to show my feeble efforts compared to the complex scripts I've found on this board; however, here is what I have so for:

http://eetee.us/station/wutable.php

Is there a way to redirect the daily links in this table to the WU page for that day? I was thinking there might be a way to use my .htaccess file (since the links are being directed to MY site). However, the links are dynamic and change for each day (however, they ARE all going to the same unused directory on my site). Can anyone help with this one?

EDIT:

Ok, I got the redirect to work as a TEMPORARY fix; however, I'd like to get the daily link to somehow go to a daily graph page (like the one  kenmtrue helped me make above):

http://eetee.us/station/wutable.php

Is there a way to do that without creating 365 pages? (one for each day of the year) :(
South of the Tracks, Anchorage, KY

Offline anchorageweather

  • Forecaster
  • *****
  • Posts: 445
    • http://eetee.us/station/station.php
Anyone know of a php script that can capture a WU page?
« Reply #7 on: September 25, 2006, 04:00:27 PM »
Ok, I've gotten a little farther;  Looking at kenmtrue's code for the daily graph:

Code: [Select]
<?php 
   
//Set the Timezone 
 
putenv&#40;"TZ=US/Eastern"&#41;; 
   
$now getdate&#40;&#41;; 
   
$mon $now['mon'&#93;; 
   
$year $now['year'&#93;; 
   
$day  $now['mday'&#93;; 
?>


<html>
<body bgcolor="#FFFFAA" link="#0000FF" vlink="#0000FF" alink="#0000FF">

<CENTER><p><img src="http://www.weatherunderground.com/cgi-bin/wxStationGraphAll?day=<?php echo $day?>&amp;year=<?php echo $year?>&amp;month=<?php echo $mon?>&amp;ID=KKYLOUIS20&amp;type=5" width="600" height="645" />
</p></CENTER>



</body>
<html>


it looks like the parameters to call the graph image are the same  as the parameters to call the corresponding table page on WU's site -  year=2004, month=4, day=13
ie:

http://www.wunderground.com/weatherstation/WXDailyHistory.asp?ID=KKYLOUIS20&year=2006&month=9&day=8


  If I could write a php script (like Saratoga-Weather.org's earthquake script):

http://eetee.us/station/quake-USA-test.php?magnitude=1.0&distance=500

where you can specify the parameters in the URL, I could redirect the WXDailyHistory.asp file to my php file, on my site, that uses the suppled parameters, and it should display the graph for that day.  Something like:

http://eetee.us/weatherstation/YOUPICKTHEDAYSCRIPT.php?year=2006&month=9&day=8

That php script, however, is WAY over my head - currently.  Can anyone give me a couple of hints to get started, or does anyone know where I might be able to go on the web to learn how to do it?
South of the Tracks, Anchorage, KY

Offline kray1000

  • Purveyor of wry
  • Forecaster
  • *****
  • Posts: 1336
    • http://www.roanokevalleyweather.com
Anyone know of a php script that can capture a WU page?
« Reply #8 on: September 25, 2006, 04:39:21 PM »
Quote from: "anchorageweather"
That php script, however, is WAY over my head - currently.  Can anyone give me a couple of hints to get started, or does anyone know where I might be able to go on the web to learn how to do it?


Here's one place...
http://www.tizag.com/

Offline saratogaWX

  • Administrator
  • Forecaster
  • *****
  • Posts: 9257
  • Saratoga, CA, USA Weather - free PHP scripts
    • Saratoga-Weather.org
Anyone know of a php script that can capture a WU page?
« Reply #9 on: September 25, 2006, 05:00:43 PM »
Don't underestimate your own talent to learn.. you've done a great job with the script already!

I only started to learn PHP programming in March of this year.. I admit to being fairly expert with Perl (having done programming in it for over 10 years), but PHP syntax is just a bit different, so it has a learning curve too.

I learn best by seeing and analyzing code others have written, and for that start, I used the Tom's (carterlake) forecast script and Larry's (anole) sticker script.  Feel free to bag and reuse any of my code from my scripts page ( http://saratoga-weather.org/scripts.php ) in your own quest :-)

Now.. you're on the right track.. you already have a script that will do the scraping of the WU page for the data for 6 months.. all you have to add (I think) is a bit a parameter handling and a simple preg_replace to subsitute your page for the /weatherstation/WXDailyHistory.asp page.

Parameters can be handled like this (at the top of your page):
Code: [Select]
if (isset($_REQUEST['month')) { $mon = $_REQUEST['month']; }
if (isset($_REQUEST['year')) { $year = $_REQUEST['year']; }
if (isset($_REQUEST['day')) { $day = $_REQUEST['day']; }

   //Set the Timezone
 putenv("TZ=US/Pacific");
   $now = getdate();
// set the day/month/year to current if not passed as parms
if(!$mon) { $mon = $now['mon'];}
if(!$year) { $year = $now['year'];}
if(!$day)  { $day  = $now['mday'];}


Then, where you're scraping the lines of the page, just include this statement to change the URL references:
Code: [Select]
$new = preg_replace('|/weatherstation/WXDailyHistory.asp|Uis','/station/wutable.php ',$line);

Since I can't see your PHP code, I've assumed the line your looking for is in $line and the replacement will be placed in $new.. you'll have to change the variable names to match the ones you're using to make it work, but I hope you get the picture.

This way, if /station/wutable.php is invoked without parameters, it will do the default of looking up using today's date.  The internal links returned on the page will be rewritten to use your page, and the parameters will be passed through you and down to the URL for WeatherUnderground in your code.

Hope this helps...

Best regards,
Ken
Ken True/Saratoga, CA, USA main site: saratoga-weather.org
Davis VP1+ FARS, Blitzortung RED, GRLevel3, WD, WL, VWS, Cumulus, Meteobridge
Free weather PHP scripts/website templates - update notifications on Twitter saratogaWXPHP

Offline anchorageweather

  • Forecaster
  • *****
  • Posts: 445
    • http://eetee.us/station/station.php
Anyone know of a php script that can capture a WU page?
« Reply #10 on: September 26, 2006, 12:24:56 AM »
I think I got it :D   If you click here:

http://www.eetee.us/station/wumonthlytable.php

or here:

http://www.eetee.us/station/wutable.php

from my website and then click on any day, it opens a graph for that day!!  you can also open the php without the URL parameter and it shows todays graph.

kenmtrue, THANKS for taking the time to help me through it.  I learned alot!
South of the Tracks, Anchorage, KY

Offline saratogaWX

  • Administrator
  • Forecaster
  • *****
  • Posts: 9257
  • Saratoga, CA, USA Weather - free PHP scripts
    • Saratoga-Weather.org
Anyone know of a php script that can capture a WU page?
« Reply #11 on: September 26, 2006, 01:30:50 AM »
Congratulations, and welcome to the ranks of the page scrapers   :lol:

You're very welcome for whatever minor assistance I provided.. keep on trying and learning, it's very fun.

Best regards,
Ken
Ken True/Saratoga, CA, USA main site: saratoga-weather.org
Davis VP1+ FARS, Blitzortung RED, GRLevel3, WD, WL, VWS, Cumulus, Meteobridge
Free weather PHP scripts/website templates - update notifications on Twitter saratogaWXPHP

Offline katlon

  • LONNIE'S WEATHER STATION
  • Contributor
  • ***
  • Posts: 142
  • GO HEELS!!!
    • LONNIE'S WEATHER STATION
Anyone know of a php script that can capture a WU page?
« Reply #12 on: September 26, 2006, 06:49:15 AM »
anchorageweather

Could you post your script for the tabular data. That looks really nice.
katlon

Offline anchorageweather

  • Forecaster
  • *****
  • Posts: 445
    • http://eetee.us/station/station.php
Anyone know of a php script that can capture a WU page?
« Reply #13 on: September 26, 2006, 10:04:36 AM »
Quote from: "katlon"
anchorageweather

Could you post your script for the tabular data. That looks really nice.
katlon


Not much to it really:

Code: [Select]

<?php

$html 
implode&#40;'', file&#40;'http&#58;//www.wunderground.com/weatherstation/WXDailyHistory.asp?ID=STATIONID&graphspan=year'&#41;&#41;;

     

      
preg_match&#40;'|<h3>Yearly Tabular Data</h3>&#40;.*&#41;<form action="/weatherstation/WXDailyHistory.asp"|s', $html, $betweenspan&#41;; 
      
$dailytable  $betweenspan[1&#93;; 

 //Set the Timezone 
 
putenv&#40;"TZ=US/Eastern"&#41;; 
   
$now getdate&#40;&#41;; 
   
$mon $now['mon'&#93;; 
   
$year $now['year'&#93;; 
   
$day  $now['mday'&#93;; 


     
?>


<html>
<head>
</head>
<body>





<CENTER>

<table border="0" align="center" cellpadding="0" cellspacing="0">

<tr valign ="top" align="center">
<td>
<font face="verdana,arial" size="1">
<h3>Yearly Tabular Data for <?echo $year?></h3>
</td>
</tr>



<tr valign ="top" align="center">
<td>
<font face="verdana,arial" size="1">
<?echo $dailytable?>
</td>
</tr>
</table>



</CENTER>


</body>
</html>


You will need to change the station ID.  Hope it helps. :)

P.S.  Now that I'm such a coding fool - I added an advisory status line in my sticker :wink:
South of the Tracks, Anchorage, KY

Offline katlon

  • LONNIE'S WEATHER STATION
  • Contributor
  • ***
  • Posts: 142
  • GO HEELS!!!
    • LONNIE'S WEATHER STATION
Anyone know of a php script that can capture a WU page?
« Reply #14 on: September 26, 2006, 12:33:53 PM »
Thanks! I got that part but how do you get the individual days to point to daily graphs?

katlon

Offline anchorageweather

  • Forecaster
  • *****
  • Posts: 445
    • http://eetee.us/station/station.php
Anyone know of a php script that can capture a WU page?
« Reply #15 on: September 26, 2006, 12:59:37 PM »
Quote from: "katlon"
Thanks! I got that part but how do you get the individual days to point to daily graphs?

katlon


I used my .htaccess file to redirect the link that the script was generating:

http://eetee.us/weatherstation/WXDailyHistory.asp?ID=STATIONID&year=2006&month=9&day=6

The link, once redirected, goes to a file on my site called wugraphs.php that uses this code:

Code: [Select]
<?php 

if &#40;isset&#40;$_REQUEST['month'&#93;&#41;&#41; &#123; $mon = $_REQUEST['month'&#93;; &#125; 
if &#40;isset&#40;$_REQUEST['year'&#93;&#41;&#41; &#123; $year = $_REQUEST['year'&#93;; &#125; 
if &#40;isset&#40;$_REQUEST['day'&#93;&#41;&#41; &#123; $day = $_REQUEST['day'&#93;; &#125; 




     //Set the Timezone 
 
putenv&#40;"TZ=US/Eastern"&#41;; 
   
$now getdate&#40;&#41;; 
// set the day/month/year to current if not passed as parms 
if&#40;!$mon&#41; &#123; $mon = $now['mon'&#93;;&#125; 
if&#40;!$year&#41; &#123; $year = $now['year'&#93;;&#125; 
if&#40;!$day&#41;  &#123; $day  = $now['mday'&#93;;&#125; 

?>



<CENTER><H3>Daily Graph for Anchorage, KY on <?php echo $mon?>-<?php echo $day?>-<?php echo $year?></H3></CENTER>

<CENTER><p><img src="http://www.weatherunderground.com/cgi-bin/wxStationGraphAll?day=<?php echo $day?>&amp;year=<?php echo $year?>&amp;month=<?php echo $mon?>&amp;ID=STATIONID&amp;type=3" width="600" height="645" />
</p></CENTER>






Again, you will need to change your station ID

Ken, gave me some code that I think would have redirected the link without changing the .htaccess file, but this workaround was easier for my simple mind.  If your host supports ASP (mine does not) you could just create an .asp file in the /weatherstation/ directory that would do the same thing as wugraphs.php without having to redirect.  

-E
South of the Tracks, Anchorage, KY

Offline Lars-K

  • Member
  • *
  • Posts: 10
Re: Anyone know of a php script that can capture a WU page?
« Reply #16 on: June 10, 2010, 03:21:04 AM »
Hi, I have a request: I tried this code
Code: [Select]
<?php 
   
//Set the Timezone 
 
putenv&#38;#40;"TZ=US/Eastern"&#38;#41;; 
   
$now getdate&#38;#40;&#38;#41;; 
   
$mon $now['mon'&#38;#93;; 
   
$year $now['year'&#38;#93;; 
   
$day  $now['mday'&#38;#93;; 
?>


<html>
<body bgcolor="#FFFFAA" link="#0000FF" vlink="#0000FF" alink="#0000FF">

<CENTER><p><img src="http://www.weatherunderground.com/cgi-bin/wxStationGraphAll?day=<?php echo $day?>&amp;year=<?php echo $year?>&amp;month=<?php echo $mon?>&amp;ID=KKYLOUIS20&amp;type=5" width="600" height="645" />
</p></CENTER>



</body>
<html>

and it worked. Then I tried to change "type" from 5 to 3. (Daily) and now the date shows 0/0/0 in the web page.

Here are the complete code I'm using:
Code: [Select]
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
    "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">

<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">

<head>
  <title></title>
</head>

<?php
   
//Set the Timezone
 
putenv&#38;#40;"TZ=US/Eastern"&#38;#41;;
   
$now getdate&#38;#40;&#38;#41;;
   
$mon $now['mon'&#38;#93;;
   
$year $now['year'&#38;#93;;
   
$day  $now['mday'&#38;#93;;
?>


<html>
<body bgcolor="#FFFFAA" link="#0000FF" vlink="#0000FF" alink="#0000FF">

<CENTER><p><img src="http://www.weatherunderground.com/cgi-bin/wxStationGraphAll?day=<?php echo $day?>&amp;year=<?php echo $year?>&amp;month=<?php echo $mon?>&amp;ID=KKYLOUIS20&amp;type=3" width="600" height="645" />
</p></CENTER>



</body>
<html>

Anybody that can help me, so the page shows the data from the present day?

Offline DaculaWeather

  • WxElement panel
  • Forecaster
  • *****
  • Posts: 3206
    • North Georgia Weather
Re: Anyone know of a php script that can capture a WU page?
« Reply #17 on: June 10, 2010, 08:43:59 AM »
Are you trying to do something like this? http://www.daculaweather.com/daculahistory.php

Offline Lars-K

  • Member
  • *
  • Posts: 10
Re: Anyone know of a php script that can capture a WU page?
« Reply #18 on: June 10, 2010, 09:41:09 AM »
That is how I have it today. What I wanted was only the daily data.
Like this:
Code: [Select]
http://www.wunderground.com/cgi-bin/wxStationGraphAll?day=8&year=2010&month=6&ID=IGILDESK2&type=3But with auto insertion of the date of today.

Offline ELDoradoWx

  • El Dorado Weather
  • Senior Contributor
  • ****
  • Posts: 298
  • Gentoo X = Complete_Control
    • El Dorado Weather
Re: Anyone know of a php script that can capture a WU page?
« Reply #19 on: June 27, 2010, 03:44:08 AM »
That is how I have it today. What I wanted was only the daily data.
Like this:
Code: [Select]
http://www.wunderground.com/cgi-bin/wxStationGraphAll?day=8&year=2010&month=6&ID=IGILDESK2&type=3But with auto insertion of the date of today.


I think this is what you're looking for to plug into your above code.

Code: [Select]
Change this:
<img src="http://www.weatherunderground.com/cgi-bin/wxStationGraphAll?day=<?php echo $day?>&amp;year=<?php echo $year?>&amp;month=<?php echo $mon?>&amp;ID=KKYLOUIS20&amp;type=5" width="600" height="645" />

To this:
<img src="http://www.wunderground.com/cgi-bin/wxStationGraphAll?day=<?php echo date("d"); ?>&amp;year=<?php echo date("Y"); ?>&amp;month=<?php echo date("m"); ?>&amp;ID=KKYLOUIS20&type=3" width="600" height="645" />
« Last Edit: June 27, 2010, 04:08:09 AM by eldoradowx »
-Danny
-root is everything

 

anything