Author Topic: NOAA's forecast/xml/xml.php  (Read 458 times)

0 Members and 1 Guest are viewing this topic.

Offline mpotts95420

  • Member
  • *
  • Posts: 6
    • websites designed by the Caspar Institute
NOAA's forecast/xml/xml.php
« on: August 13, 2017, 04:37:17 PM »
Until recently, I posted a small but powerful weather forecast table parsed from http://www.wrh.noaa.gov/forecast/xml/xml.php?duration=168&interval=6&lat=39.33691&lon=-123.81027.
 [ You are not allowed to view attachments ]
My half-dozen family members used it to plan activities -- have all that yummy data was much more useful to us than a simple verbal forecast. In particular, the wind into helps us here on the Northern California Coast. And parsing XML is easier than the similar but much less reliably formatted data at http://www.wrh.noaa.gov/forecast/wxtables/index.php?duration=168&interval=6&lat=39.33691&lon=-123.81027
My script is now getting refused by the NOAA server with the cryptic message "If you have automated scripts, that are now unable to pull the data, Don't for get to: 1/ Set the ACCEPT header, and 2/ Add user-agent to the header!"
Anybody got any clues about how to satisfy their new requirements? I read their arglebargle about what they were protecting their data from exploits, but anything I tried in the way of PHP headers and specifying user_agents didn't seem to make their server happy.
Alternatively, is there a source for forecasting information this granular? Thank you for any help.

Offline wvdkuil

  • Wim van der kuil
  • Forecaster
  • *****
  • Posts: 1121
    • Support site Leuven Template and scripts
Re: NOAA's forecast/xml/xml.php
« Reply #1 on: August 13, 2017, 04:48:09 PM »
Until recently, I posted a small but powerful weather forecast table parsed from http://www.wrh.noaa.gov/forecast/xml/xml.php?duration=168&interval=6&lat=39.33691&lon=-123.81027.
 [ You are not allowed to view attachments ]
My half-dozen family members used it to plan activities -- have all that yummy data was much more useful to us than a simple verbal forecast. In particular, the wind into helps us here on the Northern California Coast. And parsing XML is easier than the similar but much less reliably formatted data at http://www.wrh.noaa.gov/forecast/wxtables/index.php?duration=168&interval=6&lat=39.33691&lon=-123.81027
My script is now getting refused by the NOAA server with the cryptic message "If you have automated scripts, that are now unable to pull the data, Don't for get to: 1/ Set the ACCEPT header, and 2/ Add user-agent to the header!"
Anybody got any clues about how to satisfy their new requirements? I read their arglebargle about what they were protecting their data from exploits, but anything I tried in the way of PHP headers and specifying user_agents didn't seem to make their server happy.
Alternatively, is there a source for forecasting information this granular? Thank you for any help.
If you post the script (rename it from xyz.php to xyz.txt we can take a look.
The latest requirements are:
1. Use CURL  to obtain the data
2. Add headers to identify yourself and the scripts requesting the data.
So that will take some adaptions necessary,

Wim
Vantage VUE with a WLIP used for https://weer.sluispark.be/
Vantage VP2 with a USB logger/Meteobridge for https://www.weerstation-herent.be/

Offline mpotts95420

  • Member
  • *
  • Posts: 6
    • websites designed by the Caspar Institute
Re: NOAA's forecast/xml/xml.php
« Reply #2 on: August 13, 2017, 08:48:00 PM »
Wim, thank you for your quick reply and your willingness to look at my self-taught spaghetti code! Haven't used cURL but I'm willing to learn. I have had a problem attaching a text file, so I'm just pasting in the code:

<?php // nowwxtbl.php : mrp/Ci 5Apr15 : parses weather xml stream from
   // http://www.wrh.noaa.gov/forecast/xml/xml.php?duration=168&interval=6&lat=39.33691&lon=-123.81027
error_reporting(E_ALL & ~(E_STRICT|E_NOTICE)) ;
ini_set("display_errors", 1) ;

$testflag = 0 ;
$tz = -8 ;
date_default_timezone_set("America/Los_Angeles") ;

function frag( $txt, $key ) { // extracts fragment following key
   global $diags, $truncpos ;
   $stxt = "{$key}" ;
   $ntxt = "/{$key}" ;
   $s = strpos( $txt, "<{$stxt}" ) + strlen( $stxt ) + 2 ;    // finds start text position
   $truncpos = strpos( $txt, $ntxt ) + 1 ;
   $n = $truncpos - $s - 2 ;
   $diags .= "stxt, ntxt, s, n: {$stxt}, {$ntxt}, {$s}, {$n}<br>" ;
   $frag = substr( $txt, $s, $n ) ;                                     // chops out fragment between start and end
   return $frag ;   
}

function xsfx( $txt ) { // chops off the suffix/tail
   global $diags ;
   $x = strpos( $txt, ">" ) + 1 ;
   $xfrag = substr( $txt, 0, $x ) ;
   $frag = substr( $txt, $x ) ;
   $diags .= "x, xfrag: {$x}, |{$xfrag}|<br>" ;
   return $frag ;
}

// gets requester's desired weather lat/lon : $wthrlatlon
if ( $_COOKIE['op'] ) { $op = $_COOKIE['op'] ; } else { $op = "now" ; }
$diags .= "op: {$op}<br>\n" ;
require_once 'time/cn2db.php';
$sql = "SELECT `wthr` FROM `now` WHERE `op`=\"{$op}\" LIMIT 1" ;
$diags .= "sql: {$sql}<br>\n" ;
$opResource = mysql_query( $sql, $dbConn );
$nowvars = mysql_fetch_array( $opResource, MYSQL_ASSOC ) ;
$wthrlatlon = $nowvars['wthr'] ;

$wthrxml ="http://www.wrh.noaa.gov/forecast/xml/xml.php?duration=168&interval=6&{$wthrlatlon}" ;
//$wthrurl="http://www.wrh.noaa.gov/forecast/wxtables/index.php?{$wthrlatlon}" ;
$wthrxml = "w.xml" ; // for testing
$wsrc = file_get_contents( $wthrxml ) ; // Caspar Weather
$wrows = array() ;
   // 0=days 1=hours 2=weather 3=temp 4=cloud 5=chance 6=precip 7=windspeed, gust, windirection 8=RH 9=dewpt 10=ssnowlevel
$wttls = array( "","","Weather","temp","cloud","chance","precip","wind","RH","dewpt","snow lvl") ;
$wdirs = array( "N","NNE","NE","ENE","E","ESE","SE","SSE","S","SSW","SW","WSW","W","WNW","NW","NNW" ) ;
$wxs = array( "chc rw","chc r","lkly rw","lkly r","def rw","def r","num rw","schc rw","schc r","patchy l","patchy f") ;
$wxphr = array( "Chance<br>Rain+","Chance<br>Rain","Rain+<br>Likely","Rain<br>Likely","Rain<br>TStrm","Rain","Sctrd<br>Shwrs","Slight<br>Chance+","Slight<br>Chance","Patchy<br>Drizzle","Patchy<br>Fog") ;
$wxclr = array( " bgcolor='DDDDFF'"," bgcolor='AAAAFF'"," bgcolor='8888FF'"," bgcolor='888888'"," bgcolor='FF0000' class='invtny'", " bgcolor='000088' class='invtny'"," bgcolor='DDDDFF'") ;
$days = array("Sun","Mon","Tue","Wed","Thu","Fri","Sat") ;
$daynum = date( "w" ) ;
$cparm = " align='center'" ;
$bg0 = " bgcolor='FFFFFF'" ;
$bgb1 = " bgcolor='DDDDFF'" ;
$bgb2 = " bgcolor='AAAAFF'" ;
$bgb3 = " bgcolor='8888FF'" ;
$bgb4 = " bgcolor='000088' class='invtny'" ;
$bgg1 = " bgcolor='DDDDDD'" ;
$bgg2 = " bgcolor='AAAAAA'" ;
$bgg3 = " bgcolor='888888'" ;
$bgg4 = " bgcolor='444444' class='invtny'" ;
$bgb = " bgcolor='000000' class='invtny'" ;
$bgt0 = " bgcolor='DDDDFF'" ;
$bgt1 = " bgcolor='000088' class='invtny'" ;
$bgt2 = " bgcolor='AAAAFF'" ;
$bgt3 = " bgcolor='FFFFAA'" ;
$bgt4 = " bgcolor='FF0000' class='invtny'" ;
foreach( $wttls as $ttl ) {
   $wrows[] = "<td align='right'>{$ttl}</td>" ;
}
$tblttl = "NOAA Forecast for " . frag( $wsrc, "location" ) ;
$tblttl .= " created " . frag( $wsrc, "forecastCreationTime" ) ;
// testing:
$tblttl = "NOAA is broken - This is NOT live weather<br>
this is Sun Apr 05 09:38:08 2015 UTC" ;
$parsing = TRUE ;
$dayctr = 0 ;
while ( $parsing ) {
   $dayctr ++ ;
   $day = frag( $wsrc, "forecastDay" ) ;
   $wsrc = substr( $wsrc, $truncpos ) ;
   $dt = frag( $day, "validDate" ) ;
   $diags .= "dt: |{$dt}|<br>" ;
   $perctr = 0 ;
   $dayparse = TRUE ;
   while ( $dayparse ) {
      if ( $perctr == 2 ) {
         for ( $i=1; $i<=10 ; $i++ ) {
            $wrows[ $i ] .= "<td{$bgg1} width='1'></td>" ;
         }       
      }
      $perctr ++ ;
      $per = frag( $day, "period" ) ;
      $day = substr( $day, $truncpos ) ;
      $hr = frag( $per, "validTime" ) ;
      $hr = xsfx( $hr ) + $tz;
      if ( $hr < 0 ) { $hr = $hr + 24 ; }
      $wrows[1] .= "<td{$cparm}>{$hr}</td>" ;
      $diags .= "hr: |{$hr}|<br>" ;
      $wx =  frag( $per, "wx" ) ;
      $lwx = trim( strtolower( $wx ) ) ;
      $bg = $bg0 ;
      if ( $wx == -999 ) { $pwx = "" ; $bg = $bg0 ; }
      else {
         if ( in_array( $lwx, $wxs ) ) {
            $phrptr = array_search( $lwx, $wxs ) ;
            $diags .= "wx, lwx, phrptr: |{$wx}|, |{$lwx}|, {$phrptr}<br>" ;
            $pwx = $wxphr[ $phrptr ] ;
            $bg = $wxclr[ $phrptr ] ;
         } else { $pwx = $wx ; }
      }
      $wrows[2] .= "<td{$cparm}{$bg}>{$pwx}</td>" ;
      $t = frag( $per, "temperature" ) ;
      $t = xsfx( $t ) ;
      $bg = $bgt0 ;
      if ( $t < 36 ) { $bg = $bgt1 ; }
      if ( $t < 50 ) { $bg = $bgt2 ; }
      if ( $t > 60 ) { $bg = $bgt3 ; }
      if ( $t > 75 ) { $bg = $bgt4 ; }
      if ( $t < .01 ) { $t = "" ; $bg = $bg0 ; }
      $wrows[3] .= "<td{$cparm}{$bg}>{$t}</td>" ;
      $cvr = frag( $per, "skyCover" ) ;
      $cvr = xsfx( $cvr ) ;
      $bg = $bgg1 ;
      if ( $cvr < .01 ) { $cvr = "" ; $bg = $bg0 ; }
      if ( $cvr > 25 ) { $bg = $bgg2 ; }
      if ( $cvr > 50 ) { $bg = $bgg3 ; }
      if ( $cvr > 75 ) { $bg = $bgg4 ; }
      if ( $cvr > 95 ) { $bg = $bgb ; }
      $wrows[4] .= "<td{$cparm}{$bg}>{$cvr}</td>" ;
      $pop = frag( $per, "pop" ) ;
      $pop = xsfx( $pop ) . "%" ;
      $bg = $bgb1 ;
      if ( $pop < .01 ) { $pop = "" ; $bg = $bg0 ; }
      if ( $pop > 25 ) { $bg = $bgb2 ; }
      if ( $pop > 50 ) { $bg = $bgb3 ; }
      if ( $pop > 75 ) { $bg = $bgb4 ; }
      $wrows[5] .= "<td{$cparm}{$bg}>{$pop}</td>" ;
      $qpf = frag( $per, "qpf" ) ;
      $qpf = xsfx( $qpf ) ;
      $bg = $bgb1 ;
      if ( $qpf < .01 ) { $qpf = "" ; $bg = $bg0 ; }
      if ( $qpf > .05 ) { $bg = $bgb2 ; }
      if ( $qpf > .10 ) { $bg = $bgb3 ; }
      if ( $qpf > .25 ) { $bg = $bgb4 ; }
      $wrows[6] .= "<td{$cparm}{$bg}>{$qpf}</td>" ;
      $w = frag( $per, "windSpeed" ) ;
      $w = xsfx( $w ) ;
      if ( $w < 1 ) { $w = -999 ; }
      $gust = frag( $per, "windGust" ) ;
      $gust = xsfx( $gust ) ;
      $bg = $bgt2 ;
      if ( $gust < 7 ) { $bg = $bgt1 ; }
      if ( $gust < 15 ) { $bg = $bgt0 ; }
      if ( $gust > 34 ) { $bg = $bgt3 ; }
      if ( $gust > 50 ) { $bg = $bgt4 ; }
      if ( $gust < 1 ) { $gust = -999 ; }
      $wdir = frag( $per, "windDirection" ) ;
      $wdir = xsfx( $wdir ) ;
      if ( $wdir < 0 ) { $wdir = -999 ; }
      else {
         $wptr = intval( $wdir / 22.5 ) ;
         $wdir = $wdirs[ $wptr ] ;
      }
      if ( $w<0 AND $gust<1 AND $wdir<0 ) { $wrow7 = "" ; $bg = $bg0 ; }
      else { $wrow7 = "{$w}-{$gust}<br>\n{$wdir}" ; }
      $wrows[7] .= "<td{$cparm}{$bg}>{$wrow7}</td>" ;
      $rh = frag( $per, "rh" ) ;
      $rh = xsfx( $rh ) . "%" ;
      if ( $rh < .01 ) { $rh = "" ; $bg = $bg0 ; }
      $wrows[8] .= "<td{$cparm}>{$rh}</td>" ;
      $dew = frag( $per, "dewpoint" ) ;
      $dew = xsfx( $dew ) ;
      if ( $dew < .01 ) { $dew = "" ; $bg = $bg0 ; }
      $wrows[9] .= "<td{$cparm}>{$dew}</td>" ;
      $slvl = frag( $per, "snowLevel" ) ;
      $slvl = xsfx( $slvl ) ;
      if ( $slvl < .01 ) { $slvl = "" ; $bg = $bg0 ; }
      $wrows[10] .= "<td{$cparm}>{$slvl}</td>" ;
      if ( $perctr > 3 ) { $dayparse = FALSE ; }
   }
/*   for ( $i=1; $i<=10 ; $i++ ) {
      $wrows[ $i ] .= "<td{$bgg1} width='1'></td>" ;
   } */
   $wkdy = $days[ $daynum ] ;
   if ( $dayctr == 1 ) { $wrows[0] .= "<td colspan='2'></td><td{$bgg1} width='1'></td>" ; }
   $wrows[0] .= "<td colspan='4' align='center'>{$wkdy} {$dt}</td>" ;
   if ( $dayctr < 7 ) { $wrows[0] .= "<td{$bgg1} width='1'></td>" ; }
   $daynum ++ ;
   if ( $daynum > 6 ) { $daynum = 0 ; }
   if ( $dayctr > 6 ) { $parsing = FALSE ; }
}


?><!DOCTYPE html>
<html>
<head>
   <title>Weather Table</title>
<link rel='STYLESHEET' type='text/css' href='now.css'>
</head>

<body>
<?php
echo "<table class='tny' cellpadding='1' cellspacing='0' border='0' align='center' bgcolor='#FFFFDD'>
<tr bgcolor='#FFFF00' align='center'><td class='txtni' colspan='99'><a href='{$wthrurl}' target='guest'><strong>{$tblttl}</strong></a></td></tr>" ;
foreach ( $wrows as $row ) {
   echo "<tr>{$row}</tr>" ;
}
echo "</table>" ;   
if ( $testflag ) { echo "<br><br><br>{$diags}" ;   }
   ?>
</body>
</html>

Offline saratogaWX

  • Administrator
  • Forecaster
  • *****
  • Posts: 6226
  • Saratoga, CA, USA Weather - free PHP scripts
    • Saratoga-Weather.org
Re: NOAA's forecast/xml/xml.php
« Reply #3 on: August 13, 2017, 09:06:10 PM »
Likely all you need is to change how you use file_get_contents() with the NWS sites.

Change
Code: [Select]
$wsrc = file_get_contents( $wthrxml ) ; // Caspar Weather
to
Code: [Select]
   $STRopts = array(
  'http'=>array(
  'method'=>"GET",
  'protocol_version' => 1.1,
  'header'=>"Cache-Control: no-cache, must-revalidate\r\n" .
"Cache-control: max-age=0\r\n" .
"Connection: close\r\n" .
"User-agent: Mozilla/5.0 (nowwxtbl.php - casparinstitute.org)\r\n" .
"Accept: text/xml,text/plain\r\n"
  ),
  'https'=>array(
  'method'=>"GET",
  'protocol_version' => 1.1,
  'header'=>"Cache-Control: no-cache, must-revalidate\r\n" .
"Cache-control: max-age=0\r\n" .
"Connection: close\r\n" .
"User-agent: Mozilla/5.0 (nowwxtbl.php - casparinstitute.org)\r\n" .
"Accept: text/xml,text/plain\r\n"
  )
);

$STRcontext = stream_context_create($STRopts);

$wsrc = file_get_contents( $wthrxml,false,$STRcontext ) ; // Caspar Weather
The context provided should produce the correct headers needed to retrieve the data from the NWS site.
Ken True/Saratoga, CA, USA main site: saratoga-weather.org
Davis VP1+ FARS, Boltek-PCI/NexStorm, microSferics ToA, Blitzortung RED, GRLevel3, WD, WL, VWS, Cumulus, Meteobridge/hub
Free weather PHP scripts/website templates - update notifications on Twitter saratogaWXPHP

Offline mpotts95420

  • Member
  • *
  • Posts: 6
    • websites designed by the Caspar Institute
Re: NOAA's forecast/xml/xml.php
« Reply #4 on: August 14, 2017, 01:50:51 PM »
THAT WORKS! There is NO WAY I would have figured that out without your help. Thank you, Ken.
« Last Edit: August 14, 2017, 01:52:23 PM by mpotts95420 »

Offline mpotts95420

  • Member
  • *
  • Posts: 6
    • websites designed by the Caspar Institute
Re: NOAA's forecast/xml/xml.php
« Reply #5 on: August 15, 2017, 07:44:42 PM »
UPDATE: at 16:46 it's working again. Could it really be down for 25 minutes while it's refreshing?

Well, at least it worked for awhile! I rejiggered the code to cope with the new data format -- much cleaner than it was, and with a bit more useful information -- and as late as noon today I was getting a good result:
 [ You are not allowed to view attachments ]
...being careful to work from a saved version of the xml until I was sure it was working properly, so as not to overload their server (as requested.) Four hours and no reloads later, this is what I'm getting:
 [ You are not allowed to view attachments ]
I usually run the script in an iframe of a more complicated portal script here: http://casparinstitute.org/now.php
but the forecast script also runs alone at http://casparinstitute.org/wxml.php
Is the iframe part of the problem?
The URL for the xml works fine by itself from a browser (Chrome) but not in the portal / iframe context. I can get the xml, save it as a file, and then run the script on it (that's how I got the clean image #1 above).
« Last Edit: August 15, 2017, 07:52:20 PM by mpotts95420 »

Offline saratogaWX

  • Administrator
  • Forecaster
  • *****
  • Posts: 6226
  • Saratoga, CA, USA Weather - free PHP scripts
    • Saratoga-Weather.org
Re: NOAA's forecast/xml/xml.php
« Reply #6 on: August 15, 2017, 09:15:54 PM »
The NWS does cause point forecasts (including the XML feeds) to be unavailable at the times the point forecast data is being refreshed.  For the real point-forecasts via www.weather.gov, that usually means a redirect to the Zone forecast for that point.  I imagine that the XML forecasts do not have such a handling -- you'd have to capture the XML page at the time it happens to see what it says (or what the HTTP response code is) in order to diagnose it further.

One issue is that your script does not use caching (you make a NWS request each time your script is loaded), and that may be a longer-term issue.  You should consider adding caching code to fetch/store the XML every hour or so, which would result in only 24 fetches a day (at max) instead of a fetch-each-load-of-the-page.
Ken True/Saratoga, CA, USA main site: saratoga-weather.org
Davis VP1+ FARS, Boltek-PCI/NexStorm, microSferics ToA, Blitzortung RED, GRLevel3, WD, WL, VWS, Cumulus, Meteobridge/hub
Free weather PHP scripts/website templates - update notifications on Twitter saratogaWXPHP

Offline mpotts95420

  • Member
  • *
  • Posts: 6
    • websites designed by the Caspar Institute
Re: NOAA's forecast/xml/xml.php
« Reply #7 on: August 15, 2017, 09:38:28 PM »
Thank you again for your reassurance ...and for your quick reply. After putting several hours into the parsing and color-coding, I freaked when it all went south.
I have considered caching ... but I have six or eight users in six locations (Sacramento, Oakland, Chico, 2 in Caspar, one in Fort Bragg) and even the most active (me) only refreshes twice a day ... so I figured until the usage gets heavier, I'm doing NWS a favor by not caching regularly. I have considered caching each time more than an hour has elapsed, and providing the data from the cache if someone refreshes more often.

 

anything