WXforum.net

Web Weather => Weather Website PHP/AJAX scripting => Topic started by: gwwilk on March 11, 2018, 09:20:28 AM

Title: NOAA IOD No Longer Active?
Post by: gwwilk on March 11, 2018, 09:20:28 AM
There has been no new IOD posted by the NOAA ENVVL (https://www.nnvl.noaa.gov/) since February 5, 2018.

Has this been abandoned?  Maybe the person or persons responsible for this feature were promoted/transferred and those who remain aren't aware of it?  Or something???
Title: Re: NOAA IOD No Longer Active?
Post by: Bunty on March 11, 2018, 04:11:51 PM
I gather it's not a very popular feature since it's taken over a month before someone complained about it.
Title: Re: NOAA IOD No Longer Active?
Post by: gwwilk on March 11, 2018, 05:18:42 PM
I gather it's not a very popular feature since it's taken over a month before someone complained about it.
Or I’m just very patient. :shock:

I look at it every day.
Title: Re: NOAA IOD No Longer Active?
Post by: dwhitemv on March 12, 2018, 02:02:39 PM
Image of the Day is over here:

https://www.nesdis.noaa.gov/satellites-image-of-the-day
Title: Re: NOAA IOD No Longer Active?
Post by: gwwilk on March 12, 2018, 03:23:31 PM
Image of the Day is over here:

https://www.nesdis.noaa.gov/satellites-image-of-the-day
Thanks!

However, looking at that site, it's more like image-of-some-days-maybe-today-maybe-not.  I couldn't find a way to easily scrape the most recent image.  The image directory isn't accessible for reading, so it's going to require a retrieval of the latest image page's html output from the main page, and then scraping the latest image URL from there.  Not a 3 minute project.

Maybe Ken can figure it out.  I think it's his script.
Title: Re: NOAA IOD No Longer Active?
Post by: Bunty on March 13, 2018, 03:14:41 PM
I'll remove the Image of the Day from my front page menu for now.
Title: Re: NOAA IOD No Longer Active?
Post by: gwwilk on March 13, 2018, 03:51:50 PM
I'll remove the Image of the Day from my front page menu for now.
As I have done.
Title: Re: NOAA IOD No Longer Active?
Post by: saratogaWX on March 13, 2018, 04:48:51 PM
The new page doesn't seem to have a real 'image of the day', so I'll likely deprecate the script as the source has dried up.  :(

The main page now says
Quote
The NOAA Environmental Visualization Laboratory site will no longer be updated. Please visit https://www.nesdis.noaa.gov for the latest information on the Nation's environmental satellites and data. The staff wishes to thank you for your support since 1999.

and the new main page says
Quote
NESDIS is proposing to phase out dissemination of our data to non-governmental users via the PRODUCT DISTRIBUTION AND ACCESS system (PDA). We welcome comments through March 23, 2018.

So it looks like so-long to this feature.
Title: Re: NOAA IOD No Longer Active?
Post by: lddaly on March 13, 2018, 04:57:30 PM
Removed here too. We just checked our stats and the page had received 434 unique pageviews during 2017.
Title: Re: NOAA IOD No Longer Active?
Post by: saratogaWX on March 13, 2018, 05:40:50 PM
I've now removed wxnoaaiod.php/get-nnvl-iod.php from the Base-USA template distribution and marked the script as discontinued in the scripts page.  Sorry folks.. no source data, ergo no script needed...
Title: Re: NOAA IOD No Longer Active?
Post by: gwwilk on March 13, 2018, 06:46:49 PM
The new page doesn't seem to have a real 'image of the day', so I'll likely deprecate the script as the source has dried up.  :(

The main page now says
Quote
The NOAA Environmental Visualization Laboratory site will no longer be updated. Please visit https://www.nesdis.noaa.gov for the latest information on the Nation's environmental satellites and data. The staff wishes to thank you for your support since 1999.

and the new main page says
Quote
NESDIS is proposing to phase out dissemination of our data to non-governmental users via the PRODUCT DISTRIBUTION AND ACCESS system (PDA). We welcome comments through March 23, 2018.

So it looks like so-long to this feature.
I understand, Ken, but I was hoping you would be able to figure out how to get from here (https://www.nesdis.noaa.gov/satellites-image-of-the-day) to here (https://www.nesdis.noaa.gov/content/third-noreaster-10-days-impacts-new-england) to here (https://www.nnvl.noaa.gov/NESDIS/site/images/20180313-noreaster.png).

In reading the relevant verbiage (https://www.nesdis.noaa.gov/content/partnership-policy) I'm not sure if they mean images by 'Data'.  Who is to know?  Oh well, I liked the IOD very much. :-(
Title: Re: NOAA IOD No Longer Active?
Post by: Forever on March 13, 2018, 11:01:44 PM
What a shame, I had hoped that the script could be linked to the new site.  :-(
Title: Re: NOAA IOD No Longer Active?
Post by: gwwilk on April 22, 2018, 06:47:14 PM
I've given little thought to scraping the new NOAA IOD primarily because they've restricted access to the image directory, i.e. no directory listing is available to outsiders, but individual images are available when directly referenced.  So for now rather than not provide any way to access the IOD I've placed a link to the Satellite Image of the Day Gallery (https://www.nesdis.noaa.gov/satellites-image-of-the-day) at the bottom of my Saratoga Menu's Almanac section.

This will have to do for now.  These are great images that shouldn't go without being seen!
Title: Re: NOAA IOD No Longer Active?
Post by: jgillett on April 23, 2018, 11:20:57 AM
Great images. Thanks for that link, Jerry.
Title: Re: NOAA IOD No Longer Active?
Post by: smokie on April 25, 2018, 10:40:54 AM
Been working on this, only a novice at php so I  expect better ways to do it

Amend the settings to your website, displays the newest image, working on the text

Hopefully somebody can can improve this.
Might get error first time run as no files fetched/saved

Working here http://www.newquayweather.com/wxnoaaIOD.php (http://www.newquayweather.com/wxnoaaIOD.php)  click on image for full size


Code: [Select]
<?php
// PHP script by newquayweather.com
// 
// Version iotd.php  V1.0 - 24-April-2018
// 
  
$Version 'iotd.php V1.0 - 24-April-2018';
//
// cacheName is name of file used to store cached webpage
  
$cacheName "/home/xxxx/public_html/xxxx/iotd/data/iotdcache.txt";
// filename is url of webpage to get
 
$fileName "https://www.nesdis.noaa.gov/satellites-image-of-the-day";
// file to save processed data to
 
$record "/home/xxx/public_html/xxxx/iotd/data/iotd.html";  // used to store the file 
 
 
$refetchSeconds 14;   // refetch every nnnn seconds 3600 = 1 hour, 7200 = 2 hrs, 14400 = 4 hrs etc

// end of settings
//
// ------ start of code -------

if (file_exists($cacheName) and filemtime($cacheName) + $refetchSeconds time()) {
      
$html implode(''file($cacheName));
    } else {
        
$html fetchUrlWithoutHanging($fileName);
      if (
strlen($html) > ) {
      
$fp fopen($cacheName"w");
      if (
$fp) {
        
$write fputs($fp$html);
        
fclose($fp);
      } else {
            print 
"error";
      }
}
}

// get image name
 
preg_match('|https://www.nesdis.noaa.gov/sites/default/files/styles/large/public/(.*)jpg|Uis',$html,$pic);

   
preg_match('|<div class="field-satellite-image-thumbnail">(.*)</div>|Uis',$html,$matches);

  
$pattern '|<div class="field-satellite-image-thumbnail">|'// new
  
$replacement='';  // new
  
$matches=preg_replace($pattern$replacement$matches);
  
  
$pattern '/typeof="foaf:Image" class="img-responsive"/';
  
$replacement='';  // new
  
$matches=preg_replace($pattern$replacement$matches);
  
  
$pattern '|</div>|'// new
  
$replacement='';  // new
  
$matches=preg_replace($pattern$replacement$matches);
// update url  
  
$pattern '/\/content\/(.*)">/';
  
$replacement="https://www.nnvl.noaa.gov/NESDIS/site/images/" $pic[1]. "png\"  target=\"_blank\" />";  // new url for large image
  
$matches=preg_replace($pattern$replacement$matches);
  
$data=$matches[0];

file_put_contents($record$data);

print 
$data;

// ----------------------------functions ----------------------------------- 
 
 
function fetchUrlWithoutHanging($url
   {
$numberOfSeconds=5;  
$ch curl_init();    
curl_setopt($chCURLOPT_URL$url); // set url
curl_setopt($chCURLOPT_SSL_VERIFYPEERfalse);  
curl_setopt($chCURLOPT_USERAGENT"Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.6) Gecko/20070725 Firefox/2.0.0.6"); // set browser/user agent    
curl_setopt($chCURLOPT_RETURNTRANSFERtrue);
curl_setopt($chCURLOPT_CONNECTTIMEOUT$numberOfSeconds);
curl_setopt($chCURLOPT_TIMEOUT$numberOfSeconds);
$data=curl_exec($ch);
   return(
$data);

   }    

?>

Title: Re: NOAA IOD No Longer Active?
Post by: gwwilk on April 25, 2018, 11:16:50 AM
Looks great, smokie!  The big stumbling block for me is the REGEX statements, and you've nailed them. =D&gt; \:D/ =D&gt;

Looking forward to seeing how the text shapes up. :grin:
Title: Re: NOAA IOD No Longer Active?
Post by: smokie on April 25, 2018, 12:21:14 PM
Thanks gwwilk

With some small changes, ie change preg_match to preg_match_all, and some looping code you could show any number of past images as well, this was what I started with but like you the REGEX  got me so I went with get newest image only first.

I will have a go at multiple images next
Title: Re: NOAA IOD No Longer Active?
Post by: yamiacaveman on April 27, 2018, 08:06:34 AM
I know this is not the same but instead of the NOAA, I would like to use NASA IOD example: https://www.nasa.gov/multimedia/imagegallery/iotd.html

Would it be possible to change something like that? Those of you who know me, know that I know nothing about this stuff, but rather than give up on the NOAA pic of the day I would like to have something.

So if it is possible, please let me know -- I can't really screw up something that doesn't work any more, but even I wouldn't bet on that.

Thanks for any input.

yamiacaveman