Topic: Heads-up.. US Government websites to require HTTPS by end of 2016...

[saratogaWX]

As this article says:

Following the lead of many major Web services, the White House on Monday announced that it would move all of the federal government’s public sites and services to HTTPS-only.

Tony Scott, the federal CIO, has issued a memorandum to all federal agencies and departments instructing them to move all of their publicly accessible Web sites and services to HTTPS-only by the end of 2016.

The memorandum is available here .

What it means for weather enthusiasts running software that access US Government resources (NWS, USGS, etc.) is:

• All of the PHP scripts using http:// would have to change to use https:// and that is non-trivial. The scripts using FetchUrlWithoutHanging() routines would have to be updated with the functionality replaced (and I'll do that for my scripts).
• Native weather software (Weather-Display, etc) would have to change to support https:// for data from US Government sites if they use them.

Fortunately, with any US Government program the timescale is fairly long (1.5 years) for compliance with the directive to use HTTPS to be fully implemented and HTTP 'cut off' (or more likely, a 301-Moved redirect to the HTTPS content) implemented, so we have some time to prepare.

This is just the first 'heads-up' that the change is coming, and the coding is more complex to do -- likely many of my scripts will switch to use cURL or file_get_contents() with a context as the way of adapting to use native PHP instead of the low-level fconnect()/fget() with metrics for diagnostics timings.

More as I find out the details for specific site's plans.

Best regards,
Ken

[Andy G]

Thanks Ken for being on top of this an for all your hard work. I don't know where we would be without you.

Thanks again Andy

[cospringswx]

Thanks for the heads up Ken.

[saratogaWX]

Just did a bit of testing with the key sites:

www.weather.gov uses an invalid security certificate.
The certificate is only valid for the following names:
 *.akamaihd.net, *.akamaihd-staging.net, a248.e.akamai.net, *.akamaized.net, *.akamaized-staging.net
 (Error code: ssl_error_bad_cert_domain)

forecast.weather.gov uses an invalid security certificate.
The certificate is only valid for the following names:
 *.akamaihd.net, *.akamaihd-staging.net, a248.e.akamai.net, *.akamaized.net, *.akamaized-staging.net
 (Error code: ssl_error_bad_cert_domain)

dualicons-forecast.weather.gov has not enabled HTTPS ..
just times out.

earthquake.usgs.gov uses an invalid security certificate.
The certificate is only valid for the following names:
*.akamaihd.net, *.akamaihd-staging.net, a248.e.akamai.net, *.akamaized.net, *.akamaized-staging.net
(Error code: ssl_error_bad_cert_domain)

So they aren't quite ready for testing.. just have to wait for announcements.

It's my understanding that the NWS is deprecating all FTP access to data in favor of HTTP (and now maybe HTTPS).

I found the site https://https.cio.gov/ to be quite helpful in explaining in more detail what is mandated and caveats for implementation on Federal websites.

Best regards,
Ken

[chief-david]

sticky this!

[Andy G]

sticky this!

+1

[gwwilk]

sticky this!

+1

+2

Your guidance on this issue is going to be crucial, Ken.  Thanks!

[gwwilk]

...

• All of the PHP scripts using http:// would have to change to use https:// and that is non-trivial. The scripts using FetchUrlWithoutHanging() routines would have to be updated with the functionality replaced (and I'll do that for my scripts).
• Native weather software (Weather-Display, etc) would have to change to support https:// for data from US Government sites if they use them.

Fortunately, with any US Government program the timescale is fairly long (1.5 years) for compliance with the directive to use HTTPS to be fully implemented and HTTP 'cut off' (or more likely, a 301-Moved redirect to the HTTPS content) implemented, so we have some time to prepare.
...

The 'function fetchUrlWithoutHanging($url)' fell over this morning in my templated WxUSAlerts script so that it no longer displayed the legend.  This may have to do with the coming changes.  I implemented a modified 'fetchUrlWithoutHanging()', but the returned data uses chunked encoding, so I borrowed your 'QJ_fetchUrlWithoutHanging()' function from 'quake-json.php' to de-chunk the return.  Voila!  The WxUSAlerts page works again.  Thanks!

I first encountered this problem today with my US Ridge Radar HAniS animation.  After I queried the Webmaster and a back-and-forth exchange, Paul Kirkwood at NOAA received this message back from their Support Group:

We enabled rules that block requests that don't supply a user-agent. Here's what we recommend:

You can use any user agent string you want, but we typically follow this format:
appName/appVersion (Company Name, company contact/url)
For example:
myWeatherApp/1.0 (Weather Company, www.weathercompany.com)

Sure enough, when I used 'QJ_fetchUrlWithoutHanging' in this script's 'wxusradar-hanis-iframe3.php' component, it too was restored to normal functionality.  Bravo, Ken!

The user agent string was required in both feeds, and your 'QJ' function modified with the user agent, etc. as in 'advforecast2.php'

Code: [Select]

fputs($socketConnection, "GET $resourcePath HTTP/1.1\r\nHost: $domain\r\nCache-Control: no-cache, must-revalidate\r\nCache-control: max-age=0\r\nConnection: close\r\nUser-agent: Forecast loader (advforecast2.php saratoga-weather.org)\r\nAccept: text/plain,text/html\r\n\r\n");to

Code: [Select]

$getString = "GET $resourcePath HTTP/1.1\r\nHost: $domain\r\nCache-Control: no-cache, must-revalidate\r\nCache-control: max-age=0\r\nConnection: close\r\nUser-agent: Ridge Radars (wxusradars-hanis3.php  gwwilkins.org)\r\nAccept: */*,text/html\r\n\r\n";
for the radars and to

Code: [Select]

$getString = "GET $resourcePath HTTP/1.1\r\nHost: $domain\r\nCache-Control: no-cache, must-revalidate\r\nCache-control: max-age=0\r\nConnection: close\r\nUser-agent: Alerts Legend (wxusalerts.php gwwilkins.org)\r\nAccept: */*,text/html\r\n\r\n";
in the WxUSAlerts script.  And no, an accurate User-agent string isn't required, probably just a good idea.

[NCHurricane]

I wonder if this is why the NWS obs on my site quit working yesterday? I'm using simplexml_load_file to load the xml for each metar site and then echo each parameter to display. I haven't changed anything on the pages in two weeks.

If it may be something else, I'll delete this post and start a new thread.

[colonieweather]

I wonder if this is why the NWS obs on my site quit working yesterday? I'm using simplexml_load_file to load the xml for each metar site and then echo each parameter to display. I haven't changed anything on the pages in two weeks.

If it may be something else, I'll delete this post and start a new thread.

I had the same issue using simplexml_load_file.  I began getting an 403 error in log file on Tuesday:
simplexml_load_file(the NWS XML URL): failed to open stream: HTTP request failed! HTTP/1.0 403 Forbidden 

I just converted the simple XML load to cURL and worked again.

[gwwilk]

I wonder if this is why the NWS obs on my site quit working yesterday? I'm using simplexml_load_file to load the xml for each metar site and then echo each parameter to display. I haven't changed anything on the pages in two weeks.

If it may be something else, I'll delete this post and start a new thread.

I had the same issue using simplexml_load_file.  I began getting an 403 error in log file on Tuesday:
simplexml_load_file(the NWS XML URL): failed to open stream: HTTP request failed! HTTP/1.0 403 Forbidden 

I just converted the simple XML load to cURL and worked again.

This morning I realized that cURL would work, at least for now, so I converted both scripts to use it rather than the much more cumbersome 'QJ_fetchUrlWithoutHanging'.  The changed scripts seem to load faster.

[NCHurricane]

I wonder if this is why the NWS obs on my site quit working yesterday? I'm using simplexml_load_file to load the xml for each metar site and then echo each parameter to display. I haven't changed anything on the pages in two weeks.

If it may be something else, I'll delete this post and start a new thread.

I had the same issue using simplexml_load_file.  I began getting an 403 error in log file on Tuesday:
simplexml_load_file(the NWS XML URL): failed to open stream: HTTP request failed! HTTP/1.0 403 Forbidden 

I just converted the simple XML load to cURL and worked again.

Through PM's, colonieweather helped solve my issue, but I wanted to publicly thank him here for the help.

[wingwing]

Thanks for this info.. my site "died" too.

I use an aJax script and I'm trying to figure how to add th Agent Header.
..could use some help if someone knows.
..I've tried a few different ways but nothing works so far.. (guessing does not work too well .. lol..)

code snip
This WAS working for the past three years:
   $.ajax({
   async: true,
   cache: false,       
   "url": "myPhpScript.php?&Myproduct="+product+"&rid1="+ns1+"&rid2="+ns2+"&rid3="+ns3+"&rid4="+ns4,
   "type": "GET",
   "dataType": "json",
      "success": function( data, status ){

I've tried some variations like this, but so far it's still not working:
   $.ajax({
   async: true,
   cache: false,       
    headers: { 'Api-User-Agent': 'myapp/5.0 (MyWxSite, www.MyUrl.com)' },   

thanks in advance
Dennis

[saratogaWX]

Try using

headers: { 'User-agent': 'myapp/5.0 (MyWxSite, www.MyUrl.com)' },

instead.  The User-agent header is now required by the NWS for accesses -- it has nothing to do with http v.s. https however, just a new requirement to have them prevent bots (w/o a User-agent header) from accessing their site(s).

[wingwing]

try..
headers: { 'User-agent': 'myapp/5.0 (MyWxSite, www.MyUrl.com)' },

Thank you for your quicki reply!

I tried that "as is" and also with my actual app and site info..
..no joy.. yet...

both result in:
"Refused to set unsafe header "User-agent""
.. but that IS farther than I got before..

The aJAX function calls my php script.
I wonder if I need to change anything there too?

thanks again,
Dennis

[wingwing]

still working on it.. no joy yet.

does anyone have a link (or example) for a cURL script that can grab the most recent 5 ridge files?

thanks,
Dennis

[Jáchym]

For NWS I use this and it works, you can give it a try:

Code: [Select]

function curl_get_contentsNWS($url){
$ch = curl_init();
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/5.0 (Windows NT 6.1; rv:19.0) Gecko/20100101 Firefox/19.0");
$data = curl_exec($ch);
curl_close($ch);
return $data;
}

[wingwing]

Thanks for that.

Now, I'm getting past the USER AGENT security change but I suspect they also altered their php,ini file.

The rest of my script had been successfully using  "file_get_contents" to get the latest ridge N0R files.
I believe (at this time) that also has been disallowed for cross domain requests.

Thanks,
Dennis

[saratogaWX]

The file_get_contents() defaults to have no user agent.   Use something like

Code: [Select]

$opts = array(
  'http'=>array(
'method'=>"GET",
'protocol_version' => 1.1,
'header'=>"Cache-Control: no-cache, must-revalidate\r\n" .
  "Cache-control: max-age=0\r\n" .
  "Connection: close\r\n" .
  "User-agent: Mozilla 5.0 ($Version saratoga-weather.org)\r\n" .
  "Accept: text/plain,text/html\r\n"
  )
);

$context = stream_context_create($opts);

   $xml = file_get_contents($url,0,$context);

Change the line

Code: [Select]

"User-agent: Mozilla 5.0 ($Version saratoga-weather.org)\r\n" .to have your own user agent name.

[wingwing]

re:
"The file_get_contents() defaults to have no user agent.   Use something like
Code: [Select]"

That did it..

million thanks

Dennis