Author Topic: Local METAR reports Not loading (Read 11695 times)

saratogaWX · « **Reply #100 on:** October 02, 2019, 04:29:59 PM »

Ed,
You're getting the 403 error, but from a different GoDaddy egress server

Quote

My guess is that the NWS webmaster for tgftp.nws.noaa.gov has put in a firewall block for queries from 72.167.190.0/24 block (which I think is the set of egress servers based on the whois registration (a /24 network).

Are you also on an 'old' GoDaddy hosting (and not a newer cPanel hosting)?

PaulMy · « **Reply #101 on:** October 02, 2019, 04:32:50 PM »

My http://www.komokaweather.ca/wxmetar.php had worked after doing the mod a couple weeks ago, but now get the RC=403 Forbidden. I have not yet contacted GoDaddy. I upgraded to GoDaddy cPanel in 2018.

Enjoy,
Paul

mkutche · « **Reply #102 on:** October 02, 2019, 05:39:32 PM »

This is nuts.. i refuse to switch hosts because its a pain in the butt. there has to be some kinda fix for this!!

saratogaWX · « **Reply #103 on:** October 02, 2019, 05:41:21 PM »

I have a cPanel host on GoDaddy. Just ran some requests via terminal:

Quote

swnmaint2@a2plcpnl0172 [~]$ hostname
a2plcpnl0172.prod.iad2.secureserver.net
swnmaint2@a2plcpnl0172 [~]$ hostname -i
198.71.224.69
$swnmaint2@a2plcpnl0172 [~]$ nslookup tgftp.nws.noaa.gov
Server: 10.255.250.30
Address: 10.255.250.30#53

Non-authoritative answer:
tgftp.nws.noaa.gov canonical name = tgftp.op.ncep.noaa.gov.
tgftp.op.ncep.noaa.gov canonical name = tgftp.bldr.ncep.noaa.gov.
Name: tgftp.bldr.ncep.noaa.gov
Address: 140.172.138.79

swnmaint2@a2plcpnl0172 [~]$ cat /etc/resolv.conf
# Generated via Puppet
nameserver 10.255.250.30
nameserver 10.255.251.30
options rotate
swnmaint2@a2plcpnl0172 [~]$ curl -v https://tgftp.nws.noaa.gov/data/observations/metar/stations/KHUF.TXT
* About to connect() to tgftp.nws.noaa.gov port 443 (#0)
* Trying 140.172.138.79... connected
* Connected to tgftp.nws.noaa.gov (140.172.138.79) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
* SSL connection using TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
* Server certificate:
* subject: CN=dev.ncep.noaa.gov,OU=Domain Control Validated
* start date: Feb 13 16:27:33 2019 GMT
* expire date: Jan 27 15:33:01 2020 GMT
* common name: dev.ncep.noaa.gov
* issuer: CN=Go Daddy Secure Certificate Authority - G2,OU=http://certs.godaddy.com/repository/,O="GoDaddy.com, Inc.",L=Scottsdale,ST=Arizona,C=US
> GET /data/observations/metar/stations/KHUF.TXT HTTP/1.1
> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.27.1 zlib/1.2.3 libidn/1.18 libssh2/1.4.2
> Host: tgftp.nws.noaa.gov
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Wed, 02 Oct 2019 21:30:24 GMT
< Server: Apache
< X-Frame-Options: SAMEORIGIN
< X-Content-Type-Options: nosniff
< X-XSS-Protection: 1; mode=block
< Last-Modified: Wed, 02 Oct 2019 20:55:28 GMT
< ETag: "4f73a8c-5d-593f3b05286dd"
< Accept-Ranges: bytes
< Content-Length: 93
< Vary: Accept-Encoding
< Content-Type: text/plain; charset=UTF-8
< Strict-Transport-Security: max-age=31536000; includeSubdomains; preload
<
2019/10/02 20:53
KHUF 022053Z 24009KT 10SM FEW055 33/20 A2990 RMK AO2 SLP120 T03280200 56013
* Connection #0 to host tgftp.nws.noaa.gov left intact
* Closing connection #0
swnmaint2@a2plcpnl0172 [~]$

and it worked fine on the actual webserver. Note that the server has two private IP address DNS resolvers.

Paul, you appear to be using a different GoDaddy egress server

Quote

That IP is part of the 50.62.0.0/15 IP space assigned to GoDaddy, and 50.62.176.93 is not the IP of your site (23.229.161.0) so it's an egress server also.

saratogaWX · « **Reply #104 on:** October 02, 2019, 05:52:17 PM »

Quote from: mkutche on October 02, 2019, 05:39:32 PM

This is nuts.. i refuse to switch hosts because its a pain in the butt. there has to be some kinda fix for this!!

There seems to be two issues:
1) the caching DNS servers for some GoDaddy websites cannot reliably do a IP lookup (gethostbyname()) for tgftp.nws.noaa.gov -- that would seem to be firmly in GoDaddy's court to solve. With the existing data from y'all, it seems to be on older GoDaddy shared hosting for that issue.
Jasiu's bypass fix does address that problem.
2) a new problem (403-Forbidden) returns from tgftp.nws.noaa.gov appear to be cause by firewall rules likely implemented by the NWS on tgftp.nws.noaa.gov.

The NWS server is in the netblock

Quote

NetRange: 140.172.0.0 - 140.172.255.255
CIDR: 140.172.0.0/16
NetName: NOAA-BOULDER
NetHandle: NET-140-172-0-0-1
Parent: NET140 (NET-140-0-0-0-0)
NetType: Direct Assignment
OriginAS:
Organization: NOAA-Boulder (NOAABO)

they list
OrgTechHandle: RN326-ARIN
OrgTechName: NOAA-Boulder NOC
OrgTechPhone: +1-812-856-7477
OrgTechEmail: nb-noc@noaa.gov

as the Tech/Abuse contacts for the IP space.

I'll send them a note asking for clarification about the 403-Forbidden responses for GoDaddy IP-space requests.

saratogaWX · « **Reply #105 on:** October 02, 2019, 06:05:55 PM »

I've sent off the following note:

Quote

Gentlebeings,

I have several scripts that access data from https://tgftp.nws.noaa.gov/ for raw METAR data. Lately, several users of the script have begun encountering 403-Forbidden responses from the site for previously-working queries.

The common factor seems to be that requests are being issued from GoDaddy egress servers in the 72.167.190.0/24 or in 50.62.176.0/24 IP addresses. Have you blacklisted requests from those IP ranges? If so, would you please consider allowing access again?

My script uses a User-agent: Mozilla/5.0 (get-metar-conditions-inc.php - saratoga-weather.org)

The script can be viewed at https://saratoga-weather.org/get-metar-conditions-inc.php?sce=view

Thanks in advance for any assistance you can provide.

I'll let you know what they say.

mkutche · « **Reply #106 on:** October 02, 2019, 06:10:52 PM »

I ran a Track DNS from godaddy cpanel and got this.. http://prntscr.com/pe2lgl

saratogaWX · « **Reply #107 on:** October 02, 2019, 06:49:40 PM »

Yep.. that's problem (1) above (failure to resolve tgftp.nws.noaa.gov).
If you have terminal access on your cPanel do a

Code: [Select]

cat /etc/resolv.conf and it will id the nameservers having the problem.
You can also do a

Code: [Select]

nslookup -debug tgftp.nws.noaa.gov and it will show the problem also.

Your hostname/ip can be found doing

Code: [Select]

hostname
hostname -i

All that info should be given to GoDaddy tech support to have them resolve the DNS lookup problem.

BTW.. no need to capture images.. just copy/paste the responses as text into a [quote]...[/quote] entry in a post.

gateway2capecod · « **Reply #108 on:** October 02, 2019, 07:15:44 PM »

hi all....

I did the work around fix that jasiu provided a few weeks ago but now today my site is also not getting Metar data once again.....

unable to load KEWB data RC=403 Forbidden is being displayed instead.

I have had Godaddy hosting since E-rice went out as a Host, so it has been quite a few years now.

I have tried calling them and explaining to them a few times and got nowhere with their Stupid Tech Support.

...chris

saratogaWX · « **Reply #109 on:** October 02, 2019, 07:24:03 PM »

Chris,
You're having problem (2) - the 403-Forbidden response

Your site is showing

Quote

and is using the 72.167.190.52 server for GoDaddy egress -- same problem IP range as previously cited.
I've not received a reponse from the NWS yet about my blacklisting query.

saratogaWX · « **Reply #110 on:** October 03, 2019, 11:17:44 AM »

I've done a bit of extra diagnostics to Jasiu's testdns.php script.

Please try the following on your problem websites (and let me know the URL for the test script)

Code: [Select]

<?php
error_reporting(E_ALL);

echo "<p>\n";
echo "Running on " . $_SERVER["SERVER_NAME"] . " (" . $_SERVER["SERVER_ADDR"] . ")\n";
echo "</p>\n";

$NWS = "tgftp.nws.noaa.gov";
  

echo "<p>dns_get_record(\"$NWS\", DNS_A) returns:</p>\n";
$result = dns_get_record($NWS, DNS_A);
echo "<pre>\n";
print var_export($result,true);
echo "</pre>\n";

$hostaddr = gethostbyname($NWS);
echo "<p>gethostbyname(\"$NWS\") returns: '" . $hostaddr . "'</p>\n";

$resolver = file_get_contents('/etc/resolv.conf');
echo "<p>Contents of /etc/resolv.conf nameserver specification</p>\n";
echo "<pre>$resolver</pre>\n";

if(!empty($hostaddr)) {
	echo "<p>Output of 'curl -v https://$NWS/data/observations/metar/stations/KSJC.TXT 2>&1' command</p>\n";
	echo "<pre>\n";
	$curl = system('curl -v https://'.$NWS.'/data/observations/metar/stations/KSJC.TXT 2>&1');
	echo "\n";
	echo "</pre>\n";
} else {
	echo "<p>Curl test skipped as DNS lookup for $NWS failed.</p>\n";
}
?>

ed_heaton · « **Reply #111 on:** October 03, 2019, 11:26:56 AM »

Ken - I loaded mine at www.atglenweather.net/testdns.php

saratogaWX · « **Reply #112 on:** October 03, 2019, 11:41:53 AM »

Thanks Ed,
That shows the problem

Running on www.atglenweather.net (166.62.75.129)

dns_get_record("tgftp.nws.noaa.gov", DNS_A) returns:

false <== indicates failure of the query

gethostbyname("tgftp.nws.noaa.gov") returns: 'tgftp.nws.noaa.gov' <== should be an IP address (140.172.138.79)

Contents of /etc/resolv.conf nameserver specification

# Generated via Puppet
nameserver 72.167.234.213 <== these are the IP addresses of the GoDaddy caching DNS servers with the problem!
nameserver 72.167.234.214 <== and both these servers have the problem!!
options rotate

Oddly enough, the native curl command succeeded (but really slowly) where the PHP gethostbyname() lookup failed!

92merc · « **Reply #113 on:** October 03, 2019, 12:10:53 PM »

Here is my results from my GD server. I'm not having issues, for comparison.

https://www.bismarckweather.net/testdns.php

PaulMy · « **Reply #114 on:** October 03, 2019, 12:30:58 PM »

Maybe I've missed something but just uploaded http://www.komokaweather.ca/testdns.php
and doesn't show much.
Enjoy,Paul

saratogaWX · « **Reply #115 on:** October 03, 2019, 01:02:49 PM »

Paul,
I think that's Jasiu's original script, not the modified one (since it doesn't show the contents of /etc/resolv.conf).
Try uploading it again (the modified script above).

saratogaWX · « **Reply #116 on:** October 03, 2019, 01:05:25 PM »

Quote from: 92merc on October 03, 2019, 12:10:53 PM

Here is my results from my GD server. I'm not having issues, for comparison.

https://www.bismarckweather.net/testdns.php

You're using GoDaddy DNS servers:
# Generated via Puppet
nameserver 10.255.250.30
nameserver 10.255.251.30
options rotate

and they're working. The problem DNS servers (so far) are

# Generated via Puppet
nameserver 72.167.234.213 <== these are the IP addresses of the GoDaddy caching DNS servers with the problem!
nameserver 72.167.234.214 <== and both these servers have the problem!!
options rotate

Thanks for sharing!

saratogaWX · « **Reply #117 on:** October 03, 2019, 01:18:54 PM »

Here's some more interesting stuff about tgftp.nws.noaa.gov .. the actual DNS settings to get an A (address) record involves two CNAME hops:

Code: [Select]

dig @8.8.8.8 tgftp.nws.noaa.gov

; <<>> DiG 9.11.4-P2-RedHat-9.11.4-9.P2.el7 <<>> @8.8.8.8 tgftp.nws.noaa.gov
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 19063
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;tgftp.nws.noaa.gov.            IN      A

;; ANSWER SECTION:
tgftp.nws.noaa.gov.     271     IN      CNAME   tgftp.op.ncep.noaa.gov.
tgftp.op.ncep.noaa.gov. 287     IN      CNAME   tgftp.bldr.ncep.noaa.gov.
tgftp.bldr.ncep.noaa.gov. 21587 IN      A       140.172.138.79

;; Query time: 59 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Thu Oct 03 13:15:57 EDT 2019
;; MSG SIZE  rcvd: 116

Maybe the failing GoDaddy DNS servers are not able to navigate the two CNAME hops to get the A record with the IP address.

As a test, for those having the GoDaddy DNS lookup issue issue, try changing the get-metar-conditions-inc.php script from

Code: [Select]

    $host = 'tgftp.nws.noaa.gov';

to

Code: [Select]

$host = 'tgftp.bldr.ncep.noaa.gov'; and see if that restores the functionality.
This fix won't work for the folks having the 403-Forbidden issue (and may just result in you seeing the 403-Forbidden issue too), but it's worth a try.

Jasiu · « **Reply #118 on:** October 03, 2019, 01:58:29 PM »

Ken,

The first pass as the testdns script had the ability to set a host (still works at e.g. https://www.hsnpar.com/testdns.php?host=radar3pub.ncep.noaa.gov) and pretty much any noaa.gov URL we tried failed in the same way, unfortunately.

PaulMy · « **Reply #119 on:** October 03, 2019, 02:09:27 PM »

Quote from: saratogaWX on October 03, 2019, 01:02:49 PM

Paul,
I think that's Jasiu's original script, not the modified one (since it doesn't show the contents of /etc/resolv.conf).
Try uploading it again (the modified script above).

Thanks Ken,
Trying to find the modified script but no success

Enjoy,
Paul

saratogaWX · « **Reply #120 on:** October 03, 2019, 02:16:02 PM »

Quote from: PaulMy on October 03, 2019, 02:09:27 PM

Quote from: saratogaWX on October 03, 2019, 01:02:49 PM
Paul,
I think that's Jasiu's original script, not the modified one (since it doesn't show the contents of /etc/resolv.conf).
Try uploading it again (the modified script above).
Thanks Ken,
Trying to find the modified script but no success

Enjoy,
Paul

The modified script is in this post.

ed_heaton · « **Reply #121 on:** October 03, 2019, 02:42:59 PM »

Ken - I tried the change ($host = 'tgftp.bldr.ncep.noaa.gov'

and things are working again! Thanks!

saratogaWX · « **Reply #122 on:** October 03, 2019, 03:16:14 PM »

Quote from: ed_heaton on October 03, 2019, 02:42:59 PM

Ken - I tried the change ($host = 'tgftp.bldr.ncep.noaa.gov' and things are working again! Thanks!

I see on your wxmetar.php page that loading is working there too:

Quote

and nicely fast also.

That kinda confirms my suspicion that the GoDaddy caching DNS servers weren't handling the two CNAME lookups correctly and just timing out instead of returning the IP address in an A record. Fault is DEFINITELY with GoDaddy and their DNS servers setup for your particular website hosting.

PaulMy · « **Reply #123 on:** October 03, 2019, 04:55:34 PM »

Thanks again Ken,
By running the updated http://www.komokaweather.ca/testdns.php I get the further details.

So I thought I would try it using our local METAR CYXU instead of KSJC in the script http://www.komokaweather.ca/testdns-1.php and that does not give a similar result.

Enjoy,
Paul
p.s. www.komokaweather.ca/wxmetar.php is still not working

saratogaWX · « **Reply #124 on:** October 03, 2019, 05:09:57 PM »

Paul,
Your website is using the problematic GoDaddy DNS servers:

Contents of /etc/resolv.conf nameserver specification

# Generated via Puppet
nameserver 72.167.234.213
nameserver 72.167.234.214
options rotate

You should try changing the get-metar-conditions-inc.php script from

Code: [Select]

    $host = 'tgftp.nws.noaa.gov';

to

Code: [Select]

$host = 'tgftp.bldr.ncep.noaa.gov'; and see if that restores the functionality.

News:

Author Topic: Local METAR reports Not loading (Read 11695 times)