Author Topic: Website slow to load issue with METAR and tgftp.nws.noaa.gov  (Read 345 times)

0 Members and 1 Guest are viewing this topic.

Offline sjforster

  • Member
  • *
  • Posts: 13
    • Westerhope Weather
Website slow to load issue with METAR and tgftp.nws.noaa.gov
« on: August 05, 2017, 02:34:53 PM »
Hi,

Since Thursday, or at least that's when I first noticed, my website has been loading really slowly, and sometimes not at all.  Initially I thought it was the issue with the UNSO but that appears to be back up and running normally now but there's no change to my website www.westerhopeweather.co.uk.

I'm running Ken's Saratoga Template which has been fine since installing it back in March.  The website is hosted by GoDadday on a linux shared server.

When the website eventually loads there's a message on the Ajax dashboard that displays where the METAR current conditions should be that states:

Code: [Select]
unable to load EGNT data RC=
This morning I've managed to undertake some basic troubleshooting which I think has narrowed it down to the tgftp.nws.noaa.gov website where the METAR data is downloaded from.

I've deleted the cached METAR data and its not recreating the cache files. I've manually created a cached file with data from tgftp.nws.noaa.gov and that allowed the website to load a bit quicker but it was still slow compared to normal.

I've checked the version of get-meter-conditions-inc.php and it's the latest Version 1.4.

I know I've made quite a few customisations to the templates so I've also backed up and wiped the website and loaded a complete fresh Saratoga Template and that also had the same issue so I've reloaded the original website back on.

I've accessed the web server via SHH and tried pinging tgftp.nws.noaa.gov but that just returned

Code: [Select]
ping: unknown host tgftp.nws.noaa.gov
I've also tried changing the host in the get-meter-conditions-inc.php from tgftp.nws.noaa.gov to the IP address 140.172.138.79 but that also failed to work and using the IP address in the browser gives an Internal Server Error.

I've also tried a DNS Lookup through cpanel but that also fails with the following error data:

Code: [Select]
<<>> DiG 9.8.2rc1-RedHat-9.8.2-0.62.rc1.el6_9.2 <<>> tgftp.nws.noaa.gov
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 32252
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;tgftp.nws.noaa.gov. IN A

;; Query time: 5879 msec
;; SERVER: 188.121.43.232#53(188.121.43.232)
;; WHEN: Sat Aug  5 11:15:06 2017
;; MSG SIZE  rcvd: 36

Which to me looks like there's a DNS issue on the GoDaddy side of things as I can access the tgftp.nws.noaa.gov site from my computer at home fine and also download the METAR text files.

I've contacted GoDaddy support by phone but they weren't that helpful saying there's nothing they can do with a DNS issue as it's outside their server and they suggested I check the code thoroughly on the website to make sure it's correct.  He did confirm however that he was having intermittent access issues to the tgftp.nws.noaa.gov site.

Looking around the various sites using the Saratoga Template it looks like it's a isolated occurrence with my site but I'm now at a loss of what I can do next to resolve it.

My only thought if there's another site that also has the METAR in text files that can be downloaded via http, but my searches so far have been fruitless.

Any thoughts would be greatly appreciated.

Apologies for the long post but thought it worth going over everything I've done to date to try and resolve it and thanks for getting this far if you're still reading.

Thanks

Simon

Online wvdkuil

  • Wim van der kuil
  • Forecaster
  • *****
  • Posts: 1103
    • Support site Leuven Template and scripts
Re: Website slow to load issue with METAR and tgftp.nws.noaa.gov
« Reply #1 on: August 05, 2017, 03:46:25 PM »
Please check this topic also.  https://www.wxforum.net/index.php?topic=32322.0
That is also a "DNS-problem" with services.swpc.noaa.gov
This is my conclusion about this: https://www.wxforum.net/index.php?topic=32322.msg326480#msg326480

At GoDaddy US and Belgium and Netherlands, they  ignore all complaints.

I now wonder if the servers of GoDaddy are blocked by noaa.gov or that it is really a "cloud / multiple IP problem"
http://dnscheck.pingdom.com/?domain=tgftp.nws.noaa.gov
http://dnscheck.pingdom.com/?domain=www.ndbc.noaa.gov
http://dnscheck.pingdom.com/?domain=services.swpc.noaa.gov
All three same error
Quote
Not enough nameserver information was found to test the zone services.swpc.noaa.gov, but an IP address lookup succeeded in spite of that.
I attach a smal test script.
It tries to read three different files from noaa.gov:
http://services.swpc.noaa.gov/text/3-day-forecast.txt
http://www.ndbc.noaa.gov/data/latest_obs/latest_obs.txt
http://tgftp.nws.noaa.gov/data/observations/metar/stations/EBBR.TXT

All fail when executed from a GoDaddy server, example  at http://www.weerstation-leuven.be/test_curl.php
Running this script takes some time as the wait time is set to 5 seconds, so you have tob wait 15 seconds before you see the 3 failures
Code: [Select]
Following link retrieves swpc and is executed on your server:
http://services.swpc.noaa.gov/text/3-day-forecast.txt
WAIT TIME 5 seconds max
Return codes:

Array
(
    [url] => http://services.swpc.noaa.gov/text/3-day-forecast.txt
    [content_type] =>
    [http_code] => 0
    [header_size] => 0
    [request_size] => 0
    [filetime] => -1
    [ssl_verify_result] => 0
    [redirect_count] => 0
    [total_time] => 5.000038
    [namelookup_time] => 0
    [connect_time] => 0
    [pretransfer_time] => 0
    [size_upload] => 0
    [size_download] => 0
    [speed_download] => 0
    [speed_upload] => 0
    [download_content_length] => -1
    [upload_content_length] => -1
    [starttransfer_time] => 0
    [redirect_time] => 0
    [certinfo] => Array
        (
        )

    [primary_ip] =>
    [redirect_url] =>
)
Error codes: Resolving timed out after 5000 milliseconds

No data was retrieved for http://services.swpc.noaa.gov/text/3-day-forecast.txt

Following link retrieves ndbc and is executed on your server:
http://www.ndbc.noaa.gov/data/latest_obs/latest_obs.txt
WAIT TIME 5 seconds max
Return codes:

Array
(
    [url] => http://www.ndbc.noaa.gov/data/latest_obs/latest_obs.txt
    [content_type] =>
    [http_code] => 0
    [header_size] => 0
    [request_size] => 0
    [filetime] => -1
    [ssl_verify_result] => 0
    [redirect_count] => 0
    [total_time] => 5.00144
    [namelookup_time] => 0
    [connect_time] => 0
    [pretransfer_time] => 0
    [size_upload] => 0
    [size_download] => 0
    [speed_download] => 0
    [speed_upload] => 0
    [download_content_length] => -1
    [upload_content_length] => -1
    [starttransfer_time] => 0
    [redirect_time] => 0
    [certinfo] => Array
        (
        )

    [primary_ip] =>
    [redirect_url] =>
)
Error codes: Resolving timed out after 5001 milliseconds

No data was retrieved for http://www.ndbc.noaa.gov/data/latest_obs/latest_obs.txt

Following link retrieves nws and is executed on your server:
http://tgftp.nws.noaa.gov/data/observations/metar/stations/EBBR.TXT
WAIT TIME 5 seconds max
Return codes:

Array
(
    [url] => http://tgftp.nws.noaa.gov/data/observations/metar/stations/EBBR.TXT
    [content_type] =>
    [http_code] => 0
    [header_size] => 0
    [request_size] => 0
    [filetime] => -1
    [ssl_verify_result] => 0
    [redirect_count] => 0
    [total_time] => 5.001759
    [namelookup_time] => 0
    [connect_time] => 0
    [pretransfer_time] => 0
    [size_upload] => 0
    [size_download] => 0
    [speed_download] => 0
    [speed_upload] => 0
    [download_content_length] => -1
    [upload_content_length] => -1
    [starttransfer_time] => 0
    [redirect_time] => 0
    [certinfo] => Array
        (
        )

    [primary_ip] =>
    [redirect_url] =>
)
Error codes: Resolving timed out after 5001 milliseconds

No data was retrieved for http://tgftp.nws.noaa.gov/data/observations/metar/stations/EBBR.TXT

To check the code of this script, add ?sce=view to the url and the script will display itself

They all work OK when executed on servers of other providers, example https://sluispark.be/test_curl.php
No provider as I am aware of has the same problem. Only GoDaddy has.
All users of my template either have to remove some pages from the menu, or they move to other providers. GoDaddy does not care about that, "your type of hosting account at GoDaddy does not support that". Which is a non-answer for  200 euro /  year  "Ultimate Secure Linux" hosting at GoDaddy.

Please tell GoDaddy to execute the script and investigate the cause of the DNS failures, they did not want to do that last with us,

Wim
« Last Edit: August 05, 2017, 04:07:37 PM by wvdkuil »
Vantage VUE with a WLIP used for https://weer.sluispark.be/
Vantage VP2 with a USB logger/Meteobridge for https://www.weerstation-herent.be/

Offline saratogaWX

  • Administrator
  • Forecaster
  • *****
  • Posts: 6136
  • Saratoga, CA, USA Weather - free PHP scripts
    • Saratoga-Weather.org
Re: Website slow to load issue with METAR and tgftp.nws.noaa.gov
« Reply #2 on: August 05, 2017, 04:27:38 PM »
If you have SSH access to your site try
Code: [Select]
cat /etc/resolv.conf and it will display the nameservers used by the webserver for DNS resolution.  If more than one nameserver entry exists, then they are searched in order (with a timeout usually of 2 to 4 seconds each).  Since Wim's code shows it is a 'Resolving timed out after 5001 milliseconds' issue, it is DEFINITELY in their internal network DNS lashup.  Call and insist that they log on the webserver via SSH and try 'dig' for the several domains.. they (the server admins) have to do it as the command is restricted from mere mortal customers.  At least, that gives the server admins a place to start diagnosing the problem with their DNS structure.
Ken True/Saratoga, CA, USA main site: saratoga-weather.org
Davis VP1+ FARS, Boltek-PCI/NexStorm, microSferics ToA, Blitzortung RED, GRLevel3, WD, WL, VWS, Cumulus, Meteobridge/hub
Free weather PHP scripts/website templates - update notifications on Twitter saratogaWXPHP

Offline sjforster

  • Member
  • *
  • Posts: 13
    • Westerhope Weather
Re: Website slow to load issue with METAR and tgftp.nws.noaa.gov
« Reply #3 on: August 05, 2017, 04:36:02 PM »
Wim / Ken,

Thanks for taking the time to assist.  I'll contact GoDaddy again in the morning and try the cat command and post back.

As far as I know this issue only started Thursday on my site and had been working flawlessly before that.

One thing when I run the script it returns in 11 seconds despite say all three timed out at 5001ms.  Does that mean it's getting a not available response before it times out?

Regards,

Simon

Offline sjforster

  • Member
  • *
  • Posts: 13
    • Westerhope Weather
Re: Website slow to load issue with METAR and tgftp.nws.noaa.gov
« Reply #4 on: August 05, 2017, 06:59:12 PM »
Hi Ken,

I've logged in via SSH and ran cat /etc/resolv.conf the response I got was:

Code: [Select]
# Generated via Puppet
nameserver 188.121.43.231
nameserver 188.121.43.232
options rotate

So there's 2 DNS name server entries.  I'll ask about dig, with regards to the different sites, is that the three sites Wim's script is accessing, as below?

http://services.swpc.noaa.gov/text/3-day-forecast.txt
http://www.ndbc.noaa.gov/data/latest_obs/latest_obs.txt
http://tgftp.nws.noaa.gov/data/observations/metar/stations/EBBR.TXT

WIM, with regards to the dynamic IP address every time I try to resolve tgftp.nws.noaa.gov from my home computer it comes back as 140.172.138.79 so from my perspective that doesn't appear to be altering.

Regards,

Simon



Offline saratogaWX

  • Administrator
  • Forecaster
  • *****
  • Posts: 6136
  • Saratoga, CA, USA Weather - free PHP scripts
    • Saratoga-Weather.org
Re: Website slow to load issue with METAR and tgftp.nws.noaa.gov
« Reply #5 on: August 05, 2017, 07:03:55 PM »
For the 'dig' command, it would be
Code: [Select]
dig services.swpc.noaa.gov
dig www.ndbc.noaa.gov
dig tgftp.nws.noaa.gov

On my VPS system that results in
Quote
bash-4.1$ dig services.swpc.noaa.gov

; <<>> DiG 9.8.2rc1-RedHat-9.8.2-0.62.rc1.el6_9.4 <<>> services.swpc.noaa.gov
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 16930
;; flags: qr rd ra; QUERY: 1, ANSWER: 12, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;services.swpc.noaa.gov.                IN      A

;; ANSWER SECTION:
services.swpc.noaa.gov. 41616   IN      CNAME   edge-nws.woc.noaa.gov.
edge-nws.woc.noaa.gov.  207     IN      CNAME   edge-p1.l.noaa.gov.
edge-p1.l.noaa.gov.     30      IN      A       140.90.200.11
edge-p1.l.noaa.gov.     30      IN      A       140.172.17.21
edge-p1.l.noaa.gov.     30      IN      A       129.15.96.21
edge-p1.l.noaa.gov.     30      IN      A       129.15.96.11
edge-p1.l.noaa.gov.     30      IN      A       216.38.80.81
edge-p1.l.noaa.gov.     30      IN      A       216.38.80.71
edge-p1.l.noaa.gov.     30      IN      A       140.90.33.21
edge-p1.l.noaa.gov.     30      IN      A       140.90.33.11
edge-p1.l.noaa.gov.     30      IN      A       140.90.200.21
edge-p1.l.noaa.gov.     30      IN      A       140.172.17.11

;; Query time: 76 msec
;; SERVER: 10.255.255.3#53(10.255.255.3)
;; WHEN: Sat Aug  5 19:02:37 2017
;; MSG SIZE  rcvd: 251

bash-4.1$ dig www.ndbc.noaa.gov

; <<>> DiG 9.8.2rc1-RedHat-9.8.2-0.62.rc1.el6_9.4 <<>> www.ndbc.noaa.gov
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 33476
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;www.ndbc.noaa.gov.             IN      A

;; ANSWER SECTION:
www.ndbc.noaa.gov.      300     IN      CNAME   www.ndbc.l.noaa.gov.
www.ndbc.l.noaa.gov.    30      IN      A       140.90.238.27

;; Query time: 1521 msec
;; SERVER: 10.255.255.3#53(10.255.255.3)
;; WHEN: Sat Aug  5 19:02:38 2017
;; MSG SIZE  rcvd: 76

bash-4.1$ dig tgftp.nws.noaa.gov

; <<>> DiG 9.8.2rc1-RedHat-9.8.2-0.62.rc1.el6_9.4 <<>> tgftp.nws.noaa.gov
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 39046
;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;tgftp.nws.noaa.gov.            IN      A

;; ANSWER SECTION:
tgftp.nws.noaa.gov.     78      IN      CNAME   tgftp.op.ncep.noaa.gov.
tgftp.op.ncep.noaa.gov. 78      IN      CNAME   tgftp.bldr.ncep.noaa.gov.
tgftp.bldr.ncep.noaa.gov. 86178 IN      A       140.172.138.79

;; Query time: 0 msec
;; SERVER: 10.255.255.3#53(10.255.255.3)
;; WHEN: Sat Aug  5 19:02:44 2017
;; MSG SIZE  rcvd: 105

bash-4.1$
which happens in a couple of seconds.  Likely, they'll see a timeout message (and that's the problem they have to fix).
Ken True/Saratoga, CA, USA main site: saratoga-weather.org
Davis VP1+ FARS, Boltek-PCI/NexStorm, microSferics ToA, Blitzortung RED, GRLevel3, WD, WL, VWS, Cumulus, Meteobridge/hub
Free weather PHP scripts/website templates - update notifications on Twitter saratogaWXPHP

Offline sjforster

  • Member
  • *
  • Posts: 13
    • Westerhope Weather
Re: Website slow to load issue with METAR and tgftp.nws.noaa.gov
« Reply #6 on: August 05, 2017, 07:22:40 PM »
Ken,

Thanks for confirming.

I'll ask the question tomorrow with GoDaddy support, seems a fairly simple request, although from WIM's previous experience I somehow don't hold out much hope.

From your dig commands I notice that the tgftp.nws.noaa.gov also resolves to 140.172.138.79 but goes through three changes of name to get there.  Is that also part of the issue?  Sorry for the basic questions, I'm no expert.

Regards,

Simon

Offline saratogaWX

  • Administrator
  • Forecaster
  • *****
  • Posts: 6136
  • Saratoga, CA, USA Weather - free PHP scripts
    • Saratoga-Weather.org
Re: Website slow to load issue with METAR and tgftp.nws.noaa.gov
« Reply #7 on: August 05, 2017, 07:35:29 PM »
The other CNAME entries are 'Canonical Name' DNS entries (those map an alias name to a true or canonical domain name. CNAME records are typically used to map a subdomain such as www or mail to the domain hosting that subdomain's content.)  In the tgftp.nws.noaa.gov , there are 2 CNAME records that cause access to those names to land here with the return of the A (address) record.   A nslookup returns the A record IP address, and that's what curl is using to establish (or try to establish) a connection -- the CNAME records don't matter in this case.

But with your GoDaddy server, the DNS resolution is broken and so upstream lookups are failing to return an IP address to use.

dig is just the diagnostic version of a nslookup .. it returns much more info and a handy timing for how long the query took.
Ken True/Saratoga, CA, USA main site: saratoga-weather.org
Davis VP1+ FARS, Boltek-PCI/NexStorm, microSferics ToA, Blitzortung RED, GRLevel3, WD, WL, VWS, Cumulus, Meteobridge/hub
Free weather PHP scripts/website templates - update notifications on Twitter saratogaWXPHP

Offline sjforster

  • Member
  • *
  • Posts: 13
    • Westerhope Weather
Re: Website slow to load issue with METAR and tgftp.nws.noaa.gov
« Reply #8 on: August 05, 2017, 08:03:08 PM »
Ken,

Thanks for the explanation.  I'll let you know how my discussions with GoDaddy go.

Regards,

Simon

Offline mldenison

  • Member
  • *
  • Posts: 30
    • York, PA Weather
Re: Website slow to load issue with METAR and tgftp.nws.noaa.gov
« Reply #9 on: August 05, 2017, 10:34:06 PM »
Good luck.  Wim diagnosed the timeout problem several months ago for me.  I spent countless hours with GoDaddy tech support and never once did they concede they had a DNS issue with those web sites.  Nor would they escalate past their first tier support.  Which is just about worthless, considering what I pay every year.
Windows 7 x64
Weather PC: Intel NUC
Davis Vantage Pro 2
Weather Underground: KPAYORK30
York, PA Weather (Saratoga)
York, PA Weather (Leuven)

Offline sjforster

  • Member
  • *
  • Posts: 13
    • Westerhope Weather
Re: Website slow to load issue with METAR and tgftp.nws.noaa.gov
« Reply #10 on: August 06, 2017, 06:13:56 AM »
Well after just under an hour on the phone with GoDaddy Support they're basically saying it's the website code that's the issue not their servers.  They loaded a default website to prove it.......  I couldn't make them understand that their default website wasn't requesting data from tgftp.nws.noaa.gov so no wonder it worked which led to some funny conversations about redirecting to different providers I still don't think they understood that it was trying to pull data from other websites.

When I tried to explain about running a DNS lookup from cPanel to tgftp.nws.noaa.gov they just said it was an issue with the code or the other website, definetly not theirs, couldn't make them see that when running the DNS lookup from the cPanel it's completley bypassing the code on the website.

I asked about running a dig command via SSH on the server to the website addresses but was told that was outside of the support agents scope and they weren't intrested in the test script.

They also stated my redirects (.com to .co.uk and http to https) in the .htaccess where causing an issue with it loading slow.  I've tried it with the redirects removed and it's still the same.

Don't know what I expected really, it's inline with WIM's and mldension experience but now I've got nowhere left to go other than either put up with / edit the website to remove the calls to tgftp.nws.noaa.gov or move to another hosting provider that doesn't have the same issue.  I asked about a refund for the remaining part of the service I've paid for which just got the stock response of it's outside of policy and the 30 day window for cancellation.

Thanks for everbody's assitance on this it's much appreciated. Shame GoDaddy can't be as helpful.

Simon

Offline sjforster

  • Member
  • *
  • Posts: 13
    • Westerhope Weather
Re: Website slow to load issue with METAR and tgftp.nws.noaa.gov
« Reply #11 on: August 06, 2017, 11:49:52 AM »
After messing around most of the day trying to switch off the METAR part of the dashboard it would appear that Ken has already got that covered also.

 :oops: It's as simple as commenting out the $SITE['conditionsMETAR'] = 'EGNT' statement in Setting-weather.php as below

Code: [Select]
#$SITE['conditionsMETAR'] = 'EGNT';  // set to nearby METAR for current conditions icon/text
#  comment out conditionsMETAR if no nearby METAR.. conditions icon/text will not be displayed

Site's back to loading fast but without the METAR Icon and text.

Thanks Ken.

Regards,

Simon