Author Topic: api.weather.gov down?  (Read 1903 times)

0 Members and 1 Guest are viewing this topic.

Offline daneast

  • Member
  • *
  • Posts: 48
api.weather.gov down?
« on: January 01, 2019, 08:26:22 AM »
I'm getting 502 Bad Gateway when attempting to access anything at api.weather.gov.

I'm not sure if this was because of changes set to take affect Jan 1 with the API, or because of the partial government shutdown.  It appeared to start at exactly 04:00 EST, so it has been down for 4.5 hours now.

When I pull up a community at forecast.weather.gov all the forecast data appears to be there as normal, so this looks like purely an API related thing (IE they are still making forecasts).

Anyone know what is going on?

Offline Jasiu

  • Forecaster
  • *****
  • Posts: 947
    • LexMAWeather
Re: api.weather.gov down?
« Reply #1 on: January 01, 2019, 08:41:53 AM »
Seems to be consistently giving 502s. I sent a tweet to my local office - they are usually pretty responsive. If / when I hear back I'll let you know.
https://lexmaweather.info
On Mastodon: @LexMAWeather@toot.community

Offline ValentineWeather

  • Forecaster
  • *****
  • Posts: 6364
    • Valentine Nebraska's Real-Time Weather
Re: api.weather.gov down?
« Reply #2 on: January 01, 2019, 08:45:15 AM »
Just getting this.

Sorry.. the forecast for Valentine is not available at this time.

NWS server https://api.weather.gov/zones/JSON-LD/NEZ005/forecast has an error.

Getting BAD GATEWAY
« Last Edit: January 01, 2019, 08:47:31 AM by ValentineWeather »
Randy

Offline ValentineWeather

  • Forecaster
  • *****
  • Posts: 6364
    • Valentine Nebraska's Real-Time Weather
Re: api.weather.gov down?
« Reply #3 on: January 01, 2019, 09:08:31 AM »
Looks like both are now working and server back up....This version is working.........$Version = 'advforecast-json.php (multi) - V5.00 (beta)- 21-Jan-2017';
This version will not get the 502 error...$Version = 'advforecast2.php (JSON) - V5.09 - 26-May-2018';
« Last Edit: January 01, 2019, 09:13:51 AM by ValentineWeather »
Randy

Offline daneast

  • Member
  • *
  • Posts: 48
Re: api.weather.gov down?
« Reply #4 on: January 01, 2019, 10:40:31 AM »
It's working for me now as well.  Came back up just after 8:55 EST time.  So it was down from 04:00 - 08:55 EST.

The timing with it being a new year, and the partial government shutdown was interesting.  I have a hunch it was some kind of year rollover bug.  It happened at 01:00 GMT.

Offline miraculon

  • Sunrise Side Weather
  • Forecaster
  • *****
  • Posts: 4107
  • KE8DAF
    • Sunrise Side Weather in Rogers City MI USA
Re: api.weather.gov down?
« Reply #5 on: February 05, 2020, 08:35:41 AM »
I am getting this 502 Bad Gateway error at https://api.weather.gov/zones/forecast/MIZ018/forecast. Friom the history on this thread, it seems to be a recurring thing.

Anyone else getting this for the NWS forecasts currently?

Greg H.


Blitzortung Stations #706 and #1682
CoCoRaHS: MI-PI-1
CWOP: CW4114 and KE8DAF-13
WU: KMIROGER7
Amateur Radio Callsign: KE8DAF

Offline miraculon

  • Sunrise Side Weather
  • Forecaster
  • *****
  • Posts: 4107
  • KE8DAF
    • Sunrise Side Weather in Rogers City MI USA
Re: api.weather.gov down?
« Reply #6 on: February 05, 2020, 08:36:52 AM »
Never mind. As soon as I posted this, I tried again and it was working. Figures....

Greg H.


Blitzortung Stations #706 and #1682
CoCoRaHS: MI-PI-1
CWOP: CW4114 and KE8DAF-13
WU: KMIROGER7
Amateur Radio Callsign: KE8DAF

Offline Maumelle Weather

  • Forecaster
  • *****
  • Posts: 1824
    • Maumelle Weather
Re: api.weather.gov down?
« Reply #7 on: February 05, 2020, 08:42:51 AM »
I was about to ask the same question, Greg. Mine just started working again as well.
GR2AE, GR3, Cumulus

Offline daneast

  • Member
  • *
  • Posts: 48
Re: api.weather.gov down?
« Reply #8 on: February 05, 2020, 11:27:40 AM »
It's intermittent, and still happening for me.  Some calls fail, others are processed.

The NWS had a major outage a couple weeks ago.  Cooling systems went out at a data center in CO, and HTTP errors started happening intermittently as their capacity to handle volume decreased. Eventually the new API went down completely.  Apparently their fallover systems did not work correctly to transfer processing to other centers.  Then the new API didn't come back online like it should.  Any existing weather alerts that occurred during the outage were not in the API data when it came back online. Only new alerts from that point forward came through.

So they have some serious deficiencies for sure in their redundancy and ability to handle outages at key data centers.  I'm not sure what these current failures are about.

Offline Jasiu

  • Forecaster
  • *****
  • Posts: 947
    • LexMAWeather
Re: api.weather.gov down?
« Reply #9 on: February 05, 2020, 11:54:22 AM »
Appears to have almost completely gone down again. Timeouts, 502 errors....

They do know about it.
https://lexmaweather.info
On Mastodon: @LexMAWeather@toot.community

Offline HamiltonNJWX

  • Senior Member
  • **
  • Posts: 60
    • HamiltonWx
Re: api.weather.gov down?
« Reply #10 on: February 05, 2020, 12:21:00 PM »
Just noticed my forecast from NWS down as well. Thanks for posting the info that y’all have come across so far.

Offline Bunty

  • Forecaster
  • *****
  • Posts: 2429
  • Stillwater, home of Oklahoma State University
    • Welcome to Stillwater Weather
Re: api.weather.gov down?
« Reply #11 on: February 05, 2020, 05:27:12 PM »
Same here.  Changing to other version didn't help for long.

Also the 2nd home page using modified AltDashboard 6.95 at http://stillwaterweather.com/2ndhome.php

Offline saratogaWX

  • Administrator
  • Forecaster
  • *****
  • Posts: 9257
  • Saratoga, CA, USA Weather - free PHP scripts
    • Saratoga-Weather.org
Re: api.weather.gov down?
« Reply #12 on: February 05, 2020, 05:43:57 PM »
Both old/new are back for my area.  I noticed the issues earlier this morning.. I think it was due to back-end data porting from the models to the database for the API.  Sadly, those issues seem to continue happening with the NWS servers.
Ken True/Saratoga, CA, USA main site: saratoga-weather.org
Davis VP1+ FARS, Blitzortung RED, GRLevel3, WD, WL, VWS, Cumulus, Meteobridge
Free weather PHP scripts/website templates - update notifications on Twitter saratogaWXPHP

Offline Jasiu

  • Forecaster
  • *****
  • Posts: 947
    • LexMAWeather
Re: api.weather.gov down?
« Reply #13 on: February 05, 2020, 08:36:51 PM »
Mail thread I have says they've moved the API services from College Park to Boulder (if I'm reading it right) and they are testing a fix there, but I'm still getting a slew of 502 errors as of a few minutes ago.
https://lexmaweather.info
On Mastodon: @LexMAWeather@toot.community

Offline Jasiu

  • Forecaster
  • *****
  • Posts: 947
    • LexMAWeather
Re: api.weather.gov down?
« Reply #14 on: February 07, 2020, 10:45:06 AM »
Latest from where I'm at: Still seeing a lot of timeouts and stale data. No 502 errors anymore though.

One case that had me banging my head on the desk for a while... My site is showing stale data but if I fetch the URL from my browser, it's current. Tracked it down to different DNS resolutions for api.weather.gov - so not all of the caches are in sync. Sigh.

I reported the latter problem.
https://lexmaweather.info
On Mastodon: @LexMAWeather@toot.community

Offline saratogaWX

  • Administrator
  • Forecaster
  • *****
  • Posts: 9257
  • Saratoga, CA, USA Weather - free PHP scripts
    • Saratoga-Weather.org
Re: api.weather.gov down?
« Reply #15 on: February 07, 2020, 03:52:44 PM »
Well, it's kinda a tortuous path from api.weather.gov to the Akamai services proxying the actual NWS services providing the data.  Here's the dig traces

Quote
02/07/20 12:44:29 dig api.weather.gov @ 208.67.222.222
Dig api.weather.gov@208.67.222.222 ...
Non-authoritative answer
Recursive queries supported by this server
 Query for api.weather.gov type=255 class=1
  api.weather.gov CNAME (Canonical Name) sancert.weather.gov.edgekey.net  <=== first redirect


02/07/20 12:44:43 dig sancert.weather.gov.edgekey.net @ 208.67.222.222
Dig sancert.weather.gov.edgekey.net@208.67.222.222 ...
Non-authoritative answer
Recursive queries supported by this server
 Query for sancert.weather.gov.edgekey.net type=255 class=1
  sancert.weather.gov.edgekey.net CNAME (Canonical Name) e278.dscg.akamaiedge.net  <=== second redirect


02/07/20 12:44:55 dig e278.dscg.akamaiedge.net @ 208.67.222.222
Dig e278.dscg.akamaiedge.net@208.67.222.222 ...
Non-authoritative answer
Recursive queries supported by this server
 Query for e278.dscg.akamaiedge.net type=255 class=1
  e278.dscg.akamaiedge.net A (Address) 104.84.224.50                                       <=== ahh.. IP addresses at last
  e278.dscg.akamaiedge.net AAAA (IPv6 Address) 2600:1406:1400:589:0:0:0:116
  e278.dscg.akamaiedge.net AAAA (IPv6 Address) 2600:1406:1400:584:0:0:0:116


02/07/20 12:46:16 dig dscg.akamaiedge.net @ 208.67.222.222
Dig dscg.akamaiedge.net@208.67.222.222 ...
Non-authoritative answer
Recursive queries supported by this server
 Query for dscg.akamaiedge.net type=255 class=1
  dscg.akamaiedge.net SOA (Zone of Authority)
        Primary NS: n0dscg.akamaiedge.net
        Responsible person: hostmaster@akamai.com
        serial:1581108097
        refresh:1000s (160 minutes)      <=== reasonably speedy timeouts so changes should be propagated fairly quickly.
        retry:1000s (160 minutes)
        expire:1000s (160 minutes)
        minimum-ttl:1800s (30 minutes)

Ken True/Saratoga, CA, USA main site: saratoga-weather.org
Davis VP1+ FARS, Blitzortung RED, GRLevel3, WD, WL, VWS, Cumulus, Meteobridge
Free weather PHP scripts/website templates - update notifications on Twitter saratogaWXPHP

Offline Jasiu

  • Forecaster
  • *****
  • Posts: 947
    • LexMAWeather
Re: api.weather.gov down?
« Reply #16 on: February 13, 2020, 07:44:49 AM »
Via email this morning:

Quote
Due to a planned upgrade last week there were unintended consequences to the API and it was down for several days. We restored services to 100% yesterday. I'll request that if you are still seeing any issues to notify Tech Control, but at this point I'll close this specific ticket since we have restored services. We apologize for the inconvenience this caused.
https://lexmaweather.info
On Mastodon: @LexMAWeather@toot.community