Author Topic: Ouch, maybe rate limiting to NOAA/NWS sites in future...  (Read 481 times)

0 Members and 1 Guest are viewing this topic.

Offline saratogaWX

  • Administrator
  • Forecaster
  • *****
  • Posts: 8114
  • Saratoga, CA, USA Weather - free PHP scripts
    • Saratoga-Weather.org
Ouch, maybe rate limiting to NOAA/NWS sites in future...
« on: November 18, 2020, 06:08:59 PM »
They're looking for comments at this time...

Announcement: https://www.weather.gov/media/notification/pdf2/pns20-85ncep_web_access.pdf

Quote
NOUS41 KWBC 182006
PNSWSH

Public Information Statement 20-85
National Weather Service Headquarters Silver Spring MD
406 PM EST Wed Nov 18 2020

To:     Subscribers:
     -NOAA Weather Wire Service
     -Emergency Managers Weather Information Network
     -NOAAPort
     Other NWS Partners, Users and Employees

FROM:     Brian Gross
     Acting Director
     National Centers for Environmental Prediction

SUBJECT:  Seeking Comments on Proposed Changes to
User Access of NCEP Web Services through December 18, 2020

The NWS National Centers for Environmental Prediction (NCEP) is
seeking public comments through December 18, 2020, on the
management of user access of the NCEP Web Services.

As demand for data continues to grow across NCEP websites, we
are proposing to put new limits into place to safeguard our web
services. The frequency of how often these websites are accessed
by the public has created limitations and infrastructure
constraints. To add new or upgraded streams of data, there has
to be a reduction in the number of connections into our system.
The mitigation will reduce the strain on our infrastructure,
ensuring a more robust level of service for all users.

For example, a possible mitigation would lower the limit of the
number of connections to 60 per minute for a given user
accessing any of the websites below. This would be a cumulative
limit for any type of data request across any website.
A "head" request will count towards the limit the same as
a "get" request.

Websites affected, both ftp and https, would include:
 nomads.ncep.noaa.gov
 ftp.ncep.noaa.gov
 www.ftp.ncep.noaa.gov
 ftpprd.ncep.noaa.gov
 mag.ncep.noaa.gov
 tgftp.nws.noaa.gov
 aviationweather.gov
 weather.gov
 alerts.weather.gov
 w1.weather.gov
 w2.weather.gov/climate
 w2.weather.gov/dmawds
 airquality.weather.gov
 graphical.weather.gov
 digital.weather.gov
 ra4-gifs.weather.gov
 hamradio.noaa.gov
 weather.gov/spot
 water.weather.gov
 marine.weather.gov
 forecast.weather.gov
 mobile.weather.gov
 ocean.weather.gov
 radar.weather.gov
 api.weather.gov
 tsunami.gov
 madis.ncep.noaa.gov
 madis-data.ncep.noaa.gov
 hads.ncep.noaa.gov
 amdar.ncep.noaa.gov
 mrms.ncep.noaa.gov/data
 water.noaa.gov
 inws.ncep.noaa.gov
 iris.ncep.noaa.gov
 nowcoast.ncep.noaa.gov
 idpgis.ncep.noaa.gov
 Gisc-washington.ncep.noaa.gov
 cts.ncep.noaa.gov
 charts.noaa.gov
 tileservice.charts.noaa.gov
 Radar2pub.ncep.noaa.gov
 Radar3pub.ncep.noaa.gov
 Ocean.ncep.noaa.gov
 www.cpc.ncep.noaa.gov
 vlab.ncep.noaa.gov
 emc.ncep.noaa.gov
 polar.ncep.noaa.gov
 hysplit.ncep.noaa.gov
 ctbto.ncep.noaa.gov
 www.ncep.noaa.gov
 www.nco.ncep.noaa.gov
 www.lib.ncep.noaa.gov
 mageval.ncep.noaa.gov

This list is not all inclusive. Any direct URLs such as "origin,
" or any non-operational URLs such as "dev," "qa," or "preview"
will also be impacted.

NCEP will be holding a virtual public forum to discuss this and
other mitigation options on December 8, 2020, at 2 pm EST.
Please register for the meeting at:
https://register.gotowebinar.com/register/6331491940012140301

Please send questions and comments regarding this mitigation
plan to:

NCO Implementation and Data Services Branch
Carissa Klemmer, Chief
idp.feedback@noaa.gov

We will review any feedback we receive, and will notify users
via a Service Change Notice (SCN) of any operational changes.

National Weather Public Information Statements are online at:
https://www.weather.gov/notification/

This would likely not impact the Saratoga scripts which use caching (advforecast2, get-metar-conditions-inc, nws-alerts, radar-status, etc.), it could effect the performance of scripts which do not have caching.

Personally, I think that if they concentrate on having a robust front-end cache (updated per data refresh rates) in front of their dedicated infrastructure servers, they (and we) would have better service.  The limiting by IP address is a brute-force technique that mostly causes great user dissatisfaction and impedes their mission to deliver timely data.

Note that the weather.gov and tgftp.nws.noaa.gov servers are included and we use them a lot on our websites.
Ken True/Saratoga, CA, USA main site: saratoga-weather.org
Davis VP1+ FARS, Boltek-PCI/NexStorm, microSferics ToA, Blitzortung RED, GRLevel3, WD, WL, VWS, Cumulus, Meteobridge/hub
Free weather PHP scripts/website templates - update notifications on Twitter saratogaWXPHP

Offline WeatherHost

  • Forecaster
  • *****
  • Posts: 3554
Re: Ouch, maybe rate limiting to NOAA/NWS sites in future...
« Reply #1 on: November 18, 2020, 06:23:16 PM »
Kinda torn between 'public access' and abuse of that access.  One reason (among many) I shut down all my sites using their data was that it wasn't necessary with all the media and NWS sites.  It just seemed redundant.  A PWS site with original data is fine.  One that just regurgitates NWS data is a different matter.

I won't be surprised if they do something with all the L3 and L2 radar data feeds.  I hope they don't because I use it quite a bit during storms, but I also respect the 'free' access by turning GR off when no storms are present.



ICDHost, Webhosting:  https://www.icdsoft.com/c/webhost

Offline saratogaWX

  • Administrator
  • Forecaster
  • *****
  • Posts: 8114
  • Saratoga, CA, USA Weather - free PHP scripts
    • Saratoga-Weather.org
Re: Ouch, maybe rate limiting to NOAA/NWS sites in future...
« Reply #2 on: November 18, 2020, 06:32:39 PM »
I agree we should all be frugal in our use of external resources for our personal weather websites.  I think that having a decent forecast (with image icons) and a local sky condition (based on a nearby metar) make our PWS sites generally more useful than just a plain package of our weather station's data readouts.

As Jerry and I found out that the current static radar images from the NWS are being phased out (https://radar.weather.gov/) and their replacement is (currently) a slow and painful mess (https://preview-radar.weather.gov/).  Not sure what to do about that for PWS website scripting quite yet.
Ken True/Saratoga, CA, USA main site: saratoga-weather.org
Davis VP1+ FARS, Boltek-PCI/NexStorm, microSferics ToA, Blitzortung RED, GRLevel3, WD, WL, VWS, Cumulus, Meteobridge/hub
Free weather PHP scripts/website templates - update notifications on Twitter saratogaWXPHP

Offline Jasiu

  • Forecaster
  • *****
  • Posts: 794
    • LexMAWeather
Re: Ouch, maybe rate limiting to NOAA/NWS sites in future...
« Reply #3 on: November 18, 2020, 07:50:30 PM »
Does it count as an "access" if you get a 50x return code?  :?

Offline saratogaWX

  • Administrator
  • Forecaster
  • *****
  • Posts: 8114
  • Saratoga, CA, USA Weather - free PHP scripts
    • Saratoga-Weather.org
Re: Ouch, maybe rate limiting to NOAA/NWS sites in future...
« Reply #4 on: November 19, 2020, 12:03:37 AM »
Hmm... even fail messages will likely 'count' as a message.  :(
Ken True/Saratoga, CA, USA main site: saratoga-weather.org
Davis VP1+ FARS, Boltek-PCI/NexStorm, microSferics ToA, Blitzortung RED, GRLevel3, WD, WL, VWS, Cumulus, Meteobridge/hub
Free weather PHP scripts/website templates - update notifications on Twitter saratogaWXPHP

Offline mcrossley

  • Forecaster
  • *****
  • Posts: 627
    • Wilmslow Astro
Re: Ouch, maybe rate limiting to NOAA/NWS sites in future...
« Reply #5 on: November 19, 2020, 03:23:58 AM »
It seems fair enough to me, 60 accesses per minute would seem to be fair use.

It is incumbent on users to use these sort of resources responsibly and local caching is part of that. I do not use their products but I do use data from the UK Met Office and previously Wund, and like the Saratoga template I implemented caching from day one. It's just the responsible thing to do.

The alternative is that these public bodies spend their scarce money providing additional infrastructure to support people who do not use appropriate techniques.

Opinions are free and often worth as much!
Mark

Online wvdkuil

  • Wim van der kuil
  • Forecaster
  • *****
  • Posts: 1705
    • My PWS at Leuven Belgium Europe
Re: Ouch, maybe rate limiting to NOAA/NWS sites in future...
« Reply #6 on: November 19, 2020, 04:19:51 AM »
IMHO the largest problem lies with checking the IP-address and not the website-name:

For a small test of template users  IP-address, I did a reverse lookup of their website:
Most IP-numbers hosted 30+ websites
Some 300+
and a few "over 1000 websites"
The large numbers are OVH and similar website providers.
But those "inexpensive" providers are frequently used for weather-websites.

The second problem is that one needs to access 3 (forecast) or 4 (with alerts) URL's to get a new-style NWS-point-forecast.
Luckily the detailed 3-hour forecast still needs 1 access.
Although I cache all forecast for an hour, not all users leave the caching as default.
METAR values are valid fo 30 minutes, some stations have 2 or 3 "nearby" METAR stations as not all stations are 24/7 operational.

Wim
« Last Edit: November 19, 2020, 04:26:33 AM by wvdkuil »
Support for Leuven-Template & Scripts  at this forum via https://support.leuven-template.eu/
Script sponsors support.leuven-template.eu/userlistScripts.php
Template sponsors support.leuven-template.eu/userlist.php

Support for the PWS_Dashboard  pwsdashboard.com

Offline gwwilk

  • Southeast Lincoln Weather
  • Forecaster
  • *****
  • Posts: 2361
    • SouthEast Lincoln, NE Weather
Re: Ouch, maybe rate limiting to NOAA/NWS sites in future...
« Reply #7 on: November 19, 2020, 07:38:17 AM »
Have you have registered for the Dec. 8th Webinar, Ken?  If you're willing I think that you could act as a proxy for many of us, particularly those like me who don't have anything like your grasp of the problem and a viable way forward to solve it.  Intermediaries such as Allison House whose radar images I use for my GRLevel3 radar display, might help, and they could be a new 'industry' that helps take the load off of the NOAA-NWS servers.  Just a nascent idea that's only half-formed and maybe half-baked.
Regards, Jerry Wilkins
gwwilk@gmail.com

Offline saratogaWX

  • Administrator
  • Forecaster
  • *****
  • Posts: 8114
  • Saratoga, CA, USA Weather - free PHP scripts
    • Saratoga-Weather.org
Re: Ouch, maybe rate limiting to NOAA/NWS sites in future...
« Reply #8 on: November 19, 2020, 11:03:26 AM »
Yes, I did register for the webinar .. I'll do my best to represent the weather website hobbyist interests from a user and developer perspective.  I hope it's not prophetic that they picked Microsoft Patch Tuesday in December for the webinar.

I, too, use Allison House feeds for my GRLevel3 radar, and I'm sure they'll continue undiminished by the changes since they likely have an agreed-to set of feeds.

Like Wim, I'm concerned that the NWS API canonicalization of data calls has resulted in requiring more-than-one access to retrieve the needed data, each of which counts on a rate-limit scheme.  Wim's points about shared webservers is also good -- with a lot of personal weather websites hosted on shared servers, they generally have one IP address as an egress point for the queries by the scripts, so a rate-limit by IP address could take a bunch of websites out.  If they instead insist on an API key in the query, then it would be possible to separate those queries by requestor.  The API developers for api.weather.gov have intimated that a API key is a future project.
Ken True/Saratoga, CA, USA main site: saratoga-weather.org
Davis VP1+ FARS, Boltek-PCI/NexStorm, microSferics ToA, Blitzortung RED, GRLevel3, WD, WL, VWS, Cumulus, Meteobridge/hub
Free weather PHP scripts/website templates - update notifications on Twitter saratogaWXPHP

Offline galfert

  • Global Moderator
  • Forecaster
  • *****
  • Posts: 5869
Re: Ouch, maybe rate limiting to NOAA/NWS sites in future...
« Reply #9 on: November 19, 2020, 12:46:54 PM »
Are they using a CDN? If not would that not be a solution? Or is that cost prohibitive?

Ecowitt GW1000 | Meteobridge on Raspberry Pi
WU: KFLWINTE111  |  PWSweather: KFLWINTE111
CWOP: FW3708  |  AWEKAS: 14814
Windy: pws-f075acbe
Weather Underground Issue Tracking
Tele-Pole

Offline saratogaWX

  • Administrator
  • Forecaster
  • *****
  • Posts: 8114
  • Saratoga, CA, USA Weather - free PHP scripts
    • Saratoga-Weather.org
Re: Ouch, maybe rate limiting to NOAA/NWS sites in future...
« Reply #10 on: November 19, 2020, 02:08:17 PM »
They're already using Akamai for delivery of many weather.gov sites.  They're having continuing issues with cache refresh (particularly for api.weather.gov) in a timely manner (cache refresh when new data is available, and 50x returns from the cache servers).
Also, they seem to have occassional issues with internal rsync amongst the various internal data sources and sinks and then to cache.  Sigh.  Reading the NCO status is a bit of doomscrolling.
Ken True/Saratoga, CA, USA main site: saratoga-weather.org
Davis VP1+ FARS, Boltek-PCI/NexStorm, microSferics ToA, Blitzortung RED, GRLevel3, WD, WL, VWS, Cumulus, Meteobridge/hub
Free weather PHP scripts/website templates - update notifications on Twitter saratogaWXPHP

Offline WeatherHost

  • Forecaster
  • *****
  • Posts: 3554
Re: Ouch, maybe rate limiting to NOAA/NWS sites in future...
« Reply #11 on: November 20, 2020, 07:49:48 AM »
Is this why the MesoWest site can't keep the 5 minute readings?


ICDHost, Webhosting:  https://www.icdsoft.com/c/webhost