WXforum.net
Weather Related Organizations => NOAA National Weather Service => Topic started by: saratogaWX on November 18, 2020, 06:08:59 PM
-
They're looking for comments at this time...
Announcement: https://www.weather.gov/media/notification/pdf2/pns20-85ncep_web_access.pdf
NOUS41 KWBC 182006
PNSWSH
Public Information Statement 20-85
National Weather Service Headquarters Silver Spring MD
406 PM EST Wed Nov 18 2020
To: Subscribers:
-NOAA Weather Wire Service
-Emergency Managers Weather Information Network
-NOAAPort
Other NWS Partners, Users and Employees
FROM: Brian Gross
Acting Director
National Centers for Environmental Prediction
SUBJECT: Seeking Comments on Proposed Changes to
User Access of NCEP Web Services through December 18, 2020
The NWS National Centers for Environmental Prediction (NCEP) is
seeking public comments through December 18, 2020, on the
management of user access of the NCEP Web Services.
As demand for data continues to grow across NCEP websites, we
are proposing to put new limits into place to safeguard our web
services. The frequency of how often these websites are accessed
by the public has created limitations and infrastructure
constraints. To add new or upgraded streams of data, there has
to be a reduction in the number of connections into our system.
The mitigation will reduce the strain on our infrastructure,
ensuring a more robust level of service for all users.
For example, a possible mitigation would lower the limit of the
number of connections to 60 per minute for a given user
accessing any of the websites below. This would be a cumulative
limit for any type of data request across any website.
A "head" request will count towards the limit the same as
a "get" request.
Websites affected, both ftp and https, would include:
nomads.ncep.noaa.gov
ftp.ncep.noaa.gov
www.ftp.ncep.noaa.gov
ftpprd.ncep.noaa.gov
mag.ncep.noaa.gov
tgftp.nws.noaa.gov
aviationweather.gov
weather.gov
alerts.weather.gov
w1.weather.gov
w2.weather.gov/climate
w2.weather.gov/dmawds
airquality.weather.gov
graphical.weather.gov
digital.weather.gov
ra4-gifs.weather.gov
hamradio.noaa.gov
weather.gov/spot
water.weather.gov
marine.weather.gov
forecast.weather.gov
mobile.weather.gov
ocean.weather.gov
radar.weather.gov
api.weather.gov
tsunami.gov
madis.ncep.noaa.gov
madis-data.ncep.noaa.gov
hads.ncep.noaa.gov
amdar.ncep.noaa.gov
mrms.ncep.noaa.gov/data
water.noaa.gov
inws.ncep.noaa.gov
iris.ncep.noaa.gov
nowcoast.ncep.noaa.gov
idpgis.ncep.noaa.gov
Gisc-washington.ncep.noaa.gov
cts.ncep.noaa.gov
charts.noaa.gov
tileservice.charts.noaa.gov
Radar2pub.ncep.noaa.gov
Radar3pub.ncep.noaa.gov
Ocean.ncep.noaa.gov
www.cpc.ncep.noaa.gov
vlab.ncep.noaa.gov
emc.ncep.noaa.gov
polar.ncep.noaa.gov
hysplit.ncep.noaa.gov
ctbto.ncep.noaa.gov
www.ncep.noaa.gov
www.nco.ncep.noaa.gov
www.lib.ncep.noaa.gov
mageval.ncep.noaa.gov
This list is not all inclusive. Any direct URLs such as "origin,
" or any non-operational URLs such as "dev," "qa," or "preview"
will also be impacted.
NCEP will be holding a virtual public forum to discuss this and
other mitigation options on December 8, 2020, at 2 pm EST.
Please register for the meeting at:
https://register.gotowebinar.com/register/6331491940012140301
Please send questions and comments regarding this mitigation
plan to:
NCO Implementation and Data Services Branch
Carissa Klemmer, Chief
idp.feedback@noaa.gov
We will review any feedback we receive, and will notify users
via a Service Change Notice (SCN) of any operational changes.
National Weather Public Information Statements are online at:
https://www.weather.gov/notification/
This would likely not impact the Saratoga scripts which use caching (advforecast2, get-metar-conditions-inc, nws-alerts, radar-status, etc.), it could effect the performance of scripts which do not have caching.
Personally, I think that if they concentrate on having a robust front-end cache (updated per data refresh rates) in front of their dedicated infrastructure servers, they (and we) would have better service. The limiting by IP address is a brute-force technique that mostly causes great user dissatisfaction and impedes their mission to deliver timely data.
Note that the weather.gov and tgftp.nws.noaa.gov servers are included and we use them a lot on our websites.
-
Kinda torn between 'public access' and abuse of that access. One reason (among many) I shut down all my sites using their data was that it wasn't necessary with all the media and NWS sites. It just seemed redundant. A PWS site with original data is fine. One that just regurgitates NWS data is a different matter.
I won't be surprised if they do something with all the L3 and L2 radar data feeds. I hope they don't because I use it quite a bit during storms, but I also respect the 'free' access by turning GR off when no storms are present.
-
I agree we should all be frugal in our use of external resources for our personal weather websites. I think that having a decent forecast (with image icons) and a local sky condition (based on a nearby metar) make our PWS sites generally more useful than just a plain package of our weather station's data readouts.
As Jerry and I found out that the current static radar images from the NWS are being phased out (https://radar.weather.gov/) and their replacement is (currently) a slow and painful mess (https://preview-radar.weather.gov/). Not sure what to do about that for PWS website scripting quite yet.
-
Does it count as an "access" if you get a 50x return code? :?
-
Hmm... even fail messages will likely 'count' as a message. :(
-
It seems fair enough to me, 60 accesses per minute would seem to be fair use.
It is incumbent on users to use these sort of resources responsibly and local caching is part of that. I do not use their products but I do use data from the UK Met Office and previously Wund, and like the Saratoga template I implemented caching from day one. It's just the responsible thing to do.
The alternative is that these public bodies spend their scarce money providing additional infrastructure to support people who do not use appropriate techniques.
Opinions are free and often worth as much!
-
IMHO the largest problem lies with checking the IP-address and not the website-name:
For a small test of template users IP-address, I did a reverse lookup of their website:
Most IP-numbers hosted 30+ websites
Some 300+
and a few "over 1000 websites"
The large numbers are OVH and similar website providers.
But those "inexpensive" providers are frequently used for weather-websites.
The second problem is that one needs to access 3 (forecast) or 4 (with alerts) URL's to get a new-style NWS-point-forecast. (https://www.weerstation-herent.be/wsfct4/wsFctNoaaPPage4.php)
Luckily the detailed 3-hour forecast (https://www.weerstation-herent.be/wsfct4/wsFctNoaaDPage4.php) still needs 1 access.
Although I cache all forecast for an hour, not all users leave the caching as default.
METAR values are valid fo 30 minutes, some stations have 2 or 3 "nearby" METAR stations as not all stations are 24/7 operational.
Wim
-
Have you have registered for the Dec. 8th Webinar, Ken? If you're willing I think that you could act as a proxy for many of us, particularly those like me who don't have anything like your grasp of the problem and a viable way forward to solve it. Intermediaries such as Allison House whose radar images I use for my GRLevel3 radar display, might help, and they could be a new 'industry' that helps take the load off of the NOAA-NWS servers. Just a nascent idea that's only half-formed and maybe half-baked.
-
Yes, I did register for the webinar .. I'll do my best to represent the weather website hobbyist interests from a user and developer perspective. I hope it's not prophetic that they picked Microsoft Patch Tuesday in December for the webinar.
I, too, use Allison House feeds for my GRLevel3 radar, and I'm sure they'll continue undiminished by the changes since they likely have an agreed-to set of feeds.
Like Wim, I'm concerned that the NWS API canonicalization of data calls has resulted in requiring more-than-one access to retrieve the needed data, each of which counts on a rate-limit scheme. Wim's points about shared webservers is also good -- with a lot of personal weather websites hosted on shared servers, they generally have one IP address as an egress point for the queries by the scripts, so a rate-limit by IP address could take a bunch of websites out. If they instead insist on an API key in the query, then it would be possible to separate those queries by requestor. The API developers for api.weather.gov have intimated that a API key is a future project.
-
Are they using a CDN? If not would that not be a solution? Or is that cost prohibitive?
-
They're already using Akamai for delivery of many weather.gov sites. They're having continuing issues with cache refresh (particularly for api.weather.gov) in a timely manner (cache refresh when new data is available, and 50x returns from the cache servers).
Also, they seem to have occassional issues with internal rsync amongst the various internal data sources and sinks and then to cache. Sigh. Reading the NCO status is a bit of doomscrolling.
-
Is this why the MesoWest site can't keep the 5 minute readings?