Author Topic: NWS Hourly forecast data  (Read 1172 times)

0 Members and 1 Guest are viewing this topic.

Offline daneast

  • Member
  • *
  • Posts: 48
NWS Hourly forecast data
« on: December 28, 2018, 11:04:41 AM »
I thought I had everything I needed from the new API's NWS hourly data to make a transition from the Weather Underground API.  However, the information returned from the NWS API is rather sparse.  For example, it does not include the precipitation percentage.  Looking at this page, the NWS appears to have a great deal of hourly data, including precipitation potential:
https://forecast.weather.gov/MapClick.php?lat=36.9537&lon=-81.087&unit=0&lg=english&FcstType=graphical

So it looks like the new API doesn't have that in the hourly data.  Looking at the older XML API I don't see hourly data like this.

Does anyone know if it is available in the new API, and if not how do I access it in the XML REST API?

Thanks!

Offline saratogaWX

  • Administrator
  • Forecaster
  • *****
  • Posts: 9257
  • Saratoga, CA, USA Weather - free PHP scripts
    • Saratoga-Weather.org
Re: NWS Hourly forecast data
« Reply #1 on: December 28, 2018, 02:38:37 PM »
Actually, that data does exist as a query on api.weather.gov (for /points/{lat,long}/forecast/hourly)  which returns JSON like
Code: [Select]
{
    "@context": {
        "wx": "https://api.weather.gov/ontology#",
        "geo": "http://www.opengis.net/ont/geosparql#",
        "unit": "http://codes.wmo.int/common/unit/",
        "@vocab": "https://api.weather.gov/ontology#"
    },
    "geometry": "GEOMETRYCOLLECTION(POINT(-122.0220167 37.2668315),POLYGON((-122.0385572 37.275548,-122.0330058 37.2536879,-122.005479 37.2581132,-122.011025 37.2799738,-122.0385572 37.275548)))",
    "updated": "2018-12-28T17:11:06+00:00",
    "units": "us",
    "forecastGenerator": "HourlyForecastGenerator",
    "generatedAt": "2018-12-28T18:26:47+00:00",
    "updateTime": "2018-12-28T17:11:06+00:00",
    "validTimes": "2018-12-28T11:00:00+00:00/P8DT2H",
    "elevation": {
        "value": 121.0056,
        "unitCode": "unit:m"
    },
    "periods": [
        {
            "number": 1,
            "name": "",
            "startTime": "2018-12-28T10:00:00-08:00",
            "endTime": "2018-12-28T11:00:00-08:00",
            "isDaytime": true,
            "temperature": 48,
            "temperatureUnit": "F",
            "temperatureTrend": null,
            "windSpeed": "12 mph",
            "windDirection": "NNW",
            "icon": "https://api.weather.gov/icons/land/day/few?size=small",
            "shortForecast": "Sunny",
            "detailedForecast": ""
        },
        {
            "number": 2,
            "name": "",
            "startTime": "2018-12-28T11:00:00-08:00",
            "endTime": "2018-12-28T12:00:00-08:00",
            "isDaytime": true,
            "temperature": 51,
            "temperatureUnit": "F",
            "temperatureTrend": null,
            "windSpeed": "12 mph",
            "windDirection": "NNW",
            "icon": "https://api.weather.gov/icons/land/day/few?size=small",
            "shortForecast": "Sunny",
            "detailedForecast": ""
        },
        {
            "number": 3,
            "name": "",
            "startTime": "2018-12-28T12:00:00-08:00",
            "endTime": "2018-12-28T13:00:00-08:00",
            "isDaytime": true,
            "temperature": 54,
            "temperatureUnit": "F",
            "temperatureTrend": null,
            "windSpeed": "12 mph",
            "windDirection": "NNW",
            "icon": "https://api.weather.gov/icons/land/day/few?size=small",
            "shortForecast": "Sunny",
            "detailedForecast": ""
        },
        {
            "number": 4,
            "name": "",
            "startTime": "2018-12-28T13:00:00-08:00",
            "endTime": "2018-12-28T14:00:00-08:00",
            "isDaytime": true,
            "temperature": 55,
            "temperatureUnit": "F",
            "temperatureTrend": null,
            "windSpeed": "14 mph",
            "windDirection": "N",
            "icon": "https://api.weather.gov/icons/land/day/few?size=small",
            "shortForecast": "Sunny",
            "detailedForecast": ""
        },
        {
            "number": 5,
            "name": "",
            "startTime": "2018-12-28T14:00:00-08:00",
            "endTime": "2018-12-28T15:00:00-08:00",
            "isDaytime": true,
            "temperature": 57,
            "temperatureUnit": "F",
            "temperatureTrend": null,
            "windSpeed": "14 mph",
            "windDirection": "N",
            "icon": "https://api.weather.gov/icons/land/day/few?size=small",
            "shortForecast": "Sunny",
            "detailedForecast": ""
        },
...
    ]
}

In their wisdom, the request to /points/{lat,long}/forecast/hourly  results in a 301 redirect to /gridpoints/{WFO}/{gridpoint}/forecast/hourly which has the real data.  Grumble.  But that's the way they've set it up.

I have a mod to the advforecast2.php JSON version which with auto-fetch the associated hourly forecast JSON every hour and save it as forecast-{ZONE}-{N}-json-hourly.txt in the cache location.  I've attached the file if you are interested.
Ken True/Saratoga, CA, USA main site: saratoga-weather.org
Davis VP1+ FARS, Blitzortung RED, GRLevel3, WD, WL, VWS, Cumulus, Meteobridge
Free weather PHP scripts/website templates - update notifications on Twitter saratogaWXPHP

Offline Jasiu

  • Forecaster
  • *****
  • Posts: 947
    • LexMAWeather
Re: NWS Hourly forecast data
« Reply #2 on: December 28, 2018, 02:46:41 PM »
I think the question was along the lines that while there is hourly data as posted above, it is missing some of the details that you can see both in the old and new NWS graphical forecasts - an example being the precip percentage.
https://lexmaweather.info
On Mastodon: @LexMAWeather@toot.community

Offline daneast

  • Member
  • *
  • Posts: 48
Re: NWS Hourly forecast data
« Reply #3 on: December 28, 2018, 03:05:05 PM »
saratogaWX, thanks, but as Jasiu pointed out, my problem is that the hourly data from that API does not contain a fraction of the information the NWS has at the hourly level, as indicated in the graphic I linked to above.  All the API has for hourly is the temp, wind speed / direction, and current condition.  The graphic shows they model lots of things at the hourly level like POP, relative humidity, sky cover percentage, dew point, etc.

The thing I mainly need at the hourly level is the POP.  I'm giving Dark Sky a try, as their free API seems generous (1000 API calls per day) and their data (at least as far as the format and information they provide - I can't speak to the accuracy) looks good and easy to work with.  One API call gives me all the data I need for a location.

Offline daneast

  • Member
  • *
  • Posts: 48
Re: NWS Hourly forecast data
« Reply #4 on: December 28, 2018, 03:09:43 PM »
I think I spoke too soon.  In the link in my original post there is an option to view tabular data. When you switch to that view there is an XML button, which leads to...
https://forecast.weather.gov/MapClick.php?lat=36.9537&lon=-81.087&FcstType=digitalDWML

It looks like all that data is there, although the format is very clunky - using a "time-layout" key to reference values by index to another set of values which are the times.  Also, that does not appear to be a part of the formal APIs, new or old.  Boy I wish they could get their act together with the new API...

Offline saratogaWX

  • Administrator
  • Forecaster
  • *****
  • Posts: 9257
  • Saratoga, CA, USA Weather - free PHP scripts
    • Saratoga-Weather.org
Re: NWS Hourly forecast data
« Reply #5 on: December 28, 2018, 03:26:25 PM »
True.. the /points/{lat,long}/forecast/hourly results in a truncated version without some of the data that is shown in the tables/graphs.

For that, you need the gridpoint hourly data (also can be enabled for advforecast2.php JSON V5.09+) which results in a LOT more data (including thunder, etc), but in a time-compressed format.. each datum has a time+repetition factor so has to be expanded to one data point per time.   That query uses gridpoints/{WFO}/{gridpoint} as the query and is stored in forecast-{ZONE}-{N}-json-gridpoint.txt in the cache location.
I've attached a sample of the JSON.
Ken True/Saratoga, CA, USA main site: saratoga-weather.org
Davis VP1+ FARS, Blitzortung RED, GRLevel3, WD, WL, VWS, Cumulus, Meteobridge
Free weather PHP scripts/website templates - update notifications on Twitter saratogaWXPHP

Offline saratogaWX

  • Administrator
  • Forecaster
  • *****
  • Posts: 9257
  • Saratoga, CA, USA Weather - free PHP scripts
    • Saratoga-Weather.org
Re: NWS Hourly forecast data
« Reply #6 on: December 28, 2018, 03:29:13 PM »
I think I spoke too soon.  In the link in my original post there is an option to view tabular data. When you switch to that view there is an XML button, which leads to...
https://forecast.weather.gov/MapClick.php?lat=36.9537&lon=-81.087&FcstType=digitalDWML

It looks like all that data is there, although the format is very clunky - using a "time-layout" key to reference values by index to another set of values which are the times.  Also, that does not appear to be a part of the formal APIs, new or old.  Boy I wish they could get their act together with the new API...

Keep in mind that the MapClick.php access to forecast.weather.gov will disappear when they do the cutover to the new API-driven site (sample at https://forecast-v3.weather.gov/ ) with API docs at  https://forecast-v3.weather.gov/documentation?redirect=legacy#
so any script you develop using MapClick.php access will have to be discarded when they eventually cutover the new site.  It's been over a year now, and no forecast on when the cutover will take place.
Ken True/Saratoga, CA, USA main site: saratoga-weather.org
Davis VP1+ FARS, Blitzortung RED, GRLevel3, WD, WL, VWS, Cumulus, Meteobridge
Free weather PHP scripts/website templates - update notifications on Twitter saratogaWXPHP

Offline wvdkuil

  • Wim van der kuil
  • Forecaster
  • *****
  • Posts: 1986
    • My PWS at Leuven Belgium Europe
Re: NWS Hourly forecast data
« Reply #7 on: December 28, 2018, 03:31:49 PM »
This is indeed data from another source.
I wrote  a forecast-script for both the "point click" API and the older "National Digital Forecast Database".
You can compose your request for that here https://graphical.weather.gov/xml/SOAP_server/ndfdXML.htm
and then use it in your scripts.
As soon as you "grab" the idea about all the different time-series, it works nicely to have so much detailed data to play with.

Here you find all my forecast scripts: https://www.weerstation-herent.be/wsfct4/
This is the script using NDFD data: https://www.weerstation-herent.be/wsfct4/wsFctNoaaDPage4.php

Wim

Offline daneast

  • Member
  • *
  • Posts: 48
Re: NWS Hourly forecast data
« Reply #8 on: February 15, 2019, 05:47:40 PM »
To bump this thread... I'm using DarkSky and I'm pretty happy with their data.  However, even at $1 per 10k requests, it will become too pricey for me.  Basically what they are doing is grabbing data from NWS and some other sources and packaging it nicely, plus doing some text generation based on the data.

NWS has the data I need but it will require processing, which is perfectly fine.  That's right up my alley.  What I need is... the data.

This has the raw data I need: https://forecast.weather.gov/MapClick.php?lat=36.9537&lon=-81.087&FcstType=digitalDWML

However saratogaWX indicated that at some time this is going away.  The new API has hourly data, but it is very sparse (doesn't even have POP).  I can't imagine the NWS pulling the data feed above until they provide the data through their new API, otherwise there is no way at all to access that data.

Questions...
Who at the NWS can I contact to get some degree of assurance about the above MapClick data before I commit to using it?
Just in case I'm overlooking anything, is the complete hourly data available anywhere else on the NWS or with the new API?

Thanks.