Topic: SPC Convective Outlook for Specific Location Script

[UpstateWeather.com]

I'm looking to write / use a script / api that can produce the SPC categorical and/or probability convective outlook for a specific location (i.e. Atlanta). I can't not find any apis on the NWS SPC site that may actually provide that data for a specific location, but I'm also not the most knowledgeable with this.

can anyone point me in the right direction?

Thanks!
eric

[Jasiu]

Can you point to an example of what you are looking for on the SPC site?  If there is no API you might just need to bite the bullet and scrape the HTML of what you can see there.

[UpstateWeather.com]

Can you point to an example of what you are looking for on the SPC site?  If there is no API you might just need to bite the bullet and scrape the HTML of what you can see there.

That's the problem - I haven't been able to find an example.

Basically, here's what I want. I live in Atlanta and I want to display, in text, what the current convective outlook is for the day. Thunderstorm, Marginal, Slight, etc.

Regarding scraping html - I haven't for the life of me been able to figure that out! Though I would love to.

[Jasiu]

The only convective outlooks I've ever seen are national in scale. Mesoscale discussions (as warranted) zoom in on particular areas.

Scraping is the hacky art of identifying a pattern in the HTML of a page and then using that knowledge to extract the bit that you want. For example:

Code: [Select]

view-source:https://forecast.weather.gov/product.php?site=NWS&issuedby=BOX&product=PNS&format=CI&version=1&glossary=1&highlight=off
The meat of the public information statement is between <pre ...> and </pre>:

Code: [Select]

<pre id="proddiff" class="glossaryProduct "><span><div></div></span>
387
NOUS41 KBOX 131906
PNSBOX
CTZ002>004-MAZ002>024-026-RIZ001>008-140706-

<a href="/glossary.php?word=Public Information Statement" onClick="return popup(this, 'notes')">Public Information Statement</a>
Spotter Reports

.....

</pre>

Additionally, if you want to avoid the metadata, you can clip it at the "&&".

The obvious problem and downside compared to an API is that if the HTML format of the page changes, you are screwed.

The version 4.0x advforecast2.php code does scraping if you want to look at an example (although not a simple one).  I have an version of code that pulled the PNS in this way (I'm using the API now) and can dig it up if you want me to send it (direct message..).

[saratogaWX]

If you'd like the 'bones' of a basic scraper script, try my NWS forecast discussion script .. already selects the <pre>...</pre> part of a page and uses HTTPS to access the NWS website.
That basic script has been repurposed by others for a variety of page scraping scripts.

[UpstateWeather.com]

saratogaWX and Jasiu -

Thanks so much! I didn't know those scripts were essentially scrapers, so I'll play around and figure out the whole scraping thing. Thanks again!

[saratogaWX]

Only some of my older scripts are scrapers now.. over time, I've shifted to using API access (where the website provides it).  It means far fewer changes in the future.

As Jasiu noted, scrapers depend on having a fixed/parsable HTML format, and webmasters are prone to tinker with designs, which end up breaking a scraper script until it is adjusted to follow the new design.

The advforecast2.php V4.xx script for NWS forecasts is a (complicated) scraper.  The V5.xx version uses an API.
The ec-forecast.php script now uses an API .. the old script was updated frequently as Environment Canada tweaked with their website much more frequently than the NWS did.

Happy programming...