Author Topic: Alternative Dashboard WU Issue (Resolved by the beteljuice) (Read 6033 times)

Silversword · « **on:** June 06, 2019, 01:06:19 PM »

Hi All,

It seem that on my Dashboard an issue has surfaced. The history coming from WU for the Official Record temperatures are not current and seems to be stuck. See below witch box I am talking about.

Hope that someone will be able to suggest or recommend on a resolution for this issue.

Regards,

--Stan Y.
Maui, Hawaii

saratogaWX · « **Reply #1 on:** June 06, 2019, 01:30:34 PM »

Alas, the data that came from Wunderground for the record high/low is no longer available, and is not offered to the WU PWS API users, so that part of the dashboard should just be commented-out. Weather-Display cannot access that data anymore.

gwwilk · « **Reply #2 on:** June 06, 2019, 02:45:19 PM »

WU still has the Record High and Low Temps available, but without the dates, e.g. Maui:https://www.wunderground.com/weather/us/hi/maui If you look at the 'Records' for the date you'll see that no date is provided to indicate when the record was set.

I was fortunate to find a local source at the University of Nebraska-Lincoln that has the record Highs and Lows with dates. That's how I display them on my dashboard.

Here is an NWS site with records and dates they were set: https://w2.weather.gov/climate/index.php?wfo=hnl This site will display data for any valid WFO. Set radio button to the Daily Climate Report (CLI) at your airport of choice, select 'Most Recent' and push the 'GO' button. A window pops up that contains the record hi-lo for today along with the dates they were set. I'm not sure how useful this information is because I don't see a way to capture it. Use 'AutoIt' and then scrape? I don't know.

Silversword · « **Reply #3 on:** June 06, 2019, 02:46:14 PM »

Thanks Ken for your reply. Now that WU does not support this data, is there a way to find another way to garnish this data from our airport NWS and have it displayed on the dashboard?

Hope that there may be a work around.

Thank you again for your support and guidance for our hobby.

Aloha,

--Stan Y.
Maui, Hawaii

Silversword · « **Reply #4 on:** June 06, 2019, 02:51:20 PM »

Thanks Jerry for your suggestions and will see if there's a way to find our local data that can be used for my site.

Aloha,

--Stan Y.
Maui, Hawaii

jgillett · « **Reply #5 on:** June 06, 2019, 04:44:33 PM »

Thanks for the post, Stan, and the replies. I have the same problem as Stan, and I suspect there must be others as well.

PSJohn · « **Reply #6 on:** June 07, 2019, 02:01:22 AM »

Quote from: gwwilk on June 06, 2019, 02:45:19 PM

~
Here is an NWS site with records and dates they were set: https://w2.weather.gov/climate/index.php?wfo=hnl This site will display data for any valid WFO. Set radio button to the Daily Climate Report (CLI) at your airport of choice, select 'Most Recent' and push the 'GO' button. A window pops up that contains the record hi-lo for today along with the dates they were set. I'm not sure how useful this information is because I don't see a way to capture it. Use 'AutoIt' and then scrape? I don't know.

I too am looking for a solution to replace record and normal data for a single NWS site.

With my limited knowledge of PHP and scripting my attempt to try to scrape this data from the NWS climate page for a single site has failed. I am hoping someone can create a script that would scrape it once a day and cache it, then maybe put it into a PHP variable to use for standalone sites also!

the beteljuice · « **Reply #7 on:** June 07, 2019, 06:51:37 PM »

Quote from: PSJohn on June 07, 2019, 02:01:22 AM

Quote from: gwwilk on June 06, 2019, 02:45:19 PM
... I'm not sure how useful this information is because I don't see a way to capture it. Use 'AutoIt' and then scrape? I don't know.

With my limited knowledge of PHP and scripting my attempt to try to scrape this data from the NWS climate page for a single site has failed. I am hoping someone can create a script that would scrape it once a day and cache it, then maybe put it into a PHP variable to use for standalone sites also!

Edit:
If you don't want to 'select' particular lines and don't mind the formatting, then a simple 'read' of eg. ~~https://w2.weather.gov/climate/getclimate.php?wfo=hnl~~ https://w2.weather.gov/climate/getclimate.php?wfo=hnl&pil=CLI&recent=yes&sid=HNL and only accepting lines within the <pre> </pre> tags and then writing to eg. mydaily.txt should be fairly straight-forward

Silversword · « **Reply #8 on:** June 08, 2019, 06:56:47 PM »

Quote from: the beteljuice on June 07, 2019, 06:51:37 PM

Quote from: PSJohn on June 07, 2019, 02:01:22 AM
Quote from: gwwilk on June 06, 2019, 02:45:19 PM
... I'm not sure how useful this information is because I don't see a way to capture it. Use 'AutoIt' and then scrape? I don't know.

With my limited knowledge of PHP and scripting my attempt to try to scrape this data from the NWS climate page for a single site has failed. I am hoping someone can create a script that would scrape it once a day and cache it, then maybe put it into a PHP variable to use for standalone sites also!

If you don't want to 'select' particular lines and don't mind the formatting, then a simple 'read' of eg. https://w2.weather.gov/climate/getclimate.php?wfo=hnl and only accepting lines within the <pre> </pre> tags and then writing to eg. mydaily.txt should be fairly straight-forward

Hi,

I could not access https://w2.weather.gov/climate/getclimate.php?wfo=hnl as it requires a valid product and station ID. How do I obtain this info?

Thanks for sharing and maybe it will be useful for my site as well as others that may the same issue.

Regards,

--Stan Y.
Maui, Hawaii

gwwilk · « **Reply #9 on:** June 08, 2019, 07:00:08 PM »

The link in my post way above works, Stan.

Silversword · « **Reply #10 on:** June 08, 2019, 07:37:04 PM »

Hi Jerry,

Hmm, that is interesting, yours works and his does not.

Now on how to glean this info:

...THE KAHULUI HI CLIMATE SUMMARY FOR JUNE 7 2019...

CLIMATE NORMAL PERIOD 1981 TO 2010
CLIMATE RECORD PERIOD 1954 TO 2019

WEATHER ITEM OBSERVED TIME RECORD YEAR NORMAL DEPARTURE LAST
VALUE (LST) VALUE VALUE FROM YEAR
NORMAL
..................................................................
TEMPERATURE (F)
YESTERDAY
MAXIMUM 95R 149 PM 94 1953 86 9 MM
MINIMUM 68 416 AM 60 1952 69 -1 MM
AVERAGE 82 78 4 MM

From this highlighted area to make it work for the Alternate Dashboard of ours.

Thanks for your input and hope one of these days a solution will be found.

Aloha,

--Stan Y.
Maui, Hawaii

the beteljuice · « **Reply #11 on:** June 08, 2019, 08:24:52 PM »

Sloppy work by the beteljuice

It's really a two stage form verification:

Real format of query url: https://w2.weather.gov/climate/getclimate.php?wfo=hnl&pil=CLI&recent=yes&sid=HNL (Note two location entries)

Edit
@stan - so are you saying all you want is yesterdays max / min temp values (F) ?

gwwilk · « **Reply #12 on:** June 08, 2019, 10:08:51 PM »

I suspect Stan, like me, wants the current day's records which are found at the bottom of the page you linked. Good work, by the way, beteljuice! I think you have the beginning of a way to ferret out the current date's average Hi/Lo's and record Hi/Lo's for many NWS sites.

https://w2.weather.gov/climate/getclimate.php?date=&wfo=oax&sid=LNK&pil=CLI&recent=yes links to my Lincoln, NE climate data.

It appears that at least for oax/LNK tomorrow's averages and records are delivered rather than today's after 5:00 pm local time.

the beteljuice · « **Reply #13 on:** June 08, 2019, 11:33:12 PM »

@Jerry

I'm still not at all sure what info you want, or what you want to do with it !

Here's the temp stuff for your locale 'scraped' and reformatted http://beteljuice.co.uk/noaa.php

the beteljuice · « **Reply #14 on:** June 09, 2019, 04:03:28 AM »

Thought ....

What (date) figures do you actually want ?

Using the 'recent' option will pull up typically sooner than 4 hours ago, which doesn't really fit in any particular window.

Wouldn't it be better to ask for 'yesterday' so that the figures make sense ?

gwwilk · « **Reply #15 on:** June 09, 2019, 07:25:53 AM »

See my dashboard's upper left temperatures. Today's averages and records are displayed, and this is what others who are using Scott Thompson's alternate ajax-dashboard need: today's values. These values should be available at midnight in order to change over to the current day's values.

I already scrape these values from other sources, but they're unique to my location and not a general solution. It looks like you may be close to a solution provided you also scrape the average high and low.

the beteljuice · « **Reply #16 on:** June 09, 2019, 08:24:48 AM »

This page does yesterday and before.
It can provide:

Yesterday Max(and time) / Min(and time) / Normal[max / min] / Average
Record Max value / year / last year value
Record Min value / year / last year value

Yesterday value / Normal
Record value / year
Month to date value / Normal / Last year value
Since Jun 1st value / Normal / Last year value
Since Jan 1st value / Normal / Last year value

Yesterday value / Normal
Record value / year
Month to date value / Normal / Last year value
Since Jun 1st value / Normal / Last year value
Since Jul 1st value / Normal / Last year value
Snow Depth - unsure, only one int entry, see below

Max Avg / direction
Max Gust / direction
Avg (value only)

Max
Min
Avg

So eg. Temp avg High / Low = Normal max / min

Something strange going on ...
Note Precip says Since Jun 1 and Jan 1, but Snow says Since Jun 1 (which is Month To Date !) and Jul 1
- also since Jul 1st = 55.5 In, but Snow Depth = 0 ?

PRECIPITATION (IN) YESTERDAY T 2.92 1964 0.15 -0.15 0.00 MONTH TO DATE 1.37 1.14 0.23 1.25 SINCE JUN 1 1.37 1.14 0.23 1.25 SINCE JAN 1 14.78 11.48 3.30 8.01 SNOWFALL (IN) YESTERDAY 0.0 T 1953 0.0 0.0 0.0 2001 MONTH TO DATE 0.0 0.0 0.0 0.0 SINCE JUN 1 0.0 0.0 0.0 0.0 SINCE JUL 1 55.5 25.9 29.6 21.4 SNOW DEPTH 0

- Is this not suitable for your purposes

jgillett · « **Reply #17 on:** June 09, 2019, 10:15:03 AM »

What is needed to make this work as before are the record high/low temperatures for today's date, which is what WU used to provide.

92merc · « **Reply #18 on:** June 09, 2019, 12:27:05 PM »

This is basically what I was looking for as well. I've done it with an Excel file, exported to a bunch of TXT files, then PHP fetched based on today's month/day.

http://www.wxforum.net/index.php?topic=35922.msg381625#msg381625

I have normal rain for THIS month, and normal rain YTD which is tabulated everyday.

Then I have normal high/low.
Then record high/low.

It was a bit of work to put it all into a running spreadsheet. But now that I have it done, about all I should have to do is once a year make sure the NOW data from NWS hasn't changed my rain data. And check my record high/lows.

Silversword · « **Reply #19 on:** June 09, 2019, 03:43:31 PM »

Hi All,

Seems that we are getting close to finding a solution for this issue for the High and Low record for the the start of the day at 12:00/2400 hours midnight local times.

Thank you all for finding a possible solution.

BTW, my local airport that is about 10 miles away is "OGG" that supplies data to the NWS.

Aloha,

--Stan Y.
Maui, Hawaii

the beteljuice · « **Reply #20 on:** June 09, 2019, 04:35:03 PM »

@ 92merc

So .. Have you already done everything (from a different dataset) ?

Although I'm not sure why you need eXcel.

92merc · « **Reply #21 on:** June 09, 2019, 05:24:05 PM »

Yeah, I'll attach the Excel sheet. The biggest reason for the spreadsheet is I wanted my YTD rain to increment every day. I used to have it do it just on the first of the month, add the whole month. But being an anal retentive person, I wanted to compare my actual YTD rain compared to normal. So using the data from the NWS NOW data section, I added in what is typical rain for each day of the year. I then added it all up to get my normal monthly rain. Then normal yearly rain.

The other fields for normal high/low and record high/low are regular text fields I can maintain. Lastly, I had some Excel macro help in exporting those out into files with the month/day in them. Since I left total for the month as just 12 monthly files, I had one file for each of the 5 stats, for each day of the year.

So then here is the code I added to my ajaxdashboard file. Down in the empty area of line 982.

Code: [Select]

<table width="630" border="0" cellpadding="0" cellspacing="0" style = "font-weight: normal";>
			<tr style="text-align:center">
				<td>Normal Rain Month: &nbsp;<?php $mydate = date('m'); $filename = "./history/rain".$mydate."mo.txt"; include($filename);?>&nbsp; in.</td>
				<td>Normal Rain YTD: &nbsp;<?php $mydate = date('md'); $filename = "./history/".$mydate."ytd.txt"; include($filename);?>&nbsp; in.</td>
			</tr>
			<tr style="text-align:center">
				<td>Normal High: &nbsp;<?php $mydate = date('md'); $filename = "./history/".$mydate."nh.txt"; include($filename);?>&deg;</td>
				<td>Normal Low: &nbsp;<?php $mydate = date('md'); $filename = "./history/".$mydate."nl.txt"; include($filename);?>&deg;</td>
			</tr>
			<tr style="text-align:center">
				<td>Record High: &nbsp;<?php $mydate = date('md'); $filename = "./history/".$mydate."rh.txt"; include($filename);?></td>
				<td>Record Low: &nbsp;<?php $mydate = date('md'); $filename = "./history/".$mydate."rl.txt"; include($filename);?></td>
			</tr>
</table>

So basically all the files are in my /history folder. The files are generated by 5 Excel macros. But this way I can update the spreadsheet. Run the macro. Files relative to the Excel file are created. Upload the updated files.

Oh, and the first two columns in the Excel sheet decide what the beginning of the files are named. Followed by nh for Normal high, nl for Normal low, rh for Record high, rl for Record low, and YTD for YTD rainfall. The other 12 files are normal rain for the month, rain01mo.txt, etc..

the beteljuice · « **Reply #22 on:** June 09, 2019, 05:59:08 PM »

Quote from: Jerry

What is needed to make this work as before are the record high/low temperatures for today's date,

~~I can't find that data for 'today'~~, nor is it clear if the Record values are 'All time' or (Yesterdays) date. I suspect 'All time' ...

Edit: Ah ....
A different section gives 'Today' normal max / min, record max[and year] / record min[and year] - Rain Only !

THE LINCOLN NE CLIMATE NORMALS FOR TODAY NORMAL RECORD YEAR MAXIMUM TEMPERATURE (F) 82 101 1933 MINIMUM TEMPERATURE (F) 59 44 1915
A little more complicated to detect but do-able.
Please create a 'wish list' of:
a/ The data you want.
b/ The PHP variable name for it.

... and I'll think about putting something together
It would be a simple php include file that creates / updates a cache-file (ie. 'todays' grab) which in turn returns PHP variables for '<? echo value ?> statements'.
- No promisies

gwwilk · « **Reply #23 on:** June 09, 2019, 07:07:40 PM »

@ beteljuice
Here's an image of the temperature section in my (HIGHLY MODIFIED) version of Scott Thompson's Alternate Ajax-Dashboard6:

Here's the relevant dashboard code:

Code: [Select]

<?php if ($useWU) { // Use Local, i.e. UNL-KLNK High/lows  ?>
                        <tr>
                            <td align="center" valign="top" style="text-align: center; border: 1px solid red;">
                                    <?php langtrans('Rec** High'); ?><br/>
                                <strong><span style="font-size: 16px"><span class="convTemp"><?php echo strip_units($theHigh) . $uomTemp; ?></span></span></strong><br/>
    <?php echo $highDate; ?>
                            </td>
                            <td align="center" valign="top" style="text-align: center; border: 1px solid blue;">
    <?php langtrans('Rec** Low'); ?><br/>
                                <strong><span style="font-size: 16px"><span class="convTemp"><?php echo strip_units($theLow) . $uomTemp; ?></span></span></strong><br/>
                                    <?php echo $lowDate; ?>
                            </td>
                        </tr>
                        <tr>
                            <td align="center" colspan="2" style="font-size: 8px">
                                    **Records from <a href="https://lincolnweather.unl.edu/records" title="New Tab" target="_blank">University of Nebr-Lincoln</a>
                            </td>
                        </tr>
                        <tr>
                            <td align="center" valign="top" style="text-align: center; border: 1px solid red;">
                                    <?php langtrans('Avg*** High'); ?><br/>
                                <strong><span style="font-size: 16px"><span class="convTemp"><?php echo strip_units($avgHigh) . $uomTemp; ?></span></span></strong><br/>
                            </td>
                            <td align="center" valign="top" style="text-align: center; border: 1px solid blue;">
    <?php langtrans('Avg*** Low'); ?><br/>
                                <strong><span style="font-size: 16px"><span class="convTemp"><?php echo strip_units($avgLow) . $uomTemp; ?></span></span></strong><br/>
                            </td>
                        </tr>
                        <tr>
                            <td align="center" colspan="2" style="font-size: 8px">
                                    ***Daily Averages from <a href="https://www.usclimatedata.com/climate/lincoln/nebraska/united-states/usne0283" title="New Tab" target="_blank">US Climate Data</a>
                            </td>
                        </tr>
<?php } // End WU High/Low  ?>

NOTE: I left the '$useWU' switch as is even though WU no longer plays a role in this display. This switch is in 'AltAjaxDashboardConfig6.php'.

Ideally the next day's 'normals' and records could be gleaned after 5:00 pm via an appropriate script. Or they could be captured at midnight. This is a process that would only need to be run once a day.

the beteljuice · « **Reply #24 on:** June 09, 2019, 07:58:11 PM »

Values not shown in code snippet ...
Your Current temp - cannot be done from 'scrape'
Today (so far) High / Low - cannot be done from 'scrape'
Station (this day) High / Low - cannot be done from 'scrape'

Values shown in code snippet .....
Yesterday High / Low with time - OK
This Day record High / Low with year - OK
This Day Average High / Low - OK

So, in the snippet, is that all you require ?
.. plus - how is metric handled (I was hoping to give a straight-forward drop-in, but it looks like there will have to be a section rewrite

)

News:

Author Topic: Alternative Dashboard WU Issue (Resolved by the beteljuice) (Read 6033 times)