WXforum.net

Web Weather => Weather Website PHP/AJAX scripting => Topic started by: Silversword on June 06, 2019, 01:06:19 PM

Title: Alternative Dashboard WU Issue (Resolved by the beteljuice)
Post by: Silversword on June 06, 2019, 01:06:19 PM
Hi All,

It seem that on my Dashboard an issue has surfaced.  The history coming from WU for the Official Record temperatures are not current and seems to be stuck.  See below witch box I am talking about.

(http://[attachment id=1 msg=382276][/attachment])

Hope that someone will be able to suggest or recommend on a resolution for this issue.

Regards,

--Stan Y.
   Maui, Hawaii
Title: Re: Alternative Dashboard WU Issue
Post by: saratogaWX on June 06, 2019, 01:30:34 PM
Alas, the data that came from Wunderground for the record high/low is no longer available, and is not offered to the WU PWS API users, so that part of the dashboard should just be commented-out.  Weather-Display cannot access that data anymore. 
Title: Re: Alternative Dashboard WU Issue
Post by: gwwilk on June 06, 2019, 02:45:19 PM
WU still has the Record High and Low Temps available, but without the dates, e.g. Maui:https://www.wunderground.com/weather/us/hi/maui  If you look at the 'Records' for the date you'll see that no date is provided to indicate when the record was set.

I was fortunate to find a local source at the University of Nebraska-Lincoln that has the record Highs and Lows with dates.  That's how I display them on my dashboard (https://www.gwwilkins.org/).

Here is an NWS site with records and dates they were set:  https://w2.weather.gov/climate/index.php?wfo=hnl  This site will display data for any valid WFO.  Set radio button to the Daily Climate Report (CLI) at your airport of choice, select 'Most Recent' and push the 'GO' button.  A window pops up that contains the record hi-lo for today along with the dates they were set.  I'm not sure how useful this information is because I don't see a way to capture it.  Use 'AutoIt' and then scrape?  I don't know.
Title: Re: Alternative Dashboard WU Issue
Post by: Silversword on June 06, 2019, 02:46:14 PM
Thanks Ken for your reply. Now that WU does not support this data, is there a way to find another way to garnish this data from our airport NWS and have it displayed on the dashboard?

Hope that there may be a work around.

Thank you again for your support and guidance for our hobby.

Aloha,

--Stan Y.
  Maui, Hawaii
Title: Re: Alternative Dashboard WU Issue
Post by: Silversword on June 06, 2019, 02:51:20 PM
Thanks Jerry for your suggestions and will see if there's a way to find our local data that can be used for my site.

Aloha,

--Stan Y.
  Maui, Hawaii
Title: Re: Alternative Dashboard WU Issue
Post by: jgillett on June 06, 2019, 04:44:33 PM
Thanks for the post, Stan, and the replies. I have the same problem as Stan, and I suspect there must be others as well.
Title: Re: Alternative Dashboard WU Issue
Post by: PSJohn on June 07, 2019, 02:01:22 AM
~
Here is an NWS site with records and dates they were set:  https://w2.weather.gov/climate/index.php?wfo=hnl  This site will display data for any valid WFO.  Set radio button to the Daily Climate Report (CLI) at your airport of choice, select 'Most Recent' and push the 'GO' button.  A window pops up that contains the record hi-lo for today along with the dates they were set.  I'm not sure how useful this information is because I don't see a way to capture it.  Use 'AutoIt' and then scrape?  I don't know.

I too am looking for a solution to replace record and normal data for a single NWS site.

With my limited knowledge of PHP and scripting my attempt to try to scrape this data from the NWS climate page for a single site has failed. I am hoping someone can create a script that would scrape it once a day and cache it, then maybe put it into a PHP variable to use for standalone sites also!
Title: Re: Alternative Dashboard WU Issue
Post by: the beteljuice on June 07, 2019, 06:51:37 PM
... I'm not sure how useful this information is because I don't see a way to capture it.  Use 'AutoIt' and then scrape?  I don't know.

With my limited knowledge of PHP and scripting my attempt to try to scrape this data from the NWS climate page for a single site has failed. I am hoping someone can create a script that would scrape it once a day and cache it, then maybe put it into a PHP variable to use for standalone sites also!

Edit:
If you don't want to 'select' particular lines and don't mind the formatting, then a simple 'read' of eg. https://w2.weather.gov/climate/getclimate.php?wfo=hnl https://w2.weather.gov/climate/getclimate.php?wfo=hnl&pil=CLI&recent=yes&sid=HNL and only accepting lines within the <pre> </pre> tags and then writing to eg. mydaily.txt should be fairly straight-forward
Title: Re: Alternative Dashboard WU Issue
Post by: Silversword on June 08, 2019, 06:56:47 PM
... I'm not sure how useful this information is because I don't see a way to capture it.  Use 'AutoIt' and then scrape?  I don't know.

With my limited knowledge of PHP and scripting my attempt to try to scrape this data from the NWS climate page for a single site has failed. I am hoping someone can create a script that would scrape it once a day and cache it, then maybe put it into a PHP variable to use for standalone sites also!

If you don't want to 'select' particular lines and don't mind the formatting, then a simple 'read' of eg. https://w2.weather.gov/climate/getclimate.php?wfo=hnl and only accepting lines within the <pre> </pre> tags and then writing to eg. mydaily.txt should be fairly straight-forward

Hi,

I could not access https://w2.weather.gov/climate/getclimate.php?wfo=hnl (https://w2.weather.gov/climate/getclimate.php?wfo=hnl) as it requires a valid product and station ID.  How do I obtain this info?

Thanks for sharing and maybe it will be useful for my site as well as others that may the same issue.

Regards,

--Stan Y.
   Maui, Hawaii
Title: Re: Alternative Dashboard WU Issue
Post by: gwwilk on June 08, 2019, 07:00:08 PM
The link in my post way above works, Stan.
Title: Re: Alternative Dashboard WU Issue
Post by: Silversword on June 08, 2019, 07:37:04 PM
Hi Jerry,

Hmm, that is interesting, yours works and his does not.

Now on how to glean this info:

...THE KAHULUI HI CLIMATE SUMMARY FOR JUNE 7 2019...

CLIMATE NORMAL PERIOD 1981 TO 2010
CLIMATE RECORD PERIOD 1954 TO 2019


WEATHER ITEM   OBSERVED TIME   RECORD YEAR NORMAL DEPARTURE LAST
                VALUE   (LST)  VALUE       VALUE  FROM      YEAR
                                                  NORMAL
..................................................................
TEMPERATURE (F)
 YESTERDAY
  MAXIMUM         95R   149 PM  94    1953  86      9       MM
  MINIMUM         68    416 AM  60    1952  69     -1       MM
  AVERAGE         82                        78      4       MM

From this highlighted area to make it work for the Alternate Dashboard of ours.

Thanks for your input and hope one of these days a solution will be found.

Aloha,

--Stan Y.
   Maui, Hawaii

Title: Re: Alternative Dashboard WU Issue
Post by: the beteljuice on June 08, 2019, 08:24:52 PM
Sloppy work by the beteljuice  :oops:

It's really a two stage form verification:

Real format of query url: https://w2.weather.gov/climate/getclimate.php?wfo=hnl&pil=CLI&recent=yes&sid=HNL (Note two location entries)

Edit
@stan - so are you saying all you want is yesterdays max / min temp values (F) ?
Title: Re: Alternative Dashboard WU Issue
Post by: gwwilk on June 08, 2019, 10:08:51 PM
I suspect Stan, like me, wants the current day's records which are found at the bottom of the page you linked.  Good work, by the way, beteljuice!  I think you have the beginning of a way to ferret out the current date's average Hi/Lo's and record Hi/Lo's for many NWS sites.

https://w2.weather.gov/climate/getclimate.php?date=&wfo=oax&sid=LNK&pil=CLI&recent=yes links to my Lincoln, NE climate data.

It appears that at least for oax/LNK tomorrow's averages and records are delivered rather than today's after 5:00 pm local time.
Title: Re: Alternative Dashboard WU Issue
Post by: the beteljuice on June 08, 2019, 11:33:12 PM
@Jerry

I'm still not at all sure what info you want, or what you want to do with it !

Here's the temp stuff for your locale 'scraped' and reformatted http://beteljuice.co.uk/noaa.php
Title: Re: Alternative Dashboard WU Issue
Post by: the beteljuice on June 09, 2019, 04:03:28 AM
Thought ....

What (date) figures do you actually want ?

Using the 'recent' option will pull up typically sooner than 4 hours ago, which doesn't really fit in any particular window.

Wouldn't it be better to ask for 'yesterday' so that the figures make sense ?
Title: Re: Alternative Dashboard WU Issue
Post by: gwwilk on June 09, 2019, 07:25:53 AM
See my dashboard (https://www.gwwilkins.org/)'s upper left temperatures.  Today's averages and records are displayed, and this is what others who are using Scott Thompson's alternate ajax-dashboard need: today's values.  These values should be available at midnight in order to change over to the current day's values.

I already scrape these values from other sources, but they're unique to my location and not a general solution.  It looks like you may be close to a solution provided you also scrape the average high and low.
Title: Re: Alternative Dashboard WU Issue
Post by: the beteljuice on June 09, 2019, 08:24:48 AM
This page does yesterday and before.
It can provide:

So eg. Temp avg High / Low = Normal max / min

Something strange going on ...
Note Precip says Since Jun 1 and Jan 1, but Snow says Since Jun 1 (which is Month To Date !) and Jul 1
 - also since Jul 1st = 55.5 In, but Snow Depth = 0 ?

PRECIPITATION (IN)
  YESTERDAY        T             2.92 1964   0.15  -0.15     0.00
  MONTH TO DATE    1.37                      1.14   0.23     1.25
  SINCE JUN 1      1.37                      1.14   0.23     1.25
  SINCE JAN 1     14.78                     11.48   3.30     8.01

SNOWFALL (IN)
  YESTERDAY        0.0           T    1953   0.0    0.0      0.0
                                      2001
  MONTH TO DATE    0.0                       0.0    0.0      0.0
  SINCE JUN 1      0.0                       0.0    0.0      0.0
  SINCE JUL 1     55.5                      25.9   29.6     21.4
  SNOW DEPTH       0



 - Is this not suitable for your purposes   [tup]
Title: Re: Alternative Dashboard WU Issue
Post by: jgillett on June 09, 2019, 10:15:03 AM
What is needed to make this work as before are the record high/low temperatures for today's date, which is what WU used to provide.
Title: Re: Alternative Dashboard WU Issue
Post by: 92merc on June 09, 2019, 12:27:05 PM
This is basically what I was looking for as well.  I've done it with an Excel file, exported to a bunch of TXT files, then PHP fetched based on today's month/day.

http://www.wxforum.net/index.php?topic=35922.msg381625#msg381625

I have normal rain for THIS month, and normal rain YTD which is tabulated everyday.

Then I have normal high/low.
Then record high/low.

It was a bit of work to put it all into a running spreadsheet.  But now that I have it done, about all I should have to do is once a year make sure the NOW data from NWS hasn't changed my rain data.  And check my record high/lows.
Title: Re: Alternative Dashboard WU Issue
Post by: Silversword on June 09, 2019, 03:43:31 PM
Hi All,

Seems that we are getting close to finding a solution for this issue for the High and Low record for the the start of the day at 12:00/2400 hours midnight local times.

Thank you all for finding a possible solution.

BTW, my local airport that is about 10 miles away is "OGG" that supplies data to the NWS.

Aloha,

--Stan Y.
   Maui, Hawaii
Title: Re: Alternative Dashboard WU Issue
Post by: the beteljuice on June 09, 2019, 04:35:03 PM
@ 92merc

So .. Have you already done everything (from a different dataset) ?

Although I'm not sure why you need eXcel.
Title: Re: Alternative Dashboard WU Issue
Post by: 92merc on June 09, 2019, 05:24:05 PM
Yeah, I'll attach the Excel sheet.  The biggest reason for the spreadsheet is I wanted my YTD rain to increment every day.  I used to have it do it just on the first of the month, add the whole month.  But being an anal retentive person, I wanted to compare my actual YTD rain compared to normal.  So using the data from the NWS NOW data section, I added in what is typical rain for each day of the year.  I then added it all up to get my normal monthly rain.  Then normal yearly rain. 

The other fields for normal high/low and record high/low are regular text fields I can maintain.  Lastly, I had some Excel macro help in exporting those out into files with the month/day in them.  Since I left total for the month as just 12 monthly files, I had one file for each of the 5 stats, for each day of the year.

So then here is the code I added to my ajaxdashboard file. Down in the empty area of line 982. 
Code: [Select]
<table width="630" border="0" cellpadding="0" cellspacing="0" style = "font-weight: normal";>
<tr style="text-align:center">
<td>Normal Rain Month: &nbsp;<?php $mydate date('m'); $filename "./history/rain".$mydate."mo.txt"; include($filename);?>&nbsp; in.</td>
<td>Normal Rain YTD: &nbsp;<?php $mydate date('md'); $filename "./history/".$mydate."ytd.txt"; include($filename);?>&nbsp; in.</td>
</tr>
<tr style="text-align:center">
<td>Normal High: &nbsp;<?php $mydate date('md'); $filename "./history/".$mydate."nh.txt"; include($filename);?>&deg;</td>
<td>Normal Low: &nbsp;<?php $mydate date('md'); $filename "./history/".$mydate."nl.txt"; include($filename);?>&deg;</td>
</tr>
<tr style="text-align:center">
<td>Record High: &nbsp;<?php $mydate date('md'); $filename "./history/".$mydate."rh.txt"; include($filename);?></td>
<td>Record Low: &nbsp;<?php $mydate date('md'); $filename "./history/".$mydate."rl.txt"; include($filename);?></td>
</tr>
</table>

So basically all the files are in my /history folder.  The files are generated by 5 Excel macros.  But this way I can update the spreadsheet.  Run the macro.  Files relative to the Excel file are created.   Upload the updated files.

Oh, and the first two columns in the Excel sheet decide what the beginning of the files are named.  Followed by nh for Normal high, nl for Normal low, rh for Record high, rl for Record low, and YTD for YTD rainfall.  The other 12 files are normal rain for the month, rain01mo.txt, etc..
Title: Re: Alternative Dashboard WU Issue
Post by: the beteljuice on June 09, 2019, 05:59:08 PM
Quote from: Jerry
What is needed to make this work as before are the record high/low temperatures for today's date,
I can't find that data for 'today', nor is it clear if the Record values are 'All time' or (Yesterdays) date. I suspect 'All time' ...

Edit: Ah ....
A different section gives 'Today' normal max / min, record max[and year] / record min[and year] - Rain Only !

THE LINCOLN NE CLIMATE NORMALS FOR TODAY
                         NORMAL    RECORD    YEAR
 MAXIMUM TEMPERATURE (F)   82       101      1933
 MINIMUM TEMPERATURE (F)   59        44      1915

A little more complicated to detect but do-able.
Please create a 'wish list' of:
  a/ The data you want.
  b/ The PHP variable name for it.

... and I'll think about putting something together
It would be a simple php include file that creates / updates a cache-file (ie. 'todays' grab) which in turn returns PHP variables for '<? echo value ?> statements'.
- No promisies  :shock:
Title: Re: Alternative Dashboard WU Issue
Post by: gwwilk on June 09, 2019, 07:07:40 PM
@ beteljuice
Here's an image of the temperature section in my (HIGHLY MODIFIED) version of Scott Thompson's Alternate Ajax-Dashboard6:
(https://www.gwwilkins.org/temp-images/Alt-AjaxDashboardTempPanelExample.png)

Here's the relevant dashboard code:
Code: [Select]
<?php if ($useWU) { // Use Local, i.e. UNL-KLNK High/lows  ?>
                        <tr>
                            <td align="center" valign="top" style="text-align: center; border: 1px solid red;">
                                    <?php langtrans('Rec** High'); ?><br/>
                                <strong><span style="font-size: 16px"><span class="convTemp"><?php echo strip_units($theHigh) . $uomTemp?></span></span></strong><br/>
    <?php echo $highDate?>
                            </td>
                            <td align="center" valign="top" style="text-align: center; border: 1px solid blue;">
    <?php langtrans('Rec** Low'); ?><br/>
                                <strong><span style="font-size: 16px"><span class="convTemp"><?php echo strip_units($theLow) . $uomTemp?></span></span></strong><br/>
                                    <?php echo $lowDate?>
                            </td>
                        </tr>
                        <tr>
                            <td align="center" colspan="2" style="font-size: 8px">
                                    **Records from <a href="https://lincolnweather.unl.edu/records" title="New Tab" target="_blank">University of Nebr-Lincoln</a>
                            </td>
                        </tr>
                        <tr>
                            <td align="center" valign="top" style="text-align: center; border: 1px solid red;">
                                    <?php langtrans('Avg*** High'); ?><br/>
                                <strong><span style="font-size: 16px"><span class="convTemp"><?php echo strip_units($avgHigh) . $uomTemp?></span></span></strong><br/>
                            </td>
                            <td align="center" valign="top" style="text-align: center; border: 1px solid blue;">
    <?php langtrans('Avg*** Low'); ?><br/>
                                <strong><span style="font-size: 16px"><span class="convTemp"><?php echo strip_units($avgLow) . $uomTemp?></span></span></strong><br/>
                            </td>
                        </tr>
                        <tr>
                            <td align="center" colspan="2" style="font-size: 8px">
                                    ***Daily Averages from <a href="https://www.usclimatedata.com/climate/lincoln/nebraska/united-states/usne0283" title="New Tab" target="_blank">US Climate Data</a>
                            </td>
                        </tr>
<?php // End WU High/Low  ?>

NOTE: I left the '$useWU' switch as is even though WU no longer plays a role in this display.  This switch is in 'AltAjaxDashboardConfig6.php'.

Ideally the next day's 'normals' and records could be gleaned after 5:00 pm via an appropriate script.  Or they could be captured at midnight.  This is a process that would only need to be run once a day.
Title: Re: Alternative Dashboard WU Issue
Post by: the beteljuice on June 09, 2019, 07:58:11 PM
Values not shown in code snippet ...
Your Current temp - cannot be done from 'scrape'
Today (so far) High / Low - cannot be done from 'scrape'
Station (this day) High / Low - cannot be done from 'scrape'

Values shown in code snippet .....
Yesterday High / Low with time - OK
This Day record High / Low with year - OK
This Day Average High / Low - OK

So, in the snippet, is that all you require ?
.. plus - how is metric handled (I was hoping to give a straight-forward drop-in, but it looks like there will have to be a section rewrite  :-( )
Title: Re: Alternative Dashboard WU Issue
Post by: gwwilk on June 09, 2019, 08:11:56 PM
Thanks! I’m pretty much set with my local values for the average high and low as well as the record high and low with dates for each day. Others who are also using Scott’s alternate dashboard need to be able to display their local averages and records as well. The graphic just illustrates how I display them. A general solution that uses a variable for the locale and then scrapes those values with the dates for the records would be very useful. :-) A cron job or curl could run the script once a day at the appropriate time.
Title: Re: Alternative Dashboard WU Issue
Post by: the beteljuice on June 09, 2019, 09:27:36 PM
Quote from: Jerry
A general solution that uses a variable for the locale and then scrapes those values with the dates for the records would be very useful. :-) A cron job or curl could run the script once a day at the appropriate time.
Let's be clear about requirements ....

Yesterday High / Low with time - handled by other code / station data

This Day (regional) record High / Low with year [$theHigh, $highDate /  $theLow, $lowDate]
This Day (regional) Average High / Low [$avgHigh / $avgLow]

... and those six values are the only info you require ?
Unit conversion I can do from inspecting $uomTemp

Cron not necessary, simple date check and cache file.

How would the data get displayed if you aren't using WU ? - I'm thinking of the "if ($useWU)" statement. - is WU data used elsewhere ?, otherwise how would someone stop the WU grab routine ? - is that standard code or one of your mods ?

If ($useWU) is false is there an existing alternative or can this 'block' be an "} else {" for ease of implementation ?
Title: Re: Alternative Dashboard WU Issue
Post by: gwwilk on June 09, 2019, 10:02:35 PM
...
Let's be clear about requirements ....

Yesterday High / Low with time - handled by other code / station data

This Day (regional) record High / Low with year [$theHigh, $highDate /  $theLow, $lowDate]
This Day (regional) Average High / Low [$avgHigh / $avgLow]

... and those six values are the only info you require ?
Unit conversion I can do from inspecting $uomTemp
Yes.  Remember, though, that the records may have occurred  in more than one past year.

Quote
Cron not necessary, simple date check and cache file.
That would work, too.
Quote
How would the data get displayed if you aren't using WU ? - I'm thinking of the "if ($useWU)" statement. - is WU data used elsewhere ?, otherwise how would someone stop the WU grab routine ? - is that standard code or one of your mods ?
The '$useWU' switch only occurs once in the original ajax-dashboard6.php, and it is set in the 'AltAjaxDashboardConfig6.php'.  I just didn't bother changing the switch's name for my purposes.  WU now has nothing to do with my ajax-dashboard6.
Quote
If ($useWU) is false is there an existing alternative or can this 'block' be an "} else {" for ease of implementation ?
No.  People can get creative if they want to use the space for something else.

Here (http://tiggrweather.net/) is John Gillett's example of an obviously erroneous Record Hi/Lo set because WU no longer provides them.  He has contributed to this thread.

The Average Hi and Average Lo Temps were added to my dashboard in order to provide some historical perspective.  They weren't in Scott's original script.

If you want to examine Scott's ajax-dashboard6 script set, Ken True has kindly undertaken to update it and host it on his SaratogaWx site (http://saratoga-weather.org/scripts-legacy.php#scott).
Title: Re: Alternative Dashboard WU Issue
Post by: the beteljuice on June 09, 2019, 10:17:43 PM
Quote
The '$useWU' switch only occurs once in the original ajax-dashboard6.php,
It must occur again in order to execute the 'grab' routine and call any unit conversion surely ?
Quote
Remember, though, that the records may have occurred  in more than one past year.
My intention would be to only show the earliest year.
Quote
The Average Hi and Average Lo Temps were added to my dashboard in order to provide some historical perspective.  They weren't in Scott's original script.
Hmm ... whether to add the html / php code or not - probably 'break' peoples existing layout.

Might need a 'vote' on that.

Thoughts please dear readers .....
Title: Re: Alternative Dashboard WU Issue
Post by: jgillett on June 09, 2019, 11:26:34 PM
Here (http://tiggrweather.net/) is John Gillett's example of an obviously erroneous Record Hi/Lo set because WU no longer provides them.  He has contributed to this thread.

I did finally manage to turn off the $useORHL   = false; item in the config file (otherwise the ORH image would have shown for today), but because of problems with my hands I am not yet ready to mess with the dashboard .js file to comment out the bad 'Records from WUnderground' data. For me (and others I think), the only thing I need is record high and low for today's date. Scott's (now Ken's) script will still do the compare if the data is saved with the same var name as it was from WU.
Title: Re: Alternative Dashboard WU Issue
Post by: gwwilk on June 09, 2019, 11:37:13 PM
Here (http://tiggrweather.net/) is John Gillett's example of an obviously erroneous Record Hi/Lo set because WU no longer provides them.  He has contributed to this thread.

I did finally manage to turn off the $useORHL   = false; item in the config file (otherwise the ORH image would have shown for today), but because of problems with my hands I am not yet ready to mess with the dashboard .js file to comment out the bad 'Records from WUnderground' data. For me (and others I think), the only thing I need is record high and low for today's date. Scott's (now Ken's) script will still do the compare if the data is saved with the same var name as it was from WU.
Just change '$useWU = true;' to '$useWU = false;' in 'AltAjaxDashboardConfig6.php', John.  Best leave the .js file alone.
Title: Re: Alternative Dashboard WU Issue
Post by: gwwilk on June 09, 2019, 11:47:50 PM
Quote
The '$useWU' switch only occurs once in the original ajax-dashboard6.php,
Quote
It must occur again in order to execute the 'grab' routine and call any unit conversion surely ?
No.  Thinking about this problem, since grabbing the 'latest' gives tomorrow's results after 5pm, you can't leave it to chance as to when the 'grab' occurs.  It must be initiated at a specific time and in a specific way if the results are to be valid.  Also, since this is the US NWS all units will be Imperial, not Metric.  Any UOM conversions will be handled automatically by the ajaxWDwx3.js script.
 Click on the blue IMPERIAL at the top of my dashboard to see this in action.
Quote
Quote
Remember, though, that the records may have occurred  in more than one past year.
Quote
My intention would be to only show the earliest year.
The data is there.  Why not use all of it?
Quote
Quote
The Average Hi and Average Lo Temps were added to my dashboard in order to provide some historical perspective.  They weren't in Scott's original script.
Quote
Hmm ... whether to add the html / php code or not - probably 'break' peoples existing layout.
The code should just follow the pattern in my dashboard as in the snippet I posted earlier.  No need to 'reinvent the wheel'.
Title: Re: Alternative Dashboard WU Issue
Post by: jgillett on June 10, 2019, 12:12:03 AM
Just change '$useWU = true;' to '$useWU = false;' in 'AltAjaxDashboardConfig6.php', John.  Best leave the .js file alone.

Heh - perfect, Jerry. Right in front of my eyes (which is why I missed it  :!: ).

Thank you.

This should also fix Stan.
Title: Re: Alternative Dashboard WU Issue
Post by: Silversword on June 10, 2019, 01:22:40 AM
Hi All,

I turned off my WU part of my High and Low records from my Dashboard for now.  Thanks Jerry for where to do that editing.  Hope that one of these days it will come to show the "Official" temperature records.

Regards,

--Stan Y.
  Maui, Hawaii


Title: Re: Alternative Dashboard WU Issue
Post by: the beteljuice on June 10, 2019, 09:36:50 AM
OK

Got something that should work (Using WU vars is actually set-up within WD - the dashboard script itself does NOT call WU, just 'reads' what WD has put out. So you may want to turn that off).

Here is my effort running in debug (forces a fresh grab from NOAA amongst other things)

EDIT: Redundant material removed ... post for historic information only

I want to keep an eye on it for a day or two to see what Jerry is talking about 'rollover' time, plus of course I'm caching in a different timezone, so need to confirm that works alright.

Looking at the original scripts, it should allow unit change and use of 'new record' graphic option.
Title: Re: Alternative Dashboard WU Issue
Post by: gwwilk on June 10, 2019, 09:49:29 AM
Looks good! [tup]  Interestingly the values from NOWData are rounded to whole numbers, whereas my local sources include decimal tenths.

I think Stan in HI, John in CA, and John in AZ (all contributors to this thread) should be your real test targets because they have no local sources such as mine.
Title: Re: Alternative Dashboard WU Issue
Post by: jgillett on June 10, 2019, 11:01:47 AM
I'd be happy to try it. Note that at least 95% of AZ does NOT observe DST (hooray!), so my NWS office (PSR) is always -0700 GMT. Thinking maybe 0030 MST or 0100 MST might be a good time for it to run once/day as Jerry suggested.
Title: Re: Alternative Dashboard WU Issue
Post by: 92merc on June 10, 2019, 11:14:59 AM
If it is confirmed working, I'd be interested in the PHP file if I can get it incorporated into the standard dashboard.
Title: Re: Alternative Dashboard WU Issue
Post by: K3JAE on June 10, 2019, 11:31:31 AM
Am I then to understand based on a very quick scanning of this thread,  that if the record high/lows can be located, that I now need to somehow create an Excel file to import into the dashboard?

I'll read this thread through a but closer in a few hours (work taking my time at moment) and digest what I read a little longer.
Title: Re: Alternative Dashboard WU Issue
Post by: 92merc on June 10, 2019, 11:37:26 AM
I had created a method using an Excel sheet.

Beteljuice's method will scrape the data from the NWS NOW data section to be posted on your site.  So there are 2 different methods.

But if Betel's method will work for the standard dashboard, I'll probably switch my 4 temp records over to his method.  That way I won't have to keep up my database as much.
Title: Re: Alternative Dashboard WU Issue
Post by: K3JAE on June 10, 2019, 11:44:23 AM
Yeah, I prefer an auto scrape as I guarantee I will forget to update over many days and get it called into question.  :lol: Gotta love Weather.com - that company is as worthless as a ...........

Where is the USE WU switch located at?  I can at least disable that part for now until a more permanent resolution is found.
Title: Re: Alternative Dashboard WU Issue
Post by: jgillett on June 10, 2019, 12:21:48 PM
Where is the USE WU switch located at?  I can at least disable that part for now until a more permanent resolution is found.

John, it's in the AltAjaxDashboardConfig6.php file, line 129 in mine.
Title: Re: Alternative Dashboard WU Issue
Post by: K3JAE on June 10, 2019, 12:24:27 PM
OK, Yeah, I'm not using the ALT dashboard.  But I'll sniff it out. LOL
Title: Re: Alternative Dashboard WU Issue
Post by: jgillett on June 10, 2019, 12:28:17 PM
OK, Yeah, I'm not using the ALT dashboard.  But I'll sniff it out. LOL

Sorry, I should have checked your site. Scott put in a very nice routine to hide the ORHL and bad record data by simply going to false.
Title: Re: Alternative Dashboard WU Issue
Post by: the beteljuice on June 10, 2019, 05:52:39 PM
@all

The beteljuice 'scraper' is a simple php 'include' file. It gleans the values requested as php variables with which you can do whatever you like.

ie. It can be used as a 'stand alone' - It has no dependency on any other code or resource.

In the Alt dashboard template there are two adjacent well described areas to replace, but that is only because of the WU 'switch' (Now redundant), accreditation change, and I have created different variable names. As stated above there will be an additional option of 'This Day' Norm (Averages), and the  ORHL should work if wanted.

Still to check what 'Time' the noaa donor page rolls over (Fixed TZ, UTC, Locale requested TZ ?)
... and how that affects the grab file / cache file routine on (your) local server - maybe some extra setting up to do.

ATM the only set-up info required is wfo and sid - instructions in the file ...
Code: [Select]
// get this days record max / min temps with year and this days max / min avg temps
// wfo= location in lower_case, sid= location in UPPER_CASE as per https://w2.weather.gov/climate/
// click on map, next screen pick (2) nearest location click (4) Go - pop-up screen
// view source and look at wfo (lowercase) and sid (UPPERCASE) values in <input> section  ;-)

$wfo = "oax";
$sid = "LNK";

// only need info every new day so use a cachefile

Edit:

Arrgh ...
Now I see what Jerry was saying.

EDIT: Redundant material removed ... post for historic information only


@ 16:00 10th June locale TZ by report but now 17:00 (UTC -5) - and their query is 10th 21:00

Yesterday values block have become Today,
... and ..
Today values block have become Tomorrow

More importantly when do they 'flip' back again ?

There's going to have to be some serious logic checking on which area of data to scrape when.

... there's going to some delays guys  #-o
Title: Re: Alternative Dashboard WU Issue
Post by: gwwilk on June 10, 2019, 06:13:51 PM
 [tup] beteljuice :!:

Anxious to see it whenever you're ready. :-)

Edit:  I understand the problems, so take whatever time you need.

I think (hope) they flip back around midnight after having been 'tomorrow' from 1700 to 2400?
Title: Re: Alternative Dashboard WU Issue
Post by: jgillett on June 10, 2019, 06:19:08 PM
... there's going to some delays guys  #-o

Not a problem here.

Thanks for your efforts.
Title: Re: Alternative Dashboard WU Issue
Post by: jgillett on June 10, 2019, 07:43:38 PM
Got an SRH Mon afternoon on my site. At least that still works. Nice.   :grin:
Title: Re: Alternative Dashboard WU Issue
Post by: the beteljuice on June 10, 2019, 08:13:57 PM
beteljuice fuzzy logic may yet prevail  ;)

Edit:
New Day on the test page ...

... By George I think I've Got It  :lol:

Anyone want to play ????
Title: Re: Alternative Dashboard replace WU - Test Pilot Wanted
Post by: the beteljuice on June 11, 2019, 07:58:24 AM
Test Pilot wanted ....

Must be able to feed-back problems with instructions, html layout, actual functionality ..

(Keep an eye on data to see if it 'rolls over' at an acceptable time)

NB: This can be used as a 'stand-alone', no dependencies on other scripts - data available for US only !

 EDIT: Redundant material removed ... post for historic information only
Title: Re: Alternative Dashboard WU Issue
Post by: gwwilk on June 11, 2019, 01:30:27 PM
Thanks, beteljuice!  I have it running on my test server (http://www.selincolnwx.info/). [tup]

I will check the results periodically throughout the day and evening.

I misled you somewhat about what to include in the patch in that this should not have been included:
Code: [Select]

                   
                    <td align="center" class="data1" valign="top">
                <table align="center" border="0" cellpadding="0" cellspacing="4" style="width:180px">
<?php if ($useORHL) {  //  Show NOAA Based New Record HIGH/LOW Image ?>
                <tr>
                    <td colspan ="2">
                        <?php if (strip_units($maxtemp) > strip_units($thisDayMax)) {
                            echo 
"<img src=\"${imagesDir}OfficialRec_HIGH.gif\" alt=\"Official Record High\"/>"; }
                        
?>

                        <?php if (strip_units($mintemp) < strip_units($thisDayMin)) {
                            echo 
"<img src=\"${imagesDir}OfficialRec_LOW.gif\" alt=\"Official Record Low\"/>"; }
                        
?>

                    </td>
                </tr>
<?php // end $useORHL  ----------------------------------------  ?>

After bringing in the patch this segment occurred twice, messing me up until I figured it out. #-o  Actually I just had a problem following your instructions which were excellent.  My bad.
Title: Re: Alternative Dashboard WU Issue
Post by: gwwilk on June 11, 2019, 02:15:44 PM
The presence of '</body></html>' in 'noaarec.php' caused multiple validation errors at https://validator.w3.org/ until I removed them from the end of the script.  Validation of my dashboard on selincolnwx.info (https://validator.w3.org/check?uri=http%3A%2F%2Fselincolnwx.info%2F&charset=%28detect+automatically%29&doctype=Inline&group=0&accept=text%2Fhtml%2Capplication%2Fxhtml%2Bxml%2Capplication%2Fxml%3Bq%3D0.9%2Cimage%2Fwebp%2Cimage%2Fapng%2C*%2F*%3Bq%3D0.8%2Capplication%2Fsigned-exchange%3Bv%3Db3&accept-language=en-US%2Cen%3Bq%3D0.9&user-agent=W3C_Validator%2F1.3+http%3A%2F%2Fvalidator.w3.org%2Fservices) now succeeds without errors.
Title: Re: Alternative Dashboard replace WU - Test Pilot Wanted
Post by: PSJohn on June 11, 2019, 02:23:18 PM
Test Pilot wanted ....

Must be able to feed-back problems with instructions, html layout, actual functionality ..

(Keep an eye on data to see if it 'rolls over' at an acceptable time)

NB: This can be used as a 'stand-alone', no dependencies on other scripts - data available for US only !

 [ You are not allowed to view attachments ]  (readme typos fixed)

I'm willing to test it. I don't use any templates on my site, standalone. I will also check it periodically to see it is running as expected!

thanks for your time & effort with this!


EDIT: I have it running on my site https://desertweather.com/temprecavg/ (https://desertweather.com/temprecavg/)
Title: Re: Alternative Dashboard WU Issue
Post by: ed2kayak on June 11, 2019, 03:01:49 PM
beteljuice

I have implemented your code and changes.  =D&gt;

http://cvweather.org/

Looks good right now, I'll monitor, and report if issues.

I really missed the WU history, thanks for you efforts.

Ed
Title: Re: Alternative Dashboard WU Issue
Post by: jgillett on June 11, 2019, 04:08:52 PM
Mine is running at http://tiggrweather.net/. Lookin' good. Keeping an eye out for the 'next day' shift.

Thanks for your efforts.
Title: Re: Alternative Dashboard WU Issue
Post by: gwwilk on June 11, 2019, 05:05:58 PM
I noticed that the precision is wrong on the four values for extremes and averages after ajax kicks in.  The fix was to add a separate class, 'convTemp0', to ajaxWDwx3.js and change the class of the values to 'convTemp0'.  You can see the results of this change on my test server (http://www.selincolnwx.info).
Code: [Select]
<!-- used to be if ($useWU) { // Use WU High/lows -->

<tr>
<td align="center" valign="top" style="text-align: center; border: 1px solid red;">
<?php echo 'Rec* High'?><br/>
<strong><span style="font-size: 16px"><span class="convTemp0"><?php echo strip_units(intval($thisDayMax)) . $uomTemp?></span></span></strong><br/>
<?php echo $thisDayMaxYear?>
</td>
<td align="center" valign="top" style="text-align: center; border: 1px solid blue;">
<?php echo 'Rec* Low'?><br/>
<strong><span style="font-size: 16px"><span class="convTemp0"><?php echo strip_units(intval($thisDayMin)) . $uomTemp?></span></span></strong><br/>
<?php echo $thisDayMinYear?>
</td>
</tr>

<!-- Optional this day Max / Min Avg -->
<?php if($showNoaaAvg) { ?>
<tr>
<td align="center" valign="top" style="text-align: center; border: 1px solid red;">
                    <?php echo 'Avg* High'?><br/>
<strong><span style="font-size: 16px"><span class="convTemp0"><?php echo strip_units(intval($thisDayMaxAvg)) . $uomTemp?></span></span></strong><br/>

</td>
<td align="center" valign="top" style="text-align: center; border: 1px solid blue;">
<?php echo 'Avg* Low'?><br/>
<strong><span style="font-size: 16px"><span class="convTemp0"><?php echo strip_units(intval($thisDayMinAvg)) . $uomTemp?></span></span></strong><br/>

</td>
</tr>
<?php ?>
<!-- END optional Avg -->

I've attached the compressed version of my modified ajaxWDwx.js.txt along with the uncompressed version.

And, yes, I'm fussy about numerical precision...
Title: Re: Alternative Dashboard WU Issue
Post by: jgillett on June 11, 2019, 05:12:03 PM
Jerry, I still see .0°F on all four of the new values on your server, or am I missing something.

Thanks.
Title: Re: Alternative Dashboard WU Issue
Post by: gwwilk on June 11, 2019, 05:22:55 PM
Jerry, I still see .0°F on all four of the new values on your server, or am I missing something.

Thanks.
My main site still displays my locally derived values with the proper numerical precision.  Go to http://selincolnwx.info to see the adjusted values derived from NOAA.
Title: Re: Alternative Dashboard WU Issue
Post by: jgillett on June 11, 2019, 05:30:36 PM
My main site still displays my locally derived values with the proper numerical precision.  Go to http://selincolnwx.info to see the adjusted values derived from NOAA.

Using the link you provided in your original post is where I saw the .0°F temps. Now using the new link immediately above I see no decimal temp numbers - or is it FF again?
Title: Re: Alternative Dashboard WU Issue
Post by: gwwilk on June 11, 2019, 05:31:55 PM
My main site still displays my locally derived values with the proper numerical precision.  Go to http://selincolnwx.info to see the adjusted values derived from NOAA.

Using the link you provided in your original post is where I saw the .0°F temps. Now using the new link immediately above I see no decimal temp numbers - or is it FF again?
I think it was a browser cache issue with FF?  I don't use FF much, and just upgraded to the Developer Edition here.
Title: Re: Alternative Dashboard WU Issue
Post by: jgillett on June 11, 2019, 05:35:10 PM
I think it was a browser cache issue with FF?

Tried clearing and also a completely different browser -same result.
Title: Re: Alternative Dashboard WU Issue
Post by: gwwilk on June 11, 2019, 05:37:07 PM
I think it was a browser cache issue with FF?

Tried clearing and also a completely different browser -same result.
What's the URL?
Title: Re: Alternative Dashboard WU Issue
Post by: the beteljuice on June 11, 2019, 05:47:00 PM
@all Test Pilots ...

Please REMOVE

</body>
</html>

bottom lines in noaarec.php

Original zip has been updated,EDIT: Redundant material removed ... post for historic information only

Good to see so many getting it looking OK (Means the instructions were sound)
Glad to see a 'stand-aloner' - this could spread .....

Now if only it reports the right data for 'this day' for most of the time, we are on a winner.

Fingers crossed ..
Title: Re: Alternative Dashboard WU Issue
Post by: jgillett on June 11, 2019, 06:03:40 PM
I think it was a browser cache issue with FF?

Tried clearing and also a completely different browser -same result.
What's the URL?

In #55 I click directly on the 'my test server' link - http://www.selincolnwx.info/
In #57 you provided the direct URL - http://selincolnwx.info/

Both show the same result.
Title: Re: Alternative Dashboard WU Issue
Post by: K3JAE on June 11, 2019, 07:45:09 PM
I'm gonna hold off implementing this until the bugs are worked out. This also appears to be only for NON-ajax "Saratoga templates" type sites?  While I do use Saratoga's template it is highly customized and does not follow the normal v2 look/layout.

If this ends up being a drop and play script then I can implement it in the appropriate location n my site.
Title: Re: Alternative Dashboard WU Issue
Post by: the beteljuice on June 11, 2019, 07:58:36 PM
Quote
This also appears to be only for NON-ajax "Saratoga templates" type sites?
NO, this is an anywhere script, but the html patch is specifically for alt dashboard.php (which has it's own js / css unit conversion routine)

In any other 'site' if you wish to display metric units you would have to investigate what you have or create a script to convert values (and / or language, but I think that is probably an unnecessary overhead)

If your page is reloaded on a unit change, and that uom is posted in the url, OR if there is an available php 'flag' set BEFORE include('noaarec.php') then you could put a conversion script directly in noaarec.php

EDIT:
A quick look at your ajaxWDwx.js says ..
Code: [Select]
For static field that don't have an "ajaxtag" you need to add one of the
following classes to your fields:
    ----------------------------------------------------------------------------
    <span class="convTemp">82.7&deg;F</span>   ...for temp
So the html should be very similar to the patch, just change eg.
Code: [Select]
echo strip_units($thisDayMinAvg)
to
Code: [Select]
echo $thisDayMinAvg I think uom is carried around, but you may have to 'hard code' &deg;F is it isn't seen.

EDIT: Anyone willing to try it with the 'standard' dashboard ?
I don't know if convTemp exists in anything other than the WD ajax variant ????
Title: Re: Alternative Dashboard WU Issue
Post by: the beteljuice on June 12, 2019, 06:51:14 AM
Dear Readers ..

Jerry has discovered that for his locale data is incorrect between midnight and 01:30 local time.

Any others please .....
Title: Re: Alternative Dashboard WU Issue
Post by: jgillett on June 12, 2019, 10:31:41 AM
Sometime between 0030 MST and 0700 MST my data changed from what I had starting at 1400 MST yesterday. Will see if I can pin it down tonight (i.e., stay awake that long!).

Thanks again, BJ, for your work on this.

EDIT: Just checked the noaarecdat file. It shows 6/12 @ 0653 server time. Allowing for that gives 0553 MST which is too wide a gap with 'old' (yesterday) data.

Also, I'd like to see that file in /cache/, which I believe most of us are using.
Title: Re: Alternative Dashboard WU Issue
Post by: PSJohn on June 12, 2019, 01:41:33 PM
Dear Readers ..

Jerry has discovered that for his locale data is incorrect between midnight and 01:30 local time.

Any others please .....

Beta update: I monitored data last night just before and after midnight local time and saw no errors. My local NWS office updates the Climatological Report just after 5am. I checked at 5:30 with no update to data, but after the cache expired it updated and working as expected.


a thought.... with my local NWS office, the Climatological Report doesn't get updated until after 5am and sometimes later on weekends. Could you scrape the time the report gets updated so it could become a variable (e.t. $timeupdated)?

Title: Re: Alternative Dashboard WU Issue
Post by: the beteljuice on June 12, 2019, 03:40:43 PM
Notes:

Time of data update (and therefore syntax and position of required data 'block') = unknown for any specific locale and maybe flexible.

Location of cache file - you can do what you like once I can successfully second-guess them !

The beteljuice fuzzy logic will prevail over NOAA ...
... be patient !
Title: Re: Alternative Dashboard WU Issue
Post by: the beteljuice on June 12, 2019, 05:13:43 PM
OH NOAA do like to be totally confusing ...

Code: [Select]
Local Server: Wednesday 12th of June 2019 03:15:04 PM CDT UTC -05:00

...THE LINCOLN NE CLIMATE SUMMARY FOR JUNE 11 2019...
So it's midafternoon and they are giving the data for yesterday

.. and ATM at the bottom of the page ....
Code: [Select]
THE LINCOLN NE CLIMATE NORMALS FOR TODAY (NB: this sometimes says TOMORROW !)
                         NORMAL    RECORD    YEAR
 MAXIMUM TEMPERATURE (F)   83       101      1987
 MINIMUM TEMPERATURE (F)   60        43      1974
So .. is this 'Today' the report date 11th or 'real' today 12th ?
EDIT: The figures seem to be 'real' today checked against another source, but over the past / next few days figures for test locale only change by +/- 1 so can't be sure
also the sunrise / set times below are for the 12th and 13th !!!

So dear test pilots ....
You need to run noaarec.php?naaobug and check;
a) Your local time date
b) report time date
c) Wording at the bottom (ie. TODAY or TOMORROW)
d) check if sunrise / set are yesterday / today OR today / tomorrow

Got that - clear as mud  :lol: :lol:  ](*,)

PS. Anyone know of a separate page / resource to get a locales 'this day' so that we can double check which 'day' this page is really giving us ?

EDIT: Just had another thought ....
What if

...THE LINCOLN NE CLIMATE SUMMARY FOR JUNE 11 2019...

...............
TEMPERATURE (F)
(which was) YESTERDAY

That would make some kind of crazy sense
Title: Re: Alternative Dashboard WU Issue
Post by: gwwilk on June 12, 2019, 05:37:08 PM
I think they don't quite have their scripts in order.  While you see the 11th at the beginning of the Summary, it clearly states on down below, 124 AM CDT WED JUN 12 2019, that it's the 12th they're talking about.  Here's the entire summary as of 4:30pm CDT:
Code: [Select]
Running in DeBug Mode
...THE LINCOLN NE CLIMATE SUMMARY FOR JUNE 11 2019...

CLIMATE NORMAL PERIOD 1981 TO 2010
 MAXIMUM TEMPERATURE (F)   83       101      1987
This Day Record Max. Temp: 101°F @ 1987
Norm. 83°F
 MINIMUM TEMPERATURE (F)   60        43      1974
This Day Record Min. Temp: 43°F @ 1974
Norm. 60°F
Original Document ....

These data are preliminary and have not undergone final quality control by the National Climatic Data Center (NCDC). Therefore, these data are subject to revision. Final and certified climate data can be accessed at the NCDC - http://www.ncdc.noaa.gov.
Climatological Report (Daily)

566
CDUS43 KOAX 120626
CLILNK


CLIMATE REPORT
NATIONAL WEATHER SERVICE OMAHA NEBRASKA
124 AM CDT WED JUN 12 2019


...................................

...THE LINCOLN NE CLIMATE SUMMARY FOR JUNE 11 2019...

CLIMATE NORMAL PERIOD 1981 TO 2010
CLIMATE RECORD PERIOD 1887 TO 2019


WEATHER ITEM   OBSERVED TIME   RECORD YEAR NORMAL DEPARTURE LAST
                VALUE   (LST)  VALUE       VALUE  FROM      YEAR
                                                  NORMAL
..................................................................
TEMPERATURE (F)
 YESTERDAY
  MAXIMUM         77   1134 AM 103    1918  83     -6       94
  MINIMUM         57   1132 PM  44    1903  60     -3       68
  AVERAGE         67                        71     -4       81

PRECIPITATION (IN)
  YESTERDAY        0.20          3.03 2008   0.16   0.04     0.02
  MONTH TO DATE    1.72                      1.59   0.13     1.57
  SINCE JUN 1      1.72                      1.59   0.13     1.57
  SINCE JAN 1     15.13                     11.93   3.20     8.33

SNOWFALL (IN)
  YESTERDAY        0.0           0.0  2007   0.0    0.0      0.0
                                      2001
  MONTH TO DATE    0.0                       0.0    0.0      0.0
  SINCE JUN 1      0.0                       0.0    0.0      0.0
  SINCE JUL 1     55.5                      25.9   29.6     21.4
  SNOW DEPTH       0

DEGREE DAYS
 HEATING
  YESTERDAY        0                         1     -1        0
  MONTH TO DATE    2                        12    -10        0
  SINCE JUN 1      2                        12    -10        0
  SINCE JUL 1   6523                      6052    471     5999

 COOLING
  YESTERDAY        2                         7     -5       16
  MONTH TO DATE   73                        62     11      122
  SINCE JUN 1     73                        62     11      122
  SINCE JAN 1    145                       141      4      326
..................................................................


WIND (MPH)
  HIGHEST WIND SPEED    26   HIGHEST WIND DIRECTION     S (200)
  HIGHEST GUST SPEED    35   HIGHEST GUST DIRECTION    SW (210)
  AVERAGE WIND SPEED     8.2




RELATIVE HUMIDITY (PERCENT)
 HIGHEST    93
 LOWEST     33
 AVERAGE    63

..........................................................


THE LINCOLN NE CLIMATE NORMALS FOR TODAY
                         NORMAL    RECORD    YEAR
 MAXIMUM TEMPERATURE (F)   83       101      1987
 MINIMUM TEMPERATURE (F)   60        43      1974


SUNRISE AND SUNSET
JUNE 12 2019..........SUNRISE   555 AM CDT   SUNSET   859 PM CDT
JUNE 13 2019..........SUNRISE   555 AM CDT   SUNSET   859 PM CDT


-  INDICATES NEGATIVE NUMBERS.
R  INDICATES RECORD WAS SET OR TIED.
MM INDICATES DATA IS MISSING.
T  INDICATES TRACE AMOUNT.




BCM





The U.S. Naval Observatory (USNO) computes astronomical data. Therefore, the NWS does not record, certify, or authenticate astronomical data. Computed times of sunrise, sunset, moonrise, moonset; and twilight, moon phases and other astronomical data are available from USNO's Astronomical Applications Department (http://www.usno.navy.mil). See http://www.usno.navy.mil/USNO/astronomical-applications/astronomical-information-center/litigation for information on using these data for legal purposes.

As a double-check I have an alternate source for the data on my main site (https://www.gwwilkins.org/).  The link for the averages there has averages available  for almost everywhere in the US (https://www.usclimatedata.com/climate/united-states/us).
Title: Re: Alternative Dashboard WU Issue
Post by: the beteljuice on June 12, 2019, 06:07:47 PM
@Jerry ...

Your locale has just switched report dates  :shock:
Code: [Select]
Local Server: Wednesday 12th of June 2019 04:53:37 PM CDT UTC -05:00

...THE LINCOLN NE CLIMATE SUMMARY FOR JUNE 12 2019...
We have now switched the data block we are scraping and that is giving correct values ('yesterday' has become 'today', and 'today' has become 'tomorrow') (sun rise / sets still 12th / 13th)

See what they do (If anything) at midnight local ....

It's beginning to look like:
Quote
EDIT: Just had another thought ....
What if

...THE LINCOLN NE CLIMATE SUMMARY FOR JUNE 11 2019...

...............
TEMPERATURE (F)
(which was) YESTERDAY

That would make some kind of crazy sense
Might be the 'key' ....
Title: Re: Alternative Dashboard WU Issue
Post by: gwwilk on June 12, 2019, 06:13:23 PM
Right you are, BJ!  It says
'...THE LINCOLN NE CLIMATE SUMMARY FOR JUNE 12 2019...
VALID TODAY AS OF 0400 PM LOCAL TIME'
but it didn't update until around 5:00 pm CDT.

Your script has been and is still pulling up the right values for today, so you're doing something right!
Title: Re: Alternative Dashboard WU Issue
Post by: PSJohn on June 12, 2019, 06:14:00 PM

So dear test pilots ....
You need to run noaarec.php?naaobug and check;
a) Your local time date
b) report time date
c) Wording at the bottom (ie. TODAY or TOMORROW)

as of 3:10pm... report date: 6/12 5:19am...

My data is displaying correct for today(12th) (117° and 47°). But Running ?noaabug shows yesterday the 11th. As mention by gwwilk, the date is pulled from top section when it should pull from the lower "THE PALM SPRINGS INTL AIRPORT CA CLIMATE NORMALS FOR TODAY" section.

Take a look: https://desertweather.com/temprecavg/index.php?noaabug


 

Title: Re: Alternative Dashboard WU Issue
Post by: jgillett on June 12, 2019, 06:30:49 PM
Things got a bit toasty today but there's more to come.

Seems it's not sure about what the date is - report for June 12 but summary for June 11.
Title: Re: Alternative Dashboard WU Issue
Post by: PSJohn on June 12, 2019, 06:35:14 PM
Things got a bit toasty today but there's more to come.

Glad to see someone else enjoying the southwest heat! 113° here! :lol:
Title: Re: Alternative Dashboard WU Issue
Post by: jgillett on June 12, 2019, 06:43:18 PM
Things got a bit toasty today but there's more to come.
Glad to see someone else enjoying the southwest heat! 113° here! :lol:

Wildfires already started for both of us and unfortunately the really nice weather we had is now gone. Gettin' too old for this!
Title: Re: Alternative Dashboard WU Issue
Post by: CW2274 on June 12, 2019, 06:58:00 PM
And here I am whining about 108F..... :roll:
Title: Re: Alternative Dashboard WU Issue
Post by: gwwilk on June 12, 2019, 06:59:02 PM
Ahem, I'm not the net police, but we're straying from the topic here, guys. :roll:
Title: Re: Alternative Dashboard WU Issue
Post by: CW2274 on June 12, 2019, 07:00:13 PM
Ahem, I'm not the net police, but we're straying from the topic here, guys. :roll:
Please....
Title: Re: Alternative Dashboard WU Issue
Post by: PSJohn on June 12, 2019, 07:01:55 PM
edit: uh oh, back to the topic...   :oops: :grin:

So doing some research I figured out that my NWS office updates the report at least twice a day. I hope this makes sense...

Around 5:15am we get the first data for that day. The"top" main section displays yesterday's data and near the bottom displays the climate norms for today.
then around 4 to 5pm the upper main section will display the results of today and the bottom section switched to tomorrow's climate norms.

I'm not sure if all NWS locales follow that same format.

Maybe be there was some logic that checks if line 24 equal "TODAY" grab the data there BUT if it equals "YESTERDAY" then grab from line 81. (line numbers maybe off by a few but just use as a <pre> location reference)

Just some of my thoughts... :)
Title: Re: Alternative Dashboard WU Issue
Post by: the beteljuice on June 12, 2019, 07:53:31 PM
@ PSJohn Unfortunately your record date has now rolled over (But you are still getting the correct data)

In terms of yesterday / today / tomorrow we are still trying to work out with reference to what / which bit.

Such is the semantics / syntax of NOAA
Title: Re: Alternative Dashboard WU Issue
Post by: gwwilk on June 13, 2019, 01:15:51 AM
Via Splinterware's System Scheduler (https://www.splinterware.com/products/scheduler.html) I scheduled a 'curl (https://curl.haxx.se/windows/)' to run 'noaarec.php?noaabug=true' at every 12:04 am, and after it ran just now the correct numbers for today were displayed on my testing server (http://selincolnwx.info).

At least for me, the 'noaarec.php' script works, but with my modifications of which BJ is aware.
Title: Re: Alternative Dashboard WU Issue
Post by: Silversword on June 13, 2019, 01:39:46 AM
Hi Jerry,

As John pointed out on your test site the numbers do not show the degree decimals for some reason.  I am using FF here as well. See below the snapshot that I just took early this evening HST.

 [ You are not allowed to view attachments ]

I hope to get mine up when my time permits.

WOW, what great inputs to find a solution by all contributors. I posted this on June 6 and less than a week great progress to a beta testing for this issue.

Thank you all.

On another note of some NWS updates of data that takes place at 5:00 AM is because, I think, that is when they collect data from various recording stations. Just my thought on this.

Regards,

--Stan Y.
   Maui, Hawaii
Title: Re: Alternative Dashboard WU Issue
Post by: the beteljuice on June 13, 2019, 03:48:57 AM
Dear All .... patience.

@Jerry
Quote
and after it ran just now the correct numbers for today
The latest version was OK after midnight anyway .... It's the next record update that screws things.

EDIT: Redundant material removed ... post for historic information only
 OK so far, just need to wait for the next change ...

In the release (If that happens  :lol:) I will be changing the html slightly as I believe the uom should be hard coded to deg F - but we'll see.

the beteljuice has been wrong before - honest  \:D/
Title: Re: Alternative Dashboard WU Issue
Post by: gwwilk on June 13, 2019, 07:34:13 AM
@Stan:
The absence of the decimal points on my testing server's hi/lo/avg values is due to a modification I made in the ajax javascript file that's on that machine.  I referenced it earlier in this thread (https://www.wxforum.net/index.php?topic=37154.msg382765#msg382765).

@BJ:
Hard-coded uom's would disable the ajax-dashboard695's ability to convert the dashboard between Metric and Imperial displays.

Regarding the NOAA update times, the times on the dropdown selections seem to me to be the important parameter, not the text itself.  So long as TODAY and TOMORROW are clearly delineated we should have a shot at scraping them.  Here, TOMORROW's values are valid until the next update between 1:29 am and 1:39 am.  Are you saying that the next update caused the displayed numbers to revert back to yesterday's again?
Title: Re: Alternative Dashboard WU Issue
Post by: the beteljuice on June 13, 2019, 12:29:09 PM
Quote
Hard-coded uom's would disable the ajax-dashboard695's ability to convert the dashboard between Metric and Imperial displays.
I don't think so. initial uom appears to be what you have set up as station default (but regardless of that. initial state of theses values is deg F)
ajax js gets involved and convTemp takes the string eg. 12.6&deg;F and replaces it if necessary. ie. the page is NOT being refreshed and PHP plays no part. It appears to be working because US 'choice' of units is F and the figures we are scraping are F.

Try eg.
Code: [Select]
<span class="convTemp"><?php echo $thisDayMax?>&deg;F</span>and tell me  what happens please.


Title: Re: Alternative Dashboard WU Issue
Post by: gwwilk on June 13, 2019, 12:53:08 PM
Here's the code from that part of the line in my dashboard on my testing server:
Code: [Select]
<span class="convTemp0"><?php echo strip_units(intval($thisDayMax)) . $uomTemp?></span>
Changing it to
Code: [Select]
<span class="convTemp0"><?php echo $thisDayMax?>&deg;F</span>produces the desired results:
(https://www.gwwilkins.org/temp-images/ImperialRecords.png) and
(https://www.gwwilkins.org/temp-images/MetricRecords.png)

However if for some reason the default uomTemp is set to METRIC, I'm not sure it would display correctly.  Since Scott designed the ajax-dashboard6 to be more or less universal, I've reverted to the prior line.
Title: Re: Alternative Dashboard WU Issue
Post by: Silversword on June 13, 2019, 01:00:07 PM
Hi All,

Got my site running for testing out here in Central Pacific zone last night and this morning. Check out www.makawaoweather.com (http://www.makawaoweather.com)

Hope all goes well with modifications.

Thanks Jerry and BJ for the time to resolve this issue that WU dropped.

Regards,

--Stan Y.
   Maui, Hawaii
Title: Re: Alternative Dashboard WU Issue
Post by: the beteljuice on June 13, 2019, 01:51:25 PM
@Stan

I note that 'ajax' is adding val .0 - is the number of decimals an option somewhere ?
Title: Re: Alternative Dashboard WU Issue
Post by: PSJohn on June 13, 2019, 02:00:56 PM
Here is an idea... Why don't we simplify it?

I spent a few moments and took a look at several NWS locales and how their Climate reports look like at the end of the day around 11:45pm. They ALL have a tomorrow section towards the bottom at the end of the day. (see attached photo).

Make the script scrape the data from there. Then each one of us will create a cron to run the script once a day around 11:55pm. That way we get the data for tomorrow and we don't have to worry about the "flips" in the report format.

What do you think BJ?
Title: Re: Alternative Dashboard WU Issue
Post by: gwwilk on June 13, 2019, 02:04:55 PM
@Stan

I note that 'ajax' is adding val .0 - is the number of decimals an option somewhere ?
The numerical precision is a result of the ajaxWDwx3.js script.  See above for a fix, since I'm fussy about falsely precise numbers being displayed.

With this ajaxWDwx3.js using convTemp0 instead of convTemp displays no decimals for the BJ-scripted values.  The modified dashboard code is in post #55 of this thread.
Title: Re: Alternative Dashboard WU Issue
Post by: Silversword on June 13, 2019, 02:13:26 PM
@BJ,

I really don't know where the settings are located for the .0 is coming from. It could be coming from the config script for the Ajax alternative dashboard? I just loaded it as per your instructions with guidance from John G.
 
Thanks for all that you have done with this issue.

Regards,

--Stan Y.
  Maui, Hawaii
Title: Re: Alternative Dashboard WU Issue
Post by: the beteljuice on June 13, 2019, 04:41:58 PM
@PSJohn

What is this fixation you all seem to have with cron ?

Quote
end of the day around 11:45pm.
Whose / which day ?
Unlikely to be your local TZ, and it's looking like the reports are generated to some UTC schedule.

the beteljuice believes he has cracked the semantics and record day issues. The coming hours will tell.

Forgot to mention
Quote from: you said
Maybe be there was some logic that checks if line 24 equal "TODAY" grab the data there BUT if it equals "YESTERDAY" then grab from line 81. (line numbers maybe off by a few but just use as a <pre> location reference)
If you look at the source of noaarec that is indeed very similar to the imperfect logic employed - so well done !
Title: Re: Alternative Dashboard WU Issue
Post by: PSJohn on June 13, 2019, 08:16:00 PM
@PSJohn

What is this fixation you all seem to have with cron ?

Cron is our leader, the ruler of all! He helps us control the chaos in the code!

ha ha, Just kidding    :grin:

Thanks again for you efforts with this!  =D&gt;
Title: Re: Alternative Dashboard replace WU - Test Pilot Wanted
Post by: Bunty on June 14, 2019, 06:49:29 AM
Test Pilot wanted ....

Must be able to feed-back problems with instructions, html layout, actual functionality ..

(Keep an eye on data to see if it 'rolls over' at an acceptable time)

NB: This can be used as a 'stand-alone', no dependencies on other scripts - data available for US only !

 [ You are not allowed to view attachments ]  (errors fixed)
Sorry, but I don't understand how to  the "strip_units($thisDayMax)) { echo "\"Official"; } ?> "; } ?>" part.
Title: Re: Alternative Dashboard WU Issue
Post by: gwwilk on June 14, 2019, 07:10:39 AM
@Bunty:

The 'strip_units' function is found in Scott's ajax-dashboard6 which I see you have a highly modded version of available as your '2nd Home Page'.  You're not displaying the extreme/average daily hi/lo's that I can see.
Title: Re: Alternative Dashboard WU Issue
Post by: the beteljuice on June 14, 2019, 08:37:36 AM
'Release' will be shortly available under a new thread, which will lead you to http://beteljuice.co.uk/DEMOS/_recnoaa.php (I think all the correct versions are in place  :roll:)
Example html patch has also been updated / simplified.
Test bed checked over last 40hrs without problems.

@bunty

Do you mean the display graphics part ?
I have no idea if you have those graphics or not, or if you have the 'option' to set anywhere - sorry
You use an ajaxWD js so convTemp should work, no need for strip_units( .... ).

As far as I can see, if you want the records below 2nd home page yesterday / today info, then paste new <tr> .... </tr> inbetween </tr> in a new <table> @ view-source line #693 below existing one as the 'style' does not match yours, upto you to format.
Title: Re: Alternative Dashboard WU Issue
Post by: gwwilk on June 14, 2019, 09:07:04 AM
@BJ:

Thanks!  Looking at and running the latest script it looks like you nailed it [tup]  Great job =D&gt; \:D/ =D&gt;
Title: Re: Alternative Dashboard WU Issue
Post by: the beteljuice on June 14, 2019, 11:00:39 AM
OK Folks ...

the beteljuice believes his work is done !!!

Script can be found in NEW TOPIC ... here (https://www.wxforum.net/index.php?topic=37220)

Hope it works for you and original post has been resolved  \:D/
Title: Re: Alternative Dashboard WU Issue
Post by: mikeym2m on June 17, 2019, 05:24:13 PM
Working good here - THANKS    [tup]

Cheers

 :grin:

MikeyM

www.mikeymsweather.com