WXforum.net

Web Weather => Weather Website PHP/AJAX scripting => Custom Website Templates => Topic started by: the beteljuice on June 14, 2019, 10:54:44 AM

Title: 'This day' Max / Min / Avg Temperatures (NOAA) - PHP Script
Post by: the beteljuice on June 14, 2019, 10:54:44 AM
beteljuice NOAA this day temperature records / 'norms' - requires inclusion in a .php page

Ver #4.0 now available ... 27 Aug 2021

Data available for US only - sorry

This was originally meant to replace WU data in ALT dashboard, but is quite happy as a 'stand-alone'.
See this thread for discussion > https://www.wxforum.net/index.php?topic=37154.0

See this post for explanation of Ver #3.0 > https://www.wxforum.net/index.php?topic=37220.msg384607#msg384607

The script file can be used as a stand-alone - NO dependences - Be aware units are deg F
For stand-alone use php variables available are:
   $thisDayMax, $thisDayMaxYear
   $thisDayMin, $thisDayMinYear
   $thisDayMaxAvg (Norm)
   $thisDayMinAvg (Norm)
   
See running in debug mode and get zip > FROM HERE < (http://beteljuice.co.uk/DEMOS/_recnoaa.php)

betel_noaa.zip contains:

noaarec.php    include file that does all the work
patch.html      replacement for section of ALT dashboard / example of useage
README.txt     Do what it says !

Upon execution a cachefile will be created called noaarecdat.txt

TIP - Get noaarec.php?noaabug working BEFORE you commit to modifying dashboard

Ver #3.0
noaarec.php - Improved debug, error handling, comments.
                    copes with upto 48hrs of NOAA errors.
patch.html   - No change
README.txt  - Minor change

Ver #4.0
noaarec.php - Mods by Ken True for url changes and call methodology (Thanks  ;) )

Use or abuse - upto you  :-P
Title: Re: 'This day' Max / Min / Avg Temperatures (NOAA) - PHP Script
Post by: the beteljuice on June 20, 2019, 08:11:18 AM
Version #2 - 20th June 2019

See above - improved debug and instructions ...
Title: Feedback Please ..... Temperatures (NOAA) - PHP Script
Post by: the beteljuice on June 24, 2019, 07:01:10 AM
Knock ... Knock ...

Feedback please.

Topics concerning WD alt-dashboard and loss of WU 'this date' temps.
 views > 2000, posts > 200

This replacement, which can also be used in any (US) template / page.
 views 200, others posts 0  #-o

Is anyone using it ?
Is there anything that readers of this post / debug / demo page don't understand ?  :shock:
Is there anybody out there   :???:
Title: Re: 'This day' Max / Min / Avg Temperatures (NOAA) - PHP Script
Post by: mikeym2m on June 25, 2019, 11:21:28 AM
Working good here - www.mikeymsweather.com

Thanks

Cheers

 [tup]

MikeyM
Title: Re: 'This day' Max / Min / Avg Temperatures (NOAA) - PHP Script
Post by: tbrasel on June 25, 2019, 01:07:11 PM
Yes, I am using your script without any issues.

Thank you.
Title: Re: 'This day' Max / Min / Avg Temperatures (NOAA) - PHP Script
Post by: Silversword on June 25, 2019, 05:31:15 PM
Hi beteljuice,

Got this started on the other topic and thank you for finding a solution to this issues that WU has dropped.

Working just fine so far as long as our NWS does not do any changes as they sometimes do.

I see on the other WD Forum that they are having issues with this. Here: https://www.weather-watch.com/smf/index.php/topic,68068.0.html (https://www.weather-watch.com/smf/index.php/topic,68068.0.html)

Thank you again for all that you have done and I will post it on the other part as resolved.

Best Regards,

--Stan Y.
   Maui, Hawaii
Title: Re: 'This day' Max / Min / Avg Temperatures (NOAA) - PHP Script
Post by: gwwilk on June 25, 2019, 09:35:03 PM
You should know that I'm using your script on my testing site (http://www.selincolnwx.info), and it's working just fine. [tup]
Title: Re: 'This day' Max / Min / Avg Temperatures (NOAA) - PHP Script
Post by: jgillett on June 26, 2019, 11:36:25 AM
Working fine here - http://tiggrweather.net/. And I know it's right - one does not forget temps like this ORH (record setting anniversary is today, 26 June 1990)...
Title: Re: 'This day' Max / Min / Avg Temperatures (NOAA) - PHP Script
Post by: thebigboi on June 26, 2019, 12:18:01 PM
Can you please look at my site http://murfreesboroweather.com/ and tell me what I'm doing wrong. thanks.
Title: Re: 'This day' Max / Min / Avg Temperatures (NOAA) - PHP Script
Post by: Silversword on June 26, 2019, 12:29:45 PM
Can you please look at my site http://murfreesboroweather.com/ and tell me what I'm doing wrong. thanks.

Hi,

I think you need to put this line into your dashboard script:

include_once('noaarec.php');

Hope that is the issue with yours as I had that issue.

Regards,

--Stan Y.
   Maui, Hawaii
Title: Re: 'This day' Max / Min / Avg Temperatures (NOAA) - PHP Script
Post by: the beteljuice on June 26, 2019, 05:35:47 PM
@thebigboi

Strange .... ATM NOAA don't have any record highs for you (although they do tomorrow)
http://murfreesboroweather.com/noaarec.php?noaabug

Code: [Select]
...............................................................
TEMPERATURE (F)
 TODAY
  MINIMUM         70    406 AM  51    1974  68      2

PRECIPITATION (IN)
  TODAY            0.00          3.23 1994   0.12  -0.12
  MONTH TO DATE    6.60                      3.63   2.97
  SINCE JAN 1     36.07                     24.93  11.14

It looks like noaarec.php is crashing in an unexpected way.

.... the beteljuice is investigating - thank you.
Title: Re: 'This day' Max / Min / Avg Temperatures (NOAA) - PHP Script
Post by: the beteljuice on June 26, 2019, 05:45:08 PM
@thebigboi

Ahhh . your TZ has been set to UTC

Try moving the include further up the page OR setting your TZ in noaarec.php

See if that fixes things .....

EDIT: Suddenly working on both your site and my test - must have been some unforeseen corruption on NOAA page.

You still need to fix your TZ problem ..
Title: Re: 'This day' Max / Min / Avg Temperatures (NOAA) - PHP Script
Post by: thebigboi on June 26, 2019, 06:58:57 PM
It looks like that did the trick. Thanks so much. I will watch and compare to the NOAA site. Right now, I have been locked out for being on their site too long. Like you noticed I don't know where it would get the high.
Title: Re: 'This day' Max / Min / Avg Temperatures (NOAA) - PHP Script
Post by: the beteljuice on June 26, 2019, 09:36:22 PM
Anyone a member of this forum https://www.weather-watch.com/smf/index.php/topic,68068.0.html and can give them "the nod" ?
Title: UPDATE: 'This day' Max / Min / Avg Temperatures (NOAA) - PHP Script
Post by: the beteljuice on June 27, 2019, 06:52:45 AM
UPDATE V2.1

noaarec.php - error handling for complete line of data missing (NOAA problem)

See first post for link .........
Title: Re: 'This day' Max / Min / Avg Temperatures (NOAA) - PHP Script
Post by: ed2kayak on June 27, 2019, 08:47:09 AM
I updated to v2.1
Only made changes in 1 file noaarec.php
Getting error -
 ** NO MAXIMUM temp line found !! **

** NO MINIMUM temp line found !! **

Worked previously, debug mode  does shows max/min temps.

cvweather.org

Thanks for your work on this project. Appreciate it!

Ed
Title: Re: 'This day' Max / Min / Avg Temperatures (NOAA) - PHP Script
Post by: 92merc on June 27, 2019, 12:54:06 PM
OK, I had it working for like 5 minutes.  Then it quit.  What am I missing?

https://www.bismarckweather.net/noaarec.php?noaabug

I'm at 2.1, but getting:
** NO MAXIMUM temp line found !! **

** NO MINIMUM temp line found !! **
Title: Re: 'This day' Max / Min / Avg Temperatures (NOAA) - PHP Script
Post by: 92merc on June 27, 2019, 01:11:18 PM
I'm guessing it has to be something in the way it's included into my standard ajax-dashboard.  I've added in the TZ setting to noaarec.php.  I've moved the include basically to the top.

But the debug page shows the right data.
https://www.bismarckweather.net/noaarec.php?noaabug
Title: Re: 'This day' Max / Min / Avg Temperatures (NOAA) - PHP Script
Post by: thebigboi on June 27, 2019, 02:16:33 PM
Well, it looks like the issues from yesterday have come back again today. It does not appear it is anything wrong with the script but rather the fact that the NOAA site does not publish the high temperatures (max and avg.) for much of the day. I ran the debug and it is returning the exact same thing as the NOAA site shows which is no max and avg. high temperatures. Oddly, it does show that very information for the next day. Is there any way to harvest that information and save it for the following day? The only reason I say that is that NOAA site is consistent in that it seems to always show the information for the next day.
Title: Re: 'This day' Max / Min / Avg Temperatures (NOAA) - PHP Script
Post by: 92merc on June 27, 2019, 04:01:11 PM
I've reverted my dashboard back to my own homebrewed solution until an answer is found.  It's easy for me to rename and flip back for testing.
Title: update fix: 'This day' Max / Min / Avg Temperatures (NOAA) - PHP Script
Post by: the beteljuice on June 27, 2019, 04:36:44 PM
Apologies to all !

beteljuice schoolboy error - missed $ on a var name  :oops: #-o

I don't have a working site, so can only test with the debug, which bulldozed through the problem.

Should be OK now. You'll all soon tell me if it isn't  \:D/
Title: Re: 'This day' Max / Min / Avg Temperatures (NOAA) - PHP Script
Post by: 92merc on June 27, 2019, 04:54:54 PM
After blowing away the noaarecdat.php file, I'm still getting erroneous data.

The debug pulls the right data.  But the noaarecdat.php file has the bad data, and the info on screen has the bad data.

I have the include statement in my dashboard.  And then this data for displaying it.

Code: [Select]
<tr style="text-align:center">
<td>Normal High: &nbsp;<?php echo $thisDayMaxAvg?>&deg;F</td>
<td>Normal Low: &nbsp;<?php echo $thisDayMinAvg?>&deg;F</td>
</tr>
<tr style="text-align:center">
<td>Record High: &nbsp;<?php echo $thisDayMax?>&deg;F &nbsp;<?php echo $thisDayMaxYear?></td>
<td>Record Low: &nbsp;<?php echo $thisDayMin?>&deg;F &nbsp;<?php echo $thisDayMinYear?></td>
</tr>
Title: Re: 'This day' Max / Min / Avg Temperatures (NOAA) - PHP Script
Post by: the beteljuice on June 27, 2019, 05:32:55 PM
Everybody hold fire whilst I triple check my new error handling ....  :oops:
Title: Re: 'This day' Max / Min / Avg Temperatures (NOAA) - PHP Script
Post by: the beteljuice on June 27, 2019, 05:48:39 PM
Must be having a bad bald day .... o/p logic in totally the wrong place.

Hopefully correct now ???
Title: Re: 'This day' Max / Min / Avg Temperatures (NOAA) - PHP Script
Post by: ed2kayak on June 27, 2019, 06:13:47 PM
Not working for me? :?

cvweather.org (http://cvweather.org)

Ed
Title: Re: 'This day' Max / Min / Avg Temperatures (NOAA) - PHP Script
Post by: the beteljuice on June 27, 2019, 06:18:36 PM
Oh yes it is ......
Title: Re: 'This day' Max / Min / Avg Temperatures (NOAA) - PHP Script
Post by: ed2kayak on June 27, 2019, 06:20:47 PM
Yes, you are correct. I'm must have been too impatient.  =D&gt;
Title: Re: 'This day' Max / Min / Avg Temperatures (NOAA) - PHP Script
Post by: the beteljuice on June 27, 2019, 06:38:15 PM
Note to all:

If you run noaarec.php?noaabug it will delete the existing cache-file and force a new data 'grab'.

(A bit of a two edged sword, I might put a 'flag' on it to 'see' cache regardless of age ....)
Title: Re: 'This day' Max / Min / Avg Temperatures (NOAA) - PHP Script
Post by: 92merc on June 27, 2019, 06:41:34 PM
Huzzah!  Looks like that did the trick.  I'm pulling data.
Title: Re: 'This day' Max / Min / Avg Temperatures (NOAA) - PHP Script
Post by: thebigboi on June 27, 2019, 08:30:03 PM
The Beteljuice, I have watched the NOAA site where the script pulls its information and the site looks like it refreshes several times a day. The early morning refreshes, up to around 7:00 a.m. include maximum and minimum temperature records whereas every refresh after that until approximately 6:00 p.m. does not include the maximum temperature records. Therefore, if I access the site between approximately 7:00 a.m. and 6:00 p.m. I get an error because apparently every time the website is accessed it pulls new info from the NOAA site.  I don't know if there is a work around but it looks like several sites have this same anomaly.
Title: Re: 'This day' Max / Min / Avg Temperatures (NOAA) - PHP Script
Post by: the beteljuice on June 27, 2019, 09:27:49 PM
The site(s) seem to typically update twice in 24hrs.
If there are subsequent updates because of incomplete data I don't know ?

Your cache-file is set for an 'age' of 1 hour (unless you request after local midnight upto 1 hour)

Are you sure the data is missing for your requested location ?
Quote
.... does not include the maximum temperature records.
Only the Max ? - very odd ...


The NOAA page is a little confusing because they are actually using English correctly  :shock:

You have to look at the date the report was created, together with the date the report is for ....

Generally speaking if you see "TODAY" (rather than Yesterday) in the top quarter of the report, those are the figures used, else the figures towards the bottom of the report which maybe labeled TODAY - OR - TOMORROW will be used.

I don't know if one of your locations is a part-time reporter, or why that would cause an omission of 'record' data ?

NOAA do say that the page is unconfirmed data awaiting ratification for publication elsewhere.
... or perhaps they are just having a bad few days, just because we are watching them !!!


Storing "Todays" and "Tomorrows" values (In the time window they both appear) in case of a failure tomorrow, together with date / time check would require a complete change of philosophy and effectively a new script.

Q. Have any 'missing' tomorrow values been discovered ?

If anyone else is having the same problem please report it. - some people may have to change their locale or not use the prog.
Title: Re: 'This day' Max / Min / Avg Temperatures (NOAA) - PHP Script
Post by: thebigboi on June 28, 2019, 08:56:08 AM
The Beteljuice, I will continue to monitor. My thought was to only have it update in the early morning hours and stay that way for the entire day since historic highs and lows don't change as the day progresses. This morning the NOAA site updated at 1:41 a.m. with all the "today" data being available. It did the same yesterday. However, later and I don't have the exact time it updated and only the lows were available with no reference to the highs. Is there a way to keep the cache that was fetched at certain hours for the balance of the day?
Title: Re: 'This day' Max / Min / Avg Temperatures (NOAA) - PHP Script
Post by: 92merc on June 28, 2019, 09:05:46 AM
I noticed mine hasn't updated yet.  It's 8:05 am local time.

noaarecdat was previously modified: Friday 28th of June 2019 12:07:23 PM UTC

Local Server NOW: Friday 28th of June 2019 01:04:35 PM UTC UTC +00:00

$wfo = bis
$sid = BIS

...THE BISMARCK ND CLIMATE SUMMARY FOR JUNE 27 2019...
Title: Re: 'This day' Max / Min / Avg Temperatures (NOAA) - PHP Script
Post by: thebigboi on June 28, 2019, 11:03:46 AM
The Betlejuice, you indicated the cache was held for one hour in noaarec.php which it appears this line "$check_it = time() - (60 * 60); // 1hr ago - should be able to increase to *6 or *12 if wanted" is what sets the cache reset. How would I increase the cache hold to eight hours? Would that simply mean modifying (60 * 60) with (480 * 480) or is it actual hours such as (8 * 8)? I just figured I would try that since the NOAA data changes during the day.
Title: Re: 'This day' Max / Min / Avg Temperatures (NOAA) - PHP Script
Post by: the beteljuice on June 28, 2019, 07:06:19 PM
@92merc AND @thebigboi
Quote
noaarecdat was previously modified: Friday 28th of June 2019 12:07:23 PM UTC

Local Server NOW: Friday 28th of June 2019 01:04:35 PM UTC UTC +00:00
Something has left server time in UTC.
As stated in prog and thread, try either moving 'include' higher up in your page OR set your TZ in the prog.

@thebigboi

Code: [Select]
$check_it = time() - (60 * 60); // 1hr ago - should be able to increase to *6 or *12 if wanted

// would become

$check_it = time() - (60 * 60 * 8); // 8hr ago - should be able to increase to *6 or *12 if wanted

... but that is not going to fix your problem, just 'offset' it.
Have you noticed yet if 'Tomorrow' data is always complete ?
Title: Re: 'This day' Max / Min / Avg Temperatures (NOAA) - PHP Script
Post by: thebigboi on June 28, 2019, 07:26:09 PM
The Betlejuice, I am still watching the NOAA output. So far today every update includes the tomorrow language. The last update was at 4:33 p.m. and included the following:
THE NASHVILLE CLIMATE NORMALS FOR TOMORROW
                         NORMAL    RECORD    YEAR
 MAXIMUM TEMPERATURE (F)   89       109      2012
 MINIMUM TEMPERATURE (F)   68        54      1974

It also included the information for today. So far it appears every update after around 8:00 a.m. to the afternoon update does not include full today data. It only includes the info for the minimums and not the maximums. I will keep checking and see what happens.
Thanks for being so patient.
Title: Re: 'This day' Max / Min / Avg Temperatures (NOAA) - PHP Script
Post by: thebigboi on June 28, 2019, 07:50:41 PM
The Betlejuice, in looking at the NOAA site, I did notice that the site has a NOWDATA category that includes a daily almanac that includes the information you scraped from the Observed Weather data but only changes once a day. First, I have no idea how you capture the information and second, I have no idea if there is a way to access this information via a php script. It was just an observation while perusing the NOAA site. Thanks.
Title: Re: 'This day' Max / Min / Avg Temperatures (NOAA) - PHP Script
Post by: the beteljuice on June 28, 2019, 08:29:06 PM
@thebigboi ...

... you still haven't got your TZ right !!!!

As an aside: I think NOAA reports are being done for 00:00 UTC and 12:00 UTC with an hour or so for them to to collate and 'publish'.
Title: Re: 'This day' Max / Min / Avg Temperatures (NOAA) - PHP Script
Post by: thebigboi on June 28, 2019, 08:59:21 PM
I moved the "include" script line to the top of ajax-dashboard6. How do I check it myself? How do I set the tz in the program?
Title: Re: 'This day' Max / Min / Avg Temperatures (NOAA) - PHP Script
Post by: the beteljuice on June 28, 2019, 09:45:15 PM
@thebigboi

using noaarec.php?noaabug gives you the current local server time - including TZ
Quote
Running in DeBug Mode
noaarecdat was previously modified: Saturday 29th of June 2019 01:35:01 AM UTC

Local Server NOW: Saturday 29th of June 2019 01:43:01 AM UTC UTC +00:00

In the script approx. line #20 > 23...
Code: [Select]
// there has been one case where a previous 'include' changed TZ to UTC
// If this happens 'include' noaarec.php higher uo the page OR uncomment and set TZ BELOW
// date_default_timezone_set("America/??????????");
Title: Re: 'This day' Max / Min / Avg Temperatures (NOAA) - PHP Script
Post by: thebigboi on June 29, 2019, 07:44:25 PM
I did get the tz issue resolved. So far, it appears the NOAA site updates three times a day at around 2:00 a.m. or thereabouts, at around 8:00 a.m. or thereabouts, and at around 5:00 p.m. or thereabouts. The only update that does not include the maximum record temperature is the 8:00 a.m. update. Oddly, the tomorrow records for both max and min temperature records are included in all updates. I will continue to monitor.
Title: Re: 'This day' Max / Min / Avg Temperatures (NOAA) - PHP Script
Post by: the beteljuice on June 29, 2019, 07:47:26 PM
@thebigboi ...

cross post whilst I was editing this ....

I haven't forgotten your plight !!!

I see you've finally got your TZ set  =D&gt;

... keeping an eye on your locale reports

quoted (not 'published') report times:

01:20 AM CDT = OK
08:07 AM CDT = FAIL all TODAY max values missing, and min 'so far' (null) with time (MM) - neither of which we use.
04:45 PM CDT = OK ('published' within 10 min)

Experimenting with a change of code / philosophy - It won't be foolproof, but should fix your problem as long as there is one 'visit' in a 'good' period in today and yesterday server time.

No promises, but if I get it working I'll PM you to test it  :shock:
Title: Re: 'This day' Max / Min / Avg Temperatures (NOAA) - PHP Script
Post by: thebigboi on June 29, 2019, 08:01:58 PM
@The Betlejuice

Thanks so much. I really do appreciate your continued patience with my questions.
Title: Re: 'This day' Max / Min / Avg Temperatures (NOAA) - PHP Script
Post by: the beteljuice on June 29, 2019, 08:12:22 PM
Quote from: thebigboi
...  continued patience with my questions.

It's the only way we all learn - Not a problem  #-o
Title: Re: 'This day' Max / Min / Avg Temperatures (NOAA) - PHP Script
Post by: the beteljuice on June 29, 2019, 09:08:21 PM
... and now they've updated again @ 07:50 PM CDT, only a 3hr gap (all data OK)

What a crazy schedule !!
Title: Re: 'This day' Max / Min / Avg Temperatures (NOAA) - PHP Script
Post by: thebigboi on June 29, 2019, 09:28:06 PM
@The Betlejuice

Yep, I just spotted it also. The only update I've seen so far that does not include the maximum temperature happens each day around 8:00 a.m. Makes no sense at all.
Title: UPDATE: 'This day' Max / Min / Avg Temperatures (NOAA) - PHP Script
Post by: the beteljuice on July 06, 2019, 03:53:09 PM
NEW ! - IMPROVED, Low Carbon, Eco friendly, nut free, self healing

OK Ver #3 now available ...

Currently being "field tested" and evaluated - available from link on first post or from here (http://beteljuice.co.uk/DEMOS/_recnoaa.php)

Humungous amount of beteljuice logic to store "this" server day and "tomorrow" (when available) server day historic temp data from NOAA.

As well as identifying correct parts of report to 'scrape', it will also try to 'heal' any missing data / data lines.

This is a bit confusing because NOAA say "report for (date) ..." then refers relatively to the report created date (eg. yesterday, today, tomorrow).

In the case where local server day has rolled over, but report created date has not - the references are incorrect.
Further - there is at least one location where for eight hours or more each day there are two lines of data completely missing (and of course one of them is what we want)

All far too lengthy to explain fully in a meaningful way - but if you're interested see the real life example below of two consecutive reports, one of which has incomplete data, and how the program handles the situation ...



A report just after 00:00 local .. (but still being 'published' for 8 hours plus)

Local Server Time NOW:  Friday 5th of July 2019 07:24:52 AM CDT, America/Mexico_City - UTC -05:00

CLIMATE REPORT
NATIONAL WEATHER SERVICE NASHVILLE TN
12:10 AM CDT FRI JUL 5 2019

...

...THE NASHVILLE CLIMATE SUMMARY FOR JULY 4 2019...

[program decides: recDay: 4, serverDay: 5, Tomorrow: 6, Use second block]

...

[1st block]
WEATHER ITEM   OBSERVED TIME   RECORD YEAR NORMAL DEPARTURE
                VALUE   (LST)  VALUE       VALUE  FROM
                                                  NORMAL
...............................................................
TEMPERATURE (F)
 YESTERDAY [program decides: correct, day(4) not wanted]
  MAXIMUM         91   1214 PM 103    2012  89      2
  MINIMUM         71    302 AM  55    1968  69      2
  AVERAGE         81                        79      2
 
 ...
 
[2nd block]
THE NASHVILLE CLIMATE NORMALS FOR TODAY
[program decides: correct, day(5) already stored, but checked for any missing values]
                         NORMAL    RECORD    YEAR
 MAXIMUM TEMPERATURE (F)   89       104      2012
 MINIMUM TEMPERATURE (F)   69        54      1968
 
 [program decides: NO genuine Tomorrow day(6) Data Available, only Ring NOAA upto once per hour]
 
 
 
 Now a report 09:00 local that has problems ... (still being 'published' upto 17:00 local - 8hrs)
 
  Local Server Time NOW:  Friday 5th of July 2019 10:48:49 AM CDT, America/Mexico_City - UTC -05:00

CLIMATE REPORT
NATIONAL WEATHER SERVICE NASHVILLE TN
9:22 AM CDT FRI JUL 5 2019

...THE NASHVILLE CLIMATE SUMMARY FOR JULY 5 2019
VALID AS OF 0900 AM LOCAL TIME.

[program decides: recDay: 5, serverDay: 5, Tomorrow: 6, Use first block]

...

[1st block]
TEMPERATURE (F)
TODAY [program decides: correct, day(5) already stored, but checked for any missing values]
[ALL MAXIMUM TEMP DATA MISSING !!]
  MINIMUM         72    512 AM  54    1968  69      3
[ALL AVERAGE TEMP DATA MISSING !!] 
PRECIPITATION (IN)
  TODAY            0.00          3.33 1950   0.12  -0.12
[program decides: use stored day(5) in cache-file]
...

[2nd block]
THE NASHVILLE CLIMATE NORMALS FOR TOMORROW
[program decides: correct, day(6) store in cache-file]
                         NORMAL    RECORD    YEAR
 MAXIMUM TEMPERATURE (F)   89       105      2012
 MINIMUM TEMPERATURE (F)    69        54      1972

[program decides: Now we have complete data for day(5), day(6) stored in cache-file, - don't Ring NOAA until day(6) (when we look for day7 data)]

It's pretty much impossible to second guess all the possible NOAA responses to report errors, but hopefully in the majority of cases this prog allows for upto 48 hours of NOAA 'failure' as long as your site has a visitor in the morning and one at night  ;)

For those already using this prog noaarec.php is a direct compatible replacement.
noaarecdat..php on your server is now redundant and may be deleted.
New noaarecdat.txt will be created / maintained.
Title: Re: 'This day' Max / Min / Avg Temperatures (NOAA) - PHP Script
Post by: 92merc on July 06, 2019, 09:21:38 PM
Looks good.  I have mine up and running.  Only different I noticed is there is a TXT file instead of a PHP file for the cache.  Hey, what ever works!

Thanks for your work.
Title: Re: 'This day' Max / Min / Avg Temperatures (NOAA) - PHP Script
Post by: thebigboi on July 07, 2019, 07:17:21 AM
Very good work. My site www.murfreesboroweather.com is working great. Thanks to The Betlejuice.
Title: Re: 'This day' Max / Min / Avg Temperatures (NOAA) - PHP Script
Post by: mikeym2m on July 07, 2019, 10:05:06 AM
Also working well here www.mikeymsweather.com

Thanks to Bettlejuice   [tup]

Cheers

  :-)

MikeyM
Title: Re: 'This day' Max / Min / Avg Temperatures (NOAA) - PHP Script
Post by: ed2kayak on July 09, 2019, 10:28:24 AM
Thanks Beteljuice, update in place and working. [tup]
Title: Re: 'This day' Max / Min / Avg Temperatures (NOAA) - PHP Script
Post by: thebigboi on July 14, 2019, 10:53:11 AM
The Betle Juice,

Everything has been working fine until yesterday and I noticed after the morning NOAA update I started getting odd information. I pulled the text file for today and it is doing it again. Using an NOAA update doesn't work in that it simply repeats the same text file. You can see the info on the webpage www.murfreesboroweather.com
Title: Re: 'This day' Max / Min / Avg Temperatures (NOAA) - PHP Script
Post by: the beteljuice on July 14, 2019, 06:43:14 PM
Too late .....

Everything has 'cleared' now !

So I don't know what the corruption was on the NOAA page.

Because there were figures to 'parse' the prog accepted them as genuine and stored them, and when those figures persisted with no obvious errors everything had to wait until next 'tomorrow' check.

If it happens again do noaarec.php?noaabug - view source and copy the actual report section either as txt, or if you do it from the <head> section as html.

Was it the same time time-slot that normally has missing data ? - perhaps they partially fixed it, but made things worse for us  ](*,)

Why doesn't NOAA like your location ??  :roll:
Title: Re: 'This day' Max / Min / Avg Temperatures (NOAA) - PHP Script
Post by: thebigboi on July 14, 2019, 06:53:28 PM
The Beteljuice,

Yes, it happens during the period when there is insufficient info. It certainly does look like they don't want us to use the data. Thanks.
Title: Re: 'This day' Max / Min / Avg Temperatures (NOAA) - PHP Script
Post by: the beteljuice on July 14, 2019, 07:03:51 PM
BTW ..

From your attached cachefile:

14,104,3,56,4,1,1

Translates as:

Day 14, MaxTemp 104, MaxYear 3, MinTemp 56, MinYear 4, MaxAvg 1, Minavg 1

So at least the Min/Max temp seems to be reported at last, but the rest  :lol:
Title: Re: 'This day' Max / Min / Avg Temperatures (NOAA) - PHP Script
Post by: the beteljuice on July 15, 2019, 10:36:22 AM
Hmpph ..

Today your 08:00 report has gone back to all Max and average data missing (Which the prog copes with) So someone must be playing somewhere ....

Watch that space  :roll:

Edit: Difficult to say without knowing what was in that 'crashing' report, but I suspect that only the first digits of year(s) and averages(s) were being produced else the cache values would look different (the 'fields' are in a different order to the report, but the min / max values were being parsed correctly)
Title: Re: 'This day' Max / Min / Avg Temperatures (NOAA) - PHP Script
Post by: K3JAE on July 29, 2019, 01:10:20 PM
Is there a specific time when these reports actually roll over to the current date? If so I will build a CRON event to auto grab these at a specific time. Based on what I've seen, MEG (assuming local WFO is the ones who actually run this) does them whenever they feel like it and may run it multiple times per day, once per day or within seconds of the first run.

Would be hard to schedule a event to grab the data unless I force a CRON event to run say every 4 hours.
Title: Re: 'This day' Max / Min / Avg Temperatures (NOAA) - PHP Script
Post by: thebigboi on July 29, 2019, 02:12:56 PM
My NOAA site (Nashville) usually puts out four different reports, early morning, late morning, early evening and late evening. It is  the late morning report that does not list the maximum high or average high. To date, the last version issued by The betlejuice has worked good.
Title: Re: 'This day' Max / Min / Avg Temperatures (NOAA) - PHP Script
Post by: K3JAE on July 29, 2019, 03:04:53 PM
I will check Memphis' reports to ensure all reports show the Daily HIGH/LOW records. I had not paid that too much attention. I believe I am using the latest version (v3) so I should be OK. I have not noticed any discrepancies and script appears to be working properly.
Title: Re: 'This day' Max / Min / Avg Temperatures (NOAA) - PHP Script
Post by: the beteljuice on July 29, 2019, 05:26:31 PM
... and script appears to be working properly.
... So why do you feel the need to cron  #-o
Title: Re: 'This day' Max / Min / Avg Temperatures (NOAA) - PHP Script
Post by: MackerelSky on December 07, 2019, 04:24:29 AM
Sir beteljuice,
Hope things are well.

If you get a moment, could you point me into the right direction to take care of a little hiccup with this script.

Day before yesterday and today, the record low temp for the Bradford Airport was 0°, but the script was throwing a -100° reading as seen below
(https://www.mckeanweather.com/1207ex.PNG)

This is from the noaarecdat.txt file
Code: [Select]
7,63,1998,,1970,36,21with the low temp missing.

From the URL to the Bradford site, this is what I see

(https://www.mckeanweather.com/1207exx.PNG)

Again, no rush as I don't see many more '0°' days coming down the pike, but just wanted to see if there's an easy fix.

All the Best beteljuice,

Brad
Title: Re: 'This day' Max / Min / Avg Temperatures (NOAA) - PHP Script
Post by: the beteljuice on December 07, 2019, 10:37:25 AM
The -100 value is an error 'response' from the script when neither the current value nor the cached value exist.

It is deliberately an absurd value and the other clue is the 'epoch' year of 1970  ;)
Title: Re: 'This day' Max / Min / Avg Temperatures (NOAA) - PHP Script
Post by: K3JAE on December 07, 2019, 10:45:09 AM
... and script appears to be working properly.
... So why do you feel the need to cron  #-o

.... I don't use a CRON for this script?  Where'd you get that idea? I use CRON only for the NWS-Alerts script. My reference above to CRON was if I had to pull that script multiple times per day.
Title: Re: 'This day' Max / Min / Avg Temperatures (NOAA) - PHP Script
Post by: the beteljuice on December 07, 2019, 10:50:03 AM
.... If so I will build a CRON event to auto grab these ..... and may run it multiple times per day, ...

... unless I force a CRON event to run say every 4 hours.
Title: Re: 'This day' Max / Min / Avg Temperatures (NOAA) - PHP Script
Post by: MackerelSky on December 07, 2019, 05:44:53 PM
The -100 value is an error 'response' from the script when neither the current value nor the cached value exist.

It is deliberately an absurd value and the other clue is the 'epoch' year of 1970  ;)

Hmmm,

Like I mentioned beteljuice, the day before yesterday and today both had a record low temp of 0°, for both the 5th and the 7th of December.

Not only did it show up in the URL for your script to the Bradford site, but they also show up for two different years from a NWS site at https://www.weather.gov/ctp/climateRecordsBradford if you scroll down to December in the list of records.
(https://mckeanweather.com/1207p4.PNG)

I truly believe that the actual record was 0° in both instances, but if there's nothing we can do, so be it.

Brad
Title: Re: 'This day' Max / Min / Avg Temperatures (NOAA) - PHP Script
Post by: the beteljuice on December 07, 2019, 07:15:59 PM
As you stated earlier ... data was missing ...

You give no link to your site nor page  :sad:

You can always get some info by view source of your page, and in html <!-- comments
there will be 'feed-back' from the prog OR you can noaarec.php?noaabug which will clear the cache file and start from scratch and show the now 'page' that is being scraped.
Title: Re: 'This day' Max / Min / Avg Temperatures (NOAA) - PHP Script
Post by: MackerelSky on December 07, 2019, 07:33:52 PM
As you stated earlier ... data was missing ...

You give no link to your site nor page  :sad:

You can always get some info by view source of your page, and in html <!-- comments
there will be 'feed-back' from the prog OR you can noaarec.php?noaabug which will clear the cache file and start from scratch and show the now 'page' that is being scraped.
Your script shows up on my home page about 1/2way down.

Not to get into an argument here, as it's totally not worth it, but the data was missing in the noaarecdat.txt file, not on the website from your URL in the script. It read 0° on both days. I believe, somehow, that a 0° reading is throwing an error, the -100° value in your script, instead of scraping the 0° as an actual figure.

And looking at the debug, https://mckeanweather.com/mwnoaarec.php?noaabug it's showing a 0° reading, but outputting -100
(https://mckeanweather.com/1207p5.PNG)

Like I said, not a big deal, but it's not picking up a Zero as a reading and throwing an error INMO.

Brad
Title: Re: 'This day' Max / Min / Avg Temperatures (NOAA) - PHP Script
Post by: the beteljuice on December 07, 2019, 08:18:38 PM
Quote
... but it's not picking up a Zero as a reading and throwing an error IMHO
YHO is correct  :shock:

My version 3 validation checks are incomplete  :oops: - regarding zero values as the same as null.

BTW I was looking for a banner / text link for your site, never thought to look at the little icon on the right left  :roll:.

I'll get to work on a new release ......
Title: Re: 'This day' Max / Min / Avg Temperatures (NOAA) - PHP Script
Post by: MackerelSky on December 07, 2019, 09:39:39 PM
Fixed the link on my banner, didn't realize it wasn't working.
I'm sure you'll work out the 0°/null thingy, no shot for a shooter like you.
Thx...

Brad
Title: Re: 'This day' Max / Min / Avg Temperatures (NOAA) - PHP Script
Post by: the beteljuice on December 07, 2019, 11:34:19 PM
Please try this, if it isn't too late ..
 [ You are not allowed to view attachments ]
Title: Re: 'This day' Max / Min / Avg Temperatures (NOAA) - PHP Script
Post by: MackerelSky on December 08, 2019, 03:32:10 AM
Please try this, if it isn't too late ..
 [ You are not allowed to view attachments ]

Yeah, I missed that chance, but I have it installed and the next day looking ahead that has a 0° for a thisDayMin is on the 16th of this month. Will let you know as soon as I can on that day!

Thx,
Brad
Title: Re: 'This day' Max / Min / Avg Temperatures (NOAA) - PHP Script
Post by: gwwilk on November 20, 2020, 08:25:50 AM
Unfortunately this script has bombed on my site overnight:  noaarec.php?noaabug=true (https://www.gwwilkins.org/noaarec.php?noaabug=true)

I've disabled this section of my alternate dashboard (https://www.gwwilkins.org) under 'Temperature', because Beteljuice's code is beyond me. :?
Title: Re: 'This day' Max / Min / Avg Temperatures (NOAA) - PHP Script
Post by: the beteljuice on November 20, 2020, 12:10:55 PM
Seems to be alright now ?
Title: Re: 'This day' Max / Min / Avg Temperatures (NOAA) - PHP Script
Post by: gwwilk on November 20, 2020, 12:34:14 PM
Indeed it does.  Now to back out of my changes to restore my dashboards 'Temperature' offerings.

Thanks for looking!  Much appreciated.
Title: Re: 'This day' Max / Min / Avg Temperatures (NOAA) - PHP Script
Post by: the beteljuice on November 20, 2020, 04:03:43 PM
*** IMPORTANT ***

I just ran my test bed and thought "That doesn't look right"

at the bottom of the report it says ..
Quote
The U.S. Naval Observatory (USNO) data is currently unavailable. The links provided are from other US Government sources. When USNO data is returned to service, the links will be updated.
Quite what that means, and how much impact it may have, and for how long ?  :roll:
Title: Re: 'This day' Max / Min / Avg Temperatures (NOAA) - PHP Script
Post by: Bunty on November 27, 2020, 08:47:49 PM
It doesn't work for me in debug mode.  What did I do wrong?

https://stillwaterweather.com/wx/noaarec.php?noaabug (https://stillwaterweather.com/wx/noaarec.php?noaabug)
Title: Re: 'This day' Max / Min / Avg Temperatures (NOAA) - PHP Script
Post by: the beteljuice on November 27, 2020, 09:23:32 PM
You didn't ....

If you view-source you'll see in the comments
Code: [Select]
<!-- RING NOAA -->
<!-- NOAA data FAIL -->
... and your last cache file is too old to be of any use.

Seems they are still all over the place ATM  :roll:
Title: Re: 'This day' Max / Min / Avg Temperatures (NOAA) - PHP Script
Post by: Bunty on December 25, 2020, 01:08:28 AM
Sorry, but I still don't understand what I did or didn't do:

https://stillwaterweather.com/wx/noaarec.php?noaabug (https://stillwaterweather.com/wx/noaarec.php?noaabug)
Title: Re: 'This day' Max / Min / Avg Temperatures (NOAA) - PHP Script
Post by: the beteljuice on December 25, 2020, 07:44:13 AM
Same reply as I gave before Bunty - you aren't doing anything 'wrong' - It's them having problems with their servers and either redirecting to a different source (in various formats) or returning nothing !

Read the line at the bottom ....
https://w2.weather.gov/climate/getclimate.php?pil=CLI&recent=yes&wfo=oun&sid=OKC (https://w2.weather.gov/climate/getclimate.php?pil=CLI&recent=yes&wfo=oun&sid=OKC)

Title: Re: 'This day' Max / Min / Avg Temperatures (NOAA) - PHP Script
Post by: the beteljuice on August 27, 2021, 06:20:16 PM
IMPORTANT UPDATE v4.0 27 Aug 2021

It seems that the urls have permanently changed and a few other little things  #-o

Many thanks to his worship Ken True for finding the answers and modifying the code.  \:D/

Script available from link in first post.
Title: Re: 'This day' Max / Min / Avg Temperatures (NOAA) - PHP Script
Post by: K3JAE on December 09, 2021, 04:15:01 PM
OK, I updated the script to 4.0.

Data being received - somewhat. Running DEBUG displays all items as intended.

URL to data: https://forecast.weather.gov/product.php?site=meg&issuedby=mkl&product=CLI&format=txt&version=1&glossary=0 (https://forecast.weather.gov/product.php?site=meg&issuedby=mkl&product=CLI&format=txt&version=1&glossary=0)

Results in (only posting area of interest):
THE JACKSON CLIMATE NORMALS FOR TODAY
                                             NORMAL    RECORD    YEAR
 MAXIMUM TEMPERATURE (F)    53              75        2007
 MINIMUM TEMPERATURE (F)     33                9       1995

The Average High/Lows are printing to the page perfectly fine... however the MAX/MIN RECORD with year is not.

Actual page code:
Code: [Select]
<tr>
                    <td align="center" valign="top" style="background-color: #C2F7B9; font-size:8pt; text-align: center; border: 1px solid red; border-radius:5px; -moz-border-radius:5px; -webkit-border-radius:5px; padding:2px;">
<span style="font-size:8pt"><?php langtrans('Date Record High<span style="font-weight: bold; color: red">*</span>'); ?></span><br />
<span style="font-size: 12pt; font-weight:bold"><?php echo strip_units($thisDayMax) . $uomTemp?></span><br/>
<span style="font-size:8pt"><?php echo $thisDayMaxYear?></span>
                    </td>
                    <td align="center" valign="top" style="background-color: #C2F7B9; font-size:8pt; text-align: center; border: 1px solid blue; border-radius:5px; -moz-border-radius:5px; -webkit-border-radius:5px; padding:2px;">
<span style="font-size:8pt"><?php langtrans('Date Record Low<span style="font-weight: bold; color: red">*</span>'); ?></span><br />
<span style="font-size: 12pt; font-weight:bold"><?php echo strip_units($thisDayMin) . $uomTemp?></span><br/>
<span style="font-size:8pt"><?php echo $thisDayMinYear?></span>
                    </td>
</tr>

I even went as far as commenting out my original code (as it is above) and copy/pasting the code from the DEBUG section of noaarec.php and dropped it in it's place - exact same result.

Guidance please?


**** UPDATE *****
Problem was resolved. I had commented out the include_once('noaarec.php');  in my dashboard page and forgot to re-enable it.  ](*,)  #-o  :roll: