Author Topic: 'This day' Max / Min / Avg Temperatures (NOAA) - PHP Script  (Read 3533 times)

0 Members and 1 Guest are viewing this topic.

Online the beteljuice

  • the beteljuice
  • Senior Contributor
  • ****
  • Posts: 244
    • test site
Re: 'This day' Max / Min / Avg Temperatures (NOAA) - PHP Script
« Reply #25 on: June 27, 2019, 06:18:36 PM »
Oh yes it is ......
Imagine what you will KNOW tomorrow !

Offline ed2kayak

  • Ed
  • Forecaster
  • *****
  • Posts: 600
  • Davis Vantage Pro2 w/12 hr FARS, solar/uv
    • Cumberland Valley Weather
Re: 'This day' Max / Min / Avg Temperatures (NOAA) - PHP Script
« Reply #26 on: June 27, 2019, 06:20:47 PM »
Yes, you are correct. I'm must have been too impatient.  =D>
Ed
CoCoRaHS PA-CD-6,  Weather Underground KPAMECHA7, CWOP DW5425

Online the beteljuice

  • the beteljuice
  • Senior Contributor
  • ****
  • Posts: 244
    • test site
Re: 'This day' Max / Min / Avg Temperatures (NOAA) - PHP Script
« Reply #27 on: June 27, 2019, 06:38:15 PM »
Note to all:

If you run noaarec.php?noaabug it will delete the existing cache-file and force a new data 'grab'.

(A bit of a two edged sword, I might put a 'flag' on it to 'see' cache regardless of age ....)
Imagine what you will KNOW tomorrow !

Offline 92merc

  • BismarckWeather.net
  • Forecaster
  • *****
  • Posts: 928
  • BismarckWeather.net
    • BismarckWeather.net
Re: 'This day' Max / Min / Avg Temperatures (NOAA) - PHP Script
« Reply #28 on: June 27, 2019, 06:41:34 PM »
Huzzah!  Looks like that did the trick.  I'm pulling data.
https://www.BismarckWeather.net
Davis VP2, Cumulus, WeatherDisplay, Blitzortung, Bloomsky, Saratoga Scripts, NOAA Stream via PI

Offline thebigboi

  • Member
  • *
  • Posts: 22
    • Murfreesboro Weather Station
Re: 'This day' Max / Min / Avg Temperatures (NOAA) - PHP Script
« Reply #29 on: June 27, 2019, 08:30:03 PM »
The Beteljuice, I have watched the NOAA site where the script pulls its information and the site looks like it refreshes several times a day. The early morning refreshes, up to around 7:00 a.m. include maximum and minimum temperature records whereas every refresh after that until approximately 6:00 p.m. does not include the maximum temperature records. Therefore, if I access the site between approximately 7:00 a.m. and 6:00 p.m. I get an error because apparently every time the website is accessed it pulls new info from the NOAA site.  I don't know if there is a work around but it looks like several sites have this same anomaly.

Online the beteljuice

  • the beteljuice
  • Senior Contributor
  • ****
  • Posts: 244
    • test site
Re: 'This day' Max / Min / Avg Temperatures (NOAA) - PHP Script
« Reply #30 on: June 27, 2019, 09:27:49 PM »
The site(s) seem to typically update twice in 24hrs.
If there are subsequent updates because of incomplete data I don't know ?

Your cache-file is set for an 'age' of 1 hour (unless you request after local midnight upto 1 hour)

Are you sure the data is missing for your requested location ?
Quote
.... does not include the maximum temperature records.
Only the Max ? - very odd ...


The NOAA page is a little confusing because they are actually using English correctly  :shock:

You have to look at the date the report was created, together with the date the report is for ....

Generally speaking if you see "TODAY" (rather than Yesterday) in the top quarter of the report, those are the figures used, else the figures towards the bottom of the report which maybe labeled TODAY - OR - TOMORROW will be used.

I don't know if one of your locations is a part-time reporter, or why that would cause an omission of 'record' data ?

NOAA do say that the page is unconfirmed data awaiting ratification for publication elsewhere.
... or perhaps they are just having a bad few days, just because we are watching them !!!


Storing "Todays" and "Tomorrows" values (In the time window they both appear) in case of a failure tomorrow, together with date / time check would require a complete change of philosophy and effectively a new script.

Q. Have any 'missing' tomorrow values been discovered ?

If anyone else is having the same problem please report it. - some people may have to change their locale or not use the prog.
« Last Edit: June 27, 2019, 09:34:08 PM by the beteljuice »
Imagine what you will KNOW tomorrow !

Offline thebigboi

  • Member
  • *
  • Posts: 22
    • Murfreesboro Weather Station
Re: 'This day' Max / Min / Avg Temperatures (NOAA) - PHP Script
« Reply #31 on: June 28, 2019, 08:56:08 AM »
The Beteljuice, I will continue to monitor. My thought was to only have it update in the early morning hours and stay that way for the entire day since historic highs and lows don't change as the day progresses. This morning the NOAA site updated at 1:41 a.m. with all the "today" data being available. It did the same yesterday. However, later and I don't have the exact time it updated and only the lows were available with no reference to the highs. Is there a way to keep the cache that was fetched at certain hours for the balance of the day?

Offline 92merc

  • BismarckWeather.net
  • Forecaster
  • *****
  • Posts: 928
  • BismarckWeather.net
    • BismarckWeather.net
Re: 'This day' Max / Min / Avg Temperatures (NOAA) - PHP Script
« Reply #32 on: June 28, 2019, 09:05:46 AM »
I noticed mine hasn't updated yet.  It's 8:05 am local time.

noaarecdat was previously modified: Friday 28th of June 2019 12:07:23 PM UTC

Local Server NOW: Friday 28th of June 2019 01:04:35 PM UTC UTC +00:00

$wfo = bis
$sid = BIS

...THE BISMARCK ND CLIMATE SUMMARY FOR JUNE 27 2019...
https://www.BismarckWeather.net
Davis VP2, Cumulus, WeatherDisplay, Blitzortung, Bloomsky, Saratoga Scripts, NOAA Stream via PI

Offline thebigboi

  • Member
  • *
  • Posts: 22
    • Murfreesboro Weather Station
Re: 'This day' Max / Min / Avg Temperatures (NOAA) - PHP Script
« Reply #33 on: June 28, 2019, 11:03:46 AM »
The Betlejuice, you indicated the cache was held for one hour in noaarec.php which it appears this line "$check_it = time() - (60 * 60); // 1hr ago - should be able to increase to *6 or *12 if wanted" is what sets the cache reset. How would I increase the cache hold to eight hours? Would that simply mean modifying (60 * 60) with (480 * 480) or is it actual hours such as (8 * 8)? I just figured I would try that since the NOAA data changes during the day.

Online the beteljuice

  • the beteljuice
  • Senior Contributor
  • ****
  • Posts: 244
    • test site
Re: 'This day' Max / Min / Avg Temperatures (NOAA) - PHP Script
« Reply #34 on: June 28, 2019, 07:06:19 PM »
@92merc AND @thebigboi
Quote
noaarecdat was previously modified: Friday 28th of June 2019 12:07:23 PM UTC

Local Server NOW: Friday 28th of June 2019 01:04:35 PM UTC UTC +00:00
Something has left server time in UTC.
As stated in prog and thread, try either moving 'include' higher up in your page OR set your TZ in the prog.

@thebigboi

Code: [Select]
$check_it = time() - (60 * 60); // 1hr ago - should be able to increase to *6 or *12 if wanted

// would become

$check_it = time() - (60 * 60 * 8); // 8hr ago - should be able to increase to *6 or *12 if wanted

... but that is not going to fix your problem, just 'offset' it.
Have you noticed yet if 'Tomorrow' data is always complete ?
« Last Edit: June 28, 2019, 07:25:24 PM by the beteljuice »
Imagine what you will KNOW tomorrow !

Offline thebigboi

  • Member
  • *
  • Posts: 22
    • Murfreesboro Weather Station
Re: 'This day' Max / Min / Avg Temperatures (NOAA) - PHP Script
« Reply #35 on: June 28, 2019, 07:26:09 PM »
The Betlejuice, I am still watching the NOAA output. So far today every update includes the tomorrow language. The last update was at 4:33 p.m. and included the following:
THE NASHVILLE CLIMATE NORMALS FOR TOMORROW
                         NORMAL    RECORD    YEAR
 MAXIMUM TEMPERATURE (F)   89       109      2012
 MINIMUM TEMPERATURE (F)   68        54      1974

It also included the information for today. So far it appears every update after around 8:00 a.m. to the afternoon update does not include full today data. It only includes the info for the minimums and not the maximums. I will keep checking and see what happens.
Thanks for being so patient.

Offline thebigboi

  • Member
  • *
  • Posts: 22
    • Murfreesboro Weather Station
Re: 'This day' Max / Min / Avg Temperatures (NOAA) - PHP Script
« Reply #36 on: June 28, 2019, 07:50:41 PM »
The Betlejuice, in looking at the NOAA site, I did notice that the site has a NOWDATA category that includes a daily almanac that includes the information you scraped from the Observed Weather data but only changes once a day. First, I have no idea how you capture the information and second, I have no idea if there is a way to access this information via a php script. It was just an observation while perusing the NOAA site. Thanks.
« Last Edit: June 28, 2019, 08:10:34 PM by thebigboi »

Online the beteljuice

  • the beteljuice
  • Senior Contributor
  • ****
  • Posts: 244
    • test site
Re: 'This day' Max / Min / Avg Temperatures (NOAA) - PHP Script
« Reply #37 on: June 28, 2019, 08:29:06 PM »
@thebigboi ...

... you still haven't got your TZ right !!!!

As an aside: I think NOAA reports are being done for 00:00 UTC and 12:00 UTC with an hour or so for them to to collate and 'publish'.
Imagine what you will KNOW tomorrow !

Offline thebigboi

  • Member
  • *
  • Posts: 22
    • Murfreesboro Weather Station
Re: 'This day' Max / Min / Avg Temperatures (NOAA) - PHP Script
« Reply #38 on: June 28, 2019, 08:59:21 PM »
I moved the "include" script line to the top of ajax-dashboard6. How do I check it myself? How do I set the tz in the program?
« Last Edit: June 28, 2019, 09:01:25 PM by thebigboi »

Online the beteljuice

  • the beteljuice
  • Senior Contributor
  • ****
  • Posts: 244
    • test site
Re: 'This day' Max / Min / Avg Temperatures (NOAA) - PHP Script
« Reply #39 on: June 28, 2019, 09:45:15 PM »
@thebigboi

using noaarec.php?noaabug gives you the current local server time - including TZ
Quote
Running in DeBug Mode
noaarecdat was previously modified: Saturday 29th of June 2019 01:35:01 AM UTC

Local Server NOW: Saturday 29th of June 2019 01:43:01 AM UTC UTC +00:00

In the script approx. line #20 > 23...
Code: [Select]
// there has been one case where a previous 'include' changed TZ to UTC
// If this happens 'include' noaarec.php higher uo the page OR uncomment and set TZ BELOW
// date_default_timezone_set("America/??????????");
Imagine what you will KNOW tomorrow !

Offline thebigboi

  • Member
  • *
  • Posts: 22
    • Murfreesboro Weather Station
Re: 'This day' Max / Min / Avg Temperatures (NOAA) - PHP Script
« Reply #40 on: June 29, 2019, 07:44:25 PM »
I did get the tz issue resolved. So far, it appears the NOAA site updates three times a day at around 2:00 a.m. or thereabouts, at around 8:00 a.m. or thereabouts, and at around 5:00 p.m. or thereabouts. The only update that does not include the maximum record temperature is the 8:00 a.m. update. Oddly, the tomorrow records for both max and min temperature records are included in all updates. I will continue to monitor.

Online the beteljuice

  • the beteljuice
  • Senior Contributor
  • ****
  • Posts: 244
    • test site
Re: 'This day' Max / Min / Avg Temperatures (NOAA) - PHP Script
« Reply #41 on: June 29, 2019, 07:47:26 PM »
@thebigboi ...

cross post whilst I was editing this ....

I haven't forgotten your plight !!!

I see you've finally got your TZ set  =D>

... keeping an eye on your locale reports

quoted (not 'published') report times:

01:20 AM CDT = OK
08:07 AM CDT = FAIL all TODAY max values missing, and min 'so far' (null) with time (MM) - neither of which we use.
04:45 PM CDT = OK ('published' within 10 min)

Experimenting with a change of code / philosophy - It won't be foolproof, but should fix your problem as long as there is one 'visit' in a 'good' period in today and yesterday server time.

No promises, but if I get it working I'll PM you to test it  :shock:
Imagine what you will KNOW tomorrow !

Offline thebigboi

  • Member
  • *
  • Posts: 22
    • Murfreesboro Weather Station
Re: 'This day' Max / Min / Avg Temperatures (NOAA) - PHP Script
« Reply #42 on: June 29, 2019, 08:01:58 PM »
@The Betlejuice

Thanks so much. I really do appreciate your continued patience with my questions.

Online the beteljuice

  • the beteljuice
  • Senior Contributor
  • ****
  • Posts: 244
    • test site
Re: 'This day' Max / Min / Avg Temperatures (NOAA) - PHP Script
« Reply #43 on: June 29, 2019, 08:12:22 PM »
Quote from: thebigboi
...  continued patience with my questions.

It's the only way we all learn - Not a problem  #-o
Imagine what you will KNOW tomorrow !

Online the beteljuice

  • the beteljuice
  • Senior Contributor
  • ****
  • Posts: 244
    • test site
Re: 'This day' Max / Min / Avg Temperatures (NOAA) - PHP Script
« Reply #44 on: June 29, 2019, 09:08:21 PM »
... and now they've updated again @ 07:50 PM CDT, only a 3hr gap (all data OK)

What a crazy schedule !!
« Last Edit: June 29, 2019, 09:10:11 PM by the beteljuice »
Imagine what you will KNOW tomorrow !

Offline thebigboi

  • Member
  • *
  • Posts: 22
    • Murfreesboro Weather Station
Re: 'This day' Max / Min / Avg Temperatures (NOAA) - PHP Script
« Reply #45 on: June 29, 2019, 09:28:06 PM »
@The Betlejuice

Yep, I just spotted it also. The only update I've seen so far that does not include the maximum temperature happens each day around 8:00 a.m. Makes no sense at all.

Online the beteljuice

  • the beteljuice
  • Senior Contributor
  • ****
  • Posts: 244
    • test site
UPDATE: 'This day' Max / Min / Avg Temperatures (NOAA) - PHP Script
« Reply #46 on: July 06, 2019, 03:53:09 PM »
NEW ! - IMPROVED, Low Carbon, Eco friendly, nut free, self healing

OK Ver #3 now available ...

Currently being "field tested" and evaluated - available from link on first post or from here

Humungous amount of beteljuice logic to store "this" server day and "tomorrow" (when available) server day historic temp data from NOAA.

As well as identifying correct parts of report to 'scrape', it will also try to 'heal' any missing data / data lines.

This is a bit confusing because NOAA say "report for (date) ..." then refers relatively to the report created date (eg. yesterday, today, tomorrow).

In the case where local server day has rolled over, but report created date has not - the references are incorrect.
Further - there is at least one location where for eight hours or more each day there are two lines of data completely missing (and of course one of them is what we want)

All far too lengthy to explain fully in a meaningful way - but if you're interested see the real life example below of two consecutive reports, one of which has incomplete data, and how the program handles the situation ...



A report just after 00:00 local .. (but still being 'published' for 8 hours plus)

Local Server Time NOW:  Friday 5th of July 2019 07:24:52 AM CDT, America/Mexico_City - UTC -05:00

CLIMATE REPORT
NATIONAL WEATHER SERVICE NASHVILLE TN
12:10 AM CDT FRI JUL 5 2019

...

...THE NASHVILLE CLIMATE SUMMARY FOR JULY 4 2019...

[program decides: recDay: 4, serverDay: 5, Tomorrow: 6, Use second block]

...

[1st block]
WEATHER ITEM   OBSERVED TIME   RECORD YEAR NORMAL DEPARTURE
                VALUE   (LST)  VALUE       VALUE  FROM
                                                  NORMAL
...............................................................
TEMPERATURE (F)
 YESTERDAY [program decides: correct, day(4) not wanted]
  MAXIMUM         91   1214 PM 103    2012  89      2
  MINIMUM         71    302 AM  55    1968  69      2
  AVERAGE         81                        79      2
 
 ...
 
[2nd block]
THE NASHVILLE CLIMATE NORMALS FOR TODAY
[program decides: correct, day(5) already stored, but checked for any missing values]
                         NORMAL    RECORD    YEAR
 MAXIMUM TEMPERATURE (F)   89       104      2012
 MINIMUM TEMPERATURE (F)   69        54      1968
 
 [program decides: NO genuine Tomorrow day(6) Data Available, only Ring NOAA upto once per hour]
 
 
 
 Now a report 09:00 local that has problems ... (still being 'published' upto 17:00 local - 8hrs)
 
  Local Server Time NOW:  Friday 5th of July 2019 10:48:49 AM CDT, America/Mexico_City - UTC -05:00

CLIMATE REPORT
NATIONAL WEATHER SERVICE NASHVILLE TN
9:22 AM CDT FRI JUL 5 2019

...THE NASHVILLE CLIMATE SUMMARY FOR JULY 5 2019
VALID AS OF 0900 AM LOCAL TIME.

[program decides: recDay: 5, serverDay: 5, Tomorrow: 6, Use first block]

...

[1st block]
TEMPERATURE (F)
TODAY [program decides: correct, day(5) already stored, but checked for any missing values]
[ALL MAXIMUM TEMP DATA MISSING !!]
  MINIMUM         72    512 AM  54    1968  69      3
[ALL AVERAGE TEMP DATA MISSING !!] 
PRECIPITATION (IN)
  TODAY            0.00          3.33 1950   0.12  -0.12
[program decides: use stored day(5) in cache-file]
...

[2nd block]
THE NASHVILLE CLIMATE NORMALS FOR TOMORROW
[program decides: correct, day(6) store in cache-file]
                         NORMAL    RECORD    YEAR
 MAXIMUM TEMPERATURE (F)   89       105      2012
 MINIMUM TEMPERATURE (F)    69        54      1972

[program decides: Now we have complete data for day(5), day(6) stored in cache-file, - don't Ring NOAA until day(6) (when we look for day7 data)]

It's pretty much impossible to second guess all the possible NOAA responses to report errors, but hopefully in the majority of cases this prog allows for upto 48 hours of NOAA 'failure' as long as your site has a visitor in the morning and one at night  ;)

For those already using this prog noaarec.php is a direct compatible replacement.
noaarecdat..php on your server is now redundant and may be deleted.
New noaarecdat.txt will be created / maintained.
« Last Edit: July 07, 2019, 07:17:17 AM by the beteljuice »
Imagine what you will KNOW tomorrow !

Offline 92merc

  • BismarckWeather.net
  • Forecaster
  • *****
  • Posts: 928
  • BismarckWeather.net
    • BismarckWeather.net
Re: 'This day' Max / Min / Avg Temperatures (NOAA) - PHP Script
« Reply #47 on: July 06, 2019, 09:21:38 PM »
Looks good.  I have mine up and running.  Only different I noticed is there is a TXT file instead of a PHP file for the cache.  Hey, what ever works!

Thanks for your work.
https://www.BismarckWeather.net
Davis VP2, Cumulus, WeatherDisplay, Blitzortung, Bloomsky, Saratoga Scripts, NOAA Stream via PI

Offline thebigboi

  • Member
  • *
  • Posts: 22
    • Murfreesboro Weather Station
Re: 'This day' Max / Min / Avg Temperatures (NOAA) - PHP Script
« Reply #48 on: July 07, 2019, 07:17:21 AM »
Very good work. My site www.murfreesboroweather.com is working great. Thanks to The Betlejuice.

Offline mikeym2m

  • Contributor
  • ***
  • Posts: 133
    • MikeyM's Thornhurst PA Weather
Re: 'This day' Max / Min / Avg Temperatures (NOAA) - PHP Script
« Reply #49 on: July 07, 2019, 10:05:06 AM »
Also working well here www.mikeymsweather.com

Thanks to Bettlejuice   [tup]

Cheers

  :-)

MikeyM