Author Topic: WU-History now works with WU/TMC API...  (Read 13388 times)

0 Members and 1 Guest are viewing this topic.

Offline saratogaWX

  • Administrator
  • Forecaster
  • *****
  • Posts: 9257
  • Saratoga, CA, USA Weather - free PHP scripts
    • Saratoga-Weather.org
Re: WU-History now works with WU/TMC API...
« Reply #125 on: June 07, 2019, 06:04:45 PM »
I've also found an anomaly with the Daily query -- the date=YYYYMMDD seems to be responding to UTC date, not date-in-timezone-of-station for queries with the API.  This happens late afternoon for me in TZ=America/Los_Angeles after the UTC date switches to the new day.  I've sent off a query to Victoria (and filed a feedback to the official reporting site).

It means that the JSON contents of the cache files may be off-by-one due to this.  Unfortunately, the API docs do not say what variety of date is used (UTC or Timezone) and the old WU/API and the old CSV queries all used date-in-timezone-of-station.

I'll keep you posted on their response.  Meanwhile, if the old CSV query is working.. keep using it.  Use my WXDailyHistory.php as a backup (and maybe likely future).
« Last Edit: June 07, 2019, 06:52:28 PM by saratogaWX »
Ken True/Saratoga, CA, USA main site: saratoga-weather.org
Davis VP1+ FARS, Blitzortung RED, GRLevel3, WD, WL, VWS, Cumulus, Meteobridge
Free weather PHP scripts/website templates - update notifications on Twitter saratogaWXPHP

Offline saratogaWX

  • Administrator
  • Forecaster
  • *****
  • Posts: 9257
  • Saratoga, CA, USA Weather - free PHP scripts
    • Saratoga-Weather.org
Re: WU-History now works with WU/TMC API...
« Reply #126 on: June 07, 2019, 08:26:38 PM »
Ok.. more info on the bug.  Today (20190607 in both PDT and UTC) the url of
Code: [Select]
https://api.weather.com/v2/pws/history/all?stationId=KCASARAT1&format=json&units=e&date=20190607&apiKey=<my-api-key> returned data like
Code: [Select]
0:       "stationID": "KCASARAT1",
      "tz": "America/Los_Angeles",
      "obsTimeUtc": "2019-06-07T07:04:55Z",
      "obsTimeLocal": "2019-06-07 00:04:55",
      "epoch": 1559891095,
...

last:       "stationID": "KCASARAT1",
      "tz": "America/Los_Angeles",
      "obsTimeUtc": "2019-06-07T22:50:01Z",
      "obsTimeLocal": "2019-06-07 15:50:01",
      "epoch": 1559947801,

Same URL after 5pm PDT the rollover of UTC date to 20190608 returns
Code: [Select]
0:       "stationID": "KCASARAT1",
      "tz": "America/Los_Angeles",
      "obsTimeUtc": "2019-06-06T07:04:54Z",
      "obsTimeLocal": "2019-06-06 00:04:54",
      "epoch": 1559804694,
...
last:       "stationID": "KCASARAT1",
      "tz": "America/Los_Angeles",
      "obsTimeUtc": "2019-06-07T06:59:56Z",
      "obsTimeLocal": "2019-06-06 23:59:56",
      "epoch": 1559890796,
.. in other words, the COMPLETE data for 20190606 instead of the data for 20190607 after 5pm PDT.

Looks like a valid API bug to me...

Then trying with 20190608 (tomorrow for me) returns
Code: [Select]
0:       "stationID": "KCASARAT1",
      "tz": "America/Los_Angeles",
      "obsTimeUtc": "2019-06-07T07:04:55Z",
      "obsTimeLocal": "2019-06-07 00:04:55",
      "epoch": 1559891095,
...
208:      "stationID": "KCASARAT1",
      "tz": "America/Los_Angeles",
      "obsTimeUtc": "2019-06-08T00:24:54Z",
      "obsTimeLocal": "2019-06-07 17:24:54",
      "epoch": 1559953494,
which is the data I'd expect to continue seeing with a 20190607 query.  Definitely an API bug.


« Last Edit: June 07, 2019, 08:32:26 PM by saratogaWX »
Ken True/Saratoga, CA, USA main site: saratoga-weather.org
Davis VP1+ FARS, Blitzortung RED, GRLevel3, WD, WL, VWS, Cumulus, Meteobridge
Free weather PHP scripts/website templates - update notifications on Twitter saratogaWXPHP

Offline saratogaWX

  • Administrator
  • Forecaster
  • *****
  • Posts: 9257
  • Saratoga, CA, USA Weather - free PHP scripts
    • Saratoga-Weather.org
Re: WU-History now works with WU/TMC API...
« Reply #127 on: June 07, 2019, 09:26:52 PM »
I worked up a bypass for the API today's daily data issue and pushed V1.21 of WXDailyHistory.php to https://github.com/ktrue/WU-History

It adds DAY processing code to specifically address the above issue.  I know it works for timezones West of GMT .. I'm not sure of the effect on timezones that are East of GMT .. I hope someone in Europe or Oceania would kindly test it and let me know the effect there.

Thanks!
Ken True/Saratoga, CA, USA main site: saratoga-weather.org
Davis VP1+ FARS, Blitzortung RED, GRLevel3, WD, WL, VWS, Cumulus, Meteobridge
Free weather PHP scripts/website templates - update notifications on Twitter saratogaWXPHP

Offline gwwilk

  • Southeast Lincoln Weather
  • Forecaster
  • *****
  • Posts: 2578
    • SouthEast Lincoln, NE Weather
Re: WU-History now works with WU/TMC API...
« Reply #128 on: June 07, 2019, 11:39:04 PM »
It looks like you and WU are making some real progress, Ken!

The graphs are back--the initial creation was slow but after that they are quite snappily displayed.  The weekly, monthly, and yearly graphs are all there.  (The June graph doesn't display for lack of data since this is only the seventh, but the May graph displays.)

And the data is showing up promptly.

 [tup] =D&gt; \:D/ [tup]
Regards, Jerry Wilkins
gwwilk@gmail.com

Offline rlee171

  • Contributor
  • ***
  • Posts: 124
Re: WU-History now works with WU/TMC API...
« Reply #129 on: June 16, 2019, 05:44:16 PM »
The latest README on Ken's GitHub page regarding the recent WU history issues has some comments about "Help for users of Radomir Luza's wxwugraphs scripts". He also mentions "•please note that I did not write, nor directly support Radomir's wxwugraphs scripts." I can absolutely appreciate where he is coming from with multitude of other issues he helps us all with. I made all of the suggested changes and have been unable to have any success so far. I know this is an old script, however there are some of us that still use them. Personally I really like them and never had any issues until the WU fiasco. If I hadn't spent many many hours trying to fix them I would have abandoned them and went another direction. However, I found a couple of sites that have them up and running. That and my thick skull has me determined to continue on. The interesting thing (and what prompted this post) is that the ones that are working did not update to Ken's WXDailyHistory fix. Doing a view source on wxwugraphs I find the old station source as wunderground/weather/WXDailyHistory.asp? I assume these folks didn't respond to the fix for one reason or another and now that WU is back to their old format again, their sites are back up and running with no changes. I personally don't want to go back, I think it wiser to follow Ken's lead on this one and stay with his changes. I hope the site owner does not mind but here is an example of what I'm talking about, check the source code, mine is nearly a mirror image with the exception of "Station data source: <a href="http://www.wunderground.com/weatherstation/WXDailyHistory.asp?ID=KCTNEWMI12" and mine of "Station data source: <a href="www.hsnpar.com/WXDailyHistory.php?ID=KARHOTSP16". I believe this is what Ken was addressing in his README file.
http://www.newmilfordctweather.com/wxwugraphs.php and mine https://www.hsnpar.com/wxwugraphs.php
Do I not see the forest for the trees? If any one has had success with Ken's fix, please jump in!

Update - I have found something interesting that may be bringing me closer to finding what is happening. Looking back through this thread I noticed where Ken did ?debug=y after a view source and found an issue I had. (first time I had seen that, shows you where I am on the learning curve!) I first did a debug on wxwugraphs.php before the view source and it returned
Warning: Cannot modify header information - headers already sent by (output started at /home/ln0pka3d6few/public_html/common.php:352) in /home/ln0pka3d6few/public_html/top.php on line 43

Then I did another debug after a view-source and it showed the same error with additional information on line 143 thru 147 (using Firefox) also I noticed that on line 684 that it is attempting to use the new WXDailyHistory.php as my data source. I believe I am inching towards something, albeit caveman style!

Thanks
« Last Edit: June 17, 2019, 02:21:48 AM by rlee171 »
Davis VP2 
Blitzortung Stations: 1387  1445  2315
Cumulus
GRLevel3

Offline rlee171

  • Contributor
  • ***
  • Posts: 124
Re: WU-History now works with WU/TMC API...
« Reply #130 on: July 13, 2019, 11:14:42 PM »
Are we back to having issues with the wxwuhistory again? I have looked at a few of our fellow users sites and they seem to be having problems too. The odd thing is my WU History graphs are complete for every day, but the data is not. it has been erratic looking back now since the 3rd of July. That was the last full 24 hrs. of data. Since the 4th it is incomplete, it stops, in day order: 23:10, 12:20, 15:39, 22:10, 11:25, 14:34, 15:14, 15:50, 14:05 yesterday and none today. Maybe there is discussion some where else under a different topic, but I can't find it.





Davis VP2 
Blitzortung Stations: 1387  1445  2315
Cumulus
GRLevel3

Offline saratogaWX

  • Administrator
  • Forecaster
  • *****
  • Posts: 9257
  • Saratoga, CA, USA Weather - free PHP scripts
    • Saratoga-Weather.org
Re: WU-History now works with WU/TMC API...
« Reply #131 on: July 17, 2019, 07:27:32 PM »
I've just released WXDailyHistory.php V1.22 - 12-Jul-2019.  The WU API was returning bad obsTimeLocal values for any date from June, 2018 or before.  The epoch date was being returned in milliseconds (13 digits) instead of the expected 10-digit seconds.

https://github.com/ktrue/WU-History -- only WXDailyHistory.php (and the README.md) were changed.  No need to reload your cache files for June, 2018 and before.. the script handles the date by processing the epoch date (after pruning if need be) to generate the local date based on the tz entry in the returned JSON.

This may also fix some strange errors for Radomir's wugraphs in month and year mode.

My thanks to Holger of http://dl5ark.heliohost.org for spotting the issue leading to the fix.

Best regards,
Ken
Ken True/Saratoga, CA, USA main site: saratoga-weather.org
Davis VP1+ FARS, Blitzortung RED, GRLevel3, WD, WL, VWS, Cumulus, Meteobridge
Free weather PHP scripts/website templates - update notifications on Twitter saratogaWXPHP

Offline saratogaWX

  • Administrator
  • Forecaster
  • *****
  • Posts: 9257
  • Saratoga, CA, USA Weather - free PHP scripts
    • Saratoga-Weather.org
Re: WU-History now works with WU/TMC API...
« Reply #132 on: July 18, 2019, 06:31:21 PM »
Ahh.. one more major change to Version 1.30 - 18-Jul-2019.  This set of changes now fixes the display issues for Year and Month in Radomir's wxwugraphs.  Only WXDailyHistory.php (and the README.md) were changed.

If you're using Radomir's wxwugraphs scripts, you'll need to clean out your wxwugraphs/cache/ files to get good CSV files stored by WXDailyHistory.php returns.  See the README.md on the distribution for details.  Thanks again to Holger for testing this out.
Ken True/Saratoga, CA, USA main site: saratoga-weather.org
Davis VP1+ FARS, Blitzortung RED, GRLevel3, WD, WL, VWS, Cumulus, Meteobridge
Free weather PHP scripts/website templates - update notifications on Twitter saratogaWXPHP

Offline rlee171

  • Contributor
  • ***
  • Posts: 124
Re: WU-History now works with WU/TMC API...
« Reply #133 on: July 24, 2019, 02:44:19 PM »
Ken thanks for continuing to help us few die-hards who are trying to continue using Radomirs's scripts.

In the Read.me there is the following: 
Quote

    Delete ./wxwugraphs/cache/ entries with station data (since they have the bad formats that are causing the problems). Delete files/directories:
        {WUID}-*.txt and all the
        YYYY-DD directories

Radomir's scripts should fetch/cache what's needed for proper month/year displays by calling WXDailyHistory.php using your existing JSON caches and save the resulting CSV files in the wxwugraphs cache directory. You may have to call the wugraphs.php/wxwugraphs.php page with ?force=1 to get it to regenerate the local cache files needed.
  I emptied the wxwugraphs cache and did the ?force=1 and it only partially populated the wxwugraphs/cache. Any ideas?



« Last Edit: July 24, 2019, 02:49:07 PM by rlee171 »
Davis VP2 
Blitzortung Stations: 1387  1445  2315
Cumulus
GRLevel3

Offline rlee171

  • Contributor
  • ***
  • Posts: 124
Re: WU-History now works with WU/TMC API...
« Reply #134 on: July 24, 2019, 03:18:18 PM »
I have just noticed that we may be fighting a loosing battle. When viewing Radomir's graph it stops at approx 11:09, the same time WU's tabular data stopped, almost three hours ago. Their data stops at random times nearly every day, there have only been 3 or 4 complete 24 hours of tabular data this month. The crazy thing is WU's graph's are complete every day. Why the graphs continue but the data stops is baffeling to me. Also, if we are depending on the tabular data, we are trying to feed oats to a dead horse! Notice the times in the attached pics, they were all snipped within minutes of each other.



Davis VP2 
Blitzortung Stations: 1387  1445  2315
Cumulus
GRLevel3

Offline 92merc

  • BismarckWeather.net
  • Forecaster
  • *****
  • Posts: 1308
  • BismarckWeather.net
    • BismarckWeather.net
Re: WU-History now works with WU/TMC API...
« Reply #135 on: July 24, 2019, 03:52:39 PM »
It may be unrelated, but when I used WeatherDisplay's tool just a couple of weeks ago to import a bunch of my old WU data, there were holes in it too.  I was missing a day or two sometimes.  Whole half of a month another time.
https://www.BismarckWeather.net
Davis VP2, Cumulus, WeatherDisplay, Blitzortung, Saratoga Scripts, NOAA Stream via PI

Offline rlee171

  • Contributor
  • ***
  • Posts: 124
Re: WU-History now works with WU/TMC API...
« Reply #136 on: July 24, 2019, 04:29:20 PM »
I'm relatively new to the hobby, put a Davis up and joined the forum in 2013. I guess I was just too complacent tagging along, now I can truly understand the frustration and outrage that most folks are feeling towards WU.
Davis VP2 
Blitzortung Stations: 1387  1445  2315
Cumulus
GRLevel3

Offline saratogaWX

  • Administrator
  • Forecaster
  • *****
  • Posts: 9257
  • Saratoga, CA, USA Weather - free PHP scripts
    • Saratoga-Weather.org
Re: WU-History now works with WU/TMC API...
« Reply #137 on: July 24, 2019, 04:41:50 PM »
Ken thanks for continuing to help us few die-hards who are trying to continue using Radomirs's scripts.

In the Read.me there is the following: 
Quote

    Delete ./wxwugraphs/cache/ entries with station data (since they have the bad formats that are causing the problems). Delete files/directories:
        {WUID}-*.txt and all the
        YYYY-DD directories

Radomir's scripts should fetch/cache what's needed for proper month/year displays by calling WXDailyHistory.php using your existing JSON caches and save the resulting CSV files in the wxwugraphs cache directory. You may have to call the wugraphs.php/wxwugraphs.php page with ?force=1 to get it to regenerate the local cache files needed.
  I emptied the wxwugraphs cache and did the ?force=1 and it only partially populated the wxwugraphs/cache. Any ideas?

With the WXDisplayHistory.php and the Radomir wxwugraphs, there are two separate cache file sets to worry about.

First, the WXDisplayHistory.php will cache the returned JSON from WU API queries in the ./cache/ directory (normally).  It's important for you to follow the README.md instructions to pre-load all your prior months data back to 2008 (or whenever your station started submitting data to WU) whichever is LATER.. WU API has no data prior to 2008 on the API.   Once those cache files for each month are successfully cached, then queries to WXDailyHistory.php for prior months/years will return the CSV data promptly (without accessing WU API) from the cached JSON responses.  If you've already done the pre-load with an older version of WXDailyHistory.php, then you don't need to redo the pre-load -- the stored JSON for WXDailyHistory.php in ./cache/ is ok and doesn't need to be updated.

Second: Radomir's wxwugraphs has its own caching strategy/algorithm for calling WXDailyHistory to get the CSV files needed and processing them.  If you had files in the wxwugraphs/cache/ directory that were created by calling an older version of WXDailyHistory.php, then those are the files/directories that need to be deleted so wxwugraphs can get updated copies cached to make the graphs.

Whew.. complicated yes, but doable.

And, yes, sometimes the JSON from the WU API is missing data.. I've seen it on my site (July 17 was missing chunks).  The WXDailyHistory.php just decodes the returned JSON into a CSV format .. it doesn't check for missing or erroneous data.
« Last Edit: July 24, 2019, 04:44:50 PM by saratogaWX »
Ken True/Saratoga, CA, USA main site: saratoga-weather.org
Davis VP1+ FARS, Blitzortung RED, GRLevel3, WD, WL, VWS, Cumulus, Meteobridge
Free weather PHP scripts/website templates - update notifications on Twitter saratogaWXPHP

Offline rlee171

  • Contributor
  • ***
  • Posts: 124
Re: WU-History now works with WU/TMC API...
« Reply #138 on: July 24, 2019, 05:42:15 PM »
Thanks Ken...   what you have done works great. There is nothing we can do about missing or incomplete data which unfortunately is a regular occurrence these days. When all of this started a couple of months ago, I started using the graphs generated by Cumulus. They are not as functional or pleasing to the eye, but always complete!! There is probably a way to start from scratch using Highcharts, but it is far beyond my capabilities. Thanks again for your extra effort making Radomir's graphs functional including the monthly and yearly data, but unless WU starts providing complete data in the future (not holding my breath) incomplete, inaccurate graphs is just too frustrating.

Davis VP2 
Blitzortung Stations: 1387  1445  2315
Cumulus
GRLevel3

Offline rlee171

  • Contributor
  • ***
  • Posts: 124
Re: WU-History now works with WU/TMC API...
« Reply #139 on: July 24, 2019, 11:51:42 PM »
I think I have been starring at the monitor too long today or my mind is playing tricks on me. I guess someone needs to talk to me like a 9 year old. Is the data that is pulled from wu using the api not the same as the data displayed going directly to the dashboard? The dashboard shows a complete day of data (so far today) yet my root cache file for today is less than half the normal size and going to the wxwuhistory I am getting the all to familiar "no data available for this period". I am lost as a goose. I updated the WXDailyHistory today, cleared the wxwugraphs (that is re-populating now). I have already done the pre-load from the previous WXDailyHistory update. What am I missing here?

 
Davis VP2 
Blitzortung Stations: 1387  1445  2315
Cumulus
GRLevel3

Offline saratogaWX

  • Administrator
  • Forecaster
  • *****
  • Posts: 9257
  • Saratoga, CA, USA Weather - free PHP scripts
    • Saratoga-Weather.org
Re: WU-History now works with WU/TMC API...
« Reply #140 on: July 25, 2019, 12:58:26 PM »
I think the issue is related to the YYYYMMDD date in the queries to the API.  I filed a question with WU support about it, but no answer yet.

The API docs just say 'date' but don't specify whether that date is UTC or Local Timezone YYMMDD -- I found that when UTC date is different from Local timezone date, the problem about 'missing' data exists.  I suspect that the query may have to be for UTC date in order to reliably pull data.  Grrrr.
Ken True/Saratoga, CA, USA main site: saratoga-weather.org
Davis VP1+ FARS, Blitzortung RED, GRLevel3, WD, WL, VWS, Cumulus, Meteobridge
Free weather PHP scripts/website templates - update notifications on Twitter saratogaWXPHP

Offline saratogaWX

  • Administrator
  • Forecaster
  • *****
  • Posts: 9257
  • Saratoga, CA, USA Weather - free PHP scripts
    • Saratoga-Weather.org
Re: WU-History now works with WU/TMC API...
« Reply #141 on: July 26, 2019, 11:53:07 AM »
I discovered (by watching XHR requests to a WU PWS page) that the magic of decimal values is enabled by a (currently undocumented) addition of &numericPrecision=decimal the the WU/TWC API URL -- it causes the values to be returned with decimal numbers.

So..I've updated the WXDailyHistory.php to V1.31 - 26-Jul-2019 at https://github.com/ktrue/WU-History with that change along with some additional cURL headers trying to have Akamai not cache the WU/TWC API returns so much.

After you install the update, use the 'After Initial Installation' instructions in the README.md file to reload your monthly cache files with decimal precision in the JSON.

ALSO.. July looks like a 'bad month' for API returns so far.. even though my station continues to send data without pause, large chunks of July, 2019 data are missing from the WU PWS monthly table page (and, of course from the WXDailyHistory.php request since it uses the same API source).
Ken True/Saratoga, CA, USA main site: saratoga-weather.org
Davis VP1+ FARS, Blitzortung RED, GRLevel3, WD, WL, VWS, Cumulus, Meteobridge
Free weather PHP scripts/website templates - update notifications on Twitter saratogaWXPHP

Offline wvdkuil

  • Wim van der kuil
  • Forecaster
  • *****
  • Posts: 1986
    • My PWS at Leuven Belgium Europe
Re: WU-History now works with WU/TMC API...
« Reply #142 on: July 29, 2019, 11:06:05 AM »
I discovered (by watching XHR requests to a WU PWS page) that the magic of decimal values is enabled by a (currently undocumented) addition of &numericPrecision=decimal the the WU/TWC API URL -- it causes the values to be returned with decimal numbers.

. . . shortened . . .

Thank you for this briljant detective work,  this really helps the WU-only-websites to get a decent dashboard.  =D&gt;
I added it to a test-site and it looks very nice => screenshot attached

On another WU-problem:
The min-max values for today which are only updated once /day : 
I replaced the today summary which never updates because of "cache-control: max-age=86392"
https://api.weather.com/v2/pws/history/daily?stationId=KCAOAKLA44&format=json&units=m&date=20181001&apiKey=yourApiKey
with the 7 day summary as it has a far shorter "cache-control: max-age=854" in the response.
https://api.weather.com/v2/pws/dailysummary/7day?stationId=KMAHANOV10&format=json&units=e&apiKey=yourApiKey

Wim
« Last Edit: July 29, 2019, 11:08:13 AM by wvdkuil »

Offline saratogaWX

  • Administrator
  • Forecaster
  • *****
  • Posts: 9257
  • Saratoga, CA, USA Weather - free PHP scripts
    • Saratoga-Weather.org
Re: WU-History now works with WU/TMC API...
« Reply #143 on: July 29, 2019, 11:19:38 AM »
Hi Wim,
I, too, noticed the extraordinarily long "cache-control: max-age=nnnnn" returns from API results that should only have a max-age of 300 seconds or so.

I sent a message to Victoria about it.  She also suggested I use the official reporting page to put the issue in their system.  Using Google Chrome, I tried several times to send the feedback having selected "Weather Underground" and "Weather API Key", filled in the form with my key.  The 'Send your email' button returns with transient display at the page top of 'validation failed: weather_underground is invalid'.  Sigh..  The Weather Channel selector does not offer an API Key setting.  So I reported that problem to Victoria too.

My thought is the Weather.com folks responsible for setup/configuring the Akamai cache for api.weather.com have screwed up the configuration so Akamai is not refreshing their cache in a timely manner for what should be near-realtime data.  I can understand long cache expire times for historical data (yesterday and before), but NOT for today's data .. that should be un- or minimally cached so it maintains 'freshness'.

I added additional headers into the cURL request of
Code: [Select]
  $reqHeader = array (
  "Cache-Control: max-age=0,no-cache,must-revalidate",
"Pragma: no-cache",
"Accept-Encoding: gzip",
    "Accept: text/html,application/json;q=0.9,*/*;q=0.8"
  );
and
Code: [Select]
  curl_setopt($ch, CURLOPT_ENCODING, 'gzip,deflate');          // decode compressed returns
, but the Akamai cache refuses to honor those requests for no caching.

Sigh.  When the official report form is fixed, I'll use it to report it officially.

« Last Edit: July 29, 2019, 12:18:32 PM by saratogaWX »
Ken True/Saratoga, CA, USA main site: saratoga-weather.org
Davis VP1+ FARS, Blitzortung RED, GRLevel3, WD, WL, VWS, Cumulus, Meteobridge
Free weather PHP scripts/website templates - update notifications on Twitter saratogaWXPHP

Offline dasman

  • Forecaster
  • *****
  • Posts: 491
    • Wx Peotone
Re: WU-History now works with WU/TMC API...
« Reply #144 on: September 07, 2019, 08:21:19 PM »
Ok, I'm late to the party. I just updated the WXDailyHistory script and I am now missing daily data. I have graphs and weekly and monthly data.

http://www.peotoneweather.com/wu.php

EDIT... It appears it is only the daily data for the current month that is missing
« Last Edit: September 07, 2019, 08:26:22 PM by dasman »
Dave Sommerfed
Peotone Illinois USA
CW7762, KILPEOT1
NWS COOP, CoCoRaHS, Spotter Network

Offline saratogaWX

  • Administrator
  • Forecaster
  • *****
  • Posts: 9257
  • Saratoga, CA, USA Weather - free PHP scripts
    • Saratoga-Weather.org
Re: WU-History now works with WU/TMC API...
« Reply #145 on: September 07, 2019, 08:41:25 PM »
I fear it's a WU API issue.. I'm still getting incomplete data on my raw-JSON queries too.  The script just formats what is returned from the API call.
Ken True/Saratoga, CA, USA main site: saratoga-weather.org
Davis VP1+ FARS, Blitzortung RED, GRLevel3, WD, WL, VWS, Cumulus, Meteobridge
Free weather PHP scripts/website templates - update notifications on Twitter saratogaWXPHP

Offline dasman

  • Forecaster
  • *****
  • Posts: 491
    • Wx Peotone
Re: WU-History now works with WU/TMC API...
« Reply #146 on: September 07, 2019, 09:04:11 PM »
The strange thing is I had the data immediately before the update. since it was a matter of minutes I figured I might have done something wrong.
Dave Sommerfed
Peotone Illinois USA
CW7762, KILPEOT1
NWS COOP, CoCoRaHS, Spotter Network

Offline AWL

  • Forecaster
  • *****
  • Posts: 727
    • Ardmore Weather Live
Re: WU-History now works with WU/TMC API...
« Reply #147 on: September 08, 2019, 03:16:02 PM »
Finally got around to updating this script. Great work for sure! Have one problem. Today's data only updates on initial load and that's it. Stuck at 11:05 today. All past data loads correctly. Thoughts?
https://www.ardmoreweatherlive.com/wugraphs.php

* This morning first view at 5:45.
*After 8pm is does not load either. Have a time zone issue apparently.

Thanks Doug
« Last Edit: September 09, 2019, 09:08:02 AM by AWL »

Offline saratogaWX

  • Administrator
  • Forecaster
  • *****
  • Posts: 9257
  • Saratoga, CA, USA Weather - free PHP scripts
    • Saratoga-Weather.org
Re: WU-History now works with WU/TMC API...
« Reply #148 on: September 09, 2019, 12:35:21 PM »
It's a WU/Akamai caching issue, I think.

The response to a daily query (for today) returns with a header of
Code: [Select]
cache-control: max-age=84692 and despite being requested with all the cache busting headers, only returns the cached version which seems to be fetched the first time it's accessed on a date.

While that response (cached) should be fine for PAST data (like yesterday and before), it certainly isn't good for TODAYs data.  It's not something I can fix -- all the request headers to not cache the result are being ignored for today's daily data.  Sigh.
Ken True/Saratoga, CA, USA main site: saratoga-weather.org
Davis VP1+ FARS, Blitzortung RED, GRLevel3, WD, WL, VWS, Cumulus, Meteobridge
Free weather PHP scripts/website templates - update notifications on Twitter saratogaWXPHP

Offline dasman

  • Forecaster
  • *****
  • Posts: 491
    • Wx Peotone
Re: WU-History now works with WU/TMC API...
« Reply #149 on: September 09, 2019, 01:32:19 PM »
Ken my daily data for September is still screwed up. Some of my days the data ends in the morning?
Dave Sommerfed
Peotone Illinois USA
CW7762, KILPEOT1
NWS COOP, CoCoRaHS, Spotter Network

 

anything