Author Topic: WU-History now works with WU/TMC API...  (Read 5173 times)

0 Members and 1 Guest are viewing this topic.

Offline saratogaWX

  • Administrator
  • Forecaster
  • *****
  • Posts: 7418
  • Saratoga, CA, USA Weather - free PHP scripts
    • Saratoga-Weather.org
WU-History now works with WU/TMC API...
« on: May 23, 2019, 04:52:54 PM »
Whew.. I've written a shim script to replace the now-missing WXDailyHistory.asp page on the WU website with a WXDailyHistory.php script on your website that returns the same formatted CSV file for use with Jim McMurry's (jcweather.us) script set.  Only two files change if you have a working installation:

WU-History-inc.php  - V3.4d - 23-May-2019 and
WXDailyHistory.php  -  V1.00 - 23-May-2019  (added file .. initial release).

The WXDailyHistory.php file uses the WU/TWC API to api.weather.com to retrieve (and cache monthly data), then format to the CSV file expected by WU-History-inc.php.

Released on GitHub at https://github.com/ktrue/WU-History distribution .. please read the README.md for more info.

Best regards,
Ken
Ken True/Saratoga, CA, USA main site: saratoga-weather.org
Davis VP1+ FARS, Boltek-PCI/NexStorm, microSferics ToA, Blitzortung RED, GRLevel3, WD, WL, VWS, Cumulus, Meteobridge/hub
Free weather PHP scripts/website templates - update notifications on Twitter saratogaWXPHP

Offline txweather.org

  • Forecaster
  • *****
  • Posts: 1526
    • Texas Weather
Re: WU-History now works with WU/TMC API...
« Reply #1 on: May 23, 2019, 04:56:44 PM »
Yeeeeeeeeeeeeehhhhhhhhaaaaaaaaaa!!!! Thanks Ken!!!!
You are da man!

----
Davis Vantage Pro2 Plus +FARS|Meteobridge Nano SD|Meteohub|Meteobridge MR-3020|WU KTXSPRIN75/PWS JRARGWX75/CWOP EW2972/WBB TXWDVUE75/Blitzortung ID: 1142|AWEKAS: 12095
Donations are welcome: https://paypal.me/ffuentesb

Offline tmabell

  • Senior Contributor
  • ****
  • Posts: 269
Re: WU-History now works with WU/TMC API...
« Reply #2 on: May 24, 2019, 06:01:10 AM »
I must have fat-fingered something?  This is all I get:

Quote
Sorry... the WU data for this date is not currently available. Please try again later.



The data seems to have been successfully cached:



 (I've removed my API key for the moment).
« Last Edit: June 05, 2019, 05:42:46 PM by tmabell »

Offline txweather.org

  • Forecaster
  • *****
  • Posts: 1526
    • Texas Weather
Re: WU-History now works with WU/TMC API...
« Reply #3 on: May 24, 2019, 10:08:14 AM »
Heads up for those who like to copy and paste...

The section in the readme of the github has a small typo:

Code: [Select]
If running in a Saratoga template, the following can be used in Settings.php instead of customizing WXDailyHistory.php

$SITE['WCAPIkey'] = 'specify-your-api-key-here';
$SITE['WCunits']  = 'e'; // ='e' for F,mph,inHg,in; ='m' for C,km/h,hPa,mm
$SITE['WUID']  = 'KCASARAT1' // your WU PWS ID
$SITE['tz']   = 'America/Los_Angeles'; // timezone
$SITE['cacheFileDir'] = './cache/';  // cache file directory

(the same WCAPIkey and WCunits used for the WC-forecast.php script are reused with this script)

You need a ";" at the end of $SITE['WUID']  = 'KCASARAT1'
Like this: $SITE['WUID']  = 'KCASARAT1';

Just a heads up :)

----
Davis Vantage Pro2 Plus +FARS|Meteobridge Nano SD|Meteohub|Meteobridge MR-3020|WU KTXSPRIN75/PWS JRARGWX75/CWOP EW2972/WBB TXWDVUE75/Blitzortung ID: 1142|AWEKAS: 12095
Donations are welcome: https://paypal.me/ffuentesb

Offline ed2kayak

  • Ed
  • Forecaster
  • *****
  • Posts: 593
  • Davis Vantage Pro2 w/12 hr FARS, solar/uv
    • Cumberland Valley Weather
Re: WU-History now works with WU/TMC API...
« Reply #4 on: May 24, 2019, 10:21:51 AM »
Ken Thank you!! =D>
Ed
CoCoRaHS PA-CD-6,  Weather Underground KPAMECHA7, CWOP DW5425

Offline saratogaWX

  • Administrator
  • Forecaster
  • *****
  • Posts: 7418
  • Saratoga, CA, USA Weather - free PHP scripts
    • Saratoga-Weather.org
Re: WU-History now works with WU/TMC API...
« Reply #5 on: May 24, 2019, 10:29:47 AM »
Heads up for those who like to copy and paste...

The section in the readme of the github has a small typo:

Code: [Select]
If running in a Saratoga template, the following can be used in Settings.php instead of customizing WXDailyHistory.php

$SITE['WCAPIkey'] = 'specify-your-api-key-here';
$SITE['WCunits']  = 'e'; // ='e' for F,mph,inHg,in; ='m' for C,km/h,hPa,mm
$SITE['WUID']  = 'KCASARAT1' // your WU PWS ID
$SITE['tz']   = 'America/Los_Angeles'; // timezone
$SITE['cacheFileDir'] = './cache/';  // cache file directory

(the same WCAPIkey and WCunits used for the WC-forecast.php script are reused with this script)

You need a ";" at the end of $SITE['WUID']  = 'KCASARAT1'
Like this: $SITE['WUID']  = 'KCASARAT1';

Just a heads up :)
Oops... thanks for finding the typo.. I've corrected the example on GitHub. 
Ken True/Saratoga, CA, USA main site: saratoga-weather.org
Davis VP1+ FARS, Boltek-PCI/NexStorm, microSferics ToA, Blitzortung RED, GRLevel3, WD, WL, VWS, Cumulus, Meteobridge/hub
Free weather PHP scripts/website templates - update notifications on Twitter saratogaWXPHP

Offline txweather.org

  • Forecaster
  • *****
  • Posts: 1526
    • Texas Weather
Re: WU-History now works with WU/TMC API...
« Reply #6 on: May 24, 2019, 10:37:03 AM »
Thanks Ken!
Ken, Are the graphs coming and going normal?
My graphs display ok yesterday but today they dont.
https://www.txweather.org/wxwuhistory.php

----
Davis Vantage Pro2 Plus +FARS|Meteobridge Nano SD|Meteohub|Meteobridge MR-3020|WU KTXSPRIN75/PWS JRARGWX75/CWOP EW2972/WBB TXWDVUE75/Blitzortung ID: 1142|AWEKAS: 12095
Donations are welcome: https://paypal.me/ffuentesb

Offline saratogaWX

  • Administrator
  • Forecaster
  • *****
  • Posts: 7418
  • Saratoga, CA, USA Weather - free PHP scripts
    • Saratoga-Weather.org
Re: WU-History now works with WU/TMC API...
« Reply #7 on: May 24, 2019, 12:14:52 PM »
The graphs are likely to vanish permanently -- the WU script that creates them (cgi-bin/wxStationGraphAll ) is a CGI script (likely a Perl script) and is likely to be deprecated in the future.
Ken True/Saratoga, CA, USA main site: saratoga-weather.org
Davis VP1+ FARS, Boltek-PCI/NexStorm, microSferics ToA, Blitzortung RED, GRLevel3, WD, WL, VWS, Cumulus, Meteobridge/hub
Free weather PHP scripts/website templates - update notifications on Twitter saratogaWXPHP

Offline saratogaWX

  • Administrator
  • Forecaster
  • *****
  • Posts: 7418
  • Saratoga, CA, USA Weather - free PHP scripts
    • Saratoga-Weather.org
Re: WU-History now works with WU/TMC API...
« Reply #8 on: May 24, 2019, 12:32:04 PM »
I must have fat-fingered something?  This is all I get:

Quote
Sorry... the WU data for this date is not currently available. Please try again later.

https://mymishawakaweather.com/WU-History-Inc.php

The data seems to have been successfully cached:

https://mymishawakaweather.com/wxdailyhistory.php

 (I've removed my API key for the moment).
Hi Tom,
Looking at view-source:https://mymishawakaweather.com/WU-History-Inc.php?debug=y shows:
Quote
<!-- curl fetching '<!-- curl fetching 'https://mymishawakaweather.com/WXDailyHistory.php?ID=KINMISHA2&month=05&day=24&year=2019&format=1&graphspan=day' -->
<!-- Error: Connection timed out after 4000 milliseconds -->
<!-- HTTP stats:  RC=0 dest= port=0 (from sce=)
      Times: dns=0.001 conn=0.000 pxfer=0.000 get=4.000 total=4.000 secs -->' -->
<!-- Error: Connection timed out after 4000 milliseconds -->
<!-- HTTP stats:  RC=0 dest= port=0 (from sce=)
      Times: dns=0.001 conn=0.000 pxfer=0.000 get=4.000 total=4.000 secs -->
While going to https://mymishawakaweather.com/WXDailyHistory.php?ID=KINMISHA2&month=05&day=24&year=2019&format=1&graphspan=day
does return data (as expected).

The curl timeout is troubling.. is your site running into a concurrent number of PHP processes allowed issue?  Do your have a web application firewall that might be inhibiting the script from execution?

BTW.. I ran WXDailyHistory.php?graphspan=year and the cache files filled successfully in your ./cache/ for Jan-Apr, 2019 so the script works...
Ken True/Saratoga, CA, USA main site: saratoga-weather.org
Davis VP1+ FARS, Boltek-PCI/NexStorm, microSferics ToA, Blitzortung RED, GRLevel3, WD, WL, VWS, Cumulus, Meteobridge/hub
Free weather PHP scripts/website templates - update notifications on Twitter saratogaWXPHP

Offline tmabell

  • Senior Contributor
  • ****
  • Posts: 269
Re: WU-History now works with WU/TMC API...
« Reply #9 on: May 24, 2019, 12:54:16 PM »
Thanks for looking into this for me.  Yes there is something going on that I have yet to identify but to answer your questions, I run my own server locally and have zero restrictions in place that would prevent the script form working.  The firewall should not be an issue but I will look into it.  All other scripts work fine.  After seeing what you tried, I did the same thing from that machine and got the same result.  However what I found interesting was that by using "localhost" as the source for the url as opposed to my web address, it did work.  I have some investigative work to do on my end.

Offline rlee171

  • Senior Member
  • **
  • Posts: 85
    • Peaceful Hills Weather
Re: WU-History now works with WU/TMC API...
« Reply #10 on: May 25, 2019, 09:30:32 PM »
Ken, I have accomplished the updates...   your documentation made it a cake walk for those of us still who have very minimal coding skills. I still have a problem that I was hoping it would be fixed with the new changes since it appears the reference to WUG-inc.day.php leads me to believe it may be WU related. I am using the HighCharts for my station graphs. I have attached a jpg of the error I am receiving. You have been plenty busy taking care of what has needed to be done, so I am not purposely asking you to solve this issue, it just seemed like an appropriate place to ask while lots of folks are focused on WU.  Thank you sincerely for the additional time you have been spending especially taking the position of being our "spokesperson" and giving WU the luxury of not having to address us as an angry mob. If one of you guys (or gals) have a direction to point me on for this one, any advice would be greatly appreciated.

Ralph  hsnpar.org

 [ You are not allowed to view attachments ]



 
Davis Vantage Pro 2
Cumulus v.1.9.4
Hot Springs National Park, AR
hsnpar.com

Offline saratogaWX

  • Administrator
  • Forecaster
  • *****
  • Posts: 7418
  • Saratoga, CA, USA Weather - free PHP scripts
    • Saratoga-Weather.org
Re: WU-History now works with WU/TMC API...
« Reply #11 on: May 26, 2019, 01:15:43 AM »
You appear to be running on PHP Version: 5.4.19 which is quite old.. any chance of updating to 5.6 or 7.3?

The image shows you aren't trying to run my script (WXDailyHistory.php) on your site, but you're still configured to use the WU WXdailyHistory.asp page call, and that won't work anymore.

From the distribution, take a look at my WU-History-inc.php and make corresponding changes to your WUG-inc-day.php script. 
Ken True/Saratoga, CA, USA main site: saratoga-weather.org
Davis VP1+ FARS, Boltek-PCI/NexStorm, microSferics ToA, Blitzortung RED, GRLevel3, WD, WL, VWS, Cumulus, Meteobridge/hub
Free weather PHP scripts/website templates - update notifications on Twitter saratogaWXPHP

Offline rlee171

  • Senior Member
  • **
  • Posts: 85
    • Peaceful Hills Weather
Re: WU-History now works with WU/TMC API...
« Reply #12 on: May 26, 2019, 06:16:16 PM »
I was able to get up to PHP Ver 7.2.7 - thank you for pointing that out. I am still working through the other changes you have addressed.
Thank you very much for your continued support. 

Ralph
Davis Vantage Pro 2
Cumulus v.1.9.4
Hot Springs National Park, AR
hsnpar.com

Offline saratogaWX

  • Administrator
  • Forecaster
  • *****
  • Posts: 7418
  • Saratoga, CA, USA Weather - free PHP scripts
    • Saratoga-Weather.org
Re: WU-History now works with WU/TMC API...
« Reply #13 on: May 26, 2019, 06:24:59 PM »
I now see on the view-source of the page:
Quote
<!-- curl fetching 'http://hsnpar.com/WXDailyHistory.php?ID=KARHOTSP16&month=05&day=26&year=2019&format=1&graphspan=day' -->
<!-- HTTP stats:  RC=200 dest=23.229.178.136 port=80 (from sce=23.229.178.136)
      Times: dns=0.004 conn=0.005 pxfer=0.005 get=0.105 total=0.110 secs -->
<!-- headers:
HTTP/1.1 200 OK
Date: Sun, 26 May 2019 22:12:43 GMT
Server: Apache
X-Powered-By: PHP/7.2.16
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate
Pragma: no-cache
Set-Cookie: PHPSESSID=36c95746f24068c1fce815435ba30727; path=/
Upgrade: h2,h2c
Connection: Upgrade
Vary: User-Agent
Content-Length: 0
Content-Type: text/html; charset=UTF-8
 -->
<p>Sorry... the WU data for this date is not currently available.  Please try again later.</p>
It's odd that a 200-OK reply would result in no JSON content.  Your https://www.wunderground.com/dashboard/pws/KARHOTSP16 PWS page shows data.  Digging deeper, and running a view-source for WXDailyHistory.php with &debug=y shows
Quote
<!-- curl fetching 'https://api.weather.com/v2/pws/history/all?stationId=KARHOTSP16&format=json&units=e&date=20190526&apiKey=**redacted**' -->
<!-- HTTP stats:  RC=401 dest=23.63.48.252 port=443 (from sce=50.62.176.52)
      Times: dns=0.004 conn=0.014 pxfer=0.047 get=0.046 total=0.093 secs -->
<!-- headers returned:
HTTP/2 401
mime-version: 1.0
content-type: text/html
content-length: 299
expires: Sun, 26 May 2019 22:18:52 GMT
date: Sun, 26 May 2019 22:18:52 GMT
 -->
a 401 response.  The API is returning content of
Quote
Access Denied
You don't have permission to access "http://api.weather.com/v2/pws/history/all?" on this server.

Reference #18.5cef6368.1558909171.1226b63
So I suspect your WU/TWC API Key is in error.. Please double check (by wunderground, Profile, API KEY, show and copy that key to use.
Ken True/Saratoga, CA, USA main site: saratoga-weather.org
Davis VP1+ FARS, Boltek-PCI/NexStorm, microSferics ToA, Blitzortung RED, GRLevel3, WD, WL, VWS, Cumulus, Meteobridge/hub
Free weather PHP scripts/website templates - update notifications on Twitter saratogaWXPHP

Offline neavilag

  • Member
  • *
  • Posts: 3
    • ClimaYa.com
Re: WU-History now works with WU/TMC API...
« Reply #14 on: May 27, 2019, 01:08:26 AM »
Thanks for the script, I was hitting WU site to store my history data and present from my DB directly in my site. I did this via the CSV option, but I saw it stop working.

Seems based on what you have posted, the only option now is via the API, do you know the limits of this API if I want to fetch history every 10 minutes in order to have my latest values, will this hit the limits ?

thanks again.

Offline txweather.org

  • Forecaster
  • *****
  • Posts: 1526
    • Texas Weather
Re: WU-History now works with WU/TMC API...
« Reply #15 on: May 27, 2019, 01:28:56 AM »
Ken do we have to put this on a cron?

WXDailyHistory.php?&year=2019&graphspan=year&debug=y

I have pulled the data and I was able to see it but now is no longer displaying.

Sorry... the WU data for this date is not currently available. Please try again later.

----
Davis Vantage Pro2 Plus +FARS|Meteobridge Nano SD|Meteohub|Meteobridge MR-3020|WU KTXSPRIN75/PWS JRARGWX75/CWOP EW2972/WBB TXWDVUE75/Blitzortung ID: 1142|AWEKAS: 12095
Donations are welcome: https://paypal.me/ffuentesb

Offline rlee171

  • Senior Member
  • **
  • Posts: 85
    • Peaceful Hills Weather
Re: WU-History now works with WU/TMC API...
« Reply #16 on: May 27, 2019, 04:59:32 PM »

<p>Sorry... the WU data for this date is not currently available.  Please try again later.</p>[/quote]
It's odd that a 200-OK reply would result in no JSON content.  Your https://www.wunderground.com/dashboard/pws/KARHOTSP16 PWS page shows data.  Digging deeper, and running a view-source for WXDailyHistory.php with &debug=y shows
Quote
<!-- curl fetching 'https://api.weather.com/v2/pws/history/all?stationId=KARHOTSP16&format=json&units=e&date=20190526&apiKey=**redacted**' -->
<!-- HTTP stats:  RC=401 dest=23.63.48.252 port=443 (from sce=50.62.176.52)
      Times: dns=0.004 conn=0.014 pxfer=0.047 get=0.046 total=0.093 secs -->
<!-- headers returned:
HTTP/2 401
mime-version: 1.0
content-type: text/html
content-length: 299
expires: Sun, 26 May 2019 22:18:52 GMT
date: Sun, 26 May 2019 22:18:52 GMT
 -->
a 401 response.  The API is returning content of
Quote
Access Denied
You don't have permission to access "http://api.weather.com/v2/pws/history/all?" on this server.

Reference #18.5cef6368.1558909171.1226b63
So I suspect your WU/TWC API Key is in error.. Please double check (by wunderground, Profile, API KEY, show and copy that key to use.
[/quote]

I double checked my key and it is correct, however there may be a problem with the way I placed it in the WXDailyHistory script.
It's there twice, I didn't know what to put in the " 2nd line - print "replacing. I only have the one key, nothing to replace.


if(preg_match('|specify|i',$WCAPIkey)) {
   print "<p>Note: the WXDailyHistory script requires an API key from WeatherUnderground to operate.<br/>";
   print "Visit <a href=\"https://www.wunderground.com/member/api-keys\">Weather Underground</a> to ";
   print "register for an API key. You must have a PWS submitting data to WU to acquire an API key.</p>\n";
   if( isset($SITE['fcsturlWC']) ) {
      print "<p>Insert in Settings.php an entry for:<br/><br/>\n";
      print "\$SITE['WCAPIkey'] = '<i>**key inserted here**</i>';<br/><br/>\n";
      print "replacing <i>**and also here**</i> with your WU/TWC API key.</p>\n";
   }
   return;

Thanks,
Ralph
Davis Vantage Pro 2
Cumulus v.1.9.4
Hot Springs National Park, AR
hsnpar.com

Offline BCJKiwi

  • Senior Contributor
  • ****
  • Posts: 290
    • Silver Acorn Weather - N.Z.
Re: WU-History now works with WU/TMC API...
« Reply #17 on: May 27, 2019, 07:59:14 PM »
Also only seeing:-
           Sorry... the WU data for this date is not currently available. Please try again later.

All the historic data data has been downloaded into the cache including the current month in which the last day is 27/May 2019 (presumably the UTC date as it is the same date in the regularly downloaded wunderground forecast used in ajax dashboard - .cache/WC-forecast-json-0-m-en.txt ).

So, the api is correct (already in settings.php)

Any suggestions?

Thanks

Offline Stegrie

  • Senior Member
  • **
  • Posts: 86
    • Wetterstation Johanngeorgenstadt
Re: WU-History now works with WU/TMC API...
« Reply #18 on: May 28, 2019, 12:29:40 PM »
The Script will not work yet  :?:
Yesterday got working, today came: Sorry... the WU data for this date is not currently available. Please try again later.
Can anybody look or help?
Cache work and has Data
WD send to WU correctly
I dont know what's wrong...

https://www.wetterstation-johanngeorgenstadt.de/wxwuhistory.php

Stefan

Offline saratogaWX

  • Administrator
  • Forecaster
  • *****
  • Posts: 7418
  • Saratoga, CA, USA Weather - free PHP scripts
    • Saratoga-Weather.org
Re: WU-History now works with WU/TMC API...
« Reply #19 on: May 28, 2019, 04:10:10 PM »
I don’t see a $SITE[‘WCAPIkey’] entry in your Settings.php .. there’s a $SITE[‘WUAPIkey’] entry.. maybe that should be WCAPIkey ?
Ken True/Saratoga, CA, USA main site: saratoga-weather.org
Davis VP1+ FARS, Boltek-PCI/NexStorm, microSferics ToA, Blitzortung RED, GRLevel3, WD, WL, VWS, Cumulus, Meteobridge/hub
Free weather PHP scripts/website templates - update notifications on Twitter saratogaWXPHP

Offline BCJKiwi

  • Senior Contributor
  • ****
  • Posts: 290
    • Silver Acorn Weather - N.Z.
Re: WU-History now works with WU/TMC API...
« Reply #20 on: May 28, 2019, 04:51:54 PM »
Ken,
Presume your reply is to Stegrie as I do have all the required WC and WU/TWC entries in settings.php

Thanks

Offline saratogaWX

  • Administrator
  • Forecaster
  • *****
  • Posts: 7418
  • Saratoga, CA, USA Weather - free PHP scripts
    • Saratoga-Weather.org
Re: WU-History now works with WU/TMC API...
« Reply #21 on: May 28, 2019, 06:12:21 PM »
Ken,
Presume your reply is to Stegrie as I do have all the required WC and WU/TWC entries in settings.php

Thanks
I think your issue is that WXDailyHistory.php is not in the same directory as wxwuhistory.php and WU-History-inc.php

I get file-not-found when trying to do https://silveracorn.nz/weather/WXDailyHistory.php?sce=view
Ken True/Saratoga, CA, USA main site: saratoga-weather.org
Davis VP1+ FARS, Boltek-PCI/NexStorm, microSferics ToA, Blitzortung RED, GRLevel3, WD, WL, VWS, Cumulus, Meteobridge/hub
Free weather PHP scripts/website templates - update notifications on Twitter saratogaWXPHP

Offline baedkesj

  • Member
  • *
  • Posts: 11
    • Harrisonburg, VA Weather
Re: WU-History now works with WU/TMC API...
« Reply #22 on: May 28, 2019, 07:31:17 PM »
Like a few others..I'm now having issues displaying data using this script.  I also get "Sorry... the WU data for this date is not currently available. Please try again later." message.  However I pretty sure I see files in the ./cache directory.

Link: https://www.hburg-wx.net/wxwuhistory.php

All help appreciated!
steve

Offline zeppline

  • Member
  • *
  • Posts: 13
    • Las Vegas Weather Guy
Re: WU-History now works with WU/TMC API...
« Reply #23 on: May 28, 2019, 09:03:14 PM »
I have mine working after a couple of hours. I went back and read instructions. I was missing a file (WXDailyHistory.php). It work if you follow the instruction.  Thanks Ken

Rainwise MKlll w/ cc3000 logger Win 7

Offline txweather.org

  • Forecaster
  • *****
  • Posts: 1526
    • Texas Weather
Re: WU-History now works with WU/TMC API...
« Reply #24 on: May 28, 2019, 09:09:45 PM »
I have mine working after a couple of hours. I went back and read instructions. I was missing a file (WXDailyHistory.php). It work if you follow the instruction.  Thanks Ken

All of my files are in place. So this is not the case for all. What I have notice is that I have to run WXDailyHistory.php at least once a day.

----
Davis Vantage Pro2 Plus +FARS|Meteobridge Nano SD|Meteohub|Meteobridge MR-3020|WU KTXSPRIN75/PWS JRARGWX75/CWOP EW2972/WBB TXWDVUE75/Blitzortung ID: 1142|AWEKAS: 12095
Donations are welcome: https://paypal.me/ffuentesb

 

anything