WXforum.net

Web Weather => Weather Web Site Help => Topic started by: Stryder87 on November 28, 2018, 11:33:04 AM

Title: 'Rain Yesterday ' not accurate.
Post by: Stryder87 on November 28, 2018, 11:33:04 AM
Hi

I’m hoping someone can help with a small problem that seems to have cropped up about three weeks ago on my web page (www.moodyweather.ca).

Other than the obvious (at this time) errors about the USNO-sunmoon items (this has been happening for a while but I think it’s a problem on Environment Canada’s end as it seems to clear up after a few days), the problem I’m seeing is that the ‘Rain Yesterday’ figure hasn’t changed for about three weeks.  It’s stuck, and it’s definitely not accurate.  Everything else seems to be working fine except that.

It’s odd because everything had been working fine, and it just quit.  I’m seeing a lot of errors concerning ‘Undefined variables’ in the ajax-dashboard.php file in the error log.  Would it help if I attached the error log, or maybe just copied it here for examination?  The file is nearly 270kb now since I last deleted it (Aug 22/18), so it’s chocked full of interesting things.   :lol:

Thanks.
Title: Re: 'Rain Yesterday ' not accurate.
Post by: PaulMy on November 28, 2018, 12:47:40 PM
Hi,
This may not be helpful but wanted to say that I don't seem to get the same warning message www.komokaweather.ca but it is a slightly modified home page.  Also notice that our versions of get-USNO-sunmoon.php are the same.

Enjoy,
Paul


Title: Re: 'Rain Yesterday ' not accurate.
Post by: Jasiu on November 28, 2018, 04:51:49 PM
USNO: Get Ken's latest code (version 3.02) so you use https rather than http for the call.

Will look into the other thing when I can (if no one else gets to it before me).
Title: Re: 'Rain Yesterday ' not accurate.
Post by: saratogaWX on November 28, 2018, 10:18:17 PM
Yes, you do need the get-USNO-sunmoon.php update to fix the issue with the Notice errata.

The 'Yesterday' data issue is due to not running the saveYesterday.php script via cron.  Since you're using the Ambientweather.net plugin, the only way to capture yesterday data for use tomorrow is via the cron job setup.

Looking at http://www.moodyweather.ca/wxstatus.php shows

Component    Status    Age h:m:s      Latest update time as of Wed, 11-28-2018 7:14pm PST
AmbientWeather.net Network weather data    Current    0:00:41    Wed, 11-28-2018 7:14pm PST
AmbientWeather.net Network Yesterday Data [ View Log] ]    NOT Current    596:15:33    > 24:00:15 Sat, 11-03-2018 11:59pm PDT

and the view log link shows indeed that it last ran successfully Sat, 11-03-2018 11:59pm PDT so your 'Yesterday' data is very stale.
Title: Re: 'Rain Yesterday ' not accurate.
Post by: Stryder87 on November 29, 2018, 11:20:26 AM
Ah... I never noticed that the cron job stopped working.  Weird... wonder why it's failing now.  Guess I'll go look for an error log somewhere.  Thanks for that.

I'll go see if I can find where you're hiding that updated USNO file.  :grin:

Thanks.
Title: Re: 'Rain Yesterday ' not accurate.
Post by: saratogaWX on November 29, 2018, 11:40:29 AM
Using http://www.moodyweather.ca/check-fetch-times.php?show=versions is a good place to start :)
Title: Re: 'Rain Yesterday ' not accurate.
Post by: Stryder87 on November 29, 2018, 11:50:28 AM
Man... you thought of everything!

Do I just download the whole template packs and just replace the files that are updated?  I'm a little concerned that there may be files that were modified.  I'd have to go through each of them and check every line.
Title: Re: 'Rain Yesterday ' not accurate.
Post by: saratogaWX on November 29, 2018, 12:20:47 PM
There is a link near the bottom of the check-fetch-times.php?show=versions page .. Note the date listed, then click the link to use the update tool page on my site.  Select Base, Plugin and the date from that prior page and submit.

It will generate a customized .zip with all the updates you need, and instructions on each file for what to do with it (replace existing, replace after customization).

For now, if you get a get-metar-conditions-inc.php in the update, leave the one you have on your site for now.  I have to issue an update for that script anyway.
Title: Re: 'Rain Yesterday ' not accurate.
Post by: Stryder87 on November 29, 2018, 12:51:36 PM
So I checked the update site, but I don't see any link to download a customized update package (the only link was the update history link).  Not a big deal as I just downloaded the two packs it mentioned.  I then picked out the two files listed on the check-fetch-times page that it showed were out of date (only the ec-radar and the USNO files).

I replaced the USNO one and the errors are gone now (thanks for that!).

Checking the ec-radar file, I noticed a spot where the file directory paths are different:

New line 396 - if(preg_match('|src="/data/radar/temp_image//\S+/(\S+).GIF"|is',$site,$tmatch)) {
Old line 395 - if(preg_match('|src="/data/radar/detailed/temp_image/\S+/(\S+).GIF"|is',$site,$tmatch)) {

There's also a section where the names of image files are different:

New lines 469-477
if($reloadImages) {
      // Now generate overlays in the same order as used by EC
      /*
      /cacheable/images/radar/layers/rivers/wkr_rivers.gif
      /cacheable/images/radar/layers/roads/WKR_roads.gif
      /cacheable/images/radar/layers/road_labels/wkr_labs.gif
      /cacheable/images/radar/layers/radar_circle/radar_circle.gif
      /cacheable/images/radar/layers/additional_cities/wkr_towns.gif
      /cacheable/images/radar/layers/default_cities/wkr_towns.gif
      
Old line 468-476
if($reloadImages) {
      // Now generate overlays in the same order as used by EC
      /*
      /cacheable/images/radar/layers/rivers/WRN_rivers.gif
      /cacheable/images/radar/layers/roads/WRN_roads.gif
      /cacheable/images/radar/layers/road_labels/WRN_labs.gif
      /cacheable/images/radar/layers/radar_circle/radar_circle.gif
      /cacheable/images/radar/layers/additional_cities/WRN_towns.gif
      /cacheable/images/radar/layers/default_cities/WRN_towns.gif

Do I need to worry about the different paths or file names?  I can't find any of them in my site anyway (browsing through FTP).  Even though the file check page didn't say some of the others were out of date, should I be replacing every file from the packs I downloaded (Base-Canada & AWN-plugin)?

Title: Re: 'Rain Yesterday ' not accurate.
Post by: saratogaWX on November 29, 2018, 01:04:44 PM
The directory paths did change on the EC site.  The listings of filenames cited in the last comparison are in comments.. not executed.  Just to remind me where (I think) stuff is so I can correct for future updates to the EC site.

The https://saratoga-weather.org/wxtemplates/updates.php page is the "Update Tool" page .. that's where you can search for and get updates from a specific date (to current) for a Base and Plugin combination.  Much easier than just downloading the Base and Plugin distribution files :)

See the bottom of https://saratoga-weather.org/wxtemplates/maint.php for more info about maintaining your website the easy way.
Title: Re: 'Rain Yesterday ' not accurate.
Post by: Stryder87 on November 29, 2018, 01:10:49 PM
Ok, sounds good.  All I needed to change in the ec-radar file was the first line, changing the $siteID = to WUJ for my area.

Now, concerning the cron job, my command line is "cd $HOME/public_html;/usr/local/bin/php -q saveYesterday.php >> cache/LOG_saveYesterday.php.txt 2> /dev/null".

I have no idea if GoDaddy changed something on their end.  Unfortunately, the log file only shows if it ran or not, not if there was any errors thrown. I'm wondering if I could change the flag (-q may be the one) to something else so it would include any errors as well as if it ran or not.  This, however, is way over my pay grade and I know nothing about this kind of scripting.  Do you know if there is a flag that would accomplish this so I could get an idea why the job quit running?
Title: Re: 'Rain Yesterday ' not accurate.
Post by: saratogaWX on November 29, 2018, 01:16:17 PM
They may have changed the location of PHP from /usr/local/bin/php to something else.  Just remove the ' 2> /dev/null' from the command and try running it again via the cpanel -- (that assumes you've checked the box to send an email to you when the cron runs.. the email should contain any error messages encountered).
Title: Re: 'Rain Yesterday ' not accurate.
Post by: Stryder87 on November 29, 2018, 01:30:02 PM
Alright.  Done.  I wish there was a way to force the cron job to run so I don't have to wait until tomorrow to see what's going on! haha

It's odd though... Storm Rain, This Month and Seasonal Total all seem to be working, yet I would think they would be tied to the same script that Rain Yesterday information comes from.  Is Rain Yesterday the only information that the cron job supplies?
Title: Re: 'Rain Yesterday ' not accurate.
Post by: saratogaWX on November 29, 2018, 01:37:02 PM
In the dashboard on the home page, the min/max temps and rain for yesterday are displayed (if available).  What the saveYesterday.php script does is cache a copy of ALL the $WX[] variables at 23:58 to use as the 'yesterday' values.  The rain numbers (daily, storm, this month, this season) are displayed from current data only as retrieved from the AmbientWeather.net API.  The API doesn't provide 'yesterday' data, that's why the saveYesterday.php script needs to be run hourly at 58 minutes.. it only does the save/capture when local time is 23:58.
Title: Re: 'Rain Yesterday ' not accurate.
Post by: Stryder87 on November 30, 2018, 11:22:35 AM
So I never got an email this morning about the failed cron job, so I took another look at GoDaddy's Cron Job page on how to format the command.  I'm not sure if it's changed, but the example it shows is actually tailored to my site (I guess they customize it for whatever site you're viewing).  Instead of what I had listed above, it says it should be"/usr/local/bin/php /home/<folder>/public_html/path/to/cron/script".

It even lists a Domain-Specific version: "/usr/local/bin/ea-php56 /home/<folder>/domain_path/path/to/cron/script".  I'm not sure if I should use the domain-specific one or not, but I'll try the first one as "/usr/local/bin/php /home/<folder>/public_html -q saveYesterday.php >> cache/LOG_saveYesterday.php.txt" and see what happens.  If it doesn't work, I'll switch to the domain-specific one and see.

Oh the joys....   :roll:
Title: Re: 'Rain Yesterday ' not accurate.
Post by: saratogaWX on November 30, 2018, 12:16:12 PM
Writing a tiny one-line cron task is tricky business in the best of circumstances.  You have to remember that the one line is actually a sh or bash script and must follow Unix command sequence which consists of one or more commands, separated by semi-colons.

The bash command has the form "<executable> <arguments-to-command>".  Since cron runs as 'you' but defaults to your primary directory (not the directory where your website is), you first have to change directory to the place your website and script are located

cd $HOME/public_html

Then you have to run the PHP command-line-processor which may not be in the default path for your userid.  Based on the above, the PHP seems to be '/usr/local/bin/ea-php56'.   The command should run with the -q option (omit HTTP headers). The run the saveYesterday.php script as input, directing the output to append to cache/LOG_saveYesterday.php.txt.  So that command would be (assuming the cd command ran before it)

/usr/local/bin/ea-php56 -q saveYesterday.php >> cache/LOG_saveYesterday.php.txt

So the FULL cron command would be to combine the two bash commands (the cd and the PHP) into one string:

Code: [Select]
cd $HOME/public_html; /usr/local/bin/ea-php56 -q saveYesterday.php >> cache/LOG_saveYesterday.php.txt
Try that one for your cron command.
Title: Re: 'Rain Yesterday ' not accurate.
Post by: Stryder87 on November 30, 2018, 02:18:18 PM
Ok, changed.  let's see how that goes.   :grin:
Title: Re: 'Rain Yesterday ' not accurate.
Post by: Stryder87 on December 03, 2018, 12:44:03 PM
Alright, I let it run a over the weekend, but it still isn't working.  i was wondering if I should change the directory structure of the cron job.  Then I got to looking at the directory line itself.  Something about the php56 struck me.  The instructions said "In the above example, replace “ea-php56” with the PHP version assigned to the domain you wish to use.".  I went back and double-checked the version of the PHP my site is using and it's not 5.6 but 7.2.  So, I changed the line to be php72 instead of php56.  Let's see if it works now.

I'm also not getting any email about the results of the attempt to run even though I input my email address.  Blah.

Guess I'll find out tomorrow.   :lol:
Title: Re: 'Rain Yesterday ' not accurate.
Post by: Jasiu on December 03, 2018, 12:50:05 PM
A couple things to ensure it'll work... ssh onto the server and:

1) which php

This will tell you what version of php runs by default.

2) ls -l /usr/local/bin/*php*

This will show you all of the php versions available and, in particular, if what you put into the cron line exists.
Title: Re: 'Rain Yesterday ' not accurate.
Post by: Stryder87 on December 04, 2018, 11:48:05 AM
Well I finally got an email from GoDaddy about the cron job (first one since I told it to a few days ago).  This is what it said:

Subject Line: "cd $HOME/public_html; /usr/local/bin/ea-php72 -q saveYesterday.php >> cache/LOG_saveYesterday.php.txt"

Message: "/usr/local/cpanel/bin/jailshell: /usr/local/bin/ea-php72: No such file or directory"

The only thing I changed was the 'ea-php72' from 'ea-php56'.  Odd that when it was set to php56 it didn't error out, yet it didn't run, yet after I change it to match the version of PHP running (7.2), it errors out.  This makes no sense to me.

Maybe I'll try the non-domain-specific version and see what happens.
Title: Re: 'Rain Yesterday ' not accurate.
Post by: saratogaWX on December 04, 2018, 02:20:15 PM
If you have SSH access to your site, try the suggestions of Jasiu .. the message you cited simply indicates the PHP command is not that name.. you do have to find out the exact path/name of the PHP command for your hoster.

In an SSH terminal session you can also try:

php -v

and see if that results in a PHP V5+ command.  If it does, then changing the command line to:

cd $HOME/public_html; php -q saveYesterday.php >> cache/LOG_saveYesterday.php.txt

would probably work.  If the php -v command shows PHP4.. it will NOT work.  You must run PHP5 or PHP7.

Call the tech support for your hoster and ask them for the full path/name of the PHP 7.2 command, then use that in place of '/usr/local/bin/ea-php72' in the one line command.
Title: Re: 'Rain Yesterday ' not accurate.
Post by: Stryder87 on December 04, 2018, 04:42:39 PM
So I enabled SSH on my account (I didn't know I could do that) and used Putty to get in.  I tried the php -v command and this is what came back:

ea-php-cli Copyright 2017 cPanel, Inc.
PHP 7.2.6 (cli) (built: May 25 2018 07:56:12) ( NTS )
Copyright (c) 1997-2018 The PHP Group
Zend Engine v3.2.0, Copyright (c) 1998-2018 Zend Technologies

I also ran the command Jasiu suggested (ls -l /usr/local/bin/*php*) and this was the result:

lrwxrwxrwx 1 root root    37 Jun 12 15:25 /usr/local/bin/ea-php55 -> /opt/cpanel             /ea-php55/root/usr/bin/php*
lrwxrwxrwx 1 root root    37 Jun 12 15:25 /usr/local/bin/ea-php56 -> /opt/cpanel             /ea-php56/root/usr/bin/php*
lrwxrwxrwx 1 root root    37 Jun 12 15:25 /usr/local/bin/ea-php70 -> /opt/cpanel             /ea-php70/root/usr/bin/php*
-rwxr-xr-x 1 root root 24421 Aug 10  2017 /usr/local/bin/lsphp*
-rwxr-xr-x 1 root root 24421 Aug 10  2017 /usr/local/bin/php*
-rwxr-xr-x 1 root root  3565 Nov  9  2017 /usr/local/bin/php-config*
-rwxr-xr-x 1 root root  4522 Nov  9  2017 /usr/local/bin/phpize*

So with all that, I'm thinking I should change the command line to the following (since my site is set to use PHP 7.2):
"cd $HOME/public_html; /usr/local/bin/ea-php70 -q saveYesterday.php >> cache/LOG_saveYesterday.php.txt"

Sound about right?
Title: Re: 'Rain Yesterday ' not accurate.
Post by: saratogaWX on December 04, 2018, 06:27:39 PM
If just typing

php -v

at the command prompt via SSH resulted in
Quote
ea-php-cli Copyright 2017 cPanel, Inc.
PHP 7.2.6 (cli) (built: May 25 2018 07:56:12) ( NTS )
Copyright (c) 1997-2018 The PHP Group
Zend Engine v3.2.0, Copyright (c) 1998-2018 Zend Technologies

Then you should use

cd $HOME/public_html; php -q saveYesterday.php >> cache/LOG_saveYesterday.php.txt

as the cron command.
Title: Re: 'Rain Yesterday ' not accurate.
Post by: Stryder87 on December 04, 2018, 09:18:07 PM
Alright.  I'll put that one in and see what tomorrow brings.
 :grin:
Title: Re: 'Rain Yesterday ' not accurate.
Post by: Stryder87 on December 05, 2018, 01:49:36 PM
Soooooo... no change, and no email from GoDaddy on what error it generated.

Since I can now SSH into my site, is there anything I can do to find out what the issue is... access the log file perhaps?

Title: Re: 'Rain Yesterday ' not accurate.
Post by: saratogaWX on December 05, 2018, 07:22:22 PM
I did a saveYesterday.php on your site and forced a collection just to check that the script was running correctly (if invoked).  It ran fine and saved today's values at that time to yesterday's values.

So the problem seems to be with trying to run the script via cron.  Do you have an email address in cPanel to receive the errors if a cron job fails.. if not, please add it.

On my GoDaddy cPanel site, /usr/bin/php is PHP 7.2 so try:

cd $HOME/public_html;/usr/bin/php -q saveYesterday.php >> cache/LOG_saveYesterday.php.txt

as the command.. if it fails, then, we'll have to use a cron batch file instead (and that's a bit trickier to make work as the batch files have to only have \n (new-line) characters, and \r\n (carriage-return,new-line) like windows uses will blow up a batch job).

Title: Re: 'Rain Yesterday ' not accurate.
Post by: Stryder87 on December 05, 2018, 08:36:20 PM
Yes, I added my email a few days ago, but it doesn't seem to send out emails reliably.  I got one in the last week.

I've changed the cron job to be what you suggested.  Let's see what happens tomorrow.

Sorry for all this trouble.   :oops:
Title: Re: 'Rain Yesterday ' not accurate.
Post by: Stryder87 on December 06, 2018, 02:19:40 PM
Ok, so it looks like 'cd $HOME/public_html;/usr/bin/php -q saveYesterday.php >> cache/LOG_saveYesterday.php.txt' didn't work.  Strangely enough, there are a couple errors listed on the Network Status/View Log page.  Not sure what they are referring to.  They seemed to show up after I had turned on showing PHP errors on the PHP configuration page in CPanel.  I've since turned that off.

Just wondering, in the command you had me try, there's no space between 'public_html;' and 'usr/bin...'.  Should there be one?  Looking at the examples they list, it looks like there should be a space there.  Maybe it's just a simple typo now?
Title: Re: 'Rain Yesterday ' not accurate.
Post by: saratogaWX on December 06, 2018, 03:15:14 PM
No, the space between public_html;/usr/bin/php shouldn't matter.

Is your full cron entry like:

Code: [Select]
58 * * * * cd $HOME/public_html;/usr/bin/php -q saveYesterday.php >> cache/LOG_saveYesterday.php.txt
Title: Re: 'Rain Yesterday ' not accurate.
Post by: Stryder87 on December 06, 2018, 04:20:18 PM
59 23 * * * cd $HOME/public_html;/usr/bin/php -q saveYesterday.php >> cache/LOG_saveYesterday.php.txt
Title: Re: 'Rain Yesterday ' not accurate.
Post by: saratogaWX on December 06, 2018, 04:54:06 PM
Ahh... there's the issue.  Change it to

59 * * * * cd $HOME/public_html;/usr/bin/php -q saveYesterday.php >> cache/LOG_saveYesterday.php.txt

It will run once per hour (at :59), but only save if the LOCAL TIME (based on your timezone setting) is in hour=23.

Your setting of 59 23 * * * meant run it at the SERVER time of 23:59, which is likely not the same timezone as your site, so the script ran and just ignored the save request as it wasn't 23:59 in your timezone.
Title: Re: 'Rain Yesterday ' not accurate.
Post by: Stryder87 on December 06, 2018, 05:30:35 PM
And yet it had worked for a while....   :?

Ok, changed.  Now the waiting again...   :lol:
Title: Re: 'Rain Yesterday ' not accurate.
Post by: saratogaWX on December 07, 2018, 12:39:28 AM
And.... I can see it running hourly, but an error message is coming out in the log
Quote
<br />
<b>Warning</b>:  session_start(): Cannot start session when headers already sent in <b>/home/d6bsim7o954q/public_html/include-style-switcher.php</b> on line <b>31</b><br />

This is caused by a trailing space in the last line of both Settings.php and common.php .. change BOTH from
Code: [Select]
?> to be
Code: [Select]
# -- end and that should stop the premature header caused by the rogue space(s) and thereby stop that error message in the log.
Title: Re: 'Rain Yesterday ' not accurate.
Post by: Stryder87 on December 07, 2018, 11:24:27 AM
Yup, looks like it ran to me, too. I've replaced the last line in both files with '# -- end'.  I'm going to delete the LOG_saveTesterday.php.txt file to make it easier to see anything that may now come up (I'll keep a copy just in case).

If everything looks good over the next couple of days, I'll put the '2> /dev/null' back in at the end of the cron job line.

I think I'm going to start a new thread to get some assistance with the huge amount of 'Undefined variable/offset/index' errors I'm seeing in the error_log.  They are mostly in the ajax-dashboard.php and AWNtags.php files with a couple ajax-gizmo.php lines.


Thanks for all the help with this one.  I really wish GoDaddy would be a little more informative when they make changes and how it could affect one's site.  The job had been running fine for quite a while.
Title: Re: 'Rain Yesterday ' not accurate.
Post by: Stryder87 on December 08, 2018, 12:35:29 PM
And.... I can see it running hourly, but an error message is coming out in the log
Quote
<br />
<b>Warning</b>:  session_start(): Cannot start session when headers already sent in <b>/home/d6bsim7o954q/public_html/include-style-switcher.php</b> on line <b>31</b><br />

This is caused by a trailing space in the last line of both Settings.php and common.php .. change BOTH from
Code: [Select]
?> to be
Code: [Select]
# -- end and that should stop the premature header caused by the rogue space(s) and thereby stop that error message in the log.

I made these changes, but I'm seeing this error again.

It's mentioning line 31 in 'include-style-switcher.php'... which says '    session_start();'.
I notice that the last line in this file is the same as the other two you had me change.  Should I change this file too?

Title: Re: 'Rain Yesterday ' not accurate.
Post by: saratogaWX on December 08, 2018, 12:58:24 PM
Yes, give it a try.
Title: Re: 'Rain Yesterday ' not accurate.
Post by: Stryder87 on December 11, 2018, 11:13:40 AM
So it looks like the cron job is successfully up and running again.  I'm still getting those weird session header errors though.  I adjusted all of the files you mentioned, but it's still throwing them.  I think I'm going to put the ' 2> /dev/null' back on the end of the cron job command line and see if that stops the errors.
Title: Re: 'Rain Yesterday ' not accurate.
Post by: Stryder87 on December 12, 2018, 11:46:53 AM
Ok, so I put that flag back into the cron job command line, but I'm still getting 2 instances of the 'Cannot start session when headers' issue, one set of 16 lines before the line saying the job ran and another 8 lines after the job runs.

I've fixed all of the files mentioned earlier (where it seemed to need it), so I'm rather stumped on what it means.