Author Topic: Web sites failures  (Read 7041 times)

0 Members and 1 Guest are viewing this topic.

Offline PaulMy

  • Forecaster
  • *****
  • Posts: 5519
    • KomokaWeather
Web sites failures
« on: July 28, 2016, 12:18:06 AM »
Last 2 days I have been having issues with my template sites not working correctly:
www.komokaweather.com extremely slow to load (more than 1 minute)
www.komokaweather.ca (Saratoga template) now not loading and giving Internal Server Error and a few times over the past days it did load
www.komokaweather.com/weather28 (Leuven template) not loading and giving Internal Server Error
www.komokaweather.com/j-template (Meteotemplate) not loading and giving Internal Server Error
www.komokaweather.com/pws (HomeWeatherStation) loading but not correctly as cron jobs fail

I've had hours of discussion with GoDaddy who say it is a site optimization issue.  They have no other answer of why all of a sudden this has happened.  Error logging has been turned on and about 400+ log entries in each of 4 files in the last 24 hours, and a recent snipped of the log is:
Quote
[Wed Jul 27 20:12:26 2016] [5379896] [fcgid:warn] [client 75.126.119.130:52852] mod_fcgid: read data timeout in 120 seconds, referer http://www.komokaweather.com/komokaweather-ca
[Wed Jul 27 20:12:26 2016] [5379896] [fcgid:warn] (110)Connection timed out: [client 75.126.119.130:52852] mod_fcgid: ap_pass_brigade failed in handle_request_ipc function, referer http://www.komokaweather.com/komokaweather-ca
[Wed Jul 27 20:12:33 2016] [5379896] [fcgid:warn] [client 64.202.160.161:38204] mod_fcgid: read data timeout in 120 seconds
[Wed Jul 27 20:12:33 2016] [5379896] [fcgid:warn] (110)Connection timed out: [client 64.202.160.161:38204] mod_fcgid: ap_pass_brigade failed in handle_request_ipc function
[Wed Jul 27 20:15:03 2016] [5379896] [fcgid:warn] [client 136.243.171.254:40549] mod_fcgid: read data timeout in 120 seconds
[Wed Jul 27 20:15:03 2016] [5379896] [core:error] [client 136.243.171.254:40549] End of script output before headers: stationcron.php
[Wed Jul 27 20:15:32 2016] [5379896] [fcgid:warn] [client 164.132.161.44:46786] mod_fcgid: read data timeout in 120 seconds
[Wed Jul 27 20:15:32 2016] [5379896] [core:error] [client 164.132.161.44:46786] End of script output before headers: index.php
[Wed Jul 27 20:16:42 2016] [5379896] [fcgid:warn] [client 64.202.160.161:45954] mod_fcgid: read data timeout in 120 seconds, referer http://komokaweather.com/weather/index.htm
[Wed Jul 27 20:16:42 2016] [5379896] [core:error] [client 64.202.160.161:45954] End of script output before headers: bloomSkyLatest.php, referer http://komokaweather.com/weather/index.htm
[Wed Jul 27 20:17:13 2016] [5379896] [fcgid:warn] (104)Connection reset by peer: [client 40.77.167.60:13046] mod_fcgid: ap_pass_brigade failed in handle_request_ipc function
[Wed Jul 27 20:23:41 2016] [5379896] [fcgid:warn] [client 64.202.160.161:48794] mod_fcgid: read data timeout in 120 seconds, referer http://komokaweather.com/weather/index.htm
[Wed Jul 27 20:23:41 2016] [5379896] [core:error] [client 64.202.160.161:48794] End of script output before headers: bloomSkyLatest.php, referer http://komokaweather.com/weather/index.htm
[Wed Jul 27 20:24:29 2016] [5379896] [fcgid:warn] [client 64.202.160.161:51932] mod_fcgid: read data timeout in 120 seconds, referer http://komokaweather.com/weather/index.htm
[Wed Jul 27 20:24:29 2016] [5379896] [core:error] [client 64.202.160.161:51932] End of script output before headers: bloomSkyLatest.php, referer http://komokaweather.com/weather/index.htm

I've apparently gone as high as one can with support, but no resolution.  Other than throw in the towel and get another host, any suggestion on what could be the problem from the error log.

Paul

Offline weather34

  • Forecaster
  • *****
  • Posts: 1068
    • https://weather34.com/homeweatherstation
Re: Web sites failures
« Reply #1 on: July 28, 2016, 03:18:50 AM »
hi paul good morning

sorry to hear and see your problems .

i would suggest is not dump the host at this stage because up to now you have not had these issues so widespread and frequently.

i would start all over again completely clean start from scratch(remove everything a bit like reformatting your windows machine), backup what data you can for historical purposes and re-introduce the templates one by one .
you pretty much have all the popular templates running all of these require resources(my gut feeling you are on the limit of your allocated server resources with a shared hosting package) and all have many files so to diagnose all of that would be a long teething process.

alternatively make a decision to reduce the number of templates you actually run less maintenance going forward easier to diagnose issues when they arise smoother running server for all your data output for the end user.

I know its easy for me to sit here and say but my experience of dealing with popular shared hosting packages has been fine for running normal websites (1 template be it joomla,wordpress etc) but running multiple templates that requires constant server resources has proven to be problematic and often its been case of server resources maxed out and endless logs of internal 500 errors and other failures.

for many years i have had account at http://icdsoft.com using there european servers not until recently i set up mine there and bought a new domain for $5 and added to my existing hosting account this hosting company is very very knowledgeable in many aspects and support is probably the best I had in 20 odd years of using web hosting services.

prior to that i bought a package early 2015 from a company called hostmetro ($88 for 3 years) solely for the purpose of developing a template for weather everything was fine when i was using just one main page but soon as i started to run 2 or 3 variants of the template for developing and using it for other personal stuff  then the issues of internal 500 generated due to scripts failing to load etc  the resources constantly maxed out and no matter how much i tried to get a good diagnostic of what was happening at server level they would always
respond your exceeding your allocated shared hosting resources . so obviously i removed everything bar 1 template problem gone , i recently just dumped this host as it doesnt meet the requirements  .

if you decide on moving host dont transfer everything over may be you just be transferring the issue to another server
here is my personal current experiences with hosts last 10 years

IFASTNET outstanding but costly currently have a italian hotel portal with aprox 178 hotels constantly updating daily (prices)
HOSTGATOR very good previously hosted hotel portal ..
HOSTMETRO forget low cost you get what you pay for..
ICDSOFT very good and knowledgeable where i recommend clients as all round performance and support is very good.
1&1 ok but dependent on what hosting package you choose


weather templates demand more from resources than the most popular or common web sites setups that don't require constant updating and are 95% static in all content. most websites probably run 3 jquery scripts one for a slider , one for a menu and load up others from a jquery cdn . in most cases weather templates and users want to see data update in seconds or updating as frequent as the hardware or software does and have 20 or more scripts in some way .

food for thought paul just a view from afar ...

have a good day..

Offline wvdkuil

  • Wim van der kuil
  • Forecaster
  • *****
  • Posts: 1986
    • My PWS at Leuven Belgium Europe
Re: Web sites failures
« Reply #2 on: July 28, 2016, 03:49:39 AM »
Last 2 days I have been having issues with my template sites not working correctly:
www.komokaweather.com extremely slow to load (more than 1 minute)
www.komokaweather.ca (Saratoga template) now not loading and giving Internal Server Error and a few times over the past days it did load
www.komokaweather.com/weather28 (Leuven template) not loading and giving Internal Server Error
www.komokaweather.com/j-template (Meteotemplate) not loading and giving Internal Server Error
www.komokaweather.com/pws (HomeWeatherStation) loading but not correctly as cron jobs fail

I've had hours of discussion with GoDaddy who say it is a site optimization issue.  They have no other answer of why all of a sudden this has happened.  Error logging has been turned on and about 400+ log entries in each of 4 files in the last 24 hours, and a recent snipped of the log is:
Quote
. . . sohrtened . . .

I've apparently gone as high as one can with support, but no resolution.  Other than throw in the towel and get another host, any suggestion on what could be the problem from the error log.

Paul
1. There are two different "webservers" involved, IP 50.63.51.1 for the .com site and 50.63.202.12 for .ca site.
The .ca site seems to be redirected to the .com server to a folder http://www.komokaweather.com/komokaweather-ca/
So all sites run at the same box/webserver?
http://www.komokaweather.ca/  => redirected using frameset to http://www.komokaweather.com/komokaweather-ca/

2. Lets test the Leuven site, what works and what does not
2.1 loading a text file http://www.komokaweather.com/weather28/uploadWD/clientraw.txt
2.2 running a stand alone php script http://www.komokaweather.com/weather28/printSite.php
2.3 checking uploaded data  http://www.komokaweather.com/weather28/printSite.php?tags&pw=#data-area
2.4 checking the source of a script http://www.komokaweather.com/weather28/wsFooter.php?sce=view
All Ok until now, all less then 10 seconds response time
2.5 Running a real page without overhead http://www.komokaweather.com/weather28/?p=ws_snow&ipad&debug
Also OK, less then 3 seconds
2.6 Running the same page with full menu http://www.komokaweather.com/weather28/?p=ws_snow&debug
2.7 Running a debug page with full menu http://www.komokaweather.com/weather28/index.php?p=0000&lang=en#data-area

Sometimes OK, sometimes time outs on external data loaded on the webserver.

2.7 Running a forecast  http://www.komokaweather.com/weather28/index.php?p=wu_fct_page&lang=en#data-area
Normal error messagewith long response time for loading the external data, probably you did not update the http->https for WU
2.7 Loading met.no forecast http://www.komokaweather.com/weather28/index.php?p=ws_metno_page&lang=en#data-area
OK
All pages have a 404 error for the anything_weather image
no other errors found
===
Saratoga template at http://www.komokaweather.ca/
3.1 http://www.komokaweather.com/komokaweather-ca/check-fetch-times.php runs OK but all time out errors
3.2 http://www.komokaweather.ca => redirected http://www.komokaweather.com/komokaweather-ca  FAILS 500 error
3.3 http://www.komokaweather.ca another time loads after 1.9 minutes but missing data for usno and others
Code: [Select]
Warning: Invalid argument supplied for foreach()
in /home/content/96/5379896/html/komokaweather-ca/get-USNO-sunmoon.php on line 359
Code: [Select]
<!-- get-UV-forecast-inc.php V1.07 - 30-Mar-2012 -->
<!-- UV forecast courtesy of and Copyright &copy; KNMI/ESA (http://www.temis.nl/). Used with permission. -->
<!-- UV data load from from cache ./cache/uv-forecast.txt -->
<!-- data not available -->
At the same time as these errors occur all text loading and displaying sources works OK  for Saratoga and Leuven

Nearly all loading of external data fails, which often results in script errors or 500 errors.

I can test the others also but my preliminary conclusions:

1. you could be banned by some suppliers for data , example uv forecast
2. cache is not always working, data for usno seems to be loaded almost every time
3. when running multiple templates the number of php loads from the browser seems to be overloading your allowance.

Solution:
1. Allow running of the different templates one by one until you find the reason
You can do this by renaming the index.php for every template/website and uploading a 1 line index.php
Code: [Select]
<?php  echo "this site is in maintenance mode"?>Also stop running ALL cron-jobs  immediately!
Wait until all templates work next to each other and then add the cron-jobs one at a time.

2. Start testing with only Saratoga, your first template.
If that works ALL the time after intensive testing add the next one, template-B.
When running template-B check often if  the test scripts return data.
http://www.komokaweather.com/komokaweather-ca/check-fetch-times.php

3. If Saratoga and template-B runs fine add the next one, template-C.
Again test all three and check if normal external loading  scripts work OK

ASK HELP if you need so from the author of the template you are testing (be aware that we are in Europe).
It is in the interest of all parties to have those scripts running again.
You could also make temporary  user-id&passwords, one for every  template folder, so we can inspect and add debug code if needed.

At some point all templates will fail as they do now. Probably because the maximum number of concurrent php process is reached.
Be aware that gauges a.s.o. and ajax-realtime data all execute php scripts even when you are only looking at a page. 
Setting the realtime refreshes to low (to a few seconds) will overload your webserver, this occurs also when running frequent cron-jobs.  Please double-check your settings in javascript as javascript uses milli-seconds not seconds to define an interval.

If all 4 templates work OK together you can then AND only then let the cron-jobs do their work.
Specifying cron-jobs is not easy depending on which provider you are using.

Wim
« Last Edit: July 28, 2016, 04:14:19 AM by wvdkuil »

Offline weather34

  • Forecaster
  • *****
  • Posts: 1068
    • https://weather34.com/homeweatherstation
Re: Web sites failures
« Reply #3 on: July 28, 2016, 03:55:54 AM »
Last 2 days I have been having issues with my template sites not working correctly:
www.komokaweather.com extremely slow to load (more than 1 minute)
www.komokaweather.ca (Saratoga template) now not loading and giving Internal Server Error and a few times over the past days it did load
www.komokaweather.com/weather28 (Leuven template) not loading and giving Internal Server Error
www.komokaweather.com/j-template (Meteotemplate) not loading and giving Internal Server Error
www.komokaweather.com/pws (HomeWeatherStation) loading but not correctly as cron jobs fail

I've had hours of discussion with GoDaddy who say it is a site optimization issue.  They have no other answer of why all of a sudden this has happened.  Error logging has been turned on and about 400+ log entries in each of 4 files in the last 24 hours, and a recent snipped of the log is:
Quote
. . . sohrtened . . .

I've apparently gone as high as one can with support, but no resolution.  Other than throw in the towel and get another host, any suggestion on what could be the problem from the error log.

Paul
1. There are two different "webservers" involved, IP 50.63.51.1 for the .com site and 50.63.202.12 for .ca site.
The .ca site seems to be redirected to the .com server to a folder http://www.komokaweather.com/komokaweather-ca/
So all sites run at the same box/webserver?
http://www.komokaweather.ca/  => redirected using frameset to http://www.komokaweather.com/komokaweather-ca/

2. Lets test the Leuven site, what works and what does not
2.1 loading a text file http://www.komokaweather.com/weather28/uploadWD/clientraw.txt
2.2 running a stand alone php script http://www.komokaweather.com/weather28/printSite.php
2.3 checking uploaded data  http://www.komokaweather.com/weather28/printSite.php?tags&pw=#data-area
2.4 checking the source of a script http://www.komokaweather.com/weather28/wsFooter.php?sce=view
2.5 Running a real page without overhead http://www.komokaweather.com/weather28/?p=ws_snow&ipad&debug
2.6 Running the same page with full menu http://www.komokaweather.com/weather28/?p=ws_snow&debug
2.7 Running a debug page with full menu http://www.komokaweather.com/weather28/index.php?p=0000&lang=en#data-area

All Ok until now, all less then 10 seconds response time

2.7 Running a forecast  http://www.komokaweather.com/weather28/index.php?p=wu_fct_page&lang=en#data-area
Normal error messagewith long response time for loading the external data, probably you did not update the http->https for WU
2.7 Loading met.no forecast http://www.komokaweather.com/weather28/index.php?p=ws_metno_page&lang=en#data-area
OK
All pages have a 404 error for the anything_weather image
no other errors found
===
Saratoga template at http://www.komokaweather.ca/
3.1 http://www.komokaweather.com/komokaweather-ca/check-fetch-times.php runs OK but all time out errors
3.2 http://www.komokaweather.ca => redirected http://www.komokaweather.com/komokaweather-ca  FAILS 500 error
3.3 http://www.komokaweather.ca another time loads after 1.9 minutes but missing data for usno and others
Warning: Invalid argument supplied for foreach()
in /home/content/96/5379896/html/komokaweather-ca/get-USNO-sunmoon.php on line 359
Code: [Select]
<!-- get-UV-forecast-inc.php V1.07 - 30-Mar-2012 -->
<!-- UV forecast courtesy of and Copyright &copy; KNMI/ESA (http://www.temis.nl/). Used with permission. -->
<!-- UV data load from from cache ./cache/uv-forecast.txt -->
<!-- data not available -->
At the same time as these errors occur all text loading and displaying sources works OK  for Saratoga and Leuven

Nearly all loading of external data fails, which often results in script errors or 500 errors.

I can test the others also but my preliminary conclusions:

1. you could be banned by some suppliers for data , example uv forecast
2. cache is not always working, data for usno seems to be loaded almost every time
3. when running multiple templates the number of php loads from the browser seems to be overloading your allowance.

Solution:
1. Allow running of the different templates one by one until you find the reason
You can do this by renaming the index.php for every template/website and uploading a 1 line index.php
Code: [Select]
<?php  echo "this site is in maintenance mode"?>
2. Start with only Saratoga, your first one.
If that works ALL the time after intensive testing add the next one, template-B.
When running template-B check often if  the test scripts return data.
http://www.komokaweather.com/komokaweather-ca/check-fetch-times.php

3. If Saratoga and template-B runs fine add the next one, template-C.
Again test all three and check if normal external loading  scripts work OK

At some point all templates will fail as the maximum number of concurrent php process is reached.
Be aware that gauges a.s.o. and ajax-realtime data all execute php scripts even when you are only looking at a page. 
Setting the realtime refreshes to low (to a few seconds) will overload your webserver, this occurs also when running frequent cron-jobs.

Wim

absolutely 100% Wim agree particularly
quote
1. you could be banned by some suppliers for data , example uv forecast
2. cache is not always working, data for usno seems to be loaded almost every time
3. when running multiple templates the number of php loads from the browser seems to be overloading your allowance.


process of elimination will help you thoroughly ..





Offline Jáchym

  • Meteotemplate Developer
  • Forecaster
  • *****
  • Posts: 8605
    • Meteotemplate
Re: Web sites failures
« Reply #4 on: July 28, 2016, 05:16:29 AM »
I am most inclined into thinking this is a server overload problem. Especially since there are suddenly so many unexpected errors in pages that worked perfectly fine.

I also know that Paul uses probably everything from everything - in case of Meteotemplate, he has all the blocks and all the plugins that exist.

You have to check exactly what your hosting parameters are.

Offline wvdkuil

  • Wim van der kuil
  • Forecaster
  • *****
  • Posts: 1986
    • My PWS at Leuven Belgium Europe
Re: Web sites failures
« Reply #5 on: July 28, 2016, 05:35:10 AM »
I am most inclined into thinking this is a server overload problem. Especially since there are suddenly so many unexpected errors in pages that worked perfectly fine.

I also know that Paul uses probably everything from everything - in case of Meteotemplate, he has all the blocks and all the plugins that exist.

You have to check exactly what your hosting parameters are.
The remaining problem is then to explain why the Saratoga template with one user  without any other user or any other script running fails to load.
It is 99% sure that there is an "overload" problem, we all agree.
But not caused by Paul using all scripts/blocks a.s.o as this morning there were no users on-line but the problems were real.

I think that the underlying problem comes from an "automatic AKA cron AKA refresh" script loading lots of external data in a far to high frequency. We already had one user setting its refresh at 5 milliseconds in javascript. Even Chrome on my fast iMac had problems displaying the error messages fast enough.

Also some scripts tend to load internal website data by use of an file_load(http://full url)
But that kind of loading data is counted as an external load and queued and handled accordingly.

Lets see and solve this as I do not like postings about problems with the word Leuven. Or with Saratoga, HomeWeatherStation or Meteotemplate either.

wim




Offline Jáchym

  • Meteotemplate Developer
  • Forecaster
  • *****
  • Posts: 8605
    • Meteotemplate
Re: Web sites failures
« Reply #6 on: July 28, 2016, 06:05:20 AM »
Yes, well by using all scripts/blocks I meant that many of them also use Ajax.

From my perspective, looking at Meteotemplate:

The site seems to be waiting for about 20-30 seconds which does not make much sense because the index.php is nothing more than a redirect page, with no external data or scripts and only detecting your screen width.

Interestingly though, when I tried the direct URL to the desktop - ie. just use the URL to your homepage that the redirect script should load:
http://www.komokaweather.com/j-template/indexDesktop.php

I get the site loading normally - despite the fact this one has many MBs and the other one a few kbs.

All blocks seem to be loading. though it takes time and they sort of load one by one. Interactive banner also loaded and all the blocks that use external data as well.

Offline wvdkuil

  • Wim van der kuil
  • Forecaster
  • *****
  • Posts: 1986
    • My PWS at Leuven Belgium Europe
Re: Web sites failures
« Reply #7 on: July 28, 2016, 06:27:55 AM »
Yes, well by using all scripts/blocks I meant that many of them also use Ajax.

From my perspective, looking at Meteotemplate:

The site seems to be waiting for about 20-30 seconds which does not make much sense because the index.php is nothing more than a redirect page, with no external data or scripts and only detecting your screen width.

Interestingly though, when I tried the direct URL to the desktop - ie. just use the URL to your homepage that the redirect script should load:
http://www.komokaweather.com/j-template/indexDesktop.php

I get the site loading normally - despite the fact this one has many MBs and the other one a few kbs.

All blocks seem to be loading. though it takes time and they sort of load one by one. Interactive banner also loaded and all the blocks that use external data as well.
First time I tried your link, no success.
Second time wait > 20 seconds, page loads until last part is loaded or failed after 160 seconds.

AND 55 errors most of which 503 service unavailable. One example
Code: [Select]
ttp://www.komokaweather.com/j-template/homepage/blocks/summary/summaryBlock.php 503 (Service Unavailable)And a few 500 errors:
Code: [Select]
http://www.komokaweather.com/j-template/homepage/blocks/warningsCA/warningsCABlock.php 500 (Internal Server Error)
http://www.komokaweather.com/j-template/homepage/blocks/partnerStations/partnerStationsBlock.php 500 (Internal Server Error)
http://www.komokaweather.com/j-template/homepage/blocks/currentUS/currentUSBlock.php 500 (Internal Server Error)
http://www.komokaweather.com/j-template/homepage/blocks/currentAustralia/currentAustraliaBlock.php 500 (Internal Server Error)
See attached screenshot:

And I attached a textfile with all chrome console messages.
Wim
« Last Edit: July 28, 2016, 06:38:02 AM by wvdkuil »

Offline Maumelle Weather

  • Forecaster
  • *****
  • Posts: 1825
    • Maumelle Weather
Re: Web sites failures
« Reply #8 on: July 28, 2016, 06:57:00 AM »
Hi Paul,

Sorry to hear about your webhost issues. Like Brian, I can highly recommend ICDSoft. I have used them for the past 5 years. The ONLY time my site was down was when Superstorm/Hurricane Sandy came ashore in NYC. My site was down about 6 hours, max. Their support is excellent. All of the issues I have had were resolved in under 15 minutes.

I started my first website on GoDaddy in 2009. Stayed with them about 6 months. Had issues with them while uploading both Weather Display and GR3. It was then I found out they allowed only 2 concurrent FTP connections at a time. It was then I moved everything to E-Rice. Only I reason I had left E-Rice in late 2011 was because I ran out of disk space for my site.

Best of luck to you,

John
GR2AE, GR3, Cumulus

Offline Jáchym

  • Meteotemplate Developer
  • Forecaster
  • *****
  • Posts: 8605
    • Meteotemplate
Re: Web sites failures
« Reply #9 on: July 28, 2016, 06:58:43 AM »
Thanks for trying Wim, yes i see that too, but when I tried now, many blocks also timed out, but different ones than in your case. It seems like it just randomly loads a few and the other ones time out meanwhile.

I would say the most important thing right now is for Paul to go to the specifications of the hosting plan he uses and look what exactly the hosting parameters are

Offline Jáchym

  • Meteotemplate Developer
  • Forecaster
  • *****
  • Posts: 8605
    • Meteotemplate
Re: Web sites failures
« Reply #10 on: July 28, 2016, 07:01:32 AM »
GoDaddy T&C:

Quote
Our Web Hosting plans are designed to host most personal, small business and organization websites. We don't limit the amount of storage and bandwidth your site can use as long as it complies with our Hosting Agreement. Should your website bandwidth or storage usage present a risk to the stability, performance or uptime of our servers, we will notify you via email and you may be required to upgrade to a Virtual Private Server or Dedicated (Private) Server, or we may restrict the resources your website is using. It’s very rare that a website violates our Hosting Agreement and is typically only seen in sites that use hosting for file sharing or storage.

"...as it complies with our Hosting Agreement" - this is tricky... reading behind the lines this is a common phrase companies use and it basically means "unlimited but only to a point that we accept, and where that threshold is is completely up to us...."

But what I would assume is that they would notify him as it says in the next sentence so it is strange that even when he contacted the Support department, they didnt say anything.

Offline Jáchym

  • Meteotemplate Developer
  • Forecaster
  • *****
  • Posts: 8605
    • Meteotemplate
Re: Web sites failures
« Reply #11 on: July 28, 2016, 07:03:32 AM »
And here is the relevant paragraph from T&C I found:

Quote
All Web Hosting and WordPress Hosting plans, including the unlimited plans, are subject to a limit of no more than 250,000 inodes per account for Linux® hosting accounts or 500,000 files and folders per account for Windows® hosting accounts. The plans are also limited to no more than 1,000 tables per database and no more than one gigabyte of storage per database. Any account or database that exceeds these limits may be issued a network violation warning and will be required to reduce the number of inodes, files and folders, tables or gigabytes (as the case may be), or may be temporarily or permanently suspended, in our sole discretion.  All Linux hosting plans are subject to the following limitations: no more than a) 25% of one CPU core; b) 512MB of RAM; c) 100 website connections; d) 100 active processes; e) 1 MB/s disk IO.  In the event these limitations are exceeded, your site may slow down or not be served until more resources are added.  More resources may be added for additional fees.   

Offline PaulMy

  • Forecaster
  • *****
  • Posts: 5519
    • KomokaWeather
Re: Web sites failures
« Reply #12 on: July 28, 2016, 07:26:07 AM »
Wow, thanks for your excellent responses and suggestions.  Today is a work day for me so will get at it again this evening.

Regards,
Paul


Offline saratogaWX

  • Administrator
  • Forecaster
  • *****
  • Posts: 9288
  • Saratoga, CA, USA Weather - free PHP scripts
    • Saratoga-Weather.org
Re: Web sites failures
« Reply #13 on: July 28, 2016, 11:17:47 AM »
No Fair... all the Europeans are awake and working the issue while I was asleep .. so I've a bit of catching up to do :)

Looking at check-fetch-times.php?show=info on the komokaweather.ca site shows
Quote
PHP Version: 5.4.19
...
Directories/files status for Base-Canada, CU-Plugin

Status of needed subdirectories
Settings.php Cache file directory in $SITE['cacheFileDir']='./cache/' exists, with permissions=drwx---r-x [0705]
..Wrote 104 bytes to ./cache/test.txt successfully, then deleted test file. Cache directory is fully functional.
Settings.php ajax-images file directory in $SITE['imagesDir']='./ajax-images/' exists; and appears to have contents.
Settings.php ec-icons file directory $SITE['fcsticonsdirEC']='./ec-icons/' exists; but does not have contents. Be sure to upload contents for proper template operation.
Settings.php forecast images file directory in $SITE['fcsticonsdir']='./forecast/images/' exists; and appears to have .jpg image contents.
Settings-weather.php $SITE['graphImageDir']='../weather/images/' exists; and appears to have contents.
Settings-weather.php $SITE['NOAAdir']='../weather/reports/' exists; and appears to have contents.

and only the ec-icons may have an issue.

Running check-fetch-times.php?show=versions on the site had an issue
Quote
..fetching recent version information.
GET /wxtemplates/template-version-info.txt HTTP/1.1
      Host: saratoga-weather.org  Port: 80 IP=216.250.120.252
Network error: Connection timed out (110)
HTTP stats: dns=0.081 conn=2.004 put=n/a get( blocks)=n/a close=n/a total=2.084 secs
fetch function elapsed= 2 secs.
RC=, bytes=0
trying to fetch the current version info .. a connection time out.  The ./cache/template-version-info.txt shows the last successful fetch on Sat, 16 Jul 2016 01:37:19 GMT and I've not done any Cumulus or Base-Canada updates since 03-Jul-2016 so the version check of the files looks good (except for the thermometer.php.. not a biggie).

The check-fetch-times.php is more troubling .. every check, to temis.nl (for UV) or weatheroffice.gc.ca fails with a "Connection timed out (110)" error.

So, as Wim/Brian/Jachym pointed out, your webserver is having an outbound HTTP/HTTPS connection issue so page loads will attempt the outbound connections to refresh the data, wait the 2 to 4 seconds before timing out (each) and it all adds to the overall page load time.

I'm fairly certain that it is not something you can fix -- the hoster will have to fix the issue (unable to connect outbound HTTP to external websites).

Sorry for being late to the party .. I hope this helps.

Best regards,
Ken
Ken True/Saratoga, CA, USA main site: saratoga-weather.org
Davis VP1+ FARS, Blitzortung RED, GRLevel3, WD, WL, VWS, Cumulus, Meteobridge
Free weather PHP scripts/website templates - update notifications on Twitter saratogaWXPHP

Offline PaulMy

  • Forecaster
  • *****
  • Posts: 5519
    • KomokaWeather
Re: Web sites failures
« Reply #14 on: July 28, 2016, 11:56:29 AM »
Quote
So, as Wim/Brian/Jachym pointed out, your webserver is having an outbound HTTP/HTTPS connection issue so page loads will attempt the outbound connections to refresh the data, wait the 2 to 4 seconds before timing out (each) and it all adds to the overall page load time.

I'm fairly certain that it is not something you can fix -- the hoster will have to fix the issue (unable to connect outbound HTTP to external websites).

Sorry for being late to the party .. I hope this helps.

Best regards,
Ken

Thanks Ken, it is not evening yet so not at home but got a couple of minutes to read through the other responses.

You have sparked a couple of things that was mentioned in mydiscussion with Tech Support:
Everything had worked pretty well in its normal way up to Monday night.  I found the problem on Tuesday morning and after some local checking called GoDaddy.  After a little bit of discussion and friendly pushing the Tech Support he did admit "GoDaddy had a problem from the previous evening that was had posted and 2 admins working on it - usually these finds of problems are fixed within 24 hours".  I asked to get a case number but as it was a posted issue he said no case needs to be noted.  He did mention that their problem was to do with "outbound connections" on the server where I was hosted.

After waiting 12 hours I called them to follow up, and then the Tech said "there was no record of my earlier call (which was close to an hour) and there was no record of their server problem".  Next time I will demand a case number.  So after again a complete description of the problem and testing my sites, asking for why all of a sudden everything seems to be failing, etc. They said it was in need of optimization and that my site was very large, etc.  The Tech eventually mentioned that the server problem they had earlier in the day was not the cause of my issue (at least he finally admitted there had been a problem).  The best they could do was to turn on error log and wait 24 hours and see what the errors show and see what should then be done.

After 24 hours, and the issue still exist from trying the templates from an outside computer and from home, I had another long discussion with Tech Support.  Unfortunately that started by them initially quoting the exact same comments from the night before (I am sure she was reading the text noted by the earlier Tech before even asking anything about the current status - i.e. didn't even ask about what the error log was showing).  I should say the 3 Techs I have talked to have been very polite, kind, and saying they want to help, but am feeling they are given the script to continue to repeat to my questions.

Ken, I will pursue this evening re " the hoster will have to fix the issue (unable to connect outbound HTTP to external websites)."

I will also get more details on what I am allowed, or not allowed from my hosting plan.  The Techs had mentioned the resources I was using as being very large but didn't say it was over any limit or if they had any alternatives to overcome the issue.

In hindsight, the issue started the evening after I tried to update Meteotemplate from v7.0 to v8.0.  I did that by first downloading the complete j-template v7.0 folder so I had a full 'webserver' copy of it, which took hours with over 20,000 files copied, then uploading it again so that I could begin replacing the v8.0 folders/files.  This upload also took hours.  Possibly that large volume has put me in the bad book?  I will pursue that as well.

Wim, your suggestion for restarting everything may become necessary and I don't look forward to the task.

Again, thanks for your wealth of information and details and great ammunition when talking to Tech Support.  Very much appreciated.

Paul








Offline PaulMy

  • Forecaster
  • *****
  • Posts: 5519
    • KomokaWeather
Re: Web sites failures
« Reply #15 on: July 28, 2016, 12:12:55 PM »
Another thought as I prepare for this evening.

I also have other domains on my hosting plan with godaddy for a couple of non-profits with which I am involved.  These are on my root folder at the same level as komokaweather-ca (to which the Saratoga template www.komokaweather.ca is redirected)
www.komokawerather.ca is folder komokaweather-ca
www.delkobrydgecanadaday.ca is folder delkobrydgecanadaday (the initial load is now much longer than a week ago - from 5-10 seconds to a minute or more, likely due to large images.  Reload is nearly instant)
www.komokaseniorsapartments.com is folder komokaseniorsapartments (this loads in a few seconds, about the same as it always has)

Just asking if this may add anything to the issue?

Thanks,
Paul

Offline Jáchym

  • Meteotemplate Developer
  • Forecaster
  • *****
  • Posts: 8605
    • Meteotemplate
Re: Web sites failures
« Reply #16 on: July 28, 2016, 12:44:13 PM »
Whatever the problem is, GoDaddy´s attitude is unacceptable

Offline saratogaWX

  • Administrator
  • Forecaster
  • *****
  • Posts: 9288
  • Saratoga, CA, USA Weather - free PHP scripts
    • Saratoga-Weather.org
Re: Web sites failures
« Reply #17 on: July 28, 2016, 01:27:11 PM »
Paul,
All your subsites are hosted on the same (problematic) webserver using the common filesystem, and just separated by having the webserver use a different subdirectory as the document root for each different website.

It all boils back to an outbound connection problem for port 80/443 that your webserver is having an issue with.  I don't think it's a general GoDaddy infrastructure issue, it may just be the upstream routing for that particular server is having an issue.

You can have them refer to the network techs (instead of the server techs) by having 1st level tech support simply log on the webserver with SSH and in the console try:

wget -d weatheroffice.gc.ca -O tst.txt

traceroute weatheroffice.gc.ca

ping weatheroffice.gc.ca

try it several times and they should see the diagnostic messages and the output saved in txt.txt
That should allow them to properly research the connectivity issue without so much wasted effort.  Unfortunately,
most hosters prohibit mere users from running traceroute or ping, so the system admin will have to do that.

Hope this helps...
Best regards (and good luck!)
Ken
Ken True/Saratoga, CA, USA main site: saratoga-weather.org
Davis VP1+ FARS, Blitzortung RED, GRLevel3, WD, WL, VWS, Cumulus, Meteobridge
Free weather PHP scripts/website templates - update notifications on Twitter saratogaWXPHP

Offline Jáchym

  • Meteotemplate Developer
  • Forecaster
  • *****
  • Posts: 8605
    • Meteotemplate
Re: Web sites failures
« Reply #18 on: July 28, 2016, 01:38:03 PM »
If it was an optimization issue then:

1. it would become apparent gradually (page gradually becoming slow)
2. only one domain (template) - the unoptimized one - would show this problem.

But it wouldnt be the case that everything suddenly stops working.

Offline PaulMy

  • Forecaster
  • *****
  • Posts: 5519
    • KomokaWeather
Re: Web sites failures
« Reply #19 on: July 29, 2016, 12:20:45 AM »
Well, no progress after more than 2 hours on the phone, and I am tired!  They did run the tests as Ken suggested
Quote
You can have them refer to the network techs (instead of the server techs) by having 1st level tech support simply log on the webserver with SSH and in the console try:

wget -d weatheroffice.gc.ca -O tst.txt

traceroute weatheroffice.gc.ca

ping weatheroffice.gc.ca

try it several times and they should see the diagnostic messages and the output saved in txt.txt
That should allow them to properly research the connectivity issue without so much wasted effort.  Unfortunately,
most hosters prohibit mere users from running traceroute or ping, so the system admin will have to do that.

and say they were successful, outbound connection were made, that they are not blocking... they no longer have the outbound issue so not a GoDaddy issue.  Tech says he has had 3 Admins involved and that he has gone as high in support as possible.  But I am not convinced of that!

Tech did make some changes to my php5.ini and now it is:
Quote
short_open_tag = off
max_execution_time = 300
memory_limit = 512M
upload_max_filesize = 256M
post_max_size = 256M
max_input_vars = 5000
wait_timeout = 300

Apparently my komokaweather.com uses 14.75 mg (he couldn't explain what that represents but that it is very large)

To my repeated comment that everything was fine as off Monday evening and why all of a sudden now 5 of my templates don't load or don't load properly and have similar issues, he would only say "they are not blocking outbound, and not a GoDaddy issue".  On the logged errors he says the time out it is not at the GoDaddy server.

The only suggestion they have, other than for me to look at the pages content, is to move to a newer server (buy a newer plan) and c-panel which can use PHP 5.6.

Does anyone have any direct contact at GoDaddy that I can try to appeal to?

Paul

Offline wvdkuil

  • Wim van der kuil
  • Forecaster
  • *****
  • Posts: 1986
    • My PWS at Leuven Belgium Europe
Re: Web sites failures
« Reply #20 on: July 29, 2016, 04:25:02 AM »
Well, no progress after more than 2 hours on the phone, and I am tired!  They did run the tests as Ken suggested
Quote
You can have them refer to the network techs (instead of the server techs) by having 1st level tech support simply log on the webserver with SSH and in the console try:

wget -d weatheroffice.gc.ca -O tst.txt

traceroute weatheroffice.gc.ca

ping weatheroffice.gc.ca

try it several times and they should see the diagnostic messages and the output saved in txt.txt
That should allow them to properly research the connectivity issue without so much wasted effort.  Unfortunately,
most hosters prohibit mere users from running traceroute or ping, so the system admin will have to do that.

and say they were successful, outbound connection were made, that they are not blocking... they no longer have the outbound issue so not a GoDaddy issue.  Tech says he has had 3 Admins involved and that he has gone as high in support as possible.  But I am not convinced of that!

Tech did make some changes to my php5.ini and now it is:
Quote
short_open_tag = off
max_execution_time = 300
memory_limit = 512M
upload_max_filesize = 256M
post_max_size = 256M
max_input_vars = 5000
wait_timeout = 300

Apparently my komokaweather.com uses 14.75 mg (he couldn't explain what that represents but that it is very large)

To my repeated comment that everything was fine as off Monday evening and why all of a sudden now 5 of my templates don't load or don't load properly and have similar issues, he would only say "they are not blocking outbound, and not a GoDaddy issue".  On the logged errors he says the time out it is not at the GoDaddy server.

The only suggestion they have, other than for me to look at the pages content, is to move to a newer server (buy a newer plan) and c-panel which can use PHP 5.6.

Does anyone have any direct contact at GoDaddy that I can try to appeal to?

Paul
@PaulMy
In IT since 1968, with at least 10 years  responsibillity for local and wide networking and application support for the largest companies, I want to add my remarks in defense of the support  people. I also had to counter the effects of overloaded queues multiple times: https://en.wikipedia.org/wiki/Queueing_theory

Often with performance problems the customer can not connect the cause = "a 'minor' change somewhere" with the results = "all sites down".
What customers did before the problems started is in their eyes so irrelevant compared to the problem that they do not accept that they themselves introduced the cause for the failure of their system.

The evening before the day all sites started to fail:
1. you installed new / updated versions of Meteotemplate and of ... and maybe ...
2. Probably you adjusted a cron-job or maybe two
And
3. You are running 2 heavy cron driven sites and multiple other sites on the same small Apache webserver shared by 1000 (normal number for a Godaddy host) domains.
4. Maybe one of these "other" sites were changed also?

I  you are really 100% convinced that the cause for these problems is not in one of your changes for the last days,  skip the rest of this post, PM me and I will delete this post.

To be honest, I am not convinced, as when I look to enormous amount of php scripts loaded, any webserver has to queue all requests (in multiple queues) and will handle them one at a time.  Those scripts using only internal data will work OK. But scripts using external data will have to wait until it is their turn.  Even if there were an error in the queue handling, it would occur on all sites sometimes. Not ALWAYS on your site  within a few minutes of browsing.

Your response times are often good in the morning here in Europe, but always suddenly change to very bad.
When I am browsing either Saratoga and Leuven response is all OK. For minutes. External data is loaded and so on. And then suddenly total failure.
How come?  Not because of other users in the middle of your night.  Maybe a cron-job starting? Maybe a automatic backup every ** minutes?


Some scripts loading external data seem to have no timeout set or the system setting is far to large. Example bloomSkyLatest.php: time-out after 120 seconds . 
Maybe there are more of those scripts which are the cause?

And why do some scripts always fail on your site, but you are still loading them:
Code: [Select]
http://a.sitemeter.com/analytics.js Failed to load resource: the server responded with a status of 503 (Service Unavailable: Back-end server is at capacity)
I still think the best way to go is: Test and Test  and Test until you know what is happening and you can prove what is the cause
1. Switch off all template sites => You do not need to un-install or remove them, rename the (folder and the) index.php only
2. Switch off ALL cron-jobs
3. Test your http://www.komokaweather.com/ site and the two other non-template sites.
3. Remove all errors when there are errors on the console

4. Let Saratoga run as the first template and test also check-fetch-times.php
It that has timeout errors it is a GoDaddy problem for sure.
Check also if the cache is functional
Check for sites which maybe banned you are the IP address of your server

5. If Saratoga runs OK lets run Leuven, that  template loads slightly more external data but needs no cron as the remaing two do.
Test with running Saratoga and Leuven and your Cumulus site and do all checks possible

6. If all that is Ok start Meteotemplate WITHOUT  its cron-job
7. If that combination of three templates runs OK,  start the cron-job for Meteotemplate.

and so on.

8. If one of those templates is causing problems, switch it off (including its cron-jobs) and continue with the others.

Your 7 websites of which  4 are templates with ALL possible pages and ALL automatic loads / cron-jobs / external data: that can probably not be handled by a 1000 sites shared server.  At some point the server will collapse, maybe the required memory is higher than the allowed, maybe the queue to the outside world is to small. 

But after testing until the system fails instead of talking about it,  we know far better what the cause is then we do now.
And in this case I think this idiom is very relevant:  https://en.wikipedia.org/wiki/Straw_that_broke_the_camel%27s_back

Wim
« Last Edit: July 29, 2016, 04:59:37 AM by wvdkuil »

Offline Jáchym

  • Meteotemplate Developer
  • Forecaster
  • *****
  • Posts: 8605
    • Meteotemplate
Re: Web sites failures
« Reply #21 on: July 29, 2016, 07:48:50 AM »
One thing that you mentioned is the BloomSkyLatest - that could be problematic because I know the API was down several times in the recent past and if you try to load the webcam image very frequently then that could also be an issue.

Offline PaulMy

  • Forecaster
  • *****
  • Posts: 5519
    • KomokaWeather
Re: Web sites failures
« Reply #22 on: July 29, 2016, 12:17:18 PM »
Thanks Wim for your knowledgeable and thoughtful comments.  A good night's sleep does wonders for the body and mind, and I was tired late last night.

I am following your advise to the best I can reasonably handle.
1. I have removed (renamed) the index files in Saratoga, Leuven, Meteotemplate and PWS folders.
2. I have removed the Cumulus index.htm link in the www.komokaweather.com root index
3. I have edited the Cumulus www.komokaweather/weather/index.htm to remove some of the content that I had added in it over time, and particular those that appeared on the GoDaddy server error log.

Both 2 and 3 seemed to work ok independently.
4. Then put the Cumulus index.htm back in the root index.php iframe and www.komokaweather.com seems ok \:D/

5.  Retried the Saratoga www.komokaweather.ca (renamed back index.php) and it loaded partially but some items missing.  A retry eventually gave an Internal Server Error.

6. Retried the Leuven www.komokaweather.com/weather28 (also renamed back index.php) and that gave an Internal Server Error.

So will go back to 5. Saratoga and start removing items from index.php as I had done with the Cumulus index.htm.  If I can get Saratoga working again, then onto Leuven the same way.

I have been checking my GoDaddy server error logs each time I get a page load issue and can see some errors from my IP but also a number from other IPs which I assume is someone going to my site(s) and getting some failures.  Like these for example:
Quote
[Fri Jul 29 08:15:12 2016] [5379896] [fcgid:warn] [client 157.55.39.66:20194] mod_fcgid: read data timeout in 120 seconds
[Fri Jul 29 08:15:12 2016] [5379896] [core:error] [client 157.55.39.66:20194] End of script output before headers: wxwuhistory.php
[Fri Jul 29 08:16:36 2016] [5379896] [fcgid:warn] [client 157.55.39.225:18737] mod_fcgid: read data timeout in 120 seconds
[Fri Jul 29 08:16:36 2016] [5379896] [core:error] [client 157.55.39.225:18737] End of script output before headers: wxwuhistory.php
[Fri Jul 29 08:59:12 2016] [5379896] [fcgid:warn] [client 51.255.65.27:24399] mod_fcgid: read data timeout in 120 seconds
[Fri Jul 29 08:59:12 2016] [5379896] [core:error] [client 51.255.65.27:24399] End of script output before headers: index.php
[Fri Jul 29 09:01:19 2016] [5379896] [fcgid:warn] [client 164.132.161.23:20272] mod_fcgid: read data timeout in 120 seconds
[Fri Jul 29 09:01:19 2016] [5379896] [core:error] [client 164.132.161.23:20272] End of script output before headers: index.php
[Fri Jul 29 09:02:39 2016] [5379896] [fcgid:warn] [client 51.255.65.72:15100] mod_fcgid: read data timeout in 120 seconds
[Fri Jul 29 09:02:39 2016] [5379896] [core:error] [client 51.255.65.72:15100] End of script output before headers: index.php
[Fri Jul 29 09:07:52 2016] [5379896] [fcgid:warn] [client 51.255.65.76:30575] mod_fcgid: read data timeout in 120 seconds
[Fri Jul 29 09:07:52 2016] [5379896] [core:error] [client 51.255.65.76:30575] End of script output before headers: index.php

And I trust my posting of these web site issues is not taken as any issue with you, Wim, Ken, Jachym and Brian, the developers but just trying to identify where the problem occurs on my sites so I can get it rectified.  I appreciate all the help and guidance you have given.

Paul

Offline Jáchym

  • Meteotemplate Developer
  • Forecaster
  • *****
  • Posts: 8605
    • Meteotemplate
Re: Web sites failures
« Reply #23 on: July 29, 2016, 12:24:48 PM »
Just an idea, but could it be that someone is trying to constantly access your site and thus blocks the free PHP slots and that is the source of all problems? A sort of "DDoS" attack on your site? I can imagine that if someone tried to retrieve data from your site at very short intervals, or even in parallel, then that could very easily overload your server.

Offline Jáchym

  • Meteotemplate Developer
  • Forecaster
  • *****
  • Posts: 8605
    • Meteotemplate
Re: Web sites failures
« Reply #24 on: July 29, 2016, 12:27:02 PM »
51.255.65.72 - France
157.55.39.225 - Washington, Redmond - this is interesting, isnt that Microsoft headquarters?
164.132.161.23 - France
51.255.65.76 - France