Topic: 500 server error

[Maumelle Weather]

Hi Ken,

I've been getting a 500 server error quite a bit today, both early this morning and happening some currently. Just thought I would let ya know.

John

[cospringswx]

Me too.

[Bunty]

The error page I get says:

Internal Server Error

The server encountered an internal error or misconfiguration and was unable to complete your request.

Please contact the server administrator, and inform them of the time the error occurred, and anything you might have done that may have caused the error.

More information about this error may be available in the server error log.

Additionally, a 404 Not Found error was encountered while trying to use an ErrorDocument to handle the request.

[DanS]

Same here, more Chinese bots getting through?

[SlowModem]

Same here, more Chinese bots getting through?

Maybe it's Big Brother watching us watch the weather.   

[mkutche]

me too but it hasn't happened to me in a while.. earlier it was like every 5 seconds for me.

[Dr Obbins]

Here too... At least 10 times today alone.

[PaulMy]

Got that a few times earlier this evening but only in trying to open this particular thread, but now fine.

Paul

[saratogaWX]

Yep, we're still getting flooded with simultaneous requests (likely from bad bots) -- I'll have to analyze the logs to see if there are some more IP addresses to block...

Best regards,
Ken

[W3DRM]

Have been getting them consistently this evening. Hard to read anything due to the 500 Internal Server Error message popping-up with almost every read.

The message I am getting is as follows:

Internal Server Error

The server encountered an internal error or misconfiguration and was unable to complete your request.

Please contact the server administrator, and inform them of the time the error occurred, and anything you might have done that may have caused the error.

More information about this error may be available in the server error log.

Additionally, a 404 Not Found error was encountered while trying to use an ErrorDocument to handle the request.

[saratogaWX]

Yep, we were beaten up by yet-another-set-of-rogue spiders.

One set in China, one in the Netherlands and one in the US.  All have been blocked with .htaccess deny on their CIDRs

See.. I can play whack-a-mole with the best of 'em

[Randall Kayfes]

Ken - Keep up the good work!

[Dr Obbins]

Got the error message again this morning, so there may be more moles out there yet. Do these moles also attack our websites, but we are unaware of it?

[moehoward4]

4 AM today....so many errors I had to close my browser just to get off the forum.

[saratogaWX]

It's not so much an 'attack' as it is a rapid series of accesses to various links on the site.  Yes, they probably check your site too, but if you have pages that don't create more links for historical pages, you have likely no issues.

Each wxforum.net page has links to other pages on the forum and if too many requests are made at the same time, the number of PHP threads available is exhausted and you receive the 500 error as Apache is unable to launch another thread to satisfy the request.
The Google, Bing, Yahoo bots are well-behaved (no more than 4 bots crawling at the same time and frequency limited) -- the rogue bots can spew a stream of requests which back up the PHP instances on the webserver, and they tend to ignore the conventions of using robots.txt and in-page <meta name="Robots" content="index,nofollow"/> on pages.  They get blocked (by CIDR) in .htaccess.

[BigOkie]

It's not so much an 'attack' as it is a rapid series of accesses to various links on the site.  Yes, they probably check your site too, but if you have pages that don't create more links for historical pages, you have likely no issues.

Each wxforum.net page has links to other pages on the forum and if too many requests are made at the same time, the number of PHP threads available is exhausted and you receive the 500 error as Apache is unable to launch another thread to satisfy the request.
The Google, Bing, Yahoo bots are well-behaved (no more than 4 bots crawling at the same time and frequency limited) -- the rogue bots can spew a stream of requests which back up the PHP instances on the webserver, and they tend to ignore the conventions of using robots.txt and in-page <meta name="Robots" content="index,nofollow"/> on pages.  They get blocked (by CIDR) in .htaccess.

Which can be problematic because sometimes (just sometimes) a valid IP exists within a block.  But typically these days, they don't, as these !$%*(^*^ buy up blocks of IP addresses.  Finding the right scope is challenging sometimes...yes, I work in IT.

[saratogaWX]

I used to work in IT too

I look up who controls the IP address range of the offending access, and if it turns out to be a country (like China, Ukraine, Russia), I just block the entire CIDR and be done with it.  If the range is controlled by a hosting company, then they get blocked too.
If it is in an ISP-provided range, then a specific IP address block is used instead.  

I've added a

Crawl-delay: 3

to the robots.txt in the hope that the Google, Bing and Yahoo bots will take a more leisurely stroll through the links.  Bing is more agressive in its crawling activity and they do honor that spec so that should reduce the 500 errors and still allow the big-3 to index the site.

BTW.. there's a great IP-range to CIDR converter at http://ip2cidr.com/bulk-ip-to-cidr-converter.php and I tend to use the APNIC for my IP lookups as they report ranges for addresses worldwide.

[BigOkie]

I used to work in IT too

I look up who controls the IP address range of the offending access, and if it turns out to be a country (like China, Ukraine, Russia), I just block the entire CIDR and be done with it.  If the range is controlled by a hosting company, then they get blocked too.
If it is in an ISP-provided range, then a specific IP address block is used instead.  

I've added a

Crawl-delay: 3

to the robots.txt in the hope that the Google, Bing and Yahoo bots will take a more leisurely stroll through the links.  Bing is more agressive in its crawling activity and they do honor that spec so that should reduce the 500 errors and still allow the big-3 to index the site.

BTW.. there's a great IP-range to CIDR converter at http://ip2cidr.com/bulk-ip-to-cidr-converter.php and I tend to use the APNIC for my IP lookups as they report ranges for addresses worldwide.

Hate to see the .htaccess file for this site.   

[saratogaWX]

Actually, thanks to CIDRs, it's only about 40 lines long

[SlowModem]

6:19 pm EDT

Internal Server Error

The server encountered an internal error or misconfiguration and was unable to complete your request.

Please contact the server administrator, and inform them of the time the error occurred, and anything you might have done that may have caused the error.

More information about this error may be available in the server error log.

Additionally, a 404 Not Found error was encountered while trying to use an ErrorDocument to handle the request.

[DanS]

4 AM today....so many errors I had to close my browser just to get off the forum.

06:45 GMT 24 June, I was blocked with this "Chrome" error message stating either the forum was down, had moved, or experiencing an "unknown error.  Error 130". I just love Google Chrome.

[saratogaWX]

It looks like there was a networking issue with 1and1 overnight.  The webserver reports it is up 128 days, but analysis of the Apache logs shows it was having no activity between 01:54:46 and 03:09:10 Eastern time on 24-June-2013.  I had called 1and1 tech support about the issue and it was escalated to the system admins who resolved it (but, don't know what they did or what the root cause was).

Sorry for the inconvenience, and thanks for keeping watch overnight, Dan

[SLOweather]

Interesting... I think that's almost exactly the start and end times of a county wide power failure here in SLO last night/this morning. ~145,000 without power.

It looks like there was a networking issue with 1and1 overnight.  The webserver reports it is up 128 days, but analysis of the Apache logs shows it was having no activity between 01:54:46 and 03:09:10 Eastern time on 24-June-2013.  I had called 1and1 tech support about the issue and it was escalated to the system admins who resolved it (but, don't know what they did or what the root cause was).

Sorry for the inconvenience, and thanks for keeping watch overnight, Dan

[saratogaWX]

Oooh... that is a strange coincidence in timing...  During the outage, wxforum.net wouldn't respond to HTTP, FTP or SSH, and ping failed too.  Must have been a networking issue (bad route? bad BGP entry?) .. we probably won't know as 1and1 does not report root causes for outages generally.
Their Twitter feed has https://twitter.com/1and1/status/273175224231555073 which says

Due to a network outage in a 1&1 data center some services may still be temporarily unavailable.

Edit: the local news said PG&E claimed a problem with the Morro Bay power substation for the SLO county outage last night (145K+ customers)

The California Utilities Emergency Association has advised that the PG&E power outage was due to a "flash" at a sub-station at Morro Bay, resulting in a loss of transmission to several sub-stations.

[jgillett]

Crawl-delay: 3

to the robots.txt in the hope that the Google, Bing and Yahoo bots will take a more leisurely stroll through the links.  Bing is more agressive in its crawling activity and they do honor that spec so that should reduce the 500 errors and still allow the big-3 to index the site.

Having nothing but problems with bingbot here. They were not following any directives I put up (including a crawl delay of 30 as suggested by the bing support folks). It also ignored IP blocks (both specific and ranges), robots.txt, and even a RewriteCond statement (properly formatted). Finally seems to be calming down a bit, however.

Interesting that my Wx site was the only one 'attacked'. I also have 5 WP sites on the same account and they were left alone.