Topic: Script to Help Control Bot Attacks

[jmcmurry]

I put the attached script together to try to do something about the bots that don't honor robots.txt, and that don't provide a decent user agent so we can combat them in our .htaccess file.  The recent abuse of a few Wunderground scripts highlights the need for the best defense possible.

I've included it at the very beginning of my problem scripts and it's working well in testing.  It uses PHP session() to try to sense and control the miscreants, but won't help with bots that have "patience".  The last flood that I observed on my server had many very rapid hits so this is to stop those. 

The comments in the script should explain everything that's going on.  Feel free to use it and let me know if you have any ideas for improvement.

- Jim

[Maumelle Weather]

A very good idea, Jim.  Does your webhost give you the ability to block countries within your .htaccess? My host ICDSoft started allowing us to block entire countries using .htaccess. I have cut my Bot issues and various miscreants by about 95%. It reduced the size of my .htaccess from 200Kb to about 65Kb or so. I have a few U.S. IP's blocked from trying to be complete a**holes on my site.


John

[SoMDWx]

John,
  I also have ICDSoft....How are  you blocking entire countries? Use of CIDRs (sp?) is helpful but can really increase the size of your .htaccess file, thus increasing web page loading times....

Jim

[Maumelle Weather]

John,
  I also have ICDSoft....How are  you blocking entire countries? Use of CIDRs (sp?) is helpful but can really increase the size of your .htaccess file, thus increasing web page loading times....

Jim

Hi Jim,

Came across that accidentally when I logged into my Control Panel with ICDSoft. I had been using CIDR's and that did make my file quite large. They may have removed it when they upgraded the control panel, because I am no longer seeing or they changed the name of it. Not sure which. I'll shoot you a PM on it.


John

[SoMDWx]

Thanks! Got it.......

Wonder if the country match directives work as well as the CIDR denys.......

Jim

[Maumelle Weather]

I believe so, Jim. It sure has made my life a lot easier dealing with all of that. Those Amazon IP's may need to be updated as well. It's been a few months. I think I have the may site address for those at home. If I do, I'll send it to you as well.

[SoMDWx]

My big issue is not the bots per se... I have a PHP script which detects then permanently blocks those IPs...

The ones i see in my logs are all the .ru sites.... Thankfully there are not too many and they don't keep hitting, dragging my site down.

Jim

[PSJohn]

I've been using a script that blocks bots that ignore htaccess. It has working pretty well for me.
here is the concept:

"The concept is simple: include a hidden link to a robots.txt-forbidden directory somewhere on your pages. Bots that ignore or disobey your robots rules will crawl the link and fall into the honeypot trap, which then performs a WHOIS Lookup and records the event in the blackhole data file. Once added to the blacklist data file, bad bots immediately are denied access to your site. "

Link to creator: https://perishablepress.com/blackhole-bad-bots/

[SoMDWx]

Yep, that's the one....

[rrrick8]

My cPanel on my Bluehost VPS has an IP Blocker builtin and works very well. Just a couple clicks to block entire IP ranges from known bad players countries. Has reduced my bots and other "trespassing" by 99%.
 [ You are not allowed to view attachments ]

That being said, great script Jim to help people on this front.

[ConligWX]

ZB Block is another script you can use.

https://www.zb-block.net/zbf/index.php

[mcrossley]

ZB Block is another script you can use.

https://www.zb-block.net/zbf/index.php

I use that one on my website.

[ConligWX]

Yep I too have used it in the past. Though it not that informative in setting up or getting the correct file updates. The forums content/installation guides are all over the place.

Sent from my GM1913 using Tapatalk