Topic: Blocked GoogleBot

[Maumelle Weather]

I finally had to block GoogleBot.  Despite setting the crawl rate in both my robots.txt file and my account with them (one should not have to create an account to set a bot crawl rate, IMHO), it has continued to be very aggressive, to the point my daily logs were showing anywhere between 10%-25% of my traffic was GoogleBot (The other one on my site that shows more is TwitterBot, which has been blocked for years, and will not give up, despite been given a 403). I know folks are going to say you'll lose your SEO. I am not worried about my SEO. When I implemented my site back in 2009, it was and still is for my enjoyment. If other folks, like a number of users here, find it useful, that's great and I thank you for looking.

John

[Jáchym]

I understand your point John, it probably is not a big deal for you, but the question is - even if it was in the range of 10-20%, was it a major issue? Was your page running very slow because of that? Because that would be the only time I would seriously worry about it, if my site was running relatively smoothly, then I dont really mind how many times Googlebot visited it. Did you notice any major difference in speed now that you disabled it?

[weatherc]

I have had my dedicated server killed by GoogleBot not long time ago when it hitted my site with 10+ hits/second 24/7.  So, yes, it can be really aggressive.
I had to block it in firewall to get the server online again and had to keep it blocked until the lower rates in Webmaster-tools took affect.
The intresting was that it hitted with random nonsense url-parameters what by-passed completely the in-use caching by the webserver.
 

[Maumelle Weather]

Hi Jachym,

Seems to be a little faster now that I've blocked them outright. Googlebot ignored my crawl rate adjustment in robots.txt so they left me little choice actually but to put them in my .htaccess. Within the past 2 weeks, I had seen crawl rates of 100-200 a second. Didn't happen often, but..... I can only imagine the crawl rate on larger sites 

John

[DoctorKnow]

When I click on the site now, it loads right up. For the last several weeks, when I click on it, to first enter the main page, it would just sit for a while, sometimes a half a minute.

[Jáchym]

Hi Jachym,

Seems to be a little faster now that I've blocked them outright. Googlebot ignored my crawl rate adjustment in robots.txt so they left me little choice actually but to put them in my .htaccess. Within the past 2 weeks, I had seen crawl rates of 100-200 a second. Didn't happen often, but..... I can only imagine the crawl rate on larger sites 

John

Maybe an idea would be to enable it say once per month for a day, so that it has time to still index your site and then disable it again

[Maumelle Weather]

@DoctorKnow - Thanks for the confirmation.

@Jachym - That is a good idea, thanks!!!!!

[azkiwi]

It may not be Google.

Search for false google bot .... False bots are up over 60% and are very aggressive in looking for site flaws..


Ken

[Maumelle Weather]

It's Google, Ken. That is based off of the IP's I've traced via Whois.com. 90%+ start with 66.249.66.xxx. I lot of the fake Google bots I've seen have been blocked.