Had a new one to me hit my site, and hard! The company running the robot has a FAQ that seems fishy as to how their robot scans your site.
Why is your web crawler trying to access pages that don't exist on my website?
Our web crawler attempts to extract links to valid web pages from javascript and other scripting languages. The crawler may misinterpret the information in these scripts and request a page that does not actually exist. These requests are attempts to retrieve valid web content, and are not an attempt to circumvent your webserver security.
In my books, what I saw in the server logs didn't look like a robot scan from Google or Bing but more like a site scraper hammering my site, and going into the javascript, well that is a bit sneaky.
Yes I could have been nice an added their bot to the robots.txt on the site, but when I saw the scraping action of the bot, it gained a place in my user agent block list in the .htacess file, sorry, bad mannered robot action I don't support. Bye-bye!!
Here's their URL:
http://panscient.com/index.htm George
Added note: This bot grabbed my robots.txt a minimum of 5 times within a few minutes!! Can you say "forgetful"?!!
Do a Google search on "panscient com bot" and see the bad writeup this bot has got over the past 5 years, bad bot!
