Had a new one to me hit my site, and hard! The company running the robot has a FAQ that seems fishy as to how their robot scans your site.
Why is your web crawler trying to access pages that don't exist on my website?
Yes I could have been nice an added their bot to the robots.txt on the site, but when I saw the scraping action of the bot, it gained a place in my user agent block list in the .htacess file, sorry, bad mannered robot action I don't support. Bye-bye!!
Here's their URL: http://panscient.com/index.htm
Added note: This bot grabbed my robots.txt a minimum of 5 times within a few minutes!! Can you say "forgetful"?!!
Do a Google search on "panscient com bot" and see the bad writeup this bot has got over the past 5 years, bad bot!