Yes, well the problem is that these text files tell the search engine - "do not index this", but thats it - in other words - it is only up to that bot if i will really do that. And obviously, the most aggressive bots are usually the ones that ignore this completely, there is nothing preventing them from doing it.
Yes there are some scripts that try to detect them, but I dont have very good experience with these, they either dont work, or they block even normal users.