Spidering The Spiders.

Publicated on : 1186362926
Wouldn't it be nice to spider some spiders? especially those who spider you without permission, and really those who ignore the robots.txt? Well today I had the joy to spider them back. You can have some fun also, go take a peek and learn how they operate the spider. Cool stuff, they might have left this open on purpose but then again maybe not. Who knows, all I know is that configuration files should not be public.

Snoop onto them as they snoop onto us, http://crawling01.us.archive.org/0