Something tells me that Ask Jeeves and Teoma are indexing the Web today. My logs have been swamped with huge spider crawls like the below all day long:
-
65.214.36.113 – – [14/Apr/2003:19:22:06 -0400] “GET /life/archives/000109.php HTTP/1.0” 200 10890 “-” “Mozilla/2.0 (compatible; Ask Jeeves/Teoma)”
65.214.36.113 – – [14/Apr/2003:19:22:43 -0400] “GET /life/archives/000302.php HTTP/1.0” 200 8765 “-” “Mozilla/2.0 (compatible; Ask Jeeves/Teoma)”
65.214.36.113 – – [14/Apr/2003:19:23:19 -0400] “GET /life/archives/000119.php HTTP/1.0” 200 9303 “-” “Mozilla/2.0 (compatible; Ask Jeeves/Teoma)”
65.214.36.113 – – [14/Apr/2003:19:23:58 -0400] “GET /life/archives/000290.php HTTP/1.0” 200 7958 “-” “Mozilla/2.0 (compatible; Ask Jeeves/Teoma)”
65.214.36.113 – – [14/Apr/2003:19:24:38 -0400] “GET /life/archives/000110.php HTTP/1.0” 200 9911 “-” “Mozilla/2.0 (compatible; Ask Jeeves/Teoma)”
65.214.36.113 – – [14/Apr/2003:19:25:14 -0400] “GET /life/archives/000115.php HTTP/1.0” 200 8072 “-” “Mozilla/2.0 (compatible; Ask Jeeves/Teoma)”
65.214.36.113 – – [14/Apr/2003:19:25:50 -0400] “GET /life/archives/000014.php HTTP/1.0” 200 11672 “-” “Mozilla/2.0 (compatible; Ask Jeeves/Teoma)”
65.214.36.113 – – [14/Apr/2003:19:26:26 -0400] “GET /life/archives/000108.php HTTP/1.0” 200 8113 “-” “Mozilla/2.0 (compatible; Ask Jeeves/Teoma)”
65.214.36.113 – – [14/Apr/2003:19:27:06 -0400] “GET /life/archives/000303.php HTTP/1.0” 200 7843 “-” “Mozilla/2.0 (compatible; Ask Jeeves/Teoma)”
65.214.36.113 – – [14/Apr/2003:19:27:43 -0400] “GET /life/archives/000288.php HTTP/1.0” 200 8158 “-” “Mozilla/2.0 (compatible; Ask Jeeves/Teoma)”
65.214.36.113 – – [14/Apr/2003:19:28:21 -0400] “GET /life/archives/000111.php HTTP/1.0” 200 10061 “-” “Mozilla/2.0 (compatible; Ask Jeeves/Teoma)”
65.214.36.113 – – [14/Apr/2003:19:28:57 -0400] “GET /life/archives/000118.php HTTP/1.0” 200 7897 “-” “Mozilla/2.0 (compatible; Ask Jeeves/Teoma)”
65.214.36.113 – – [14/Apr/2003:19:29:33 -0400] “GET /life/archives/000291.php HTTP/1.0” 200 8673 “-” “Mozilla/2.0 (compatible; Ask Jeeves/Teoma)”
3 responses so far ↓
1 Izzy Anozzy // Jul 1, 2003 at 11:22 pm
Ask Jeeves/Teoma is a very rude bot and should be banned from website invasion. Here’s why:
first it asks for robots.txt which in my case is banning all bots at the moment. Then it goes ahead and ignores the instructions in the robots.txt and crawls my site anyway. That is the rudest bot that visits my site. Here is a fix for your .htaccess file – watch out for word wrap, single lines only, below are 4 lines:
RewriteEngine on
RewriteBase /
RewriteCond %{HTTP_REFERER} web\.ask\.com [OR] # rude bot
RewriteCond %{HTTP_USER_AGENT} Teoma [NC,OR] # rude bot
Use the above then check your log files. Lets stamp out this out right pestilence from the web. Google is well behaved and follows robot.txt instructions. So should ALL bots.
Regards
Izzy
2 Thomas A Vincent // Apr 16, 2004 at 10:54 am
Ask Jeeves went to town on my site today too. Weird the way it did it too.
My logs have:
“GET /title/Bill+Stoddard HTTP/1.0” 200 11387 “-” “Mozilla/2.0 (compatible; Ask Jeeves/Teoma)”
Why would it be searching for those names unless someone is using the ask jeeves web site for spidering?
It is really wierd.
Cheers,
Tom
3 Matthew Wilson // Jun 24, 2004 at 3:17 pm
I’m getting hit big-time by the Jeeves/Teoma spider. It is using up about 1GB per day. It’s used up about 16GB on my site, and I only have about 4GB of bandwidth left. So my site will be going down shortly.
Matt