Comments on: Is Slurp Behaving Badly? http://www.bruceclay.com/blog/2007/01/is-slurp-behaving-badly/ SEO and Marketing Blog Sat, 20 Mar 2010 13:43:49 -0700 http://wordpress.org/?v=2.9.2 hourly 1 By: Keith http://www.bruceclay.com/blog/2007/01/is-slurp-behaving-badly/comment-page-1/#comment-15854 Keith Thu, 25 Jan 2007 22:22:53 +0000 http://www.bruceclay.com/blog-test/2007/01/is-slurp-behaving-badly/#comment-15854 <p>We've also noticed Inktomi behaving badly (ignoring robots.txt and MORE.)</p> <p>As of today, we've banned Inktomi, Yahoo, Slurp, and all their IPs permanently from hundreds of servers and thousands of domains.</p> <p>I don't know what they're doing, or who is in control -- but they are acting like the WORST of what we see operating on the Internet. </p> We’ve also noticed Inktomi behaving badly (ignoring robots.txt and MORE.)

As of today, we’ve banned Inktomi, Yahoo, Slurp, and all their IPs permanently from hundreds of servers and thousands of domains.

I don’t know what they’re doing, or who is in control — but they are acting like the WORST of what we see operating on the Internet.

]]> By: matt http://www.bruceclay.com/blog/2007/01/is-slurp-behaving-badly/comment-page-1/#comment-15853 matt Fri, 19 Jan 2007 08:22:10 +0000 http://www.bruceclay.com/blog-test/2007/01/is-slurp-behaving-badly/#comment-15853 don't know much about this, but in trying to figure out who/what inktomi is i found your post. they pop up (from sunnyvale, ca) as a regular visitor to my and many other bloggers i've spoken to. we guessed everything from gov't spying to simple marketing (which is what it's beginning to look like) also noticed with yahoo past couple months that my sign-in page always flashes & duplicates itself as i'm signing in... an odd event that never happened before don’t know much about this, but in trying to figure out who/what inktomi is i found your post.
they pop up (from sunnyvale, ca) as a regular visitor to my and many other bloggers i’ve spoken to. we guessed everything from gov’t spying to simple marketing (which is what it’s beginning to look like)
also noticed with yahoo past couple months that my sign-in page always flashes & duplicates itself as i’m signing in… an odd event that never happened before

]]> By: Andy Redfern http://www.bruceclay.com/blog/2007/01/is-slurp-behaving-badly/comment-page-1/#comment-15852 Andy Redfern Wed, 17 Jan 2007 22:23:53 +0000 http://www.bruceclay.com/blog-test/2007/01/is-slurp-behaving-badly/#comment-15852 <p>I have seen the similar strange behaviour with Yahoo! but also I have noticed that slurp is spidering a couple of my URL's a large number of times. </p> <p>Anyone else noticing this? </p> I have seen the similar strange behaviour with Yahoo! but also I have noticed that slurp is spidering a couple of my URL’s a large number of times.

Anyone else noticing this?

]]> By: Derrick Wheeler http://www.bruceclay.com/blog/2007/01/is-slurp-behaving-badly/comment-page-1/#comment-15851 Derrick Wheeler Wed, 17 Jan 2007 21:57:53 +0000 http://www.bruceclay.com/blog-test/2007/01/is-slurp-behaving-badly/#comment-15851 One more thing... I've seen instances of Google also "partially indexing" pages that are excluded using robots.txt files and using the text from an inbound link as the "Title" One more thing…
I’ve seen instances of Google also “partially indexing” pages that are excluded using robots.txt files and using the text from an inbound link as the “Title”

]]> By: Derrick Wheeler http://www.bruceclay.com/blog/2007/01/is-slurp-behaving-badly/comment-page-1/#comment-15850 Derrick Wheeler Wed, 17 Jan 2007 21:49:08 +0000 http://www.bruceclay.com/blog-test/2007/01/is-slurp-behaving-badly/#comment-15850 <p>It could also be a case of other websites linking to Will's URLs with extra query strings in an attempt to generate duplicate content indexed. Yahoo! might only be crawling the URLs with the extra query strings because someone else is linking to them. Yahoo discovers the URLs through links and then makes a request for the URL. A question for Yahoo is... At what point does Yahoo cross reference the URLs it is requesting with the exclusions listed in the robots.txt files? On a related note, I have seen recent examples of URLs that are excluded using the robots.txt file being "partially indexed." Meaning Yahoo discovered the URL but didnt actually index any content from the URL. In cases like this, I've noticed that Yahoo uses the anchor text of an inbound link as the "Title" of the link in the search results. Danny Sullivan discusses this on his Search Engine Land website. </p> It could also be a case of other websites linking to Will’s URLs with extra query strings in an attempt to generate duplicate content indexed. Yahoo! might only be crawling the URLs with the extra query strings because someone else is linking to them. Yahoo discovers the URLs through links and then makes a request for the URL. A question for Yahoo is… At what point does Yahoo cross reference the URLs it is requesting with the exclusions listed in the robots.txt files? On a related note, I have seen recent examples of URLs that are excluded using the robots.txt file being “partially indexed.” Meaning Yahoo discovered the URL but didnt actually index any content from the URL. In cases like this, I’ve noticed that Yahoo uses the anchor text of an inbound link as the “Title” of the link in the search results. Danny Sullivan discusses this on his Search Engine Land website.

]]>