Get Free Quote
« Are You Ready For... | Blog home | Lee Odden’s... »
May 2, 2007

Block Page Sections With Sectional Robots.txt Tag

Print Friendly

Over at WebProNews David Utter reports that Yahoo has created a new tag called [robots-nocontent] that will enable webmasters to mark off sections of a page that they don’t want to show up in the search results. The tag can be used to demark useless boilerplate pages, ad text, navigation, or anything else that would bring irrelevant traffic.

The optimization geek in me is finding this all very exciting.

I like that Yahoo has gone ahead and created this. During last months Robots.txt Summit at Search Engine Strategies New York, Sean Suchter from Yahoo polled the audience on whether they wanted a tag that they could use to tell the engines not to use certain areas of their pages for matching. At the time, I commented in the blog that I very much liked this idea of being able to tell the search engines that, here, this is the article, make this searchable, but this is the page header, make this unsearchable because its irrelevant to the theme of the site.

If I remember correctly, the response of the audience was also overwhelmingly positive and Yahoo seems to have taken that to heart. This tag reflects 100 percent what Sean was discussing. The folks worried about optimization are going to be very happy about this.

This could really help site owners with their search engine optimization campaigns because it increases the significance of your core test by removing all the excess "stuff" that’s on the page. For example, if you’re running a lengthy legal footer or you have a lot of navigation, you can tell Yahoo not to make this information searchable, thereby strengthening your site theme and relevance. This tag will both help optimization and make Yahoo’s search results more relevant. Everyone wins.
David got to speak to received a writeup from Yahoo about the new tag and provided some examples as to how it can be used:

<div class="robots-nocontent">This is the navigational menu of the site and is common on all pages. It contains many terms and keywords not related to this site</div>

<span class="robots-nocontent">This is the site header that is present on all pages of the site and is not related to any particular page</span>

<p class="robots-nocontent">This is a boilerplate legal disclaimer required on each page of the site</p>

<div class="robots-nocontent">This is a section where ads are displayed on the page. Words that show up in ads may be entirely unrelated to the page contents</div>

Two important things to note:

First, don’t worry about this being classified as cloaking – it’s not. The tag can’t be used to "hide" content. Yahoo will still see and index the entirely of your site; the content you flag just won’t be searchable to users.

Second, remember that Yahoo is the only engine currently supporting this tag. Google does not recognize this and will not allow you to make certain portions of your site unsearchable. It’s still optimization as usual over there. At least for now.

Print Friendly




3 responses to “Block Page Sections With Sectional Robots.txt Tag”

  1. David Utter writes:

    Hi Lisa,

    I haven’t spoken to Priyank about this. Yahoo sent along his writeup of robots-nocontent, which was supposed to be online today at 9 am ET. I waited until just about 1 pm ET for the posting to go live, but it hadn’t at that time. I went ahead and posted my story since Yahoo said it was ok to post after 9 anyway.

    Now it looks like the official word will come later this week or after the weekend.

    david

  2. Lisa writes:

    Thanks for the clarification, David. I misread.

  3. David Utter writes:

    Not a problem. Priyank’s post finally went up at Yahoo. For your clicking pleasure:
    http://www.ysearchblog.com/archives/000444.html



Learn SEO
Content Marketing Book
Free Executives Guide To SEO
By continuing to use the site, you agree to the use of cookies. AcceptDo Not Accept
css.php