Block Page Sections With Sectional Robots.txt Tag - Bruce Clay, Inc.

Block Page Sections With Sectional Robots.txt Tag

Over at WebProNews David Utter reports that Yahoo has created a new tag called [robots-nocontent] that will enable webmasters to mark off sections of a page that they don’t want to show up in the search results. The tag can be used to demark useless boilerplate pages, ad text, navigation, or anything else that would bring irrelevant traffic.

The optimization geek in me is finding this all very exciting.

I like that Yahoo has gone ahead and created this. During last months Robots.txt Summit at Search Engine Strategies New York, Sean Suchter from Yahoo polled the audience on whether they wanted a tag that they could use to tell the engines not to use certain areas of their pages for matching. At the time, I commented in the blog that I very much liked this idea of being able to tell the search engines that, here, this is the article, make this searchable, but this is the page header, make this unsearchable because its irrelevant to the theme of the site.

If I remember correctly, the response of the audience was also overwhelmingly positive and Yahoo seems to have taken that to heart. This tag reflects 100 percent what Sean was discussing. The folks worried about optimization are going to be very happy about this.

This could really help site owners with their search engine optimization campaigns because it increases the significance of your core test by removing all the excess "stuff" that’s on the page. For example, if you’re running a lengthy legal footer or you have a lot of navigation, you can tell Yahoo not to make this information searchable, thereby strengthening your site theme and relevance. This tag will both help optimization and make Yahoo’s search results more relevant. Everyone wins.
David got to speak to received a writeup from Yahoo about the new tag and provided some examples as to how it can be used:

<div class="robots-nocontent">This is the navigational menu of the site and is common on all pages. It contains many terms and keywords not related to this site</div>

<span class="robots-nocontent">This is the site header that is present on all pages of the site and is not related to any particular page</span>

<p class="robots-nocontent">This is a boilerplate legal disclaimer required on each page of the site</p>

<div class="robots-nocontent">This is a section where ads are displayed on the page. Words that show up in ads may be entirely unrelated to the page contents</div>

Two important things to note:

First, don’t worry about this being classified as cloaking – it’s not. The tag can’t be used to "hide" content. Yahoo will still see and index the entirely of your site; the content you flag just won’t be searchable to users.

Second, remember that Yahoo is the only engine currently supporting this tag. Google does not recognize this and will not allow you to make certain portions of your site unsearchable. It’s still optimization as usual over there. At least for now.

Lisa Barone is a writer, content marketer & VP of strategy at Overit Media. She's also a very active Twitterer, much to the dismay of the rest of the world.

See Lisa's author page for links to connect on social media.

Comments (3)
Filed under: SEO
Still on the hunt for actionable tips and insights? Each of these recent SEO posts is better than the last!

3 Replies to “Block Page Sections With Sectional Robots.txt Tag”

Not a problem. Priyank’s post finally went up at Yahoo. For your clicking pleasure:

Thanks for the clarification, David. I misread.

Hi Lisa,

I haven’t spoken to Priyank about this. Yahoo sent along his writeup of robots-nocontent, which was supposed to be online today at 9 am ET. I waited until just about 1 pm ET for the posting to go live, but it hadn’t at that time. I went ahead and posted my story since Yahoo said it was ok to post after 9 anyway.

Now it looks like the official word will come later this week or after the weekend.



Your email address will not be published. Required fields are marked *

Serving North America based in the Los Angeles Metropolitan Area
Bruce Clay, Inc. | 2245 First St., Suite 101 | Simi Valley, CA 93065
Voice: 1-805-517-1900 | Toll Free: 1-866-517-1900 | Fax: 1-805-517-1919