Priyank Garg on Bot Herding
Next up on the SMX Sydney Technical SEO Track is "Bot Herding" by Priyank Garg, Director, Product Management at Yahoo! Web Search
Priyank Garg is Director of Product Management at Yahoo! India. He is leading Product Management efforts for the Yahoo! Search team based out of Bangalore. [Aiden, our local Yahoo search universe master, made Garg's visit happen for SMX this year so a big thanks for him for that effort. - Kate] Priyank has published multiple papers and filed for more than 12 patents over his professional career.
Gard explains that Robots.txt is the baseline must do for “bot herding” (telling search engines what to do when visiting a site. Robots.txt allows publishers to tell search engine crawlers which content should be included in search results. The independent site http://www.robotstxt.org/faq.html answers a lot of the most frequently asked robots.txt related questions
Priyank suggests that all webmasters sign up for Yahoo Site Explorer, it’s a useful tool to let you privately see how Yahoo is indexing your content.
More information at http://www.seomoz.org/blog/how-much-do-you-know-about-use-yahoos-site-explorer-and-link-research and an explanation of the Yahoo Dynamic URL Rewriting content tool http://searchengineland.com/yahoo-site-explorer-adds-dynamic-url-rewriting-tool-11991