Priyank Garg on Bot Herding

Next up on the SMX Sydney Technical SEO Track is "Bot Herding" by Priyank Garg, Director, Product Management at Yahoo! Web Search

Priyank Garg is Director of Product Management at Yahoo! India. He is leading Product Management efforts for the Yahoo! Search team based out of Bangalore. [Aiden, our local Yahoo search universe master, made Garg’s visit happen for SMX this year so a big thanks for him for that effort. – Kate] Priyank has published multiple papers and filed for more than 12 patents over his professional career.

Gard explains that Robots.txt is the baseline must do for “bot herding” (telling search engines what to do when visiting a site. Robots.txt allows publishers to tell search engine crawlers which content should be included in search results. The independent site answers a lot of the most frequently asked robots.txt related questions

Priyank suggests that all webmasters sign up for Yahoo Site Explorer, it’s a useful tool to let you privately see how Yahoo is indexing your content.

More information at and an explanation of the Yahoo Dynamic URL Rewriting content tool

See Bruce's author page for links to connect on social media.

Comments (0)
Filed under: SEO
Still on the hunt for actionable tips and insights? Each of these recent SEO posts is better than the last!
Bruce Clay on May 8, 2024
The Always Up-to-Date SEO Checklist
Bruce Clay on April 4, 2024
What Makes an SEO Program Successful?
Bruce Clay on April 3, 2024
How Can I Improve My Website Rankings Through SEO?


Your email address will not be published. Required fields are marked *

Serving North America based in the Los Angeles Metropolitan Area
Bruce Clay, Inc. | PO Box 1338 | Moorpark CA, 93020
Voice: 1-805-517-1900 | Toll Free: 1-866-517-1900 | Fax: 1-805-517-1919