Get Free Quote

ACK TO BASICS: The Big Three Webmaster Tools Roundup

By Virginia Nussey, August 15, 2008

Search engine optimization professionals are part artist, part scientist. An artist's inventiveness comes into play when finessing modifications and creatively addressing site visibility. The scientist's skill is required to perform tests, analyze stats and manage technical aspects. Like any scientist, for an SEO to do her job effectively, she needs to be armed with a powerful arsenal of tools. Fortunately for SEOs, there are a number of tools at their disposal, including those provided by the three major search engines.

This overview of the webmaster tools offered by Google, Yahoo and Live Search describes the diagnostic, statistic gathering and submission tools available to webmasters and SEOs who sign up for accounts and verify their site control. It's worth noting that all three sets of tools allow the data to be downloaded into spreadsheet applications.

Overview and Diagnostic Tools

These tools report when the page was last crawled and any Web crawl errors encountered by the search engine spider. As webmasters and SEOs will want the crawling of a site to be as comprehensive as possible, awareness about crawl errors is key to addressing and remedying any issues that may cause incomplete crawling.

Google Webmaster Tools

After you sign in and select your site in the dashboard, you will be brought to the Overview page. Here you'll see listed the date of the last home page crawl, the index status of the site and the number of Web crawl errors. Crawl error information is broken down as follows:

  • Errors for URLs in Sitemaps
  • HTTP errors
  • Pages not found
  • URLs not followed
  • URLs restricted by robots.txt
  • URLs timed out
  • Unreachable URLs

Related to crawl statistics, a robots.txt file will instruct spiders not to crawl or index certain pages. The Analyze robots.txt tool can be reached through the "Tools" tab and can help webmasters ensure they are not blocking pages that they do want crawled and indexed. Under the "Tools" tab you can also set the crawl rate, remove URLs from the index and set your preferred domain (i.e., indicate a preference for the www version over the non-www version of the site).

Yahoo! Site Explorer

As with Google, sign in and click "Explore" next to the site you want to research. This will bring you to what will be referred to as the site's main Explore page for the remainder of this article. This page provides the number of pages on the site that have been indexed, the number of backlinks (or inlinks) to the site and any sub-domains of the site. All indexed pages are listed, and, when hovered over, options to see more details are available, such as the date the page was last crawled.

Live Search Webmaster Tools

Again you will need to sign in and choose your site from the Site List page. You will arrive at the Summary page that lists the date the home page was last crawled, the number of pages indexed, a domain score, any pages blocked by robots.txt and the last crawl date of the top five pages. Clicking on the "Crawl Issues" tab will bring you to the following reports, which can be additionally filtered by sub-domain or sub-folder:

  • Pages not found
  • URLs restricted by robots.txt
  • Long dynamic URL
  • Unsupported content-type

Similar to Google, Live Search Webmaster Tools includes a robots.txt validator that reports any incompatibility issues with the file and checks which pages are being blocked for crawling by MSNBot. The validator can be reached through the left-hand navigation on the main tools page of Live Search Webmaster Center.

Site Performance, Link and Statistic Gathering Tools

Top search queries, crawl stats, index stats and internal and external links are provided by the tools that fall under this category.

Google Webmaster Tools

Clicking on the "Statistics" tab will bring you to an overview page, from which you can choose to see reports on:

  • Top search queries: Results are based on the last week. The top 20 search queries for which your site appeared and the top 20 queries that users clicked on to reach your site are listed along with the ranking of the page as it was listed in the search results.
  • What Googlebot sees: Available statistics in this category include the common words and phrases in the site's content and external anchor text to your site.
  • Crawl stats: This category provides the Googlebot activity on your site for the past 90 days. You will be informed of the number of pages crawled per day, the PageRank of your pages and the page with the highest PageRank.
  • Index stats: This tool will provide you with information similar to what you can find using Google Advanced Search. Examples of results for the site:, link:, cache:, info:, and related: operators are provided.
  • Subscriber stats: Sites that publish feeds will be provided with the number of users subscribed to the feed using iGoogle, Google Reader and Orkut.

By clicking on the "Links" tab, you will see an overview of the link statistics available: pages with external links, pages with internal links, and pages that may be used as Sitelinks in Google results.

Yahoo! Site Explorer

After selecting "Explore" next to your site URL on the My Sites page, click "Inlinks" for a list of the pages linking to your site. Along with offering a comprehensive list of inlinks from all pages on the Web, links from within the domain or a sub-domain can be filtered out of the results. Inlinks can also be limited to a single URL or provided for the whole site. By filtering out the backlinks from your own domain you can view inbound links coming from external sources.

Unique to the three sets of tools is Yahoo Site Explorer's ability to view the backlinks going to URLs other than your own. While information on the inlink data of your own site may prove useful, the ability to gather parallel information for competitors is equally valuable.

Live Search Webmaster Tools

Reports on backlinks, outbound links and keyword research can all be collected with Live Search's set of tools. After selecting the "Backlinks" tab, all Web pages linking to your site will be listed. Backlink results can be filtered to include or exclude links from top-level domains, sub-domains or sub-folders. Clicking on the "Outbound Links" tab serves up a similar list of the pages your site links to, which can again be filtered to include or exclude links from pages of a specific TLD, sub-domain or sub-folder.

Finally, analyze the performance of your site for certain keywords by clicking on the "Keywords" tab. From here, enter a keyword into the search field to see what pages of your site Live Search considered most relevant for queries using that keyword.

Submission Tools

By way of Sitemap, site feed and Web page submission, the major search engines help webmasters and SEOs share information about their site for the potential of more complete indexing.

Google Webmaster Tools

From the "Sitemaps" tab you can submit your XML Sitemap feed to Google. As mentioned earlier, go to the "Tools" tab to submit your preferred domain or remove URLs from the index.

Yahoo! Site Explorer

Click on "Submit Webpage or Site Feed" from the site's main Explore page to submit the URL of a Web site, Web page or site feed that you would like to be crawled by Yahoo!

Live Search Webmaster Tools

Under the "Sitemaps" tab, submit your XML Sitemap feed to Live Search. The search engine also offers a ping service to notify MSNBot of changes to an existing Sitemap or the addition of a new one; simply enter http://webmaster.live.com/webmaster/ping.aspx?siteMap=[your sitemap web address] in your browser.

Conclusion

The various webmaster tools provided by the three major search engines show how the search engine views the site, aid in diagnosing problems and allow webmasters to share info in an effort to support more complete indexing of a site. Just as search gives users a tool to conduct research, gather information and purchase products and services, the search engines offer the Web's content providers, webmasters and SEOs the tools necessary to efficiently direct the power of search.


For permission to reprint or reuse any materials, please contact us. To learn more about our authors, please visit the Bruce Clay Authors page. Copyright 2008 Bruce Clay, Inc.