BACK TO BASICS: The Big Three Webmaster Tools Roundup

ACK TO BASICS: The Big Three Webmaster Tools Roundup

By Virginia Nussey, August 15, 2008

Search engine optimization professionals are part artist, part
scientist. An artist’s inventiveness comes into play when finessing
modifications and creatively addressing site visibility. The scientist’s
skill is required to perform tests, analyze stats and manage technical
aspects. Like any scientist, for an SEO to do her job effectively, she
needs to be armed with a powerful arsenal of tools. Fortunately for
SEOs, there are a number of tools at their disposal, including those
provided by the three major search engines.

This overview of the webmaster tools offered by Google, Yahoo and
Live Search describes the diagnostic, statistic gathering and submission
tools available to webmasters and SEOs who sign up for accounts and
verify their site control. It’s worth noting that all three sets of
tools allow the data to be downloaded into spreadsheet applications.

Overview and Diagnostic Tools

These tools report when the page was last crawled and any Web crawl
errors encountered by the search engine spider. As webmasters and SEOs
will want the crawling of a site to be as comprehensive as possible,
awareness about crawl errors is key to addressing and remedying any
issues that may cause incomplete crawling.

Google Webmaster Tools

After you sign in and select your site in the dashboard, you will be
brought to the Overview page. Here you’ll see listed the date of the
last home page crawl, the index status of the site and the number of Web
crawl errors. Crawl error information is broken down as follows:

Errors for URLs in Sitemaps
HTTP errors
Pages not found
URLs not followed
URLs restricted by robots.txt
URLs timed out
Unreachable URLs

Related to crawl statistics, a robots.txt file will instruct spiders
not to crawl or index certain pages. The Analyze robots.txt tool can be
reached through the “Tools” tab and can help webmasters ensure they are
not blocking pages that they do want crawled and indexed. Under the
“Tools” tab you can also set the crawl rate, remove URLs from the index
and set your preferred domain (i.e., indicate a preference for the www
version over the non-www version of the site).

Yahoo! Site Explorer

As with Google, sign in and click “Explore” next to the site you want
to research. This will bring you to what will be referred to as the
site’s main Explore page for the remainder of this article. This page
provides the number of pages on the site that have been indexed, the
number of backlinks (or inlinks) to the site and any sub-domains of the
site. All indexed pages are listed, and, when hovered over, options to
see more details are available, such as the date the page was last
crawled.

Live Search Webmaster Tools

Again you will need to sign in and choose your site from the Site
List page. You will arrive at the Summary page that lists the date the
home page was last crawled, the number of pages indexed, a domain score,
any pages blocked by robots.txt and the last crawl date of the top five
pages. Clicking on the “Crawl Issues” tab will bring you to the
following reports, which can be additionally filtered by sub-domain or
sub-folder:

Pages not found
URLs restricted by robots.txt
Long dynamic URL
Unsupported content-type

Similar to Google, Live Search Webmaster Tools includes a robots.txt
validator that reports any incompatibility issues with the file and
checks which pages are being blocked for crawling by MSNBot. The
validator can be reached through the left-hand navigation on the main
tools page of Live Search Webmaster Center.

Site Performance, Link and Statistic Gathering Tools

Top search queries, crawl stats, index stats and internal and
external links are provided by the tools that fall under this category.

Google Webmaster Tools

Clicking on the “Statistics” tab will bring you to an overview page, from which you can choose to see reports on:

Top search queries: Results are based on the last week. The top 20
search queries for which your site appeared and the top 20 queries that
users clicked on to reach your site are listed along with the ranking of
the page as it was listed in the search results.
What Googlebot sees: Available statistics in this category include
the common words and phrases in the site’s content and external anchor
text to your site.
Crawl stats: This category provides the Googlebot activity on your
site for the past 90 days. You will be informed of the number of pages
crawled per day, the PageRank of your pages and the page with the
highest PageRank.
Index stats: This tool will provide you with information similar to what you can find using Google Advanced Search. Examples of results for the site:, link:, cache:, info:, and related: operators are provided.
Subscriber stats: Sites that publish feeds will be provided with the
number of users subscribed to the feed using iGoogle, Google Reader and
Orkut.

By clicking on the “Links” tab, you will see an overview of the link
statistics available: pages with external links, pages with internal
links, and pages that may be used as Sitelinks in Google results.

Yahoo! Site Explorer

After selecting “Explore” next to your site URL on the My Sites page,
click “Inlinks” for a list of the pages linking to your site. Along
with offering a comprehensive list of inlinks from all pages on the Web,
links from within the domain or a sub-domain can be filtered out of the
results. Inlinks can also be limited to a single URL or provided for
the whole site. By filtering out the backlinks from your own domain you
can view inbound links coming from external sources.

Unique to the three sets of tools is Yahoo Site Explorer’s ability to
view the backlinks going to URLs other than your own. While information
on the inlink data of your own site may prove useful, the ability to
gather parallel information for competitors is equally valuable.

Live Search Webmaster Tools

Reports on backlinks, outbound links and keyword research can all be
collected with Live Search’s set of tools. After selecting the
“Backlinks” tab, all Web pages linking to your site will be listed.
Backlink results can be filtered to include or exclude links from
top-level domains, sub-domains or sub-folders. Clicking on the “Outbound
Links” tab serves up a similar list of the pages your site links to,
which can again be filtered to include or exclude links from pages of a
specific TLD, sub-domain or sub-folder.

Finally, analyze the performance of your site for certain keywords by
clicking on the “Keywords” tab. From here, enter a keyword into the
search field to see what pages of your site Live Search considered most
relevant for queries using that keyword.

Submission Tools

By way of Sitemap, site feed and Web page submission, the major
search engines help webmasters and SEOs share information about their
site for the potential of more complete indexing.

Google Webmaster Tools

From the “Sitemaps” tab you can submit your XML Sitemap feed to
Google. As mentioned earlier, go to the “Tools” tab to submit your
preferred domain or remove URLs from the index.

Yahoo! Site Explorer

Click on “Submit Webpage or Site Feed” from the site’s main Explore
page to submit the URL of a Web site, Web page or site feed that you
would like to be crawled by Yahoo!

Live Search Webmaster Tools

Under the “Sitemaps” tab, submit your XML Sitemap feed to Live
Search. The search engine also offers a ping service to notify MSNBot of
changes to an existing Sitemap or the addition of a new one; simply
enter http://webmaster.live.com/webmaster/ping.aspx?siteMap=[your
sitemap web address] in your browser.

Conclusion

The various webmaster tools provided by the three major search
engines show how the search engine views the site, aid in diagnosing
problems and allow webmasters to share info in an effort to support more
complete indexing of a site. Just as search gives users a tool to
conduct research, gather information and purchase products and services,
the search engines offer the Web’s content providers, webmasters and
SEOs the tools necessary to efficiently direct the power of search.