BACK TO BASICS: The Search Engine Family Tree

by Susan Esparza, November 2006

Information doesn’t exist in a vacuum and neither do search engine
indexes. In order to get the most out of your search engine marketing
efforts, you need to know how each of the engines interact with each
other. Useful for both organic search engine optimization as well as
pay per click advertising, Bruce Clay’s Search Engine Relationship
Chart® is a map to the ecosystem of the engines, ad networks and
important directories on the Web.

Though most people look at the chart and inherently understand the
value, they’re often not certain how to use it in order to get the most
out of it. Some of the common questions about the chart are related to
the position of the circles on the page. Is Google on top because
they’re the most important? DMOZ is right in the middle; does that make
them the most important? What exactly are we trying to say by putting
MSN way at the bottom? In all cases the answer is simply: it fit
better.

Bruce Clay first published his Search Engine Relationship Chart® in
2000. Back then, there were more major players in the search game and
things were, to say the least, somewhat cluttered. The chart had 26
companies on it, everyone from Yahoo! to Magellan to that upstart
company Google. Fifteen of those companies took their primary results
from their own index; five of those supplied secondary results to other
engines. Without a roadmap, it was an impossible task to keep it all
straight. But over the years, things started to change. What was once a
cluttered mess became a tidy interplay of a select group of companies.

The organization of the chart mirrors the organization of the search
industry. Where once there were many major players, today there are
only a few. The power of search is growing by leaps and bounds. With
mass adoption by consumers and increased interested by advertisers,
search engine marketing is the most popular online marketing strategy
today and estimates predict only more growth in the future. Already
eMarketer estimates that $7.1 billion is spent in the search marketing
space, dollars that translate into increased competition from all
quarters.

But while the power of search grows, it’s also consolidating in the
hands of fewer companies. No longer do search experts talk about being
in all the top engines. At most, they talk about only four, and in many
cases, just one: Google.

The most recent version of the chart is a clean white document with
just two clusters on it, Google above and Yahoo below. From these
center hubs, several other engines receive primary, secondary and
directory results. Paid results from AdWords and Yahoo! Search Marketing
are served to ad partners.

The charts contains several layers of information, first is the
question of where the results come from. Though the engines may have
different algorithms, many smaller engines have contracts with the
larger engines to supply the actual index. Google supplies AOL, for
example. When a search engine has its own index, the circle on the
chart is bordered in red. Should they be sharing those results out, a
red arrow points from the primary engine to the receiving engine.
Additionally, some engines get results from other engines merely to
supplement their own. These connections are shown in blue and indicate
that while the receiving engine may have their own index, they are not
solely reliant on it.

Directory results are also shown on the chart via green arrows.
Directories like DMOZ and the Yahoo directory are considered quality
links and are nice things to have. Because they supply listings to the
search engines, your site is likely to appear in the indexes. However,
they are not the be all and end all of the search engine game, they do
not guarantee rankings and it is not necessary to beg, borrow and buy
your way into them. Additionally, the directories often have title and
descriptions that are vastly different (and usually not as well
crafted) as your own optimized Meta data. If you are showing up in an
engine with your DMOZ tag and you would rather see the on-page
information, the search engines have added support for the Meta robots
command “noodp” which will instruct the engines to use the on page Meta
data. It does not negate the value of the directory link, just prevents
the directory’s title and description for your site from being shown.
Yahoo is considering adding similar support for the Yahoo Directory. If
you are conducting your own search engine optimization efforts, it is
highly recommended that you employ this command on your pages.

The last arrow on the chart is yellow. Yellow arrows indicate paid
results, like AdWords or Yahoo! Search Marketing. Check out these
distribution networks carefully because your PPC campaign has the option
of appearing on these sites. If you are using AdWords, you can choose
to appear either only on Google’s search results or all Google’s search
partners (AOL, Ask, AdSense for Search) and content network (AdSense
for Publishers). For Yahoo, your options are to appear on Yahoo and its
search partners’ results pages or on its content network (Yahoo
Publishing Network).

The online version of the Search Engine Relationship Chart® is a Flash document and incorporates several other features.

You can click on any company and be taken to information about that
engine, including information about search engine submission policies,
how long they’ve been in business, what kind of technology they are
using and how to get into the index via either free or paid methods.
The page also summarizes a few tips for best results and lists the
content received or provided by other companies. Links on the
information page will take you directly to the submission page for the
listed engine. As always, it is good to remember that submission is not a
guaranteed method of being indexed.

In order to make the relationships between companies even more
transparent, hover your mouse over a company’s circle without clicking
on it. The other unconnected companies will fade out, leaving behind
the content partners. This system works both ways whether the search
company or directory is feeding out information, as with Google below,
or merely pulling in data from elsewhere, as with Netscape.

With the growth of the search industry and the narrowing of the
field, it is more important than ever to know where and how search
engines are connected to each other and how you can achieve the most
visibility for the least cost. The SERC won’t do your optimization for
you but it’s a valuable tool in guiding your efforts. Conversion and
quality are of paramount importance in your search engine marketing
campaigns.