Study performs Coke vs. Pepsi Test on Engines

According to a new study out of France, today’s search engines are more interested in banging out fancy new features than improving search relevance and customer satisfaction.

French professor Jean Veronis and his students used three American search engines: Google, Yahoo! and MSN and three French search engines: Exalead, Voila and to compare the results from fourteen different queries. The search topics were selected to reflect a broad range of subject areas, including animals, cinema and current events. Participants were then able to construct their own search queries, using their own format (with or without quotation marks, querying one or multiple terms, etc.) to perform a ‘blind test’ on the search engines. To keep the identity of the engines a mystery, the students never performed an actual search. Instead, they wrote down their queries and Veronis performed all the searches himself. The search-URL result pairs where then given to the students to evaluate. After receiving the results, the students were asked to rate the engines in terms of relevance, scoring them on a scale of zero to five, with five being the highest. A passing grade was considered to be 2.5.

Result: All of the engines failed.

Google and Yahoo tied for ‘first place’ in terms of relevancy (MSN came in third), with a rating of 2.3, but when it came to user satisfaction, the engines failed universally with flying colors. Many engines, including Google with 28.6 percent and Yahoo with 27.7 percent, had links that scored a pathetic 0, classifying them as ‘totally useless’. Nice.

The results hint at some interesting conclusions. First, there is no difference between Google and Yahoo! in terms of search relevancy. If the study is accurate, users deem them 100 percent identical. So why is Google the hands-down favorite? Is it merely a result of their branding? A user is loyal to their engine because they are also using their email system, desktop, toolbar and exploring Google Earth? Second, are the search engines placing too much energy in their extras department instead of refining their search engines? Would a true, search-only engine truly satisfy user needs without the extra features? We’ll have to wait for someone to design one to find out.

The full study is available in PDF format for those interested.

(Hat-tip to Andy Beal)

Lisa Barone is a writer, content marketer & VP of strategy at Overit Media. She's also a very active Twitterer, much to the dismay of the rest of the world.

See Lisa's author page for links to connect on social media.

Comments (0)
Filed under: SEO
Still on the hunt for actionable tips and insights? Each of these recent SEO posts is better than the last!
Bruce Clay on May 8, 2024
The Always Up-to-Date SEO Checklist
Bruce Clay on April 4, 2024
What Makes an SEO Program Successful?
Bruce Clay on April 3, 2024
How Can I Improve My Website Rankings Through SEO?

Comments are closed

Serving North America based in the Los Angeles Metropolitan Area
Bruce Clay, Inc. | PO Box 1338 | Moorpark CA, 93020
Voice: 1-805-517-1900 | Toll Free: 1-866-517-1900 | Fax: 1-805-517-1919