Multivariate Testing and Conversion Tweaking
Back from lunch. That was one yummy chocolate chip cookie! This time we’re at the Multivariate Testing and Conversion Tweaking session with panelists Tom Leung, Glenn Alsup, Philippe Lang and Rand Fishkin. Moderating will be Gillian Muessig, aka Rand’s mommy.
Gillian isn’t wasting any time. She introduces Glenn. Hi Glenn.
Glenn talks about qualitative vs. quantitative data
Qualitative researchers: Reject the idea that social sciences can be studied like natural or physical sciences. They feel that human behavior is always bound to the context in which it occurs. And it’s usually personal and subjective.
Quantitative researches: Argue that both the natural and social sciences are testable and confirmable theories. They also do lots of other stuff I didn’t have time to get down. Sorry. There was a lot of text on that one slide.
Glenn says qualitative is Gods Fuel:
They look at:
Effects on Goals
It’s effective to look at this as a sales funnel. Identify all your drivers – Offline, Online, Explicit and Extract. Then take the data and group attributes based on filters.
Look at the stages in the funnel. Weight them by the value of that event. Downloading a white paper is more valuable than looking at a banner. Identify when the action took place. When you convert, those drivers are calculated in different ways. You want to increase conversions by identifying the right mix of marketing drivers.
Tom from Google is next to talk about smart testing. I have a feeling he’s going to talk about Google’s Web site optimizer.
Ooo, I’m right! Tom asks: What do you do when the visitor lands on your site and what can you do to get them to convert?
Driving traffic is just the beginning. You invest in search engine optimization and SEM resources for 100 percent of visitors. Your pages may lose more than half their visitors in seconds. Most that do stay choose not convert. (Industry average for converting customers is 2-3 percent.) Why bother bringing more visitors to a site that convert poorly?
Evolve with continuous improvement. Drive the right traffic to your site. Measure & analyze site activity. Test changes and implement winners. Repeat steps 1-3 until conversation rate is 100 percent.
How Testing Works
Visitors arrive on your site. They’re shown a random version of your site and the testing tool will tell you what percentage of users converted based on what version of your page.
He shows an example of a page Google uses their Web site optimizer on. It’s the home page for Picasa. In the first version of the page, it uses "free" a lot and is very picture and action oriented. They found that the second page, on that is cleaner, includes a Try button and states a clear value proposition converted 30 percent more.
How do you set up a test?
Best Practices in testing
- Test a small number of variations: Rule of thumb is less than 100 conversions per combination.
- Test big changes: If you can’t see difference between two combos in 8 seconds, visitors probably won’t either.
- Consider early indicators if you don’t have enough conversions: If you’re selling a $100k software package or a small business with modest volume, optimize for conversion indicators such as request info, view product details, etc.
- Don’t jump to conclusions: Less than 2 weeks is no good, focus on absolute conversion difference, don’t get too excited by sliver of green.
More Testing Ideas:
Conversion Cocktail: Headline/Image/Call to action. Those are the best three sections to test.
Inspirational or fact-based pitch
Philippe is up next.
Analytics provides you a ton of data about visitor behavior on your site. This information is very valuable but it can be hard to interpret. Your customers also provide you valuable information when you talk to them.
What if you could see in real time what your visitors are doing on your Web site? Start a dialogue with those visitors. Listen to them. Understand what they are doing and why. It’s like a usability test in the real world.
Things to look for when you monitor your Web site: Visitor referrer information like search engine used, marketing campaign, geolocation, and navigation behavior like page views time on page, shopping cart content, and shopping cart abandonment.
Merge the two sets of data (real time monitoring and feedback) by engaging specific visitors into a chat or leverage the customer support chat transcripts (what if you don’t offer a customer chat feature?). Try to see the patterns or trends that are coming out.
Ultimate Feedback Look
- Start by monitoring visitor on your site
- Identify visitors with unexpected behavior
- Engage visitors by listening to them
- Convert customer feedback into action
- Refine and optimize your Web site
Phillipe presents two case studies. The first is about a university offering an online degree.
Problem: High abandonment rate in the online inquiry form
Cause: Customers annoyed by specific question about the age of applicant. Customers abandoned the form instead of answering the question.
Solution: Removed the question
Result: 20 percent increase in form completion.
Online retailed focused on outdoor gear
Problem: Low conversion rate on sunglasses
Cause: Customers were confused by the sizing chart for the sunglass
Solution: Redesigned the sizing section
Result: 25 percent increase in conversion rate and 20 percent decrease in product returns.
The morale of the story: If you know what’s wrong, you can fix it!
Testing helps companies increase conversion rates, improve the sales process, increase average order value and builds relationships with customers.
Last but not least is Rand Fishkin.
Rand opens by talking about the landing page contest SEOmoz ran a few months back. They let blog readers submit landing pages to them. They got lots of submissions and ended up testing 10 of the 40 entries they received. Rand shares the stats for each page. It’s exciting. Ultimately, the page where you had to scroll forever to get to the end of the page converted the best. That’s why you see so many of those pages.
Once the contest was over, the SEOmoz crew tweaked the winning page to make it convert even better.
- If you think it will work, it probably won’t
- Landing page design is not universal
- Testing is the only way to get better
- Doubling your conversion rate is far easier than doubling your traffic.
Question & Answer
How much did you pay for Offermatica?
He thinks its $10K a month and up.
Rand, it looks like you tested a handful of completely different pages. Can you tell us how Offermatica fit into that? What page elements did each have that helped it convert better?
Rand doesn’t actually answer the question and just says that they were doing A/B testing (testing two different pages), not multivariate testing where you test different images or different headers or different layouts.
I always thought that I should change pages incrementally so that Google wouldn’t think it was a different site. Is that right or am I paranoid?
Rand: Basically conversion testing is a legal form of cloaking. What you’re really doing is showing a bunch of different versions to people to find the best one, but you’re only showing one version to the search engines.
Glenn: Typically you see smaller changes.
Multivariate testing about pleasing more of the people more of the time. When visitors come back to the page, is there any way to ensure that they see a different version of the page?
Tom: When the experiment is running, users will continue to see the same version of the page they saw the first time to protect the integrity of the experiment.