SMX East 2011: Best Practices for Paid Search Testing
Sid Shah will start us off. Public service announcement: Sex ed is wrong. Babies are delivered by storks. Let’s prove this with data:
The cross section analysis of population of stork and babies shows hat they follow. Is he mad? In the biz world, biz make the wrong conclusions based on inaccurate analysis all the time. He’s going to talk about measuring, testing and analyzing in a way that will
- Formal testing methodology.
- Mathematical intuition behind testing and analysis.
- Explain how to do experiments without spending time and money (just math).
Analyzing results is most often done wrong.
A good hypothesis is about specificity: adding “official site” to ad copy improves ROI.
Better: adding “official site” to ad copy of branded terms in search campaign improves ROI.
Best: adding “official site” to ad copy of branded terms in search campaign of my auto business improves ROI.
Hypothesis, performance metrics, external factors, testing parameters, constraints like HiPPOs.
The highest paid person’s opinion can be a constraint you need to work around. It’s best to have as much data as possible and hope their reasonable.
It’s nice to hope your results will be nice and clean and clear. But real data is more difficult to read, and it can be hard to say.
He’s showing graphs that show that the average, or mean, isn’t always the data that matters. The distribution may be very revealing for success or failure as well.
Hypothesis testing: with the collected data, can I tel with statistical confidence if the pre and post results are from statistically different populations? P value shows the probability that the data is right.
Models and Simulations
They’re quick, cheap, forces a review of assumptions and answers “what if” quesitons almost instantly.
One can simulate the performance of a search campaign using sophisticated mathematics. Simulations can be used as a planning tool in any area where it’s expensive to run an experiment.
- Be data-driven
- Compute with care
- Simulate experiments
- Constantly re-assess your assumptions
- Experiment as often as possible
Shannon Anderson is next. Keep in mind:
- Build best practices, then rebuild them.
- Don’t test for the sake of testing – have a plan and purpose.
- Remember, it’s marketing! Speak to the end user.
Her testing process:
- Purpose of test, testing variables, timing, launch, analysis, use the data, repeat
- Establish a budget for the test up front and be okay with the fact you may not get any revenue out of the test and may lose the money with no results. You have to let the test run all the way through.
Example: ABC Travel
Purpose of test: determine best practices to launch during peak booking season
Testing variables: purchase cycle, call to action, headline, targeting, day parting
- Using “official site” in the headline
- Using registered trademark symbol
They found a true improvement with the trademark symbol.
Call to action testing:
- $500 off urgency with offer end date vs. “Limited time!”
Saw worse results with end date in the test. It may have been too aggressive to have the end date.
- Used “escape the cold” language in copy of tests in Chicago
- Conversions happened more in evenings
Click Assist Report helped understand the data, especially keywords that aren’t meant to convert, but help to assist conversions further down the line.
Chris Goward will talk about how to get max conversions once traffic is on your site. These methods work for all industries in all countries.
6 conversion factors that influence LIFT (landing page influence function for tests):
- Value proposition
When hypothesizing for tests, turn problems into hypotheses. Make sure you’re addressing problems with your tests, not “what color button?”
- Solves conversion problems
- Builds on marketing insights
Funnel Experiment Map: a document that outlines the analysis, hypotheses, variations, the test plan before going to the design department.
Test plan includes:
- Test structure: A/B/n, MVT, alternative flow
- Traffic sources and volume estimates
- Calculation of number of variations
- Wire frames
Graphic design and copy: Make the wire frame look like the control and reflect the strategy of the test.
- Do not use the before and after method. Use a controlled testing method.
- Do not start with segmentation, targeting or multivariate. Start with A/B testing.
- Do use a controlled testing tool with statistical significance and 95% confidence.