How Much Testing Is Too Much?
I was reading Search Engine Land yesterday (as I’m known to do) and came across Jon Miller’s article how about much landing page testing is too much. Jon offers up a highly fancy-looking equation that you can use to determine when you’re done testing your landing page. It made me giggle. A landing page test calculator? Really? There are three very easy ways to know if you’ve done enough testing and none of them include a calculator:
- You’ve reached the burn rate where the test findings will breed only minimal results.
- When there are other, more critical elements or pages to test.
- When the person in charge of your testing process threatens to quit if you make them run one more darn A/B experiment.
See how easy that was? And no formula involved!
Truthfully, whether you’re doing A/B Testing where you test one element or a time or you’re using a Multivariate testing tool like Google Website Optimizer, you can test forever. It doesn’t matter if you’re talking landing pages, new site elements or better copy on interior pages, the iterations can go on forever. There will always be a better way to word something, a better place to put that image, or a new angle that your customers may die for. But at some point, the benefit of conducting the test looks like this:
(I have mad MS Paint skills, eh?)
Most search marketers will know intuitively when they’ve reached the burn rate for a particular test. It’s when you implement the change and find it didn’t improve the user experience on your Web site or result in a lower cost of conversion. When you get there, move on and start testing another page or site element. There’s no use having one perfect page if the rest of your site is falling by the wayside.
When you’re doing your testing, conduct it in one week increments so you take into account any day of week or time of day effects. Your test may run several weeks long or a even solid month depending on what you’re testing. You shouldn’t end your test until you have complete confidence in the answer it’s giving you. Sometimes that means running something more than once.
Make you sure you have enough data when you’re conducting your tests and don’t ignore your baseline measurements. Your customers may have preferred the “old” way to the new one. Don’t be afraid to admit that to yourself.
Once you’re confident with your results and reach that elusive burn rate, move on to another page element and start the process again. Search engine optimization isn’t about perfection; it’s about balancing your “good enoughs”.
But don’t forget to come back in the future. Once you think you have a page optimized, it doesn’t mean you’ll have it optimized forever. Your users and what they’re looking for change so frequently that it’s important to go back and restart old tests. You can step away, but you have to come back and retest and re-measure things like clicks, engagement indicators, user behavior, etc.