As you will find, User Centric consultants frequently publish papers in peer reviewed journals and publications, as well as present data at professional associations domestic and international. We believe that we run our studies in a sound and rigorous manner that is in line with the scientific method.

That said, we acknowledge some may argue about the sample size. Yes, we could have run more participants, but that only increases power. Would 30 have been better? 40? 100? While sample size is really a question of power, a majority believe sample size comes down to "this smells large enough". But, the problem is that this assumes that the speaker says that these results or scores can be generalized to the population. This is not a question of confidence intervals, but a question of difference. Bing and Google’s scores were significantly different. This is an inferential statistics question whether we can infer from the data that the difference is significant. If we were to run the study 20 times what is the likelihood that we could have found this difference by chance alone (p value of .05).

User Centric did not say that the scores were generalizable, but that the difference between Bing and Google were significant. We are able to make that statement and yes, sample size played a role in the statistical analysis performed.

Now, we certainly welcome discussions on new research questions or even if we think the result is due to the novelty of the Bing design and whether this will fade away. Valid points where we could run a study with experienced users and other permutations. Please forward topics for research questions to me directly.

The result is what it is. Let’s talk about the implications, habituation, usefulness of features, whether the options were copied from Google and improved or made worse.