“How many test respondents do we need for a good user test?”
We get asked this question all of the time. Noted User Researcher Jakob Nielsen already shares a great answer to this question on his blog¹, so we will just summarize his thoughts here.
Optimizely has a great product for A/B testing single variable changes in production. For example, it is easy to test things like: does changing this button size impact conversion rates?
But what if you want to test an entirely different design layout? Do you have to code it up and release the new design to production to get feedback?
The short answer is NO.
One of those limitations is that humans can be unintentionally persuaded to give you the answer you want to hear. Thus any test creator must take great care to not create leading questions.
Below are two questions that could be asked during a user test. Can you guess which one is a good question and which one is not?