Optimizely has a great product for A/B testing single variable changes in production. For example, it is easy to test things like: does changing this button size impact conversion rates?

But what if you want to test an entirely different design layout? Do you have to code it up and release the new design to production to get feedback?

The short answer is NO.


Here is how to run user tests on design layouts:


Step 1: Set a Baseline:

Run a test on your current UI. Test the same task and ask the same questions that you want to know the answers to for the new design versions.


Step 2: Test each design independently:

A common mistake when testing different design options is to show multiple versions to the same person and ask, “which version did you like the best” or something to that effect. This is a bad testing process because people are not great at recalling variation details after the fact.

The right approach is to test each option individually using the same tasks and questions and then compare results.

In my opinion, it is best to test each option on different people to keep fresh eyes on each version. However, it’s acceptable to test each version on the same person, so long as each variation is a unique test with the same tasks and questions.

Do This (for each variation)

Do NOT Do This

When Testing, be sure to measure:

  1. Qualitative responses.
    • Look for opinions on ease of use, look and feel, etc.
  2. Task completions
    • Does the conversion rate or other KPIs change?
  3. Engagement.
    • Measure how long the respondent stays on your prototype – pages and time. Are they really digging into your site and poking around unprompted? or are they getting lost because the site is confusing to them?


You can learn a ton by testing design images at the prototype stage. Just don’t show different versions to the same person in the same test and ask them to rank or give relative feedback. Run user tests on each design option individually, then compare qualitative responses, task completes and engagement stats. Tweet this