How A/B testing with Prototypes Can Save Time & Money

Our founder Steven Cohn and Alpha UX’s Nis Frome co-authored a piece in UX Magazine on How Split Testing Validates New Product Concepts without Code.

“Product managers already know that user data and behavior is a far better and more consistent predictor than hunches about the potential of a product. But while split testing prototypes can be one of the most effective weapons in the product management arsenal for answering critical product questions, it’s often underutilized.

Split testing, or “A/B testing,” is a popular marketing tactic for optimizing collateral and improving conversions. Instead of rolling out testable variations of products that are much more time-intensive and costly than landing pages, product managers simply need to be lean like marketers and split test prototypes. And this vehicle for generating feedback is the critical difference…”


How to Get Lean

I have spoken with a lot of organizations that want to bring Lean to their product development process.

Below are two transformation approaches that produce results:

  • Approach 1 is a way to start doing Lean without major process or organizational change
  • Approach 2 is how to re-organize your company to a Lean process.

Approach 1: Becoming Lean-ish

This model is for teams that aren’t quite ready for complete organizational change. This model is perfect for a head of product or a head of UX that still wants to use Lean techniques, but doesn’t want to waste energy fighting her company’s inertia.

If this is you, the good news is you can still make a major impact with a smaller, more controlled approach. Here’s how:

Lean is all about reducing waste by treating all product or feature ideas as hypotheses that need to be validated before building. That means, tasking your PMs and UX Designers with running rapid experiments to validate demand before the sprint planning meetings. We have posted before a simple technique on how to test demand with a prototype. It’s not brain surgery. It just requires discipline and the right tools.

If you want to do this, you need to give your UX Designers and PMs the time to run these experiments and interpret the results before they determine what goes into the next sprint. I suggest building your process so that the UX team is always at least 1 full sprint ahead of the sprint planning meeting. Also give them the power to say, “I know execs want this feature, but customers don’t!” Support them when the data is on their side. TWEET THIS

If your team doesn’t know how to run these experiments, have them watch the free videos in our Lean Customer Research Academy. Or you can reach out to me. I offer free workshops on how to run these experiments. If you are interested, feel free to email me.

Approach 2: Getting Religion

This model requires complete organizational change. As the section title suggests, you need your team to become true believers. Top to bottom of the organization needs to be completely committed. There will be some pain during the transition. And your resolve will be tested. Some employees will leave and others will be fired.

Step 1: I usually see this transformation start by having the entire executive team read “The Lean Startup”. Usually a group discussion follows. If everyone on the executive team agrees, then they have the next level of managers read the book and discuss it. The goal of these discussions is to gain consensus and start analyzing the changes that must be made.

While this is happening, the company often brings in a Lean process consultant. This person’s job is two-fold:

  1. Oversee and influence the new process and transformation plan.
  2. Train the trainers.

The consultant will work with the product team and senior executives on the engineering team to define the new process and build a detailed transition plan.

Typically you will pilot this new approach with a specific product team. The “product team” is comprised of a Product Manager, UX Designer, Engineers, and a User Researcher. This pilot will help work out the kinks and educate the leadership on how Lean will work within your specific company’s culture.

With the results of this pilot, you can start planning the company-wide rollout.

Obviously this is an oversimplification of what’s involved in major organizational change. It is really just a straw man of major milestones. Make no mistake this is major surgery for your company. Treat it as such. Expect old deadlines to be missed while people are re-learning how to build products. Expect some people to fight it. Expect some complainers. But once you start this, don’t turn back. It works. It really really works. So just push through.


Take the Pain Out of Feature Prioritization

“If we have data, let’s look at data. If all we have are opinions, let’s go with mine.”

Jim Barksdale, Former CEO of Netscape

The best part of my job is that I get to chat with leaders in Product and UX at a lot of great companies. So when I hear a common theme, I blog about it.

Lately, I have heard a lot of frustration around the often painful (and inefficient) process of feature prioritization. I have heard many complaints about a highly political process that is largely driven by anecdotes and debatable estimations.

“Our customer needs this feature to renew!” – Sales

“Our competitor just launched this feature and it is getting a lot of buzz!” – Marketing

“I have this great idea, it’s going to be huge!” – Sr. Exec

“This feature will have a bigger financial impact.” – Finance

When I hear this, I think of the great Jim Barksdale quote that I used to open this post.

We all have amazing customer demand analytics about the live product, yet the pre-build prioritization process is often void of customer demand data.

The good news is this is easy to fix.

I Already Know the Customers Want This! Let’s Just Build It

What customers say they will do, what customers say they did and what customers actually do are three different things.  Tweet this!

This inconvenient truth is what makes the feature prioritization process so difficult. This fact about customers leads to three common problems:

  1. An over-reliance upon estimates of value
  2. Assuming that a customer’s description of the problem is truly what they mean. Often customers more clearly define their pain point only after seeing the designed solution. It is not uncommon to hear, “oh..that’s not what I meant..I actually meant..”
  3. Confusing validation of the Problem Space with validation of the Solution Space. Just because a customer validates her pain point, doesn’t mean that using your designed solution is compelling relative to alternatives. Tweet this!

Bring Zen to Your Prioritization Process:

The point of Lean is to reduce waste. While all of the stakeholders are well meaning and I am sure truly believe their arguments, the simple fact is that the source of their anecdotes hasn’t been validated. Until you validate the potential solution with customers, all you really have is a hypothesis about demand.

Here is how you bring Customer Demand Data into the prioritization process:

  1. Create prototypes of the top features in the backlog. The ones that clearly stick out among the group.
  2. Run Demand Validation Tests against each prototype
  3. Benchmark the test respondents’ answers to the Cost Question for all tests.
    • Be sure to ask the same Cost Question for all of your tests
    • If certain features under-perform, but are highly strategic, then feel free to iterate and test again. But if the response is consistent, then build it at your own risk.
  4. Prioritize based on actual solution demand from customers, making slight adjustments based on development effort (i.e. if the #3 most valued feature is easier to build, then feel free to crank that one out first and ship it).

The Key to This Exercise:

The key to the Demand Validation test is the Cost Question. That is, it has to cost the customer something important to say, “yes, I want that feature.” The customer has to make a trade-off between something of value to them and this feature. That is when they truly evaluate its significance. Without the Cost Question, a customer’s input is counterproductive.

We have stated before that there are only three things that mean something (in this context) to a customer:

  1. Their Time
  2. Their Reputation
  3. Their Money

If you want to understand this point deeper, read the post on Demand Validation Tests.

Common objection: But I Don’t Have Time to do This!

Setting up a Demand Validation Test is very quick. You can set one up using Validately’s template within 60 seconds. We even help you recruit test respondents. There is a time lag in getting data back, so you need to start the process at least a week before the next sprint to allow for time to get the data back and interpret it. That means no more just in time delivery of designs.

So is it worth it? Should you alter your process to allow for this type of testing before the sprint planning meeting? Well, consider these stats:

  • A typical team spends 30% of their time on re-work. 30%!
  • 50% of a typical product’s features are unused.
  • Oh..and this ignores the opportunity cost of working on the highest impact features.

Still not convinced? Think about all of the time and brain power your team is wasting debating or jockeying for certain features. Moving to a data driven prioritization approach will free up the wasted time and energy in prioritization debates.

How to Get Started

I am not suggesting major organizational change. A good way to get started is to start two sprints from the one that you are in right now. Set up Demand Validation Tests in Validately (which takes a minute to do) for the top 5 features that you are considering for that sprint. If you want help, just email me. I am happy to help work with your team for free to set up the first test.

FREE EDUCATION from Validately’s Lean Customer Research Academy

Death to the Minimum Viable Product!

I am declaring death to the Minimum Viable Product!

And I don’t do that lightly considering our advisory board consists of the father of the Lean Startup, the author of Lean UX, the author of UX for Lean Startups, Marty Cagan, and several other awesome Lean thought leaders..

…so please humor me for a minute to explain why I want to kill (actually tweak, but “kill” sounds more dramatic) the core principle of Lean.
Tweet: Death to the Minimum Viable Product!  #lean #leanux

What is an MVP

Let me start by clearly defining the MVP:¹
Minimal Viable Product Definition



Sounds simple…so what’s the issue

The MVP is the most misunderstood term I have ever experienced in business.

Tweets about Minimal Viable Product

So many people have read/heard this definition and concluded that the MVP is really just a [crappy] version of your product that is an embarrassment to show to customers. I have heard so many times, “let’s just remove these features and call it the MVP version.” And so you get blog posts like this one from a company that I greatly respect, but really, really misses the point of the MVP.

Or tweets like this one..which..ugh, I don’t even know where to start.

Tweet about Minimum Viable Product

Again, the point of the MVP is to LEARN about customer demand and usability before over-committing resources. To make sure that you are only building what customers want. An MVP is NOT a fully usable product that will delight customers. It is simply a learning vehicle. A focus on Learning before scaling is one of the core Principles of Lean UX. In Jeff Gothelf’s words:

Principle: Learning over Growth. It is difficult to figure out the right thing to build and scale a business around that thing at the same time. They are contradictory activities. Lean UX favors a focus on learning first and scaling second.

Why is there a disconnect?

I think the root of the disconnect between theory and practice is the use of the word Product in the term MVP.

Lean practitioners have a very different definition of the word Product than most Product Managers that I know. When Eric Ries says, “that version of the collect the maximum..validated learning” he is much more focused is on learning, than on product. The point of an MVP is to validate or invalidate a specific hypothesis. That is why Lean uses the concept of a concierge MVP and heavily relies on user testing of prototypes. An MVP can be coded, but the key is creating a product version focused on hypothesis validation, not growth.

But for some reason, most people hear the the word Product and assume that it means the first version of a product that is coded and “launched.” And so, they build that version, release it and guess one likes it. Well…no duh!

What’s the alternative?
Tweet: Death to the MVP. Hello MVE. #lean #leanux
I would like to introduce the MVE..the Minimum Viable Experiment.

What’s the MVE you ask? Well, it is a prototype or a concierge MVP or anything else that you can use to validate or invalidate a product hypothesis.

See what I did there? Genius, right? I basically hijacked the definition of the MVP but am using a word that [hopefully] won’t be misinterpreted. My hope is that by replacing the word Product with the word Experiment, that the expectations are now better aligned with the results (ie learning) that you are likely to get.

No one expects an experiment to grow or scale. So no one should be disappointed when it doesn’t.

Again Jeff’s book Lean UX gives us great guidance here:

Lean UX makes heavy use of the notion of MVP. MVPs help test our assumptions – will this tactic achieve the desired outcome? – while minimizing the work we put into unproven ideas..This concept is an important part of how Lean UX minimizes waste.

Your prioritized list of hypotheses has given you several paths to explore. To do this exploration, you are going to want to create the smallest thing you can to determine the validity of each of these hypothesis statements. That is your MVP. You will use your MVP to run experiments. The outcome of the experiments will tell you whether your hypothesis was correct and thus whether the direction you are exploring should be pursued, refined or abandoned.

Dude, will the MVE go Viral?

I doubt it. First, MVE doesn’t sound as sexy as the MVP, an acronym that sounds like an award for a pro athlete. Second, I am not going to market it beyond this post.

But popularizing the term MVE wasn’t the point of this post anyway…happy experimenting!


New Validately Features: Get More Out of Your Lean Customer Research

We’ve listened to your feedback, tested prototypes, and launched a series of features that will allow you to get more out of Validately and your Lean Customer Research.  The highlights include:

  1. Sound recording of test respondents…without a download or plugin
  2. More analytics for quicker response analysis
  3. Easier sharing of results


Ask your test respondents to talk you through what they are doing and thinking as they go through the test. Just select sound recording during your test set-up.

sound recording copy

Your respondents will be asked to enable their microphone via their web browser before the test begins. But a download or a plugin is NOT used.


New Path Report: The order in which pages were visited by your test respondents, grouped by the number of page visits for each click with metrics such as average time on page and % exits.

Screen Shot 2015-01-20 at 11.32.37 AM

Improved Page Analytics: Additionally we show time spent on all pages and exit % in two other new reports.

Queuing of Recordings to Specific InteractionsOn the Path report we have added the ability to watch the recordings queued up to where in the recording the test respondent performed that interaction.

  • For example, let’s say less than 30% of test respondents exit on a particular page. You can now easily identify the associated recordings and watch them starting at the point where the test respondent entered that particular page. No more watching long videos!

Validately Paths pageRelevant recordings

Page Markers on Player Progress Bar:  We’ve introduced markers on the player progress bar to indicate the points in time when the tester clicked a link in your prototype to change a page. This makes it easier to see visually how long the tester spent on each page in your prototype and how often they changed pages.

Validately Player

Each vertical bar on the player progress bar represents a page change in the recording

3.  EASILY SHARE Qualitative Feedback and Videos with Teammates

You can now share the results from any test with teammates via a shareable URL. Test results are private by default. But if you want to share with teammates outside of Validately, click on the share link on the test results page.

Share survey results

You can also share specific videos with teammates outside of Validately from the Respondents tab on the test results page.

share video


We hope that these features make your lives easier. We have a lot of great features queued up but would love to hear your suggestions on how we can make Validately better for you.  Email our founder ( with any feedback or requests.