Case Study: How LeanCog Used Validately to Cut Time to Market by 55%

Project Background:

Our client, a Fortune 500 firm, needed some fast feedback on designs and features they were considering for a new social app in order to prioritize features into an MVP. Initial development was already underway, and the project requirements were growing and changing. The window for design or feature changes was quickly closing, and by using Validately, we were able to reach the right target segment and get quick feedback. Tweet this!

Testing Objectives:

  • MVP definition – Which features resonate most strongly with the target user segment?
  • Design/Usability – Can users intuitively navigate to accomplish these user stories?

Structuring the Tests:

Before launching any tests, we outlined our key hypotheses related to the product concept using tools such as the Business Model Canvas and the Value Proposition Canvas. We used these high level frameworks as a baseline to define more detailed test objectives for the most important aspects of our product concept. Our test objective included the who (who we were targeting), the what (what we were testing), and the how (how to run the test). Defining the test objective and success criteria up front is very important to evaluating the results later on.

Using Validately, we setup tests for each core user story and persona to solicit both feature feedback and test usability. Because interactivity was essential to our product concept, we created clickable HTML mockups with embedded video to give the user a better sense of the experience. We then defined specific tasks for the user to complete and used the survey feature to ask questions around feature feedback (ranking 1-5) and purchasing behavior.

Once the tests were setup, we used the recruitment panel to attract users who matched our personas. Before launching more broadly to panel users, we had a couple of friends and family members complete the tests to ensure they made sense.

Interpreting the Results:

Even before we ran the tests, the process of converting the designs into clickable mockups for goal-based user stories highlighted important design gaps, which forced improvements to be addressed and saved valuable development rework time.

After launching our tests in Validately, we watched the initial screen captures and quickly found that many users were not able to complete the tasks we had outlined. This helped us identify where the designs were not intuitive and led to additional design and workflow improvements.

We also reviewed the analytics and identified that there was a distinct difference in feature interest based on the different segments we were targeting, which helped us better define our user personas. By creating a separate test for each target segment, we were able to uncover these insights more easily.

Key Takeaways:

  • Define the concept, tests, and success criteria up front before diving into Validately – make sure you clearly understand the learnings and outputs you are trying to get
  • Keep the tests simple and focused on one objective at a time – create separate tests for use cases and target segments
  • Simple mocks and wireframes work for most cases, but sometimes you may need a higher fidelity mockup to properly convey the true user experience
  • Be sure to view the screen captures to gain more insight behind the quantitative results and uncover potential usability issues
  • Test iteratively to make sure the results you’re getting are providing learnings you can interpret and incorporate

Ultimately, Validately was an important tool to quickly and efficiently collect the evidence we needed to provide direction to our client as well as setup our other market testing efforts, including user interviews, landing pages, etc. Collectively, these efforts enabled our client to reduce time to market by 55%.  Tweet this!

 

leancog copyZain Yousaf (zain@leancog.com) is a Managing Consultant at LeanCog, a product development and innovation firm that helps companies iteratively develop, test, build, and launch new products using lean methodologies.

 

BECOME A USABILITY TESTING EXPERT:
SIGN UP FOR OUR WORKSHOP SERIES

or

email Sales@validately.com for details

Upcoming Validately Webinars!

New Features to Take the Pain Out of User Research

COMING SOON: ON-DEMAND LIVE MODERATED TESTS

  • Are you spending too much time recruiting testers?
  • Tired of using Skype, GoToMeeting, and lots of other tools to conduct a moderated session?
  • Do you deal with the hassle of coordinating payment to testers?

Our On-Demand Live Moderated Tests will solve these headaches. 
1. You create a live moderated test in Validately
2. We find you a live test respondent within 10 minutes
3. You watch the respondent’s screen live and ask follow up questions via voice through your browser. The recording, analytics, and Q&A will be saved in the Reports tab just like any other Validately test.

Want early access? Sign up for a demo! Tweet this!

demo

 

TWO NEW LIVE FEATURES

We’ve launched two new features to help you further customize tests so you are getting the feedback you need!

Add More than 3 Tasks
We’ve removed the 3 task limitation allowing for longer tests.

Conditional Questions
Ask follow-up questions based on respondent answers for more in-depth feedback.

UPCOMING WEBINARS

We have two webinars in April. Hope you can join us. Tweet this!

Thursday, April 23rd at 1:00pm ET
“Lean Product Management for Enterprises: The Art of Known Unknowns”
Natalie Hollier
Validately VP UX

Tuesday, April 28th at 1:00pm ET
“How to Validate Customer Demand Before You Code”
Steven Cohn
Validately Founder & CEO

SIGNUP FOR WEBINAR

Podcast: Demand Validation is Product Management

Check out my podcast on AlphaUX’s “This is Product Management”

I share tactics for validating demand for a product idea before building it, why I think Minimum Viable Products should die, and what I mean when I say product managers are in the business of homeruns.
DEMAND VALIDATION IS PRODUCT MANAGEMENT

 

Below are some other podcasts in the series:

COMMUNITY IS PRODUCT MANAGEMENT
Sarah Judd Welch, Founder of Loyal

USER ENGAGEMENT IS PRODUCT MANAGEMENT
Nir Eyal, serial entrepreneur and author of Hooked: How to Build Habit Forming Products

Case Study: How Google Search Stays on Top

The below chart is the market share over the past 6 years of search engines…and it is pretty astounding!

 

StatCounter-search_engine-US-monthly-200807-201503

 

As you look at it, keep in mind, search is a product that has no network effects and no switching costs. Actually, one could argue, that the privacy risk of data history is a reason to NOT use one search engine a lot. Yet, Google has maintained dominate market share in the face of large companies throwing massive amounts of money to displace it.

 

The multi-billion dollar question is…How?

There are likely several factors, but I recently heard a statistic that I think explains a significant part of the reason. You ready for the stat?

 

Google Search prototypes and tests every feature idea they are considering for their backlog but only codes and releases 25% of what they prototype. Tweet this!

 

Think about that for a second. Google Search has world class engineers, product managers and designers! This elite team has access to massive amounts of data and analytics to inform feature ideas. Yet the Search team is disciplined enough to only build and release 25% of the features it prototypes and tests on customers.

 

That’s why, after all of these years, its product looks like this…Simplicity is hard work!  Tweet this!

 

Screen Shot 2015-03-10 at 3.54.21 PM

 

What does Google Search’s Product Development Process Look Like?

I am not an insider, so I do not know Google’s Search’s product development process. However, if Google’s Design Sprint or Pretotoyping is any indication, below is a pretty good guess of its process.

Screen Shot 2015-03-10 at 4.01.03 PM

 

Here is how this process works:

Step 1:

Start with ideation. Seek ideas from all team members. Use some sort of voting to narrow the list of ideas down to what is worthy of prototyping and testing.

Screen Shot 2015-03-10 at 4.02.33 PM

 

Step 2:

Prototype and test the prototypes on customers. To do this, you need to get really, really good at rapid prototyping and rapid user testing. Google’s Design Sprints give us the outline of how to do this. Google’s team likely benchmarks feedback and compares features against each other and against past successful features. At Validately, we advocate using a Cost Question when doing demand validation. We believe benchmarking a consistent Cost Question against all features gives a great way to compare.

Screen Shot 2015-03-10 at 4.02.47 PM

 

Step 3:

Only prototyped features that have received the strongest user demand feedback relative to other options get moved into sprints. And if nothing gets validated before a sprint? Then spend that sprint cleaning up technical debt. But don’t just build something without it being validated first. That’s a recipe for a cluttered product.

Screen Shot 2015-03-10 at 4.01.03 PM

 

Sounds Cool…but How Do I Apply This To My Team?

Using these techniques are not as hard as it might sound. There are four keys to this process working:

  1. Buy-in: There are two models – “Get Religion” (reorganize your company to a Lean process) and “Lean’ish” (a way to start doing Lean without major process or organizational change). I outline them in detail here.
  2. Establish A Buffer. You need to give your UX and Product team time to actually validate prototypes before a sprint. That means no just-in-time delivery of prototypes to the dev team. I recommend a full sprint buffer. If you don’t have that buffer now..then have your dev team clean up technical debt for a sprint. They will love you for it, I promise.
  3. Pick a Rapid prototyping tool. Make sure you find a great prototyping tool that allows you to quickly visualize ideas. Time bound the prototyping process so as to not allow yourself to try to create a masterpiece. The goal is to validate a hypothesis, not design a finished product.
  4. Pick a Rapid Customer Research tool. ahem…Validately baby!

 

Want More Info?

Want to learn how to apply these techniques to your organization? Sign up for one of our workshops.

 

 LEARN MORE