New Features Released: Easily Test on Mobile, Save Time Analyzing Results & More

We’re making it easier than ever for our customers to test and learn. Tweet This.

Easily Test on Mobile with Sound Recording

You can now record user actions and sound in mobile web and prototypes without adding anything to a tester’s device. Any user can instantly test on mobile and give you feedback. Learn more here.

mobile recording

Save Time Analyzing Test Results

Validately has made it even easier to analyze and learn from your test results with our new Task Reporting and Download Results to CSV.

  • Task Reporting: No more time spent annotating testing videos to see if tasks were completed. With Task Reporting you can now quickly view key metrics on task completes and watch videos (or see qualitative answers) from only completed or failed tests. Learn More.

Task Report copy

  • CSV download for easier analysis. You can now download questionnaire responses including any panelist demographic data into CSV format for easy analysis.


More Customization!

You can now customize your tests further by using Markdown in the intro message, followup questions, and tasks. Markdown is an easy way to add styles — bold, italics, font size, ordered and unordered lists — to your text.

New Pricing for B2B Panels

If you use our panel to get feedback on your prototypes, we now charge a flat fee of $10/response for premium B2B categories (Job Title / Role and Industry).

Look for more exciting feature announcements in a few weeks!





Swing for the Damn Fences: The Design Sprint

We are in the home run business. Incremental features might eat up most of your time, but you need to be swinging for the fences! Tweet This!

You need to be taking risks on crazy ideas. And you need to be doing it often.

Anyone who understands baseball knows that home run hitters strike out a lot. It is just a part of baseball…and it’s a part of new product and feature development. So to make sure you are hitting home runs every year, you need a lot of at bats. And there is only one way to get more at bats…rapid prototype testing.

The Solution:

Our good friends at Google have given us a great solution for rapid prototype testing…the Design Sprint. What’s a Design Sprint? It is a 1-week sprint that helps you swing for the fences. Tweet this.

You can read more about it here, but here is an overview:

  • MONDAY: UNPACK On the first day of the sprint, your team will “unpack” everything they know. Expertise and knowledge on most teams is asymmetrical: Sales knows things engineering doesn’t, customer support knows things design doesn’t, and so on.
  • TUESDAY: SKETCH During Sketch day, your team will work individually to draw detailed solutions on paper. After sketching, you’ll use a structured critique and weighted voting to select the best ideas from the field of possibilities.
  • WEDNESDAY: DECIDE By Wednesday, you’ll have over a dozen solutions to choose from. That’s great, but it’s also a problem, because you can’t prototype and test a dozen solutions. You’ll have to narrow down and make tough decisions.
  • THURSDAY: PROTOTYPE You’ll spend Thursday in the flow, being ridiculously productive. We show you our systematic plan for doing the impossible: build an entire realistic-looking prototype in just eight hours.
  • FRIDAY: TEST On Friday, you’ll show your prototype to real customers. By the end of the day, your ideas have all been exposed to oxygen — some will be smash hits, while others will meet an early end.

A combination of rapid prototyping tools and Validately make this tactic very easy to execute. You just need to allocate the time.

Common Objection:

I can already hear some objecting…”sounds good, but how the hell can we carve out a week to do this? We have a tight deadline to hit!”

My response: Honestly, what would happen if you paused adding incremental features for a week and invested in finding a home run? Seriously…what would happen? Tweet this!

Would your business collapse? Doubt it. Unless you are currently building a potential home run, then not much would happen. Certainly nothing that would move the needle…so swing for the damn fences!

Your Team Will Love It!

I will tell you one thing for sure that would happen, your team would be jacked up! You would see a unique energy. Creative ideas would dominate office talk in between the Design Sprints. Employee morale will go up. And your team comradery will improve…so swing for the damn fences!

Your Customers Will Love It!

Customers love to be “in the know” about your innovation. Not all customers, but a lot of customers. I have seen it over and over. They love that you are thinking creatively about how to solve their problems or bring new features/products to market. It will build a trusting relationship with them. They too will get excited about the upcoming Design Sprints…so swing for the damn fences!

Oh, by the Works!

Google puts its money where it’s mouth is. Google’s famous 20% Rule enables employees to spend considerable time swinging for the fences. The result was AdSense (which is about 25% of revenues), Gmail, Google Transit, Google Talk, and Google News, among other projects. Google also trains all companies it invests in on the Design Sprint. If it works for a company growing as fast as Google, it can work for you…so swing for the damn fences!

Want Help?

Think this sounds cool, but want help getting started? Email me, I am happy to work with you to launch your first Design Sprint.


How A/B testing with Prototypes Can Save Time & Money

Our founder Steven Cohn and Alpha UX’s Nis Frome co-authored a piece in UX Magazine on How Split Testing Validates New Product Concepts without Code.

“Product managers already know that user data and behavior is a far better and more consistent predictor than hunches about the potential of a product. But while split testing prototypes can be one of the most effective weapons in the product management arsenal for answering critical product questions, it’s often underutilized.

Split testing, or “A/B testing,” is a popular marketing tactic for optimizing collateral and improving conversions. Instead of rolling out testable variations of products that are much more time-intensive and costly than landing pages, product managers simply need to be lean like marketers and split test prototypes. And this vehicle for generating feedback is the critical difference…”


How to Get Lean

I have spoken with a lot of organizations that want to bring Lean to their product development process.

Below are two transformation approaches that produce results:

  • Approach 1 is a way to start doing Lean without major process or organizational change
  • Approach 2 is how to re-organize your company to a Lean process.

Approach 1: Becoming Lean-ish

This model is for teams that aren’t quite ready for complete organizational change. This model is perfect for a head of product or a head of UX that still wants to use Lean techniques, but doesn’t want to waste energy fighting her company’s inertia.

If this is you, the good news is you can still make a major impact with a smaller, more controlled approach. Here’s how:

Lean is all about reducing waste by treating all product or feature ideas as hypotheses that need to be validated before building. That means, tasking your PMs and UX Designers with running rapid experiments to validate demand before the sprint planning meetings. We have posted before a simple technique on how to test demand with a prototype. It’s not brain surgery. It just requires discipline and the right tools.

If you want to do this, you need to give your UX Designers and PMs the time to run these experiments and interpret the results before they determine what goes into the next sprint. I suggest building your process so that the UX team is always at least 1 full sprint ahead of the sprint planning meeting. Also give them the power to say, “I know execs want this feature, but customers don’t!” Support them when the data is on their side. TWEET THIS

If your team doesn’t know how to run these experiments, have them watch the free videos in our Lean Customer Research Academy. Or you can reach out to me. I offer free workshops on how to run these experiments. If you are interested, feel free to email me.

Approach 2: Getting Religion

This model requires complete organizational change. As the section title suggests, you need your team to become true believers. Top to bottom of the organization needs to be completely committed. There will be some pain during the transition. And your resolve will be tested. Some employees will leave and others will be fired.

Step 1: I usually see this transformation start by having the entire executive team read “The Lean Startup”. Usually a group discussion follows. If everyone on the executive team agrees, then they have the next level of managers read the book and discuss it. The goal of these discussions is to gain consensus and start analyzing the changes that must be made.

While this is happening, the company often brings in a Lean process consultant. This person’s job is two-fold:

  1. Oversee and influence the new process and transformation plan.
  2. Train the trainers.

The consultant will work with the product team and senior executives on the engineering team to define the new process and build a detailed transition plan.

Typically you will pilot this new approach with a specific product team. The “product team” is comprised of a Product Manager, UX Designer, Engineers, and a User Researcher. This pilot will help work out the kinks and educate the leadership on how Lean will work within your specific company’s culture.

With the results of this pilot, you can start planning the company-wide rollout.

Obviously this is an oversimplification of what’s involved in major organizational change. It is really just a straw man of major milestones. Make no mistake this is major surgery for your company. Treat it as such. Expect old deadlines to be missed while people are re-learning how to build products. Expect some people to fight it. Expect some complainers. But once you start this, don’t turn back. It works. It really really works. So just push through.


Take the Pain Out of Feature Prioritization

“If we have data, let’s look at data. If all we have are opinions, let’s go with mine.”

Jim Barksdale, Former CEO of Netscape

The best part of my job is that I get to chat with leaders in Product and UX at a lot of great companies. So when I hear a common theme, I blog about it.

Lately, I have heard a lot of frustration around the often painful (and inefficient) process of feature prioritization. I have heard many complaints about a highly political process that is largely driven by anecdotes and debatable estimations.

“Our customer needs this feature to renew!” – Sales

“Our competitor just launched this feature and it is getting a lot of buzz!” – Marketing

“I have this great idea, it’s going to be huge!” – Sr. Exec

“This feature will have a bigger financial impact.” – Finance

When I hear this, I think of the great Jim Barksdale quote that I used to open this post.

We all have amazing customer demand analytics about the live product, yet the pre-build prioritization process is often void of customer demand data.

The good news is this is easy to fix.

I Already Know the Customers Want This! Let’s Just Build It

What customers say they will do, what customers say they did and what customers actually do are three different things.  Tweet this!

This inconvenient truth is what makes the feature prioritization process so difficult. This fact about customers leads to three common problems:

  1. An over-reliance upon estimates of value
  2. Assuming that a customer’s description of the problem is truly what they mean. Often customers more clearly define their pain point only after seeing the designed solution. It is not uncommon to hear, “oh..that’s not what I meant..I actually meant..”
  3. Confusing validation of the Problem Space with validation of the Solution Space. Just because a customer validates her pain point, doesn’t mean that using your designed solution is compelling relative to alternatives. Tweet this!

Bring Zen to Your Prioritization Process:

The point of Lean is to reduce waste. While all of the stakeholders are well meaning and I am sure truly believe their arguments, the simple fact is that the source of their anecdotes hasn’t been validated. Until you validate the potential solution with customers, all you really have is a hypothesis about demand.

Here is how you bring Customer Demand Data into the prioritization process:

  1. Create prototypes of the top features in the backlog. The ones that clearly stick out among the group.
  2. Run Demand Validation Tests against each prototype
  3. Benchmark the test respondents’ answers to the Cost Question for all tests.
    • Be sure to ask the same Cost Question for all of your tests
    • If certain features under-perform, but are highly strategic, then feel free to iterate and test again. But if the response is consistent, then build it at your own risk.
  4. Prioritize based on actual solution demand from customers, making slight adjustments based on development effort (i.e. if the #3 most valued feature is easier to build, then feel free to crank that one out first and ship it).

The Key to This Exercise:

The key to the Demand Validation test is the Cost Question. That is, it has to cost the customer something important to say, “yes, I want that feature.” The customer has to make a trade-off between something of value to them and this feature. That is when they truly evaluate its significance. Without the Cost Question, a customer’s input is counterproductive.

We have stated before that there are only three things that mean something (in this context) to a customer:

  1. Their Time
  2. Their Reputation
  3. Their Money

If you want to understand this point deeper, read the post on Demand Validation Tests.

Common objection: But I Don’t Have Time to do This!

Setting up a Demand Validation Test is very quick. You can set one up using Validately’s template within 60 seconds. We even help you recruit test respondents. There is a time lag in getting data back, so you need to start the process at least a week before the next sprint to allow for time to get the data back and interpret it. That means no more just in time delivery of designs.

So is it worth it? Should you alter your process to allow for this type of testing before the sprint planning meeting? Well, consider these stats:

  • A typical team spends 30% of their time on re-work. 30%!
  • 50% of a typical product’s features are unused.
  • Oh..and this ignores the opportunity cost of working on the highest impact features.

Still not convinced? Think about all of the time and brain power your team is wasting debating or jockeying for certain features. Moving to a data driven prioritization approach will free up the wasted time and energy in prioritization debates.

How to Get Started

I am not suggesting major organizational change. A good way to get started is to start two sprints from the one that you are in right now. Set up Demand Validation Tests in Validately (which takes a minute to do) for the top 5 features that you are considering for that sprint. If you want help, just email me. I am happy to help work with your team for free to set up the first test.

FREE EDUCATION from Validately’s Lean Customer Research Academy