Advice: Evidence methods

Design Testing

What you can learn

First of all, a definition: what is design testing? I'm using it to mean running quick tests or gathering feedback whilst your designs are still in progress. It's usually means showing screens individually rather than as a completed website or app prototype as in guerrilla user testing. It can mean printing them out on paper or it can involve sharing the designs online.

This is a chance to gather user feedback early in the process and shape your design decisions with evidence before committing to building anything. It allows you to quickly test out a couple of options and solve arguments if your team can't agree on a way forward. It's fairly lightweight and won't give you huge insight but it is suitable for answering targeted questions and helping you course-correct.

How to do it

I'm a big fan of UsabilityHub for carrying out this method (see tools), and for that reason I'll use their test types to cover the different ways to do design testing. If you use their site it's a case of uploading a design and creating a simple test around it with a few questions before putting it out to testers. If you're doing this offline yourself then you’d just show people the design and ask the questions verbally.

In all cases I'd recommend testing either a whole page design or a section of a page that would be visible without scrolling. Don’t test just individual elements without the context of the page around them.

When you get your results back, it’s worth doing a bit of analysis yourself—I find the ‘word clouds’ provided by default on UsabilityHub aren’t that useful. I export the results into a spreadsheet and do a bit of sentiment analysis. For example, if I’ve asked what people think of a certain design, I’ll classify the results as positive, negative, or neutral. This helps compare if I run the same test with another design.

Watch out for

The main challenges with running good design tests are around giving users simple instructions and asking good questions. These are brief tests and to get useful data out of them you need to be specific, as you don't have the time of a user test to cover a lot. With that in mind here are my tips for good test writing:

Give the user the right amount of context. Before each test you get to set the scene for the user. So when showing them a webpage or app screen tell them what they would have just done to reach it (e.g. "searched for a bank loan", "shopping for jewellery"). One sentence is usually enough, don't go overboard with lots of detail about what they could be doing, as part of the point is to get their fresh eyes on this and the benefits of an outside opinion.

Don't ask leading questions. People can see design testing as a chance to 'prove' that their idea is right and have some data behind it before taking to management. As a result they can write leading questions that suggest a solution or leave the user only with yes/no answers. Instead you should write questions that allow the user to express their thoughts about a design ("what do you think...", "how useful is..."). If you skew the results your design might ‘win’ in the short-run but if you come to release it and it fails, you’ll have to deal with a much bigger problem in the long-run.

Don't ask too many in-depth questions. These are short, simple tests and in the case of five second tests, the user barely gets a chance to see the design, so don't go overboard and ask too much. Most design tests are set up to answer a single question so I tend to have that plus a follow-up but certainly have no more than three questions.

Example tools (and cost)

As mentioned above I find UsabilityHub is the perfect tool for testing at this stage, as the tests are super-quick to set up and quick to get responses to. It's also free if you carry out a bunch of tests yourself or you have your own audience to distribute it to (perhaps on a mailing list). To use their panel, it costs $1 per random user or $3 per demographic-targeted user. I normally specify the demographics of my testers and just run them past 25 people.

There's a similar new tool, called Better Design, which allows you to set up quick tests in the form of polls for other users to vote on—fully free at the moment.

How long does it take?

Setting up a test and getting back results from 25 users takes a couple of hours.

How often should you use it?





Last updated on 25 April 2017

Learn how to make evidence-based improvements to your website

Sample video images from The Evidence-Based Redesign course

Learn how you can redesign a website based on evidence of how users actually behave, without spending a fortune. A new video course coming in January 2018 but register today and get a massive pre-launch discount.

Here's another method

Guerrilla User Testing

The tricks to doing quick user tests on the fly to improve your prototypes. Learn more

Last updated March 2017

View all the methods in the guide

Note: the examples in this guide are for website design, but most of the content is also applicable for native apps and software.