Advice: Evidence methods


What you can learn

Surveys promise you the opportunity to gather the thoughts of lots of users without too much effort: you can potentially reach thousands of people with just a single form. You can ask them almost anything (though some types of question are better than others, see below). One of the things surveys are often used for is Net Promoter Score, which is I cover in more detail here.

They also offer the opportunity to quantitatively assess behaviours, by asking how many people do certain things, which often appeals to the more analytical managers. However this is something you should be very careful with, as a good survey is better as a qualitative tool and a starting point for further investigation than a quant one that gives you absolute truth. It's best for asking open-ended questions and finding out issues and thoughts that users are having.

There are lots of types of survey out there, including website pop-up feedback windows. For the purposes of this I will cover the one-off type that you might run to research a subject or potential design project.

How to do it

There are plenty of tools out there that make creating online surveys easy and allow you all the different field types you might need (I cover some below). In addition the distribution shouldn't be a problem either: you can email it out to a mailing list, or share on social media, or even put a link on your site. Though be aware that the users you distribute to will affect your results. But the actual practicalities are fairly straight forward.

Most of the work is actually in the planning and preparation. You need to know what you want to ask and to determine if a survey is the right approach to take. If you're after detailed behavioural understandings then perhaps an interview or user testing is a better bet. Ideally a survey is a starting point, which gives you ideas of the issues to investigate further with interviews.

When writing the questions, it will depend on the type of question as to whether you either keep them specific (for multiple choice answers) or open-ended (for answers where you want the user's free text). The latter is often more useful for getting to the heart of real issues and gives you useful data as the user isn't limited by what they can say. If all your questions are multiple choice, it's a lot harder to find out what you don't know.

If you want to maximise your response rate, then keep the survey on the shorter side (fewer than 10 questions). In fact you should do this anyway: just because you have the opportunity to ask people anything doesn't mean that you should. Try and keep it focused around a subject and don’t attempt to capture every question you might have for your audience in one go. Also giving users the opportunity to answer anonymously can help them feel confident about opening up.

When it comes to analysing your results it's going to depend on the size of your survey. If it's a short one, you can read them all and potentially act on them too. Otherwise you're looking to analyse your results for sentiment: go through each answer and try to categorise them. For example if you're asking people about their problems with a site then you should be able to group them into things like 'navigation', 'search', 'payment' etc. This should help you order the key things that need to be solved with any new design, and you can delve back into the actual answers to get the detailed requirements.

Watch out for

As tempting as it may be, try not to quantify the qualitative. For example try and avoid questions that ask how much people 'like' something (often on a scale of 'strongly like' to 'strongly dislike'), as it's all so subjective it can be pretty meaningless. You could get back a survey with people saying they love your site but they still may not be buying your product. Erika Hall writes well about this here.

You can of course quantify quantitative data with a survey, so if you want to ask how old people are or how much they earn, this can become the stuff of charts and graphs.

Just like when interviewing, try not to ask users to predict future behaviour. Don't ask 'how many times will you go to the gym in the next month?' because you'll just get back their ideal answer or one to impress you, whereas the reality is likely to be a lot different. Ask about actual past behaviour instead ('How many times did you go to the gym last month?').

When declaring any survey results, make sure you explain who you surveyed, especially if they're not representative of your actual user base. Be careful of taking your results out of context and declaring that 'all users think this'.

Be careful of incentivising users to complete you survey. If you offer too good a prize then you're likely to get people rushing through to complete it and not caring about what they write. I think you're better off just being open and saying you need help (most people are actually happy to give help and have their opinion heard) and telling them it will only take 5-10 minutes of their time.

Example tools (and cost)

There are lots of tools for making forms and surveys out there. I've used several including Google Forms (free), which looks quite basic but collects your results in a spreadsheet for analysis. Others include Typeform (free and from £28/mo), which is arguably the best-looking form website out there, and Wufoo (free and from £12/mo), which I've used to create fairly complex forms with their conditional rules. The tool is generally less important than the content however.

How long does it take?

Writing a good survey isn't the quickest of tasks. Obviously it will depend on length but expect to spend half a day at least. Getting results can take up to a week.

How often should you use it?





Last updated on 25 March 2017

Learn how to make evidence-based improvements to your website

Sample video images from The Evidence-Based Redesign course

Learn how you can redesign a website based on evidence of how users actually behave, without spending a fortune. A new video course coming in January 2018 but register today and get a massive pre-launch discount.

Here's another method

Field Research

Sometimes you need to get out there and see how your users behave in the wild. Learn more

Last updated May 2017

View all the methods in the guide

Note: the examples in this guide are for website design, but most of the content is also applicable for native apps and software.