Category: UX advice / Research articles
I find the best way to use surveys is to ask open-ended questions to encourage stories and meaningful answers from your users. Asking people to rank things on a scales and fit into multiple choice is a bit of a false premise that misses out on the interesting details that people can express when given a blank box.
Yet have you ever run an open-ended survey and got back paragraphs and paragraphs of written answers? It can be pretty daunting and hard to know what to do with it.
Here’s how I turn those blocks of text into something usable. This approach works best with surveys from 50 or fewer people—more than this and you’re probably going to need automate it to speed things up.
There’s no real way around it other than to go through and read the results. You need to get in the weeds and understand what people are saying, at which point you can manually classify them into themes.
It might be possible to assess answers with AI and natural language processing, but for many subjects there’s a lot of context required. In addition I’ve found people write in odd ways, sometimes meaning something different to what initially seems to be the case—particularly those who don’t have English as their first language. Plus what you’re looking to get out of a survey is so dependent on your research angle and what you’re trying to understand.
The aim is to go through and summarise answers by the sentiments they express. Both the subject and whether they are being positive or negative is of interest.
People phrase things in a variety of ways but once you ask a few people the same question, repeated common themes are likely to emerge. After this you should be able to easily spot the biggest issues and things to tackle.
It’s pretty common to get your answers back from online surveys in a spreadsheet format. To start categorising, it’s important to go through the results question by question (not user by user) so I copy the answers to separate sheet for each question. I then use columns next to the answers for the categories.
It may not be obvious how to categorise at first but I find the best approach is to just start reading: you should soon get a sense of the type of answers. You should be able to summarise them in simple two or three word phrases. This kind of thing is best shown by an example, so take a look at the one below.
It’s always worth going back and checking your early categorising because as you go through the process new categories can emerge that better summarise what all the answers are saying. Once you’re happy that you’re being consistent in how you record sentiments then all you need to do is add up the number of results to get a points total for each category.
People’s answers can ramble and easily cover two or three different points, so it’s fine to categorise an answer with two or three sentiments. They can all be equally important to a person. I wouldn’t count any more than four categories of sentiment however as this is giving too much weighting to one person.
It’s certainly possible that people can bring up a topic in their answer in a negative way. I’ll note these too and count them as a minus one point. Sometimes this means reducing the score of a popular category and sometimes it means a category of its own that only has a negative points score.
Assuming you have at least 10 respondents, if only one person raises something then I don’t count it in the final standings. For something to be considered a theme it needs to be raised by at least two people (or maybe three if you have a lot more respondents).
At the end of doing your sentiment analysis you will have a leaderboard of themes for each of the questions. You’ve now boiled down masses of words into a few actionable areas to address, which is obviously great news.
However it’s important not just to present this as a few simple numbers as you’ll be missing out on the benefits of a survey. You now actually have the best of both worlds: some quantitative data that gives you a sense of the size of the things people raise, along with qualitative results that can back it up and explain things further.
Find a quote or two that best shows the sentiment that is being expressed and present it alongside your results. This often makes them much more powerful as it uses real people’s words to add in some reality that is hard to ignore, as numbers can be empty on their own. I’ve created a quick example of a way to present the results from above in a slide below.
Sign up here to get a guide to my favourite (mostly free) tools for evidence-based designing. Plus a massive, advice-filled reading list.You'll also get UX articles & content emailed to you once a week. Your email is never shared. Unsubscribe at any time.