Why you should be evidence-based when designing (and less data-driven)

A lot of business owners like to think they’re evidence-based but they’re usually paying this lip service, rather than following a process.

Most of their web design projects start off clearly enough: a problem is identified (usually low conversion), and a goal to correct this is defined. The danger comes when solutions arrive too soon after this, with vague justifications.

Without intending to, assumptions appear in decision-making (always with the best of intentions). “I’ve heard users don’t scroll, we should make this page short”, “I’m pretty sure red converts better than green so we should make our buttons red”, “We should make the search field much bigger so more users find it”, etc.

If you were a detective trying to solve a crime, you wouldn’t just guess at how it happened, based on things you’d heard before. Well you could but you can be sure your case wouldn’t stand up in a court of law.

What should come before solutions is an understanding of what is happening and why on the current version of your website. A full picture of user behaviour.

The opposite approach to assumptions would be to find metrics for everything on your site, that can tell you whether something is working or not. Otherwise known as being data-driven.

Going the other way: the dangers of being data-driven

I used the term data-driven to describe my design process in the past. This is something that I’ve moved away from, for the following reasons:

• Data-driven suggests the designer follows only metrics and everything can be reduced to a number—not a way to build a meaningful product that resonates with people. Quantitative (or quant) data is great for finding problems but it doesn’t give you a lot to work on when trying to design solutions.
• In everyone’s minds data-driven tends to mean quant data (metrics and numbers) and not qualitative, which is actually the data I find more useful in the design process. Having to explain that each time meant that the term was working against me.
• Data-driven really suggests a lot of A/B testing, as that’s the main quantitative way of measuring one design against another. However, I work with a lot of startups and none of them have enough traffic or the staff to run high quality A/B tests. I wrote about why A/B testing doesn’t make sense for small businesses here.
• It can build the wrong expectation with clients. They expect some kind of silver bullet or new data-gathering tool that they’ve not heard of before. When I tell them that most of the time all they need is to actually dig through Google Analytics, they can be disappointed.

The reality for the startups I work with is that gathering bullet-proof quant data is really hard and time-consuming and in the end may not actually give us the answers we need. The design process needs to be based on facts but needs to be much quicker and more pragmatic than large-scale data-gathering.

What we’re really looking for is proof that we have a problem and proof that we’ve got a good solution. Enough to get something out of the door that improves what was there before and keeps up growth.

What we want is evidence-based UX design.

Draw a picture with evidence

There are lots of ways you can gather evidence in a design process. Just as in a court case there is strong evidence and weak evidence, but the more sources you can get the stronger your case tends to be.

Quant data may often be one of the most powerful forms of evidence you have but if you’re a small company without much traffic then that data will be poor. With an evidence-based approach you can make up for this by gathering qualitative data and even some things you wouldn’t think of as data (interviewing clients, customer feedback).

My design process can touch on 9 or 10 evidence-gathering methods. I’ll start by talking to the client to get their knowledge; I’ll use my experience to assess their site in an expert audit; I’ll look at audience data and conversion data; I’ll study heatmaps and session recordings to understand pain points; I’ll probably run a user test to see why there are problems; I’ll assess competitor sites to get inspiration; as I design I’ll get feedback from potential users in the form of guerrilla user tests on prototypes; and I might run quick preference tests or surveys.

Using this process with clients is how we know as we go along that we’re designing something strong. The different methods build on each other to create a richer picture of what is happening and provides lots of ideas for solutions.

Evolve what you have

There are still plenty of designers who sit there in a vacuum with just a brief and Sketch open hoping to come up with some magic solution. Or startup founders who think they have a problem and code up the first solution that comes to mind.

You’re usually not designing completely from scratch. Most design work is redesign work of some kind and when you’re trying to improve something you have a lot of ways you can get hold of evidence of user behaviour.

The big challenge in redesigning is knowing the order to gather your evidence in and which method is best for finding out different things. This is a central theme that I tackle in my video course, The Evidence-Based Redesign.

Further reading

Some further interesting articles that has helped my thinking on the challenges of gathering evidence and being data-driven:

• Lets stop doing research by Erika Hall—she comes at it from the approach that too much qualitative research is a big overhead for many but lands in the same place: being evidence-based.
• Know the difference between data-informed and versus data-driven by Andrew Chen—he comes from a product point of view and warns of the danger of valuing quantitative metrics above all else.
• Metrics Versus Experience by Julie Zhuo—explains how you can measure experiences with metrics but you generally need to look at lots of different things to build a full picture.

Last updated on 6 November 2019