The Case Against UX Testing

Ben Moss.
January 17, 2022

Picture a dark office, blinds drawn. Picture a UX designer smoking a cigar. See the light filtered through the smoke whipped to fog by a spinning ceiling fan. Watch as the UX designer sits at a desk and considers the website.

The Case Against UX Testing.

The UX designer has devised a series of tests to determine if a green button is better than a red button. One of them involves tipping a tortoise onto its back. He looks the website over carefully and says, “Describe in single words, only the good things that come to mind about your mother.”

The website pauses, sweating under pressure, then replies, “Let me tell you about my mother…”

BLAM! The website pulls the trigger of an unseen gun, and the UX designer collapses, leaving the project to be rebuilt from scratch in Material by Harrison Ford, with overuse of Post-its delegated to Edward James Olmos.

Who Does UX Testing Actually Serve?

In the past’s bleak dystopian future (1982’s Blade Runner was set in 2019) no one benefitted from asking the wrong questions. And little has changed.

Designing any test to verify UX is fraught with as many complications as administering the test. Questions are skewed by bias, conscious or otherwise, and competing agendas. Even with something as apparently simple as a split test, the potential for distortion is immense.

When planned by a designer, a UX test offers little benefit to a client; the benefit is to the designer, who can then say their ideas are validated (or not).

Imagine hiring a developer to code a website, only to discover that the developer didn’t know CSS and expected to be paid to learn it before completing the work. You would hire someone else because that developer isn’t qualified.

From a client’s perspective, a UX designer should know, through experience, whether a green button is better than a red button. Designing an elaborate test to split-test the button color serves little purpose other than indemnifying the designer against mistakes.

The ROI of UX Testing

It’s widely accepted that there is substantial ROI (Return On Investment) from UX testing. We’ve all heard apocryphal stories about sites that split-tested their checkout and improved retention by 5%.

I’m going to go out on a limb and say that without user testing, that site could have improved its checkout retention by 4.9% simply by hiring a competent, experienced designer. But what about the remaining 0.1%? Well, for most sites, 0.1% represents very little profit. And the cost of recovering it via testing far exceeds the benefits.

When a company the size of Amazon, Netflix, Spotify, or Google split tests a website, it can afford to allocate $25k for user testing because it stands to gain 0.1%, and that represents far more than $25k. To meet the same 0.1% improvement, a small business has to design and run the same tests, incurring the same costs. But in the case of a small business, $25k could eat up all of its profits.

UX testing almost always works. But it is only profitable at scale.

If a good UI designer with a grounding in UX can improve checkout retention by 4.9%, tripling the project budget for just 0.1% more is a tough sell. Bluntly, that $25k is better spent on advertising.

What UX Designers Can Learn From Psychiatry

We all have the tendency to think we’re unique. It’s a survival trait attributed to our prehistoric brain. That belief in uniqueness is particularly strong in highly competitive people. We all think our site, our side-project, our approach are original. And we’re all wrong.

When a psychiatrist sits down with a patient, they have two immediate goals: categorize that patient into an established diagnosis, and assess the severity of the condition. It may be that the patient is depressed or anxious or even suffering from a potentially more debilitating condition like schizophrenia. What the psychiatrist is not trying to do, is define a new illness.

Occasionally — perhaps once per decade — a genuinely unusual patient will present themselves, and a new form of illness is considered. New treatments are found and tested. These treatments are rarely developed on behalf of individual patients; doctors work with grants from governments, medical schools, or the pharmaceutical industry and publish their results.

The vast majority of websites face similar problems. They deal with similar demographics, work within a similar culture, and deal with similar technology. As such, they can be categorized in the same manner a psychiatrist categorizes patients.

The key to delivering successful UX solutions is not UX testing in individual cases, but rather UX research, examining similar projects, and cribbing their solutions. If you categorize a project accurately, you’ll find a solution readily available.

Replacing User Testing With UX Best Practices

Your client doesn’t need to pay for UX testing to benefit from it. Enterprise sites, government sites, and even personal projects will test UX patterns. Sites like Shopify or Stripe will user-test their checkout processes at scale and enable companies to benefit from the results by adopting their platforms.

If you’re currently testing designs for small business, one of two things is true: either you’re wasting your client’s money investigating a problem someone else has already solved, or you’re designing something so original that it has no precedent (and you probably shouldn’t be).

Designers should be opinionated. Designers should know UX best practices and how they apply to a range of scenarios. Designers should be capable of making an educated guess. Designers should be self-validating.

Once or twice in your career, you may find a legitimate need to test something. However, the vast majority of the time, the correct answer is to tip the tortoise back onto its feet and choose whichever color button has the higher contrast.

Ben Moss

Ben Moss has designed and coded work for award-winning startups, and global names including IBM, UBS, and the FBI. When he’s not in front of a screen he’s probably out trail-running.

Read Next

6 Best AI Productivity Apps in 2024

There’s no escaping it: if you want to be successful, you need to be productive. The more you work, the more you…

3 Essential Design Trends, February 2024

From atypical typefaces to neutral colors to unusual user patterns, there are plenty of new website design trends to…

Surviving the Leap from College to Real-World Design

So, you’ve finished college and are ready to showcase your design skills to the world. This is a pivotal moment that…

20 Mind-Bending Illusions That Will Make You Question Reality

Mind-bending videos. Divisive Images. Eye-straining visuals. This list of optical illusions has it all. Join us as we…

15 Best New Fonts, February 2024

Welcome to February’s roundup of the best new fonts for designers. This month’s compilation includes some innovative…

The 10 Best WordPress Quiz Plugins in 2024

Whether it’s boosting your organic search visibility or collecting data for targeted email marketing campaigns, a great…

20 Best New Websites, February 2024

It’s almost Valentine’s Day, so this latest collection is a billet-doux celebrating the best of the web this month.

Everything You Need to Know About Image Formats In 2024

Always trying to walk the tightrope between image quality and file size? Looking to branch out from JPGs and PNGs this…

The 10 Best Logos of 2023 - Ranked

From vintage aesthetics to innovative new color schemes, we’ve seen a lot of creative logo designs this year. In this…

Exciting New Tools for Designers, February 2024

2024 is well underway and shaping up to be an interesting year. So, to help you keep your focus on working hard and…

The Art of Engineering AI Prompts

In the rapidly evolving world of artificial intelligence, the ability to communicate effectively with AI tools has…

15 Best New Fonts, January 2024

In this month’s roundup of the best new fonts, we find a large number of vintage, retro, and revival typefaces. Is it…