The Case Against UX Testing

Ben Moss.
January 17, 2022

Picture a dark office, blinds drawn. Picture a UX designer smoking a cigar. See the light filtered through the smoke whipped to fog by a spinning ceiling fan. Watch as the UX designer sits at a desk and considers the website.

The Case Against UX Testing.

The UX designer has devised a series of tests to determine if a green button is better than a red button. One of them involves tipping a tortoise onto its back. He looks the website over carefully and says, “Describe in single words, only the good things that come to mind about your mother.”

The website pauses, sweating under pressure, then replies, “Let me tell you about my mother…”

BLAM! The website pulls the trigger of an unseen gun, and the UX designer collapses, leaving the project to be rebuilt from scratch in Material by Harrison Ford, with overuse of Post-its delegated to Edward James Olmos.

Who Does UX Testing Actually Serve?

In the past’s bleak dystopian future (1982’s Blade Runner was set in 2019) no one benefitted from asking the wrong questions. And little has changed.

Designing any test to verify UX is fraught with as many complications as administering the test. Questions are skewed by bias, conscious or otherwise, and competing agendas. Even with something as apparently simple as a split test, the potential for distortion is immense.

When planned by a designer, a UX test offers little benefit to a client; the benefit is to the designer, who can then say their ideas are validated (or not).

Imagine hiring a developer to code a website, only to discover that the developer didn’t know CSS and expected to be paid to learn it before completing the work. You would hire someone else because that developer isn’t qualified.

From a client’s perspective, a UX designer should know, through experience, whether a green button is better than a red button. Designing an elaborate test to split-test the button color serves little purpose other than indemnifying the designer against mistakes.

The ROI of UX Testing

It’s widely accepted that there is substantial ROI (Return On Investment) from UX testing. We’ve all heard apocryphal stories about sites that split-tested their checkout and improved retention by 5%.

I’m going to go out on a limb and say that without user testing, that site could have improved its checkout retention by 4.9% simply by hiring a competent, experienced designer. But what about the remaining 0.1%? Well, for most sites, 0.1% represents very little profit. And the cost of recovering it via testing far exceeds the benefits.

When a company the size of Amazon, Netflix, Spotify, or Google split tests a website, it can afford to allocate $25k for user testing because it stands to gain 0.1%, and that represents far more than $25k. To meet the same 0.1% improvement, a small business has to design and run the same tests, incurring the same costs. But in the case of a small business, $25k could eat up all of its profits.

UX testing almost always works. But it is only profitable at scale.

If a good UI designer with a grounding in UX can improve checkout retention by 4.9%, tripling the project budget for just 0.1% more is a tough sell. Bluntly, that $25k is better spent on advertising.

What UX Designers Can Learn From Psychiatry

We all have the tendency to think we’re unique. It’s a survival trait attributed to our prehistoric brain. That belief in uniqueness is particularly strong in highly competitive people. We all think our site, our side-project, our approach are original. And we’re all wrong.

When a psychiatrist sits down with a patient, they have two immediate goals: categorize that patient into an established diagnosis, and assess the severity of the condition. It may be that the patient is depressed or anxious or even suffering from a potentially more debilitating condition like schizophrenia. What the psychiatrist is not trying to do, is define a new illness.

Occasionally — perhaps once per decade — a genuinely unusual patient will present themselves, and a new form of illness is considered. New treatments are found and tested. These treatments are rarely developed on behalf of individual patients; doctors work with grants from governments, medical schools, or the pharmaceutical industry and publish their results.

The vast majority of websites face similar problems. They deal with similar demographics, work within a similar culture, and deal with similar technology. As such, they can be categorized in the same manner a psychiatrist categorizes patients.

The key to delivering successful UX solutions is not UX testing in individual cases, but rather UX research, examining similar projects, and cribbing their solutions. If you categorize a project accurately, you’ll find a solution readily available.

Replacing User Testing With UX Best Practices

Your client doesn’t need to pay for UX testing to benefit from it. Enterprise sites, government sites, and even personal projects will test UX patterns. Sites like Shopify or Stripe will user-test their checkout processes at scale and enable companies to benefit from the results by adopting their platforms.

If you’re currently testing designs for small business, one of two things is true: either you’re wasting your client’s money investigating a problem someone else has already solved, or you’re designing something so original that it has no precedent (and you probably shouldn’t be).

Designers should be opinionated. Designers should know UX best practices and how they apply to a range of scenarios. Designers should be capable of making an educated guess. Designers should be self-validating.

Once or twice in your career, you may find a legitimate need to test something. However, the vast majority of the time, the correct answer is to tip the tortoise back onto its feet and choose whichever color button has the higher contrast.

Ben Moss

Ben Moss has designed and coded work for award-winning startups, and global names including IBM, UBS, and the FBI. When he’s not in front of a screen he’s probably out trail-running.

Read Next

15 Best New Fonts, October 2024

Welcome to our roundup of the best new fonts we’ve found online in the last four weeks. In this month’s selection we…

3 Essential Design Trends, November 2024

Touchable texture, distinct grids, and two-column designs are some of the most trending website design elements of…

20 Best New Websites, October 2024

Something we’re seeing more and more of is the ‘customizable’ site. Most often, this means a button to swap between…

Exciting New Tools for Designers, October 2024

We’ve got goodies for designers, developers, SEO-ers, content managers, and those of you who wear multiple hats. And,…

15 Best New Fonts, September 2024

Welcome to our roundup of the best new fonts we’ve found on the web in the previous four weeks. In this month’s edition…

3 Essential Design Trends, October 2024

This article is brought to you by Constantino, a renowned company offering premium and affordable website design You…

A Beginner’s Guide to Using BlueSky for Business Success

In today’s fast-paced digital world, businesses are always on the lookout for new ways to connect with their audience.…

The Importance of Title Tags: Tips and Tricks to Optimize for SEO

When it comes to on-page SEO, there’s one element that plays a pivotal role in both search engine rankings and user…

20 Best New Websites, September 2024

We have a mixed bag for you with both minimalist and maximalist designs, and single pagers alongside much bigger, but…

Exciting New Tools for Designers, September 2024

This time around we are aiming to simplify life, with some light and fast analytics, an all-in-one productivity…

3 Essential Design Trends, September 2024

September's web design trends have a fun, fall feeling ... and we love it. See what's trending in website design this…

Crafting Personalized Experiences with AI

Picture this: You open Netflix, and it’s like the platform just knows what you’re in the mood for. Or maybe you’re…