A Complete Guide to A/B Testing

A/B testing (also called split testing) is a testing method generally used in marketing to compare results between two samples with the goal to improve conversion or response rates.

In web design, A/B tests are generally used to test design elements (sometimes against the existing design) to better determine which design elements will get the best response from visitors.

A/B tests, by definition, compare only two variables (design elements) at a time. There is also multivariate testing, which compares more than one variable.

Each one serves a purpose and can help your clients make better decisions that will result in a more successful website.

Here we present a thorough guide to A/B testing, including the benefits of using it and how to administer A/B tests on your own projects.


Benefits of A/B Testing

A/B testing allows you to see how changes effect visitor behavior on a website. Many designers dive right into a new design, with or without a lot of research into current visitor habits, and hope for the best. While sometimes this results in a great final design, it can also result in a design that does no better than the original (or even worse).

A/B testing is a fairly low-risk approach to testing out website changes. While it can seem complicated, there are plenty of tools out there that can help you administer A/B tests, as well as interpret their results.

Another major benefit to A/B testing is that it can be used as proof to convince a client that one design choice is superior to another. This is particularly useful when working with a client who wants evidence backing up every decision they make, or a client who has trouble making decisions.

If you can offer them concrete proof that one design works better than another, they’re often much more comfortable making a decision.


When to Use an A/B Test

As already mentioned, an A/B test can be very useful for convincing clients who can’t decide between two design options. But it can also be helpful to designers who are unsure which of two options will work best for their client.

There are so many articles and studies released every day that tell us how to maximize results for our clients by using one design style or another. And in a lot of cases, the information contained in those articles could be conflicting when compared to an article from a different source, or released at an earlier date.

A/B tests let designers test out the different theories and recommendations in the context of their own projects, so they can decide for themselves what works best for their clients.

Full site redesigns aren’t the only time A/B tests can come in handy. They’re especially important when running a promotion or other marketing exercise. When a client wants a page designed for a particular sale or other promotion, they expect you to design something that will get results.

By setting up an A/B test, you can determine the likely success of different design elements, wording, or layouts so your client ends up with the highest number of conversions.


Things to Test

There are a variety of elements in a website design you might consider testing. Here are some of the most common:

  • Color scheme
  • Copy text
  • The general layout
  • Images
  • Headline copy
  • Text size or font

An example of a test on header images. Everything but the image itself is kept the same between tests. Images by Per Ola Wiberg – Powi and aussiegall.

Virtually any element of a website can be tested with an A/B test, though you may only want to test the most important elements (like copy, color scheme, or headlines), for both time and money’s sake.


How to Set Up an A/B Test

A/B tests consist of a few parts. Setting one up is relatively simple, especially with some of the tools listed later on in this article. There are a few basic steps included in most A/B tests:

  • Set up the two designs you want to test.
  • Randomly show one design or the other to visitors or a test group.
  • Track performance, especially related to the site’s goals, for each design.
  • Evaluate the results and decide which version to go with.

You’ll want to set up a method for tracking the results you get from each design, beyond just an analytics program. If you’re testing in a production environment, you may not have even numbers of visitors seeing each design (though they should be close).

Make sure you figure out the percentage of visitors who reach the goals, not just the concrete numbers if there’s a difference in the total number of visitors who saw each design.


Change Only One Thing at a Time

This is often the hardest thing for many designers. In true A/B testing, you should only be making one change at a time, or only testing one thing at a time. That means testing each element of the website—navigation, header design, content layout, color scheme, etc.—separately.

The test example above tests the font (Georgia vs. Verdana) of a block of body copy.

The point to testing each thing separately is to make sure you’re getting accurate results on how each design element on the site is affecting the visitor. If you change everything all at once, you don’t know if it’s the entire design that’s improved your traffic numbers (or made them drop off) or just one element. In the case of a drop in traffic numbers or sales, it’s important to be able to isolate what isn’t working.

Let’s say, for example, that you completely redesign a shopping cart on a site. Everything is different: the call to action buttons, the checkout experience, the way visitors have to enter their payment and shipping information, etc. And then let’s say there’s a big drop in sales. The problem with this is that you don’t know what caused the drop.

Sure, it could be the fact that everything is just different and return shoppers aren’t as comfortable with the new design.

But maybe it’s only because you used some cutesy wording on the “add to cart” button and it’s confusing people. If you’d tested that button separately from the rest of the shopping cart, then you could change the wording and increase sales. But instead, the client wants you to put everything back just the way it was, and they think you’re incompetent.


A/B Testing Takes Time

A/B testing isn’t something you can generally complete overnight, though it depends on exactly what you’re testing. For something simple, like a header image, you might be able to only run a short-term test. But for larger changes, especially those that will have a direct impact on conversions, you’ll want to let the test run for longer.

Determining how long to run a test is often simple. Look at the traffic patterns on the site. Most sites have cyclical traffic patterns, with some days consistently getting higher traffic than others.

For some sites, this cycle will run over a one-week period, while for others it might be a month. If possible, run your A/B test over at least one cycle to get more accurate numbers.

The goal is to get a good cross-section of visitors testing the new design options. By paying attention to the traffic cycles, you’ll be more likely to get that cross-section. If the site in question doesn’t have identifiable traffic patterns (or if they’re much longer), then try to run the test for at least a week.


How to Convince Your Clients

Sometimes clients are resistant to the extra time and money involved in running a proper A/B test.

They often think that as a designer, you should already know what’s going to work and what won’t for their website. Sometimes they think as a business-person, they already know what will and what won’t work. In either case, you need to convince them that an A/B test can help support those theories with concrete evidence.

Stress the benefits of running an A/B test. Tell them that it will help ensure their visitors are happy and more likely to purchase something, sign up for an account, or download information.

Stress that spending a bit of time and money up front on a good split test could result in much higher conversion rates in the future. Also stress that a good split test can also save time in the long run, as there will likely be fewer tweaks to the design once the site launches.


Tools for Easier A/B Testing

As already mentioned, there are tons of great tools out there for administering A/B tests on your website designs. Here are some of the best (feel free to share more in the comments):

Google Website Optimizer
Google offers their Website Optimizer as part of their Analytics package. It’s a free tool that lets you run A/B or multivariate tests. They also offer information on how to test and how to get the best results.

Visual Website Optimizer
Visual Website Optimizer is an easy-to-use A/B testing tool that’s used by both businesses and agencies. You just create multiple versions of your website, define what your visitor goals are (download, sign up, purchase, etc.), and then they split your traffic between the different versions. There’s a free trial where you can run a single test on up to 1000 visitors; paid accounts start at $49/month (for up to 10,000 visitors tested and up to 3 simultaneous tests). Visual Website Optimizer also has a number of free tools you can use, even without using their service: A/B Split Test Significance Calculator, a Landing Page Analyzer, the A/B Ideafox – Case Study Search Engine, and the A/B Split and Multivariate Test Duration Calculator.

Vertster is made specifically for multivariate testing, not just A/B testing. Their biggest advantage for agencies is that they offer private labeling of their technology, so you can offer your own testing solution to your clients.

Press9 A/B Testing Joomla Plugin
This plugin lets you run A/B tests in Joomla without the use of any outside service. It’s easy to use and can be run on any element of a Joomla-based website.

Amazon Mechanical Turk
While not specifically an A/B testing tool, Mechanical Turk can easily be used to find visitors for A/B or multivariate tests, often for only pennies per visitor. You’ll need to handle the technical aspects of the test, but it can solve the issue of finding test subjects.

Split Test Calculator
For the mathematically-challenged, this simple calculator can tell you which of your tests performed better if the total visitor numbers are different. Just enter the total number of visitors and the number who met your goals for each group of visitors and it will calculate which one did better.

ABtests.com lets designers share results from their own A/B tests, and view the results of others. This allows designers and developers to learn from what others have already tried, as well as sharing their own results to help others.

An Alternative Option
As an alternative, you can always use any number of regular usability testing tools to run an A/B test, even if they don’t officially offer the service. All you’ll need to do is set up two tests, and monitor the results from each. This is a great option if you already have a favorite usability testing tool but want to expand into A/B testing. If not, here are a few you might consider:


Multivariate Testing

Multivariate testing is similar to A/B testing, but includes more options. Where an A/B test compares two things, a multivariate test might test three, four, or five different designs.

If you opt to use multivariate tests rather than just simple A/B tests, it’s still a good idea to only test one element at a time. In fact, the more options you include in the test, the more complicated interpreting results becomes, and those complications are only exacerbated by testing more than one element.

Multivariate tests can be particularly useful if your client is unsure of how they want their site designed. You can test two or three completely different website mockups and see which one performs the best.

You may then want to run A/B tests on specific elements within the winning design to make sure it’s optimized as well as it can be.


In Review

Here are the basics you need to remember when running an A/B test:

  • Test only one thing at a time.
  • Allow ample time for testing.
  • Use available tools to make A/B testing easier.
  • Use the results to help your clients make better decisions.

Written exclusively for WDD by Cameron Chapman.

If you have more A/B or multivariate testing tips, techniques, or tools to share, please do so in the comments. We’d also love to hear success stories resulting from A/B tests!

  • http://trimbakeshwar.in/ trimbakeshwar

    WAW,its so nice article…
    i m getting new ideas for testing here…


  • http://kaplancopy.com/blog Jodi Kaplan

    It’s also important to make sure you have enough views to make the test statistically significant. For instance, 50 people is too small a sample to be meaningful.

    And yes, it’s very important to change only one thing at a time!

  • http://www.codablog.fr/ Antonin

    Hello, thank you for this article.
    I am wondering if a A/B Testing could be useful for a website with a low audience (150 visitors / day). I am afraid that to have a good overview, I will need to proceed the test for a very long times.

  • http://www.performable.com Joshua Porter

    Thanks for the nice article, Cameron, and thanks for including http://www.abtests.com in it (I’m the co-creator of the site).

    I wanted to add that while testing one thing at a time is a more scientific approach, it’s not always practical for design teams who need to move fast. That’s why I’m seeing design teams use A/B testing to test wildly different designs (and copywriting) to start…in which many variables are different…because this gives you relatively large differences in conversion.

    Then, after you’ve gotten the big parts of the design solidified, designers can start optimizing one thing at a time. Though not as scientific…it’s *much* faster and for some teams speed is everything.

    Thanks for the great post!

    • http://www.kaplancopy.com/blog Jodi Kaplan

      @Josh, do you mean multivariate testing all at once (A/B/C/D/E) or testing multiple changes on one page? Not sure how the latter could be useful. If you’ve changed say the headline, the font, and the button color how do you know which change made the difference?

      • http://www.valuepervisit.com David Oh

        @Josh totally agreed. The latter is not “useful” in terms of specific education about what variables might have affected things, but the latter may be “useful” in that it avoids being placed into a local maxima.

  • http://www.bisondisc.com an avid reader

    Such a Great article! very informative and well worth a bookmark for later referencing… Thanks!

  • http://yuliantip06.student.ipb.ac.id yulianti

    that’s A/B Testing is same with Mirror Testing?

  • http://www.virtuosimedia.com Benjamin @ Virtuosi Media

    Kudos for an article on testing, Cameron. I think that this needs to be more of a focus in the web design community. A few other points that maybe could be mentioned:

    -Test the glaringly obvious things first. Don’t waste time testing the little details until you’ve tested the major elements of the page. Work your way down the list of importance.

    -If you have low traffic volume, you can use Google AdWords or similar advertising to temporarily drive traffic to the test page. It’ll probably cost a few dollars, but the testing can be done in a much shorter time frame.

    -A few other things to test if you’re selling a product: The price or the product offering itself.

    -Testing shouldn’t be a one-time proposition.

    Great job on the article.

  • http://www.mobilebroadbandguru.co.uk mobile broadband guru AK

    that is wonderful article.

    A lot of time we never think of doing any testing but after reading this. A/B testing is worth doing it.

    Again thanks for such a nice article

  • http://knowledgecity.com Jae Xavier

    Convince business personnel to invest time and money into A/B/Multivariate testing like this…

    Speak their language

    1) Short term benefits
    2) Long term benefits
    3) Provides better risk management
    4) Proving assumption once and for all
    5) Unearthing a hidden competitive advantage their competitors might not have
    6) And finally just like in marketing, test test test

    Business folks want to take action before analysis paralysis leads to wasting time and analyzing until your blue in the face.

    Action, action, action – that is what’s on their minds.

  • http://www.blueseolosangeles.com/los-angeles-web-design/ Jim

    i have heard of people testing there websites or more specifically their landing pages for PPC advertising.

    i know for a fact if you take the time to split test you can increase your conversion you just have to put in the time to do so.

  • http://jayphen.com Jayphen

    I have played around with Google Website Optimizer & Visual Website Optimizer a little bit, but never gone in to too much depth. One thing that has always concerned me about Javascript-based A/B testing is the fact that the test is run on the client side. Does this not result in a flash on the page as the content is replaced & reflows? There has to be at least a slight delay while the page connects to Google or Amazon S3 (in the case of VWO). Isn’t server-side A/B testing a much better solution?

  • http://www.marcvonbrockdorff.com Marc von Brockdofff

    Great article! I only started AB testing a few months ago and it’s amazing just how effective some little changes can be.

    For anyone interested, I recently wrote about statistical significance on my blog, which is basically what tells you whether you have a clear winner or whether A/B is leading by random chance. Have a look: http://www.marcvonbrockdorff.com/testing/ab-testing-statistical-significance/

    Thanks again!


    Great!Thanks very much!

  • http://www.2bscene.ie Tom Doyle

    Great article and to the point. One thing I’ve learned is what Jodi says – You need to test enough people to make sure it is truly a worthy exercise.

    Convincing clients to spend time on split testing is probably the hardest part of it all. Some great tips from others comments here.

  • http://www.twitter.com/shoppr Tim Deijnen

    Great article!
    We’re going to use Visual Website Optimizer for the first time on one of our websites this week. It really looks like an easy to use tool!

  • http://www.seoessex.co.uk/ SEO Essex

    This is a very informative article. A brief, yet thorough guide on performing an effective Split test. Plus, the tools presented here can optimize the testing process. The results of A/B Testing can greatly boost a site’s profile.

    Thank you for a wondeful post.

  • http://www.michaelsaathoff.com Michael Saathoff

    awesome article – i plan on trying Press9 A/B Testing Joomla Plugin on my next project!

  • http://www.conceptfeedback.com Andrew Follett

    Hi Cameron,

    Thanks for including http://www.conceptfeedback.com! Using a qualitative tool like CF can generate a lot of new ideas for testing purposes.

    Here’s a nice post I came across this morning from Jason Cohen on A/B testing, advocating an A/A2/C approach (the C being something radically different). Worth a read – http://blog.asmartbear.com/local-minimum.html

    Andrew Follett

  • http://www.reedge.com Dennis

    Try also our tool for A/B testing (or multivariate/segmentation/personalisation). http://www.reedge.com its free

  • http://ab-testing.co.uk/ Scott Herbert

    I’m looking at building an A/B testing suite and am interested in what people think of the subject (what sucks and what doesn’t).

    If you’ve got the time (a few minutes at most) I’d really appreciate your views at ab-testing.co.uk

    Thanks. (and sorry for the shameless plug)

  • http://www.redcardfc.com Football Gifts

    A great blog and a number of interesting ideas that I can use in the future. I have used a similar concept for Google Ads with advert variations and then comparing results. Having said that there are a number of things here I would not have thought of.

  • http://www.urbansurfclothing.com Matt

    We have a dark them on our surf clothing affiliate site as we wanted something different to other affiliate sites but only get 2 visitors a day so still insure wether a design change will show any good results. Wud luve 2 do sum a/b testing after reading this, thanks for explaining also finally what google website optimizer is to me!

  • http://www.applenyt.dk Tommy

    Very useful article! I’m definately going to try it out!

  • http://www.essexseo.co.uk Essex SEO

    Very interesting article, we have used A/B split testing on some websites, but it is very hard to convince the customer this is the right thing to do.

    I’ll try out your pointer next time I have to convince a company of A/B testing.

  • http://stellarfilerepairsoftware.weebly.com/ Suzane Brandy

    It’s really worth reading article…i was lil bit confused about A/B testing, now my concept has been cleared.

    Many Thanks!

  • dude

    Try optimizely! It’s awesome.

  • http://whichtestwon.com/ Natalie Tomasso

    WhichTestWon is also a great resource if you’re interested in A/B or multivariate testing. We publish a new test each week. Many are exclusive, so you won’t find them anywhere else. It’s a great way to get some inspiration and data to inform new tests.



    Natalie Tomasso
    Senior Reporter