JeremySaid's Blog for Startups and Lead Generation

The 12 Most Common Excuses for Not A/B Testing (and Complete Destruction of Each One!)

Posted by Jeremy Smith on Jul 11, 2016 8:00:18 AM

Look, I know that testing can be a pain ...

Ever since elementary school you have been bad at testing, with teachers and parents reminding you of your shortcomings at every turn, probably causing severe mental damage that I will have to read about in the comments section. Sigh.

But this ain’t your teacher’s testing! You aren’t working on an SAT or a pointless algebra problem.

Your only grade is how much money you are making, and if you don’t want to improve that then I would kindly ask that you read another blog.

But seriously, the number of PATHETIC excuses I hear everyday about why people “can’t” get going with testing for their company website and/or marketing campaigns are absolutely appalling.

So in my traditional fashion …

I am going to demolish all of your excuses, ruthlessly, methodically, and one by one.


Let’s go!

Common Excuses for Not A/B Testing

1. I just don’t have the time to test.

Every time I hear this excuse I want to flip a table.

Cartoon illustration of two steps required of flipping a table: stand up, flip table

(Image source)

I’m sorry not sorry, but this is not an excuse at all!

Steve Jobs ran two multi-billion-dollar companies at the same time! Richard Branson owns more than 400 companies and still finds time to go kiteboarding around his private island.

Richard Branson kiteboarding

(Image source)

You have no excuse.

If you think you don’t have time to test, then I dare you to take a look at how you are spending your time each week. Download Rescue Time or a similar app and just look at how much time you spend on Facebook or watching cat videos.

Everyone has the same 24 hours in a day. If something is important to you, you will make time for it. It’s that simple.

I know this is kind of a lame rebuttal to your lame excuse, but I don’t have time to explain it further.

2. My company doesn’t allocate time for this.

No matter what phase you and your company are in, you need to be testing.

You may think that you need to focus all the efforts within your company on marketing, product creation, customer service, etc. But I promise that you have the time to do some testing.

It’s super-important.

If you are just getting started, this means doing market research. If you are in product creation, it means gauging the feedback of your audience to your offers, and if you have launched, it means testing the living crap out of your design, copy and everything else.

Just like it’s hard to lose weight if you don’t track your workouts and calorie consumption, it’s hard to make true business gains if you are not tracking the performance of your campaigns and products.

Besides, testing doesn’t have to take a lot of time. Once you align your data, buy the requisite software, and hypothesize about what should happen, you’re set to run the test.

If your company hasn’t allocated time for testing, you have a few options:

  • Allocate the time yourself. Take your lunch break or your research time, or skip out of a meeting that you don’t need to be at anyway, and do it. In an hour, you can easily set up your first split test.

3. We Don’t Have the Budget

Meme with Sean Bean as Boromir from the Lord of the Rings: The Fellowship of the Ring and legend: One Does Not Simply Have Enough Money"

Of the excuses we have examined so far, this one seems to be the most valid.

However, it it still false.


Testing does not have to be expensive.

One of the simplest ways to cut the cost is to downsize ads and marketing campaigns into smaller campaigns that are more spread out.

So, for example, instead of running one huge Facebook campaign, tweak the copy, change the design and create three new campaigns, but run each of them on a smaller scale.

While it may not be the exact same price, using this “split” campaign method is easy and cost-effective.

Or, you can do split testing in Google Analytics. It’s absolutely free!

Here’s a simple and easy-to-follow tutorial that will show you how to set up a kickass split test using the best and freest analytics tool on the planet.

4. I don’t know what to test.

Any time someone makes an excuse that starts with the words “I don’t know,” they make me want to put my head through a brick wall.


Mural of artist Vango's defeated Einstein seated with bottle of Jameson whiskey and legend: Just Google it

(Image source)

In this age of excessive information, not knowing something is just an excuse to be lazy.

Within 30 minutes, you can do more market research, discover more trends, and collect more data than whole firms could in a month back before Google.

Seriously, if you ever use this excuse around me …

Just don’t, please.

Here are some lists of things to test to get you started:

Just pick something.

5. We’ve got this figured out. We don’t need to test.

Sounds like someone needs to eat humble pie.

  • I test.
  • Fortune 500 and 1,000 companies test.
  • All billion-dollar corporations test.

What makes you think that you know more than all of them?

Sure you may have found something that works for you, but devoting a small team or small budget to regularly testing your stuff will help you discover chinks in the armor or unexploited opportunities that will make you loads more money.

At the risk of coming across as rude, I’m going to say it: You don’t have this figured out. You need to test.

6. I’m happy where I am.

Look, if you have been working hard in business and finally gotten to a point where you are content, then I am happy for you. This is a great place to be.

But what if you could improve your sales by 28 percent by changing two words?

Case in point:

Screenshot of example spilt test results from VWO site

I love using the example above, because it shows how simply changing two words in an ad can change the conversion rate by more than 25 percent (18.6 - 14.5 = 4.1; 4.1 is a shade more than 28 percent of 14.5).

Screenshot of percentage calculator indicating that 4.1 is 28.27 percent of 14.5

Here’s the point where I could throw in a half dozen disclaimers about “don’t expect the same results,” and “don’t copy others’ tests,” and etc.

However, the point I want to make is this: There is always room to improve. Always.

Even the unicorns that are enjoying sky-high conversion rates can and should improve their game and boost their rates.

The highest possible conversion rate is 100 percent – everybody converts. How far are you from that rate? Okay, then. Go out, do some tests, and see what will make it higher!

Imagine if only a few hours of work a month could yield these kind of results for you.

They can and they will if you will get over your excuses.

7. We don’t have a big enough audience to test.

Meme with distraught James Vanderbeek and legend: When she says it's not big enough

Ok, this one is probably the most valid of all the excuses, because, quite simply, if you don’t have a big enough sample size, then the results from your tests are statistically irrelevant.

You typically need sample sizes of 5,000 to 10,000 to achieve any results of real statistical significance.

But, even if you are starting with a list of only 2,000, you can still get a good idea of where to go.

While your focus at this point in your business should definitely be to boost your traffic and subscribers, doing some basic tests with the list that you already have will help you to achieve that goal.

Besides, there are some great techniques for doing optimization on low-traffic sites. Here are some resources:

Excuses, anyone?

Didn’t think so.

8. I used to test, but it doesn’t work.

Look, I get it, you ran some testing, couldn't find a clear winner, and now you are a jaded entrepreneur who believes that testing is a bunch of useless hocus-pocus.

Here’s the thing, peeps …

Regardless of the results from your test, you can always learn something. That never hurts.

Homer Simpson and quote: Every time I learn something new it pushes some old stuff out of my brain.

No matter how your test performs (or doesn’t perform), you can pick up little nuggets of wisdom about how to move forward, even if the only nugget is that your last test sucked.

Testing is about learning. Sure, you might not see that 9,000 percent increase that you were hoping for.

But you learned something.

That lesson will form the foundation of your next split test.

And the next.

And eventually you will start seeing gains.

9. Testing will hurt my conversion rates!

Meme with Keanu Reeves from Bill and Ted's Excellent Adventure and legend: Woah, dude that really hurt

Do you think that split testing is going to hurt your conversion rates?

No, it won’t.

Really … It won’t.

If anything, doing split tests will help your conversion rate. With modern testing software, once the test has been running for a few hours, the software will determine what is performing poorly and halt it, and start to boost the test that is increasing results.

Besides, a temporary split test that produces a dip in conversion rates will not matter much in the long run.

Testing is a long-term thing. You’re not just doing one test and then quitting.

You’re going to be running test after test.

Every test will have a winner. Every test will have a loser.

By picking the winner and eliminating the loser, you will gradually increase your conversion rates.

Not lower them.

10. Testing won’t guarantee results.

Damn right!

And getting into business, doesn’t guarantee success, but you did it anyways.

Life is more fun when things are not guaranteed; it adds to the suspense.

Nothing is guaranteed.

Gather what you know and create an informed experiment. In the end, that’s the best you can do.

11. I’m just not ready to start testing. (I don’t know enough. I don’t have the skills, etc.)

Were you ready to start a business when you did it?

Were you ready to take your first job?

Were you ready to take that trip?

Were you ready to go to college?

To have a child?

For all of the above, the answer is probably no.

What did you do?

You just started.

We seldom if ever feel truly “ready” to do anything big or scary, even when it has a positively awesome outcome.

You are never ready, and there is never a good time to start, but you need to do it anyways, so plug your nose because here comes the cold water.

As Nike said … Just Do It.

12. The new season of Game of Thrones is out, I really don’t have time.

Game of Thrones meme with legend: A man does not patiently wait for Season 5 of Game of Thrones, a man instead marathons seasons 1, 2, 3 and 4 for the third time

The only valid excuse on this list. Go enjoy your show, speak Dothraki to your significant other and munk on some horse heart.



There really is no reason not to test.

Sarcasm and ranting aside (it’s my style), testing is one of the most important parts of online business, and every single day that you do not test is another day that you are losing money and stunting the growth of your business.

Testing is one of the only things on this planet that I can look at and honestly say, “That is a really good thing.”

Testing is just awesome.

So get out there and get some tests going!

Topics: A/B Testing, A/B Testing, Conversion Optimization, Split Test, Split Testing

Want to hear more from us? 


Subscribe to the JeremySaid blog for Startups and Lead Generation.


You'll be notified as soon as we have something new for you to read!

Subscribe to Email Updates

Recent Posts

Posts by Topic

see all