JeremySaid's Blog for Startups and Lead Generation

How to Organize a Split Testing Map for the New Year

Posted by Jeremy Smith on Jan 12, 2015 12:56:43 PM

Split Testing Maps

Whenever a New Year rolls around, everyone wants to lose weight, eat right and get organized.

Those are great aspirations. I want to help.

My goal in this article is to explain how you can achieve greater organization in your split testing.

If you’re like a strong percentage of Internet marketers, you want to start the New Year with a spate of great tests. A/B testing is the way to get actionable results and boost your conversion rates.

But where do you start?

Organization is the missing plank in the bridge to "split testing" success. You want to get across and experience all those wonderful benefits that everyone talks about, but how the heck do you get from here to there?

The missign plank in the bridge to split testing success

After spending some time with this article, you’ll gain the framework for organizing your split testing in a manner that will pave the way to success. No missing planks.

Why is it important to organize your split testing?

First of all, let me explain a few of the more notable reason why you need to organize your split testing.

I’m a make-it-simple-for-me kind of guy, so let me just tell you why you need to organize your split testing approach in a single sentence. Then, I’ll break it down into ten points.

You need to organize your split testing, because it is the fastest and most direct way to gain higher conversion rates and revenue.

On the flip side, a disorganized A/B testing approach will give you shoddy, haphazard, and scattered results. Sure, you’ll be able to nudge the needle upward on your conversion rates.  But  you won’t be able to competently increase conversion rates with the satisfactory upward trajectory that your team desires.

Disorganization breeds lack of success.

Disorganization leads to discouragement.

Disorganization is no way to do your split testing.

Here’s why you need to get organized.

1. Split testing is at the core of conversion optimization.

This is the most important activity that you’ll ever do as a marketer. Don’t screw it up. Get organized.

2. Split testing gains value as it gains longevity, but only if it’s organized.

One of the things that’s amazing about conversion optimization, is that it gets better as time goes on. (Like wine.) Split testing is the same way. The more you do it, the more tests you run, the closer to perfection your website (and conversion rates) will become:

graph_Increasing number of complementary methods used, tests and segmentation

The thing is, this whole maturity model only works if you organize your approach! Notice the “structured approach” label in that final bar in the chart above.

3. An organized approach to split testing allows you to focus on the most valuable aspects of conversion optimization first.

You know the whole low-hanging-fruit cliché? Yeah, well it’s a thing. You do want to focus on the easy wins. Organized testing helps you do that.

Video marketing tip_grab the low-hanging fruit

4. Organizing your split testing helps you to know which areas of conversion to address sequentially.

Not all areas of conversion optimization have equal weight in the pursuit of higher conversion levels. You want to address these areas in a logical and sequential manner. I’ll tell you exactly how in the sections below.

5. Organized testing saves you money.

Yes, you will save cash by being organized. More cash. More happy.

6. Organized testing creates consistent upward movement of conversion rates.

If you never see results, you’ll quit doing something ... even split testing. I encourage you to develop an approach to testing that insists upon results. Organization will get you there.

7. Organized testing provides you with organized data, and a lot of good things will come from that.

Data is the air we breathe as marketers. The power of organized split testing is revealed in all its glory by the huge breadth of data that you’ll gain — organized data!

8. Organized split testing helps you work better as a company and as a team.

Conversion optimization isn’t something that you do alone. Most conversion optimizers work in consort with copywriters, SEOs, developers, designers and other team members. Each of these producers must be able to respond to the results of the test by implementing changes and rolling out new iterations.

For their sake, as well as for the cohesive organization of the entire organization, it’s best to have a streamlined approach to split testing.

9. Organized testing keeps you from wasting your time.

If you don’t organize your testing, you’ll run afoul — mucking about in the disorganized mess of wasted time.

Split testing is a highly time-sensitive affair. Tests must start and end on schedule. If you don’t know what you’re testing next, you’ll probably spend more time hemming and hawing then testing.

10. Organized testing keeps you from being discouraged.

I don’t know about the rest of the human population, but if I don’t see results, I tend to get discouraged. Normally, I’m a glass-half-full kind of person, but not seeing results is something that irks me. I think it’s in my DNA.

This is one of the reasons I can’t stand a disorganized testing approach. Disorganized testing leads to haphazard results, which leads to unpredictable results, which leads to discouragement. Organized split testing, on the other hand, is a powerful way to ramp up results month after month, year after year.

What should you split test first, second, third?

So far, I haven’t even touched on the actual split-testing map. How do you get organized?

First, you need to discover where you’re going to test. I suggest this approach with a word of advice.

The Advice

You want to start testing things that have a significant impact upon your revenue. For example, if you do a lot of email marketing, then it makes complete sense to start with email marketing split testing. If you don’t even do email marketing, then you can skip to PPC ad testing.

Choose to test first an area that you are both familiar with and that will affect revenue.

The List

Here is a suggested order of testing. Remember, you should spend several months in each category, conducting a series of tests.

  1. Email marketing split testing
  2. PPC ad split testing
  3. Landing page split testing
  4. Conversion funnel split testing
  5. Checkout page split testing

Now, I’ll expand on each of the above.

1. Email marketing split testing.

Email marketing is one of simplest and easiest things to split test. Most email marketing programs have built-in split testing functionality, which makes it easy for you to set up and run a split test.

Another advantage to starting with email marketing is that you stay very tactical in your approach. Emails are structured entities, with subject lines, greetings, body copy, conclusions and other features that you can test regularly and sequentially.

Email marketing remains one of the most powerful sources of marketing available in the digital age. Because it is such a source of great leads and revenue, you’ll really profit from A/B testing and optimizing your approach.

2. PPC ad split testing

For many marketers, the PPC ad spend is where they spend most of their marketing budget. That’s why I put it early in this list. If you’re spending a lot of money on getting your ads in front of people, then you need to spend a little bit of money getting those ads to get higher CTRs.

It galls me when I see a company throwing thousands of dollars into ads, but not A/B testing their ad copy or headlines. This is a wasteful game to play. Start testing as soon as you can.

3. Landing page split testing

Once you get visitors to your page, you need to convert them. This is where I spend most of my time and cognitive effort — turning landing pages into powerhouses of conversion.

Landing pages are what most people think about and talk about when they consider split testing. Please be aware, though, that split testing is more than just testing your landing pages. Landing pages are part of an overall marketing approach, and do not constitute its majority.

Yes, you should test and optimize your landing pages. However, don’t get hung up on landing pages to the exclusion of other areas where you can split test. There are a lot of other things that you can test and optimize.

4. Onsite conversion funnel split testing

Let me explain what I mean by the “onsite conversion funnel.” I’m referring to the path on a website by which a customer reaches the checkout page.

The conventional conversion funnel refers to a broad and general path of attracting customers. The conventional conversion funnel In my usage, however, I’m referring to only those things on the website that lead to conversions.

This is an example of how Nike might want to test their organization and test their onsite conversion path. We start with the main page:

Testing Nike home page
They should test the copy and buttons on the main slider. Assuming that a customer clicks on “Shop Ohio State,” they can start testing their Buckeyes paraphernalia page — perhaps “buy” buttons vs. the CSS hover effect out that they currently have.
Testing Nike product page
Then, they test this page, perhaps with a more prominent buy button.

Test nike shopping cart
And on it goes — through each step that a customer will follow as she buys a Buckeye jersey.

5. Checkout page split testing

Finally, you’ve reached the endpoint — testing the checkout page. This is an important part of testing, because it is the end of the spear. I can already hear the ominous thump-thump-thump of the drumbeats in the distance, pounding out the dark message of SHOPPING CART ABANDONMENT.

Checkout page split testing, meme

It is important to stave off shopping cart abandonments that happen during the checkout phase. You should also test the buy-related features of the page to ensure that you are converting a maximum number of users.

6. Minutiae

Save all your curiosity testing for a final stage, like when you’re bored or feeling lazy. Testing minutiae takes time and effort away from your primary objective, which is to win conversions. If it’s not a major source of conversion potential, then it’s not worth your time.

Your site's "404" pages might fall into this category, along with "unsubscribe" pages.

Taken as a whole, this organized process gives you a snowball effect. First, you start the trickle by improving your email marketing. Then, you improve your ad CTRs, getting additional traffic to the site. With higher traffic and a stream of higher conversions, you fine-tune your landing pages, driving even more people through the conversion funnel, split testing as you go. Finally, you wipe away any final form of friction by testing your checkout page.

The results build up, the conversions increase, and you win. Why do you win? Because you’re freaking organized!

The importance of being organized when split testing

Organizing Your Map

To fully organize your map, here is what you need to do.

I recommend that you start with A/A testing. Technically, this isn’t split testing, but it is the important precursor to split testing. The power of split testing can only be accurately revealed when you are able to create a benchmark of performance.

Possessing A/A testing results allows you to sidestep some of the common areas of split testing failure, see through aberrations, and better interpret the A/B test results.

  1. Choose an area to test (e.g., email, PPC, etc.)
  2. Select a time frame for testing this area. I recommend a minimum of three months.
  3. Create a list of features to test. Each area will differ, but I suggest the following
    1. Headlines, if applicable. For email marketing, this would be subject lines.
    2. Copy changes.
    3. Layout changes.
    4. Button changes
    5. Image changes.
    6. There are a bajillion different things you could test, so stay flexible. Make this list realistic but creative.
  4. Plan a duration for each test.
  5. Plan a pause after each test. This is to allow you to analyze your results, determine the statistical validity of the test, make any necessary changes, and prepare for the next test.

Your entire testing map should be a part of your work calendar. Here are dates and features to write down:

  • Exact test to run. Make it very clear what you’re testing. “Subject lines” won’t do. Instead, you need to be specific about what aspect of subject lines you’re going to test, e.g., personalized subject lines vs. non-personalized subject lines.
  • Software to use (It may differ, based on what you’re testing)
  • Test start date.
  • Test end date.
  • Analysis period. This gives you time to unravel the results and compare with your hypothesis.
  • Implementation period. This time gives the designers, developers and other contributors a chance to put changes into action.
  • Set up for new test period. During the setup, you’re working with your software and preparing your hypothesis.

This map becomes your most powerful tool in your marketing arsenal.

Rules of organization

Start simple.

You want to succeed at split testing. Do yourself a favor. Keep it simple, especially at first.

Now is not the time to whip out your multivariate full factorial discrete Taguchi response surface second-degree polynomial test with incrementally compounded multi-level variable conditional independencies. You might feel smart, but you’re actually not. example of a multivariate full factorial discrete Taguchi response surface second-degree polynomial test If you can tell what the graph above is communicating, then feel free to do something other than split testing.

Run only one test at a time.

Along with this commandment of keeping it simple, I also hereby command you to run only one test at a time.

I know, you’re all excited about split testing (and being organized), like a kid with a shiny new ten-speed bike. But hold on. You don’t want to run multiple tests at once. You want to stay organized.

Excitement can kill organization.

Test in just one area at a time.

The excited test-all-the-things approach may leave you wanting to test many different areas — email, landing pages, checkout pages, etc.

Test in just one area at a time, meme

This may be okay once you become a testing ninja. If you are not yet a ninja, do not test all the things at the same time.

Make a list. Follow it.

Here is my most sacred tip of all:  Make a list and follow it.

The list is your guiding light of split testing awesomeness. Follow it, and you will succeed. Ignore it, and you will plunge into an abyss of discouragement and possibly not petting your dog as often as you should.

Life on the edge.


If your split testing isn’t organized, then you aren’t as successful as you could be. That’s the bottom line truth of the situation.

So, here’s the takeaway:

  • Big concept:  Organize your split testing.
  • Tactical takeaway:  Create your own map.

Let’s kill it this year with a super organized approach to split testing.



Topics: A/B Testing, Conversion Optimization, Email Marketing, For beginners, Landing Page Optimization, Split Testing

Want to hear more from us? 


Subscribe to the JeremySaid blog for Startups and Lead Generation.


You'll be notified as soon as we have something new for you to read!

Subscribe to Email Updates

Recent Posts

Posts by Topic

see all