A Tale of Two Tests:
How to Structure an A/B Split
A marketing test is normally made up of two or more
"candidates," called creatives. The creatives
may be different in their look, appeal, offer or price.
The test can either be run using an A/B split (for more
information on split testing, visit http://www.vertster.com,)
or following a "one after another" approach,
proceding until one of the test members demonstrates
clearly superior performance.
When testing a new creative against an existing ad,
the older candidate is referred to as the control. It
is the element in your test that you have used the longest...
the control is what you have always done. You should
also have good documentation of how well it performs-
an ongoing accounting of its conversion rate.
The new candidate (or candidates because you can have
more than one,) is sometimes referred to as the "test."
This is somewhat confusing- because the whole process
is called the test, and the test is made up of a test
member and a control member. Thats a lot of "tests!"
In order to test against the control, you should have
a thorough understanding of how well it works, and a
documented history of its' performance. Without being
able to quantify its' effectiveness, it will be impossible
to determine if the test member is really better.
Of course, before you can run a test you have to consider
what to test.
What if you don't have a control?
If you do not have a control, then your first objective
in testing will be to try out some different ideas,
and establish one.
The circumstance of not having an established control
is actually quite common. It is handled by designating
all available candidates as "tests." Your
objective is to evenly test them against each other,
and determine a control for future activities.
One reason for not having a control would be if you
have not kept accurate records of your current ads performance
(ie you have no idea exactly how well it converts.)
This is very common, as most business owners don't understand
how or what to use as a measure of success online.
Another common occurance: launching a new product,
offer, or campaign. In this case, there is no history,
and no ads have ever run before. Launching a new campaign
is a fun time- you can get creative and try three completely
different ideas.
Track the three candidates, and eventually you will
have yourself a winner. Having a winner means you have
identified a control. This is the what you will be using
from now on, and this is the one that others will be
measured against!
Conclusion:
A control is the proven element of any test. Its' performance
is well documented, and it represents the standard by
which other candidates are measured. Just because you
don't have a control, doesn't mean you can't test. It
simply means that the best performer of your first test
will become the future control.
Lack of a control affects how you split the audience
between the test elements. Without a control, there
is nothing to lose, and evenly splitting traffic between
the candidates is the best way to go. Use an even 50%/50%
split for a two member test, or 33% each for a three
member split.
With a proven control, limiting the potential for losses
becomes a priority. In a two member test, sending 50%
of your potential profits to a poor performer is a real
risk. To combat this situation, plan on splitting your
audience such that they see the control the majority
of the time. Using a split that directs 80% or 90% of
the audience to the proven candidate reduces your risk
to current results.
It may take a little longer to get large enough numbers
to identify the better performer, but in the long run,
you will be better off.
Scott Miller
___________________________________________
Vertster.com | http://www.vertster.com/
Join
our Email newsletter
Return to Home
