For the past several months I have been working on a new series of digital marketing products, largely focused on gaps I have encountered in the marketplace over the years. The first of these were released a couple months ago now, called SplitCheck.
SplitCheck was initially designed to be an A/B Testing solution for organizations who felt the loss of a simple, straight-forward platform when Google’s Optimize existed the market a couple of years back, and the interest and use in the tools has grown steadily over the past few months.
In speaking to users in various forums after the initial launch, however, I have realized that the standard approach to A/B testing has a weakness nobody talks about: you need volume to test. Lots of it.
Almost every major A/B testing platform, as well as the tools built into most marketing stacks, use the two-proportion Z-test to determine whether a variant is winning against other contenders. This is frequentist statistics, where you set a significance threshold, collect data until you cross the finish line, and get a (usually binary) verdict. Significant or not significant. Winner or no winner.
There is a reason this method so popular in the digital marketing world. It is relatively straight forward to implement and define. And it works. Under the right conditions.
The think is that the conditions for the traditional approach generally require at least several hundred conversions per variant, stable traffic within a relatively short time period, and the patience to wait for the data to accumulate. For a large e-commerce platform or a SaaS company with very high channel traffic, this is a very reasonable and effective.
For years, however, I have been working with small to medium-sized organizations running campaigns to a landing page or a conversion action with much smaller volumes, it means waiting months for a result (by which time the competitive landscape has often shifted, and the test has become irrelevant), or simply breaking the statistics model by prematurely choosing a winner mid-test.
The issue is what “not significant” actually means. In practice, most SMB marketers interpret it as “no difference.” That is not what it means. It means “we do not have enough data to be confident either way.” Those are very different claims, and confusing them leads to bad decision making; abandoning tests that are showing real signal, or running variants indefinitely on the grounds that nothing is proven.
I have been thinking about this problem for a while, and this week I replaced SplitCheck’s traditional statistical engine with a Bayesian alternative. Well, actually I added Bayesian and then put it up front. The traditional stats are still there as well.
The change is conceptually simple. Instead of asking “is this result statistically significant?”, the Bayesian engine asks “given the data we have, what is the probability that Variant B outperforms Variant A?” The answer is a number between 0 and 1, updated continuously as visitors arrive.
The underlying model is Beta-Binomial: each variant’s true conversion rate is modelled as a Beta distribution, updated as a conjugate posterior from the Binomial likelihood of observed conversions. P(B > A) is estimated via Monte Carlo simulation: 10,000 draws from each posterior, counting the proportion of draws where B exceeds A. At 10,000 samples the Monte Carlo standard error is bounded at 0.5%, which is more than adequate for the precision required.
The practical effect is significant. A test where Variant B has converted 9 out of 50 visitors and Variant A has converted 3 out of 50 will return a Bayesian probability of around 96% that B is better. A frequentist Z-test at the same sample size will say “insufficient data.” One of these outputs is useful to small scale tests. The other isn’t.
We also report a 90% credible interval on the expected lift, the Bayesian equivalent of a confidence interval, but with the interpretation practitioners have always (incorrectly) applied to confidence intervals. A 90% credible interval of +4% to +21% means there is a 90% posterior probability that the true lift falls in that range. That is the statement people think a confidence interval makes, but it isn’t.
As I said above, the more traditional frequentist engine still runs in parallel on SplitCheck, and for larger organizations this could very well continue to be their deciding metric. It would probably be mine if I had the traffic patterns to fill it. Now both results are stored on every test, and the traditional significance output is available in a collapsed section of the dashboard for those who want it. This parallel approach serves a dual purpose: it lets customers cross-reference results, and it generates the validation dataset I need for the research work I am developing on this methodology.
That research is the longer-term motivation for me. The experimentation literature has focused almost exclusively on enterprise-scale contexts. There is real work – both practical and academic – to be done on valid causal inference for conversion experimentation at SMB traffic volumes, on what methods are calibrated, what decision thresholds are appropriate, and whether empirically derived priors from aggregate platform data can accelerate time-to-decision.
The new results panel is live at splitcheck.io. The methodology page at splitcheck.io/methodology has the full technical explanation for anyone who wants to go deeper.