Start here: Statistics for A/B testing — The Product Management Coalition. This article was first published internally at Freeletics. We sure take our decisions serious and don’t favour “hope-for-the-best” approaches. Also, we’re hiring Product Managers and Product Designers! What is an A/B test? An A/B test consists of taking two comparable groups of users and exposing them to two different versions (the control and a variation) of a software experience. By doing that and measuring how each of the groups interacts with the software we hope to infer which of the two versions best serves its purpose. Some relevant statistics concepts Overall Evaluation Criterion (OEC) Also called Primary Goal (in Optimizely and other tools) or Dependent variable, in statistics terminology.
Null hypothesis (Ho) The hypothesis that the OECs for the variants are not different in fact, and that any observed differences during the experiment are due to random fluctuations. (Most of the statistics used in A/B testing come from the field of Statistical hypothesis testing.) P-value Also: Power. Most of your AB-tests will fail. Jitbit Blog - please Subscribe if you like this post Jun 19 2013 A/B testing is overhyped. It is everywhere these days. While A/B testing is a very powerful conversion optimization instrument, it requires lots of hard work but often it is just useless. Going from absolute total crap to something decent increases your conversions.
No company have ever increased the revenue by changing an insignificant detail like a button color. Most of the tests produce no results A successful split test is a rare thing. From our experience, only one out of ten A/B tests produce results. Here is an example of a test we ran not so long ago. As you can see, the test had 85k participants and both alternatives have the exact same number of conversions. Another example: we sorted the plans in our pricing tables to go from the most expensive to the cheapest and vice versa, and tracked the most expensive plan orders.
Again 30k participants and zero difference. to this: Do you think our conversions have changed? 20 lines of code that will beat A/B testing every time. Zwibbler.com is a drop-in solution that lets users draw on your web site. A/B testing is used far too often, for something that performs so badly. It is defective by design: Segment users into two groups. Show the A group the old, tried and true stuff. Show the B group the new whiz-bang design with the bigger buttons and slightly different copy.
After a while, take a look at the stats and figure out which group presses the button more often. Sounds good, right? The problem is staring you in the face. In recent years, hundreds of the brightest minds of modern civilization have been hard at work not curing cancer. With a simple 20-line change to how A/B testing works, that you can implement today, you can always do better than A/B testing -- sometimes, two or three times better. It can reasonably handle more than two options at once..
The Multi-armed bandit problem The multi-armed bandit problem takes its terminology from a casino. Def choose(): if math.random() < 0.1: # exploration! A/B test case study: how two magical words increased conversion. Posted in A/B Split Testing on May 26th, 2010 When we saw the results Soocial had got from their latest A/B test, we were astonished! They added just two words next to the Sign up button and the conversions shot up by 28%. If we say the phrase was one of these: “Sign up for Free”, “It’s Free” or “Free Signup”, can you guess which one did the trick?
That is precisely the beauty of A/B testing, you can never guess what works – they only way out is to actually test it. This case study has direct inputs from Soocial CEO, Stefan Fountain. Background Soocial is an online address book that helps you keep your phone, computer and online services contacts sane. What was tested Their homepage features a large Sign up button. Results As you can read above, the changes on the page were extremely minor and, on the surface of it, look quite trivial.
Control: 14.5% conversion rate Variation: 18.6% conversion rate Why “It’s Free” Worked How valuable was Visual Website Optimizer for the A/B Test? Invaluable. A Beginner's Guide to A/B Testing: Better Pay-Per-Click Ads. Pay-per-click advertising is a key component of many online marketing campaigns. It can also be one of the most expensive ongoing costs in a campaign. Therefore, it’s key that you test your ads regularly, to make sure you aren’t letting conversions slip through the cracks.
To an extent, PPC testing is simpler than many other kinds of A/B tests, partly because there are fewer things to test. But that doesn’t mean any less care and planning should go into preparing and executing these tests. This is the fourth installment in our A Beginner’s Guide to A/B Testing series. Be sure to check out our previous posts: A Beginner’s Guide To A/B Testing: An Introduction, A Beginner’s Guide To A/B Testing: Exceptional Web Copy and A Beginner’s Guide to A/B Testing: Successful Email Campaigns, and stay tuned for our final installment on testing SEO landing pages.
Deciding What to Test Pay-per-click ad testing is a bit more streamlined than the other topics we’ve covered in this series. Best Practices. A Beginner’s Guide To A/B Testing: Exceptional Web Copy. Optimizing the copy on your website is at least as important as optimizing the design, especially if the primary goal of that site is to convert visitors. A pretty design can only get you so far. If you really want to gain new customers, you need to optimize the text on your site to instill trust in visitors and make them want to purchase from you.
We often spend hours or days reading about the best techniques to use to sell products. But that doesn’t give us a complete picture, and what works great for one company or one product might not work at all for another. A/B testing is the simplest type of testing you can do to figure out which variations of copy, headline, and other factors are most effective in direct relation to your site and your offerings. This is part two of a series of posts on A/B testing, which will be published in the coming weeks. Things to Test There are a ton of things you can test on your site to see what’s most effective. Tools for A/B Testing. A Beginner's Guide To AB Testing: An Introduction. A/B testing is a fantastic method for figuring out the best online promotional and marketing strategies for your business. It can be used to test everything from website copy to sales emails to search ads.
And the advantages A/B testing provide are enough to offset the additional time it takes. Well-planned A/B testing can make a huge difference in the effectiveness of your marketing efforts. Narrowing down the most effective elements of a promotion, and then combining them, can obviously make your marketing efforts much more profitable and successful. Here is the first in a series of posts covering A/B testing that we’ll be publishing in the coming weeks. How Do You Plan an A/B Test? The first thing to do when planning an A/B test is to figure out what you want to test. With off-site tests, you’re probably testing either an ad, or a sales email. Once you know what you’ll test, make a list of all the variables you’ll test. A/B Testing Can Improve Your Bottom Line Things to Test.