The ABCs of A/B Testing Mobile Game Ads

A truly great mobile game is one that is never finished. Rather, it is constantly being refined by its developers who, like the players themselves, never tire of trying to ascend to a?new level.Perfecting your game through change is only possible through experimentation. After-all, it is impossible to discern what works without first discovering what does not work. In the world of mobile gaming, this means means undergoing rigorous A/B testing.There is a ton of talk about A/B testing, but how does one actually A/B test effectively? Here are the ABCs of the simple yet profoundly important process.A: Identify what you?re trying to learnA/B testing first requires you undergo an honest assessment of what needs to be improved in your game. That pain point will inform your entire A/B testing process.Perhaps your players are abandoning the game when you serve them a 30-second video ad between levels, and you?re wondering if they?ll stick around for a 15 second spot. Maybe your iOS players make more in-app purchases than your Android players, and you want to test ways to better monetize your Android players?or you want to implement genre targeting to boost ad conversion. If your retention rate is low, you might be curious whether sending more push notifications will keep players coming back.Remember, there doesn?t necessarily have to be something ?wrong? with your game in order for you to dabble with A/B testing.You could discover an improvement that you didn?t know was possible.B: Form a hypothesisOnce you settle on what you?d like to test, produce a hypothesis. What?s paramount is to make this hypothesis as specific and quantifiable as possible.For example, "We believe that by decreasing the number of coins needed to move to the next level will adjust the game?s time loop and day one retention will increase by 25 percent" is a far better hypothesis than "shortening levels will increase player engagement.?Don?t be afraid to be outlandish, either; hypotheses were made to be tested, not proven right.C: Segment your groups accordinglyHow you segment your audience for A/B testing will derive specifically from what you?re trying to test. Say you hypothesize that changing the color of your in-app purchase buttons to red will increase the number of people who purchase in-game power-ups by 50 percent. Use random selection to parse your players into two groups, and expose only one group to the change.If you?re testing how a new version of the game will perform on Android vs. iOS, then just roll out the same version to both platforms, and voila, your test is self-segmented.Remember, though, to create a control group.For those who slept through sixth grade science class ? a control group is integral to A/B testing. The control group, or the ?A? group, should not be exposed to whatever change you?re testing for. The control group is the benchmark, the segment you will be testing your results against.Going back to the button color example: Say you?re considering changing the color of your blue buttons, and you want to test whether green, yellow or red will elicit an improvement in performance. That?s fine so long as you segment the audience into four groups, with one being served the blue buttons.D: Continue testing until you have enough dataIt?s tempting to have a knee-jerk reaction to any initial differences you notice among the groups, but in the name of good science, it?s important to let the test play out until there?s a significant amount of data. The larger the data pool, the more trustworthy the results.If you're unsure of how many data points you'll need in order to have a scientific test, there are plenty of free sample size calculators online. The more accurate you want to be, the larger your sample size needs to be.E: Implement your change, and then do it all overCongratulations, you improved your game! Now it?s time to do it all over again.As Winston Churchill said and Frank Underwood subsequently quoted, "To change is to improve. To perfect is to change often.?Stay tuned for some deeper dives on A/B testing your audience.