A/B Tests

1. There are a few use cases for A/B tests on Helium.
   1. You can use A/B tests to test the hybrid monetization model (bidding plus waterfall) against pure waterfall monetization when you first start using Helium.
   2. Even after using Helium for a while, if you want to test a new mediation network or test some different settings for line items, you can always use A/B tests.
2. A/B tests are available at each Helium placement level. You need to make sure that the basic bidding and waterfall are already set up for those Helium placements.
3. For the initial use case of testing hybrid vs waterfall, here is the best practice for setting up your first A/B test on Helium
   1. Under the Helium placement, you have finished setting up all bidding networks (including Chartboost, Meta Audience Network, AdColony, Tapjoy). You also have finished setting up the waterfall (including ad groups and line items).
   2. Enable A/B test in Advanced Settings section of the placement.
   3. Helium dashboard will automatically create the A/B Test Settings section.
   4. Control Group (A) is the existing bidding and waterfall setup for the Helium placement.
   5. Test Group (B) is a copy of the existing setup. You should deactivate all the bidding networks and add the additional line items for the bidding networks (including Meta Audience Network, AdColony, Tapjoy) in the ad groups.
4. Helium automatically allocates 50% of ad requests to Control Group (A) and 50% of ad requests to Test Group (B).
   1. Each user is randomly assigned to one of the groups and is sticky to one group, which ensures the efficiency of A/B tests
5. You can check A/B tests performance data in Analytics > A/B tests.
6. If you find one of the groups is having better performance after reviewing A/B test analytics, you can disable the A/B test by disabling the worse performance group in the Helium placement.
   1. All ad requests will automatically start going to the remaining active group once one group is being disabled.
   2. You can not disable both groups. At least one group has to be active.
7. If you want to re-activate the A/B test, you can directly edit any bidding and waterfall settings in the disabled group first and then turn on the “Active” toggle.