Simple A/B Test

This post shows how we can quickly analyze an A/B test to get an estimated treatment effect as well as understand the uncertainty around our estimates using Daggy.

Data

In the experiment that follows we have a variable grp whose values are either treatment or control. The outcome variable is converted, which is yes in the case the user bought an item and no otherwise. We are interested in testing whether the treatment group is statistically different from the control group.

Analysis in Daggy

We can model the relationship between grp and converted by setting the target to converted and then drawing an arrow between the variables, as in the graph below:

We then name this model grp-converted and train it. Since we care about the impact of the treatment relative to control on the likelihood of converted being yes, we set up the scenario as follows:

We can run this and look at the results.

We see a large difference between the estimated impact of the treatment and the control group. Looking at the "Mean Probability (New)" the treatment group has a 60.5 percent chance of converting while the "Mean Probability (Previous)" cell shows the control group only had a 40.4 percent chance of converting. The difference between the average of these two groups is 20.1 percentage points, as shown in the ATE cell. The 95 percent confidence interval for the ATE is between 16.2 and 24.3 percentage points. Because the 95 percent confidence interval does not cross zero, this difference is statistically significant at the 5 percent threshold.

Daggy automatically plots the estimated conversion rates for each group, as shown in the graph below:

As the above makes clear, we see substantial differences in the conversion rates between the treatment and control group on the likelihood of converting.

Conclusion

With a few clicks we analyzed an A/B test in Daggy. We could quickly see that the difference between the two groups was substantial and statistically significant.