How Time of Day Fooled an Email Campaign

In this study, we have two variants of a subject line: Long and Short. We want to know which of the two lines generates the most clicks. Unfortunately, our account manager ended up sending more long messages out in the evening and more short ones in the morning. This time of day effect may have also impacted whether a recipient clicked on an email. This case study shows how we can use Daggy to account for the time the email was sent and properly select the best subject line of the two.

Data

We have the following data:

The Naive Approach: Testing Subject against Time of Day

We can proceed by ignoring the impact of the Time of Day variable in our model, just as if we had a random experiment. We can define this model in Daggy by drawing an arrow from subject line to clicked, as shown below:

We can then look at the impact of going from a Long to a Short subject line. The figure below shows the results of this scenario:

Surprisingly, it looks like Short subject lines were 26 percent less likely to be clicked on that Long subjects. On average, the click-rate for short subject lines was 11.6 percent and for long subjects it was 15.7 percent. Furthermore, we can see that zero is outside the 95 percent confidence interval, as both values are negative, meaning the results are statistically significant.

The naive decision would be to run with the longer subject line. But before doing that we should look at the impact from Time of Day.

Exploring Confounders

We can begin by looking at how much Time of Day impacted whether our analyst sent a Long or a Short subject. We can define this with the following model:

After training the above model we then set the target to Long and run a simulation to look at how Time of Day impacted whether a Long or Short subject line was sent. The results of the simulation are below.

From the results, we can see that Long subject lines were almost exclusively sent in the evening while Short lines were almost always sent in the morning. In other words, Time of Day is strongly associated with Subject Line.

For a variable to be confoundin, it is not only associated with the explanatory variable, but also the outcome. We can run a similar analysis to see if Time of Day is also associated with Clicked. Now we set Clicked as a target and train the following model:

After training, we then ran a simulation comparing Morning to Evening. The results are below.

Sure enough, Time of Day is associated with a recipient opening an email. This data shows that the probability of opening an email in the evening is more than 2 times likely as opening it in the morning. Put together, Time of Day confounds the relationship between the length of the subject line and the open rate.

Accounting for Confounders

With Daggy we can easily account for these confounding variables by drawing an arrow from the confounder to the explanatory variable and the target, as shown in the figure below:

We can now train this model and name it Time Confounder. In our scenario, we explore the impact of moving subject line from Long to Short while leaving Time of Day at "Select a Value", which is the same as leaving it at the mean. Of course, we could also hold it at Morning or Evening if we were curious about the impact under those cuts of our data.

The results of this simulation are in the figure below:

Notice that if we go from a Long subject line to a Short subject line the probability of a user clicking on the email increases by 41 percent. This is the opposite of the result accounting for the Time of Day confounder. Furthermore, as the 95 percent confidence interval does not overlap zero -- the lower and upper numbers are both positive -- the result is statistically significant. The clear recommendation here is to send the shorter email -- the opposite result of the recommendation before we accounted for the confounder.

Conclusion

Comparing the result that accounted for Time of Day with the naive result above highlights the importance of accounting for confounding variables. Without accounting for the confounder, the data would have drove us to incorrectly use the longer subject line. But when we accounted for the confounder, the data correctly suggested the opposite: use the shorter variant. Confounding variables often introduce severe bias to data analysis. Fortunately, provided you have data on these confounders, tools like Daggy make accounting for them simple.