Using Back-Door Adjustment Causal Analysis to Measure Pre-Post Effects

To reach the original version of this post check out the DoorDash Engineering blog here (Using Back-Door Adjustment Causal Analysis to Measure Pre-Post Effects).

When A/B testing is not recommended because of regulatory requirements or technical limitations to setting up a controlled experiment, we can still quickly implement a new feature and measure its effects in a data-driven way. In such cases, we use the back-door adjustment method, a type of causal inference to measure pre-post effects. This type of pre-post analysis is useful because it requires the same or less analytical effort to implement metrics tracking and make a data-driven decision as would be done in typical A/B testing. Because no test setup is required, this analysis can be used when we have to release new features quickly and as an alternative to slower testing methods. Here we explain how back-door adjustments enable non-biased pre-post analysis and how we set up these analyses at DoorDash.

Which features go live without experimentation

While data-driven experimentation ensures that the impact of new features are proven before they are presented to customers, we still want to be able to fast-track some features that address existing bugs or poor user experiences. For example, when our Global Search product team detected a critical bug in DoorDash’s mobile web platform and there was a correlated drop in key product metrics, the normal…