Abstract

Benchmark tests are widely employed in testing for racial discrimination by police. Neil and Winship (2019) correctly point out that the use of such tests is threatened by the phenomenon of Simpson’s paradox. Nevertheless, their analysis of the paradox is inadequate, in ways that point to a more general problem with how they relate statistical quantities to discrimination hypotheses. Simpson’s paradox reveals that the statistics employed in benchmark tests will not, in general, be invariant to updating on new information. I argue that as a result of this, benchmark statistics should not by themselves be taken to provide any evidence for or against discrimination, absent additional modeling assumptions. Although Neil and Winship highlight ways in which benchmark statistics appearing to provide evidence for discrimination no longer appear to do so given additional assumptions, they lack an account of which sets of assumptions would ensure invariance. Causal models provide such an account. This motivates the use of causal models when using statistical methods as evidence for discrimination.