Unpacking Compound Treatments in Email Audit Experiments

This research project will improve the design and analysis of audit experiments. Audit experiments are tools used to test for discriminatory behavior in situations where surveys and interviews can induce social desirability bias and are known to make strong causal claims. However, a common issue in audit experiments is the potential confounding of interventions with other attributes. For example, researchers often use names to signal race, but names may also signal other attributes, such as socio-economic status. Using other signals like income or occupation as additional interventions may help disentangle the effect of race from socio-economic status. This project will develop improved design and methodology for disentangling the effects of potential confounders from interventions of interest by incorporating such confounders as additional interventions. The methods to be developed will be applied to the study of the effects of race and related socio-economic factors on access to legal services. Understanding discrimination in the market for legal services is important because access to lawyers is a crucial first step for individuals to participate in the justice system. The project also will support training and engagement with audit studies through the involvement of graduate students. This research project will develop methods to address the complex problems in design and analysis that result from audit experiments having a factorial structure. These methods will include designing optimal factorial experiments that maximize information under resource constraints, development of appropriate statistical models for assessing causal effects of interventions, and data analysis procedures consistent with the design. The modern theory of causal inference and randomized experiments will serve as the foundation for the new methods. Analysis methods will be developed to assess causal effects of interventions on a selected group of subjects (finite population) as well as on a larger superpopulation from which such subjects can be assumed to be randomly selected. Methods for model-free (randomization-based) and model-based assessment of causal effects will be developed. The investigators also will explore methodology to extend design and analysis of audit experiments with binary or two-level interventions to the case of multilevel interventions. The new methods will be applied to an audit experiment to assess racial discrimination in response to requests for legal services, using names on emails to signal race. The results of this research will provide critical evidence for the study of racial discrimination in law and access to the justice system. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.