Researching racial equity: Evaluating “Ban the Box” policies

In J-PAL North America’s researching racial equity blog series, we discuss how research plays a critical role in identifying structural inequities in systems and policies that disproportionately affect communities of color. In part one, we interview Dr. Agan to discuss the evaluation and explore the role of randomized evaluations in advancing racial equity.

Can you tell us a bit about the “Ban the Box” study and the goals of this research?

Millions of people across the United States acquire criminal records each year, and these records can be a barrier to employment. Because the entire criminal legal system is rife with racial inequities, members of communities of color—and Black men in particular—are overrepresented in all steps of the criminal-legal process from stops, to arrests, to charges, to convictions, and sentencing. 

In an effort to increase opportunities for people with records, jurisdictions across the country have implemented “Ban the Box” (BTB), a set of policies restricting employers from asking about applicants’ criminal histories on job applications. These policies are often presented as a means of reducing unemployment among Black men. In our BTB study, we wanted to understand how employers reacted to job applicants from different demographic groups once the criminal record question was removed from the application. Would they start to rely on other observable characteristics as proxies for criminal justice contact? In particular, would they use perceived race to stereotype applicants? 

To answer these questions, we sent out a total of 15,000 fictitious job applications both before and after BTB went into effect in New Jersey and New York City in 2015. On each application, we randomized 1) the perceived race of the applicant by using names associated with certain races, 2) whether the applicant had a criminal history, 3) whether the applicant had a GED or a high school diploma, and 4) whether the applicant had a one-year employment gap or not. Before BTB, there was little racial disparity in callback rates among applicants with similar records, though employers were 63 percent more likely to call an applicant with no record than one with a record. After BTB, however, when the employers could no longer ascertain criminal history from the job application, employers were 43 percent more likely to call back an applicant perceived to be white than one perceived to be Black. Employers were clearly stereotyping Black applicants as more likely to have a criminal record, harming Black applicants who did not have a criminal record. Interestingly, they did not seem to react differentially to applicants with one-year employment gaps or GEDs, even though those characteristics are also correlated with criminal records.

What steps did your team take to disseminate findings after results were finalized? Did the study result in any policy change that you know of?

We shared early versions of our findings with President Obama's Council of Economic Advisors when they approached us about potentially implementing a Federal BTB policy. We also spoke with several media outlets, including NPR and US News, to advertise the findings. In addition, we participated in an online policy debate on BTB hosted by the Urban Institute that featured several academics and policymakers.

I am not aware of any jurisdiction that repealed BTB laws due to this research. But, I do hope it has given policymakers pause as they look for policies that can help improve opportunities for individuals with records so that we do so in a way that does not inadvertently harm Black job seekers who do not have criminal records.

From your perspective, how did your BTB work address racial inequities?

BTB was meant to increase interviews and opportunities for people with records. We could have simply randomized criminal record status without disaggregating by race, and likely would have found, as we did, that BTB "worked" in reducing the impact of having a criminal record on employment opportunities. But given what we knew about the racial statistical discrimination and stereotyping that happens in employment and other domains, we decided to explore this aspect in our research as well, which uncovered a harm that we believe was important to document.

BTB policies appear to have exacerbated racial inequities in employer callback rates because employers held the stereotypical belief that a young, Black, male applicant was more likely to have a criminal record than a white applicant. The research implies that simply omitting information will not eliminate the negative labor impacts of criminal legal contact and may harm Black applicants without criminal histories.

What role can randomized evaluations play in promoting racial equity?

Randomized evaluations are really good at pinpointing the causal impact of a (manipulatable) characteristic, policy, or treatment on measurable outcomes. In certain instances, one can even try to pinpoint the direct impacts of race by manipulating perceived race as we did or as in a previous study by J-PAL affiliates Marianne Bertrand and Sendhil Mullainathan. By demonstrating that racism is, at least in part, driving disparate outcomes in certain fields (e.g., hiring), audit studies and other randomized evaluations have the potential to effect change by informing policy. 

However, randomized evaluations that can manipulate perceived race are usually estimating the impacts of race while holding other variables constant. Systemic racial inequality means that there are many differences between Black applicants and white applicants besides their (perceived) race, some of which reflect the direct and indirect impacts of racial discrimination in other domains or at previous time periods. These complexities are harder to measure and address in a randomized evaluation. Integrating the results of randomized evaluations with other methods, both empirical and qualitative, as well as bringing in the voices of directly impacted community members and scholars, will likely give us the strongest path forward for racial equity in policy and outcomes.

The researching racial equity blog series features the contributions of researchers and partners in examining and addressing racial inequities and offers resources and tools for further learning. Part two features work quantifying housing discrimination. Part three gives an overview of stratification economics in the context of evaluations. Part four discusses how to center lived experiences throughout the research process and in impact evaluations. Part five shares guidance for incorporating inclusive and asset-based framing throughout the research cycle. Part six examines sources of bias in administrative data bias. Lastly, in part seven, Damon Jones and J-PAL staff share progress on researching racial equity and future areas of work.