The State and Local Learning Agenda: A Blueprint for Future Evaluations

For those whose income would otherwise fall below a threshold needed for survival in the US economy, social protection programs provide critical support as a precondition for upward mobility, allowing individuals to maintain their well-being and to develop the skills and knowledge needed to find gainful employment. 

What role do evaluations play? Improving Take-Up of Tax Benefits in the United States: The results of this evaluation suggest that most individuals fail to take up social benefits because of complicated forms, confusion about program rules, or uncertainty regarding eligibility. Take-up rates could be increased by simplifying forms and clarifying eligibility criteria. 

How can we expand the evidence base? Individuals in different jurisdictions may face varied barriers to take-up of social protection programs. Research can help further determine the effectiveness of providing different information on social protection programs, an individual’s eligibility, and the application and/or recertification process through in-person assistance, text messages, postcards, and/or letters.

Many anti-poverty programs seek to ensure that families have the resources they need to achieve economic self-sufficiency. Barriers to self-sufficiency range from limited employment opportunities to informational barriers. Multiple jurisdictions have highlighted the priority of moving people to higher-paying jobs to become economically self-sufficient.

What role do evaluations play? Intensive Case Management to Overcome Barriers to Self-Sufficiency in the United States: There exist many programs seeking to alleviate discrete aspects of poverty but they often provide short-term relief. Devoting resources to comprehensive services, such as those provided by case management, could remove barriers to long-term self-sufficiency. This ongoing evaluation seeks to understand whether intensive case management can lead to long-term self-sufficiency by looking at a program in Rochester, NY that assigns low-income individuals a navigator/ mentor who helps an individual access a wide range of government services.

How can we expand the evidence base? The costs of engaging in a job search program can vary from economic costs, such as transportation costs, to time costs and psychological costs. Jurisdictions are interested in evaluating programs like distance work services appointments, simplifying the processes and requirements to apply to work services, and providing value affirmation exercises as well as personalized, targeted, and motivational support.

Federal relief packages have recently allocated billions of dollars to state and local jurisdictions, but the strain this has put on existing social programs highlights the need for infrastructure improvements to better target and deliver aid. It is also important to make sure programs are reaching the individuals that they are intended to reach.         
                
What role do evaluations play? Understanding the Effects of Text Reminders on Reducing SNAP Churn: This study assesses the impact of a Massachusetts texting reminder program in reducing the number of individuals who fail to renew for SNAP, leading them to reapply at a later point, which is known as “churn.” SNAP churn is costly to applicants and may leave them without protection before they reapply.    

How can we expand the evidence base? Testing unconditional cash transfers and universal basic income programs is a key evaluation opportunity in promoting economic security. Unconditional cash transfers are programs for a specific group of people that provide money to individuals without requiring them to meet specific conditions (i.e., participating in a specific activity). Universal basic income is a regular, guaranteed income without restrictions or requirements.

Access to employment can result in positive outcomes beyond immediate income; for instance, prior research indicates that job programs for low-income individuals may result in positive physical and mental health outcomes. Improving the quality of jobs is also a priority, as many jobs available to those who need them do not pay living wages, offer sufficient benefits or protections, or provide a path for workers to progress within their industry.
                                        
What role do evaluations play? The Effect of Tailored Information on Low-Income Workers’ Transitions, Employment, and Earnings: Researchers leading this ongoing study look at why low-income workers often do not move to higher-paying jobs, focusing on the explanation that workers might have inaccurate or uncertain perceptions about job opportunities. They test whether providing tailored information on occupations with relatively high wages to workers in Southwest Michigan can improve employment outcomes. 

How can we expand the evidence base? Pre-and post-employment services include subsidized employment, apprenticeships, volunteer work, job coaching, job search services, educational programs, and other training services. For example, a randomized evaluation could examine the effectiveness of a vocational program in helping healthcare workers acquire higher certification for promotion and career advancement.

Technological progress has had highly uneven effects on labor markets and earnings opportunities for workers. Those with at least some college education have benefited from these shifts, while those with less education have been increasingly pushed into low-wage work. The Covid-19 pandemic only exacerbated this disparity as low-wage workers experienced significant job losses. Understanding what can be achieved through training to equip workers with new skills is critical in increasing long-term employment and earnings prospects. 

What role do evaluations play? A Randomized Evaluation of the Pursuit Fellowship: Given the demand for greater evidence about when and for whom sectoral training programs are appropriate and effective, this ongoing study evaluates a tech sector employment program’s impact on earnings and other labor market outcomes among low-wage workers. 

How can we expand the evidence base? Looking at specific sectoral approaches to job training is critical in an economy that continuously relies on specialized jobs. For example, a randomized evaluation could examine the effectiveness of a healthcare apprenticeship program in equipping youth for high-skill jobs through a blend of classroom learning and on-the-job training. Take a look at Preparing for the Work of the Future for more research needs connected to re-skilling workers. 

Job search barriers can severely inhibit employees seeking work. Several commonly cited search barriers include a lack of knowledge around where and how to search for jobs efficiently, the effort required in the job search process, and potential skills mismatches between what job seekers can offer and what employers want.
                                        
What role do evaluations play? J-PAL North America reviewed the results of 19 randomized evaluations of interventions aimed at reducing job search barriers around the world. It shows that job search assistance helped job seekers look for jobs in better places and increased their search efforts. 

How can we expand the evidence base? The costs of engaging in a job search program can vary from economic costs, such as transportation costs, to time costs and psychological costs. Jurisdictions are interested in evaluating programs like distance work services appointments, simplifying the processes and requirements to apply to work services, and providing value affirmation exercises as well as personalized, targeted, and motivational support.

Past research has identified clear links between schooling and long-term life outcomes such as poverty, health, and risk of involvement with the criminal justice system, all critical components of upward mobility.^{2 3 4} Lack of access to quality education can exacerbate existing inequalities and hinder students’ opportunities for economic success. 

What role do evaluations play? Closing the Word Gap with Big Word Club: Evaluating the Impact of a Tech-Based Early Childhood Vocabulary Program: This study looks at how digital vocabulary learning programs can increase vocabulary acquisition in early childhood, an area of learning critical to reading comprehension and language/knowledge development. Researchers found that a program in the Southwest effectively increased students’ vocabulary knowledge and retention. Given its low cost and ease of use, programs like this one may be a cost-effective strategy for improving the vocabulary of young children.

How can we expand the evidence base? To address under-preparedness for college, jurisdictions focus on providing educational services to high schoolers. For states with oversubscribed programs, a lottery can randomly select from an eligible applicant pool and jurisdictions can compare educational and employment outcomes of those who participated in college readiness programs and those who did not.

Researchers predict that school closures due to Covid-19 will lead to learning loss with disparate impacts for different socioeconomic groups. Furthermore, many students are losing access to their virtual education as a result of digital inequality. These shortfalls in education accessibility may result in adverse effects on education outcomes and mobility from poverty. 

What role do evaluations play? Saga Education Tutoring: Two randomized evaluations found that students who received individualized math tutoring through Saga scored higher on exams, earned better grades, and were more likely to pass high school classes. 

How can we expand the evidence base? Researchers can explore the effectiveness of virtual tutoring programs to address potential academic backslide from the Covid-19 pandemic. An evaluation could measure the impact of participating in one-on-one tutoring, group tutoring, or a mix of the two on educational outcomes such as reading and math test scores.

Living with an opioid addiction limits an individual’s power, autonomy, and economic success.⁵ Due to the increased risk of death and the decreased standard of living, relief from and management of opioid addiction is a necessary precondition for upward mobility. 

What role do evaluations play? Minnesota Board of Pharmacy: This project looks to increase the use of Minnesota’s prescription drug monitoring program (PMP) and to measure the impact of that increased use on prescribers’ rate of controlled substance prescriptions and other tertiary outcomes. 

How can we expand the evidence base? Addiction treatment and prevention interventions can be evaluated to determine which programs are most effective in which contexts. There is a range of both programs, such as hospital-care or community-based buprenorphine treatment, and coping strategies aimed at youth to encourage them not to use drugs.

Mental health affects a number of determinants that directly relate to upward mobility: housing security, employment, education, involvement with the criminal justice system, physical health, and a sense of value and belonging in a community.⁶ Accessing culturally relevant services and navigating complex medical systems represent especially unique challenges in accessing care. 

What role do evaluations play? Online Delivery of Mental Health Care: ​​Low-income Americans have a higher risk of suffering from common mental illnesses, and lower rates of receiving treatment. This ongoing study looks to understand whether online and app-based psychotherapy tools can provide a cost-effective way to deliver mental health care to low-income Americans.

How can we expand the evidence base? Researchers can look at improving access to behavioral health treatment in traditional medical settings by embedding behavioral health support into various primary and specialty care settings.

Early childhood health and development can determine the course of one’s life. For this reason, disruptions to healthy and safe development can have profound negative consequences on mobility from poverty and other life outcomes. 

What role do evaluations play? The Impact of a Nurse Home Visiting Program on Maternal and Child Health Outcomes in the United States: Young mothers living in poverty are at greater risk for poor birth outcomes. Growing up in poverty can be harmful to a child’s cognitive development, health, school performance, and social and emotional well-being. This ongoing evaluation looks at the effects that nurse home-visiting programs can have on both the health and development of mothers and children. 

How can we expand the evidence base? Interventions providing child health-related nudging or incentives to young parents can be evaluated to determine what is most effective in which context. For example, researchers and state partners can look at text message programs that aim to encourage the adoption of healthy practices among parents and children. Likewise, programs involving financial incentives for low-income mothers, like one that aims to promote breastfeeding, can benefit from evaluation.

The expansion of telemedicine can mitigate the lack of access to healthcare often felt by individuals experiencing poverty. It removes geographic distance as a barrier, increasing accessibility especially for households in rural communities. Understanding how to best provide these services and overcome the lack of digital access can have lasting impacts. 

What role do evaluations play? Reducing Nurse Burnout through Online Social Support: There is promising evidence that online platforms can effectively be used to reduce worker burnout. Given the limited evidence on the most cost-effective method to deliver such necessary care to health care workers specifically, this ongoing evaluation tests the impact of a low-cost online program of communications on social belonging and support among nurses and, in turn, on burnout, retention, and patient outcomes.

How can we expand the evidence base? To maintain social distancing during Covid-19, some jurisdictions have loosened restrictions on telemedicine provision.⁷ An intervention can examine the varying availability of telemedicine on health care quality as well as a comparison between telemedicine and in-person health care.

Due to the disparate impact poor health can have on individuals experiencing poverty, increasing trust and take-up of preventive health products in low-income communities is critical in increasing one’s mobility from poverty. Researchers and jurisdictions are also concerned with low levels of trust and take-up of preventive health products such as vaccines in low-income neighborhoods.^{8 9} This is an especially stark reality when considering Covid-19 vaccine distribution, a case in which inadequate take-up reduces the vaccine’s overall effectiveness.

What role do evaluations play? Understanding Covid-19 Testing among Underserved Communities in Minnesota: The Minnesota Department of Public Health, the University of Minnesota, and the Mayo Clinic partnered with J-PAL researchers to study barriers to take-up of Covid-19 testing among communities of color and low-income communities. 

How can we expand the evidence base? Barriers to take-up can differ across communities. Researchers can determine whether financial incentives or nudging programs are more effective in a certain jurisdiction in increasing the take-up of health programs and vaccines and the adoption of healthy practices with randomized evaluations.

Living unhoused can have severe consequences on education, child development, health, employment, and involvement with the criminal justice system. Ensuring safe and affordable housing is an important component of upward mobility.

What role do evaluations play? Researchers are examining the impact of providing rapid rehousing to single adults experiencing homelessness in Santa Clara County on housing stability and health outcomes. 

How can we expand the evidence base? Many homelessness programs involve multiple components, and a tenant’s success in maintaining housing may depend on the type of support received during the housing search process. A randomized evaluation could measure the effectiveness of each service in isolation versus bundled.

Homes are high-value assets, and for many people are the largest contributor to their net worth. Homeowners can build equity and use this equity to better survive economic shocks. 

How can we expand the evidence base? Low- and moderate-income loan programs for down payment assistance can encourage homeownership and asset building. A randomized evaluation can provide varying payment assistance amounts and evaluate the difference in homeownership and other economic outcomes.

Neighborhoods matter for intergenerational mobility. These effects of place are also a question of choice and autonomy; efforts to promote mobility should account for the impact of place and the legacy of residential segregation.

What role do evaluations play? Creating Moves to Opportunity is an ongoing collaboration between J-PAL-affiliated researchers and public housing authorities to introduce and evaluate interventions to “create moves to opportunity” for low-income families in Seattle and King County.

How can we expand the evidence base? One barrier to families moving to areas of opportunity is fewer listings available to housing voucher recipients. A randomized evaluation can investigate the impact of outreach campaigns to landlords on their participation in the housing choice voucher program.

Strategies to mitigate pollution can reduce the health burdens on low-income populations, who are disproportionately impacted by environmental health hazards related to the location and quality of housing. 

What role do evaluations play? Heavy-duty trucks are major sources of particulate matter (PM) pollution, which threatens human health and disproportionately impacts low-income populations. In California, researchers are evaluating the impact of new remote sensing technology on regulatory compliance, emissions, emissions leakage, and enforcement costs.

How can we expand the evidence base? Programs focusing on healthy home environments provide consultations, inspections, testing, outreach, and training to protect residents from hazards in the home that can affect health. A randomized evaluation can study the impact of providing home hazard training on child health outcomes.

The effects of climate change disproportionately impact low-income communities along dimensions like health, household finances, and labor markets.

What role do evaluations play? In Michigan, researchers randomly selected a group of eligible low-income households to receive encouragement and assistance to apply for a fully subsidized residential energy efficiency program. Those who enrolled experienced energy savings equal to about half of the upfront cost of the efficiency improvements they received. 

How can we expand the evidence base? Residential energy efficiency programs are key components of many energy policies and efforts to reduce greenhouse gas emissions. A randomized evaluation can examine the impact of direct incentives on carbon emissions, such as subsidies for home energy efficiency that pay per unit of emissions avoided.

Despite the implementation of equal employment policies, there are racial and gender disparities in labor market outcomes. Equal opportunity to employment is an important factor to achieve economic security. 

What role do evaluations play? In Chicago and Boston, researchers found that resumes with white-sounding names received 50 percent more callbacks than identical resumes with Black-sounding names, indicating that considerable racial discrimination exists in the American labor market.

How can we expand the evidence base? Many jurisdictions have anti-discrimination laws that are enforced to varying degrees, with local businesses potentially unaware of all requirements. A randomized evaluation can evaluate the impact of reminding workplaces that may not comply with local anti-discrimination laws on discrimination in the local job market.

Racial disparities in the criminal justice system, education, health, and labor contribute to long-term social and economic inequities. Diversity, equity, and inclusion (DEI) training programs are one strategy governments are using to decrease bias. 

What role do evaluations play? An ongoing project throughout the United States seeks to test the effectiveness of an intervention designed to reduce racial disparities in bail decisions, by providing information and feedback to judges. 

How can we expand the evidence base? To promote DEI, many workplaces ask their employees to undergo training programs or workshops. A randomized evaluation can evaluate the impact of varied trainings on reported instances of workplace discrimination and measures of social cohesion and productivity.
 

Discrimination and discriminatory policy against Black communities resulted in large wealth gaps between Black and white individuals and families. Targeted wealth generation and asset building is a critical component of mobility from poverty for Black communities.

How can we expand the evidence base? Bonds or savings programs targeted to children in historically underinvested communities can help build wealth and provide savings for opportunities like college. 

Violent crimes have negative repercussions for the victim, the broader community, and the perpetrators themselves. Exposure to violence can put children at greater risk for mental, academic, and social challenges, while also increasing the likelihood that they will commit violence in the future.                         

What role do evaluations play? Summer youth employment programs are an increasingly popular strategy for reducing violence among low-income youth and young adults. Participating in Chicago’s One Summer Chicago Plus (OSC+) nearly halved the number of violent crime arrests among program participants. 

How can we expand the evidence base? Community-based policing involves high-level community involvement in police activities and multiple touchpoints between police officers and community residents, such as foot patrols, neighborhood outreach programs, and police monitoring. A randomized evaluation can examine the impact on crime and violence by staggering the roll-out of a community-based policing program.

Incarceration not only represents a loss of income while incarcerated; a formerly incarcerated person also has limited upward mobility upon their release, affecting their family and community. Involvement in the criminal justice system can also place an economic burden on those who are never convicted of a crime due to fines, fees, and the money bail system. 

What role do evaluations play? Researchers partnered with policymakers in New York City to evaluate the impacts of a redesigned court summons form and text message reminders on failures to appear (FTA) in court, which can result in additional jail time. Text messages reminders were able to reduce FTA by up to 26 percent, translating to 3,700 fewer arrest warrants per year. 

How can we expand the evidence base? The exposure an individual gets to the jail and prison system can greatly vary from case to case. For instance, a police officer can decide to issue a citation or place an individual under arrest at first encounter, or a judge can decide whether to deny bail as an individual awaits trial. A randomized evaluation can examine the employment and case outcomes for individuals who receive assistance from a limited cash bail fund.

People experiencing poverty are disproportionately represented in the criminal justice system, and more than half of individuals released from the US prison system are rearrested within their first year. People re-entering civil society may face systemic legal and societal restrictions. Breaking the cycle of recidivism is imperative for released individuals and their dependents to achieve upward mobility.

What role do evaluations play? In Chicago, Youth Guidance delivers Becoming a Man, a program inspired by cognitive behavioral therapy, to at-risk youth and criminally engaged men. A randomized evaluation found that arrests per student decreased by 12 percent by the end of the program, and participants were 9 percent more likely to graduate high school on time. 

How can we expand the evidence base? A randomized evaluation can investigate the impact of prisoner reentry programs on recidivism by varying the inclusion of wraparound social services such as employment preparation, education and skills training, mental health and substance use treatments, or housing assistance.