Nurturing the null: Navigating Evaluation Challenges in Community-Based Care Management

Null results—when a study does not find significant impacts on chosen outcomes—can provide valuable insights for research and policies alike. However, it can be difficult for stakeholders to identify and leverage these insights. In J-PAL's null results blog series, we highlight randomized evaluations that yielded null results in order to elevate their lessons learned and inform future research. In this piece, originally published in American Society on Aging’s Generations, Aaron Truchil, Dawn Wiest, and Kathleen Noonan of the Camden Coalition discuss their experience finding and learning from null results.

The null effect finding in the randomized controlled trial (RCT) of the Camden Coalition’s signature care-management intervention, the Camden Core Model, was a significant moment for the organization, prompting reflection on the challenges of supporting and evaluating our work for patients with complex health and social needs (Finkelstein et al., 2020).

The finding was disappointing, but we did not steer away from widely disseminating the result and engaging partners and stakeholders in dialogue around it. We saw it as an opportunity to learn and further innovate, and to move the field forward toward more effective solutions.

This mindset drove the Coalition to publish two secondary analyses that found that higher intervention dosage (Wiest et al., 2023) and engagement (Yang et al., 2023) were associated with lower readmission rates, and to partner again with MIT’s Abdul Latif Jameel Poverty Action Lab (J-PAL) to publish a new analysis looking at intermediate measures of care coordination, like connection to primary and specialty outpatient care (Finkelstein et al., 2023). Central to this continued learning has been a combination of internal data capacity investment and cultivating effective research partnerships. This article reflects on our lessons learned moving forward from the initial RCT findings.

Camden Core Model and Lead Up to the RCT
The Camden Core Model, one of the Camden Coalition’s first initiatives, is aimed at providing critical supports to patients with complex health and social needs. The Core Model’s interdisciplinary care teams provide high-touch, time-limited (roughly 90 days on average) care management and care coordination, including home/community visits and accompaniment to appointments, to high-risk patients transitioning from the hospital back to the community. Through a Centers for Medicare & Medicaid Innovation grant, the Camden Coalition began scaling up the program in 2013 and had a stable, multiyear funding window. As the Affordable Care Act and other policy shifts promoted a proliferation in value-based payment models, there continued to be significant interest in evaluating programs like the Camden Core Model. Recognizing that there were significantly more patients who were eligible than we could support, the Camden Coalition decided to partner with J-PAL North America to implement a randomized controlled trial to evaluate the program’s impact on hospital readmissions.

Measure What Matters to Program Participants
About 2 weeks after the original RCT was published, we discussed the null effect of the model on hospital readmissions with our Community Advisory Committee (CAC). The CAC consists of former program participants and residents invested in improving the well-being of the South Jersey community. It was a humbling experience to share the findings with this group: they are a committee of our Board of Trustees and many of them were graduates of the program. They were underwhelmed with both the study and its conclusions, especially with what we chose to measure.

One member said, “You were obviously asking the wrong question,” and another added, “Did you ask them if they were going to meetings, or visiting family? I knew I was getting better when I did that.”

Those were questions we had not asked. In our quest to demonstrate that we could both do what’s right for our patients and reduce spending, we put such a high premium on hospital readmissions that we overlooked other crucial aspects of value. The RCT was constrained by time and resources. We also did not want to place undue burden on our staff and patients by expanding data collection efforts and the study time frame to achieve the sample size required to address a wider range of questions. While we can’t say we have given up on measures like readmissions and cost of care, we would not embark on another RCT without also measuring what our patients care about.

Prioritize Study Readiness
The Camden Coalition’s care-management model received increased attention following a 2011 New Yorker article by Atul Gawande. As resources followed, our program grew exponentially. Further, with the passage and implementation of the Affordable Care Act around the same time, the health policy world was hungry for answers, and new funding was available through the Center for Medicare and Medicaid Innovation. All of this pushed us to move more quickly than we were ready to. In hindsight, we knew our program was still maturing and developing know-how about effectively supporting patients at scale.

We have no regrets about participating in the RCT, as the Coalition and the field have gained a tremendous amount of knowledge from it. When we decide to do another RCT, however, we will take time to lay a solid foundation for it, similar to how medical and drug trials go through many phases before the ultimate clinical trial to determine efficacy or effectiveness. We are working to answer some key questions first, including which patients are most likely to benefit, how to define and measure implementation fidelity and dosage, and which outcomes are most appropriate and achievable for patients.

Plan for Post-Study Learning
The Coalition did not have a pre-existing plan on what to do once the study findings were published. We also were not funded to engage in post-RCT analyses except for some limited funding to maintain data, IRB agreements, and other study management support. Despite this, we wanted to capitalize on the clean RCT data set for quality improvement purposes and to examine the possibility that the model effectively reduced readmissions for certain subsets of patients or achieved other benefits.

The decision to re-analyze the data resulted in important knowledge. In a follow-up analysis recently published in Health Affairs, we worked with researchers affiliated with J-PAL North America and the Rutgers Center for State Health Policy to study broader healthcare utilization measures and additional value streams for complex-care programs. With access to Medicaid claims data, we demonstrated that our program significantly increased connections to primary and specialty care, as well as access to Durable Medical Equipment (DME; Finkelstein et al., 2023).

Alongside an earlier finding of increased SNAP uptake, these outcomes highlight the impact our teams had on patients’ daily lives. Access to durable medical equipment (e.g. oxygen supplies, wheelchairs), food stamps, or a primary care or specialty physician improves people’s lives in ways that are not fully captured by measuring readmissions across an entire patient panel.

In addition to the Health Affairs findings, our team collaborated with researchers at Kaiser Permanente to explore how varying intervention-participation levels may have affected our results. In some instances, our care team dedicated at least 100 hours working with a single client across several months, whereas, in other cases, the team struggled to even complete the initial home visit (Martinez et al., 2019).

The average impact across patients with differing levels of intervention engagement does not tell the full story. In a recent JAMA Network Open publication, we applied a novel and robust methodology developed by our Kaiser Permanente partners (Adams et al., 2022) to ask if the model was more effective for patients with greater engagement. Contrary to the earlier null result, we found that greater likelihood of participation was associated with significantly lower readmission rates, and that certain clinical and social characteristics were associated with varying degrees of intervention engagement.

Moving forward, we could either: a) refine our inclusion criteria to identify those most likely to engage and benefit, or b) build strategies to better meet the circumstances and needs of participants who face engagement barriers. For example, patients with housing instability or recent criminal justice involvement had lower engagement levels. Even prior to the findings, the day-to-day experiences of our care teams identified the inherent difficulties of building relationships, maintaining contact, and providing all the necessary supports for patients with legal and housing concerns.

This led us to develop two powerful new “add-ons” to our model: a Housing First program that provides housing vouchers and longer-term support, and a Medical Legal Partnership that allows lawyers to work hand-in-hand with the care team to remedy any legal concerns of the patient. As we look to the future, we will continue to find opportunities to strengthen our ecosystem of care and support the complex care workforce so that our participants can access the services they need.

Identify Long-Term Research Partners
At the Camden Coalition, we have long recognized that community-based organizations (CBOs) cannot simply delegate their program evaluation to external researchers; they must build internal data capacity through investments in data infrastructure and skilled data staff who can steer the ship, identify research and evaluation priorities, ensure program readiness, and build research partnerships. Unfortunately, it is rare that organizations like ours have access to funding streams for investments in data infrastructure and data analysis beyond program evaluation. While funders are keen to support RCTs, they often overlook the preparatory groundwork needed for a CBO to successfully implement an RCT, or the subsequent analysis necessary to further shed light on initial findings. Consequently, much of our post-RCT work has been self-funded.

Forging strong relationships with outside researchers has been a critical strategy. Over the years we have been privileged to partner with a talented group of researchers, including MIT’s J-PAL, Rutgers Center for State Health Policy, Kaiser Permanente, the University of Massachusetts, and PolicyLab at the Children’s Hospital of Philadelphia. We are often approached by more researchers than we have the capacity to work with at any given moment, and when we decide who to partner with, we ask ourselves the following questions:

• Whose research expertise most aligns with our internal values and priorities?
• Is the researcher looking for a one-off project or are they looking for longer-term engagement?
• What is the researcher’s history in the community? Have they demonstrated the ability to gain trust with program participants and community stakeholders? Do they have a history of engagement and cultural competency with the populations we work with?
• Is the researcher willing to adapt and evolve their thinking alongside our team?
• Does the researcher see us as true partners and collaborators, rather than as conduits to the populations and data they need to pursue their research ambitions?

Our research partners often work with us based on the funding we have available for evaluation or research, and this rarely covers the true cost of their work. We are grateful they are willing to do this, but would prefer research funds to be more easily available to CBOs, especially during the critical early stages of evaluation that are needed to prepare for larger studies.

Going Forward
The Camden Coalition will continue its research and evaluation work, both for its Camden Core Model as well as its broader portfolio of demonstration projects. Working with a researcher at the University of Massachusetts, we received a National Science Foundation grant to conduct another secondary analysis of the RCT study population to refine our understanding of patient subgroups. We’ve partnered with the Walter Rand Institute at Rutgers University–Camden to conduct a mixed methods evaluation of our Medical Legal Partnership, and we are conducting an internal evaluation of our Pledge to Connect program that connects individuals with recent mental health–related emergency department encounters to outpatient behavioral health providers. With each of these efforts, we hope to replicate the learning process articulated in this article and continue to grow the evidence base for complex care. We look forward to continuing to learn from and share more with the field.