Real-time monitoring and response plans: Creating procedures

• Real-time monitoring alerts you to what you should be concerned about. If you’re not aware of any concerns at all, this may be a sign that you are missing information!
• Key indicators to monitor in randomized evaluations often include: fidelity to randomization procedures; take-up rates; and indicators of data quality.
• Automated systems for monitoring and generating red flags are useful, but only if a person is designated to actively review them. 
• There is no substitute for regular, frequent interactions with an implementing partner. Regular visits or calls can alert you to concerns as they arise.

While our companion resource on implementation monitoring covers the overall monitoring and documentation of a study (for example, interviews with staff and procedures for periodic, detailed back-checks), this resource is a more procedural guide for action plans around key indicators of changes and concerns.

A good monitoring procedure helps researchers learn about major deviations from plans by observing indicators of how the intervention, data collection, and other essential processes (e.g., data transfer or informed consent) are operating. Researchers should identify a limited set of need-to-know indicators and decide how to track them. These monitoring activities should continue throughout the design, pilot, and implementation of a randomized evaluation. This resource guides readers to:

1. Choose which indicators are most important to include in a real-time monitoring procedure,
2. Organize incoming data with automatically generated checks and flags, and
3. Set up channels of communication and plans for reviewing data and reporting concerns.

Indicators

Choosing a limited set of indicators¹ helps to cut through the noise of available information and alert the research team to changes and concerns.

Key indicators to monitor and check in real time may relate to data quality (e.g., the match rate when matching datasets), randomization fidelity, or rate of recruitment. Examples of key indicators include: “duration of each tutoring session” as an indicator of the implementation fidelity for a tutoring program; or “number of deviations from survey script” in a study of a health care intervention––as measured in a weekly random audit of audio files of phone surveys––as an indicator of the appropriateness of the script or of how well staff understand protocols. 

Selectivity in choosing key indicators enables researchers to identify important patterns and red flags quickly. Use your research questions, in conjunction with your budget and resource constraints, to identify the most important elements of the study to monitor actively. These considerations can help in the selection of indicators for a quick-response monitoring plan:

• Use research questions and practical constraints to decide which indicators to include in a real-time monitoring action plan and how frequently to check them. Consider the relative severity of possible concerns, budget, and staff capacity when choosing which indicators to monitor actively and how frequently to check them.
• Identify the points in a study when it is most important to follow protocols (or in other words, where there are the most concerns about things going wrong) and discuss these ahead of time with stakeholders. Researchers and implementers should discuss (1) what problems have arisen in previous versions of the intervention and (2) whether the study is introducing changes to the intervention. Pay extra attention to these areas in future communication.
• Assess the status quo before the study begins. Measure benchmarks of implementation for future comparison.

Data sources and data collection

Assess what data is available to report on key indicators and determine how to gather the information you need. Data sources for monitoring plans may include, for example: summary statistics of administrative data that are sent to researchers at regular intervals; survey responses sent to researchers automatically from electronic devices; or qualitative data shared during team calls.

Consider whether it is feasible to make data collection monitoring and review procedures the same for treatment and control (in order to preserve the ability to attribute differences to the treatment itself) or whether these need to be different due to practical constraints (Glennerster and Takavarasha 2013, 306). The monitoring methods themselves might affect the impacts of an intervention or policy. For example, if a monitoring plan includes asking people about the treatments they are or are not receiving, this might encourage them to seek treatment (Glennerster and Takavarasha 2013, 306). Research teams can assess these possibilities with the help of site visits, data, and conversations with partners.

Depending on the existing data systems of implementing partners, it may be mutually helpful for the research team and partners to work together to update systems and data collection procedures to support consistent implementation and monitoring.

Monitoring tables typically display a small set of indicators such as treatment intensity (if applicable) and the balance of indicators across treatment and control groups. See the appendix for an example of a monitoring table. Deviations that exceed thresholds of anticipated variation can prompt researchers to investigate further. Tables and reports should complement open communication with partners and within the research team.

Set up automatic flags and thresholds

Set up flags within monitoring tables and design procedures to ensure that implementers and research staff are empowered to report possible concerns even if they are not certain about what might be happening. Automatic red flags can prompt the designated reviewer to report concerns to the appropriate party. For example, if variable A drops below a certain number, then alert the primary investigator; if variable B increases rapidly, then talk with an implementing partner. There should also be defined ways to raise concerns based on information that falls below pre-set reporting thresholds but still causes doubt about whether all is as running as intended.

Balance checks test whether randomization is working as planned by testing for differences in observable characteristics between the treatment and control groups. High-frequency checks can be a helpful way to streamline a process of automatic flags and checks. Innovations for Poverty Action (IPA) has developed Stata templates for high-frequency checks designed for population-based survey research and randomized impact evaluations. These templates run high-frequency monitoring checks of incoming research data.

The research team should make a plan for reviewing monitoring tables and responding to concerns. The following steps will help the research team make the best use of monitoring tables and other sources of information.

Designate a primary reviewer

Designate one person to review the few absolutely key indicators in monitoring tables at regular, realistic, and clearly defined intervals. By reviewing tables regularly, the designated person will be able to detect and report if there are major changes, discrepancies, or other red flags in how the intervention is being implemented. Detailed plans for checking and responding to monitoring data are essential: without sufficient, yet manageable, procedures, monitoring tables may become overlooked and therefore ineffective. A good plan will catch major causes for concern and will be reasonable for staff to balance with their other responsibilities. 

Lower barriers to raising concerns about data and protocols

Although it is not possible to pre-specify how to respond to every possible situation, it can be helpful to set a few important thresholds (see above) and procedures to enable quick responses. This is especially important if junior staff, such as Research Associates (also known as RAs or Research Assistants), will be reviewing data and triaging concerns: staff should feel empowered to say, "hey, come look at this!" Pre-set procedures, combined with regular conversations with managers, can help RAs identify what must be flagged and reported immediately, versus what must be flagged but is not urgent, versus what should be documented but may not ultimately cause problems. Clear plans and open channels of communication help staff work together to make informed decisions.

Similarly, implementing partners should feel comfortable raising concerns about protocols, whether they are observing changes to protocols in practice, think that protocols should be changed, or both. If possible, discuss ahead of time how to communicate about and design potential changes to implementation protocols if an intervention begins to evolve over time.

Case study: In a case management intervention designed to help low-income students overcome barriers to college completion, researchers observed that differences in outcomes between treatment and control groups narrowed after two years (Evans et al. 2017). Outcomes included enrollment in college, GPA, and degree completion. After one year, outcomes for the control group were starting to look similar to the outcomes for the treatment group. In response to this type of change, researchers can ask program staff to discuss changes together. These conversations should not be framed as “the program not working,” but rather as a request to investigate the program together and explore what has changed. 

In this case management example, the study team read case notes and identified less engagement between students and counselors over time as a possible cause of the narrowing difference in outcomes between treatment and control groups. They found that the average length of time that counselors had been spending per session with students had declined starting at the end of the first year. This was not in line with the program’s guidelines and dampened the effect of the intervention. 
In order to adjust the intervention towards the program’s desired implementation, the program staff changed the guidance and requirements for counselors, asking counselors to ensure that each session met a specific minimum time requirement. A monitoring plan that flags narrowing outcomes across treatment and control groups, together with thoughtful communication between researchers and implementing partners, can prompt this type of exploration and adjustment.

Set up regular calls and visits with implementing partners

Implementing partners hold essential information about a study’s progress and any changes or challenges in the intervention or study. During the process of choosing indicators and data—and throughout the entire process of study design and implementation—research staff and implementation staff should set up regular calls for communicating updates and concerns. 

Regular calls with regular staff to discuss study details, updates, and progress in general, even if problems seem unlikely, mean that there will be opportunities to learn about problems as they occur.⁴ These regular check-ins help to create routines and establish understanding and trust that will be helpful if problems do arise. They also make it more likely that minor challenges will not worsen. Open and regular communication makes it more likely that issues that may seem minor to some, but are actually major issues, come to the attention of the team. There does not need to be an agenda for each call: the most important thing is to have a regular point of contact.

Site visits, if possible, are another way to both facilitate open communication with partners and actively observe how protocols are being implemented in practice. 

Consider designating a “field RA” or embedding a researcher on site

Designating a research associate to engage with implementing partners on the site of the study can be a valuable way to build relationships and open a channel for the research team to be aware of unexpected problems as they arise. A “field RA” may be designated to work on these tasks full time; or fielding activities may be woven into the responsibilities of an RA with more typical office-based, data-focused activities. Goals for a field RA can include both implementation monitoring and learning whether the planned protocols make sense to the partner or should be modified. 

A full-time field RA might be required for some projects, while other projects’ field monitoring might be accomplished via more infrequent visits from an RA who has both data and fielding responsibilities. Sending a research associate to a study site for a day-long visit, for example, can allow this member of the research team to learn, in real time, how protocols are being implemented. If a schedule of regular check-ins is set, it’s important to conduct these visits as planned even if no concerns have arisen yet; visits can help build good relationships and cement contingency plans for if concerns do arise.

Appendix table A: important yes/no checks

Appendix table B: custom balance checks

Appendix table C: survey duration

For example, if a survey is designed to take 45 minutes, then it may be helpful to track the number of surveys that are much shorter than expected—in this case, 15 minutes is a threshold for concern. A research team might decide that although short surveys do occasionally happen and a single short survey is not necessarily cause for concern, they want to track whether enumerators tend to frequently administer surveys shorter than 15 minutes. In this example table, one short survey is a “yellow flag.” However, if one enumerator has two or more surveys lasting less than 15 minutes within a certain monitoring period, this generates a “red flag” and may prompt a member of the research team to reach out to the enumerator to ask whether they are encountering difficulties.