Evaluating technology-based interventions

Technology-based interventions can offer many benefits, in particular the ability to standardize treatment across participants and sites. This can make it easier to replicate the interventions with fidelity across contexts, though the context itself can still change the effectiveness of the intervention (Pane et al. 2014; Roschelle et al. 2016). If interventions are shown to be effective, being technology-based also offers unique opportunities for rapid scaling and replication.

Evaluating technology-based interventions, however, poses some unique challenges. Generally, these challenges center around:

1. Research and technology design: designing both the intervention and the study well, and integrating them with one another;
2. Study implementation: executing the study, in particular being vigilant about randomization fidelity;
3. Interaction between the technology, study, and individuals: accounting for participant adoption of the technology, and ensuring that all implementers are aware of the study. 

This is by no means a comprehensive list, and even in taking all these measures, challenges may still arise. If any readers have additional studies or suggestions they would like to contribute to this resource, we encourage them to reach out to the J-PAL North America team at [email protected].

• Try to study pre-existing interventions instead of designing new ones. Researchers have repeatedly reported that developing new interventions is much slower and costlier than anticipated. One-time development of new technologies is usually insufficient, and these technologies may also have maintenance costs as underlying platforms or systems change.
• Work with a partner who has experience designing, building, and maintaining tech interventions. There is a world of technology-building expertise to draw on, and it is difficult for researchers to be experts in both creating and evaluating a technology-based intervention. Working with a partner, we recommend that researchers: 
  • Carefully assess the software platform early in the process. Software companies, especially those who are new to research, may promise more flexibility in their programming than is actually available. The more researchers probe earlier on, the faster they may be able to discover gaps between what is required for the study, and what is possible for the company to build. 
  • Ask early on about the timeline and cost of development—and budget extra. Even for updating old interventions, both timing and cost often end up being greater than anticipated. It helps to have buffer time and funding in case unexpected developer time is needed. 
  • Consider testing parts of the technology in different treatment arms. Because technological interventions can be standardized across participants in a way that interventions requiring human service delivery cannot, they allow for more precise study of mechanisms. With a large enough sample size, multiple treatment arms with different versions of the intervention may isolate these mechanisms (Oreopoulos and Petronijevic 2018). 
  • Use process and outcomes data collected from the system. For example, researchers may collect information on: whether participants open screens, how frequently they log in, how many times they click on the screen, whether they use specific software, and other process measures that give insight into how the intervention is used. 
  • Interim measures can also be actively obtained on the technology platform (e.g. through assessments or surveys). If researchers use these data as outcomes, however, they must consider whether and how to obtain similar measures for the control group. 

Triple-check the randomization procedure and sample

Before the study launches, the technology related to both intervention delivery as well as randomization should be heavily scrutinized. Implementing partners likely have expertise in running programs, but may have less experience with testing how random assignment fits within these programs. Researchers play a critical role in flagging the importance of piloting randomization and building robust and comprehensive test cases to ensure the fidelity of assignment.

• Pilot both the intervention itself and the randomization procedure. Researchers can work with implementing partners to run and explicitly document checks on treatment dosages for all individuals tracked in the study (i.e., treatment, control, refused/other). These tests should be done before the study begins. 
• Check the exact logic statement used to implement random assignment. For a given random assignment strategy, there are a number of ways it can be translated into a logic statement or code. It is essential that the researchers are provided the exact logic statement that is used to determine whether or not the intervention is active for an individual or group. All variables and technology employed in the logic statement should be examined and tested to ensure definitions are correct. See the Appendix for an example of such a statement. 
   

• Check test cases in a “live” environment, or as close to one as possible. It is important to test on the newest version of the software and on the live platform, whether this means in an online forum, in an Electronic Medical Record (EMR), or on an updated device. For example, some hospitals have a “training” environment for the EMR where doctors can practice putting in test orders and getting a sense of the workflow. However, this environment may differ slightly from the “live” EMR that has actual patient information and where physicians can put in real orders and notes. 

Verify random assignment regularly

Work with the partner to develop (preferably automated) monitoring tables, as well as a regular system for reviewing those tables. Monitoring tables display indicators of treatment intensity and/or key steps of implementation—such as proportion assigned to treatment, balance across treatment groups for different characteristics, or the duration of a certain step in an intervention—for a set of cases. An unexpected change in the results displayed in these tables could indicate a problem with delivery of the intervention. 

Be aware of software updates

Some questions to consider are: When do the operating systems on phones or computers change, does coding of the intervention need to change, or do software packages need to be re-loaded? If the intervention is embedded into something like an EMR, do version changes to the platform affect outcomes collection in any way? Operating systems can affect things like data storage, precision of digits stored, randomization algorithms, and pseudo-random seed generators.

• Understand your developer’s update schedule. Developers often have set release dates for updates —knowing their timing allows researchers to account for it in designing the study. Regardless of explicitly research-related updates, it is useful for researchers to have updates on and dates of software version changes, downtimes, and testing procedures for version upgrades.

Make sure everyone who can change the intervention knows about the randomized evaluation

With all randomized evaluations, buy-in from implementing staff is important—but with a tech intervention, even staff who are not involved directly with the randomized evaluation may make changes that affect the evaluation.

• IT staff should know how to answer help desk emails and calls related to the intervention or study. 
• The implementing partner should have a clear point of contact on the research team in case of problems or changes (e.g., power outages).

Try to think separately about questions of adoption versus efficacy

Adoption of new technology is often very low. Evaluating technology adoption and efficacy simultaneously can make it difficult to interpret results if lack of adoption causes questions of efficacy to be underpowered. This is a great reason to find implementing partners whose technology has proven high fidelity of usage (if the goal is to test efficacy). 

• Ways to improve take-up will vary depending on the intervention, but could include: social media advertisements, working with community representatives or organizations, recruiting participants in-person and setting aside time for them to engage with the interventions, and adding short-term incentives for interactions with the technology.
• Test for take-up and account for it in your power calculations. This can be done in a number of ways, including piloting the app and holding focus groups with participants (both those who do and do not use the app), building process measures into data collection (e.g., signing up versus logging in), and examining historical data based on this specific intervention. 
  • Piloting the intervention is especially critical if it is new or changed significantly. It can be especially useful to gather qualitative feedback and have discussions with participants to see how and when they use the technology. This feedback is most informative when the intervention is piloted wherever it is meant to be used (e.g., in a clinic, at a school, etc.). 
  • Low take-up has a large effect on statistical power for studies using intent-to-treat.¹
• Consider opt-out versus opt-in adoption of technology. If allowed by the IRB of record, automatic enrollment may be an option to dramatically increase take-up of an intervention.
  • It is critical to remember that repeated use of the technology (and thus delivery of the intervention) may still be low, even if all participants are automatically “enrolled.”
     

In this appendix, we explore a hypothetical research study to demonstrate how slight differences in code can have impacts on the composition of the study sample. We suggest tests designed to validate whether the code produces the intended sample.

Consider an evaluation of an education-related intervention for elementary school students. A data set of all students exists, but is not directly accessible to researchers. In order to identify the sampling universe for the study and assign students to interventions, they must describe the study inclusion criteria and randomization procedures to the holders of the data set. Researchers will receive a de-identified data set consisting of the study sample. 

The researchers decided to focus on public school students in the second, third, fourth, or fifth grades for whom English is their first language. To simplify for the data holder, they provide the following inclusion criteria:

• In 2nd, 3rd, 4th, or 5th grade
• Attends a public school
• Does not participate in English as a Second Language (ESL) classes

In 2nd, 3rd, 4th, or 5th grade

Error may be introduced if, for example, a typo results in the use of a strict rather than a non-strict inequality:

Attends a public school

Error may be introduced if the code does not properly account for all types of schools included in the dataset. For example:

Does not participate in English as a Second Language (ESL) classes

These methods may produce different results if ESL is ever missing for students not enrolled in ESL courses. 
 

Interaction of inclusion criteria

Researchers should verify that the code combines or applies inclusion/exclusion criteria appropriately. For example, if this code were implemented in Stata, it would include all charter school students, due to the syntax of Stata.

Test cases

Researchers may request the code used to create the de-identified data set in order to review the translation of inclusion criteria to code. We also suggest researchers request a codebook including data definitions, summary statistics, and all possible values of each definition included in the code. Further, we suggest the use of test cases to assess the code. For example, researchers can test edge cases – cases just inside or just outside of eligibility criteria. 
For this example, a request for a codebook and testing might include:

• Students by grade:
  • Test inclusion/exclusion of all edge cases: 1st, 2nd, 5th, and 6th grades
• ESL:
  • Test inclusion/exclusion of: student who attends ESL course, student who does not
  • Codebook for all possible values of "ESL" variable
• School type:
  • Codebook for all possible values of "School_type" variable and related indicators
  • Test inclusion/exclusion of students at each possible school type, and students where school type is missing
• Test cases for all possible interaction of the above test cases, e.g.:
  • A 3rd grade student enrolled in ESL courses at a public school
  • A 3rd grade student enrolled in ESL courses at a private school
  • Etc.

Last updated September 2021.

These resources are a collaborative effort. If you notice a bug or have a suggestion for additional content, please fill out this form.