|Home | About | Journals | Submit | Contact Us | Français|
Clinical informatics intervention research suffers from a lack of attention to external validity in study design, implementation, evaluation, and reporting. This hampers the ability of others to assess the fit of a clinical informatics intervention with demonstrated efficacy in one setting for implementation in their setting. The objective of this model formulation paper is to demonstrate the applicability of the RE-AIM (Reach, Effectiveness, Adoption, Implementation, and Maintenance) framework with proposed extensions to clinical informatics intervention research and describe the framework's role in facilitating the translation of evidence into practice and generation of evidence from practice. Both aspects are essential to reap the clinical and public health benefits of clinical informatics research.
We expanded RE-AIM through the addition of assessment questions relevant to clinical informatics intervention research including those related to predisposing, enabling, and reinforcing factors and validated it with two case studies.
The first case study supported the applicability of RE-AIM to inform real world implementation of a clinical informatics intervention with demonstrated efficacy in randomized controlled trials (RCTs) - the Choice (Creating better Health Outcomes by Improving Communication about Patients' Experiences) intervention. The second, an RCT of a personal digital assistant–based decision support system for guideline-based care, illustrated how RE-AIM can be used to inform the design of an efficacy RCT that captures essential contextual details typically lacking in RCT design and reporting.
The case studies validate, through example, the applicability of RE-AIM to inform the design, implementation, evaluation, and reporting of clinical informatics intervention studies.
Translational research has received heightened attention since the publication and initial implementation of the National Institutes of Health (NIH) Roadmap for Medical Research. 1,2 Although more focus has centered on breaking down the barriers between basic science and clinical science, several authors have emphasized the importance of removing the barriers between clinical science and translation of discoveries into routine clinical practice and healthcare policy. 3,4 Woolf, in particular, has suggested the need for additional resources to support the latter given the likelihood of greater impact on the public health. 5 The NIH Clinical and Translational Science Awards (CTSAs) 6 partially address this need through the funding of Community Engagement Resources and, in some instances, through supplemental funding to conduct pilot work related to the creation of a national network of community-based research sites. 7
Recognition of the importance of studying the real world implementation of efficacious interventions to address their effectiveness preceded these NIH initiatives. Models, such as Veteran's Affairs QUERI 8,9 and the RE-AIM framework (Reach, Effectiveness, Adoption, Implementation, and Maintenance), 10–13 have been proposed as methods to facilitate translation of research into practice. In addition, federal agencies 14 and others 15 have called for the conduct of so-called practice-based or pragmatic trials so the evidence generated through research is more likely to be appropriate for implementation in practice. The number of practice-based research networks (PBRNs) is on the rise and research conducted in practice settings has been identified as “a crucial scientific step, the blue highway, between the great medical advances of the next 25 years and the millions of Americans who want to live a long and healthy life.” 4 Common across these two approaches—using a theoretically based implementation model to apply evidence to practice and generation of evidence from practice-based research—is an increased consideration of external validity in addition to the internal validity of the study design.
Our premise is that, as with clinical science, clinical informatics intervention research suffers from a lack of attention to external validity (i.e., generalizability) in study design, implementation, evaluation, and dissemination. Moreover, lack of attention to external validity hampers the ability of others to assess the fit of a clinical informatics intervention with demonstrated efficacy in one setting for implementation in their setting. The RE-AIM framework addresses these concerns.
The purpose of this model formulation paper is to demonstrate the applicability of the RE-AIM framework to clinical informatics intervention research. First, we discuss the importance of such a framework for planning, implementing, evaluating, and reporting clinical informatics intervention studies. Second, we describe the RE-AIM framework and suggest additional assessment questions for clinical informatics intervention research. Third, we validate the use of the RE-AIM framework with its extension for clinical informatics intervention research through two clinical informatics intervention case studies. The first case study is focused on the real world implementation of a clinical informatics intervention with demonstrated efficacy in randomized controlled trials (RCTs)—the Choice (Creating better Health Outcomes by Improving Communication about Patients' Experiences) intervention. In this instance, the dimensions of the RE-AIM framework provide a model for describing the process of implementing Choice into routine care (i.e., translation of evidence into practice). The second case study is an RCT of a personal digital assistant (PDA)-based decision support system (DSS) for guideline-based screening and management of depression, obesity, and smoking cessation. This case study illustrates how the RE-AIM framework can be used to inform the design of an efficacy RCT that will generate evidence from practice through capture of essential contextual details typically lacking in RCT design and reporting.
There is a considerable gap between existing clinical research-based knowledge and its implementation into clinical practice. 16 This gap also exists for clinical informatics interventions. There are several reasons for this gap—some are related to insufficient capture and reporting of contextual details that can help others determine if the findings of the efficacy study are relevant and replicable in their setting. 17,18 This is very important in clinical informatics intervention research because in some instances most evidence for a particular intervention (e.g., computer-based provider order entry) can be attributable to only a few organizations whose organizational cultures may differ from settings in which others wish to implement such technology. In addition, because of their emphasis on internal validity, traditional RCTs have characteristics that hamper implementation of the results into practice. These include strict inclusion and exclusion criteria and ignoring or trying to control for variations in clinical practice. 18–21
There has been less attention given to the study of interventions under typical and varying, rather than optimal, conditions such as those in an RCT. 11,22 The need for studying clinical informatics interventions under such conditions has been recognized in the informatics literature. For example, in a model that matches stage of system development with level of evaluation, Stead, et al conceptualized the study of an informatics innovation under typical and varying conditions as the fifth level of evaluation, in which the focus was determination of intervention effectiveness and the reason for its effect. 23
Historically, many information systems have failed to achieve their potential because of lack of organizational leadership, organizational attitudes toward innovation, and inadequate attention to users' judgments about system feasibility, time requirements, and usefulness in clinical practice. 24–27 Thus, there is a need for research that combines intervention effectiveness with research about effective ways to integrate interventions into existing organizational contexts. This type of research is characterized by multiple names in the literature including implementation science, dissemination science, translational research, and knowledge transfer, 28–30 and the RE-AIM framework provides a model to inform such research.
The RE-AIM framework was designed to assist in the planning, conduct, evaluation, and reporting of studies with a goal of translation of research into practice. 31 As a complement to criteria that focus on internal validity such as the CONSORT criteria for RCTs 32–34 and the recommendations by Shcherbatykh, et al 35 for handling methodologic issues in health informatics trials, RE-AIM dimensions reflect a primary emphasis on external validity. 12 The framework has been mainly applied in the area of health promotion, 11,36 but has also been used more broadly as a framework for eHealth evaluation and dissemination 37 and translational research. 22
The RE-AIM framework differs from existing theories or models related to technology adoption or technology acceptance in several ways. First, although it is based on work in diffusion theory, the breadth of the framework is beyond technology adoption or acceptance. Second, the focus of the framework is research and its translation into practice. Third, the framework is intended to assist in the planning, conduct, evaluation, and reporting of research studies rather than to only guide the implementation of a specific innovation. Fourth, the dimensions of the RE-AIM framework emphasize external validity in study design and evaluation as a method for increasing the likelihood that a particular intervention will work either across settings or in a particular setting. The first aspect relates to the generalizability or robustness of findings across situations and populations. The second, which is paramount for organizational decision making, addresses the relevance of the evidence generated through research in a particular setting, population, and circumstance to another.
summarizes the dimensions discussed below with associated definitions and questions for assessment of research studies with the aim of translating research into practice. also includes our proposed questions for assessment of clinical informatics intervention studies. These expansions are particularly made for the Effectiveness and Implementation dimensions. Metrics have been developed to operationalize some RE-AIM dimensions 38 and conceptually, the impact of an intervention on an individual is a function of the Reach dimension multiplied by the Effectiveness dimension. 22
In the original RE-AIM framework the Reach dimension is an individual level measure (e.g., patient or employee) of participation, referring to the percentage of persons who receive a program. 11,36 However, there is a considerable difference between interventions that are administered to patients by a few dedicated program officers or research staff, and clinical informatics interventions that often involve at least two large target groups: clinicians who use the clinical informatics intervention as part of their care processes and their patients. This interdependency among user groups adds an additional layer of complexity that affects other RE-AIM dimensions.
Efficacy/Effectiveness refers to behavioral, quality of life, and participant satisfaction outcomes as well as physiological endpoints on an individual level, usually the patient. 11,36 In terms of Efficacy/Effectiveness for clinical informatics intervention studies, it is important to examine the impact on quality of care from both the process and outcome perspectives and to monitor unintended negative and positive consequences. For many clinical informatics interventions, the potential effect on patient outcomes is mediated by clinician behavior, i.e., if the clinician does not use the intervention, the patient cannot reap the benefits. Furthermore, when multiple clinicians use a clinical informatics intervention, the dose and consistency with which it is applied may vary greatly. In studies of these types of nested interventions it is crucial to measure outcomes on all user group levels, as well as monitor potential confounders, practice variations, and heterogeneity of patients and clinicians. 39 While this may require larger sample sizes than what is usually necessary in RCTs, inclusion of these data can greatly contribute to understanding how and why a clinical informatics intervention works, for whom and under what conditions, and how clinician and system variables may mediate effects on patient outcomes. Our proposed extension of the RE-AIM assessment questions for the Efficacy/Effectiveness dimension focuses on quality of care and unintended positive consequences. The latter is becoming increasingly important as Web 2.0 social technologies are being implemented into health care and have the potential to enable unanticipated, but positive effects, through the community of users.
Adoption relates to the absolute number, proportion, and representativeness of settings and intervention agents (people who deliver the program) who are willing to initiate a program. This differs from the typical way that the words adoption and acceptance are used in most innovation or technology acceptance models; these are integrated as part of the RE-AIM Implementation dimension and discussed in the following paragraph. 40 Multi-site studies are a rarity in the clinical informatics intervention literature. Moreover, study reports often provide insufficient information about the nature of the organization or organization in which the intervention was implemented which precludes an assessment of representativeness of the setting for the Adoption criterion.
The Implementation dimension refers to the extent and consistency to which a program is delivered across programs and settings as intended after it is implemented. 22,36 At the individual level, Implementation is the individual's use of the intervention strategies. 31 The framework does not specifically address the processes it takes to actually implement the intervention. Successful implementation depends on several factors at the individual, group, and organizational levels. 41 Our expansion of the assessment questions in the RE-AIM framework through addition of predisposing and enabling factor concepts from the PRECEDE-PROCEED model 42 is consistent with the need to capture the breadth of clinical informatics interventions and also to carefully document the clinical informatics intervention and its subsequent adaptations in new contexts. Predisposing factors occur before a behavior and influence motivation to undertake a particular behavior; for example, knowledge, attitudes, beliefs, values, self-efficacy, behavioral intentions, and existing skills. 43 Enabling factors also precede behavior and make it possible for individuals or populations to change their behavior or their environment. 43 Enabling factors for clinical informatics interventions include institutional commitment and central leadership support, extent of integration of the system into its organizational context, time to allow learning, investment in change processes, and adequate user training. 24,26,44,45
Maintenance refers to the degree to which a program becomes routine and part of the everyday culture and norms of an organization. 11,36 At the individual level, Maintenance is defined as the long-term effects of a program on outcomes for 6 or more months after the most recent intervention contact. 31 Due to the multidimensionality of clinical informatics interventions, reinforcing factors 42 are crucial to maintain the intervention. 41 For example, high turnover rates among clinicians, practice changes, and other individual and organizational factors may diminish commitment and use of a clinical informatics intervention after implementation. Conversely, factors such as remuneration, rewards, and awareness of positive attitudes and behaviors of others reinforce behaviors related to use of clinical informatics interventions. We have expanded the RE-AIM framework Maintenance assessment questions by adding a question related to reinforcing factors at the individual and setting/organizational levels.
Choice is an interactive tailored patient assessment tool designed to help clinicians (nurses and physicians) elicit and integrate patient-reported symptoms and health problems into patient care. Using a portable tablet computer, patients rate their symptoms and health problems according to physical, functional, and psychosocial dimensions, denote degree of bother, and prioritize need for symptom-related care. Choice creates an assessment summary that displays patient symptoms and distress ratings in rank-order of prioritized need for care. Choice's effectiveness in improving patient-centered care and patient outcomes has been repeatedly demonstrated in clinical trials with cancer and rehabilitation patients. 46–48 Use of Choice resulted in significantly increased frequency and precision of symptom and problem reporting, 49 higher preference achievement, functional status and patient satisfaction, 47,48 and greater symptom improvement and reduction in patients' prioritized needs. 50 In addition, when nurses and physicians in experimental groups had assessment summaries available for care planning, there was significantly greater congruence between patient-reported symptoms and those addressed in clinician documentation.
Based on the evidence for Choice efficacy, the five hospital units who had participated in the most recent RCT requested to use Choice in routine patient care. Supported by the nursing and medical leadership, the decision was made to apply for funding to implement and test the effects of Choice as part of routine clinical practice. Since previous study results had been obtained under the controlled conditions of a RCT, applicability and replication of study results in routine care were unknown.
The following paragraphs illustrate the application of the extended RE-AIM framework in the context of the Choice intervention. The specific research questions associated with each dimension are outlined in . To present a more time-oriented flow related to the implementation of Choice in routine practice, description of the Adoption, Implementation, and Maintenance dimensions precedes that of Reach and Effectiveness.
An important aspect of adoption is the match between the clinical informatics intervention and values or expectations of the organization or of the users. We conducted a set of focus groups with nurses and physicians, asking them about their experiences with Choice from the RCT and their future expectations. These evaluations demonstrated consistency between clinicians' values and purpose of the Choice intervention. Moreover, another indication of matching values was the user request to implement Choice in routine care. 51
During the implementation process, we paid careful attention to predisposing and enabling factors. Predisposing factors measured as part of the implementation process focused on clinicians' willingness to adopt Choice in their practice. Clinicians (65 nurses, 12 physicians) who had participated in the RCT completed questionnaires that addressed: (1) use, perceived usefulness and ease of use of the Choice assessment summaries to support patient care; (2) perceptions about Choice's ability to improve care planning, understanding of patient perspectives, and patient-provider communication; and (3) attitudes towards patient involvement in planning patient care. 51
In terms of patient predisposing factors, we also conducted in-depth interviews with 17 cancer patients who had used the Choice intervention during the RCT. These patients unanimously endorsed its use in practice. They reported that Choice was easy to use and that it enhanced their preparation for consultations, provided them with knowledge and understanding about their illness, and helped them to ask questions and participate more in their care. Additionally, patients felt that symptoms and problems were more “appreciated” and addressed by providers, and that they received more individualized care.
Factors that enabled implementation included: formation of an internal PBRN consisting of researchers and clinicians, adjustment of workflow processes, and modifications to Choice.
The internal PBRN consisted of the nurses from the five units, their head nurses, and members of the Choice research team. During the planning phase, network meetings were held regularly. The PBRN members identified potential barriers and benefits and defined clinician need for education and other enabling factors required to make Choice fit with their workflow.
When moving an informatics application from a RCT into routine practice, it is crucial to streamline it into the daily routines of the clinical practice setting so that its use is not perceived as a disruptive, additional task. Consequently, we needed to identify and describe processes related to outpatient consultations, patient admissions, discharges and other situations where the Choice application could be potentially used. Flowcharts were created for different use situations and delineated how patients moved through the system. This included whom they encountered, for what purpose, when, where, for how long, and at which points the Choice assessment would be most helpful and easy to administer. Based on this work, a preliminary set of workflow procedures were described and pilot tested and additional adjustments were made.
A clinical informatics intervention that works well in an RCT is not necessarily technically robust enough for large-scale clinical application and integration into existing systems such as an electronic health record (EHR). Modifications for Choice included:
In terms of reinforcing factors, we identified an unintended consequence of Choice use in routine care that required clinician education beyond mastering use of the application itself. Through focus groups, feedback from the PBRN, and monitoring participation rates, we discovered that clinicians also needed training to engage in a more patient-centered communication style, address patient problems from the patient perspective, and treat patients as equal partners. Choice challenged traditional roles which created insecurity among clinicians and some resistance to use. As a result, not all nurses offered the application to their patients, and some nurses and physicians did not use the assessment summaries and or ignored them when talking to patients. The patients who completed assessment summaries, by contrast, expected their problems to be acknowledged, and they were disappointed when this did not happen. Due to close monitoring and assistance from the internal PBRN, we were able to identify and address the problem. As a result, we conducted a series of communication training sessions for nurses and physicians with experts in communication with oncology patients. Furthermore, 4–5 nurses from each unit participated in a 4-day train-the-trainer workshop to become “experts” in patient provider communication. Subsequently, they learned how to conduct one-to-one training sessions and support their peers.
In the context of Choice, the question about the level of participation applies to two clinician user groups (physicians, nurses) and patients. On the patient level, we regularly collect data over a 2-week period on the ratio of Choice assessments in the EHR and number of admitted patients. If this number drops below a predefined acceptable rate, we plan to further study the processes related to nurses offering Choice to their patients and thus identify potential problems.
The purpose of measuring the effectiveness of Choice in routine care is to (1) assess whether effects of previous RCTs can be repeated or even strengthened under conditions of routine practice; (2) measure additional system, clinician and patient outcomes; (3) explore the relationships between these outcomes; (4) monitor unexpected outcomes that may occur; and (5) discover new, interesting research questions. We are examining these research questions using a pre-post implementation design with two cohorts.
To achieve the first purpose, we are using similar outcome measures as in the RCT plus additional outcomes such as patient-provider communication. For example, we are audio-taping and analyzing patient-provider communication style with and without the Choice intervention. Our experience to date suggests that implementing a clinical informatics intervention into routine care can provide new exciting opportunities for research and knowledge discovery. Furthermore, using Choice with all patients in the participating units generates a sufficient sample size to study effects of Choice under different practice conditions and in subgroups of patients with varying cancer diagnoses, stages of disease, or demographic characteristics. This allows us to address aspects of internal and external validity; and to answer new research questions that address important problems in clinical practice.
This example demonstrates the applicability of the RE-AIM framework to inform the translation of an efficacious clinical informatics intervention into routine care and to study its effectiveness under typical rather than tightly controlled conditions.
The PDA DSS was designed to support guideline-based care for the screening, diagnosis, and management of obesity, depression, and tobacco dependence and was evaluated in an efficacy RCT. The design included diverse participants and heterogeneous practice settings, two features identified by Tunis, et al 52 as important for increasing the value of research for clinical practice and policy, and incorporated data collection related to all RE-AIM dimensions.
The DSS was built on top of an existing PDA application for documenting patient encounters and provided: (1) a reminder to screen for one of the three conditions, (2) guideline-based screening assessment, (3) automated generation of diagnosis based upon screening, (4) documentation of patient goal (e.g., weight loss, quit smoking), and (5) guideline-based care plan template that is tailored based upon selected demographic characteristics, assessment, diagnosis, and patient goal. 53 Registered nurses in Advanced Practice Nurse training were randomized within clinical specialty (e.g., Adult Nurse Practitioner, Pediatric Nurse Practitioner) to receive the DSS for one of the three conditions and the control PDA application for the remaining two conditions. 54 The control application for each condition and its associated guideline contained all the diagnoses and interventions that were in the DSS. 55
For consistency with the Choice case study, Reach and Efficacy are described following Adoption, Implementation, and Maintenance. Research questions for each dimension of the framework for this study are shown in . The intent of this example is to illustrate how the RE-AIM dimensions were incorporated into the study design as guidance for other investigators wishing to integrate the framework, not to report the study findings.
Because we wished to recruit participants from heterogeneous practice settings, we selected guidelines that were appropriate across clinical populations, age groups, and settings. The nurse participants represented six specialties spanning acute care and ambulatory care settings in four states (Adult Nurse Practitioner, Acute Care Nurse Practitioner, Family Nurse Practitioner, Oncology Nurse Practitioner, Pediatric Nurse Practitioner, and Women's Health Nurse Practitioner). Settings ranged from small private practices in suburban settings to urban academic medical centers.
We implemented several additional study processes into the design to facilitate the implementation and maintenance of the intervention and the associated capture of data related to the Implementation and Maintenance dimensions of RE-AIM. For example, we logged training and booster training sessions, asked nurse participants to complete questionnaires regarding their perceptions of the DSS's ease of use and usefulness, and conducted focus groups to discuss barriers to use. 56
In terms of automatically capturing data related to the Implementation and Maintenance dimensions of RE-AIM, all data were stamped with time of data entry and were downloaded by the users from the PDA DSS to a central database on a regular basis. The database contained information about the users such as specialty, semester in educational program, setting, and which guideline they were randomized to receive. 55 Consequently, we could determine patterns of use and ascertain if use varied according to patient characteristics, Nurse Practitioner specialty, time, setting, or guideline. Database reports were run at the nurse-level and specialty-level approximately once per month. We iteratively refined the reports that were intended to reinforce usage of the PDA DSS.
The PDA DSS was available for use in adult and pediatric patient encounters by nurse participants. We established the age for screening eligibility at 2 years for obesity, 8 years for depression, and 9 years for smoking and captured data on the proportion of those eligible for screening that were screened. The PDA DSS also facilitated collection of data regarding patient age, gender, insurance status, and race/ethnicity, thus facilitating an assessment of the representativeness of those who were screened, diagnosed, and treated for the guideline-related conditions.
The primary outcome of the RCT was guideline adherence, which we measured in regards to three health problems with high significance to public health: obesity, depression, and cigarette smoking. Because the study design included randomization within specialty to guideline, subgroup analyses by specialty and by guideline were possible. Such analyses contribute to understanding under what circumstance an intervention is efficacious.
This case study illustrates the usefulness of the RE-AIM framework to inform the capture of relevant contextual data regarding Adoption, Implementation, and Maintenance dimensions within the context of an efficacy RCT. Inclusion of data collection and analysis related to RE-AIM dimensions often requires a variety of methods not typically integrated into efficacy RCTs and consequently additional resources. For example, we conducted focus groups to assess predisposing, enabling, and reinforcing factors. Where appropriate, data related to the RE-AIM dimensions should be automatically captured as part of the clinical informatics intervention.
Several authors have provided guidance on design and reporting of RCT results in a manner that attends to the internal validity of the research design. 32–35 This is essential in determining the efficacy of clinical informatics interventions. As a complement to criteria that focus on internal validity, we have described an expansion of the RE-AIM framework to address the external validity of clinical informatics intervention studies. This is a critical step in moving from measuring efficacy to measuring effectiveness of a particular clinical informatics intervention, i.e., Stead, et al's fifth-level of evaluation. 23 As with clinical science, 3,4 data related to the RE-AIM dimensions are vital to the understanding and removal of the barriers between clinical informatics intervention efficacy research and its translation into routine clinical practice and healthcare policy. Moreover, such data are likely to be critical for informing comparative effectiveness research of clinical informatics interventions.
The research described and preparation of the manuscript was supported by Mobile Decision Support for Advanced Practice Nursing (R01NR008903, S. Bakken, Principal Investigator) and the Center for Evidence-based Practice in the Underserved (P30NR010677, S. Bakken, Principal Investigator) funded by the National Institute of Nursing Research, and Communication and Information Sharing between Patients and Their Care Providers (Grant 176823/S10, C. Ruland, Principal Investigator) funded by the Norwegian Research Council.