PMCCPMCCPMCC

Search tips
Search criteria 

Advanced

 
Logo of nihpaAbout Author manuscriptsSubmit a manuscriptHHS Public Access; Author Manuscript; Accepted for publication in peer reviewed journal;
 
Res Nurs Health. Author manuscript; available in PMC 2010 August 17.
Published in final edited form as:
Res Nurs Health. 2009 February; 32(1): 110–121.
doi:  10.1002/nur.20309
PMCID: PMC2923040
NIHMSID: NIHMS225375

Intervention Research in Highly Unstable Environments

Kathleen C. Buckwalter, PhD, RN, FAAN, Sally Mathis Hartwig Professor
Gerontological Nursing Research University of Iowa, College of Nursing, Iowa City, IA
Margaret Grey, DrPH, RN, FAAN, Dean, Annie Goodrich Professor
Yale University, School of Nursing, New Haven, CT
Barbara Bowers, PhD, RN, Professor
University of Wisconsin-Madison, School of Nursing, Madison, WI
Ann Marie McCarthy, PhD, RN, FAAN, Professor, Chair
Parent, Child & Family Area University of Iowa, College of Nursing, Iowa City, IA
Deborah Gross, DNSc, RN, FAAN, Leonard and Helen Stulman Professor
Mental Health and Psychiatric Nursing Johns Hopkins University, School of Nursing and School of Medicine, Baltimore, MD
Marjorie Funk, PhD, RN, FAHA, FAAN, Professor, Director
Doctoral Program Yale University, School of Nursing, New Haven, CT
Cornelia Beck, PhD, RN, MSE, Professor

Abstract

This article highlights issues and presents strategies for conducting intervention research in highly unstable environments such as schools, critical care units, and long-term care facilities. The authors draw on their own experiences to discuss the challenges that may be encountered in highly unstable settings. The concept of validity provides a framework for understanding the value of addressing the many methodological issues that can emerge in settings characterized by instability. We explain unstable environments by elaborating on knowable elements that contribute to instability. Strategies are provided for improving success of intervention research in unstable settings by carrying out an environmental assessment prior to beginning a study.

Keywords: Research settings, unstable environments, intervention research, research strategies

Nurses work and carry out research in a wide range of settings, with some settings more prone to unpredictable changes compared to others. There are always challenges to carrying out research in real world practice settings. These include barriers to identifying potential participants, practice cultures that are not supportive of research, and problems outside the control of the researcher or the study setting, such as natural disasters. Intervention research, which typically includes obtaining data at multiple times, often with the assistance of employees at the study site, is particularly vulnerable to highly unstable environments, which can have negative outcomes on the success of the studies.

The purpose of this article is to highlight issues and present strategies for conducting intervention research in highly unstable environments such as schools, critical care units, and long-term care facilities. The authors draw on their own experiences in carrying out intervention research to discuss the challenges that may be encountered in these highly unstable settings.

Although all research settings have the potential for instability, the focus of this paper is on settings that are known to be highly unstable such as those undergoing rapid change characterized by high rates of attrition/ turnover of staff, students, or patients, or that are subject to unpredictable external forces, such as regulatory change. These elements combine to make it essential for researchers to assess a priori the environment of potential study sites for unstable factors that may affect research outcomes and the ability to complete investigations in a timely and rigorous manner. Although many of the issues set forth can be considered best practices in any research setting, the challenges and strategies discussed are particularly germane to unstable environments. In addition, we acknowledge that some challenges to research in these settings cannot be anticipated, but others may be identified early and effectively addressed.

The concept of validity is presented first to provide a framework for understanding the methodological issues that can emerge in settings characterized by instability. Next we explain unstable environments by elaborating on knowable elements that contribute to instability. Last, we propose a systematic, investigator- initiated, environmental assessment with strategies that may increase the likelihood of success when carrying out intervention research in unstable settings.

Validity and Significance

Intervention research requires attention to issues of internal and external validity. Otherwise, findings may be subject to alternative explanations or may not be applicable to other relevant settings. Research conducted solely in highly controlled environments (e.g., laboratory) may be questioned not only in generalizability, but also in translation to the real world. Research conducted in real world settings may fill important gaps in science with regard to care delivered in these potentially unstable settings and provides the opportunity to answer complex, rich questions. To do so, however, requires that the researcher understand the potential threats to internal and external validity inherent in conducting research in such environments.

Shadish, Cook, and Campbell (2002) provided a framework for examining internal and external validity. We use this framework to illustrate potential threats to the integrity of intervention research in unstable environments and to offer strategies to minimize threats to validity. Shadish et al., defined validity as “the approximate truth of an inference,” (p. 34) and noted that validity is a property of the inference rather than of the design or methods. There are four types of inferential validity – internal, external, construct, and statistical conclusion. All are important, but internal and external validity are particularly important in unstable environments. Internal validity refers to the degree to which inferences about the outcomes of an intervention reflect a true causal relationship and are not the result of extraneous variables (Burns & Grove, 2005). External validity refers to the degree to which inferences about cause and effect relationships hold over variation in persons, settings, treatment variables, and measurement variables (Shadish et al., p.38). That is, can the findings of the study be generalized to other individuals and settings?

Construct validity concerns “making inferences from the sampling particulars of a study to the higher order constructs they represent” (Shadish et al., 2002, p. 65). In unstable environments, it is possible that measurement properties will vary by setting, but this is less of a threat than internal or external validity. Similarly, statistical conclusion validity refers to the validity of inferences about the association between treatment and outcome. The threats to statistical conclusion validity are important, but are not likely to be higher in unstable environments. Threats to validity are the reasons inferences can be partially or completely wrong. In the following paragraphs, threats to internal and external validity that are particularly problematic in conducting research in unstable environments are highlighted.

Internal Validity

In a randomized controlled trial designed to test the efficacy of a multifaceted school-based intervention to prevent type 2 diabetes in high-risk youth, Grey and colleagues (2004) encountered a number of issues that had the potential to affect the internal validity of the study. These included multiple changes in school personnel, including principals and teachers, community mistrust of research, yearly changes in curriculum requirements, pupil mobility, and rescheduling of mastery testing (Grey et al., 2004). Grey and colleagues used a number of approaches to improve the quality of the data, including establishing long-term relationships, providing ongoing training for new teachers, and carefully assessing intervention fidelity including both adherence and competence. Although any of a number of threats to internal validity (Fogg & Gross, 2000) can affect the quality of inferences to be drawn from intervention research conducted in unstable environments, those more likely to occur are ambiguous temporal precedence, history, attrition, treatment fidelity, and compensatory equalization of treatments.

Ambiguous temporal precedence is lack of clarity about the time order of the presumed cause and effect (Shadish et al., 2002). In chaotic, busy environments, the temporal implementation of an experimental treatment may not occur as intended. For example, in a study of an intervention in an inner city school system, snow days and changes in teachers resulted in a great deal of variability in the timing of implementation of the intervention, despite careful training. Such occurrences may represent a history threat as well (Grey et al., 2004).

History refers to events that occur between the beginning of the intervention and the outcome measurement that could influence the outcome. In nursing homes, for example, the introduction of new leadership could lead to changes in practice that have nothing to do with the intervention being studied but may, nonetheless, have an impact on the intervention and/or the outcome. Another example of the influence of history on intervention research occurred in a study examining the effect of the presence of dedicated monitor watchers on patient outcomes in an acute cardiac unit (Funk, Parkosewich, Johnson, & Stukshis, 1997). Contrary to what was hypothesized, there was a significantly higher incidence of serious bradyarrhythmias during the time when monitor watchers were present. Because of changes in admitting practices in this unit between 1993 and 1994, significantly more patients had an admitting diagnosis of bradyarrhythmias during the period with monitor watchers (1993) than during the period without monitor watchers (1994). Admitting practices, rather than the presence of monitor watchers, could have accounted for the difference in occurrence of serious bradyarrhythmias. In general, threats to validity from history are best controlled with the use of an intervention and control group from which data are collected concurrently rather than sequentially, but even then, a randomly assigned experimental unit could have events occur that do not happen in the other experimental or control units.

Attrition refers to the loss of participants from a study. In unstable environments, staff turnover may be high, and patients may be more mobile or likely to die, as in nursing homes or critical care units. Nursing homes that are not supportive of staff may experience high turnover rates; thus, interventions delivered to staff may appear to be ineffective because the experimentally trained staff left the setting before the impact of the intervention could be felt. In inner city schools, not only will teachers move from school to school, but students will as well, leading to problems with follow-up. In these examples, it is difficult to tell whether an observed outcome is because of the intervention or because those who stayed were more likely to improve. Differential attrition across comparison groups is potentially more likely in unstable environments, and is particularly problematic when the unit of analysis is the setting itself (e.g., school, nursing home). That is, two nursing homes may have very different baseline attrition rates, if, for example, one home has a higher percentage of end of life residents. Similarly, schools, even in different areas of the same city, may experience vastly different student and staff turnover rates, leading to differential attrition in the samples.

Treatment fidelity refers to “methodological strategies used to monitor and enhance the reliability and validity of behavioral interventions” (Borrelli et al., 2005, p. 852) to minimize errors in interpreting outcomes, in order to ascribe those outcomes directly to the intervention (Spillane et al., 2007). In intervention research, the intervention is typically provided by either a member of the research team or a staff member at the site. Although maintaining treatment fidelity is important for all intervention studies, one critical element, provider or interventionist training (Bellg et al., 2004), is particularly important in unstable environments where staff turnover may be more likely to occur, and where complex protocols that are different from routine practice are difficult to adhere to (Fogg & Gross, 2000). If staff members need to be the interventionists, then careful attention must be paid to their training and monitoring to enhance treatment integrity (Bellg et al.).

Compensatory equalization of treatments refers to a situation where one group receives more “desirable goods and services” (Shadish et al., 2002, p. 79) resulting in staff or administrators being troubled by subsequent inequities in treatment. As a result, staff may unintentionally or intentionally intervene to provide additional benefits to control group members, effectively jeopardizing treatment integrity. This threat to internal validity may be a particular issue in unstable environments, often comprised of caring providers and vulnerable populations, such as children and frail elders (Lynch, Whitley, & Willis, 2000).

External Validity

Threats to external validity include the interaction of the causal relationship with participants, treatment variations, outcomes, and settings as well as context-dependent mediation (Shadish et al., 2002). In other words, different types of people, treatment approaches, effects, and settings may lead to differing outcomes, limiting the generalizability of study results. A train-the-trainer model may work well for teachers with certain characteristics but not others, and in another, more chaotic school environment, it may not be effective at all. Settings with variable characteristics, such as critical care units with variations in patient acuity, may make data collection inconsistent, leading to variations in outcomes that have nothing to do with the intervention. For example, nurses who are charged with recruiting study participants may recruit fewer patients during busy times, and it is possible that the patients recruited may vary on important characteristics. Context-dependent mediation refers to the potential for an explanatory mediator of a causal relationship in one context to not mediate in another. This is a particularly insidious threat to external validity in intervention research in unstable environments as it is often difficult to identify mediators that act similarly across settings, and the influence of mediators is rarely tested.

In summary, nurses who conduct intervention research in potentially unstable environments must attend to the issues of internal and external validity that are particularly salient to these settings. They need to examine the presence of these identified threats to validity, a priori, more carefully than researchers in more stable, controlled environments and to take steps to address these threats. With these concerns in mind, we next identify issues within the environment and outside the environment that can contribute to instability, followed by a discussion of strategies to address these concerns.

Elements that Contribute to Unstable Environments

Within the Environment

Staff Turnover and Attrition

A high level of staff turnover is common to many unstable environments. Ensuring that staff involved in a research study are sufficiently trained in identifying participants, implementing an intervention, and collecting data following an intervention can be serious challenges when staff turnover is high. In addition to the loss of staff with sufficient understanding of the research study, ongoing turnover can undermine commitment to the project. Knowing the turnover rates a priori for the setting (school, hospital unit, nursing home) is important, as some setting types are more affected by high turnover than others. For example, front-line staff in nursing homes have an average annual turnover rate of about 80%, but range from 20% to 150% annually (Banaszak-Holl & Hines 1996). This would be important to know before initiating a study. Many nursing homes have turnover rates of over 100% annually (Bowers, Esmond, & Jacobson, 2003; Castle, 2005). Annual turnover rates less than 40% are considered low in the nursing home industry. Thus, although 40% turnover might be considered extremely high in other types of settings, it represents a very stable nursing home environment. The researcher working in nursing homes should always plan for such turnover rates, carefully accounting for staff turnover in the research design. It would also be important to find out what an organization or unit's turnover rate has been, as the best predictor of future turnover rates is turnover rates in the recent past (Castle).

Staff turnover rates can be determined prior to selecting organizations to include in an intervention study. Turnover data must be carefully assessed because there are many different ways to calculate staff turnover. For example, one study conducted in California nursing homes revealed over 40 different formulas used by nursing homes to estimate front-line staff turnover rates (Harahan et al., 2003). Thus, the researcher may need to recalculate turnover rates, using a consistent formula, in order to make accurate comparisons across facilities. For research conducted in schools, most cities and towns keep records of teacher and staff turnover rates and vacancies. Whenever a choice is available, researchers may want to select sites with lower turnover rates.

Turnover at the management level, regardless of whether managers are directly involved in the study, is also an important consideration, as demonstrated by Agency for Healthcare and Quality (AHRQ) funded translation research studies (TRIP-II) that underscore the importance of stable leadership (Jones et al., 2004) and strengthening the infrastructure (AHRQ, 2000). When managers are actively involved in the study, high rates of management turnover would, of course, likely undermine the successful completion of the study and might even lead to termination of studies approved by previous administrations. The support of management is necessary for successful follow through by staff involved in a research study. New managers may be reluctant to commit to an ongoing study, either by active participation or by supporting the participation of others in the organization. They may have priorities that are at odds with the study or may simply be too overwhelmed by the new role to support research. Competing priorities may also lead to the postponement or suspension of a research project. Although it is not always possible to predict this in advance, inquiring about plans for major organizational changes would alert the researcher to those that can be known in advance. This might result in an alteration of the timeline or eliminating a facility altogether. The significance of management support is often overlooked or underestimated by researchers. For example, in choosing one critical care unit over another for a study employing an education intervention for nurses and evaluating the long-term effect on quality of care measures, it is preferable to use a unit with a history of strong leadership and minimal staff turnover.

Workforce Issues

In addition to turnover, there are several workforce issues that can affect the validity of an intervention study (Mentes & Tripp-Reimer, 2002). These include changes in staff scheduling systems, internal staff rotation, the development of specialty units, and changes in policies related to shift work. In addition, financial status changes in an organization often result in changes in staffing and staff configuration such as ratio of professional to ancillary staff, changes in duties of staff, and changes in reporting structures. In both hospitals and nursing homes, the ratio of RNs to other nursing staff has been correlated with a range of patient and resident outcomes (Aiken, Clarke, Silber, & Sloane, 2003). Horn, Buerhaus, Bergstrom, and Smout (2005) found adverse outcomes such as pressure ulcers were decreased when RNs spent more time on direct patient care. Aiken, Clarke, Cheung, Sloane, and Silber (2003) concluded that in hospitals with higher proportions of nurses educated at the baccalaureate level surgical patients experienced lower mortality and failure-to-rescue rates. Similarly, in schools, the recommended ratio of 1 school nurse to 750 children is increasingly recognized as optimal to providing quality care for children (National Association of School Nurses, 2006), and the American Academy of Pediatrics (2008) now recommends that all schools employ a school nurse. These and other workforce-related issues may impact the validity of a study and should be controlled for whenever possible.

Stability and Quality of Data Sources

The stability and quality of data sources are crucial considerations. Although some studies use research staff to collect data, many studies rely on data that are routinely collected by organizations for reasons other than research. Intervention research often uses data from the existing records or data sources of a setting, such as the Minimum Data Set (MDS) from nursing homes or standardized achievement scores in schools, for descriptive, matching, predictive, or outcome measures. It is important to explore whether an organization plans to continue using a particular tool or database if it is an essential component of the data collected for the study. Researchers have sometimes put tremendous effort into gaining entre, generating a commitment from the managers in an organization, and training staff in various roles only to discover that a new data base, one not compatible with the study aims, is about to be implemented. This may be anticipated in advance by checking with the department or staff overseeing the operation of the database or the selection of tools. In nursing homes the MDS is mandated by the federal government and is thus collected by all nursing homes. Although well-trained MDS assessors can provide valid and reliable data, there is wide variation in the quality of MDS assessors in the real world (Bates-Jensen, Simmons, Schnelle, & Alessi, 2005; Bowers & Nolet, 2006). Important considerations in determining the quality of MDS data are whether knowledgeable trainers had trained the assessors, if inter-rater reliability or data recording has ever been assessed, and whether there has been turnover in the MDS assessors since the assessment of data recording quality.

When the data source is the patient, the researcher needs to consider both the potential attrition of participants between multiple data collection points and the patient's physical ability to participate. Patients in critical care units and long term care facilities may have higher attrition due to death, while children in school may fail to complete a study due to changing grades, schools, or school systems. In critical care units, the data source (the patient) is usually not physically stable. Especially with critically ill or injured patients, patient needs are a priority. It is imperative that a researcher makes an accurate assessment of an individual's immediate care needs prior to examining, testing, or interviewing. These needs – whether to be suctioned, have a drink, or get medication for pain – must be met prior to collecting data from the patient. Although it is important and helpful to keep staff informed about the study, their involvement in carrying out the study should be minimal.

Organizational Mission and Structure

Although not extensively researched, organizational mission is one aspect of the setting that has been shown to exert an independent influence on multiple variables. In relation to nursing homes and hospitals, this has primarily been studied by comparing for-profit and not-for-profit organizations on a variety of staff and patient variables (Baker et al., 2000; Castle & Engberg, 2006; Harrington, Woolhandler, Mullan, Carrillo, & Himmelstein, 2002; McGregor at al., 2006; O'Neill, Harrington, Kitchener, & Saliba, 2003). Higher turnover rates, poorer care outcomes (McGregor et al., 2006), higher mortality rates (Devereaux et al., 2002), decreased patient satisfaction (Tu & Reschovsky, 2002) and more deficiencies (Harrington et al.) have been shown to be higher in for-profit facilities, potentially having a significant effect on study outcomes if not controlled for.

The fidelity of an intervention and even the survival of a research project can be affected by structural changes in an organization. Internal reorganization can sometimes be anticipated, such as when organizations are considering adopting a new model of care, a departmental reorganization, a new staffing configuration, a merger, or a significant quality improvement program. These changes might result in altered access to information, relationships among staff, or involvement of staff in key decision making, all with potential to affect an intervention study (Gifford, Zammuto, Goodman, & Hill, 2002). Sometimes these changes are in response to oversight from regulatory bodies or accrediting organizations and cannot necessarily be anticipated. Some organizations are more prone to reorganization than others, such as those with frequent changes in management. Knowing the organization's history is important.

Consistent differences in outcomes also have been found between horizontally integrated (chains) and freestanding healthcare organizations. Freestanding healthcare organizations generally have lower turnover rates of both staff and managers (Harrington et al., 2002). There are differences between the two organizational structures in relation to the rapidity with which new policies or practices are adopted, staff participation in decision making processes, and organizational commitment of staff. Evidence concerning quality differences between the two types of organizational structure is so compelling that one state (New York) has legislated the exclusion of corporate structures, effectively excluding many for-profit chains from operating within the state (NYS Assembly Bill A 04436, 2008: NYS PL # 2801).

Another feature of organizational structure that may contribute to instability is the form of governance, in particular, the locus of decision-making and the flow of information. The success and stability of practice changes in hospital settings have been shown to rely in part on staff inclusion in planning the substance of the change and the strategies that will be used to implement the change, as well as the availability of information to those who will be directly involved (Mezey et al., 2004). In schools, a setting where education, not healthcare, is the focus, principals are often the primary decision makers, even for health issues. Understanding of a research study may vary across principals and influence participation by other educational staff and families.

There is some evidence that nursing homes involved in active quality improvement (QI) processes are more likely to engage successfully in practice change than those with little or no experience in doing so (Castle, 2005; Eaton, 2000; Yeatts & Seward, 2000). Evidence is suggestive that having an active, ongoing QI process reflects a higher level of organizational sophistication and integration, which may be related to organizational stability. This would also have an impact on the success of implementation of the intervention and might be an important independent variable to include. These organizations might also be more receptive to research activities. Moreover, administrators at acute care hospitals that are applying for or that recently achieved Magnet status, for which staff participation in research is highly valued, may be more likely to agree to have a study conducted on their units.

Outside the Environment

Assessing the stability of an environment should also include examination of factors external to the setting that might have an influence on the internal environment such as pending or recent changes in ownership or public policy. Issues outside the environment are discussed next.

Changes in Ownership of Organization

Some organizations have a history of frequent changes in ownership, which brings unpredictability in general (Ghobadian & O'Regan, 2006), as well as increases in staff turnover, increased costs with subsequent impact on resources available, changes in the flow of information and decision making authority, changes in policy and staffing patterns, and changes in contracted services that may influence the organization of work (Ferlie, 1997; Holmes, 1996). Often it is not possible to predict future changes in ownership although there are some indicators, such as recent financial shortfalls, multiple or recent cuts in services and resources, and the sale of other facilities owned by the same parent organization. Researchers should be aware of the potential impact of ownership on study outcomes and attend to the aforementioned indicators.

Legislative Mandates/Quality Improvement Organization

The health care industry is highly regulated. Federal, state, and local legislation all influence how health care organizations operate, including data that are collected on a daily basis, organizational priorities, staffing patterns, benefits available to employees, and training programs offered (Rastegar, 2004). In some instances, data collection methods are prescribed, in others, specific data are mandated but the methods used to collect the data are determined by the individual organizations and may consequently vary considerably.

In addition to legislative mandates, federal and state agencies sometimes reinterpret the legislation regulating health care organizations, leading to substantial changes in oversight, and in turn, in the priorities of health care organizations. This may occur at local, regional, state, or federal levels. For example, nursing home surveyors may be directed by the state or federal government to focus on a particular issue (e.g., pressure ulcers), whereas local groups, or even individual surveyors, might decide that something else (e.g., weight loss) should have the closest scrutiny (Schwartz, Ozmiinkowski, Hoaglin, Cella, & Branch, 1994). Once it becomes known that a particular group of surveyors is focusing on a specific issue, homes in the same survey region often shift their quality improvement efforts to prepare for the anticipated survey visit. This sometimes happens very quickly. And because the nursing home environment is so strongly influenced by regulatory demands, a facility may agree to focus on research on a particular clinical issue (e.g., hydration management) but be easily distracted if the quality assurance focus is shifted to another clinical issue (e.g., depression) because of federal or state initiatives.

Shifting priorities and competing initiatives can also influence research done in critical care units. For example, a regulatory agency may force a focus on the prevention of ventilator-associated pneumonia at a time when an investigator is starting a study addressing electrocardiogram monitoring practices (ECG; Funk et al., 1997). If the staff is undergoing mandatory education on the prevention of ventilator-associated pneumonia and being evaluated on preventive interventions, they may have little interest in participating in a new study involving education related to ECG monitoring. Changing priorities within a research site may also limit sustainability of a practice change, even when it is initially successful.

Some external changes occur at a national level. For example, the federal government recently gave direction to the state Quality Improvement Organizations to invest a significant amount of their organizational effort toward the improvement of nursing home care (Centers for Medicare and Medicaid Services, 2008). Although each state approached the mandate differently, in most states this resulted in a sudden shift in focus for accountability. Similarly, the No Child Left Behind (NCLB) law (U.S. Department of Education, 2002) has had an impact on schools in a number of ways, including limiting the ability of teachers to influence school schedules. Because of NCLB, curricula are more prescribed and standardized achievement tests as measures of accountability are a primary focus. The percentage of non-proficient children is crucial to funding, and efforts to improve proficiency are paramount. Research not related to NCLB may therefore not be valued.

Surveyor citations or lawsuits filed against an organization can make the organization suddenly much more aware of a particular aspect of care delivery, leading to policy changes, loss of local control and autonomy, and a diversion of resources to address the deficiency. In turn, this may lead to fewer available resources for discretionary activities such as research. In many states nursing home surveyor visits are unscheduled and unannounced. When they occur, any essential research activities carried out by personnel (such as staff rating of behaviors or recording of particular events) become a low priority and are less likely to occur, regardless of incentives.

Thus, competing external priorities may impact the ability to successfully implement a research project. In some cases this may force the researcher to postpone project implementation or to introduce a project in stages. Despite the number of issues inside and outside the environment that can affect the validity of intervention studies in highly unstable environments, researchers can implement a variety of strategies, discussed next, to minimize threats to validity.

Strategies

Realistic design for conducting studies in highly unstable environments needs to be built on a thorough assessment of the environment and its resources in order to minimize threats to internal and external validity. This assessment should occur prior to designing and implementing a study and involves identifying (a) aspects of the environment unlikely to change during the course of the study, (b) aspects of the environment likely to change during the course of the study, (c) changes that, if they occurred, would lead to study failure, (d) benefits and barriers of the research for stakeholders, and (e) compensation and/or non-coercive incentives that might motivate people and agencies to participate in the study.

Researchers are advised to enumerate the five aspects of environmental assessment listed above for the research site, and to examine their lists for strengths, weaknesses, and potentially fatal flaws. As part of the preliminary research process, this strategy should complement and augment pilot work in the research setting. Pilot studies provide an opportunity to test the feasibility of the methods, pre-test instruments, provide valuable insights for the full scale study (van Teijlingen & Hundley, 2002) and increase the efficiency of clinical trials (Wittes & Brittain, 1990). An environmental assessment will provide an opportunity to judge the feasibility of a site for the planned research. Investigators should show the lists to others familiar with the particular agency or organization for further clarification to make realistic plans for using the site in intervention research. This section includes a brief description of each component of an environmental assessment initiated by the investigator and the intervention team and recommendations based on the authors' experiences.

Assessment of Environment Stability

When planning research in highly unstable environments, the list of stable aspects of the research environment may be short and seem unnecessary. However identifying stable aspects of potentially chaotic environments is an important step in determining whether a site should be included in a research study, or is simply too unstable. For example, one might assume that an organization that has been in existence for 60 years will remain financially solvent and viable during the course of the study. An agency whose mission has been the care of people with Alzheimer's disease might be assumed to continue to treat this population of patients. Although assisted living facilities and nursing homes are unstable environments from many perspectives, they provide an opportunity to examine more easily the longer-term effects of an intervention because the length of stay for patients is often longer than for patients in other settings, and the use of the MDS in nursing homes provides a standard set of data collected periodically on every resident. Similarly, schools provide a natural environment for the study of children, despite the potential for families and teachers to move. Critical care units in which high quality evidence-based care is valued may be perfect sites for clinical research that hold promise for improved patient care.

Assessment of Environmental Instability

History of past instability is the best predictor of future instability. An agency with a 50% staff turnover rate over the previous 5 years is likely to have an equivalent turnover rate during the next 5 years. A unionized staff that has previously threatened to strike over contract disputes is likely to do so again when their next contract is up for negotiation. The investigator should make a list of those aspects of the environment one could assume will change and how each could affect the success of the study. Colleagues experienced with the research context (or an advisory board) might review the list and provide guidance regarding solutions to potential issues.

For most highly unstable environments, this part of the pre-assessment list is likely to be long and potentially daunting. To help focus on the most important sources of environmental instability, the research team should rank each potential change twice: once for their likelihood of occurring (e.g., very likely, somewhat likely, not at all likely) and once for their potential effect on the study (e.g., major effect, somewhat of an effect, minor effect). The research team might focus first on those potential changes that are highly likely and would have a major impact on the study, followed by those that are less likely but could also have a major effect. Then the study can be designed so that it can survive these high impact problems, or the investigator may consider selection of alternative research settings.

Assessment of “Fatal” Instabilities

The researcher needs to identify those aspects of the study environment that if changed, could stop the study from progressing. Examples include an agency going bankrupt, having its funding terminated, or no longer admitting patients with the phenomenon of interest. Another example is when an investigator previously granted access by a prior leader is denied continued access by the new management. These are problems that are outside the control of the investigator and require contingency plans such as having agreements from back-up sites and timelines that can accommodate site changes and prolonged data collection phases.

Further, researchers should be mindful of how they can design studies so that even if the students, patients, or staff change, they can still answer their research questions. For example, in a study focused on a partnership approach for sustaining best practices in nursing homes, the investigators acknowledged that the residents and the staff would change from pre-test to post-test, but the researchers were interested in their characteristics as a group rather than comparing individuals (Beck et al., 2005).

Stakeholder Assessments

Stakeholders include all individuals or groups who affect the success of the study. These might include state officials, local associations, unions, caregiver groups, administrators, staff, patients, and families. Investigators should think broadly about whose support could affect their ability to conduct the planned research. The list should include the benefits of the planned research from each stakeholder's perspective, knowing that each may have different needs and interests. For example, teachers might be motivated to support a parenting intervention if it helped parents be more invested in their children's education and improved parent-teacher relationships. School administrators might be interested if it helped show they are meeting agency expectations for parent involvement. Parents might be interested if the intervention helped them feel better about themselves as parents and more effective in disciplining their children. If the investigator is not sure what the perceived benefits might be to different stakeholders, ask each stakeholder group. The information can then be used to frame the benefits of the study so stakeholders will understand why it is important to support it, even when money, resources, and time are constrained by external forces.

As important as perceived benefits are for stakeholders, it is also important to identify the barriers to participation each might perceive. Subject burden (e.g., lengthy questionnaires, frequent assessments, invasive data collection protocols) is a common example of a barrier to participation when environments are highly unstable and resources are constrained.

Investigators may wish to convene an advisory board to help them assess the research environment and collaborate in the development of the study design and procedures. If so, the stakeholder list may be particularly useful in identifying informed, interested, and influential individuals to serve on this advisory board.

Non-Coercive Compensation/Incentives

It is wise to assume that in highly unstable environments, administrators, staff, patients, and families are prioritizing their efforts according to what they need to do to get through the day safely. Under these circumstances, research takes a back seat to survival and functional maintenance. Financial compensation for the time and effort devoted to the study, for both the participating agencies and individual participants, should be included when financially feasible. Incentives such as continuing education credit for staff participating in educational interventions, gift certificates, prizes, and donated equipment or supplies might also be appropriate. However, researchers are cautioned to work closely with their respective Institutional Review Boards (IRBs) to provide incentives that are not deemed coercive. Advisory boards may be particularly helpful to researchers in providing guidance on the types of incentives most likely to motivate and maintain involvement in research under unstable conditions.

In summary, researchers must be clear about how the proposed research is dependent on the stability of the environment. Questions that should be addressed prior to beginning a study include: What is the investigator expecting the site to do as part of the research? Can these expectations survive instability – what type? how much? The fewer expectations and less dependency the researcher has on the stability of the setting for activities such as recruitment and data collection, the less the instability will affect the study.

Conclusion

The challenges of intervention research can be amplified in highly unstable environments, yet we are increasingly recognizing the value of effectiveness research carried out in real world practice settings (Bellg et al., 2004; Fogg & Gross, 2000; Lagomasino, Dwight-Johnson & Simpson, 2005). Nurse researchers need to systematically identify potential threats to internal and external validity when conducting research in these settings and employ strategies to address those threats (Maas, Kelley, Park & Specht, 2002). The experiences of the authors, who have encountered a wide range of research challenges, has resulted in a recommendation that nurse researchers contemplating studies in highly unstable settings carry out an assessment of the environment and its resources prior to designing and initiating research, and then ask themselves what kind of studies can survive these changes. This assessment should be carried out in conjunction with the more traditional pilot work of intervention research (Van Teijlingen & Hundley, 2002; Wittes & Britain, 2006). Whenever possible, nurse researchers should employ rigorous designs using control groups, randomization, and concurrent data collection that can account for many threats to internal validity. Researchers are also advised to form partnerships and involve key stakeholders from the beginning of the research process, and to provide non-coercive, IRB approved incentives and compensation that cultivate win-win situations for researchers, participants, and agencies. With these considerations in mind, the challenges of conducting research in highly unstable, but clinically important, environments can be minimized. We invite readers to comment on our environmental assessment recommendations in light of their own research experiences in unstable environments.

Acknowledgments

The authors gratefully acknowledge support from the Center for Patient-Centered Interventions NINR # P20 NR008987, Sandra Ward, Principal Investigator, which sponsored a conference at the University of Wisconsin, Madison, which served as a stimulus for development of our paper.

References

  • Agency for Healthcare Research and Quality Strengthening the health care safety net. Workshop Brief, October 25–27, 2000. User Liaison Program. 2000. Retrieved May 18, 2008, from http://www.ahrq.gov/news/ulp/safety/ulpsfty.htm.
  • Aiken LH, Clarke SP, Cheung RB, Sloane DM, Silber JH. Educational levels of hospital nurses and surgical patient mortality. Journal of the American Medical Association. 2003;290(12):1617–23. [PMC free article] [PubMed]
  • Aiken LH, Clarke SP, Silber JH, Sloane D. Hospital nurse staffing, education and patient mortality. LDI Issue Brief. 2003;9(2):1–4. [PubMed]
  • American Academy of Pediatrics The role of the school nurse in providing school health services. Pediatrics. 2008;121(5):1052–1056. [PubMed]
  • Baker CM, Messmer PL, Gyurko CG, Domagala SE, Conly FM, Eads TS, et al. Hospital ownership, performance, and outcomes. Journal of Nursing Administration. 2000;30(5):227–240. [PubMed]
  • Banaszak-Holl J, Hines MA. Factors associated with nursing home staff turnover. The Gerontologist. 1996;36(4):512–517. [PubMed]
  • Bates-Jensen BM, Simmons SF, Schnelle JF, Alessi C. Evaluating the accuracy of minimum data set bed-mobility ratings against independent performance assessments: Systematic error and directions for improvement. The Gerontologist. 2005;45(6):731–738. [PubMed]
  • Beck C, Heacock P, Mercer S, Doan R, O'Sullivan P, Stevenson J, et al. Sustaining a best-care practice in a nursing home. Journal of Healthcare Quality. 2005;27(4):5–16. [PubMed]
  • Bellg AJ, Resnick B, Minicucci DS, Ogedegbe G, Ernst D, Borrelli B, et al. Enhancing treatment fidelity in health behavior change studies: Best practices and recommendations from the NIH Behavior Change Consortium. Health Psychology. 2004;23(5):443–451. [PubMed]
  • Borrelli L, Sepinwall D, Bellg AJ, Breger R, DeFrancesco C, Sharp EL, et al. A new tool to assess treatment fidelity and evaluation of treatment fidelity across 10 years of health behavior research. Journal of Consulting and Clinical Psychology. 2005;73(5):852–860. [PubMed]
  • Bowers BJ, Esmond S, Jacobson N. Turnover reinterpreted: CNAs talk about why they leave. Journal of Gerontological Nursing. 2003;29(3):36–43. [PubMed]
  • Bowers BJ, Nolet K. Promising practices in care planning for nursing home patients. Report to Agency for Health Research and Quality, U.S. Department of Health and Social Services; Baltimore, MD: 2006.
  • Burns N, Grove SK. The practice of nursing research: Conduct, critique and utilization. 5th ed. Elsevier; St. Louis, MO: 2005.
  • Castle NG. Turnover begets turnover. The Gerontologist. 2005;45(2):186–195. [PubMed]
  • Castle NG, Engberg J. Organizational characteristics associated with staff turnover in nursing homes. The Gerontologist. 2006;46(1):62–73. [PubMed]
  • Centers for Medicare and Medicaid Services Office of Clinical Standards and Quality, Quality Improvement Organization (QIO) Program's 9th Scope of Work Social Security Act, TITLE XVIII. 2008. Retrieved July 29, 2008 from http://www.cms.hhs.gov/QualityIMprovementOrgs/04_9thsow.asp#TopOfPage http://www.cms.hhs.gov/QualityIMprovementOrgs/04_9thsow.asp - TopOfPage.
  • Devereaux PJ, Choi PTL, Lacchetti C, Weaver B, Schunemann HJ, Haines T, et al. A systematic review and meta-analysis of studies comparing mortality rates of private for-profit and private not-for-profit hospitals. Canadian Medical Association Journal. 2002;166(11):1399–1406. [PMC free article] [PubMed]
  • Eaton SC. Beyond “unloving care:” linking human resource management and patient care quality in nursing homes. International Journal of Human Resource Management. 2000;11(3):591–616.
  • Ferlie E. Large-scale organizational and managerial change in health care: A review of the literature. Journal of Health Services Research & Policy. 1997;2(3):180–189. [PubMed]
  • Fogg L, Gross D. Threats to validity in randomized clinical trials. Research in Nursing & Health. 2000;23(1):79–87. [PubMed]
  • Funk M, Parkosewich JA, Johnson CR, Stukshis I. The effects of dedicated monitor watchers on patient outcomes. American Journal of Critical Care. 1997;6(4):318–323. [PubMed]
  • Ghobadian A, O'Regan N. The impact of ownership on small firm behavior and performance. International Small Business Journal. 2006;24(6):555–586.
  • Gifford R, Zammuto F, Goodman E, Hill K. The relationship between hospital unit culture and nurses' quality of work life. Journal of Healthcare Management. 2002;47(1):13–25. [PubMed]
  • Grey M, Berry D, Davidson M, Galasso P, Gustafson E, Melkus G. Preliminary testing of a program to prevent type 2 diabetes among high risk youth. Journal of School Health. 2004;74(1):10–15. [PubMed]
  • Harahan M, Kiefer K, Burns-Johnson A, Guiliano J, Bowers B, Stone R. Addressing shortages in the direct care workforce. Institute for the Future of Aging Services; Washington, DC: 2003.
  • Harrington C, Woolhandler S, Mullan J, Carrillo H, Himmelstein DU. Does investor-ownership of nursing homes compromise the quality of care? International Journal of Health Services. 2002;32(2):315–325. [PubMed]
  • Holmes J. The effects of ownership and ownership change on nursing home industry costs. Health Services Research. 1996;31(3):327–346. [PMC free article] [PubMed]
  • Horn SD, Buerhaus P, Bergstrom N, Smout RJ. RN staffing time and outcomes of long-stay nursing home residents: Pressure ulcers and other adverse outcomes are less likely as RNs spend more time on direct patient care. American Journal of Nursing. 2005;105(11):58–70. [PubMed]
  • Jones KR, Fink R, Vojir C, Pepper G, Huff E, Clark L, et al. Translation research in long-term care: Improving pain management in nursing homes. Worldviews on Evidence-Based Nursing. 2004;Suppl 1:S 13–20. [PubMed]
  • Lagomasino IG, Dwight-Johnson M, Simpson GM. Psychopharmacology: The need for effectiveness trials to inform evidence-based psychiatric practice. Psychiatric Services. 2005;56(6):649–651. [PubMed]
  • Lynch DC, Whitley TW, Willis SE. A rationale for using synthetic designs in medical education research. Advances in Health Sciences Education. 2000;5(2):93–103. [PubMed]
  • Maas ML, Kelley LS, Park M, Specht JP. Issues in conducting research in nursing homes. Western Journal of Nursing Research. 2002;24(4):373–389. [PubMed]
  • McGregor MJ, Tate R, McGrain K, Ronald L, Broemeling A, Cohen M. Care outcomes in long term care facilities in British Columbia, Canada: Does ownership matter? Medical Care. 2006;44(10):929–935. [PubMed]
  • Mentes JC, Tripp-Reimer T. Barriers and facilitators in nursing home intervention research. Western Journal of Nursing Research. 2002;24(8):918–936. [PubMed]
  • Mezey M, Kobayashi M, Grossman S, Firpo A, Fulmer T, Mitty E. Nurses Improving Care to Health System Elders (NICHE): Implementation of best practice models. Journal of Nursing Administration. 2004;34(10):451–457. [PubMed]
  • National Association of School Nurses Position statement caseload assignments. 2006. Retrieved May 22, 2008, from http://www.nasn.org/Default.aspx?tabid=209.
  • New York State Public Health Law #2801 . Establishment of Incorporation of Hospitals. 2008.
  • O'Neill C, Harrington C, Kitchener M, Saliba D. Quality of care in nursing homes: An analysis of the relationships among profit, quality and ownership. Medical Care. 2003;41(12):1318–1330. [PubMed]
  • Rastegar DA. Health care becomes an industry. Annals of Family Medicine. 2004;2(1):79–83. [PubMed]
  • Schwartz C, Ozmiinkowski R, Hoaglin D, Cella M, Branch L. Can a survey influence quality of care in nursing homes? Journal of Aging and Health. 1994;6(4):549–572. [PubMed]
  • Shadish WR, Cook TD, Campbell DT. Experimental and quasi-experimental designs for generalized causal inference. Houghton Mifflin; Boston: 2002.
  • Spillane V, Byrne MC, Byrne M, Leathem CS, O'Malley M, Cupples ME. Monitoring treatment fidelity in a randomized controlled trial of a complex intervention. Journal of Advanced Nursing. 2007;60(3):343–352. [PubMed]
  • Tu HT, Reschovsky JD. Assessments of medical care by enrollees in for-profit and nonprofit health maintenance organizations. New England Journal of Medicine. 2002;346(17):1288–1293. [PubMed]
  • U.S. Department of Education No child left behind act of 2001. 2002. Retrieved May 18, 2008, from http://www.ed.gov/nclb/landing.jhtml.
  • Van Teijlingen E, Hundley V. The importance of pilot studies. Nursing Standard. 2002;16(40):33–36. [PubMed]
  • Wittes J, Brittain E. The role of internal pilot studies in increasing the efficiency of clinical trials. Statistics in Medicine. 1990;9(1–2):65–72. [PubMed]
  • Yeatts DE, Seward RR. Reducing turnover and improving health care in nursing homes: The potential effects of self-managed work teams. The Gerontologist. 2000;40(3):358–63. [PubMed]