PMCCPMCCPMCC

Search tips
Search criteria 

Advanced

 
Logo of jncimonoLink to Publisher's site
 
J Natl Cancer Inst Monogr. 2012 May; 2012(44): 127–133.
PMCID: PMC3482966

Multilevel Intervention Research: Lessons Learned and Pathways Forward

Abstract

This summary reflects on this monograph regarding multilevel intervention (MLI) research to 1) assess its added value; 2) discuss what has been learned to date about its challenges in cancer care delivery; and 3) identify specific ways to improve its scientific soundness, feasibility, policy relevance, and research agenda. The 12 submitted chapters, and discussion of them at the March 2011 multilevel meeting, were reviewed and discussed among the authors to elicit key findings and results addressing the questions raised at the outset of this effort. MLI research is underrepresented as an explicit focus in the cancer literature but may improve implementation of studies of cancer care delivery if they assess contextual, organizational, and environmental factors important to understanding behavioral and/or system-level interventions. The field lacks a single unifying theory, although several psychological or biological theories are useful, and an ecological model helps conceptualize and communicate interventions. MLI research designs are often complex, involving nonlinear and nonhierarchical relationships that may not be optimally studied in randomized designs. Simulation modeling and pilot studies may be necessary to evaluate MLI interventions. Measurement and evaluation of team and organizational interventions are especially needed in cancer care, as are attention to the context of health-care reform, eHealth technology, and genomics-based medicine. Future progress in MLI research requires greater attention to developing and supporting relevant metrics of level effects and interactions and evaluating MLI interventions. MLI research holds an unrealized promise for understanding how to improve cancer care delivery.

Clinical science is making significant strides and changing how clinicians care for patients along the entire cancer care continuum. These scientific advances are applied within an increasingly complex social, organizational, and environmental context. Current approaches to intervention research may be insufficient to address this complexity. It is essential that we rethink the manner and mode of interventions along the care continuum and their implications for affecting the quality, cost, and outcomes of the care being provided.

The previous 12 chapters in this monograph have attempted to fill this need by illuminating and evaluating the role of multilevel intervention (MLI) research in cancer care delivery. These chapters were commissioned for this monograph either in preparation for or in response to the National Cancer Institute (NCI)–organized conference, Multilevel Interventions in Health Care: Building Foundations for Future Research, held March 4–5, 2011 in Las Vegas, Nevada. Guided by a multidisciplinary scientific program committee drawn from government, academia, and health-care delivery organizations, and attended by more than 160 researchers and clinicians, the conference aimed to 1) describe the state of the science in MLIs, 2) clarify the issues in the conceptualization of multilevel effects, and 3) identify areas for building the foundation of MLI research (eg, taxonomy, measurement, intervention design). By addressing these goals, participants hoped to identify promising areas of application throughout the cancer control continuum.

In this concluding chapter, we draw from the reviews and findings of the preceding MLI research chapters to 1) assess the added value of MLI research; 2) reflect on what has been learned to date about the success and challenges of MLI research in cancer care delivery; and 3) identify specific ways to improve the scientific soundness, feasibility, and policy relevance of MLI research in cancer control and its contribution to the NCI research agenda (Box 1).

Box 1.

Our working definition of multilevel interventions

As noted in the introductory chapter, an intervention is a specified strategy or set of strategies designed to change the knowledge, perceptions, skills, and/or behavior of individuals, groups, or organizations, with the goal of improving health outcomes. The term “multilevel intervention” refers to an intervention targeted to influence more than one contextual level (individual, group, organization, and community). Our primary interest is in MLI research that influences at least individual, group, organizational, and societal contexts in the United States that influence health-care delivery. The purpose of multilevel interventions is to affect the critical contextual issues and create a more efficient, effective, and coordinated cancer care delivery system that achieves relevant patient outcomes, including improved survival, health-related quality of life, and patient experience with care, at a reduced cost to all involved.

The Added Value of MLI Research

One theme from the preceding chapters is that MLI research adds richness to intervention science in cancer care delivery. As Stange et al. and Charns et al. note, cancer has a significant intervention research tradition, but it is almost exclusively focused on single-level intervention studies, primarily targeted to patient and/or physician behavior (1,2). Charns et al. (2) and Yano et al. (3) argue that MLI can consider other factors and levels that, if left unaddressed, serve as potential unmeasured barriers or facilitators to health-care improvements. By emphasizing the importance of context, the Zapka et al. (4) and Yano et al. (3) articles illustrate another way in which MLI research deepens our understanding of intervention science. Most cancer care intervention studies lack details about the multilevel context in which care is provided, making it difficult to understand how or why the intervention succeeds or fails or was not adopted (1,2). Environmental or policy factors, whether at the organizational, community, state, or national level, are often important to specify as mediators and moderators of behavioral or system-level interventions. Even when they are not modifiable, these factors frame the intervention problem at the appropriate level of complexity necessary to understand how the impact, direction, and sustainability of intervention effects are shaped. Although MLIs are often more complex than single-level interventions, they actually may be less costly to health-care systems in the long run if they facilitate successful and sustainable improvements in the quality of care.

Reviews by Stange et al. (1) and Sheinfeld Gorin et al. (5) illustrate that a few good MLI research examples exist, especially in community-based intervention research like tobacco control, and these examples demonstrate the potential power of MLI research to facilitate positive and sustainable change. However, these studies are the exception rather than the rule in cancer intervention research. In particular, almost all authors recognized that few MLI research studies are done in actual care delivery settings and that organizational- and policy-level interventions are systematically underrepresented in this literature, though they offer great potential. Studies that intervene and measure effects on three or more levels of the ecological model are virtually nonexistent (1,2). The challenge remains to understand how to take advantage of MLI research’s scientific potential to inform cancer care delivery across the entire continuum.

Lessons Learned

Conceptual and Theoretical Issues

In the introduction to this monograph, we asked how theories and definitions of context could be brought to bear on an MLI framework (6). At least three frameworks for defining levels are used in the monograph, including an ecological model (6), a psychological model (7), and, as presented in the commentary by Scott, the concept of an organizational field (8). These frameworks provide a useful heuristic either explicitly or implicitly and are consistently cited as useful frames of reference for conceptualizing MLIs. Yet, no single unified theory explains how the community context affects the behaviors of individuals seeking health care, the health professionals providing care, and/or organizations providing health care services, and no single conceptual framework predominates (4,69).

The authors also note important gaps in the application of theory to MLI studies. The cancer care intervention literature is most developed at the individual level (ie, patients, consumers or specific individual health-care providers) (1,2). This level is guided by strong behavioral and psychological theories of individual behaviors. Two areas found to be underdeveloped in cancer intervention research that are replete in the general health-care literature are “teams” and “organizations” (913). Team-based care has been associated with mortality reductions in hospital settings (12), improvements in geriatric inpatient care (13), and as a critical element in successful collaborations to improve the quality of depression, hypertension, and diabetic care in ambulatory settings (14). Success with team approaches to care has become so well documented that the National Quality Forum has issued a recommendation to “establish a proactive systematic, organization-wide approach to developing team-based care through team-work training” (15), and expand the Medical Home Model as a “team-based approach to delivering care led by a personal physician” (16).

Organizations are a third level within our framework that needs further consideration in cancer intervention research. In his commentary, Scott (8) recognizes the centrality of organizations in modern societal systems and suggests that MLI occurs within an organization field. A field is heuristically composed of actors (patients, families, providers, and organizations), institution logics (norms, values, culture), and prevailing governance structures that exercise control and formalize relationships (8,9). Organizations are central and could include a single office practice with multiple provider teams or a set of practices that each includes a set of teams. The characteristics of the most common health-care organizations are that the activities of the members are organized around achieving a specific set of goals, consistent with prevailing norms and values, and that the structure is highly formalized. In health care, formalization includes a financial and legal structure that manages and controls reimbursement and the dispersion of resources. By this definition, health-care organizations as part of an organizational field could include a single practitioner's office or clinics within a managed care system. These organizations must facilitate the activities of teams within them that actually deliver care. Organizations provide some of the inputs that affect the processes of teams and their outputs, so understanding and evaluating how organizations facilitate teamwork is critical to improving cancer care.

Although several monograph authors noted the importance of health-care teams and organizations in MLI studies, none of them developed the connection between these constructs and improved quality of care. Much more of what is known about how health-care teams and organizations function is in the management, social psychology, and sociology literature rather than the medical literature. This separation of knowledge is a critical limitation to improvements in the quality of health care. Donabedian (17) noted long ago that quality was determined by both technical and interpersonal aspects, where the latter were both contextual issues and issues of interactions between individuals. Recently, these issues have been the focus of a separate supplement on the interfaces of primary and subspecialty oncology care (18). A critical area for future research is to bridge the lessons from management and the technical aspects of care to solve some of the problems that we face in delivering high-quality care. For example, Donald Berwick argues that improving the US health-care system requires simultaneous pursuit of three aims: improving the experience of care, improving the health of populations, and reducing per capita costs of health care (19). Preconditions for this include interventions on multiple levels, including the patient and family, organizational redesign of physician practices to align financial management with population health management, and macro-system integration so payment incentives reward organizations that improve cost and quality of care. The goal of improving the health of populations is influenced by many forces well beyond the clinical encounter such as public health efforts related to health behaviors and control of environmental exposure. However, examining cancer care organizations in the context of policy and practice is essential as well. Nowhere is this need more clear than with multimodality cancer care where new practice systems, training, and reimbursement incentives are needed to help cancer specialists work more effectively across the disciplines of surgery, radiation therapy, and chemotherapy to optimize care delivery for cancer patients (20). Viewing the problem as a whole rather than reducing it to its individual parts conceptualizes care delivery as a complex adaptive system. The opening article to this supplement argues this approach, and some think it is not possible to address quality issues without taking this holistic approach (6,21).

A source of conceptual confusion at the conference was how to determine whether a study is multilevel or single level when it targets physicians and patients. Traditional organizational and psychological theory suggests that individuals are one level of analysis that could include patients and/or the individual care provider. The care provider interacts with people seeking care, and Engel acknowledged this in his ecological model by recognizing the “physician–patient” dyad as a level of care above the individual (22). People seeking care along the cancer care continuum interact with individuals, groups, and organizations in the course of their care. Care providers interact with each other, with patients, with groups and organizations when delivering care. The ecological and organizational framework helps conceptualize the levels and relevant components, and the perspective that interpersonal interaction is one mechanism of effect between individuals and the group and between individuals and the organization. Thus, the number of levels in a study depends on the number of contextual levels targeted (ie, what social units are being targeted—individuals, teams, organizations, and/or communities).

A holistic approach can be intimidating. Stepping back and realizing that everything is connected in an interactive web of communication and relationships may lend itself to paralysis rather than intervention building. No single theory explains everything, and no single disciplinary approach is sufficient. At some point, the complex web must be broken into manageable parts, and investigators need to assemble a group that can design an intervention. Breaking the system into individual, team, and organizational levels may be a place to begin. Then, build approaches that account for time, consider organizational and team effects, conceptualize the mechanism of the intervention, and conceive of the appropriate measures.

Methodological Issues of Design and Measurement

Because of our interest in the causal relationships between the timing of MLIs and health outcomes, we consider MLI research designs emphasizing longitudinal data collection and interventions inherently superior to cross-sectional approaches (23). Randomized designs have long been the “gold standard” in cancer intervention research, and cluster randomized designs are often effective when measuring and intervening at multiple levels, especially when the effects of organizations, groups, and individuals need to be separately estimated (24). However, Cleary et al. note that in many MLI research applications, randomized approaches are often infeasible (eg, randomizing interventions among physicians within a group practice) or inferior (eg, where generalizability or scalability of MLIs are the priority research question) (25). Multimethod designs (eg, using qualitative and quantitative scientific approaches), alternative evaluation models, and sophisticated statistical designs show promise, especially in MLI research related to implementation science and quality improvement research (35).

Many studies suffer from statistical limitations in estimating effects across levels. However, existing statistical design principles and approaches enable examination of how multilevel effects interact on outcomes simultaneously (eg, path analysis, hierarchical linear [or nonlinear] modeling, or other structural equation estimation methods and modeling) (25). Still, collecting the breadth of data necessary to support these complex models requires a major investment in data, often without theoretical confidence that the data will support the complexities involved in these models or produce expected results. In this case, Morrissey et al. make a good case that simulation models may be a method to pretest intervention models (although sufficient data to assess critical assumptions and model sensitivity are still needed) or to examine the anticipated scale effects of the MLI (26).

In the near term, Charns et al. note that the real challenge to MLI research may be the limited experience in deploying measures across levels, especially when applied to group-, organizational-, and community-level measurement, in cancer care delivery research (2). All statistical models presume the accurate and complete measurement of the multiple contexts that are hypothesized to influence cancer outcomes in various ways. Investigators need to draw from other fields to begin testing and modifying these measures to fit the circumstances of cancer care delivery. The consequence of these limitations is that in the short term, it may be necessary to improve multilevel measurements of effects, rather than conduct MLIs.

The chapter by Charns et al. (2) identified several measurement issues related to effective execution of MLI studies. These issues include the need for congruency between theory, construct, and measures; the lack of independence of some measures; potentially nonhierarchical and nonlinear relationships between variables; the need to confirm that intended effects have been/are achieved over time and that unintended effects and barriers to implementation have been/are captured over time; and the need to capture nontraditional aspects of interventions (eg, practicality, feasibility, cost efficiency). Few MLI research applications in cancer have used sophisticated approaches to measure group- or organizational-level constructs, such as leadership and team cohesion, although these measures are well developed and used extensively in the management, social psychology, and general health literature (2). As discussed earlier, modern cancer care is increasingly a team activity. Patients and families work together to assure that informal supports are adequate to cope with and manage their cancer (27). Treatments are becoming multimodal, which involves multiple specialists (and increasingly, primary care physicians) engaging in the execution of complex sequencing of therapies that, for many tumors, last months if not years (28). Measurement is not yet able to fundamentally capture and quantify the reality of patient/family “teams” interacting with clinical teams, and this is an area of real need in cancer care intervention research. Measures of policy influences at multiple levels (organizational, state, and national) also are lacking. In particular, they lack the sensitivity to the characteristics of policy (eg, its restrictiveness or leniency) that serve as barriers or facilitators of organizational, clinical, or patient self-management approaches that lead to improved health outcomes (2). With the ability to assess individual, group, organization, and community effects, MLI research is uniquely suited to deploy such measures to support these studies.

A final methodological issue is the argument that the complexity, and thus cost of MLI research is beyond the capacity (ie, design, recruitment, organizational, analytical, and funding capacity) of the current grant funding approaches of most research organizations. Cleary et al. (25), Alexander et al. (23), and Morrissey et al. (26) note that many design options are suited to a variety of intervention research issues. Simulation modeling, mixed-method designs, sophisticated experimental designs that test multiple interventions and their interactions rather than a single intervention at a time, and interdisciplinary action research designs, which engage key constituents in rapid learning cycles, can provide pragmatic and practical approaches to MLI research (29). The challenges of selecting theory and evidence-based interventions may be addressed by using any of a variety of conceptual frameworks (eg, Put Prevention into Practice; Agency for Healthcare Research and Quality checklists; Strengths, Weaknesses, Opportunities, Threats; force field analysis; Chronic Care Model; and Reach, Efficacy/Effectiveness, Adoption, Implementation and Maintenance) (3037). The use of Internet-based computer adaptive technology platforms also can reduce the cost of standardized data collection considerably (38). Many studies that target multiple measures at multiple levels in multiple organizations will be expensive. MLI research may be an area especially suited to conducting smaller pilot projects that test various aspects of the broader model before funders are approached with large-scale multisite intervention studies. Weiner et al. (7) describe methods for more efficient selection and sequencing of interventions in MLI studies.

Building Capacity, Systems, and MLI Research Infrastructure

Building capacity to move the field toward adopting, implementing, evaluating, and sustaining MLI research must be deliberate and comprehensive and may require some specialized infrastructure and skills. This will require disciplines to work synergistically and across traditional boundaries to examine the relevance of team, systems, transdisciplinary science, and participatory research. Consideration needs to be given to identifying crucial stakeholders from the managerial, clinical, and policy communities to build capacity and develop strategies for shifting organizational culture, norms, and values to adopt and sustain MLIs.

Identifying Key Stakeholders

Identifying key stakeholders may seem simple or obvious. However, in research and with the diffusion of a novel and complex MLI idea, it is important to think carefully about the multiplicity of stakeholders who must engage in all phases of the research process. This is especially important in MLI research given the frequent need to get buy in from organizational leaders and policy makers. As Yano et al. (3) noted, if we neglect to invite appropriate stakeholders to planning, implementation, and evaluation processes associated with MLIs, then we risk reinventing the wheel, wasting resources, or missing an opportunity to have expertise that will facilitate the success of MLIs.

Working Synergistically to Build MLI Capacity

Although challenges exist both conceptually and methodologically, there is precedence for addressing the cancer research enterprise in the context of MLI research. We have learned many lessons from funded research platforms about interventions, their context, and the pros and cons of models of collaborative science.

For example, NCI funds multiple networks and systems to support intervention and epidemiological studies, some of which include MLI approaches relevant to understanding cancer care. These platforms include the Cancer Intervention and Surveillance Modeling Network, Cancer Research Network (CRN), NCI Comprehensive/Designated Cancer Centers, NCI Community Cancer Centers Program (NCCCP), Community Clinical Oncology Program, Patient Navigation Research Program, Cancer Consortium for Outcomes Research and Surveillance, the Breast Cancer Surveillance Consortium, and Centers of Excellence for Cancer Communication Research Studies (3946). These research platforms include multiple partners who take into consideration how team, systems, transdisciplinary science, and participatory research interact and support the goals and objectives of their research. Yet, NCI-sponsored practice-based networks like the CRN and the NCCCP mostly support intervention studies at one or two levels. With the exception of the Veterans Health Administration, few delivery-based studies include three levels of intervention. Knowledge from multiple fields will be needed to respond effectively to the new demands and expectations of funders on research networks engaged in MLI research.

Furthermore, health-care organizations are facing increasing costs, dwindling resources, and greater accountability from the federal to local levels. Funder demands on the various delivery systems through the grant or contract process may be incompatible with developing research designs given existing operations or prevailing norms. Therefore, as Yano et al. (3) note, engaging stakeholders in the process at the onset is critical to implementation and sustainability. The delivery organizations may have to identify new resources at the onset to facilitate change toward MLI approaches. Although the research is critical, the practical components of data collection in clinical settings are as critical if the goal is to sustain change to support successful MLIs.

For example, we have learned from the field of tobacco control that MLIs (comprehensive tobacco control) can reduce the prevalence rate of tobacco use and secondhand smoke exposure (47). The challenge with this evidence-based MLI approach is that state resources have diminished and funds set aside for comprehensive control are now being used to fill state budget gaps. Comprehensive tobacco control is no longer being implemented at the recommended levels and declines in smoking rates in the United States have stalled (48,49). Therefore, as we think about how to move forward in building MLI capacity for the continuum of cancer care delivery, it is important to consider strategies for sustaining and optimizing research infrastructures and strategies for carrying out and sustaining the practice of MLIs.

Funders and health/community systems also need to work synergistically and equitably with providers, patients, and consumers of MLIs. Patients and consumers may include those who have not benefited from single-level interventions, have been slow to benefit, or who fall through the cracks within existing delivery systems. Working with providers, consumers, and patients requires participatory processes that engage them in research conceptualization, design, implementation, and evaluation. Furthermore, communities and health system organizations will need to develop communication and policy strategies to ensure patient access, use, and benefits. This requires a clear understanding of the patients’ access to insurance, interpersonal factors, interpersonal relationships, social support systems, and their history, culture, geography, and perceptions about different interventions. Researchers need to consider theories, measures, and methods that fit the populations targeted for MLI.

Shifting Organizational Culture, Norms, and Values for Adoption and Sustainability

Changing organizational cultures, norms, and values involves multiple processes and channels by which consumers and members of social systems affected by MLIs are informed, engaged in the idea that these interventions have the potential to enhance cancer care for the system and its populations, and involved in their implementation. The acknowledgement of the process by which “innovation” is diffused and disseminated is critical (50), particularly in an environment where single-level approaches are standard, resources are scarce, risk aversion is common practice, and stakeholders have competing ideas on how to best provide cancer care across the continuum.

The NCI consulted with leading members of the extramural research community to develop a plan for “diffusing” consideration of MLI science among multiple stakeholders, including potential funders and researchers. This plan included developing this monograph, hosting the meeting in March 2011, and engaging new and existing NCI/NIH leadership in discussions to increase knowledge on the value and benefits of MLI approaches. As with any initiative of this size and scope, the process of persuading decision makers within the NCI and the larger research community of the need to adopt this idea was met with mixed reactions. Clearly, the NCI acknowledges that additional stakeholders, including policy makers, health-care systems, insurers, patients, funders, journals, providers, and many others, are needed to influence thinking around the utility of MLIs. It is important to carefully consider the existing diversity of social norms around cancer care in interacting systems, the perceived costs/benefits of MLIs, organizational culture and decision-making processes, and particular obstacles and facilitators within systems that affect the successful adoption and sustainability of MLIs.

For example, changing social norms regarding how best to facilitate cancer care may require system-wide policy changes among funders and individuals within the larger clinical, research, and care delivery community. This may include training for peer reviewers of grants; development of new study sections; and training and educational efforts among researchers, journal editors, providers, and staff to familiarize stakeholders with MLI concepts, methodologies, and design. In addition, the implementers must consider short- and long-term costs/benefits because MLIs are challenging, could initially be disruptive to systems, and gains to the patient may take years.

An MLI Research Agenda

An important cross-cutting theme in this supplement is that MLI research should be conceptualized, planned, and executed carefully and will require enhanced infrastructure, training, and innovative partnerships throughout the policy, clinical, and research communities to effectively build a sustainable approach to its widespread application. The issues and opportunities identified in this supplement for pursing these conceptual, methodological, and research infrastructure challenges are summarized in Table 1. It is clear that this transition will take time. Yet, major policy and research changes are emerging in cancer care delivery that may serve as leverage points to facilitate this transformation. The development of genomic-based, individual targeted therapies is a potential “game changer” in cancer care delivery because it fundamentally changes early detection, diagnosis, and therapeutic processes associated with cancer (51). Khoury et al. (52) argue that genomic medicine is an excellent area for implementing MLI research, and point out how little translational research has been conducted in this growing area of medicine. They illustrate by describing how specific genomic applications related to conditions such as Lynch Syndrome are especially affected by influences at the state health policy, organizational, provider team, and patient and family levels. Similarly, Flood et al. (53) illustrate that MLI research applications are especially promising for health reform, especially in terms of assessing optimal approaches to delivering cancer care in new delivery mechanisms, such as affordable care organizations. Other opportunities for MLI research are present in eHealth interventions, which have rapidly proliferated and can materialize simultaneously at the patient, organizational, and community levels (54). We live in an age of rapidly growing e-technology and policy change at the national, state, and local levels. Reaching the tipping point can occur rapidly when appropriate channels of communications are used to market and diffuse the idea of MLIs in an effort to reach a critical mass of change agents, decision makers, and patients.

Table 1
MLI cancer research: issues and opportunities*

Conclusions

The next generation of research must address three persistent challenges to progress in MLIs addressing health-care delivery: 1) explaining interactions between the levels theoretically and practically, 2) measuring the interactions and contextual effects in quantifiable ways that build theory and contribute to stronger interventions over time, and 3) developing the research infrastructure and training opportunities for MLI investigators. This is a long-term agenda. In the short run, it will be important to define levels, or organizational frames preferably as units of human aggregation, develop measures, and show more evidence regarding the relative effects of MLIs on health-care delivery to individuals and the larger population being served. Consideration needs to be given to the context of health-care reform, eHealth technology, and genomics-based medicine that hold extraordinary opportunities to improve cancer care outcomes. We also need to consider how this context interacts with research initiatives to support the implementation and sustainability of MLIs.

Funding

This project has been funded in whole or in part with federal funds from the National Cancer Institute, National Institutes of Health, under Contract No. HHSN261200800001E. The content of this publication does not necessarily reflect the views or policies of the Department of Health and Human Services, nor does mention of trade names, commercial products, or organizations imply endorsement by the US government.

References

1. Stange KC, Breslau ES, Dietrich AJ, Glasgow RE. State-of-the-art and future directions in multilevel interventions across the cancer control continuum. J Natl Cancer Inst Monogr. 2012;44:20–31. [PMC free article] [PubMed]
2. Charns MP, Foster MK, Alligood EC, et al. Multilevel interventions: measurement and measures. J Natl Cancer Inst Monogr. 2012;44:67–77. [PMC free article] [PubMed]
3. Yano EM, Green LW, Glanz K, et al. Implementation and spread of interventions into the multilevel context of routine practice and policy: implications for the cancer care continuum. J Natl Cancer Inst Monogr. 2012;44:86–99. [PMC free article] [PubMed]
4. Zapka J, Taplin SH, Ganz P, Grunfeld E, Sterba K. Multilevel factors affecting quality: examples from the cancer care continuum. J Natl Cancer Inst Monogr. 2012;44:11–19. [PMC free article] [PubMed]
5. Sheinfeld Gorin S, Badr H, Krebs P, Prabhu Das I. Multilevel interventions and racial/ethnic health disparities. J Natl Cancer Inst Monogr. 2012;44:100–111. [PMC free article] [PubMed]
6. Taplin SH, Anhang Price R, Edwards HM, et al. Introduction: understanding and influencing mulitlevel factors across the cancer care continuum. J Natl Cancer Inst Monogr. 2012;44:2–10. [PMC free article] [PubMed]
7. Weiner BJ, Lewis MA, Clauser SB, Stitzenberg KB. In search of synergy: strategies for combining interventions at multiple levels. J Natl Cancer Inst Monogr. 2012;44:34–41. [PMC free article] [PubMed]
8. Scott WR. Embedding the examination of multilevel factors in an organization field context. J Natl Cancer Inst Monogr. 2012;44:32–33. [PMC free article] [PubMed]
9. Scott WR. Institutions and Organizations. Thousand Oaks, CA: Sage Publications; 2001.
10. West MA, Borrill CS, Dawson JF, Brodbeck F, Shapiro DA, Haward B. Leadership clarity and team innovation in health care. Leadersh Q. 2003;14(4–5):393–410.
11. Salas E, Diaz Granados D, Weaver SJ, King H. Does team training work? Principles for health care. Acad Emerg Med. 2008;15(11):1002–1009. [PubMed]
12. West MA, Guthrie JP, Dawson JF, Borrill CS, Carter M. Reducing patient mortality in hospitals: the role of human resource management. J Organ Behav. 2006;27(7):983–1002.
13. Scott J, Gade G, Venohr I, McKenzie M. Cooperative health care clinics: a group approach to individual care. Geriatrics. 1998;53(5):68. [PubMed]
14. Lemieux-Charles L, McGuire WL. What do we know about health care team effectiveness? A review of the literature. Med Care Res Rev. 2006;63(3):263–300. [PubMed]
15. National Quality Forum. Nursing Performance Measurement and Reporting: A Status Report. Washington, DC: National Quality Forum; 2007.
16. Rosenthal TC. The medical home: growing evidence to support a new approach to primary care. J Am Board Fam Med. 2008;21(5):427–440. [PubMed]
17. Donabedian A. Evaluating the quality of medical care. Milbank Q. 2005;83(4):691–729. [PubMed]
18. Taplin SH, Rodgers AB. Toward improving the quality of cancer care: addressing the interfaces of primary and oncology-related subspecialty care. J Natl Cancer Inst Monogr. 2010;40:3–10. [PMC free article] [PubMed]
19. Berwick DM, Nolan TW, Whittington J. The triple aim: care, health, and cost. Health Aff (Milwood). 2008;27(3):759–769. [PubMed]
20. Bach PB, Mirkin JN, Luke JJ. Episode-based payment for cancer care: a proposed pilot for Medicare. Health Aff (Milwood). 2011;30(3):500–509. [PubMed]
21. Pronovost PJ, Goeschel CA. Time to take health delivery research seriously. JAMA. 2011;306(3):310–311. [PubMed]
22. Engel GL. The need for a new medical model: a challenge for biomedicine. Fam Syst Med. 1992;10(3):317–331.
23. Alexander J, Prabhu Das I, Johnson TP. Time issues in multilevel interventions for cancer treatment and prevention. J Natl Cancer Inst Monogr. 2012;44:42–48. [PMC free article] [PubMed]
24. Donner A, Klar N. Design and Analysis of Cluster Randomization Trials in Health Research. London, UK: Arnold; 2000.
25. Cleary PD, Gross CP, Zaslavsky AM, Taplin SH. Multilevel interventions: study design and analysis issues. J Natl Cancer Inst Monogr. 2012;44:49–55. [PMC free article] [PubMed]
26. Morrissey JP, Hassmiller Lich K, Anhang Price R, Mandelblatt J. Computational modeling and multilevel cancer control interventions. J Natl Cancer Inst Monogr. 2012;44:56–66. [PMC free article] [PubMed]
27. Adler NE, Page AEK, editors. Institute of Medicine. Cancer Care for the Whole Patient: Meeting Psychosocial Health Needs. Washington, DC: The National Academies Press; 2008. [PubMed]
28. Fennell ML, Prabhu Das I, Clauser SB, Petrelli N, Salner A. The organization of multidisciplinary care teams: modeling internal and external influences on cancer care quality. J Natl Cancer Inst Monogr. 2010;40:72–80. [PMC free article] [PubMed]
29. Eisenberg EM, Baglia J, Pynes JE. Transforming emergency medicine through narrative: qualitative action research at a community hospital. Health Commun. 2006;19(3):197–208. [PubMed]
30. Prevention and chronic care resources. Agency for Healthcare Research and Quality Prevention and Care Management Web site. http://www.ahrq.gov/clinic/ppipix.htm. Accessed September 29, 2011.
31. Ko H, Turner T, Finnigan M. Systematic review of safety checklists for use by medical care teams in acute hospital settings—limited evidence of effectiveness. BMC Health Serv Res. 2011;11(1):211–211. [PMC free article] [PubMed]
32. Hill T, Westbrook R. SWOT analysis: it's time for a product recall. Long Range Plan. 1997;30(1):46–52.
33. Helms MM, Nixon J. Exploring SWOT analysis—where are we now?: a review of academic research from the last decade. J Strategy Manag. 2010;3(3):215–251.
34. Lewin K. Field Theory in Social Science and Selected Theoretical Papers. New York, NY: Harper; 1951.
35. The chronic care model. Improving Chronic Illness Care Web site. http://www.improvingchroniccare.org/index.php?p=The_Chronic_Care_Model&s=2. Accessed September 25, 2011.
36. Wagner EH, Austin BT, Davis C, Hindmarsh M, Schaefer J, Bonomi A. Improving chronic illness care: translating evidence into action. Health Aff (Milwood). 2001;20(6):64–78. [PubMed]
37. Glasgow RE. RE-AIMing research for application: ways to improve evidence for family medicine. J Am Board Fam Med. 2006;19(1):11–19. [PubMed]
38. Wright KB. Researching internet-based populations: advantages and disadvantages of online survey research, online questionnaire authoring software packages, and web survey services. J Computer-Mediated Commun. 2005;10(3)
39. Cronin KA, Feuer EJ, Clarke LD, Plevritis SK. Chapter 15: impact of adjuvant therapy and mammography on U.S. mortality from 1975 to 2000: comparison of mortality results from the CISNET breast cancer base case analysis. J Natl Cancer Inst Monogr. 2006;36:112–121. [PubMed]
40. Wagner EH, Greene SM, Hart G, et al. Building a research consortium of large health systems: the cancer research network. J Natl Cancer Inst Monogr. 2005;35:3–11. [PubMed]
41. Cancer centers list. National Cancer Institute Office of Cancer Centers Web site. http://cancercenters.cancer.gov/cancer_centers/cancer-centers-list.html. Accessed September 29, 2011.
42. Clauser SB, Johnson MR, O’Brien DM, Beveridge JM, Fennell ML, Kaluzny AD. Improving clinical research and cancer care delivery in community settings: evaluating the NCI community cancer centers program. Implement Sci. 2009;4:63–63. [PMC free article] [PubMed]
43. Ayanian JZ, Chrischilles EA, Fletcher RH, et al. Understanding cancer treatment and outcomes: the Cancer Care Outcomes Research and Surveillance Consortium [Erratum in J Clin Oncol. 2004;22(24):5026] J Clin Oncol. 2004;22(15):2992–2996. [PubMed]
44. National Cancer Institute. Breast Cancer Surveillance Consortium: Evaluating Screening Performance in Practice. NIH Publication 04-5490. http://breastscreening.cancer.gov/espp.pdf. Published April 2004. Accessed September 15, 2011.
45. Health communication and informatics research. National Cancer Institute Web site. http://cancercontrol.cancer.gov/hcirb/index.html. Updated May 4, 2011. Accessed September 29, 2011.
46. Centers of excellence in cancer communication research initiative. National Cancer Institute Web site. http://cancercontrol.cancer.gov/hcirb/ceccr/ceccr-index.html. Updated March 9, 2011. Accessed September 29, 2011.
47. Babb S, Tynan M, MacNeil A. State preemption of local smoke-free laws in government work sites, private work sites, and restaurants – United States, 2005-2009. MMWR. 2010;59(4):105–108. [PubMed]
48. King B, Dube S, Kaufmann R, Shaw L, Pechacek T. Vital signs: current cigarette smoking among adults aged ≥ 18 years—United States, 2005-2010. MMWR. 2011;60(35):1207–1212. [PubMed]
49. Griffin M, Babb SD, Tynan M, MacNeil AE. State preemption of local tobacco control policies restricting smoking, advertising, and youth access—United States, 2000-2010. MMWR. 2011;60(33):1124–1127. [PubMed]
50. Rogers EM. Diffusion of Innovations. 3rd ed. New York, NY: The Free Press; 1983.
51. Collins FS. The Language of Life: DNA and the Revolution in Personalized Medicine. New York, NY: Harper; 2010.
52. Khoury MJ, Coates RJ, Fennell ML, et al. Multilevel research and the challenges of implementing genomic medicine. J Natl Cancer Inst Monogr. 2012;44:112–120. [PMC free article] [PubMed]
53. Flood AB, Fennell ML, Devers KJ. Health reforms as examples of multilevel interventions in cancer care. J Natl Cancer Inst Monogr. 2012;44:80–85. [PMC free article] [PubMed]
54. Clauser SB, Wagner EH, Aiello Bowles EJ, Tuzzio L, Greene SM. Improving modern cancer care through information technology. Am J Prev Med. 2011;40:S198–S207. [PMC free article] [PubMed]

Articles from Journal of the National Cancer Institute. Monographs are provided here courtesy of Oxford University Press