PMCCPMCCPMCC

Search tips
Search criteria 

Advanced

 
Logo of nihpaAbout Author manuscriptsSubmit a manuscriptHHS Public Access; Author Manuscript; Accepted for publication in peer reviewed journal;
 
Am J Prev Med. Author manuscript; available in PMC 2014 March 3.
Published in final edited form as:
PMCID: PMC3940421
NIHMSID: NIHMS59439

Systems Thinking to Improve the Public’s Health

Abstract

Improving population health requires understanding and changing societal structures and functions, but countervailing forces sometimes undermine those changes, thus reflecting the adaptive complexity inherent in public health systems. The purpose of this paper is to propose systems thinking as a conceptual rubric for the practice of team science in public health, and transdisciplinary, translational research as a catalyst for promoting the functional efficiency of science. The paper lays a foundation for the conceptual understanding of systems thinking and transdisciplinary research, and will provide illustrative examples within and beyond public health. A set of recommendations for a systems-centric approach to translational science will be presented.

Introduction

Public health asks of systems science, as it did of sociology 40 years ago, that it help us unravel the complexity of causal forces in our varied populations, and the ecologically layered community and societal circumstances of public health practice.1

Green’s quote suggests that to improve public health, it will be necessary to gain a greater understanding of the complex adaptive systems involved in both causing and solving public health problems.2 For example, preventing and containing pandemic influenza requires collaboration across a wide array of disciplines and fields, including global surveillance to catch new outbreaks, rapid laboratory analysis of new viral strains so that effective medications can be developed, and the creation of expansive communications and informatics infrastructures so that communities can prepare and react effectively. Each separate activity to address pandemic influenza is necessary but insufficient in itself. However, when viewed together, the structures and functions to prevent and contain pandemic influenza represent an ever-changing complex adaptive system whose sum is greater than the parts. Indeed, millions—and perhaps billions—of lives depend on how well that complex system works.

The increasing emphasis on systems thinking as an organizing rubric reflects a confluence of trends among very different fields that have begun to emphasize systems thinking, including business, engineering, physics, military science, agriculture, weather forecasting and public health.3,4 While there is no single discipline for systems thinking, there are some fundamental systems-thinking perspectives and approaches that are shared across fields: (1) increased attention to how new knowledge is gained, managed, exchanged, interpreted, integrated, and disseminated; (2) emphasis on a network-centric approach that encourages relationship-building among and between individuals and organizations across traditional disciplines and fields in order to achieve relevant goals and objectives; (3) the development of models and projections, using a variety of analytic approaches (e.g., differential equations, agent-based modeling, system-dynamics modeling) in order to improve strategic decision making; and (4) systems organizing in order to foster improvements in organizational structures and functions.24

Consistent with this systems perspective, and echoing Rosenfield’s5 benchmark definitions of multidisciplinarity, interdisciplinarity, and transdisciplinarity, Stokols6 in this supplement to the American Journal of Preventive Medicine describes transdisciplinary research as a “process in which team members representing different fields work together over extended periods to develop shared conceptual and methodologic frameworks that not only integrate but also transcend their respective disciplinary perspectives.” Given the profoundly different ways that scientists collect data and define new knowledge within disciplines, along with the many different discipline-based assumptions about the nature of that knowledge, transdisciplinarity reflects an epistemology, or theory of knowledge, that has profound implications for how new knowledge is collected, synthesized, interpreted, and disseminated. This is not to suggest that unidisciplinary, reductionist science is no longer relevant. Rather, the increased emphasis on science that is transdisciplinary, translational, and network-centric reflects a recognition that much, if not most, disease causation is multifactorial, dynamic, and nonlinear.7 Indeed, scientific silos, or compartmentalized knowledge, have the potential to impede understanding of the complex inter-relationships among variables.8

It is perhaps neither possible nor desirable to eliminate the silos of science, but there is increasing recognition that it is essential to link them and to recognize that they represent components of a larger system.2 That is, transdisciplinary science represents a necessary but insufficient aspect of complex adaptive public health systems. Achieving effective and lasting advances in public health clearly depends on the knowledge gained through transdisciplinary science (e.g., the biological and behavioral causes of tobacco dependence, or social and biological factors that cause the spread of communicable diseases). But achieving those gains also requires making strategic decisions about which complex scientific questions will lead to the greatest public health gains, how new discoveries can be disseminated effectively, and what structures and functions are needed to deliver the new knowledge. The opinion that complex challenges cannot be solved by reductionist approaches alone reflects an orientation toward systems thinking that Senge9 called a “fifth discipline.” And this fifth discipline is highly consistent with the principles of systems thinking and cybernetics that were discussed long ago by von Bertalanffy,10,11 Wiener,12 and Ackoff,13 and more recently by Leischow and Milstein,2 Sterman,14 Midgely,15 and Green.1

Systems Thinking for Hurricanes and H5N1 Avian Influenza

Because systems thinking is often difficult to conceptualize, exemplars of both systems design and systems analysis can serve as valuable models for those who are unfamiliar or even perplexed by what is meant by the term. While many examples exist, weather forecasting and the prevention of communicable disease will be described here.

Weather Forecasting

Perhaps one of the most advanced transdisciplinary collaboratives that is fundamentally oriented toward the conceptual framework of systems thinking is weather modeling and forecasting.16 Networks of organizations and scientists from around the world work together to understand the complexity of weather patterns so that more accurate and timely weather forecasts can be made. The Weather Research & Forecasting Model group employs a type of translational model whereby new discoveries made within a particular discipline (e.g., oceanography) are linked together, so that complex relationships can be determined by transdisciplinary teams of scientists (i.e., physicists, atmospheric chemists, geographers). Models can be developed that explain the data, and optimized models can then be disseminated to specific end-users and the public. Understanding the interplay of solar activity, land masses, water temperatures, wind flow, and other natural forces has made it possible—via complex and intensive computational modeling—to develop predictive weather models that have both saved lives and reduced economic devastation. Indeed, the National Oceanic and Atmospheric Administration, in collaboration with more than 150 universities, implemented a new computer system that can model ever-more-complex data (e.g., wind activity at specific elevations, humidity differences between night and day, the amount of Arctic ice) in order to develop improved forecasts.17

An integral part of the weather forecasting system is communication with the public. The example of Hurricane Katrina serves as a reminder that having accurate forecasting and analysis of a complex weather system does not necessarily translate into an effective use of that information. Indeed, Katrina was a tragic example of the dire consequences of a failed delivery component of the system. Many years of investment into collecting data from a variety of sources led to accurate forecasts, which in turn gave millions of people in Katrina’s path time to escape; however, the application of that knowledge by federal, state, and local officials failed. The devastating outcome was a reminder that a complex system worth investigating lends itself to large-scale organizational change as a result of new knowledge. This phenomenon is both the promise and the challenge of systems thinking.

Preventing the Next Global Pandemic

In 1918–1919, the Spanish influenza pandemic spread globally in waves, killing between 50 and 100 million people worldwide.18,19 This viral infection was the last pandemic in the U.S., and if history is consistent, there will be additional pandemics in the future. In recent years, the H5N1 Avian influenza has been of paramount concern because it is deadly to humans and could rapidly spread if mutations allow it to easily pass from human to human. Fortunately, as in the weather forecasting example above, public health agencies worldwide have recognized this risk and have implemented systems—including transdisciplinary teams of scientists—to prevent or minimize the risk of a future communicable-disease pandemic.

In the U.S., the CDC coordinates a comprehensive surveillance-and-response system to anticipate and manage influenza outbreaks. One component of this system is BioSense,20 a real-time surveillance system that links data from local and national sources to identify and track new and existing influenza outbreaks. Another component, also under the supervision of the CDC, is the Laboratory Response Network,21 an integrated system of laboratories at the local, national, and international level, that serves as a rapid reporting-and-response infrastructure for communicable disease and bioterrorism. This comprehensive system assures that “hot spots” of influenza will be identified early, so that local healthcare systems can mobilize, and policymakers can take appropriate action to prevent the spread of disease. In addition, the NIH has increased its investment in the development of new drugs to treat influenza, and has created an initiative called Models of Infectious Disease Agent Study, a “collaboration of research and informatics groups to develop computational models of the interactions between infectious agents and their hosts, disease spread, prediction systems, and response strategies.”22

The overall goal of these and other efforts is to bring together those who are critical to the discovery, development, and delivery of the knowledge, products, and services that will most effectively prevent and treat communicable disease. This comprehensive and multidisciplinary systems approach to preventing a massive outbreak of disease that could kill millions of people depends, like the weather-forecasting system, on (1) massive and rapid data collection from many different sources; (2) rapid communication to a broad array of sources; (3) transdisciplinary science, in order to understand and analyze data from many sources; and (4) modeling of the complex relationships among the components in the system. These four elements are necessary for the creation of more accurate predictions and recommendations that can be used by policymakers to protect the health of the public.

Systems Thinking in Public Health and Learning from ISIS

Despite the promise that systems approaches hold for improved understanding of the complex factors that contribute to health and disease, few systems initiatives have been developed at one of the premier U.S. center for health research—the NIH—to address chronic disease or its causal factors. A recent exception is the pilot Initiative on the Study and Implementation of Systems (ISIS). Aware of the systems-thinking approaches that have been applied in other areas and given the complex nature of tobacco use and tobacco-related disease, the National Cancer Institute (NCI) funded ISIS to explore how systems-thinking approaches might improve the understanding of the factors contributing to tobacco use; to inform strategic decision making about which efforts might be most effective for reducing tobacco use and tobacco-related disease; and to serve as an exemplar for addressing other public health problems. More specifically, ISIS was intended to become a long-term, multi-agency collaboration to create and implement transdisciplinary-systems principles and methods for the discovery, development, and delivery of program and policy interventions within a research-to-practice paradigm.

Developing and Defining the Four Key Areas in Systems Thinking

Given the multiple systems approaches that have been employed to address complex problems (e.g. weather forecasting, communicable disease, managing the economy, conducting military operations), one of the goals of ISIS was to identify what they have in common, so that this information could be used to identify effective ways to improve tobacco control. More specifically, a strategic-planning and development process was put into place to consider existing literature; the efforts of experts in other fields (e.g., the military, business, system dynamics, etc.); and experts across several disciplines within the tobacco-control field.

In addition to focus groups and other formative efforts completed during the first year of ISIS, a process led by noted system-dynamics expert George Richardson23 was implemented to explore what is meant by a tobacco control system. As a result of that process, two important conclusions emerged: (1) understanding and implementing complex systems is all about the relationships among people, collections of information, and even concepts; and (2) these relationships work or do not work as a function of information and how it is communicated. Thus, as the ISIS team began exploring complex relationships via system-dynamics thinking and modeling, two of what became four key principles emerged very rapidly: Without effective information and knowledge exchange, social networks do not function effectively; in addition, when social networks oriented to public health are not functioning effectively as a result of inadequate or dysfunctional information and knowledge exchange, systems that could be effective are compromised and even prevented from achieving their potential positive impact. A perfect example of what can go wrong is the outcome of Hurricane Katrina.

Conversely, when knowledge flow is effective, network performance is better, and systems-level change is possible. An example is community-driven policy change, wherein over the last few decades there has been an increased shared awareness24 that higher cigarette taxes and restrictions on smoking in public places would result in significant drops in smoking prevalence. Consequently, many states and nations focused their tobacco-control efforts on increasing tobacco taxes and legislating bans on smoking in public places.

As a result of the activities in the strategic-planning process, the ISIS group identified four priority areas (Figure 1) that together serve as a synergistic foundation for understanding and improving the public’s health from a systems perspective. They do not represent the only possible foundations, and certainly do not represent all of the critical areas within the public health system that require attention, but they do reflect both conceptual and functional areas that together result in a sum greater than their individual contributions.

Figure 1
ISIS strategic-planning activities and key priorities

A brief summary of each area, drawn from the NCI monograph25 on systems thinking that these authors developed, summarizes the relevance of each to the systems approach that the ISIS team delineated.

1. Managing systems knowledge

The management and transfer of shared knowledge form the basis of interaction between stakeholders in a systems environment. The development of an effective system requires a comprehensive, sophisticated infrastructure for knowledge management and transfer that is based on integrating existing silos of information, and manages both explicit knowledge (what we know we know) and tacit knowledge (what we do not know we know; unconscious lessons from experience). This knowledge environment must be collaborative, in keeping with the needs of the stakeholders it supports, and able to meet the changing needs and methods underlying a systems approach to tobacco control. It must also be evolutionary.

To demonstrate the potentials of a web-based, collaborative-knowledge environment for tobacco control, the NIH and other partners created a cyber-infrastructure to improve the sharing, analysis, and dissemination of tobacco data. This tobacco web portal, currently called the Tobacco Informatics Grid (TobIG), will use state-of-the-science information technology and networking software to link tobacco data, researchers, and resources (e.g., citation indexes, data mining, and visualization software). TobIG is envisioned as a cyber-infrastructure to support a voluntary network, or grid, of tobacco-control stakeholders to data and software/analysis tools. TobIG was conceived to be part of a multicomponent strategy to speed the development and delivery of innovative approaches to tobacco control that would link directly with the larger NCI-funded cancer–bioinformatics grid (caBIG).

2. The power of transdisciplinary and multidisciplinary systems networks

Networks form the backbone of a system by harnessing the power of linking diverse stakeholder individuals and groups. Understanding the formation and management of networks and using that knowledge to foster healthy networks in tobacco control are critical components of a systems environment in public health. To better the understanding of how multidisciplinary and organizational communication and collaboration were occurring in tobacco control, several network projects were implemented by the ISIS team. These projects included Mapping the Tobacco Harm Reduction Network (presented in detail in this supplement26); the Global Tobacco Research Network (GTRN); and the Social Network Mapping of Tobacco Control at USDHHS.

Global Tobacco Research Network

The GTRN is a virtual web of interconnected scientists and organizations collaborating in the conduct, synthesis, and dissemination of tobacco-control research in support of a progressive, policy-relevant research agenda. Functioning through its web interface,27 the program provides network consolidation, information management, and information sharing. One product is the Research Assistance Matching Program (Program RAM), in which mentors are matched with novice researchers.

Social network mapping of tobacco control efforts within USDHHS

A social network analysis was used to delineate the connections among the agencies doing tobacco control work within the USDHHS to identify communication gaps and any silos of information (DA Luke, NB Mueller, Saint Louis University, unpublished technical report, 2005). Figure 2 shows the extent of contact between organizations regarding tobacco control on at least a quarterly basis. The size of each node represents betweenness, or how often the individuals within an organization act as a bridge between other organizations in the network. The isolates in the display (i.e., the Food and Drug Administration and the Center for Medicare and Medicaid Services) suggest that much can be done to strengthen the tobacco-control communications network with the USDHHS.

Figure 2
Social-network analysis of tobacco control in the USDHHS

3. Methods for analyzing complex systems

System dynamics involve methods that facilitate a more-constructive examination of complex adaptive systems by modeling the behavior of actions and their consequences, both intended and unintended. These methods are particularly well-suited to tobacco control, which encompasses an ongoing struggle with countervailing factors that change over time and can be strengthened. There is considerable promise in a range of systems approaches, including formal system-dynamics modeling techniques and group processes that harness the problem-solving capabilities of multiple stakeholders. These approaches constitute tools that help address problems that are increasingly dynamic and complex.2,14,15

To explore this methodology within the ISIS initiative, system-dynamics modeling methods were used to simulate tobacco prevalence and consumption over a 40-year period across various age groups. The ISIS system-dynamics model used a participatory team process among stakeholders to define causal factors in tobacco prevalence, as well as to provide estimates of empirical model data. Formal empirical data from sources such as Morbidity and Mortality Weekly Report were used for both model parameters and results validation. A causal-loop model of factors in tobacco prevalence and a formal simulation model of specific shards of this model were developed, using the VENSIM simulation language. One such model is an aging chain of smokers (Figure 3), which explores tobacco use across the lifespan and begins to take into account changes in smoking status, death, and outside influences, in order to inform the modeling process for predicting future tobacco-related morbidity and mortality. This figure, although a bit daunting at first glance, shows the dynamic nature of youth uptake of tobacco through the development of addiction and the potential outcomes through adulthood. Such models can be fit with data (e.g., time to addiction, relapse percentages) to better convey the complexity of the tobacco problem and to identify points in the system where interventions are likely to yield the greatest impact.

Figure 3
Aging chain of smokers

4. Systems organizing

Systems organizing reflects an evolution from traditional management theory to a learning organization,3,4 or an adaptive-systems perspective within a systems environment. Its major message is the evolution of current concepts of managing and organizing by transforming traditional top-down, command-and-control structures to encompass network-centric participatory approaches, the effective evaluation of system complexity and dynamics, and explicit attention to knowledge flow and management. Methods of organization are envisioned as a continuum from formal organization in the traditional management sense to self-organizing partnerships or collaborations. For example, in order to identify the specific genes associated with a particular disease, scientists from multiple disciplines might come together for the purpose of that project and then spin off into other groups to explore other problems. This dynamic process of systems organizing fosters not only increased collaboration to address a particular problem but also an inherent recognition that complex problems require transdisciplinary teams that will change as the problems change.

To explore how systems-organizing approaches could be used in public health contexts, the ISIS project looked at two examples (one appears in Figure 4) that utilized a collaborative, participatory, structured conceptualization methodology known as concept mapping28,29 to model and graphically depict aggregated clusters of ideas or concepts held by groups (or networks) of stakeholders. This concept-mapping methodology is a good example of a systems-organizing approach that can be utilized either in a face-to-face, real-time group process or in a distributed asynchronous process over the Internet. Concept mapping enables a diverse group of stakeholders to brainstorm a broad spectrum of specific issues that address a mapping focus, organizes these issues through individual sorting and rating, and then synthesizes this input across individuals, using several multivariate statistical methods (multidimensional scaling and hierarchical cluster analysis). The results are graphically presented as conceptual maps. Figure 4 provides an example in which stakeholders associated with state and local tobacco-control efforts developed a conceptual model of the components of a strong tobacco-control program.

Figure 4
Concept-mapping example

Taking a Systems-Centric Perspective in Science

There is a critical need for government agencies to take a leadership role in fostering increased transdisciplinary and translational collaboration and to employ an approach that recognizes that public health is the culmination of a complex, adaptive federation of systems22 that no one organization can or should control. While comprehensive, centralized, hierarchical control is not the desired system goal here, there is an essential facilitative role that needs to be played by hierarchical, centralized organizational entities like the federal government, which can provide the leadership essential to developing a framework for action, and encourage and support the process of fostering collaboration among a diverse group of stakeholders. For example, in part as a result of the ISIS effort, the NIH Office of Behavioral and Social Sciences Research has identified systems thinking as fundamental to its strategic planning.30 Similarly, the President’s Cancer Panel presented a translational model that reflects a systems approach (discovery, development and delivery), the success of which depends on collaboration both among and between scientists and, just as importantly, among scientists, clinical providers, community providers, policymakers, and the public to ensure that new discoveries can be implemented to improve health in the fastest way possible.31

At the completion of the ISIS initiative, the ISIS team developed several recommendations (Table 1) for fostering movement toward a more systems-centric approach to translational science.25 Some of the recommendations were very concrete, such as studying the networks of networks and developing cyber-infrastructures, and others were conceptual, such as encouraging transdisciplinarity and encouraging ecologic perspectives on implementation.

Table 1
ISIS recommendations

However, inherent in the ISIS initiative and the resulting recommendations was a recognition that each of the four domains are intertwined and, in fact, depend on each other. This recognition exemplifies systems thinking, because it is oriented to the identification and understanding of complex relationships, not just the dissection of them. Thus, the ISIS team further concluded, a fundamental goal must be the creation of an integrated systems-thinking environment that requires a strong orientation toward new approaches to team science (Figure 5).

Figure 5
Integrative systems-thinking framework for complex systems in public health

The interplay of systems components to improve public health presented in Figure 5 illustrates the need for new approaches to team science that have a transdisciplinary orientation, as well as new approaches to training that integrate reductionist and systems epistemology, that promote a translational orientation, and that are oriented toward the understanding of complex relationships and the fostering of teams to better address public health challenges as complex adaptive systems. Tackling complex public health problems requires transdisciplinary and multidisciplinary teams to understand and address that complexity, and systems thinking is a path for getting them there.

Acknowledgments

The authors wish to thank the many individuals whose insights led to the ISIS recommendations. In addition, they wish to thank Gregg Moor for his editorial help.

No financial disclosures were reported by the authors of this paper.

Footnotes

Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

References

1. Green LW. Public health asks of systems science: to advance our evidence-based practice, can you help us get more practice-based evidence? Am J Public Health. 2006;96:406–9. [PubMed]
2. Leischow SJ, Milstein B. Systems thinking and modeling for public health practice. Am J Public Health. 2006;96:403–5. [PubMed]
3. Best A, Moor G, Holmes B, et al. Health promotion dissemination and systems thinking: towards an integrative model. Am J Health Behav. 2003;27(3S):S206–16. [PubMed]
4. Best A, Tenkasi R, Trochim W, et al. Systematic transformational change in tobacco control: an overview of the Initiative for the Study and Implementation of Systems (ISIS) In: Casebeer AL, Harrison A, Mark AL, editors. Innovations in health care: a reality check. New York: Palgrave Macmillan; 2006.
5. Rosenfield PL. The potential of transdisciplinary research for sustaining and extending linkages between the health and social sciences. Soc Sci Med. 1992;35(11):1343–57. [PubMed]
6. Stokols D, Hall KL, Taylor BK, Moser RP. The science of team science: overview of the field and introduction to the supplement. Am J Prev Med. 2008;35(2S):XXX–XXX. [PubMed]
7. Barabasi AL. Network medicine—from obesity to the “diseasome. N Engl J Med. 2007;357:404–7. [PubMed]
8. Satterfield D, DeBruyn LM. The malignment of metaphor: silos revisited–repositiories and sanctuaries for these times. Am J Prev Med. 2005;29(3):240–1. [PubMed]
9. Senge PM. The fifth discipline: the art and practice of the learning organization. New York: Doubleday; 1994.
10. von Bertalanffy L. The theory of open systems in physics and biology. Science. 1950;11:23–9. [PubMed]
11. von Bertalanffy L. In: A systems view of man: collected essays. Violette PA, editor. Boulder CO: Westview Press; 1981.
12. Wiener N. Cybernetics or control and communication in the animal and the machine. Cambridge MA: MIT Press; 1948.
13. Ackoff RL. Systems, organizations and interdisciplinary research. In: Eckman DP, editor. Systems research and design. New York: Wiley; 1961.
14. Sterman JD. Learning from evidence in a complex world. Am J Public Health. 2006;96:505–14. [PubMed]
15. Midgley G. Systemic intervention for public health. Am J Public Health. 2006;96:466–72. [PubMed]
16. The Weather Research & Forecasting Model. www.wrf-model.org/index.php.
17. National Weather Service. NCEP’s environmental modeling center unveils new national forecast model. www.weather.gov/com/nwsfocus/fs20060711.htm#first.
18. Osterholm MT. Preparing for the next pandemic. N Eng J Med. 2005;352:1839–42. [PubMed]
19. The great pandemic: the U.S. in 1918–1919. 1918.pandemicinfluenza.gov/the_pandemic/index.htm
20. CDC. BioSense. www.cdc.gov/biosense/
21. CDC. Emergency preparedness & response: the laboratory response network: partners in preparedness. www.bt.cdc.gov/lrn/
22. National Institute of General Medical Sciences. Models of infectious disease agent study. www.nigms.nih.gov/initiatives/midas/
23. Richardson GP, Pugh AL. Introduction to system dynamics modeling with DYNAMO. Cambridge: MIT Press; 1981.
24. Krygiel AJ. Behind the wizard’s curtain: an integration environment for a system of systems. Washington DC: Institute for National Strategic Studies; 1999. www.dodccrp.org/files/Krygiel_Wizards.pdf.
25. National Cancer Institute. Tobacco Control Monograph No. 18. Bethesda MD: USDHHS, NIH, National Cancer Institute; Greater than the sum: systems thinking in tobacco control. NIH Pub. No. 06-6085, 2007. http://dccps.nci.nih.gov/TCRB/monographs/18/index.html.
26. Provan KG, Clark P, Huerta T. Transdisciplinarity among tobacco harm–reduction researchers: a network analytic approach. Am J Prev Med. 2008;35(2S):XXX–XXX. [PubMed]
27. Global Tobacco Research Network. www.tobaccoresearch.net.
28. Trochim WM. An introduction to concept mapping for planning and evaluation. Eval Prog Plann. 1989;12:1–16.
29. Kane M, Trochim WM. Concept mapping for planning and evaluation. Thousand Oaks CA: Sage Publications; 2007.
30. Office of Behavioral and Social Science Research, NIH. 2007 Mar 1; obssr.od.nih.gov/content.
31. Reuben SH. Translating research into cancer care: delivering on the promise. 2004–2005 annual report of the President’s Cancer Panel. Washington DC: USDHHS; 2005. deainfo.nci.nih.gov/advisory/pcp/pcp04-05rpt/reporttrans.pdf.