Search tips
Search criteria 


Logo of jncimonoLink to Publisher's site
J Natl Cancer Inst Monogr. 2012 May; 2012(44): 42–48.
PMCID: PMC3482974

Time Issues in Multilevel Interventions for Cancer Treatment and Prevention


The concept of time introduces important complexities in estimating intervention effects, program and evaluation design, and measurement and analysis of individual change in multilevel interventions (MLIs). Despite growing recognition that time is a critical element for assessing both individual-level outcomes and higher-level changes in organizational, community, and policy contexts, most MLI designs and evaluations have not addressed these issues. In this chapter we discuss 1) conceptualizing disease life-course and treatment theory in MLIs, 2) approaches to incorporating time in research and program design for MLIs in cancer treatment and prevention, 3) analysis of time-varying multilevel data in the context of cancer treatment and prevention, and 4) resource considerations and trade-offs of incorporating time as a dimension of MLIs and analysis. Although analytic techniques for analyzing time-related phenomena are becoming more available and powerful, there has not been corresponding progress made in the development of theory to guide the application of these techniques in program design and implementation.

The concept of time introduces important complexities in designing and evaluating programs, measuring and analyzing individual changes, and estimating intervention effects in multilevel interventions (MLIs). For example, investigators may focus on time to address whether interventions are more effective in early stages of implementation or whether their effects attenuate over time. Assessing how time is experienced by individual patients may influence both the design of MLIs and the analysis of the effects of such interventions. For example, assessing how interventions interact with individual growth trajectories to affect outcomes of interest may allow investigators to gain a more nuanced understanding of an intervention’s impact on individuals. At the environmental and organizational levels, interventions that change structures or processes often play out over extended time intervals and not always in linear fashion.

In this chapter, we discuss issues of time related to 1) conceptual relevance of time in multilevel cancer interventions, 2) time as a research and program design issue in MLIs, 3) analytic methods for dealing with time in MLIs, and 4) resource considerations and trade-offs when incorporating time as a component of multilevel research.

Conceptual Relevance of Time in Multilevel Cancer Interventions

The empirical literature is scant regarding the role of time in MLI research. The limited number of published studies have cited the challenges of insufficient time allowed for implementing the intervention (1) and for introducing changes into existing work load or routines. Some work has described different ways in which time may be treated in the context of intervention studies. Time has been conceptualized in four ways: 1) “clock and calendar time,” which is structured, and addresses “when,” “how often,” “how long,” and “in what order” activities need to occur (this conceptualization recognizes that time may be linear but its effects as experienced are frequently nonlinear); 2) “temporality,” which conveys processes that unfold in time; 3) “timing,” which is dependent on the organizational context in which an intervention is applied; and 4) “tempo,” which is related to a patient's perception of time (eg, the rate at which a patient perceives his or her disease to have progressed) (2,3).

The Symptoms Experience in Time (SET), developed for symptom management research, is another conceptual model for understanding time. It incorporates antecedents that include a precipitating event, such as cancer or a disease process. In addition to clock/calendar time and tempo, three other temporal assumptions describe time as both a concrete (ie, biological/social time) and an abstract concept (ie, sense of timelessness amid intense symptoms) (4).

This conceptual work underscores the importance of understanding complex temporal dynamics from both the patient's perspective and the care delivery standpoint when planning or analyzing MLIs. Such notions of time imply a range of dynamic interactions between the intervention operating at different levels of influence and individuals targeted by those interventions (5). Time is experienced on two dimensions, intervention target and intervention context. The intervention target is the level where change is intended (ie, individual, organization), and which defines the relevant context of the intervention. Intervention context is the external and/or internal milieu of the intervention target. Time possesses several attributes that include but are not limited to life-course (patient's developmental trajectory), disease-course, timing, duration, frequency, and sequence. Each time attribute has the potential to influence outcomes across different phases of the intervention.

Figure 1 illustrates where in the intervention process time may contribute to intervention effects. For example, if the intervention target is the patient, individual life-course or disease-course may affect the intervention outcome through their influence on the design and implementation phases. Context of an intervention targeting individual patients could include encounters with the provider team or family dynamics occurring at the time. Multiple physical and psychological processes that an individual experiences due to a physiological disease-course or a clinical care process describe the context within which time attributes need to be considered. Since these are simultaneous, individual and contextual processes, an intervention would need to address the patient's life-course and/or disease-course, while the organization considers the timing, duration, frequency, and sequence during the design and implementation phases.

Figure 1
Time in multilevel interventions illustrative example: individual as intervention target.

Time as a Dimension of the Individual

Cancer progresses through multiple stages or periods, although there may be more “stages” to cancer than to other diseases and more variations in the disease process. For example, Rasmussen and Elverdam (6) emphasize the importance of understanding that cancer survivors experience their trajectory as a process in the midst of constant changes. Similarly, Henly et al. (7) maintain that because health and illness conditions are associated with processes and changes, interventions targeting such conditions imply a temporal context. These understandings can provide deeper insight into changes in patients’ values and priorities, signaling when intervention is appropriate and consequently effective. For purposes of MLI and analysis, they also suggest rigorous assessment of time-related trajectories of individual patients during their disease life-course.

Individual Growth Trajectories in the Context of MLIs

A trajectory is simply a path, progression, or line of development in some outcome of interest. For example, Figure 2 illustrates hypothetical individual growth trajectories for level of anxiety about prostate examinations. The trajectories indicate that anxiety initially increases as a function of time since last exam but generally level off after a certain point before picking up again. These trajectories suggest that the impact of an intervention to improve screening may have differential effects depending on when individuals are introduced to the intervention. Thinking about individual trajectories is an exercise in understanding individual growth or change over time. We can think about within-individual variation (ie, changes in cancer outcomes/behaviors over time) or between-individual variation (eg, different trajectories for older vs younger cancer patients, or for patients in different racial/ethnic groups). Individual growth models typically consider factors in early life or current risk factors that may alter outcome trajectories over time. For example, patients experience a trajectory of life events that enhance or inhibit their health status contributing to creating a subjective reality of health (lifeworld) that may also change over time. Thus, patients’ readiness and responsiveness when introduced to an intervention—and consequently the outcome or effect of the intervention—is mediated through their lifeworld and where they are in their life-course (8).

Figure 2
Hypothetical individual growth trajectories for prostate screening anxiety level.

From an analytic perspective, individuals are viewed as having their own regression equations that reflect such circumstances. Typically, each individual's unique trajectory of change is an outcome and is represented as a function of person- and time-specific parameters, such as age, income, wealth, comorbidities, marital status, and social support. Age also is often considered to examine how outcome trajectories may vary by age over time (9,10). Recognizing that these parameters can either vary or remain static over time is critical when developing measurement models in MLIs.

Given the importance of improving individual-level outcomes in cancer treatment and prevention, growth trajectories might be considered one of the basic building blocks of multilevel cancer interventions. Learning about predictors of trajectories, for example, can help to inform interventions (program/policy) by suggesting when to introduce interventions or determining the expected direct or joint effects of the intervention and time. For example, studying the course of a disease over time could enable investigators to identify factors associated with onset, remission and relapse and to describe antecedents and long-term outcomes at the individual level, all of which may influence the timing and appropriateness of various interventions. Incorporating such trajectories also would allow investigators to identify factors that predict differences in mean growth rates (outcomes) between several types of cancer treatment interventions but also to identify factors that predict differences in growth rates (outcomes) between individuals over time.

Recognizing the importance of growth trajectories, at least three approaches can be used to incorporate time as a dimension of MLI models. At the most basic level, one can identify temporal patterns in the data. For example, does the outcome (ie, tumor-free survival rate, medication adherence rate) increase, decrease, or remain stable over time? Is the general pattern linear or nonlinear? Are there abrupt shifts at substantively important moments?

Second, “time-varying predictors” can be used to model how changes in factors, such as family circumstances (eg, income, social support), may influence the likelihood that the cancer intervention will have the desired effect (ie, participation in screening) and the conditions under which it is likely to do so. Third, and perhaps most important for purposes of MLIs, variables representing the intervention can be interacted with time to test whether an intervention's effect varies over time. This is particularly important given that some effects dissipate over time, some effects increase as individuals become acclimated to the procedure or condition, and some effects may be especially pronounced at particular times. For example, when comorbid conditions are not undergoing intensive treatment in the clinical setting, a cancer patient can become more adherent to a rehabilitation intervention designed to strengthen and improve physical functioning.

Time and Organizational Change in MLI Research

From an organizational perspective, time takes on a somewhat different role in multilevel research and intervention planning. Interventions are fundamentally about system change, particularly those that target the higher-level contexts in which individuals function, such as institutions, teams, or networks. Organizations vary in their structures, missions, resource availability, and staff over given time periods. And, just as individuals change over time, organizations and communities evolve, which affects outcomes and the fidelity with which an intervention is practiced (11).

In a related vein, the timescale over which change is expected may be directly influenced by the level (or levels) at which an intervention is implemented (eg, organizational, policy, system). If the impact on outcomes is mediated by pervasive organizational system effects, it may take an extended period to filter through an organization and only be observable after the program or evaluation ends. Obviously, this creates problems for assessing multilevel program effects over a short study time frame. Conversely, attributing causality to a program or intervention over longer time frames is made difficult by concurrent influences or secular changes or events that may confound any intervention effect. Collins and Graham (12) provide a useful overview of the effects of the timing of measurements on empiric findings. They caution that a mismatch between measurement intervals and anticipated outcomes may impair a researcher's ability to correctly model the processes of interest. The meaning of measurements taken across multiple time points may themselves change as a consequence of interventions. Andrykowski et al. (13), for example, have reported response shifts in the fatigue ratings of women undergoing adjuvant therapy for breast cancer as the women recalibrate the personal standards they use to evaluate current fatigue levels throughout the course of their treatment.

Another timing issue relates to potential data-related differences in institutional commitments, efficiencies, and policies across organizations, which may affect the availability of information relevant to research. For example, the interval necessary for data from electronic medical records to become accessible to researchers may differ considerably across facilities due to any of these factors. Even if available in a timely fashion, the usefulness of digitized data available from these records also may vary across organizations (14). Under most circumstances, individual health data will likely become available from electronic systems more quickly than will aggregated health information, such as prevalence data produced as part of the Surveillance, Epidemiology and End Results (SEER) program and state-level vital statistics offices.

Time as a Research and Program Design Issue in MLIs

Time and Change as a Factor in Study Design

Cross-sectional evaluations that study different groups of individuals at different points in time are inadequate to examine processes such as clinical improvement, progress, behavioral change, or other aspects of change that are inherently longitudinal. More fundamentally, cross-sectional designs in MLI research make it difficult to ascertain whether the intervention produced a change in individual patient outcomes or behavior or, alternatively, if observed associations are a result of self-selection, omitted variable problems, or other sources of bias.

However, evaluation designs that incorporate time as a dimension may be less susceptible to the influences of individual background, intake characteristics, and other confounding factors. In a repeated measures design in which subjects are tracked over time, individuals serve as their own controls. As a result, stable characteristics of the individual are constant over time and cannot confound estimation of the growth curve. This potential research design advantage is an important one in nonexperimental MLI studies, where control over confounding influences, such as context, policy, or organizational characteristics, is often insufficient.

Another advantage of applying time and growth models in the context of MLIs relates to the notion that there are at least two parameters of interest to analysts, program designers, and policy makers: initial level of performance (intercept) and the rate of change (slope) (15). These two parameters may also interact. For example, it is conceivable that differences in the extent or progression of a particular cancer at the outset of individuals’ participation in an MLI (initial level) may result in differences in the rate of improvement for those individuals exposed to the intervention (rate of change). This interaction may be such that more severely ill patients experience higher rates of improvement than less severely ill patients even though they receive the same “treatment.” This is a crucial distinction because even an effective cancer intervention may not be able to exert limitless influence over the absolute level of an individual patient's outcome but can potentially influence the rate of improvement in the same outcome.

Time as a Factor in Implementation and Sustainability of MLIs

In response to criticism levied against traditional experimental and quasi-experimental program evaluation studies, evaluators are now measuring treatment implementation and sustainability more closely. Measuring treatment implementation provides a means to determine whether key program components were delivered as specified by the program logic model/theory (16,17). Assessing sustainability is concerned with whether such components are active long enough to produce the desired effect on individuals (11). Both imply temporal assessment.

Implementation monitoring allows the evaluator to uncover service delivery breakdowns and unwanted side effects during the early stages or entropy of critical program elements over longer periods (fidelity). Both provide the potential for real-time process corrections or adjustments (18). Close monitoring of the implementation process over time also allows investigators to use program or treatment fidelity data to contextualize treatment outcomes, such as those occurring at the individual level. Careful thought about organizational structures, workflow, and other operational processes, as well as individuals’ needs, is required to reveal how time may change the observed effects in the real world during actual implementation or institutionalization.

Sustainability is the capacity to adapt to new conditions and endure once the interventions have been institutionalized or embedded as routine standard practice or lifestyle (19). Without institutionalization, sustaining the benefits of the interventions is a challenge (20). Factors such as health-care markets, societal norms and beliefs, and changes in national and state policy can all influence an intervention's sustainability. This is particularly important for MLIs because the effects of an organizational or policy intervention on individual outcomes may vary over time as a function of the entropy of particular program elements or exogenous changes in organizational contexts (21).

Greater recognition of the potential for a planned intervention to deviate from the intended design or attenuate over time requires evaluators to more closely focus on identifying essential (ie, active) program components and monitoring the extent to which the treatment protocol is adhered to or modified in practice and over time (11). This implies a longitudinal assessment of interventions (vs a single point in time assessment of intervention implementation). For example, multidisciplinary care conferences (MDCCs) are designed to plan prospective treatment that incorporates evidence and evaluate the patient's eligibility for clinical trials, thereby increasing patient confidence in their care by incorporating evidence into patient treatment planning. However, the ability of an MDCC to affect patient-level outcomes and its sustainability over time depends on the implementation climate within which the MDCC functions (22)—specifically whether the new MDCC process is viewed as an organizational priority, is supported by organizational and clinical leadership with dedicated time and resources, and has appropriate incentives (23,24).

Intervention adherence is itself often described as a multilevel multivariate phenomenon. This characterization stems from recognition that the fidelity with which a given intervention is implemented by a particular provider follows from interrelationships among a range of internal and external factors that constitute the social system surrounding the intervention. Treatment providers can be expected to systematically increase or decrease adherence to protocol on the basis of a variety of factors, including initial and ongoing training, perceived efficacy, and organizational support. Given this, the typical, single point-in-time assessment of intervention adherence may result in a misleading static representation of the intervention. Evaluation designs that allow for implementation measurements across multiple points during the course of the intervention would permit better estimation of average adherence levels as well as enable examination of factors that promote or inhibit a positive change in adherence. For example, as organizational priorities change and resources become more or less available over time, management decisions may influence the timing of intervention implementation, ongoing support for an intervention, or even whether the intervention is to be retained at all. Such considerations are especially important when MLIs require extended periods to ramp up or when effects on patient outcomes are likely to require sustained “doses” of the intervention.

In many domains, three observations (conducted at the beginning, middle, and end of the treatment period) would provide for a reasonable estimate of the level and linear change in adherence to protocol. Additional observations, of course, would further increase an evaluator's ability to identify the underlying trajectory of change and reliably estimate associated implementation trajectories.

Analytic Methods for Dealing with Time in MLI Research

Despite the conceptual, practical, and evaluation challenges, both quantitative and mixed methods approaches are available to support the analysis of time within a multilevel framework. In this section, we briefly review several current strategies for analyzing time in MLIs.

Quantitative Advances

It is now widely recognized that hierarchical linear models (HLMs) offer a powerful approach to the conduct of longitudinal analyses across three or more levels (25). These models also are commonly known by other names, such as mixed effects regression models (26) and multilevel models (27). These approaches are closely related and share several key features. Most importantly, each recognizes the nested or clustered data structures underlying longitudinal and multilevel assessments and introduces adjustments to control for these dependencies. In addition, these various approaches each permit the inclusion of time-varying as well as time-invariant (ie, fixed) covariates and independent variables. They also recognize the importance of modeling correlated errors when dealing with clustered observations.

Typically, these models include multiple time observations embedded within individual patients. In analyses of clinical trials, patients, in turn, can be nested within treatment conditions. Alternatively, in analyses of community interventions, multiple time assessments would be nested within individuals, which, in turn, could be nested within neighborhoods or other organizational structures. These models are highly flexible and offer several advantages. First, they allow for the inclusion of variable numbers of observations for each patient, an approach that helps prevent the loss of valuable information due to attrition. Because complete data are not required, as is typically the case in more traditional fixed-effects analytic models, the partial data collected from patients who drop out before completing an intervention or trial can be easily included in the final study models.

Second, time itself can be measured more precisely and in multiple ways within the HLM framework. HLMs permit the treatment of time as continuous, thereby allowing for assessments at unequal intervals that will often much more accurately represent the actual timing of measurements. For example, if patient interviews are conducted at baseline and at 3, 7, 14, 30, 180, and 365 days postintervention, these differential time intervals can be quickly and efficiently modeled within the HLM framework at these precise time points, rather than simply at times 1–2–3–4–5–6–7. Likewise, where data are abstracted from medical records at the time of unequally spaced patient visits, an HLM's ability to model time in a variable manner again proves invaluable. In addition, change can be measured other than at the patient level. Barrett et al. (28), for example, evaluated the effects of change at the census tract level on stage of breast cancer diagnosis using a HLM. HLMs allow investigators to assess patients at time points of variable spacing (29) as well as assess the effects of variables measured at other levels of analysis and also at variable time points. These features are shared with the other models outlined below.

A second, and related, approach to assessing patient data in multilevel models is latent curve modeling (LCM) (30,31). This approach is typically estimated within the structural equation modeling (SEM) framework (32). In LCM models, observed measurements at multiple time points are used to develop latent measures of construct slopes and intercepts. These can then be used to model change in individual growth or developmental trajectories as a function of individual and/or higher-level factors. A key advantage of this analytic strategy is its ability to address measurement error through the introduction of latent variables. In doing so, it is possible to distinguish actual variability from error variance, thereby estimating the true change in latent variables over time (33). LCM also easily permits assessments of mediating processes within a longitudinal framework (34). Another advantage is the ability of LCM to model and compare trajectories of change in latent variables across multiple groups. Stull (35) suggests that this feature makes LCM a particularly strong approach to analyzing clinical trial data.

Although the statistical tools necessary to assess MLIs for cancer prevention and treatment are now available, considerably more effort will be necessary to deploy them effectively. Considerations of the temporal design of these models should receive at least as much attention as do the more traditional elements of research design (12). The coding of time in hierarchical models, in particular, is not always straightforward, as multiple approaches can be taken. Biesanz et al. (36) provide a comprehensive overview of these strategies and consider some of the advantages and disadvantages of each.

Mixed-Method Approaches

Variations in the success of an intervention are likely to occur across different levels of influence. Furthermore, stakeholders beyond the researcher will determine program or intervention success differently. Eliciting stakeholder assessments and examining the various aspects of success necessitate a mixed-methods approach that combines qualitative and quantitative methods (37). A mixed-methods design can improve understanding of the effects of the intervention both within and across levels, as well as explain the variations in outcomes across the different levels.

Depending on the type of MLI, the addition of qualitative research can be invaluable in studying the processes of implementation to demonstrate the validity of a study's findings. Qualitative methods, such as interviews, case study, participant observation, record reviews, and/or focus groups, can adeptly elicit responses from patients, provider teams, clinical leadership, and community stakeholders regarding the duration, frequency, and timing of the intervention. Such rich data can help institutionalize an intervention within a health-care organization as well as engage stakeholders for a community-level intervention, thereby increasing buy-in and commitment to adopting and adapting the intervention to achieve the intended effect. An example is the ongoing evaluation of the National Cancer Institute Community Cancer Centers Program (NCCCP), a multimethod innovation intended to deliver quality cancer services where patients reside ( The evaluation of this pilot program examines program implementation and patient satisfaction at five levels: 1) national, through the network of awarded hospitals; 2) organizational, within the systems and hospitals through which the program is being implemented; 3) programmatic, for the impact on delivery of the cancer services; 4) individual, in terms of the impact on patients’ perceptions of the quality of care they are receiving within each participating hospital; and 5) with regard to each program component of the NCCCP. Mixed methods are used to assess program performance as well as effectiveness of the program model. The qualitative assessment uses case studies, site visit interviews, and patient focus groups, whereas the quantitative assessment includes a patient survey and an economic study. Changes are assessed at various time points across the pilot period, each year, and/or at 18-month intervals. The economic study draws upon the influences exerted on the program at the environmental level and reveals how hospitals have invested in the program through the leveraging of funds and cost allocations. Their persistent support in the midst of current economic challenges (a temporal issue) reveals their commitment to delivering this multilevel innovative program, and their effectiveness in organizational monitoring and adaptation. Findings from the study show that the fluctuations, historical and contextual, can significantly influence organizational decision making.

Resource Considerations and Trade-offs of Including Time as a Component of MLI Research

To conclude our discussion of time, we discuss three key resource-related issues in MLI research: analytic complexity, study duration, and theoretical development.

Analytic Complexity

As suggested by the preceding discussion, incorporating time and timing elements into MLI can place formidable data demands on the evaluation. Linear growth models are straightforward—when change in outcomes is linear over time, and such time elements are simple and easy to interpret when these assumptions are met. These models can adequately represent the growth process with a small number of measurement waves and are particularly useful when study duration is short. However, more complex growth models require more time, data, and analytic complexity.

Furthermore, a larger sample is typically required for MLIs that involve complex cross-level interactions, say between organizational-level factors and individual-level attributes. Introducing time into evaluations of MLIs adds to this complexity by increasing both the number and type of cross-level interactions. For example, more reliable estimates for individual growth models are thought to be obtained with a larger number of measurement waves (eg, eight or more). Interactions between higher-level intervention components and time may have to incorporate periodicity and nonlinear specifications to capture the shape of the growth curve. Similar interaction complexities and data demands increase if higher-level changes in organizations, communities, or policies are tracked over time to ensure fidelity and sustainability of the intervention.

Study Duration

MLI studies incorporating time as either a dimension of individuals or of higher-level entities, such as organizations, communities, or policies, are by definition longitudinal. With longer study periods come greater costs and logistical challenges, particularly as more measurement points are involved. In designing or evaluating MLIs, three or more measurement occasions that use a cohort of subjects individually matched over time are necessary, at a minimum, to allow estimation of individual growth trajectories (38). Aside from the need for more data in MLIs that incorporate time, attendant costs include attrition, loss to follow-up, and greater data collection and administrative burden on the sites carrying out the intervention. Although costs are generally greater for MLI research that incorporates a time dimension, there can be analytic benefits as well. For example, the power to estimate group-to-group variability and group-level effects are strongly dependent on the number of groups included in the analyses. Failure to observe significant group-to-group variability is a common occurrence in multilevel studies owing to small Ns at the group level but should not always be taken as an indication that groups can be ignored in the analyses. Including a time dimension to assess group or any higher-level factors will likely increase the statistical power of the analysis owing to the addition of multiple observations/measurement points for any given group, thereby effectively increasing the N.

Theoretical Development

Finally, despite the many advantages of incorporating a time dimension into MLI design and evaluation, such efforts present major challenges given the absence of robust theoretical frameworks to guide this work. It is doubtful that a “one size fits all” approach will work for all interventions, individual trajectory classes, or types of cancers. However, it seems imperative that a more refined understanding of the way the intervention works (derived from treatment theory) and the way in which different classes of individuals change over time are fundamental starting points for capturing the heterogeneity in samples of individuals exposed to MLIs in cancer prevention and treatment (39,40). This means that specifying growth trajectories or implementation approaches, for example, may need to assume an exploratory trial and error nature with all the inefficiencies and blind alleys that this entails. As theoretical development in these areas improves, inefficiencies will likely be reduced. However, in the near term, we can expect to incur significant costs as empiric work and theoretical development proceed unevenly.


The first authors, who were not federal employees, received a small stipend from the National Cancer Institute to prepare the manuscript for publication and travel expenses to participate in the meeting where this paper was discussed.


1. Lasser KE, Murillo J, Medlin E, et al. A multilevel intervention to promote colorectal cancer screening among community health center patients: results of a pilot study. BMC Fam Pract. 2009;10(37):37. [PMC free article] [PubMed]
2. Jones AR. Time to think: temporal considerations in nursing practice and research. J Adv Nurs. 2001;33(2):150–158. [PubMed]
3. Walsh K. Encounters, endings and temporality in psychiatric nursing. J Adv Nurs. 1997;25(3):485–491. [PubMed]
4. Brant JM, Beck S, Miaskowski C. Building dynamic models and theories to advance the science of symptom management research. J Adv Nurs. 2010;66(1):228–240. [PubMed]
5. Little TD, Schnabel KU, Baumert J. Modeling Longitudinal and Multilevel Data: Practical Issues, Applied Approaches, and Specific Examples. Mahwah, NJ: Erlabum; 2000.
6. Rasmussen DM, Elverdam B. Cancer survivors’ experience of time—time disruption and time appropriation. J Adv Nurs. 2007;57(6):614–622. [PubMed]
7. Henly SJ, Kallas KD, Klatt CM, et al. The notion of time in symptom experiences. Nurs Res. 2003;52(6):410–417. [PubMed]
8. Kelly MP, Stewart E, Morgan A, et al. A conceptual framework for public health: NICE's emerging approach. Public Health. 2009;123(1):e14–e20. [PubMed]
9. De Stavola BL, Nitsch D, Silva ID, et al. Statistical issues in life course epidemiology. Am J Epidemiol. 2006;163(1):84–96. [PubMed]
10. Bowen ME. Coronary heart disease from a life-course approach: findings from the health and retirement study. J Aging Health. 2010;22(2):219–241. [PubMed]
11. Benn J, Burnett S, Parand A, et al. Studying large-scale programmes to improve patient safety in whole care systems: challenges for research. Soc Sci Med. 2009;69(12):1767–1776. [PubMed]
12. Collins LM, Graham JW. The effect of the timing and spacing of observations in longitudinal studies of tobacco and other drug use: temporal design considerations. Drug Alcohol Depend. 2002;68(suppl 1):S85–S96. [PubMed]
13. Andrykowski MA, Donovan KA, Jacobsen PB. Magnitude and correlates of response shift in fatigue ratings in women undergoing adjuvant therapy for breast cancer. J Pain Symptom Manage. 2009;37(3):341–351. [PMC free article] [PubMed]
14. D’Avolio LW. Electronic medical records at a crossroads: impetus for change or missed opportunity? JAMA. 2009;302(10):1109–1111. [PubMed]
15. Seltzer M, Choi K, Thum YM. Examining relationships between where students start and how rapidly they progress: using new developments in growth modeling to gain insight into the distribution of achievement within school. Educ Eval Policy Anal. 2003;25(3):263–286.
16. Dumas JE, Lynch AM, Laughlin JE, et al. Promoting implementation fidelity: conceptual issues, methods, and preliminary results from the Early Alliance prevention trial. Am J Prev Med. 2001;20(suppl 1):38–47. [PubMed]
17. Mowbray CT, Holter MC, Teague GB, et al. Fidelity criteria: development, measurement and validation. Am J Eval. 2003;24(3):315–340.
18. Rossi PH, Lipsey MW, Freeman HE. Evaluation: A Systematic Approach. 7th ed. Thousand Oaks, CA: Sage Publications; 2004.
19. Schensul JJ. Community, culture, and sustainability in multilevel dynamic systems intervention science. Am J Community Psychol. 2009;43(3–4):241–256. [PubMed]
20. Wilson KD, Kurz RS. Bridging implementation and institutionalization within organizations: proposed employment of continuous quality improvement to further dissemination. J Public Health Manag Pract. 2008;14(2):109–116. [PubMed]
21. Riley BL, MacDonald J, Mansi O, et al. Is reporting on interventions a weak link in understanding how and why they work? A preliminary exploration using community heart health exemplars. Implement Sci. 2008;3(27):1–12. [PMC free article] [PubMed]
22. Helfrich CD, Weiner BJ, McKinney MM, et al. Determinants of implementation effectiveness: adapting a framework for complex innovations. Med Care Res Rev. 2007;64(3):279–303. [PubMed]
23. Greenhalgh T, Robert G, Macfarlane F, et al. Diffusion of innovations in service organizations: systematic review and recommendations. Milbank Q. 2004;82(4):581–629. [PubMed]
24. Look Hong NJ, Gagliardi AR, Bronskill SE, et al. Multidisciplinary cancer conferences: exploring obstacles and facilitators to their implementation. J Oncol Pract. 2010;6(2):61–68. [PMC free article] [PubMed]
25. Raudenbush SW, Bryk AS. Hiearchical Linear Models: Applications and Data Analysis Methods. 2nd ed. Thousand Oaks, CA: Sage Publications; 2002.
26. Hedeker D, Gibbons RD. Longitudinal Data Analysis. New York, NY: Wiley; 2006.
27. Goldstein H. Multilevel Statistical Models. 2nd ed. New York, NY: Halstead Press; 1995.
28. Barrett RE, Cho YI, Weaver KE, et al. Neighborhood change and distant metastasis at diagnosis of breast cancer. Ann Epidemiol. 2008;18(1):43–47. [PubMed]
29. Kwok OM, Underhill AT, Berry JW, et al. Analyzing longitudinal data with multilevel models: an example with individuals living with lower extremity intra-articular fractures. Rehabil Psychol. 2008;53(3):370–386. [PMC free article] [PubMed]
30. Bollen KA, Curran PJ. Latent Curve Models: A Structural Equation Perspective. New York, NY: Wiley; 2006.
31. Duncan EE, Duncan SC, Strycker LA, et al. An Introduction to Latent Variable Growth Curve Modeling. Mahwah, NJ: Erlabum; 1999.
32. Hox JJ. Multilevel analyses of grouped and longitudinal data. In: Little TD, Schnabel KU, Baumert J, editors. Modeling Longitudinal and Multilevel Data: Practical Issues, Applied Approaches, and Specific Examples. Mahwah, NJ: Erlabum; 2000. pp. 15–32.
33. Anstey KJ, Hofer SM. Longitudinal designs, methods and analysis in psychiatric research. Aust N Z J Psychiatry. 2004;38(3):93–104. [PubMed]
34. Roesch SC, Norman GJ, Adams MA, et al. Latent growth curve modeling of adolescent physical activity: testing parallel process and mediation models. J Health Psychol. 2010;14(2):313–325. [PubMed]
35. Stull DE. Analyzing growth and change: latent variable growth curve modeling with an application to clinical trials. Qual Life Res. 2007;17(1):47–59. [PubMed]
36. Biesanz JC, Deeb-Sossa N, Papadakis AA, et al. The role of coding time in estimating and interpreting growth curve models. Psychol Methods. 2004;9(1):30–52. [PubMed]
37. Nastasi BK, Hitchcock J. Challenges of evaluating multilevel interventions. Am J Community Psychol. 2009;43(3–4):340–376. [PubMed]
38. Raudenbush SW. Comparing personal trajectories and drawing causal inferences from longitudinal data. Annu Rev Psychol. 2001;52:501–525. [PubMed]
39. Bickman L. Using program theory in evaluation. In: Bickman L, editor. Using Program Theory in Evaluation: New Directions for Program Evaluation, #33. San Francisco, CA: Jossey-Bass; 1987. pp. 5–18.
40. Muthén B, Muthén L. The development of heavy drinking and alcohol-related problems from ages 18 to 37 in a U.S. national sample. J Stud Alcohol. 2000;61(2):290–300. [PubMed]

Articles from Journal of the National Cancer Institute. Monographs are provided here courtesy of Oxford University Press