PMCCPMCCPMCC

Search tips
Search criteria 

Advanced

 
Logo of nihpaAbout Author manuscriptsSubmit a manuscriptHHS Public Access; Author Manuscript; Accepted for publication in peer reviewed journal;
 
Am J Prev Med. Author manuscript; available in PMC 2018 January 1.
Published in final edited form as:
PMCID: PMC5571824
NIHMSID: NIHMS895056

Multilevel Interventions Targeting Obesity: Research Recommendations for Vulnerable Populations

Abstract

Introduction

The origins of obesity are complex and multifaceted. To be successful, an intervention aiming to prevent or treat obesity may need to address multiple layers of biological, social, and environmental influences.

Methods

NIH recognizes the importance of identifying effective strategies to combat obesity, particularly in high-risk and disadvantaged populations with heightened susceptibility to obesity and subsequent metabolic sequelae. To move this work forward, the National Heart, Lung, and Blood Institute, in collaboration with the NIH Office of Behavioral and Social Science Research and NIH Office of Disease Prevention convened a working group to inform research on multilevel obesity interventions in vulnerable populations. The working group reviewed relevant aspects of intervention planning, recruitment, retention, implementation, evaluation, and analysis, and then made recommendations.

Results

Recruitment and retention techniques used in multilevel research must be culturally appropriate and suited to both individuals and organizations. Adequate time and resources for preliminary work are essential. Collaborative projects can benefit from complementary areas of expertise and shared investigations rigorously pretesting specific aspects of approaches. Study designs need to accommodate the social and environmental levels under study, and include appropriate attention given to statistical power. Projects should monitor implementation in the multiple venues and include a priori estimation of the magnitude of change expected within and across levels.

Conclusions

The complexity and challenges of delivering interventions at several levels of the social—ecologic model require careful planning and implementation, but hold promise for successful reduction of obesity in vulnerable populations.

INTRODUCTION

Obesity is a major threat to public health because it increases risk for several chronic diseases, including cardiovascular disease, which is the leading cause of death in the U.S. Obesity and its adverse effects on health occur disproportionately among ethnic minorities and those with low SES, individuals living in rural areas, and some immigrant populations.13 High-risk and vulnerable populations require intensification of efforts to reduce obesity to maintain a healthy public.

The importance of research to identify strategies to combat chronic conditions, including obesity in high-risk groups, has long been recognized by NIH. In 2010, in response to a strategic priority at NIH, the National Cancer Institute collaborated with the National Heart, Lung, and Blood Institute to support the Centers for Population Health and Health Disparities Program to better address inequities in health among underserved racial, ethnic, and poor populations, with a focus on utilizing and evaluating multilevel interventions.4 Recommendations from that program highlighted the need to address underlying social and economic barriers to healthy choices and collaboration with local health delivery systems and safety net providers to improve healthcare access and quality for underserved populations. Also endorsed was use of community-based participatory research principles to improve population health outcomes, and transdisciplinary collaborations. 57

Building upon these efforts, on September 15–16, 2015, the National Heart, Lung, and Blood Institute, in collaboration with the NIH Office of Disease Prevention and the NIH Office of Behavioral and Social Science Research, convened a working group to inform future research directions for multilevel interventions in high-risk and vulnerable populations, to encourage and enrich public health action. The group was composed of pediatricians, family medicine and community health educators, nurses, epidemiologists, biostatisticians, behavioral scientists, psychologists, and project coordinators who are experts in obesity, health promotion, behavior change, multilevel and systems intervention research design, nutrition, and physical activity. The purpose of the meeting was to discuss critical issues in the design and implementation of multilevel intervention research targeting hard-to-reach, high-risk, or vulnerable populations and communities.

This report summarizes the presentations and recommendations made at the meeting and additional discussions that were initiated by those interactions. It is intended to be of use to researchers, funders, and policymakers as they consider future research on the control of obesity in vulnerable populations. It begins by expanding on the meaning and contributions to be made by multilevel interventions, with particular attention to vulnerable populations. It then reviews relevant aspects formative work, intervention planning, recruitment, retention, implementation, evaluation, and analysis. It also gives suggestions for future research.

MULTILEVEL INTERVENTIONS

Multiple factors influence obesity risk, including individual behavioral factors, social and physical environments, and state and local policies. Multilevel interventions include change strategies in two or more levels of influence in the social—ecologic model (Figure 1).810 Interventions with multiple components that address only one level of influence, such as programs that modify aspects of both the food and physical activity environments in schools, are multicomponent, but not multilevel. Multilevel interventions are delivered in more than one setting and researchers are usually (but not always) required to intervene on different people or sets of people in each setting to affect change.

Figure 1
Multiple levels and systems pertinent to obesity. Multiple levels: defined as two or more levels within the ecological system (adapted from Pratt et al., 201310).

In the ongoing Childhood Obesity Prevention and Treatment Consortium, four studies are testing the use of multilevel interventions for the prevention and treatment of obesity in high-risk, vulnerable populations.10 For example, the Stanford GOALS study is an RCT that intervenes on overweight and obese children of Hispanic descent at three levels.11 The intervention includes a sports program delivered in community centers, environmental and behavioral changes in the home, and enhanced behavioral counseling delivered by primary care practitioners in their offices.

HIGH-RISK, VULNERABLE POPULATIONS

In spite of efforts from multiple stakeholders in recent decades, the high prevalence of obesity in the U.S. has shown little improvement.3,12,13 Population segments of deep concern include children, low-income individuals, racial/ethnic minority groups, rural populations, and adults aged >65 years.3,1215 Predisposing factors for excess weight include biological, behavioral, cultural, environmental, political, sociodemographic (income, gender, race/ethnicity, age), and life stage.1,1623 There is a need to better understand the multitude of complex and context-specific factors that interact to enhance vulnerability.24

A group recently recognized as being at high risk for developing obesity is recent immigrants, and studies indicate that dietary quality and physical activity patterns often decline soon after entry into the U.S.1,2 Reasons for this decline may be environmental, such as places where immigrants settle (poor, urban food desert, high convenience food, high crime communities), which then predispose individuals to developing poor dietary habits and physical inactivity.20,21,23 Furthermore, immigrant and low-income status each amplify barriers such as language constraints, limited access to resources, and health beliefs, which may contribute to behaviors that lead to obesity. These sources of vulnerability make recent immigrants an important group to consider for new research.

FORMATIVE WORK

Understanding the determinants of health behaviors in the specific population of interest is crucial to the design of feasible, acceptable, and effective interventions. Ideally formative research would be conducted at multiple levels of influence in preparation for intervention planning, and be used to identify which determinants are: most strongly related to obesity risk in the population of interest; most susceptible to change; and mutable within the scope of the intervention, considering length of time, cost, and expertise, and other resources needed to conduct the intervention.25,26 Formative work can be both qualitative and quantitative and use existing as well as newly collected data. Examples of formative work in obesity multilevel intervention trials include collaboration with community advisory groups and conducting small, iterative pilot studies to refine interventions aimed at community, individual, and family levels.11,27,28

Recommendations

  • Designate adequate time to conduct and analyze both qualitative and quantitative formative assessment at multiple levels to understand contextual factors relevant to the population under study.
  • Engage and obtain the perspectives of members of the target population and a wide variety of stakeholders early in the process.
  • Use multidisciplinary approaches to examine influences at multiple levels.
  • Continue to collect formative data as the intervention is being developed and implemented to understand issues related to dissemination.

PRELIMINARY WORK AND INTERVENTION PLANNING

Current conceptual models suggest the role of complex systems in obesity etiologies, prevention, and control.29 Thus, the addition of systems thinking to multilevel intervention design may help to increase impact.30,31 Systems models are distinguished from multilevel models by including, for example, dynamic interactions between and across levels and components, feedback loops and tipping points that may magnify or attenuate responses, and consideration of non-linear changes and multiple causal pathways. Interventions can be designed to incorporate these same complex systems features to prompt interactions and synergies between multiple components across multiple levels and settings.29 Similarly, careful consideration of specific behavior change strategies planned for the intervention may increase its effectiveness.32,33

Perry’s planning model34 and Intervention Mapping35 are examples of guides that can be useful in intervention design. Such approaches could be used to produce a specific, standardized, but flexible multilevel intervention guide for obesity prevention in vulnerable populations. A standard intervention planning process could be applied to determinants and the levels of influence relevant to a specific group. Thus, different interventions could be developed to reflect the determinants and capacities of the specific targeted population, but the process would be generalizable and reproducible.

A consortium of investigators could conduct preliminary or evidentiary studies36 that emanate from a defined, but flexible, intervention planning process. Sharing of results could accelerate the process of finalizing the specific intervention strategies to be used. Within that work, triggers and tailored therapies could be defined for use as elements of adaptive interventions using the Multiphasic Optimization Strategy framework, including designs such as a sequential multiple assignment randomized trial (SMART).3739 The magnitude of change expected within levels and in the ultimate target (usually at the individual level) should be estimated a priori so that expectations and goals are well communicated and realistic.

Recommendations

  • Include intervention strategies that target determinants at multiple levels of the ecologic model.
  • Create linkages among the intervention levels and optimize the likelihood of synergy.
  • Consider the use of triggers to adapt or tailor interventions at multiple levels.
  • Incorporate systems thinking into the design and evaluation of multilevel interventions to try to anticipate, capture, and better understand the complexity of interacting exposures and responses.
  • Consider a priori decomposition of intervention strategies using a taxonomy such as that proposed by Michie et al.33,40 to describe and evaluate the anticipated approach.
  • Conduct appropriately powered evidentiary studies that target modest (relatively easily influenced) outcomes at several levels prior to undertaking the full trial to help select effective approaches.

RECRUITMENT AND RETENTION

For multilevel interventions, it is helpful to contact individuals in leadership positions within targeted organizations. Community groups (e.g., parks and recreation), healthcare systems (clinics, hospitals), and educational establishments (e.g., Head Start, schools) provide natural partners in these types of efforts. Recruitment of these organizations may require unique strategies and incentives, and often their early involvement in the planning processes is important.

To reach largely underserved populations, recruitment efforts need to consider locations where these populations tend to be, processes that allow for a trusted introduction to the recruiter and the research study, and materials in preferred language with attention to health literacy.41 In a systematic review of recruitment of underserved populations into research, social marketing and referral recruitment were found to be most successful.42 A potentially cost-effective recruitment strategy also includes recruiting from within ongoing cohort studies. Often, trust has already been established between the researchers and participants, and substudies can be effectively launched through this approach.43

Retention of both participants and organizations is critical to successful multilevel research. Retention efforts that maximize the building of trusting relationships with research staff can be particularly effective. Some studies have applied a “case management” approach, allowing for consistent bidirectional communication between participants and a research-staff point person.41,44 Frequent updating of contact information is critical, given that many participants change phone carriers and addresses throughout the course of the study, and organizations experience changes in personnel.

Recommendations

  • Define the population to be studied to facilitate the ability to focus on culturally important issues.
  • Cultivate relationships with key individuals and organizations at every intervention level and involve them in the planning process when possible.
  • Develop metrics for the study team to review on a regular basis, assessing objective progress and identifying what is working and what is not, in real time.
  • Plan ahead to maintain contact with study participants.

INTERVENTION IMPLEMENTATION

Because multilevel interventions often include a focus on the social and physical environments, it is useful to engage community members in intervention development and implementation. One resource that can be tapped for this service is the community health worker (CHW). CHW is an umbrella term describing community members who assist individuals and communities to adopt healthy behaviors. They usually share ethnicity, language, SES, and life experiences with the community members they serve. They have training that is recognized by health services and in some states require certification. Researchers should triage and amplify training for CHWs who show aptitude for work with community organizations to increase advocacy for the mobilization of resources and to work with policy-makers as community representatives. A review by the Centers for Disease Control and Prevention Task Force on Community Preventive Services has demonstrated sufficient evidence of effectiveness for CHWs to improve outcomes for blood pressure, cholesterol, physical activity, and healthful eating habits in patients at increased risk for cardiovascular disease.45

Process evaluation at all levels of the intervention is essential to try to understand how implementation of the intervention may be related to its effectiveness, acceptability, and dissemination potential.4650 Equally important is the need to track intervention adherence, dose (delivered and received), and fidelity.46,5153 For intervention trials that are being conducted by multiple investigative teams (such as in a multicenter trial), the ability to identify common process measures allows intervention implementation to be examined both within and across trials, thereby increasing the richness of the data.25,54

To enhance intervention effectiveness, baseline and interim measures can be collected and used as triggers for adaptations throughout the study conduct.37,55 Triggers and adaptations can be instituted at one or more levels of influence, or across levels of influence. Administered correctly, preplanned alterations in the intervention can amplify effectiveness without violating study blinding, even when the trigger is the primary outcome variable. 55,56 In addition, these preplanned changes do not inhibit the potential to disseminate an intervention, as do ad hoc, unplanned changes that are often poorly described, difficult to emulate, and constructed such that they may only be potent in the specific sample under study.

Recommendations

  • Plan for adequate resources to maintain intervention delivery at each level.
  • Consider using CHWs to engage community members and plan to provide training to enhance their role in outreach, enrollment, and intervention.
  • Track intervention participation and fidelity across settings using prespecified criteria for success, including minimal dose (delivered and received) at each level.
  • Monitor response to interventions at multiple levels and prespecify criteria for adapting the interventions to improve overall intervention effectiveness.

EVALUATION

For an obesity-related intervention, the primary evaluation of effect is usually at the individual level. However, careful thought should be given to assessment of the intervention at multiple levels. Investigators may want to use a composite variable to evaluate specific or multiple components of the intervention, as it can give insight into the achievement of a set of goals, rather than examining goals one by one. Although a global evaluation has value, composite variables can result in loss of information and be difficult to interpret.57,58 Investigators need to balance these factors and be careful to insure proper interpretation.

Recommendations

  • Collect key process and outcome data to evaluate the implementation and effectiveness of each level of the intervention.
  • Consider the use of composite variables.

STUDY DESIGN AND ANALYSIS

Multilevel interventions pose special challenges for design and analysis because they address multiple levels of influence (Figure 1). Individuals may be independent at one level but have some connection with each other or with a common change agent at another level. This is obvious in trials that randomize groups or clusters,5962 but it also occurs in trials that randomize individual participants but deliver interventions to real or virtual groups.60,6369 Such connections create the expectation for positive intraclass correlation among observations taken on participants in the same group or cluster.70 It will be difficult to avoid this problem, as investigators would have to avoid any connection among participants across the hierarchy of levels for the duration of the trial.

Positive intraclass correlation will invalidate the usual analytic procedures.5962,71 In addition, if the number of groups or clusters is limited, the df available for the test of the intervention effect will also be limited,71 which in turn will limit the power for that test.59,60,71 Finally, simple randomization of a limited number of groups or clusters to study conditions may not be enough to evenly distribute all potential sources of confounding,59,60 so special attention must be given to control confounding in the analysis. The available design and analytic options vary substantially in the way that they address these issues.

Recommended designs have some form of randomization or complete modeling of the assignment rule to protect against confounding. Examples of the former include group- or cluster-randomized trials (random assignment of identifiable groups to study conditions with observations on group members),5962 individually randomized group treatment trials (random assignment of individuals to study conditions with delivery of some intervention components in small groups or through a common change agent),60,6369 stepped wedge designs (sequential randomization of all groups to intervention, providing both intervention and comparison observations on all groups),7276 and RCTs (randomization of individuals with no interaction with each other or with a common change agent post-randomization).77 The recommended non-randomized design relies on complete modeling of the assignment rule (regression discontinuity designs: those scoring on one side of a quantitative cut point receive the treatment, whereas those on the other side are controls, with proper modeling of the relationship between the assignment score and the outcome in the analysis).7888 Designs that do not have either of these features do not provide the same strength of evidence for causal inference.79,8991 Even so, if none of the recommended designs can be used, multiple baseline designs (sequential assignment of a small number of groups to treatment, providing both intervention and comparison observations on all groups),90,92 quasi-experimental designs (all the features of RCTs except randomization), 79,91 and time series designs (many observations before and after introduction of the intervention with analysis of change associated with the intervention) 79,91,93 may be helpful. SMART designs (randomization to dose or type of intervention component based on response to prior intervention)38,9497 have been suggested as a way to refine interventions in preparation for a larger, standard randomized trial. SMART designs are well suited to assist researchers in the determination of triggers and intervention alternatives for use in a planned adaptive intervention.94

There are many examples of RCTs,98 group- or cluster-randomized trials,99 and individually randomized group treatment trials10 to evaluate obesity interventions. The authors identified one example of a stepped wedge design,100 three examples of SMART designs,37,101,102 and three examples of regression discontinuity designs103105 applied to evaluate an obesity intervention. As such, it is clear that all of the recommended designs have applications in obesity intervention research. A recently introduced adaptation of the SMART design, the SMART with adaptive randomization design (adapts randomization probabilities to favor intervention sequences that appear to be superior)106 may find future use in obesity interventions but, to the authors’ knowledge, has not yet been applied.

Recommendations

  • If connections among participants or with a common change agent are expected, employ group- or cluster-randomized trials, individually randomized group treatment trials, stepped wedge designs, or regression discontinuity designs that allocate groups or clusters to study conditions. If not, employ traditional RCTs.
  • Consider SMART or SMART with adaptive randomization designs if treatment response heterogeneity is expected.
  • Consider factorial and repeated-measures designs in the context of group- or cluster-randomized trials and RCTs to increase the information provided by, or the efficiency of, the trial.
  • Reflect the design in the analytic plan. There are a number of approaches that can provide a valid analysis, including mixed-model regression methods, randomization tests, fixed-effects modeling in two stages, and analyses based on generalized estimating equations, often with a small sample correction.63,107,108
  • Estimate expected effect sizes and power conservatively, using clinical trials data—based estimates of variance and intraclass correlation.
  • Consider tests of interactions, both within and between levels, at least as secondary analyses, even if power for those tests is limited.
  • Report results separately by gender and by race/ethnicity, even if there is limited power for such interaction effects, to support subsequent meta-analyses.

CONCLUSIONS

Although multilevel interventions have the potential to reduce obesity in vulnerable populations, they can be challenging, given the number and complexity of levels of influence requiring distinct interventions. Collaborative projects involving a group of investigators can benefit from an array of expertise pertinent to specific settings, as well as from larger sample size. Also, a consortium of studies can promote sharing of preliminary testing of intervention components applied in different settings. This type of testing is crucial, but often abbreviated by resources and circumstance. Acquired experiences could be used to inform a fully elaborated intervention in the format of a planning process that contains specific features and behavior change therapies, but is flexible to accommodate nuances of study settings and populations. The planning process used to construct different interventions tested at different sites would be amenable to dissemination owing to its flexibility.

Finally, a consortium, with common measurements enhanced and standardized by a coordinating center, can provide strong opportunities for samples studied in a randomized trial to be efficiently converted into a longitudinal cohort, thus leveraging value. Ideally, the plans for this conversion would be detailed prior to the collection of baseline trial data so that key variables can be included for longitudinal study. The questions to be addressed do not need to be limited to those expected to be impacted by the intervention, but could encompass a larger range of issues pertinent to the health of the population and organizations under study.

Acknowledgments

On September 15–16, 2015, the working group meeting was supported by the National Heart, Lung, and Blood Institute (NHLBI), and the NIH Offices of Disease Prevention and Behavioral and Social Science Research. The content is solely the responsibility of the authors and does not necessarily represent the official views of NIH or NHLBI.

Working Group Chair: June Stevens, PhD, University of North Carolina at Chapel Hill.

Working Group Members: Clifton Addison, PhD, Jackson Heart Study Community Outreach Center/Jackson State University; Jorge Armando Banda, PhD, Stanford University; Donna Antoine-LaVigne, PhD, Jackson Heart Study Community Outreach Center/Jackson State University; Shari Barkin, MD, MHS, Vanderbilt University School of Medicine; Elaine A. Borawski, PhD, Case Western Reserve University; Ying Kuen K. Cheung, PhD, Columbia University; Henry Feldman, PhD, Boston Children’s Hospital; Lawrence Green, DrPH, University of California, San Francisco; Heather Hardin, PhD, Case Western Reserve University; Sarah Jones, MS, RD, LD, Case Western Reserve University; Leslie Lytle, PhD, University of North Carolina at Chapel Hill; Donna Matheson, PhD, Stanford University; Michael Miner, PhD, University of Minnesota; Shirley M. Moore, RN, PhD, Case Western Reserve University; Judith Ottoson, PhD, San Francisco State University; Thomas N. Robinson, MD, Stanford University School of Medicine; Nancy E. Sherwood, PhD, Health-Partners Institute for Education and Research; Anna Lopez-Solerno, PhD, MS, Case Western Reserve University; Kimberly P. Truesdale, PhD, University of North Carolina at Chapel Hill; Lu Wang, PhD, University of Michigan; and Dianne S. Ward, PhD, University of North Carolina at Chapel Hill.

NIH Staff: Sonia Arteaga, PhD, Division of Cardiovascular Sciences, NHLBI; Rachel Ballard-Barbash, MD, MPH, Office of Disease Prevention, NIH; Josephine Boyington, PhD, MPH, Division of Cardiovascular Sciences, NHLBI; Susan Czajkowski, PhD, Division of Cardiovascular Sciences, NHLBI; Layla Esposito, PhD, Eunice Kennedy Shriver National Institute of Child Health and Human Development; Judy Hannah, PhD, National Institute on Aging, NIH; Christine Hunter, PhD, National Institute of Diabetes, Digestive, and Kidney Diseases, NIH; Chitra Krishnamurti, PhD, Center for Translational and Implementation Science, NHLBI; Maria Lopez-Class, PhD, MPH, MS, Center for Translation Research and Implementation Science, NHLBI; Michael Lauer, MD, Division of Cardiovascular Sciences, NHLBI; David M. Murray, PhD, Division of Program Coordination Planning and Strategic Initiatives, NIH; Cheryl Nelson, MSPH, Division of Cardiovascular Sciences, NHLBI; Holly Nicastro, PhD, Division of Cardiovascular Sciences, NHLBI; Emmanuel Peprah, PhD, Center for Translational and Implementation Science, NHLBI; Charlotte Pratt, PhD, MS, RD, Division of Cardiovascular Sciences, NHLBI; Dana Sampson, MS, MBA, Office of Behavioral and Social Science Research, NIH; and Marcel Salive, MD, MPH, Division of Geriatrics and Gerontology, National Institute of Aging.

No financial disclosures were reported by the authors of this paper.

References

1. Singh GK, Siahpush M, Hiatt RA, Timsina LR. Dramatic increases in obesity and overweight prevalence and body mass index among ethnic-immigrant and social class groups in the United States, 1976–2008. J Commun Health. 2011;36(1):94–110. http://dx.doi.org/10.1007/s10900-010-9287-9. [PMC free article] [PubMed]
2. Gordon-Larsen P, Harris KM, Ward DS, Popkin BM. Acculturation and overweight-related behaviors among Hispanic immigrants to the U.S.: the National Longitudinal Study of Adolescent Health. Soc Sci Med. 2003;57(11):2023–2034. http://dx.doi.org/10.1016/S0277-9536(03)00072-8. [PubMed]
3. Ogden CL, Carroll MD, Lawman HG, et al. Trends in obesity prevalence among children and adolescents in the United States, 1988–1994 through 2013–2014. JAMA. 2016;315(21):2292–2299. http://dx.doi.org/10.1001/jama.2016.6361. [PubMed]
4. Warnecke RB, Oh A, Breen N, et al. Approaching health disparities from a population perspective: the National Institutes of Health Centers for Population Health and Health Disparities. Am J Public Health. 2008;98(9):1608–1615. http://dx.doi.org/10.2105/AJPH.2006.102525. [PubMed]
5. Clauser SB, Taplin SH, Foster MK, Fagan P, Kaluzny AD. Multilevel intervention research: lessons learned and pathways forward. J Natl Cancer Inst Monogr. 2012;2012(44):127–133. http://dx.doi.org/10.1093/jncimonographs/lgs019. [PMC free article] [PubMed]
6. Gorin SS, Badr H, Krebs P, Prabhu Das I. Multilevel interventions and racial/ethnic health disparities. J Natl Cancer Inst Monogr. 2012;2012(44):100–111. http://dx.doi.org/10.1093/jncimonographs/lgs015. [PMC free article] [PubMed]
7. NIH. [Accessed February 16, 2016];NIH-Supported Centers for Population Health and Health Disparities (CPHHD) (P50) http://grants.nih.gov/grants/guide/rfa-files/RFA-CA-09-001.html.
8. Pratt CA, Arteaga S, Loria C. Forging a future of better cardiovascular health: addressing childhood obesity. J Am Coll Cardiol. 2014;63(4):369–371. http://dx.doi.org/10.1016/j.jacc.2013.07.088. [PubMed]
9. Story M, Kaphingst KM, Robinson-O’Brien R, Glanz K. Creating healthy food and eating environments: policy and environmental approaches. Annu Rev Publ Health. 2008;29:253–272. http://dx.doi.org/10.1146/annurev.publhealth.29.020907.090926. [PubMed]
10. Pratt CA, Boyington J, Esposito L, et al. Childhood Obesity Prevention and Treatment Research (COPTR): interventions addressing multiple influences in childhood and adolescent obesity. Contemp Clin Trials. 2013;36(2):406–413. http://dx.doi.org/10.1016/j.cct.2013.08.010. [PubMed]
11. Robinson TN, Matheson D, Desai M, et al. Family, community and clinic collaboration to treat overweight and obese children: Stanford GOALS—A randomized controlled trial of a three-year, multi-component, multi-level, multi-setting intervention. Contemp Clin Trials. 2013;36(2):421–435. http://dx.doi.org/10.1016/j.cct.2013.09.001. [PMC free article] [PubMed]
12. Flegal KM, Carroll MD, Ogden CL, Johnson CL. Prevalence and trends in obesity among U.S. adults, 1999–2000. JAMA. 2002;288(14):1723–1727. http://dx.doi.org/10.1001/jama.288.14.1723. [PubMed]
13. Ogden CL, Flegal KM, Carroll MD, Johnson CL. Prevalence and trends in overweight among U.S. children and adolescents, 1999–2000. JAMA. 2002;288(14):1728–1732. http://dx.doi.org/10.1001/jama.288.14.1728. [PubMed]
14. Fakhouri TH, Ogden CL, Carroll MD, Kit BK, Flegal KM. Prevalence of obesity among older adults in the United States, 2007–2010. NCHS Data Brief. 2012;(106):1–8. [PubMed]
15. Befort CA, Nazir N, Perri MG. Prevalence of obesity among adults from rural and urban areas of the United States: findings from NHANES (2005–2008) J Rural Health. 2012;28(4):392–397. http://dx.doi.org/10.1111/j.1748-0361.2012.00411.x. [PMC free article] [PubMed]
16. Clinical Guidelines on the Identification, Evaluation, and Treatment of Overweight and Obesity in Adults—The Evidence Report. National Institutes of Health. Obes Res. 1998;6(suppl 2):51s–209s. [PubMed]
17. Duffey KJ, Popkin BM. Energy density, portion size, and eating occasions: contributions to increased energy intake in the United States, 1977–2006. PLoS Med. 2011;8(6):e1001050. http://dx.doi.org/10.1371/journal.pmed.1001050. [PMC free article] [PubMed]
18. Powell LM, Nguyen BT. Fast-food and full-service restaurant consumption among children and adolescents: effect on energy, beverage, and nutrient intake. JAMA Pediatr. 2013;167(1):14–20. http://dx.doi.org/10.1001/jamapediatrics.2013.417. [PMC free article] [PubMed]
19. Piernas C, Popkin BM. Food portion patterns and trends among U.S. children and the relationship to total eating occasion size, 1977–2006. J Nutr. 2011;141(6):1159–1164. http://dx.doi.org/10.3945/jn.111.138727. [PubMed]
20. Sallis JF, Glanz K. Physical activity and food environments: solutions to the obesity epidemic. Milbank Q. 2009;87(1):123–154. http://dx.doi.org/10.1111/j.1468-0009.2009.00550.x. [PubMed]
21. Neckerman KM, Lovasi GS, Davies S, et al. Disparities in urban neighborhood conditions: evidence from GIS measures and field observation in New York City. J Public Health Policy. 2009;30(suppl 1):S264–S285. http://dx.doi.org/10.1057/jphp.2008.47. [PubMed]
22. Duncan DT, Johnson RM, Molnar BE, Azrael D. Association between neighborhood safety and overweight status among urban adolescents. BMC Public Health. 2009;9:289. http://dx.doi.org/10.1186/1471-2458-9-289. [PMC free article] [PubMed]
23. Taylor W, Lou D. Do all children have places to be active? Disparities in access to physical activity environments in racial and ethnic minority and lower-income communities. Princeton, NJ: Active Living Research, a National Program of the Robert Wood Johnson Foundation; Nov, 2011. [Accessed September 28, 2016]. http://activelivingresearch.org/sites/default/files/Synthesis_Taylor-Lou_Disparities_Nov2011_0.pdf.
24. Harvey JR, Ogden DE. Obesity treatment in disadvantaged population groups: where do we stand and what can we do? Prev Med. 2014;68:71–75. http://dx.doi.org/10.1016/j.ypmed.2014.05.015. [PMC free article] [PubMed]
25. Young DR, Johnson CC, Steckler A, et al. Data to action: using formative research to develop intervention programs to increase physical activity in adolescent girls. Health Educ Behav. 2006;33(1):97–111. http://dx.doi.org/10.1177/1090198105282444. [PMC free article] [PubMed]
26. Linde JA, Sevcik SM, Petrich CA, et al. Translating a health behavior change intervention for delivery to 2-year college students: the importance of formative research. Transl Behav Med. 2014;4(2):160–169. http://dx.doi.org/10.1007/s13142-013-0243-y. [PMC free article] [PubMed]
27. Moore SM, Borawski EA, Cuttler L, Ievers-Landis CE, Love TE. IMPACT: a multi-level family and school intervention targeting obesity in urban youth. Contemp Clin Trials. 2013;36(2):574–586. http://dx.doi.org/10.1016/j.cct.2013.08.009. [PMC free article] [PubMed]
28. Czajkowski SM, Powell LH, Adler N, et al. From ideas to efficacy: the ORBIT model for developing behavioral treatments for chronic diseases. Health Psychol. 2015;34(10):971–982. http://dx.doi.org/10.1037/hea0000161. [PMC free article] [PubMed]
29. UK Government Office for Science. Foresight. [Accessed September 28, 2016];Tackling obesities: future choices—project report. (2). www.gov.uk/government/uploads/system/uploads/attachment_data/file/287937/07-1184x-tackling-obesities-future-choices-report.pdf. Published October 2007.
30. Huang TT, Drewnosksi A, Kumanyika S, Glass TA. A systems-oriented multilevel framework for addressing obesity in the 21st century. Prev Chronic Dis. 2009;6(3):A82. [PMC free article] [PubMed]
31. Finegood DT, Merth TD, Rutter H. Implications of the foresight obesity system map for solutions to childhood obesity. Obesity (Silver Spring) 2010;18(suppl 1):S13–S16. http://dx.doi.org/10.1038/oby.2009.426. [PubMed]
32. Michie S, Wood CE, Johnston M, et al. Behaviour change techniques: the development and evaluation of a taxonomic method for reporting and describing behaviour change interventions (a suite of five studies involving consensus methods, randomised controlled trials and analysis of qualitative data) Health Technol Assess. 2015;19(99):1–188. http://dx.doi.org/10.3310/hta19990. [PubMed]
33. Michie S, van Stralen MM, West R. The behaviour change wheel: a new method for characterising and designing behaviour change interventions. Implement Sci. 2011;6:42. http://dx.doi.org/10.1186/1748-5908-6-42. [PMC free article] [PubMed]
34. Perry C. Creating Health Behavior Change: How to Develop Community-Wide Programs for Youth. Thousand Oaks, CA: Sage Publications; 1999.
35. Bartholomew LK, Markham C, Mullen P, Fernandez ME. Planning models for theory-based health-promotion interventions. In: Glanz K, Rimer B, Viswanath K, editors. Health Behavior: Theory, Research, and Practice. 5. San Franciso, CA: Jossey-Bass; 2015. pp. 359–388.
36. Stevens J, Taber DR, Murray DM, Ward DS. Advances and controversies in the design of obesity prevention trials. Obesity (Silver Spring) 2007;15(9):2163–2170. http://dx.doi.org/10.1038/oby.2007.257. [PubMed]
37. Almirall D, Nahum-Shani I, Sherwood NE, Murphy SA. Introduction to SMART designs for the development of adaptive interventions: with application to weight loss research. Transl Behav Med. 2014;4(3):260–274. http://dx.doi.org/10.1007/s13142-014-0265-0. [PMC free article] [PubMed]
38. Collins LM, Nahum-Shani I, Almirall D. Optimization of behavioral dynamic treatment regimens based on the sequential, multiple assignment, randomized trial (SMART) Clin Trials. 2014;11(4):426–434. http://dx.doi.org/10.1177/1740774514536795. [PMC free article] [PubMed]
39. Wyrick DL, Rulison KL, Fearnow-Kenney M, Milroy JJ, Collins LM. Moving beyond the treatment package approach to developing behavioral interventions: addressing questions that arose during an application of the Multiphase Optimization Strategy (MOST) Transl Behav Med. 2014;4(3):252–259. http://dx.doi.org/10.1007/s13142-013-0247-7. [PMC free article] [PubMed]
40. Michie S, Richardson M, Johnston M, et al. The behavior change technique taxonomy (v1) of 93 hierarchically clustered techniques: building an international consensus for the reporting of behavior change interventions. Ann Behav Med. 2013;46(1):81–95. http://dx.doi.org/10.1007/s12160-013-9486-6. [PubMed]
41. Guzman A, Richardson IM, Gesell S, Barkin SL. Recruitment and retention of Latino children in a lifestyle intervention. Am J Health Behav. 2009;33(5):581–586. http://dx.doi.org/10.5993/ajhb.33.5.11. [PubMed]
42. UyBico SJ, Pavel S, Gross CP. Recruiting vulnerable populations into research: a systematic review of recruitment interventions. J Gen Intern Med. 2007;22(6):852–863. http://dx.doi.org/10.1007/s11606-007-0126-3. [PMC free article] [PubMed]
43. Newman AB, Aviles-Santa ML, Anderson G, et al. Embedding clinical interventions into observational studies. Contemp Clin Trials. 2016;46:100–105. http://dx.doi.org/10.1016/j.cct.2015.11.017. [PMC free article] [PubMed]
44. Brueton VC, Tierney J, Stenning S, et al. Strategies to improve retention in randomised trials. Cochrane Database Syst Rev. 2013;12:MR000032. http://dx.doi.org/10.1002/14651858.mr000032.pub2. [PMC free article] [PubMed]
45. Task Force on Community Preventive Services. Using evidence to improve health outcomes. Atlanta, GA: Centers for Disease Control and Prevention; [Accessed September 28, 2016]. 2016 Annual Report to Congress, Federal Agencies and Prevention Stakeholders. www.thecommunityguide.org/annualreport/2016-congress-report-full.pdf.
46. Schneider M, Hall WJ, Hernandez AE, et al. Rationale, design and methods for process evaluation in the HEALTHY study. Int J Obes (Lond) 2009;33(suppl 4):S60–S67. http://dx.doi.org/10.1038/ijo.2009.118. [PMC free article] [PubMed]
47. Borrelli B, Sepinwall D, Ernst D, et al. A new tool to assess treatment fidelity and evaluation of treatment fidelity across 10 years of health behavior research. J Consult Clin Psychol. 2005;73(5):852–860. http://dx.doi.org/10.1037/0022-006X.73.5.852. [PubMed]
48. Young DR, Steckler A, Cohen S, et al. Process evaluation results from a school- and community-linked intervention: the Trial of Activity for Adolescent Girls (TAAG) Health Educ Res. 2008;23(6):976–986. http://dx.doi.org/10.1093/her/cyn029. [PubMed]
49. Steckler AB, Linnan L. Process Evaluation for Public Health Interventions and Research. 1. San Francisco, CA: Jossey-Bass; 2002.
50. Laska MN, Sevcik SM, Moe SG, et al. A 2-year young adult obesity prevention trial in the U.S.: process evaluation results. Health Promot Int. In press. Online June 30. 2015 http://dx.doi.org/10.1093/heapro/dav066. [PMC free article] [PubMed]
51. Tate DF, Lytle LA, Sherwood NE, Haire-Joshu D, et al. Deconstructing interventions: approaches to studying behavior change techniques across obesity interventions. Transl Behav Med. 2016;6(2):236–243. http://dx.doi.org/10.1007/s13142-015-0369-1. [PMC free article] [PubMed]
52. Baranowski T, Stables G. Process evaluations of the 5-a-day projects. Health Educ Behav. 2000;27(2):157–166. http://dx.doi.org/10.1177/109019810002700202. [PubMed]
53. Resnicow K, Davis M, Smith M, et al. How best to measure implementation of school health curricula: a comparison of three measures. Health Educ Res. 1998;13(2):239–250. http://dx.doi.org/10.1093/her/13.2.239. [PubMed]
54. Story M, Lytle LA, Birnbaum AS, Perry CL. Peer-led, school-based nutrition education for young adolescents: feasibility and process evaluation of the TEENS study. J Sch Health. 2002;72(3):121–127. http://dx.doi.org/10.1111/j.1746-1561.2002.tb06529.x. [PubMed]
55. Melnyk BM, Morrison-Beedy D, Moore SM. Nuts and bolts of designing intervention studies. In: Melnyk BM, Morrison-Beedy D, editors. Intervention Research: Designing, Conducting, Analyzing, and Funding. New York: Springer; 2012. pp. 37–64.
56. Collins LM, Murphy SA, Bierman KL. A conceptual framework for adaptive preventive interventions. Prev Sci. 2004;5(3):185–196. http://dx.doi.org/10.1023/B:PREV.0000037641.26017.00. [PMC free article] [PubMed]
57. Song MK, Lin FC, Ward SE, Fine JP. Composite variables: when and how. Nurs Res. 2013;62(1):45–49. http://dx.doi.org/10.1097/NNR.0b013e3182741948. [PMC free article] [PubMed]
58. Freemantle N, Calvert M, Wood J, Eastaugh J, Griffin C. Composite outcomes in randomized trials: greater precision but with greater uncertainty? JAMA. 2003;289(19):2554–2559. http://dx.doi.org/10.1001/jama.289.19.2554. [PubMed]
59. Murray DM. Design and Analysis of Group-Randomized Trials. New York: Oxford University Press; 1998.
60. Donner A, Klar N. Design and Analysis of Cluster Randomization Trials in Health Research. London: Arnold; 2000.
61. Hayes RJ, Moulton LH. Cluster Randomised Trials. Boca Raton, FL: Taylor & Francis Group, LLC; 2009. http://dx.doi.org/10.1201/9781584888178.
62. Campbell MJ, Walters SJ. How to Design, Analyse and Report Cluster Randomised Trials in Medicine and Health Related Research. Chichester: John Wiley & Sons Ltd; 2014. http://dx.doi.org/10.1002/9781118763452.
63. Pals SL, Murray DM, Alfano CM, et al. Individually randomized group treatment trials: a critical appraisal of frequently used design and analytic approaches. Am J Public Health. 2008;98(8):1418–1424. http://dx.doi.org/10.2105/AJPH.2007.127027. [PubMed]
64. Roberts C, Roberts SA. Design and analysis of clinical trials with clustering effects due to treatment. Clin Trials. 2005;2(2):152–162. http://dx.doi.org/10.1191/1740774505cn076oa. [PubMed]
65. Hoover DR. Clinical trials of behavioural interventions with heterogeneous teaching subgroup effects. Stat Med. 2002;21(10):1351–1364. http://dx.doi.org/10.1002/sim.1139. [PubMed]
66. Lee KJ, Thompson SG. Clustering by health professional in individually randomised trials. BMJ. 2005;330(7483):142–144. http://dx.doi.org/10.1136/bmj.330.7483.142. [PMC free article] [PubMed]
67. Heo M, Litwin AH, Blackstock O, Kim N, Arnsten JH. Sample size determinations for group-based randomized clinical trials with different levels of data hierarchy between experimental and control arms. Stat Methods Med Res. In press. Online August 14, 2014. [PubMed]
68. Baldwin SA, Bauer DJ, Stice E, Rohde P. Evaluating models for partially clustered designs. Psychol Methods. 2011;16(2):149–165. http://dx.doi.org/10.1037/a0023464. [PMC free article] [PubMed]
69. Kahan BC, Morris TP. Assessing potential sources of clustering in individually randomised trials. BMC Med Res Methodol. 2013;13:58. http://dx.doi.org/10.1186/1471-2288-13-58. [PMC free article] [PubMed]
70. Kish L. Survey Sampling. New York: John Wiley & Sons; 1965.
71. Cornfield J. Randomization by group: a formal analysis. Am J Epidemiol. 1978;108(2):100–102. [PubMed]
72. Brown CA, Lilford RJ. The stepped wedge trial design: a systematic review. BMC Med Res Methodol. 2006;6:54. http://dx.doi.org/10.1186/1471-2288-6-54. [PMC free article] [PubMed]
73. Hussey MA, Hughes JP. Design and analysis of stepped wedge cluster randomized trials. Contemp Clin Trials. 2007;28(2):182–191. http://dx.doi.org/10.1016/j.cct.2006.05.007. [PubMed]
74. Copas AJ, Lewis JJ, Thompson JA, et al. Designing a stepped wedge trial: three main designs, carry-over effects and randomisation approaches. Trials. 2015;16:352. http://dx.doi.org/10.1186/s13063-015-0842-7. [PMC free article] [PubMed]
75. Hemming K, Haines TP, Chilton PJ, Girling AJ, Lilford RJ. The stepped wedge cluster randomised trial: rationale, design, analysis, and reporting. BMJ. 2015;350:h391. http://dx.doi.org/10.1136/bmj.h391. [PubMed]
76. Hughes JP, Granston TS, Heagerty PJ. Current issues in the design and analysis of stepped wedge trials. Contemp Clin Trials. 2015;45(Pt A):55–60. http://dx.doi.org/10.1016/j.cct.2015.07.006. [PMC free article] [PubMed]
77. Meinert CL. Clinical Trials Handbook: Design, Conduct, and Analysis. 2. New York: Wiley; 2012. http://dx.doi.org/10.1002/9781118422878.
78. Thistlethwaite DL, Campbell DT. Regression-discontinuity analysis: an alternative to the ex-post facto experiment. J Educ Psychol. 1960;51(6):309–317. http://dx.doi.org/10.1037/h0044319.
79. Shadish WR, Cook TD, Campbell DT. Experimental and Quasi-Experimental Designs for Generalized Causal Inference. Boston, MA: Houghton Mifflin Company; 2002.
80. Imbens GW, Lemieux T. Regression discontinuity designs: a guide to practice. J Econ. 2008;142(2):615–635. http://dx.doi.org/10.1016/j.jeconom.2007.05.001.
81. Rubin DB. Assignment to treatment group on the basis of a covariate. J Educ Behav Stat. 1977;2(1):1–26. http://dx.doi.org/10.3102/10769986002001001.
82. Pennell ML, Hade EM, Murray DM, Rhoda DA. Cutoff designs for community-based intervention studies. Stat Med. 2011;30(15):1865–1882. http://dx.doi.org/10.1002/sim.4237. [PMC free article] [PubMed]
83. Bor J, Moscoe E, Mutevedzi P, Newell ML, Barnighausen T. Regression discontinuity designs in epidemiology: causal inference without randomized trials. Epidemiology. 2014;25(5):729–737. http://dx.doi.org/10.1097/EDE.0000000000000138. [PMC free article] [PubMed]
84. Bor J, Moscoe E, Barnighausen T. Three approaches to causal inference in regression discontinuity designs. Epidemiology. 2015;26(2):e28–e30. http://dx.doi.org/10.1097/EDE.0000000000000256. [PubMed]
85. Moscoe E, Bor J, Barnighausen T. Regression discontinuity designs are underutilized in medicine, epidemiology, and public health: a review of current and best practice. J Clin Epidemiol. 2015;68(2):122–133. http://dx.doi.org/10.1016/j.jclinepi.2014.06.021. [PubMed]
86. Venkataramani AS, Bor J, Jena AB. Regression discontinuity designs in healthcare research. BMJ. 2016;352:i1216. http://dx.doi.org/10.1136/bmj.i1216. [PubMed]
87. O’Keeffe AG, Geneletti S, Baio G, et al. Regression discontinuity designs: an approach to the evaluation of treatment efficacy in primary care using observational data. BMJ. 2014;349:g5293. http://dx.doi.org/10.1136/bmj.g5293. [PubMed]
88. Linden A, Adams JL. Combining the regression discontinuity design and propensity score-based weighting to improve causal inference in program evaluation. J Eval Clin Pract. 2012;18(2):317–325. http://dx.doi.org/10.1111/j.1365-2753.2011.01768.x. [PubMed]
89. Murray DM, Pennell M, Rhoda D, Hade EM, Paskett ED. Designing studies that would address the multilayered nature of health care. J Natl Cancer Inst Monogr. 2010;2010(40):90–96. http://dx.doi.org/10.1093/jncimonographs/lgq014. [PMC free article] [PubMed]
90. Rhoda DA, Murray DM, Andridge RR, Pennell ML, Hade EM. Studies with staggered starts: multiple baseline designs and group-randomized trials. Am J Public Health. 2011;101(11):2164–2169. http://dx.doi.org/10.2105/AJPH.2011.300264. [PubMed]
91. Cook TD, Campbell DT. Quasi-Experimentation: Design & Analysis Issues for Field Settings. Chicago, IL: Rand McNally College Publishing Company; 1979.
92. Hawkins NG, Sanson-Fisher RW, Shakeshaft A, D’Este C, Green LW. The multiple baseline design for evaluating population-based research. Am J Prev Med. 2007;33(2):162–168. http://dx.doi.org/10.1016/j.amepre.2007.03.020. [PubMed]
93. Biglan A, Ary D, Wagenaar AC. The value of interrupted time-series experiments for community intervention research. Prev Sci. 2000;1(1):31–49. http://dx.doi.org/10.1023/A:1010024016308. [PMC free article] [PubMed]
94. Collins LM, Murphy SA, Strecher V. The multiphase optimization strategy (MOST) and the sequential multiple assignment randomized trial (SMART): new methods for more potent eHealth interventions. Am J Prev Med. 2007;32(5 Suppl):S112–S118. http://dx.doi.org/10.1016/j.amepre.2007.01.022. [PMC free article] [PubMed]
95. Lavori PW, Dawson R. Introduction to dynamic treatment strategies and sequential multiple assignment randomization. Clin Trials. 2014;11(4):393–399. http://dx.doi.org/10.1177/1740774514527651. [PMC free article] [PubMed]
96. Huang X, Choi S, Wang L, Thall PF. Optimization of multi-stage dynamic treatment regimes utilizing accumulated data. Stat Med. 2015;34(26):3424–3443. http://dx.doi.org/10.1002/sim.6558. [PMC free article] [PubMed]
97. Ogbagaber SB, Karp J, Wahed AS. Design of sequentially randomized trials for testing adaptive treatment strategies. Stat Med. 2016;35(6):840–858. http://dx.doi.org/10.1002/sim.6747. [PMC free article] [PubMed]
98. Huseinovic E, Bertz F, Leu Agelii M, et al. Effectiveness of a weight loss intervention in postpartum women: results from a randomized controlled trial in primary health care. Am J Clin Nutr. 2016;104(2):362–370. http://dx.doi.org/10.3945/ajcn.116.135673. [PubMed]
99. Gans KM, Gorham G, Risica PM, et al. A multi-level intervention in subsidized housing sites to increase fruit and vegetable access and intake: rationale, design and methods of the ‘Live Well, Viva Bien’ cluster randomized trial. BMC Public Health. 2016;16:521. http://dx.doi.org/10.1186/s12889-016-3141-7. [PMC free article] [PubMed]
100. Yun L, Boles RE, Haemer MA, et al. A randomized, home-based, childhood obesity intervention delivered by patient navigators. BMC Public Health. 2015;15:506. http://dx.doi.org/10.1186/s12889-015-1833-z. [PMC free article] [PubMed]
101. Sherwood NE, Butryn ML, Forman EM, et al. The BestFIT trial: a SMART approach to developing individualized weight loss treatments. Contemp Clin Trials. 2016;47:209–216. http://dx.doi.org/10.1016/j.cct.2016.01.011. [PMC free article] [PubMed]
102. Naar-King S, Ellis DA, Idalski Carcone A, et al. Sequential Multiple Assignment Randomized Trial (SMART) to construct weight loss interventions for African American adolescents. J Clin Child Adolesc Psychol. 2016;45(4):428–441. http://dx.doi.org/10.1080/15374416.2014.971459. [PMC free article] [PubMed]
103. Hamad R, Cohen AK, Rehkopf DH. Changing national guidelines is not enough: the impact of 1990 IOM recommendations on gestational weight gain among U.S. women. Int J Obes (Lond) In press. Online June 21 2016. http://dx.doi.org/10.1038/ijo.2016.97. [PMC free article] [PubMed]
104. Almond D, Lee A, Schwartz AE. Impacts of classifying New York City students as overweight. Proc Natl Acad Sci U S A. 2016;113(13):3488–3491. http://dx.doi.org/10.1073/pnas.1518443113. [PubMed]
105. Andalon M. Oportunidades to reduce overweight and obesity in Mexico? Health Econ. 2011;20(suppl 1):1–18. http://dx.doi.org/10.1002/hec.1773. [PubMed]
106. Cheung YK, Chakraborty B, Davidson KW. Sequential multiple assignment randomized trial (SMART) with adaptive randomization for quality improvement in depression treatment program. Biometrics. 2015;71(2):450–459. http://dx.doi.org/10.1111/biom.12258. [PMC free article] [PubMed]
107. Murray DM, Varnell SP, Blitstein JL. Design and analysis of group-randomized trials: a review of recent methodological developments. Am J Public Health. 2004;94(3):423–432. http://dx.doi.org/10.2105/AJPH.94.3.423. [PubMed]
108. Li P, Redden DT. Small sample performance of bias-corrected sandwich estimators for cluster-randomized trials with binary outcomes. Stat Med. 2015;34(2):281–296. http://dx.doi.org/10.1002/sim.6344. [PMC free article] [PubMed]