Similarities across Domains
This review of reviews examined effective elements of adolescent health promotion programs in three behavioral domains – substance abuse, sexual behavior and nutrition. We specifically focused on similarities across these domains, and indeed, we identified many similarities. The results are discussed here in light of the two types of analysis that have been explained in the Methods section: an interpretation-based method and a rule-based method. Based on our interpretation-based examination of the evidence that is currently available from strong and moderate reviews, five elements were identified to be effective in all domains. These five elements have evidence from at least one strong review in each domain:
a) use of theory, with specific reference to social cognitive theory
b) addressing social influences, especially social norms
c) addressing cognitive-behavioral skills
d) training of facilitators
e) including multiple components.
When using the rule-based method of analysis, the results are similar: all five elements have at least tentative evidence in all domains. Elements b, c and d even have sufficient evidence in all domains; elements a and e have tentative evidence in one or two domains due to conflicting results between strong reviews in these domains (positive versus null or unclear results). Using the rule-based method, no other elements were identified as having sufficient evidence in all three domains, but two additional elements had at least tentative evidence for a positive contribution to effectiveness in each domain:
f) parent involvement
g) a larger number of sessions.
These two elements were not identified as similar across domains with the interpretation-based method of analysis, since we found the evidence in at least one domain to be inconclusive due to conflicting results between strong reviews; in the rule-based method such conflict leads to the conclusion that the evidence is tentative. The different results of the two methods of analysis for these two elements can thus be explained by the different approaches to handling conflicting results.
In addition to the above elements, which had evidence from strong or moderate reviews in each of the three domains, several other elements also tended to have similar results across the three domains, but their evidence involved only weak reviews in one or two domains. Although weak reviews were not included in the analysis, they were used for exploring whether there is any indication that a particular element might be effective in a particular domain. The following elements had similar results across all domains; domains with strong or moderate reviews are given between parentheses:
h) a focus on specific behavior (sexuality, nutrition)
i) addressing behavioral determinants (sexuality, nutrition)
j) a knowledge-only approach (ineffective element; substance abuse, nutrition)
k) use of interactive methods (substance abuse).
In addition to the above elements, the results for many other elements were comparable across at least two of the three domains. We did not find one element for which the results indicated opposing directions of influence between domains (e.g., a positive contribution to effectiveness in one domain and a negative contribution in another domain). In cases where the results were not similar across domains, this was usually because results in one or more domains were unclear or indicated null findings, whereas those in other domains indicated a positive contribution to effectiveness.
The results of the present review are fairly similar to those of other systematic reviews of reviews that examined the domains of substance abuse and sexuality separately and that included only high-quality reviews [19
], suggesting that the results for these domains are robust. This review adds rigor and specificity to the general observation in several reviews that effective elements in the domains of substance abuse and sexuality appear to be similar [e.g. [17
] and extends this observation to also include the nutrition domain. In contrast to the present review, these reviews did not examine the issue of similarity systematically or in detail.
Perhaps more importantly, our results are largely comparable to, and in some cases more specific than, those of a review of reviews that specifically focused on similarities across multiple domains [16
]. That review examined partly different domains (substance abuse, risky sexual behavior, school failure, and juvenile delinquency and violence), included a smaller and different set of reviews (35 narrative reviews that explicitly discussed common features of effective programs) and used a somewhat different review methodology (determining the percentage of reviews that mentioned an element as consistently effective). In that review [16
], nine elements of effective programs were identified, which were claimed to reflect general principles that transcend specific content areas. Seven of these elements coincide with the ones identified by us, although some tend to be formulated in more general terms than ours. These seven elements and, between brackets, the corresponding letters from our list, are: theory-driven [a]; socio-culturally relevant (address cultural norms and beliefs) [b, i]; varying teaching methods (skills-based component, active and interactive format) [c, k]; providing opportunities for positive relationships (parent-child communication, peer influences) [b, f]; well-trained staff [d]; comprehensive (multi-modal, multiple settings) [e]; and sufficient dosage [g]. Two of the elements they identified are not represented in our own set of eleven elements: appropriate timing and inclusion of outcome evaluation. The issue of outcome evaluation was not considered relevant for the present review, as it is an aspect of studies rather than programs. The issue of appropriate timing has to do with tuning interventions to student characteristics such as age, cognitive and social development and experience with the risk behavior. This issue is generally recommended in health promotion theory [82
], and indeed, tailoring to age was reported to be effective by strong sexuality [65
] and nutrition reviews [77
] in this paper. However, we did not include it in our empirically-based list of effective elements because in the substance abuse domain it was reported to be unclear due to absence of high-quality comparison studies [31
]. One element from our own list, a focus on specific behavior, is not represented in the list from the other review [16
]. Unfortunately, due to the limited reporting of results in that review [16
], we cannot examine the causes for this difference. Possibly, the issue of behavioral focus may only be relevant for certain domains or may have been overlooked in certain domains.
Implications for Practice
Researchers and practitioners in the three domains can use the effective elements identified in this review, and especially the ones that are similar across domains, as guidelines for developing and improving their adolescent health promotion programs. They can also look beyond the boundaries of their own domain to generate ideas for programs or research from results in other domains.
The fact that another multiple-domain review [16
] found comparable effective elements while examining partly different domains (also school failure, juvenile delinquency and violence) suggests that the effective elements may transcend broadly to other content areas. In fact, the effective elements pertaining to program development (use of theory, addressing determinants) appear to be applicable universally, as they are general recommendations from health promotion planning models and quality assurance procedures such as PRECEDE-PROCEED [82
], intervention mapping [83
] and Preffi [84
The finding that several effective elements are comparable across the three targeted domains indicates that integrative programs can address these domains with the same program characteristics. This is important in light of the recent interest in multiple health behavior research and its potential implications for integrative interventions [6
]. The results will be used for guiding the development of our own integrative program. The effective elements pertaining to program content – address not only information, but also social influences and cognitive-behavioral skills – fit well with those of a previous review that assessed similarities between behavioral determinants across the same three domains [10
]. In that review the following determinants were found to be important for all domains: attitudinal beliefs about immediate gratification and social advantages, social norms, modeling behavior and resistance skills. Together, both that review and the present one provide sufficiently valid input for the development of an integrative program that addresses all three domains.
Given our broad focus on several health-related behaviors and the already extensive body of knowledge in each domain, we applied a review-of-reviews approach, an approach that has gained acceptance in recent years [e.g., [16
]. Although the search strategy was comprehensive, it is possible that we may have missed relevant reviews. However, it is not very likely that these reviews would have discussed different sets of primary studies and would have led to different conclusions.
A limitation of the review-of-reviews approach is that it relies on 'second-hand' information and is potentially vulnerable to the interpretive and conceptual biases of previous reviewers [16
]. We attempted to limit these biases as much as possible by using a systematic review methodology, by assessing the quality and relevance of each review and relying on reviews of high to moderate quality, by carefully categorizing the results without generalizing too much, and, in case reviews had differential results, by attempting to examine the causes of the differences. We also attempted to check the results of reviews if sufficient information was provided.
Perhaps we would have identified more similarities across domains if we had combined aspects and findings into broader categories. We used a conservative categorization process and were reluctant to generalize findings, because the operationalization, interpretation or analysis of aspects seemed to differ between reviews or were sometimes unclear.
Two-thirds of the included reviews had a high quality score of 6 or 7. In line with other reviews of reviews [21
] we included only strong and moderate reviews in the analysis. Furthermore, we used two methods for analyzing the results and especially for dealing with conflicting results between reviews: one method focused on interpretation of differences and the other set a strict rule. The conclusions based on these two methods were fairly similar. Weak reviews were excluded from the analysis but were used in a speculative way: for elements that had evidence from strong or moderate reviews in at least one domain, the results of weak reviews in the other domains were used to give any indication or suggestion of effectiveness in these other domains.
The methodological aspects assessed in reviews most often pertained to study design, appropriateness of allocation procedures, comparability of groups, validity of assessment and attrition, but only a few reviewers examined additional aspects such as quality of implementation. The strictness of inclusion criteria and assessment of methodological quality varied widely, even among high-quality reviews. Although meta-analyses in several domains reported that effect sizes did not vary with the design or quality of studies [34
], reviews with the strictest methodological criteria (e.g., accepting only high-quality comparison studies) generally appeared to have more cautious conclusions than reviews with less strict criteria. Reporting the specific criteria applied by reviewers appears to be a valuable addition to the Quality Assessment Tool for Reviews. For reviews of primary studies, the Cochrane Collaboration Handbook [86
] and others [87
] recommend using the Quality Assessment Tool for Quantitative Studies, which is also developed by the Effective Public Health Practice Project, Canada.