Our examination of Leichsenring and Rabung's [2
] meta-analysis demonstrates how the failure to attend to important methodological issues can generate invalid conclusions. The most damning threat to the validity of findings in their meta-analysis was their gross miscalculation and aggregation of comparative effect sizes of included studies. Second, the meta-analysis was based on studies with small samples sizes, and overly heterogeneous comparative treatments, disorders and outcomes. Thus, one cannot know if LTPP is superior to other treatments only for certain disorders, but not for others, or for none. Given that treatment integrity was not well examined in most studies, one also cannot know if the delivery of treatments was executed in a competent manner and in the manner described in study treatment protocols. Finally, their conclusion was based on a sample of studies that failed to protect against multiple potential sources of bias, thus producing weak quality evidence.
Our assessment is consistent with Gibbons et al. [1
] who found the literature concerning dynamic therapy to be methodologically weak. Our observations also mirror other commentaries regarding the methodological limitations of meta-analyses [14
]. Despite the recognition of the importance of strategies to limit bias in the assembly and analysis of primary studies included in meta-analyses [18
], our review of Leichsenring and Rabung's [2
] study shows that meta-analytic techniques continue to be poorly implemented even in high-impact journals.
There are a number of caveats to our assessment. First, we assessed the risk of bias, rather than the true level of bias in included studies. Second, our assessment of the articles was based on what was reported rather than what was done in the research. Authors of articles were not contacted for information. It is possible that methodological procedures were omitted from the reports or were misrepresented. Third, although our critique covered an extensive range of potential methodological flaws affecting the meta-analysis, other factors were not addressed in this review, such as the reliability of measurements, criteria employed for selecting a study for inclusion into the meta-analysis, and effects of excluding relevant articles. In addition, we did not employ assessment tools such as AMSTAR [41
] to formally evaluate the reporting or methodological quality of the meta-analysis. Therefore, the quality of the meta-analysis as defined by these tools is yet to be investigated.
Our critique highlights the type of problems that can be inherent in meta-analyses and which need to be addressed by researchers employing this research strategy. As meta-analyses become more prominent in the field, specific attention should also be directed towards the quality of evidence, synthesis of the evidence, and statistical methods of aggregating effect sizes. As shown in our review, the failure to adequately attend to each of these potential problems reduces the scientific credibility of meta-analyses. Thus, the question of whether LTPP is more effective than shorter-term psychotherapy remains open. Answering that question would require more rigorous comparative trials than are currently available.