There are 2 ways in which secondary journals can make it easier for clinicians to use the medical literature to solve patient problems. First, they can report only relevant and methodologically strong articles. Second, they can use an abbreviated format to present key information. We assessed the performance of 3 secondary journals in the second of these roles. We specifically evaluated the reporting of information that is crucial for determining the validity of a study and for understanding the results sufficiently well to apply them to patient care.
With regard to RCTs, a number of studies have shown that inadequate concealment of patient allocation, such that those responsible for enrolling patients are aware of the arm to which the patients will be allocated if enrolled, may lead to overestimation of treatment effects,7,8,9,10
as may lack of blinding.7
Thus, concealment of patient allocation and blinding of participants, health care providers and those assessing outcomes are critical methodological factors that are necessary for assessing a study's validity. An intention-to-treat analysis includes participants in the group to which they were allocated, irrespective of whether they received the prescribed treatment. Failure to analyze by intention to treat defeats the purpose of randomization and may bias the results.11,12,13,14
Reporting the proportion of participants lost to follow-up is a final factor that we believe bears strongly on the likelihood studies will produce an unbiased estimate of the treatment effect.11,14
Of these determinants of validity, ACP reported the study design in all of the summaries, the blinding status of participants in 62%, the proportion of participants lost to follow-up in 46%, and other criteria in less than 40%. Where the ACP summaries differed from those in JW and IMA, the reporting in the latter 2 journals was less complete. The most significant omissions related to study design, which JW and IMA failed to specify in 28% of their summaries. Thus, all 3 secondary journals failed to provide much of the information readers need to assess study validity.
Because treatment decisions inevitably involve trade-offs between risks and benefits, clinicians require information not only about validity and whether treatment is effective, but also about the magnitude and precision of the estimate of the treatment effect. ACP provided some estimate of the magnitude of effect (relative risk reduction, risk difference or number needed to treat) in most of its summaries, whereas JW and IMA seldom provided these data. ACP always provided either a p value or confidence interval, information that JW omitted in its summaries and IMA provided in just over 50% of its summaries.
To our knowledge our study is the first to evaluate the quality of reporting in secondary journals. Previous work has demonstrated suboptimal reporting of RCTs in full-text journals.15,16,17
Those results, along with our findings, indicate the need for enhanced reporting of primary full-text articles and of secondary summaries of these studies.
For our analysis we assumed that the goal of secondary journals is to provide information that clinicians can apply directly to their clinical practice, thus enhancing their efficiency in using the original medical literature to guide patient care decisions. Secondary journals may have different goals. Indeed, JW includes a statement that its summaries are not intended for use as the sole basis for clinical treatment nor as a substitute for reading the original journal articles. These statements suggest that JW has different goals, such as simply alerting clinicians to information that they may want to explore further.
Although clinicians have not been surveyed on how they use secondary journals, we find it implausible that many use secondary journals largely as a stimulus to seek out and read original journal articles. For those who do use secondary journals in this way, having more informative summaries that identify the key methodological factors and results will aid in selecting which studies to spend time retrieving and reviewing. Finally, and most important, we would argue that, by not aiming to provide complete enough information to guide clinical practice directly, secondary journals are abandoning their most important potential role. At the same time, we acknowledge that clinicians' views of the optimal goals of secondary journals will differ with their values and are a matter for debate.
Whatever the goals of secondary journals, they should be able to provide brief and complete summaries. For example, consider the statement, “This concealed placebo-controlled RCT, with effective blinding of participants, health care providers and judicial assessors, 99% of participants available for follow-up and an intention-to-treat analysis, assessed the effect of amiodarone in patients with heart failure.” A second sentence could include all of the crucial data about the magnitude and precision of treatment effects.
Our study has limitations. We focused only on trials addressing therapeutics and prevention and therefore were unable to comment on the reporting of summaries that reviewed studies of diagnosis, prognosis or harm. Our evaluation of reporting methods and results focused on RCTs. Further work is needed to evaluate summaries of observational studies. The abstractors were aware of the secondary journal that published the summary they were assessing. However, the minimal interobserver variation in abstracting information suggests that it was unlikely that this lack of blinding influenced our findings. Finally, we evaluated only 3 secondary journals; however, we believe that most other secondary journals are no more likely than these 3 to report study design, methods and results. As such, our results may be widely generalizable.
One of us (G.H.G.) is an associate editor of ACP. Bearing in mind this possible conflict of interest, we were scrupulous in selecting criteria for optimal reporting and in conducting our assessment. Furthermore, we have presented a detailed rationale for our choice of factors that are crucial to allow clinicians to evaluate the validity and applicability of study results. Readers must decide whether our methods withstand the more intense scrutiny that is appropriate whenever issues of conflict of interest arise.
We acknowledge that brief summaries in secondary journals can never be substitutes for full-text reports, that detailed review of methodology will always raise additional issues and that, on occasion, those issues will have important implications for study validity and applicability. For instance, the term “intention-to-treat analysis” suffers from ambiguity and variability in interpretation. Nevertheless, few clinicians have either the time or the skills to conduct the detailed review required to elucidate such issues. Secondary journals, if they scrupulously report methodological details using the most transparent terms available, can provide summaries that, although not perfect, can serve clinicians well.
For those who believe that secondary journals should provide summaries that clinicians can apply directly to patient care, our results have a clear message. By implementing a systematic and easily achieved approach to reporting a small number of key features of methods and results, secondary journals could fulfil their potential to help clinicians deliver efficient, evidence-based care. For those who see other goals for secondary journals, our results are also important. First, they suggest the need for an explicit formulation of secondary journals' goals, and a debate about what the optimal goals might be. Second, they suggest the need for primary research on how, in an era of increasing pressures and time constraints, clinicians actually use information from secondary journals.