|Home | About | Journals | Submit | Contact Us | Français|
Objective: The researchers sought to assess whether the widely used 1994 Cochrane Highly Sensitive Search Strategy (HSSS) for randomized controlled trials (RCTs) in MEDLINE could be improved in terms of sensitivity, precision, or parsimony.
Methods: A gold standard of 1,347 RCT records and a comparison group of 2,400 non-trials were randomly selected from MEDLINE. Terms occurring in at least 1% of RCT records were identified. Fifty percent of the RCT and comparison group records were randomly selected, and the ability of the terms to discriminate RCTs from non-trial records was determined using logistic regression. The best performing combinations of terms were tested on the remaining records and in MEDLINE.
Results: The best discriminating term was “Clinical Trial” (Publication Type). In years where the Cochrane assessment of MEDLINE records had taken place, the strategies identified few additional unindexed records of trials. In years where Cochrane assessment has yet to take place, “Randomized Controlled Trial” (Publication Type) proved highly sensitive and precise. Adding six more search terms identified further, unindexed trials at reasonable levels of precision and with sensitivity almost equal to the Cochrane HSSS.
Conclusions: Most reports of RCTs in MEDLINE can now be identified easily using “Randomized Controlled Trial” (Publication Type). More sensitive searches can be achieved by a brief strategy, the Centre for Reviews and Dissemination/Cochrane Highly Sensitive Search Strategy (2005 revision).
Health care professionals practicing evidence-based care, and librarians and information specialists supporting them, need easy access to the best evidence. Access to best evidence from systematic reviews, meta-analyses, and other summaries of the evidence is improving all the time, through the international efforts of The Cochrane Collaboration through the publication of The Cochrane Library , health technology assessment programs such as those of the Agency for Healthcare Research and Quality , the BMJ's Clinical Evidence , the ACP Journal Club , and the work of many others. Well-designed and conducted randomized controlled trials (RCTs) often form the most reliable input to systematic reviews and meta-analyses of health care interventions and, where practical and ethical, can provide the best evidence in the absence of systematic reviews. Access to RCTs has improved greatly since problems with identifying trials in MEDLINE were presented at a meeting at the US National Institutes of Health in 1993  and subsequently reported in a systematic review by Dickersin and colleagues in the BMJ in 1994 .
Dickersin and colleagues suggested improvements in the ways that authors should describe their work and recommended better indexing of RCTs in MEDLINE and changes to the methods used to identify trials in MEDLINE . The paper included a highly sensitive search strategy, designed by one of the authors (Lefebvre), to identify RCTs in MEDLINE. This subsequently became known as the Cochrane Highly Sensitive Search Strategy (Cochrane HSSS). This paper represented important information for researchers conducting systematic reviews, for clinicians trying to find best evidence from RCTs, and for librarians and information specialists supporting them in identifying the RCTs. The Cochrane HSSS has been used extensively over the last ten years by those involved in preparing systematic reviews, meta-analyses, and health technology assessment reports and has been recommended in guidelines for Cochrane reviewers [7, 8].
However, databases change over time in content, indexing practice, and other features. The last ten years have seen several developments that follow Dickersin and colleagues' recommendations. First, the US National Library of Medicine (NLM) applied indexing improvements to MEDLINE, as a result of the paper presented to the National Institutes of Health in December 1993 . A higher proportion of RCTs were subsequently correctly indexed with the Publication Type, “Randomized Controlled Trial” [9, 10]. In addition, a new Publication Type, “Controlled Clinical Trial,” was introduced in 1995. Secondly, NLM announced that they would “retag” reports of RCTs not already indexed with the appropriate “Randomized Controlled Trial” or “Clinical Controlled Trial” Publication Types.
As a consequence, The Cochrane Collaboration embarked on an extensive program to identify “untagged” RCTs in MEDLINE by reading the titles and abstracts of candidate records, published both before and after the introduction of the “Randomized Controlled Trial” (1991) and “Clinical Controlled Trial” (1995) Publication Types. Identified trial reports that are not already indexed with the “Randomized Controlled Trial” or “Clinical Controlled Trial” Publication Types are forwarded to NLM for retagging in MEDLINE . Over the last 10 years, the number of RCTs in humans indexed with the appropriate Publication Types, and, therefore, easily and accurately identifiable in MEDLINE, has risen from 20,000 (1993) to more than 270,000 in October 2005 (of which 100,000 were published before 1993). Retagging of MEDLINE continues annually and is usually a year behind the current MEDLINE publication year.
Beyond improvements to MEDLINE indexing, RCTs have also become more accessible through The Cochrane Collaboration's Cochrane Central Register of Controlled Trials (CENTRAL), published and updated quarterly in The Cochrane Library . CENTRAL contains 455,000 records of RCTs and clinical controlled trials (CCTs) as of issue 3 in 2005. All MEDLINE records relevant to humans with the “Randomized Controlled Trial” or “Clinical Controlled Trial” Publication Types are included, along with many thousands of non-MEDLINE records identified from The Cochrane Collaboration's journal hand-searching program and databases such as EMBASE .
In addition to NLM's concentration on correctly identifying and indexing RCTs in MEDLINE, authors and journal editors are paying increased attention to reporting research methods clearly in the titles and abstracts of reports of RCTs in journals. The Consolidated Standards of Reporting Trials (CONSORT) statement gives clear guidelines on describing methods , and positive effects of these recommendations have already been reported [13, 14]. This should mean that database indexers are more likely to identify and correctly index reports of RCTs. These major efforts mean that ten years on from its publication in the Dickersin BMJ paper , the long and complex Cochrane HSSS might no longer be the most efficient tool for librarians and other searchers and should be reassessed to identify whether alternative search terms might be more efficient.
The methods of search strategy design have also developed with a move toward more objective and research-based approaches. The Cochrane HSSS was based on a subjective selection of search terms compiled from free-text terms recommended by individuals with expertise in clinical trials and experience in searching for them in MEDLINE, together with Medical Subject Headings (MeSH) identified by one of the authors (Lefebvre). It was not derived from, or tested on, a gold standard of known reports of RCTs . Haynes and colleagues used subjectively derived search terms to identify “clinically sound” studies of therapy, which focused on identifying RCTs but tested strategy performance on gold standard sets of known records . Their strategy, updated in February 2004, has been included in the PubMed version of MEDLINE as a time-saving feature for searchers (Clinical Queries) .
In recent years, researchers have developed search strategy design further to improve the objectivity of their methods. Techniques such as word frequency analysis and discriminant analysis have been used to derive objectively, through statistical analysis, the most efficient search terms to find desired types of records. Methods that the Centre for Reviews and Dissemination (CRD) and UK Cochrane Centre (UKCC) Search Filters Design Group developed and refined [17,18] form the basis of this current research to develop efficient search strategies for busy clinicians who wish to identify RCTs in MEDLINE reliably and for systematic reviewers, meta-analysts, and others requiring higher sensitivity but with acceptable precision—and for librarians and information specialists who support these activities.
The authors' research builds on methods described elsewhere in full . These methods use an approach that is similar to that used in designing and evaluating a diagnostic test. A gold standard of known desired records (in this case, MEDLINE records of RCTs) is used to identify frequently occurring free-text words in the titles, abstracts, and MeSH terms. These frequently occurring terms are then statistically tested with respect to how well they perform as search terms to discriminate a subset of known, desired gold standard records from a subset of other types of records (in this case, MEDLINE records that are not records of RCTs). To establish the internal validity of the approach, the performance of the statistically derived sets of terms as search strategies is then tested, using a statistical package, against the unused subsets of gold standard and non–gold standard records. To test external validity (in a non-test environment), the performance of the strategies is then tested in one or more “real world” scenarios, such as in the entire MEDLINE database in certain subject areas.
In the current research, 4 random samples of RCT records and matching random samples of non-trial records published in 4 different years (1970, 1980, 1990, and 2000) were identified from MEDLINE. All records with the “Randomized Controlled Trial” Publication Type in those years were identified, and 4 samples of 300 records were selected using random number generation software. A further set of RCT records randomly identified from CENTRAL (n = 147) was added to this gold standard. These additional records were available in MEDLINE but had not, at that point in time, been indexed with the “Randomized Controlled Trial” Publication Type. We included these records of RCTs to explore whether they produced different search terms and to ensure that not all of the records in the gold standard contained the “Randomized Controlled Trial” Publication Type. The gold standard set of known but randomly selected RCT records (n = 1,347) was matched with a comparison group of randomly selected records of non-trials (n = 2,400) from the same years. By using random selection, the records identified for inclusion in the gold standard collection or the comparison groups should have no systematic similarities or systematic differences.
The frequency of occurrence of words in a 50% random sample of the gold standard was ascertained using WordStat software . All words that occurred in at least 1% of the sample records were analyzed. This frequency level was chosen to return as many variables as possible while retaining adequate degrees of freedom for analysis. Once identified, the presence or absence of these words in all the gold standard records and the non-trial records was recorded in a spreadsheet. Fifty percent of each set of records were then selected randomly and analyzed using logistic regression in SPSS . The records were analyzed by year (1970, 1980, 1990, and 2000), as a single group covering all years, and as 2 subsets of trial records (MEDLINE-derived trial records and CENTRAL-derived trial records). The logistic regression analysis produced lists of terms (search strategies) that could best discriminate the RCT records from the non-trial records. The performance of these search strategies, in terms of sensitivity and precision, was then tested on the remaining 50% of gold standard RCT and non-trial records. Sensitivity was defined as:
Precision was defined as:
The best performing sets of terms (or search strategies) from this statistical analysis and testing were then tested for external validity in the real world by searching MEDLINE (Ovid interface). Performance was assessed in terms of how well the strategies performed in finding records of known RCTs (i.e., those indexed with the “Randomized Controlled Trial” Publication Type) and records of untagged RCTs and CCTs in breast cancer. The authors identified all records indexed as being about breast cancer in humans and published in four different years (1970, 1980, 1990, and 2000) by using the “exploded” MeSH term, “Breast Neoplasms.” This collection of records was then assessed by a researcher who read the titles and abstracts of the records to identify which records were reports of RCTs. The researcher was an experienced hand-searcher, trained to identify records of trials from journals and database records such as MEDLINE's. All reports of RCTs identified by the researcher were subsequently checked by another experienced researcher.
Our final test of the external validity of the strategies we had derived was to assess the yield of our 6 strategies (Figure 1) in finding unindexed RCTs in MEDLINE that had been published in 2003 and not yet assessed in The Cochrane Collaboration's annual MEDLINE retagging exercise. We also tested the yield of a number of other published strategies for identifying RCTs in MEDLINE. We only tested published strategies that claimed over 90% sensitivity and had used a gold standard set of over 100 records against which to test their strategies [8, 16, 21–24]. To identify a gold standard of indexed and unindexed records of RCTs, we searched the current issue of MEDLINE (Ovid interface, May 2004) for all records published in 2003 in each of 4 separate subject areas, identified by searching using the exploded MeSH terms, “Otitis Media,” “Migraine,” “Cataract Extraction,” and “Asthma.” We chose these 4 subject areas to provide a variety of tests for the strategies while generating a manageable number of MEDLINE records to hand-search. The search generated 4,681 records, which were hand-searched by an experienced hand-searcher (with selections verified by a second experienced researcher) to identify records that were RCTs but not indexed as either RCTs or CCTs. These records, along with the indexed (known) records of RCTs and CCTs formed the gold standard for this range of tests.
We assessed the performance of our own and the published search strategies in terms of sensitivity and precision and what we described as a “best compromise” of sensitivity and precision. Sensitivity was defined as:
Precision was defined as:
Our working definition of the best compromise strategy, for the purpose of this study, is a strategy that has a sensitivity of at least 90%, with the highest possible precision score. Ninety percent was selected because this score is likely to be the lowest possible acceptable sensitivity rate for researchers trying to identify reports of RCTs for possible inclusion in systematic reviews and technology assessments. However, readers can select their own levels of sensitivity and precision, from our results tables, to derive their own best compromise strategies.
The logistic regression showed that overall the most discriminating single search term to differentiate the 1,347 gold standard records of RCTs from the 2,400 non-trial records was “Clinical Trial” as a Publication Type.
The performance of the most discriminating term and the next most discriminating terms was tested in a series of strategies. We assessed strategy performance in terms of successful retrieval of reports of indexed and unindexed RCTs in breast cancer in humans in the years 1970, 1980, 1990, and 2000 and identified 12,255 records from the searches. After hand-searching these records, an experienced researcher identified only 54 as unindexed trials (4 RCTs and 50 CCTs). The yield of our new strategies in terms of finding unindexed trials in breast cancer in years where Cochrane retagging in MEDLINE had taken place was, therefore, very low, and ranged from none to 2.19%.
The retrieval performances of our 6 search strategies (Figure 1) and other published strategies were tested on 4 further gold standard sets of MEDLINE records published in 2003. An experienced researcher assessed 4,681 MEDLINE records published in 2003 that were retrieved by searching using the exploded MESH terms, “Otitis Media,” “Migraine,” “Cataract Extraction,” and “Asthma.” We identified 392 indexed trials and 32 unindexed trials. The performance of our strategies and published strategies in terms of identifying records in this gold standard of 424 records is presented in decreasing order of sensitivity in Table 1.
Our strategies proved highly sensitive while retaining very reasonable levels of precision. Strategy A offers a similar level of sensitivity (99.29%) to the full Cochrane HSSS (99.53%) but with a 2% improvement in precision. Our new strategies, therefore, are effectively as sensitive as the full Cochrane HSSS, slightly more precise and much briefer (a maximum of 7 search terms rather than 22). Strategy A, strategy D, the full Cochrane HSSS , and the Robinson  strategy are all highly sensitive (more than 99%) in finding reports of trials. In our view, the best compromise strategy is strategy F, which achieved 92.69% sensitivity, while retaining a very high level of precision (81.03%).
In years where the Cochrane retagging of MEDLINE records has taken place, and based on our data from hand-searching MEDLINE breast cancer records, our strategies (A to D in Figure 1) produce very few extra unindexed records of trials. This low increase in sensitivity suggests that adequate sensitivity and precision to support busy health professionals and researchers who want to identify RCTs in MEDLINE in years before the current two years can now be achieved by a very brief search using the single Publication Type, “Randomized Controlled Trial.”
In years where Cochrane retagging of MEDLINE records has not yet taken place, we recommend relying on the “Randomized Controlled Trial” Publication Type (strategy E) to achieve relatively high sensitivity (82.78%), if high precision is required (100%). For a small loss of precision (81.03%) and enhanced sensitivity (92.69%), searchers can substitute the “Clinical Trial” Publication Type (strategy F) for “Randomized Controlled Trial.” Finally, for librarians supporting systematic reviewers, meta-analysts, and other researchers requiring high sensitivity, hand-searching four topic areas shows that adding a few additional search terms can identify additional untagged trials at reasonable levels of precision (over 21% for strategy A), at sensitivity levels (99%) that almost equal the full twenty-two-search terms of the Cochrane HSSS and that outperform the sensitivity of the PubMed Clinical Queries Therapy Filter. We have named strategy A the CRD/Cochrane Highly Sensitive Search Strategy (2005 revision), and it is shown, with the other search strategies, in Figure 2.
In this research, we have attempted to be as objective as possible, to allow the strategies to be derived from the free-text terms and indexing present in MEDLINE records, rather than use methods with which we would make subjective decisions and potentially introduce bias. The requirements of the analysis, however, have sometimes meant that subjective choices had to be made. For example, we set a frequency level of word analysis in WordStat to avoid producing too many variables for the analysis. This level meant that very infrequently occurring terms, which might be highly discriminating in terms of identifying a record as an RCT, could have been excluded from the statistical analysis. We intend to continue to develop our methods to improve objectivity and to explore ways to reduce subjective limits of this type. Ideally, all hand-searching should be conducted by two independent researchers, but we were only able to use one experienced hand-searcher, with a second researcher to verify the accuracy of the first researcher's choices. This procedure meant that some studies might have been missed.
The logistic regression analysis presents terms with weightings. The ability to implement weightings attached to search terms in database interfaces would greatly enhance the precision of search strategies. At present, few search interfaces to major databases offer this option, and we would like to pursue ways to achieve this option with database producers and service providers. Alternatively, software that processes a set of search results by using weightings would also be a useful development. This approach would be a secondary software filter applied perhaps to the results of a very sensitive search of MEDLINE. In terms of making search filter design more straightforward, we are developing software that will undertake regression analysis without weighting terms but using input from searchers about their preferences for the percentage of false positives in their results.
The current research has tested the performance of our strategies in Ovid MEDLINE using five distinct test sets of RCTs (in five disease areas). The generalizability of the results to other health care areas, in particular those where RCTs are less commonly used designs, needs to be tested. The authors plan to extend their research methods to test and develop search strategies for identifying reports of RCTs more efficiently in EMBASE and other databases.
Our results suggest that a number of the recommendations of Dickersin and colleagues are bearing fruit ten years on . Librarians and information specialists searching MEDLINE today for RCTs are benefiting from a number of initiatives: the MEDLINE retagging efforts of The Cochrane Collaboration and NLM, the improvements in accuracy of indexing by NLM, the addition of “Controlled Clinical Trial” as a Publication Type in 1995 [9, 10, 25], and the impact of the CONSORT statement [12–14]. In addition, peer reviewers should be becoming increasingly aware of the issues around clear descriptions of methods . The growing use of unique numbers assigned to RCTs at their inception, such as the International Standard Randomised Controlled Trial Number (ISRCTN) , and the insistence by a number of high-impact-factor journal publishers that RCTs must be registered at inception to publish the subsequent reports of the results in their journals  should also result in easier retrieval of published reports of RCTs. In the meanwhile, the good news for searchers is that with CENTRAL available in The Cochrane Library and MEDLINE available free of charge at the point of use, reports of RCTs are now easily and reliably accessible with just a few keystrokes.
We are grateful to Teresa Clarke and Susan Hodges for hand-searching records, Anne Eisinga for verifying the reports of RCTs and identifying and compiling the results of the hands-searches, and Steve McDonald for providing information on studies of search strategies to identify reports of RCTs. We also acknowledge helpful comments on drafts of this paper from Janette Boynton, Mike Clarke, Anne Eisinga, Su Golder, Jos Kleijnen, Eric Manheimer, and Steve McDonald and the anonymous referees.
*This research received no financial support, but the preliminary stages of the project were conducted by Janette Camosso-Stefinovic as part of her master's degree in library and information studies at Loughborough University, United Kingdom. Parts of this research have been presented at the Health Technology Assessment International Conference in Krakow, Poland; May 31–June 2, 2004; and the Twelfth Cochrane Colloquium, Ottawa, Canada; October 4–6, 2004.