The primary outcome measurement was the precision of the final result set selected to answer the clinical question. Precision was defined as the ratio of relevant citations retrieved to the total number of citations retrieved in the set. The higher the precision the more efficient the search, as more of what is retrieved is also relevant. The measurement of precision is appropriate in the clinical setting, where it is often desirable to find a few good articles, as opposed to a research setting, where the measurement of recall, finding all possible relevant articles, is more important. We also examined the number of terms and filters used in the search strategy and user satisfaction with the search interface.
Thirty-one residents completed the study. Ten residents were randomized to Protocol A; 10 residents were randomized to Protocol B; and 11 residents were randomized to Protocol C. The results from one participant in Protocol C were discarded because the saved search strategies were corrupted and not useable. All three groups had similar clinical experience and searching experience, although the participants in Protocol A had less training in EBM (Table ). Participants using the PICO templates (Protocol A or Protocol B) had higher precision scores for each question than the participants who used the standard PubMed system in Protocol C. (Question 1: A = 35%, B = 28%, C = 20%; Question 2: A = 5%, B = 6%, C = 4%; Question 3: A = 1%, B = 0%, C = 0%) 95% confidence intervals were calculated for the precision for each question using a lower boundary of zero. The 95% confidence limits were overlapping, suggesting no statistical difference between the groups. Although there were no statistical differences between the groups, there may be a trend toward improved precision with the PICO search screens. (Table )
Baseline characteristics of study participants
In addition to quantitative comparisons, this pilot study provided an opportunity for more open-ended insight into how practitioners search. Some searches within each question (Question 1 = 3/30, Question 2 = 9/30, and Question 3 = 27/30) did not retrieve any relevant citations. A qualitative assessment of search strategies that produced no acceptable results revealed three common types of errors: ambiguous mapping of subject headings (MeSH); selecting the wrong publication type; and limiting search queries to just words in the title. The error that accounted for the largest number of non-productive searches was related to the MEDLINE indexing structure or MeSH (Medical Subject Headings). In question three, most searchers used the phrase "African Americans", which maps to the MeSH "African Americans" and retrieves 29077 citations [searched 4/18/07]. However, the common word "blacks" maps to the broader MeSH term "African continental ancestry group" and retrieves 48195 citations [searched 4/18/07]. Most of the relevant citations used either the word "blacks" or were indexed to the broader MeSH term "African continental ancestry group." The second most common error was related to selecting the wrong study design or Clinical Query for the type of question being searched. This was a common problem for Protocol A, which only listed therapy study designs. The third error affected Protocol C and involved limiting search terms to the title field. While searchers often select their articles based on relevant words in the title, a problem arises when more than one word or phrase is appropriate to the topic. For example, a "peg" is also called a "percutaneous endoscopic gastrostomy tube" or "feeding tube." Limiting the search to the word "peg" in the title eliminates mapping to appropriate subject headings (MeSH) such as intubation, gastrointestinal and therefore may exclude relevant articles on the topic that use alternative terminology. Understanding these errors can help in teaching effective searching and in developing better search systems.
Perceptions of ease of use and time spent searching were approximately the same across all three protocols, as were the numbers of terms used in each search, regardless of protocol used. However, of the 30 searches performed using Protocol B (PubMed/PICO/Clinical Queries), 25 (83%) actually incorporated the Clinical Queries into the strategy, as opposed to only 2 (7%) of the searches using Protocol C (PubMed/Web). Both Protocol A and B facilitated the use of publication types and the Clinical Queries by prompting the searcher to consider these elements in the strategy. While these elements were also available in Protocol C, they are either behind the Limits tab or listed in the PubMed Services menu.