Search tips
Search criteria 


Logo of jamiaAlertsAuthor InstructionsSubmitAboutJAMIA - The Journal of the American Medical Informatics Association
J Am Med Inform Assoc. 1994 Jan-Feb; 1(1): 51–60.
PMCID: PMC116184

A performance and failure analysis of SAPHIRE with a MEDLINE test collection.


OBJECTIVE: Assess the performance of the SAPHIRE automated information retrieval system. DESIGN: Comparative study of automated and human searching of a MEDLINE test collection. MEASUREMENTS: Recall and precision of SAPHIRE were compared with those attributes of novice physicians, expert physicians, and librarians for a test collection of 75 queries and 2,334 citations. Failure analysis assessed the efficacy of the Metathesaurus as a concept vocabulary; the reasons for retrieval of nonrelevant articles and nonretrieval of relevant articles; and the effect of changing the weighting formula for relevance ranking of retrieved articles. RESULTS: Recall and precision of SAPHIRE were comparable to those of both physician groups, but less than those of librarians. CONCLUSION: The current version of the Metathesaurus, as utilized by SAPHIRE, was unable to represent the conceptual content of one-fourth of physician-generated MEDLINE queries. The most likely cause for retrieval of nonrelevant articles was the presence of some or all of the search terms in the article, with frequencies high enough to lead to retrieval. The most likely cause for nonretrieval of relevant articles was the absence of the actual terms from the query, with synonyms or hierarchically related terms present instead. There were significant variations in performance when SAPHIRE's concept-weighing formulas were modified.

Full Text

The Full Text of this article is available as a PDF (1000K).

Selected References

These references are in PubMed. This may not be the complete list of references from this article.
  • Hersh WR, Greenes RA. Information retrieval in medicine: state of the art. MD Comput. 1990 Sep-Oct;7(5):302–311. [PubMed]
  • Funk ME, Reid CA. Indexing consistency in MEDLINE. Bull Med Libr Assoc. 1983 Apr;71(2):176–183. [PMC free article] [PubMed]
  • Slingluff D, Lev Y, Eisan A. An end user search service in an academic health sciences library. Med Ref Serv Q. 1985 Spring;4(1):11–21. [PubMed]
  • Hersh WR. Evaluation of Meta-1 for a concept-based approach to the automated indexing and retrieval of bibliographic and full-text databases. Med Decis Making. 1991 Oct-Dec;11(4 Suppl):S120–S124. [PubMed]
  • Haynes RB, McKibbon KA, Walker CJ, Ryan N, Fitzgerald D, Ramsden MF. Online access to MEDLINE in clinical settings. A study of use and usefulness. Ann Intern Med. 1990 Jan 1;112(1):78–84. [PubMed]
  • Hersh WR, Hickam DH. A comparison of retrieval effectiveness for three methods of indexing medical literature. Am J Med Sci. 1992 May;303(5):292–300. [PubMed]
  • Hanley JA, McNeil BJ. A method of comparing the areas under receiver operating characteristic curves derived from the same cases. Radiology. 1983 Sep;148(3):839–843. [PubMed]
  • McKibbon KA, Haynes RB, Dilks CJ, Ramsden MF, Ryan NC, Baker L, Flemming T, Fitzgerald D. How good are clinical MEDLINE searches? A comparative study of clinical end-user and librarian searches. Comput Biomed Res. 1990 Dec;23(6):583–593. [PubMed]
  • Rada R, Bicknell E. Ranking documents with a thesaurus. J Am Soc Inf Sci. 1989 Sep;40(5):304–310. [PubMed]
  • Evans DA, Hersh WR, Monarch IA, Lefferts RG, Handerson SK. Automatic indexing of abstracts via natural-language processing using a simple thesaurus. Med Decis Making. 1991 Oct-Dec;11(4 Suppl):S108–S115. [PubMed]

Articles from Journal of the American Medical Informatics Association : JAMIA are provided here courtesy of American Medical Informatics Association