Analysis of False Positive Errors of an Acute Respiratory Infection Text Classifier due to Contextual Features

AMIA Summits Transl Sci Proc. 2010; 2010: 56–60.

Published online 2010 March 1.

PMCID: PMC3041533

Analysis of False Positive Errors of an Acute Respiratory Infection Text Classifier due to Contextual Features

Brett R. South, MS,^1,^2,⁵ Shuying Shen, MStat,^1,^2,⁵ Wendy W. Chapman, PhD,³ Sylvain Delisle, MD, MBA,⁴ Matthew H. Samore, MD,^1,^2,⁵ and Adi V. Gundlapalli, MD, PhD, MS^1,^2,⁵

¹VA Salt Lake City Health Care System

²Department of Internal Medicine, University of Utah School of Medicine

³Department of Biomedical Informatics, University of Pittsburgh

⁴VA Maryland Health Care System and University of Maryland School of Medicine

⁵Department of Biomedical Informatics, University of Utah School of Medicine

This is an Open Access article: verbatim copying and redistribution of this article are permitted in all media for any purpose

Abstract

Text classifiers have been used for biosurveillance tasks to identify patients with diseases or conditions of interest. When compared to a clinical reference standard of 280 cases of Acute Respiratory Infection (ARI), a text classifier consisting of simple rules and NegEx plus string matching for specific concepts of interest produced 569 (4%) false positive (FP) cases. Using instance level manual annotation we estimate the prevalence of contextual attributes and error types leading to FP cases. Errors were due to (1) Deletion errors from abbreviations, spelling mistakes and missing synonyms (57%); (2) Insertion errors from templated document structures such as check boxes, and lists of signs and symptoms (36%) and; (3) Substitution errors from irrelevant concepts and alternate meanings for the same word (6%). We demonstrate that specific concept attributes contribute to false positive cases. These results will inform modifications and adaptations to improve text classifier performance.

Articles from AMIA Summits on Translational Science Proceedings are provided here courtesy of
American Medical Informatics Association