We generally followed the methodology of Evans.1
We: 1) defined the concepts to be extracted; 2) built the parser and parsing rules; 3) prepared data for testing; 4) had two physicians annotate the test data to create a gold standard; 5) processed test data with the parser; 6) scored the results. The parser has the following features:
- As a task-specific parser, its algorithm and parsing rules can be restricted to a discrete domain;
- It takes as parameter a lexicon of drug names—in our design of experiments we rely on a lexicon derived from RxNorm entries in the UMLS;
- It takes as parameter a set of parsing rules formulated as regular expressions, which can be tailored for specific tasks or clinical note styles;
- By looking for surrounding dosage language, the parser can find misspelled drug names or drugs that are not in the lexicon;
- The parser can extract relevant information in addition to the drug name:
- ○ whether the drug was mentioned as an allergy or as discontinued or as prescribed and taken;
- ○ where or when the drug was administered—at home, in the emergency room, preoperatively;
- ○ dose, route, frequency and necessity;
- ○ meta-data from the drug lexicon (e.g., RxNorm CUI).
Our goal is to provide accurate, comprehensive information about what drugs a patient has been on based on evidence appearing in textual notes. We generally follow Evans, et al. in conceptualizing drug information—a drug expression can consist of a drug name accompanied by dose level, route, frequency, or necessity—however we loosen the concept so that we can catch important medication events that Evans, et al. ignored. Our goal includes identifying:
- drug expressions with misspelled drug names (“Azithromyin 250 mg PO”);
- drug names appearing independent of other drug expression elements (“Patient was given K-Dur”);
- references to medication events that use drug classes rather than drug names (“Outpt course po Abx”, “Pt will remain on antiseizure medications”);
- clear implication of medication events (“The patient was subsequently digitalized.”)
In order to avoid reporting a drug event where none occurred, such as the mention of a drug the patient refused, or drugs appearing in an allergy list, Evans required a drug name to be accompanied by dose-level, frequency or another one of the attributes they deemed sufficient. We chose to relax this requirement in order to avoid missing true medication events.
We address the problem of drug name appearance where no drug event occurred by defining a context in which the drug name appears. We also use context to help determine the dates of drug administration. In capturing context information we cast a wide net and collect any language we can identify that informs the context (examples from the gold standard and the parser include: “at home”, “cur med”, “increased”, “held”, “1980s”, “since 13”, “hospital course, OR”). We then map this variety of contextual information into four classes: history, hospital course, on discharge, and not administered.
shows a sample of discharge summary text and our breakdown of the medication event into specific fields.
Sample text from discharge summary with interpretation of a medication event
Unlike many types of text faced by NLP or MLP applications, medication event information is generally not recursive; it consists of a few types of term—medication name, dose, route, frequency, etc.—strung together in a somewhat inconsistent order, but it does not contain clauses and sub-clauses: it constitutes a regular language rather than a context-free language. We modeled this language through manual analysis, and took advantage of its relative simplicity by constructing parsing rules out of regular expressions as described below.
Following our operational definition of drugs, the parser extracts three types of objects: drugs, possible drugs, and context clues. We use a drug lexicon derived from RxNorm entries in the UMLS to identify drug names. A drug is defined as a drug name optionally accompanied by dosage information (dose, route, frequency, necessity). A possible drug is defined as any non-drug-name text surrounded by enough dosage information to indicate that it refers to a misspelled drug name or a drug name not in our lexicon. Context clues are pieces of text that give us an idea of where, when, and whether the drug was administered
Drug name identification uses a multi-word lookup from the input text to the drug lexicon. The drug lexicon may contain meta-data about the drug name, which will then be connected to the drug in the parser output. As our lexicon is derived from UMLS, we attach the UMLS Concept Unique Identifier (CUI) to drug names. This is not important to the parsing process per se, but the calling program can make use of this information when data is returned regarding the drugs that have been extracted.
Parsing of a clinical note begins by identifying drug names using the lexicon. The rest of the process uses parsing rules stored in a grammar file. The grammar file allows for the definition of terminals and nonterminals. A terminal definition associates a symbol with one or more regular expressions. For instance, the terminal unitOfMeasure is defined (partly) as “(g|gm|mg|milligram|mcg|microgram)”. Nonterminals are defined essentially in the same way as terminals except that their regular expressions can include symbols for terminals and other nonterminals, which will be expanded into appropriate regular expressions and interpolated into the regular expression of the non-terminal. For instance, part of the definition of the non-terminal dose is “number\s*unitOfMeasure?\b\s*form”, which will match a number, optional whitespace, an optional unitOfMeasure, a word boundary, more optional whitespace, and finally a form.
The parser returns drugs, possible drugs and context clues in a first pass. It then passes over the returned drug and possible drug text to break these down into their constituent parts.
The context clues extracted by the parser can be supplemented by information from outside the parsed text. In our electronic records system this was necessary because discharge summaries are stored in pieces with separate, named sections, and sometimes the section name gives a better context clue than the text of the section. (Examples of section names are “Attending physician”, “Medications”, “History of present illness”.) Context clues parsed from the text and derived separately are combined, normalized, and attached to the drugs.
We created the test data set from 26 de-identified discharge summaries from patients seen in 2004 at Columbia University Medical Center/New York Presbyterian Hospital. The test data were left unseen before scoring the results. The development of the parser was carried out using a separate set of unannotated discharge summaries.
Gold Standard Annotation
Two physicians created the gold standard by sequentially annotating the test data: one physician annotated the data and her results were given to a second physician who checked and revised her work. Revisions were discussed with and accepted by the first physician.
Annotation was performed using evaluation forms containing a discharge summary and a table with columns for Drug Name, Context, Dose, Route, Frequency and Necessity. The physicians were instructed to fill in these columns with values quoted verbatim from the discharge summary.
Scoring Parsing Results
The 26 discharge summaries were processed by the parser and drug names were scored as being: 1) a match, 2) a false positive, or 3) a false negative.
The other fields—context, dose, route, frequency and necessity—were scored as one of: 1) Correct, 2) Partially Correct, and 3) Wrong. “Partially correct” was used in instances where the parser produced some but not all of the information provided by the gold standard, for example, if the gold standard recorded a dose as beginning at 5mg and decreasing to 2.5 mg while the parser only recorded 5 mg. Since the context results were catagorical, they were scored simply as Correct ot Incorrect.