We expected that most ambulatory Sigs would be relatively simple and easily accommodated by the Format. Thus, we used a large prescription sample in order to isolate and characterize failures of the Format, each of which might represent small but important subsets of prescriptions. To maximize the informational yield from these potentially rare representation failures, we categorized the prescription sample based on a multi-step pre-processing algorithm, followed by over-sampling of categories more likely to contain representation failures. The pre-processing steps were (1) normalization of lexical and other variants, (2) parsing values into each of the applicable fields of the Format, and (3) preliminary identification of representation problems. Each Sig in the final sample was reviewed by two experts to judge whether the Format was capable of fully representing the prescriber's intent. They also judged whether the Sig should be excluded as inadequately specified (would require a call to the prescriber for clarification). To evaluate the designated terminology systems (separately from the structure of the format), the authors manually searched for an appropriate term to represent each term actually used by prescribers for one of the codified fields. The study was approved by the RAND Institutional Review Board.
Two e-prescribing system vendors supplied a total sample of 20 161 de-identified (with no individually identifiable information) prescriptions from 82 providers practicing in 48 physician offices in Kansas, Michigan, and Maryland. One vendor's system only allowed free text for Sigs, while the other had several fields with drop down menus (dose quantity, dose form, and timing) and a free text field. Provider specialties included internal medicine, family practice, pediatrics, and general surgery. The target sample size (20 000) was selected to provide 95% Wald CIs of 0.1% around non-accommodated rates as low as 0.5%. The prescriptions had been written from March 2007 to February 2009 (75% being from September 2008 or later). In addition to Sig strings, we also obtained the corresponding drug names, drug strength, and dose form for each prescription.
In order to automate the pre-processing steps, we created a processing sequence that normalized common expressions and then parsed words and values from the normalized Sig strings into variables representing the fields of the Format. The normalization step removed variations due to lower/upper case, spaces, punctuation, and common spelling errors. It also converted numbers expressed in words into numerical values (eg, from ‘one’ to ‘1’), expanded abbreviations (eg, converting ‘PO BID’ to ‘by mouth 2 times per day’), and removed extraneous statements (eg, ‘thank you’). After normalization, the sample was collapsed into a list of unique Sig strings since many of them were exactly the same. Thus, each unique Sig string represented a number of raw Sig strings. The unique Sig strings were then parsed to populate the structured fields based on regular expression patterns. ‘Text’ fields were populated with the relevant words actually used in the Sig string. If a Sig string was not fully parsed, any unparsed remainder was output in a separate field. The Perl programming language was used for both normalization and parsing. For example, the program parsed ‘Inhale contents of 1 capsule once daily’ into several fields: ‘inhale’ in dose delivery method, ‘1’ in dose quantity, ‘capsule’ in dose form, ‘1’ in frequency numerical value, ‘day’ in frequency unit, and ‘contents of’ in the unparsed field.
Twelve prescription technology experts agreed to participate in reviewing the parsed representation of 100 Sig strings each. Because each expert would only be able to review a limited number of representations, we used the parsing results to substantially over-sample Sigs that were more likely to pose representation problems. provides an overview of this strategy. Both the fully parsed strings and not-fully parsed strings were further classified into preliminary problem categories based on the occurrence of string patterns that, on preliminary review by the authors, were likely to pose representation problems (appendix A (available online at www.jamia.org
) shows examples of these categories). To create the final expert review sample, we then included (1) one randomly selected example from each of 37 preliminary problem categories, (2) all unique Sig strings that were not fully parsed and not assigned to a preliminary problem category, and (3) a simple random sample from those unique Sig strings that were fully parsed and not assigned to a preliminary problem category. A final sample size of 600 was targeted to allow for each representation to have two independent reviewers.
For each Sig representation to be reviewed, the experts were presented with the complete original prescription (including the original drug name, strength, dose form, and the raw, unprocessed Sig string), the values parsed into each of Sig Format fields, a re-construction of the Sig based on those values, and the unparsed remainder. They were asked to judge (1) whether the raw Sig string was adequately specified, and (2) whether the Format could fully accommodate the prescriber's intent. ‘Adequately specified’ meant that a pharmacist could fill the prescription without needing to call the prescriber for clarification. If the two experts reviewing each prescription did not agree with each other, the authors adjudicated the judgments. Five rounds of training among experts on how to make their judgments were conducted prior to the ratings. For each round of training, experts were asked to adjudicate several Sig strings, and then a conference call was convened to reach a consensus on the correct representation of the example Sig strings.
After review by the individual experts, teleconferences with the expert panel were held to review each challenging case, and if the panel agreed upon a judgment that was different from that of the two (or three) expert reviewers, then the panel's consensus decision overrode the individual reviewers' judgments.
Terminology code mapping
To assess how well the terms used in the Sigs could be codified, the authors evaluated whether each term used could be mapped to an appropriate term in the designated vocabulary system. The fields involved were dose delivery method, dose form, route, site, vehicle, indication, and administration timing. Version 1.0 of the Format designates SNOMED CT as the vocabulary system to use for all fields except for dose form. For dose form, the Format specifies use of the relevant FMT vocabulary, which is currently the FDA's ‘Structured Product Label’ (SPL) subset of the National Cancer Institute Thesaurus (NCIt).14
We used the January 2009 release of SNOMED and the September 2009 release of the NCIt SPL subset.16
We report the proportion of terms used in the Sig strings for which a controlled vocabulary term could be found having the same meaning.
Descriptive statistics were generated for the original raw Sig string sample. Both unweighted and weighted statistics were produced based on the parsing results and the judgments made by the panelists. Each raw Sig string in the final sample for review was assigned two sampling weights from the two-stage sampling process shown in (stage 1 collapsed raw Sig strings into unique Sig strings; stage 2 was sampling within each preliminary problem category). The product of these two sampling weights (after normalization) estimates the number of original (raw Sig) prescriptions that each item in the final sample represents.