Data from clinical notes: a perspective on the tension between structure and flexible documentation

doi:10.1136/jamia.2010.007237

J Am Med Inform Assoc. 2011 Mar-Apr; 18(2): 181–186.

Published online 2011 January 12. doi: 10.1136/jamia.2010.007237

PMCID: PMC3116264

Data from clinical notes: a perspective on the tension between structure and flexible documentation

S Trent Rosenbloom, Joshua C Denny, Hua Xu, Nancy Lorenzi, William W Stead, and Kevin B Johnson

Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, Tennessee, USA

Corresponding author.

Correspondence to Dr S Trent Rosenbloom, Eskind Biomedical Library, Room 440, 2209 Garland Avenue, Nashville, TN 37232-8340, USA; Email: trent.rosenbloom/at/vanderbilt.edu

Received July 9, 2010; Accepted December 10, 2010.

See "Electronic health records and computer-based clinical decision support: are we there yet?" on page 109.

This article has been cited by other articles in PMC.

Abstract

Clinical documentation is central to patient care. The success of electronic health record system adoption may depend on how well such systems support clinical documentation. A major goal of integrating clinical documentation into electronic heath record systems is to generate reusable data. As a result, there has been an emphasis on deploying computer-based documentation systems that prioritize direct structured documentation. Research has demonstrated that healthcare providers value different factors when writing clinical notes, such as narrative expressivity, amenability to the existing workflow, and usability. The authors explore the tension between expressivity and structured clinical documentation, review methods for obtaining reusable data from clinical notes, and recommend that healthcare providers be able to choose how to document patient care based on workflow and note content needs. When reusable data are needed from notes, providers can use structured documentation or rely on post-hoc text processing to produce structured data, as appropriate.

Keywords: Medical informatics applications, medical informatics computing, medical records systems, computerized, support, US Gov't, P.H.S., user-computer interface

Introduction

The process and products of documenting clinical care occupy a critical intersection among the diverse domains of patient care, clinical informatics, workflow, research, and quality. For the current manuscript, we define clinical documentation as the process of creating a text record that summarizes the interaction between patients and healthcare providers occurring during clinical encounters. Clinical documents produced in this process may include notes from outpatient visits, inpatient admissions and discharges, procedures, protocols, and testing results. Healthcare providers generate clinical documents to achieve numerous goals, including: to create narrative reports of their observations, impressions, and actions related to patient care; to communicate with collaborating healthcare providers; to justify the level of service billed to third-party payers; to create a legal record in case of litigation; and to provide data to support clinical research and quality-assessment programs.^1–9 Increasingly widespread adoption of electronic health record (EHR) systems has facilitated reuse of clinical documents for research, quality initiatives, and automated decision support, among other uses.^10–15 A major emphasis for integrating documentation systems with EHR systems is to increase the availability of structured clinical data for automated downstream processes. This has led to a profusion of computer-based documentation (CBD) systems that promote real-time structured clinical documentation.

The myriad requirements imposed on clinical documentation compel healthcare providers to create notes that are simultaneously accurate, detailed, reusable, and readable.¹¹ ^16–20 As a result, integrating clinical documentation into workflows that contain EHR systems has proven a challenge.¹⁸ ^20–29 The complex interplay among the note characteristics healthcare providers value, the structure and standardization that data reuse requires, and the attributes of various documentation methods each affect the documentation method's adoption into clinical workflows. The flexibility of a CBD method to allow healthcare providers freedom and ensure accuracy can directly conflict with a desire to produce structured data to support reuse of the information in EHR systems. The challenge is compounded when those implementing CBD systems have different priorities from clinician users working in busy settings. In particular, we have previously demonstrated that healthcare providers prefer the ability to balance using a standardized note structure and having the flexibility to use expressive narrative text.¹⁹ ²⁵

This viewpoint paper explores the tension between requirements that documentation methods support both structure and expressivity, then reviews two general approaches for obtaining structured data through clinical documentation, and provides our perspective of how best to use CBD systems in a clinical environment.

Tension between structure and expressivity

There exists a tension between the needs of busy healthcare providers documenting clinical care and of those reusing data from healthcare information systems. Busy clinicians generally value flexibility and efficiency, while those reusing data often value structure and standardization.²¹ ²² ^30–32 In investigations evaluating healthcare providers' impressions of documentation systems, subjects articulated that the documentation methods they use should promote the quality and expressivity of notes they generate and should integrate efficiently into busy workflows.²⁵ ³²

Expressivity has been defined as how well a note conveys the patient's and provider's impressions, reasoning, and thought process; level of concern; and uncertainty to those subsequently reviewing the note.²⁵ ³² Expressivity refers to the linguistic nuance necessary for describing aspects of the patient encounter using any words or phrases that the healthcare provider deems appropriate, at whatever length is necessary. An expressive documentation method, for example, may allow a healthcare provider to create notes using nuanced words or pictures that capture the flavor of the clinical encounter. Healthcare providers may rely on expressivity to convey: (1) patients' linguistic and narrative idiosyncrasies, (2) their level of concern or acuity, (3) the appearance of competence, (4) the provider's degree of uncertainty, and (5) the unique aspects of the clinical case that distinguishes it from other similar cases. Although healthcare providers value documentation methods supporting narrative expressivity, there have been few studies published in the biomedical literature of CBD systems directly evaluating this attribute, and its value may vary based on the type of note being written. Among existing studies, investigators have demonstrated that when compared to highly structured diagnostic or impressions data, clinical notes containing naturalistic prose have been more accurate,³³ more reliable for identifying patients with given diseases,³⁴ and more understandable to healthcare providers reviewing patient records.³⁵ ³⁶ In addition, numerous structured documentation systems include components that generate natural prose notes to increase their acceptance,³⁵ ^37–39 which implies that natural-language text has value to clinical users.

While the value of expressivity is unknown, there is value in having access to reusable structured data from clinical notes.¹⁸ ^40–42 Unfortunately, systems optimized to acquire structured data from healthcare providers often have user interfaces that are idiosyncratic, inflexible, or inefficient, and thus place the burden of entering the data in a structured format on a busy healthcare provider, rather than leveraging specific computer programs to extract the data from the human-input clinical narrative.²³

Healthcare providers using EHR systems have two major methods for converting their observations and impressions of patient care episodes into machine-computable and reusable data. With the first method, healthcare providers directly create structured clinical notes by using specialized CBD systems that capture structured data in real time. With the second method, healthcare providers document patient care episodes using relatively unstructured approaches, such as using dictation with transcription or a computer-based CBD system, and then apply computer programs designed to extract clinical data from the notes' text. These approaches are reviewed in greater detail below.

Using structured documentation tools

Clinical structured entry systems are specialized CBD systems that emphasize capturing structured (ie, conforming to a predefined or conventional syntactic organization) and/or standardized (ie, conforming to a predefined semantic standard) clinical data during standard documentation processes.⁴³ Users document in structured entry systems by retrieving concepts and assigning them a status (eg, locating in a ‘knee pain’ template the concept ‘knee effusion’ and selecting ‘absent’). Selectable concepts may come from sophisticated reference terminologies,⁴⁴ ⁴⁵ from specialized interface terminologies,³² or from informal and nonstandardized term collections. Structured entry systems may allow users and developers to create highly customized templates to maximize data completeness and structure.³² However, such templates may not easily use existing interface terminologies or conform to knowledge representation formalisms.

Over the last half-century, investigators, developers, and pioneers in the field of biomedical informatics developed numerous structured entry systems. As we reported elsewhere,⁴⁶ structured entry systems described in the biomedical literature include groundbreaking documentation systems by Slack,⁴⁷ ⁴⁸ work by Ledley for documenting radiology repots,⁴⁹ and those by Stead and Hammond that allowed patients to enter their own histories and that ultimately evolved into the TMR medical record system.^50–53 Other early structured entry systems include Barnett's COSTAR system,⁵⁴ Weed's PROMIS system,⁵⁵ Wirtschafter⁵⁶ and Shortliffe's⁵⁷ independent works developing chemotherapy documentation, and decision-support systems. Additional structured entry systems have been described, including ARAMIS which served as a user interface for data collection about patients with rheumatoid arthritis⁵⁸ ⁵⁹; an endoscopy documentation tool called CORI⁶⁰ ⁶¹; Musen's T-HELPER system⁶² for HIV-related clinical trial and medical care documentation; Shultz's QUILL system³⁸ ⁴⁶ ⁶³; Johnson's pediatric documentation system, Clictate⁶⁴, and several structured entry systems in the government and private sectors, including in products by Cerner, Eclipsis, EpicCare, and the Department of Veterans Affairs.¹⁵ ^65–68 Many of these structured documentation systems also allow users to enter narrative text in situations where they cannot find appropriate structured concepts.

While numerous structured entry systems were developed, deployed and described in the biomedical literature, there are limited data demonstrating ongoing adoption or widespread dissemination of structured entry outside of niche settings or clinical domains. Research evaluating why structured entry usage is limited remains sparse, but includes qualitative work by McDonald,²¹ Ash,⁶⁹ and Johnson.¹⁶ ⁶⁴ These studies suggest that structured entry systems can have complex interfaces that slow the user down. In addition, structured entry tends to be inflexible in situations where a documentation template does not contain a needed item and may not fully integrate with other EHR system components. Johnson's studies in particular demonstrated that structured entry users believed that the system helped them comply with clinical guidelines. Although it increased documentation time when compared to paper-based forms, it did not decrease clinician or patient satisfaction.¹⁶ ⁶⁴ Additional studies identified specific attributes of structured entry systems that can attenuate their efficiency, integration, user interface navigation, and overall user satisfaction.^70–73 Structured documentation systems using guideline-based templates may help healthcare providers to be thorough.⁶⁴ ⁷⁴ ⁷⁵

Using flexible documentation tools with text processing

With flexible documentation, healthcare providers record patient care episodes using relatively unstructured approaches, such as using dictation with transcription, speech-recognition software, or typing using a loosely templated CBD system. Once the clinical documentation is complete, post-hoc text processing algorithms can be used to produce structured data. We use the term ‘text processing’ to describe any of a number of methods designed to identify specific text, data, and concepts from the natural language stored in unstructured, narrative-text computer documents.^76–81 Computer programs can then deduce concepts contained in the notes' text in a subsequent step. Investigators have worked for decades to convert natural language into structured representations of those documents.^76–80 Text-processing technologies have been developed with differing levels of machine ‘understanding,’ from simple systems that search for key words or specific text strings^82–88 to those systems that attempt to capture clinical concepts with their context.⁷⁶ ⁸⁰ ^89–100 Text-processing tools can also serve as adjuncts to structured documentation systems, as recently described by Johnson discussing ‘structured narrative.’¹⁰¹

A basic approach to text processing involves searching documents in EHR systems for key narrative-text strings or string patterns. The narrative-text string search has been successfully employed in some large-scale clinical research studies.^82–85 For example, researchers have used plain text searches of dictated or typed medical records to find rare physical exam findings⁸² and post-operative infections,⁸⁵ and identify patients with certain types of drug-induced liver injury⁸⁴. Others have employed focused ‘regular expression’ pattern matches to extract text strings that can represent possible blood pressures⁸³ and common section header labels (eg, ‘chief complaint’) in clinical notes,⁸⁶ and remove patient identifiers such as names, phone numbers, and addresses to deidentify medical records.⁸⁷ These methods are best adapted for solving focused problems. Since string-matching algorithms can be highly tuned for a given task (eg, by including manually derived synonyms, common abbreviations, and even misspellings), they can perform very well. Simple text searching also has an advantage of faster processing speed and easier implementation than complex natural-language processing (NLP) systems, as many off-the-shelf database systems and text-indexing tools support preindexed text queries. However, simple text searching is limited by a lack of generalizability and requires substantial customization for each new task. For example, finding all patients with liver injury from a single medication (such as phenytoin) is easily accomplished with a simple text search, but finding all medications that may be associated with liver injury requires a more complicated system that matches text with a controlled vocabulary that would include all medication brand and generic names. In addition, problems associated with directly entered clinical notes, such as misspellings¹⁰² and ambiguous abbreviations (eg, ‘pt’ can stand for ‘patient’ or ‘physical therapy’),¹⁰³ also limit the use of the simple string-matching methods.

A more complex approach, called concept identification or concept indexing, attempts to normalize the text phrases to standardized terms representing concepts. Successful concept indexing systems include those by Miller and Cooper⁸⁹, MetaMap⁷⁶, SAPHIRE⁹⁰, the KnowledgeMap concept identifier,⁸⁰ ⁹¹ the Multi-threaded Clinical Vocabulary Server,⁹² ⁹³ ¹⁰⁴ and the recently released clinical Text Analysis and Knowledge Extraction System (cTAKES).¹⁰⁵ A goal of concept indexing is to ‘understand’ the information in natural-language documents by mapping text to standardized concepts using terminologies such as those in the Unified Medical Language System (UMLS).¹⁰⁶ These systems have proven effective, mapping natural-language texts to concepts with recall and precisions often exceeding 80% for general tasks and near perfect for some highly focused tasks.⁸⁰ ⁹⁵ ¹⁰⁷ ¹⁰⁸

Investigators have extended concept identification systems to combine them with additional algorithms that can extract contextual elements (eg, certainty, values, and temporal information) associated with a concept to form more robust NLP systems. Combining concept identification systems with negation detection algorithms (eg, ‘no chest pain,’ which indicates the absence of the finding ‘chest pain’), investigators have created systems that automatically generate problem lists,¹⁰⁹ discover gene–disease associations,¹¹⁰ hypothesize new drug effects¹¹¹ and new drug–drug interactions from the biomedical literature,⁹⁶ and identify important findings from clinical narratives.⁹⁷ ⁹⁸ Building on the work of the Linguistic String Project,⁷⁹ Friedman and colleagues have been developing the MedLEE (Medical Language Extraction and Encoding System) NLP system since the 1990s.⁷⁸ ⁹⁵ ¹¹² It can identify UMLS concepts from clinical documents, and can discern their timing and negation status. Researchers from several institutions have used MedLEE to identify pneumonia and other clinical conditions from chest x-rays,⁹⁹ detect adverse events,¹¹² and automatically calculate the Charlson comorbidity index.¹¹³ Several of these systems have been successfully incorporated into production clinical systems, some at multiple institutions.⁷⁸ ⁹⁵ ¹⁰⁹ ¹¹² ¹¹⁴ ¹¹⁵

The uses of NLP in the clinical domain are quite diverse, including encoding clinical notes for billing purposes,⁹⁵ ¹¹⁶ facilitating clinical research by automating the data-extraction processes,¹¹⁰ ¹¹⁷ ¹¹⁸ conducting EHR-based surveillance,⁹⁷ ¹¹⁹ and enriching EHR functionalities, such as to support visualization tools¹²⁰ and clinical decision-support systems.¹²¹ ¹²² One of the earliest examples is the MedLEE system,¹²³ which has been used to process chest radiographs to generate coded data for decision-support systems at New York Presbyterian Hospital since 1995. Recently, Day and colleagues¹²¹ reported the use of the MPLUS NLP system to classify trauma patients at a Level 1 trauma center on a daily basis. For specific tasks such as those mentioned above, advanced NLP systems such as MedLEE have shown equivalent performance as domain experts. However, the performance of current clinical NLP systems is still not satisfactory for broader uses. In addition, few NLP systems have been implemented in clinical settings and used in routine workflows in hospitals. For example, Kashyap et al, reported a study of using a commercial product to automatically structure admission notes in which the results were not judged to be acceptable for clinical use.¹²⁴ With further improvements in advanced NLP technologies and new structures for clinical notes (such as the ‘structured narrative’),¹⁰¹ NLP may be able to structure clinical text such as admission notes automatically with a satisfactory performance.

Discussion

The choice of a documentation method can alter the balance between expressivity and structure in the resultant notes, hamper the healthcare provider's workflow, influence the process and products of recording clinical information, and influence how well the note can be incorporated into an EHR system in such a way that the note's contents can be automatically reused and analyzed.¹⁵ ¹⁹ ²⁶ ³⁶ ⁷⁰ ¹⁰¹ While structured documentation systems can facilitate data collection and reuse, they can be cumbersome to use during patient encounters and may lack the flexibility and expressivity required for general medical practices. Transcribed notes create documents useful for text processing, but can require a time delay for the transcription process to occur. The attributes associated with each documentation method influence how they are best used and adopted. While structured entry emphasizes data standardization and structure, human adoption of CBD systems requires an emphasis on expressivity, efficiency, flexibility, and being well adapted to a typical workflow.¹⁹ ²² ²⁷

Both structured clinical documentation and text-processing algorithms for flexible documentation continue to evolve. The rates of evolution may not be the same for each, but we are not aware of clear evidence that shows that one approach is improving faster than the other. Furthermore, we do not intend to convey an impression that one is necessarily more advanced or has greater promise than the other. Each approach offers distinct advantages to the user. For example, because structured documentation systems can do more than just create data, it may be relatively appealing to clinicians in certain settings. A template-based structured documentation system may be useful for notes that have a standard and predictable content and format, such as those recording pediatric health maintenance visits, preoperative evaluations, formalized disability examinations, and reviews of systems. By contrast, in some settings, clinicians may prefer the flexibility of complex and nuanced narrative text to document the history of a present illness or a diagnostic impression.

We collectively have over 10 decades of personal experience in viewing or using a wide variety of computer-based documentation systems around the country in public, private and academic settings. We gained this experience in the settings of actual clinical use, observations during site visits, and direct reviews of vendor products at trade shows and professional conferences. Based on this experience, we have observed that typically, for a given site's implementation, computer-based documentation systems are usually configured to take clinical input primarily in narrative or structured form, but not both. Most computer-based documentation systems that the authors have seen do not support hybrid documentation in the way that we recommend in the current manuscript. We note that, in the case where structured clinical data are needed, information entered using structured entry would be immediately available, whereas narrative text would require processing via NLP algorithms to become similarly ‘structured.’ Nevertheless, we observe that in our experience, few nonacademic sites have NLP-based text processing systems readily available to harvest EHR system data, with the possible exception of NLP-type systems that help to extract billing codes from clinical records.

Priorities for structured documentation and text processing will likely vary in different settings and with different user groups, and the priorities may be informed by the tasks the documentation supports. Certain tasks may require that the EHR systems understand only structural information from the clinical document (eg, its metadata or header information), while others may require a deeper concept-level understanding. For example, a quality-assessment program evaluating whether inpatient progress notes are being placed in the chart in a timely manner may require only the document title and date. Any document that has the correct title, including scanned and tagged paper-based notes, could support this need. Tasks that depend on a deeper understanding of the document may require healthcare providers to document using structured documentation tools or to apply computer programs to process the natural language in narrative-text documents. For example, to implement an automated colorectal cancer screening advisor, the advisor system will need to know the patient's family and past medical history, whether a prior screening test had occurred, and the result of any screening tests. Gathering these data requires that healthcare providers document the relevant information using tools that can support data extraction and analysis, including CBD systems, structured entry tools, or narrative-text typing (either directly or via a dictation/transcription model) with a subsequent application of text processing. In addition, different healthcare providers may value structured data in EHR systems differently. Some providers may be willing to sacrifice a degree of CBD usability or efficiency for the sake of having key elements of their notes available immediately as structured data in EHR systems. Others may preferentially value documentation methods that allow them to express their impressions fluidly or create notes more quickly, without regard to how well the methods support other functions of EHR systems.

Recommendation

Given the tension between structure and flexible documentation, those implementing EHR systems should assume that multiple CBD products will be needed to meet the needs of clinician users, rather than attempting to find a single best documentation method. Factors to consider in selecting CBD products include the method's fitness for a given workflow, the content in the resultant notes, time efficiency to create a note, costs, ease of using the method, flexibility for using the method to document unforeseen clinical findings or in unexpected circumstances, and support for narrative expressivity, machine readability, and document structure.¹⁹ ²⁵ ⁴⁶ Certain documentation methods feature some attributes at the expense of others, such as promoting narrative expressivity at the expense of formal note structure. Rather than waiting until documentation tools can be created to accommodate all needs and workflows, or force complex clinical workflows to change to accommodate the EHR system rollout, we recommend that healthcare providers be able to access a variety of different documentation methods and select the one that best fits their documentation, data, and workflow needs. EHR system developers and users can weigh different documentation methods in terms of how they impact relevant documentation-related outcomes such as usability, efficiency, quality, and readability against their utility for EHR systems. The value of this approach is that it allows organizations considering EHR systems to prioritize development and implementation efforts around clinical documentation.

Footnotes

Funding: The project was supported by a grant from the United States National Library of Medicine (Rosenbloom, 1R01LM009591-01A1).

Competing interests: None.

Provenance and peer review: Not commissioned; externally peer reviewed.

References

1. Engle RL., Jr The evolution, uses, and present problems of the patient's medical record as exemplified by the records of the New York Hospital from 1793 to the present. Trans Am Clin Climatol Assoc 1991;102:182–9. [PubMed]

2. Weed LL. Quality control and the medical record. Arch Intern Med 1971;127:101–5. [PubMed]

3. DeGowin EL, DeGowin RL. Bedside Diagnostic Examination. 2nd edn New York: Macmillan, 1969.

4. Solomon DH, Schaffer JL, Katz JN, et al. Can history and physical examination be used as markers of quality? An analysis of the initial visit note in musculoskeletal care. Med Care 2000;38:383–91. [PubMed]

5. Hershberg PI, Goldfinger SE, Lemon FR, et al. Medical record as index of quality of care. N Engl J Med 1972;286:725–6. [PubMed]

6. Weed LJ. The problem oriented record as a basic tool in medical education, patient care and clinical research. Ann Clin Res 1971;3:131–4. [PubMed]

7. Fessel WJ, Van Brunt EE. Assessing quality of care from the medical record. N Engl J Med 1972;286:134–8. [PubMed]

8. Murphy JG, Jacobson S. Assessing the quality of emergency care: the medical record versus patient outcome. Ann Emerg Med 1984;13:158–65. [PubMed]

9. Holder AR. The importance of medical records. JAMA 1974;228:118–19. [PubMed]

10. Wagner MM, Bankowitz RA, McNeil M, et al. The diagnostic importance of the history and physical examination as determined by the use of a medical decision support system. SCAMC 1989:139–44.

11. Davis LS, Collen MF, Rubin L, et al. Computer-stored medical record. Comput Biomed Res 1968;1:452–69. [PubMed]

12. Hammond WE, Stead WW, Straube MJ, et al. Functional characteristics of a computerized medical record. Methods Inf Med 1980;19:157–62. [PubMed]

13. Schriger DL, Baraff LJ, Rogers WH, et al. Implementation of clinical guidelines using a computer charting system. Effect on the initial care of health care workers exposed to body fluids. JAMA 1997;278:1585–90. [PubMed]

14. Schriger DL, Baraff LJ, Buller K, et al. Implementation of clinical guidelines via a computer charting system: effect on the care of febrile children less than three years of age. J Am Med Inform Assoc 2000;7:186–95. [PMC free article] [PubMed]

15. Brown SH, Hardenbrook S, Herrick L, et al. Usability evaluation of the progress note construction set. Proc AMIA Symp 2001:76–80. [PMC free article] [PubMed]

16. Johnson KB, Serwint JR, Fagan LM, et al. Computer-based documentation: effect on parent and physician satisfaction during a pediatric health maintenance encounter. Arch Pediatr Adolesc Med 2005;159:250–4. [PubMed]

17. Stetson PD, Morrison FP, Bakken S, et al. Preliminary development of the physician documentation quality instrument. J Am Med Inform Assoc 2008;15:534–41. [PMC free article] [PubMed]

18. Stead WW, Lin HS, editors. , eds. Computational Technology for Effective Health Care: Immediate Steps and Strategic Directions. Washington, DC: National Research Council, 2009.

19. Rosenbloom ST, Crow AN, Blackford JU, et al. Cognitive factors influencing perceptions of clinical documentation tools. J Biomed Inform 2007;40:106–13. [PubMed]

20. Embi PJ, Yackel TR, Logan JR, et al. Impacts of computerized physician documentation in a teaching hospital: perceptions of faculty and resident physicians. J Am Med Inform Assoc 2004;11:300–9. [PMC free article] [PubMed]

21. McDonald CJ. The barriers to electronic medical record systems and how to overcome them. J Am Med Inform Assoc 1997;4:213–21. [PMC free article] [PubMed]

22. Ash JS, Bates DW. Factors and forces affecting EHR system adoption: report of a 2004 ACMI discussion. J Am Med Inform Assoc 2005;12:8–12. [PMC free article] [PubMed]

23. Ash JS, Berg M, Coiera E. Some unintended consequences of information technology in health care: the nature of patient care information system-related errors. J Am Med Inform Assoc 2004;11:104–12. [PMC free article] [PubMed]

24. Gilbert JA. Physician data entry: providing options is essential. Health Data Manag 1998;6:84–6, 8, 90–2. [PubMed]

25. Johnson KB, Ravich WJ, Cowan JA., Jr Brainstorming about next-generation computer-based documentation: an AMIA clinical working group survey. Int J Med Inform 2004;73:665–74. [PubMed]

26. Kabir A, Hanson R, Mellis CM, et al. Is asthma documentation improved by computer-facilitated data entry? J Qual Clin Pract 1998;18:187–93. [PubMed]

27. Rosenbloom ST, Grande J, Geissbuhler A, et al. Experience in implementing inpatient clinical note capture via a provider order entry system. J Am Med Inform Assoc 2004;11:310–15. [PMC free article] [PubMed]

28. Shiffman RN, Brandt CA, Freeman BG. Transition to a computer-based record using scannable, structured encounter forms. Arch Pediatr Adolesc Med 1997;151:1247–53. [PubMed]

29. Salomon RM, Blackford JU, Rosenbloom ST, et al. Openness of patients' reporting with use of electronic records: psychiatric clinicians' views. J Am Med Inform Assoc 2010;17:54–60. [PMC free article] [PubMed]

30. Rector AL. Thesauri and formal classifications: terminologies for people and machines. Methods Inf Med 1998;37:501–9. [PubMed]

31. Rector AL, Nowlan WA, Kay S. Foundations for an electronic medical record. Methods Inf Med 1991;30:179–86. [PubMed]

32. Rosenbloom ST, Miller RA, Johnson KB, et al. Interface terminologies: facilitating direct entry of clinical data into electronic health record systems. J Am Med Inform Assoc 2006;13:277–88. [PMC free article] [PubMed]

33. Fox KM, Reuland M, Hawkes WG, et al. Accuracy of medical records in hip fracture. J Am Geriatr Soc 1998;46:745–50. [PubMed]

34. Marill KA, Gauharou ES, Nelson BK, et al. Prospective, randomized trial of template-assisted versus undirected written recording of physician records in the emergency department. Ann Emerg Med 1999;33:500–9. [PubMed]

35. Cawsey AJ, Webber BL, Jones RB. Natural language generation in health care. J Am Med Inform Assoc 1997;4:473–82. [PMC free article] [PubMed]

36. van Ginneken AM. The physician's flexible narrative. Methods Inf Med 1996;35:98–100. [PubMed]

37. Miller PL. Critiquing anesthetic management: the ‘ATTENDING’ computer system. Anesthesiology 1983;58:362–9. [PubMed]

38. Shultz E, Rosenbloom T, Kiepek W, et al. Quill: a novel approach to structured reporting. AMIA Annu Symp Proc 2003:1074. [PMC free article] [PubMed]

39. van Mulligen EM, Stam H, van Ginneken AM. Clinical data entry. Proc AMIA Symp 1998:81–5. [PMC free article] [PubMed]

40. Blumenthal D, Glaser JP. Information technology comes to medicine. N Engl J Med 2007;356:2527–34. [PubMed]

41. Jeans WD, Danton RM, Kilburn AR. An assessment of a computerized reporting system (SIREP). Br J Radiol 1980;53:421–7. [PubMed]

42. Greenes RA, Barnett GO, Klein SW, et al. Recording, retrieval and review of medical data by physician-computer interaction. N Engl J Med 1970;282:307–15. [PubMed]

43. Kahn CE., Jr Self-documenting structured reports using open information standards. Medinfo 1998;9:403–7. [PubMed]

44. Spackman K., Snomed RT and SNOMEDCT Promise of an international clinical terminology. MD Comput 2000;17:29. [PubMed]

45. Spackman KA. Rates of change in a large clinical terminology: three years experience with SNOMED Clinical Terms. AMIA Annu Symp Proc 2005:714–18. [PMC free article] [PubMed]

46. Rosenbloom ST, Kiepek W, Belletti J, et al. Generating complex clinical documents using structured entry and reporting. Stud Health Technol Inform 2004;107:683–7. [PubMed]

47. Slack WV, Hicks GP, Reed CE, et al. A computer-based medical-history system. N Engl J Med 1966;274:194–8. [PubMed]

48. Slack WV, Peckham BM, Van Cura LJ, et al. A computer-based physical examination system. JAMA 1967;200:224–8. [PubMed]

49. Ledley RS. Computer aids to medical diagnosis. JAMA 1966;196:933–43. [PubMed]

50. Stead WW, Heyman A, Thompson HK, et al. Computer-assisted interview of patients with functional headache. Arch Intern Med 1972;129:950–5. [PubMed]

51. Stead WW, Brame RG, Hammond WE, et al. A computerized obstetric medical record. Obstet Gynecol 1977;49:502–9. [PubMed]

52. Stead WW, Hammond WE, Estes EH. Evaluation of an audio mode of the automated medical history. Methods Inf Med 1977;16:20–3. [PubMed]

53. Hammond WE. How the past teaches the future: ACMI distinguished lecture. J Am Med Inform Assoc 2001;8:222–34. [PMC free article] [PubMed]

54. Barnett GO. COSTAR, a computer-based medical information system for ambulatory care. Proc IEEE 1979;67:1226–37.

55. Stratmann WC, Goldberg AS, Haugh LD. The utility for audit of manual and computerized problem-oriented medical record systems. Health Serv Res 1982;17:5–26. [PMC free article] [PubMed]

56. Wirtschafter DD, Scalise M, Henke C, et al. Do information systems improve the quality of clinical research? Results of a randomized trial in a cooperative multi-institutional cancer group. Comput Biomed Res 1981;14:78–90. [PubMed]

57. Shortliffe EH. Update on ONCOCIN: a chemotherapy advisor for clinical oncology. Med Inform (Lond) 1986;11:19–21. [PubMed]

58. Bruce B, Fries JF. The Stanford Health Assessment Questionnaire: a review of its history, issues, progress, and documentation. J Rheumatol 2003;30:167–78. [PubMed]

59. Blum RL. Computer-assisted design of studies using routine clinical data. Analyzing the association of prednisone and cholesterol. Ann Intern Med 1986;104:858–68. [PubMed]

60. CORI: Clinical Outcomes Research Initiative Publications List. http://www.cori.org/?topic=publications

61. CORI: Clinical Outcomes Research Initiative. http://www.cori.org.

62. Musen MA, Carlson RW, Fagan LM, et al. T-HELPER: automated support for community-based clinical research. Proc Annu Symp Comput Appl Med Care 1992:719–23. [PMC free article] [PubMed]

63. Rosenbloom ST, Stead WW, Denny JC, et al. Generating Clinical Notes for Electronic Health Record Systems. Appl Clin Inform 2010;1:232–43(accepted 28 Jun 2010). [PMC free article] [PubMed]

64. Johnson KB, Cowan J. Clictate: a computer-based documentation tool for guideline-based care. J Med Syst 2002;26:47–60. [PubMed]

65. Logician Ambulatory. http://www.gehealthcare.com.

66. Epic Systems Corporation. http://www.epicsys.com/

67. Physician Micro Systems, Incorporated: Practice Partner Patient Records.

68. Brown SH, Lincoln MJ, Groen PJ, et al. VistA—US Department of Veterans Affairs national-scale HIS. Int J Med Inf 2003;69:135–56. [PubMed]

69. Ash JS, Berg M, Coiera E. Some unintended consequences of information technology in health care: the nature of patient care informations system related errors. J Am Med Inform Assoc 2004;11:104–12. [PMC free article] [PubMed]

70. Poon AD, Fagan LM, Shortliffe EH. The PEN-Ivory project: exploring user-interface design for the selection of items from large controlled vocabularies of medicine. J Am Med Inform Assoc 1996;3:168–83. [PMC free article] [PubMed]

71. Cimino JJ, Patel VL, Kushniruk AW. Studying the human–computer–terminology interface. J Am Med Inform Assoc 2001;8:163–73. [PMC free article] [PubMed]

72. Elkin PL, Mohr DN, Tuttle MS, et al. Standardized problem list generation, utilizing the Mayo canonical vocabulary embedded within the Unified Medical Language System. Proc AMIA Annu Fall Symp 1997:500–4. [PMC free article] [PubMed]

73. McKnight LK, Elkin PL, Ogren PV, et al. Barriers to the clinical implementation of compositionality. Proc AMIA Symp 1999:320–4. [PMC free article] [PubMed]

74. Biondich PG, Overhage JM, Dexter PR, et al. A modern optical character recognition system in a real world clinical setting: some accuracy and feasibility observations. Proc AMIA Symp 2002:56–60. [PMC free article] [PubMed]

75. Shiffman RN, Liaw Y, Navedo DD, et al. User satisfaction and frustration with a handheld, pen-based guideline implementation system for asthma. Proc AMIA Symp 1999:940–4. [PMC free article] [PubMed]

76. Aronson AR. Effective mapping of biomedical text to the UMLS Metathesaurus: the MetaMap program. Proc AMIA Symp 2001:17–21. [PMC free article] [PubMed]

77. Chapman WW, Bridewell W, Hanbury P, et al. A simple algorithm for identifying negated findings and diseases in discharge summaries. J Biomed Inform 2001;34:301–10. [PubMed]

78. Friedman C. Towards a comprehensive medical language processing system: methods and issues. Proc AMIA Annu Fall Symp 1997:595–9. [PMC free article] [PubMed]

79. Sager N, Lyman M, Bucknall C, et al. Natural language processing and the representation of clinical data. J Am Med Inform Assoc 1994;1:142–60. [PMC free article] [PubMed]

80. Denny JC, Smithers JD, Miller RA, et al. ‘Understanding’ medical school curriculum content using KnowledgeMap. J Am Med Inform Assoc 2003;10:351–62. [PMC free article] [PubMed]

81. Meystre SM, Savova GK, Kipper-Schuler KC, et al. Extracting information from textual documents in the electronic health record: a review of recent research. Yearb Med Inform 2008:128–44. [PubMed]

82. Denny JC, Arndt FV, Dupont WD, et al. Increased hospital mortality in patients with bedside hippus. Am J Med 2008;121:239–45. [PubMed]

83. Turchin A, Kolatkar NS, Grant RW, et al. Using regular expressions to abstract blood pressure and treatment intensification information from the text of physician notes. J Am Med Inform Assoc 2006;13:691–5. [PMC free article] [PubMed]

84. Jinjuvadia K, Kwan W, Fontana RJ. Searching for a needle in a haystack: use of ICD-9-CM codes in drug-induced liver injury. Am J Gastroenterol 2007;102:2437–43. [PubMed]

85. Michelson J. Improved detection of orthopaedic surgical site infections occurring in outpatients. Clin Orthop Relat Res 2005;433:218–24. [PubMed]

86. Meystre S, Haug PJ. Automation of a problem list using natural language processing. BMC Med Inform Decis Mak 2005;5:30. [PMC free article] [PubMed]

87. Gupta D, Saul M, Gilbertson J. Evaluation of a deidentification (De-Id) software engine to share pathology reports and clinical documents for research. Am J Clin Pathol 2004;121:176–86. [PubMed]

88. Johnson K, Poon A, Shiffman S, et al. A history-taking system that uses continuous speech recognition. Proc Annu Symp Comput Appl Med Care 1992:757–61. [PMC free article] [PubMed]

89. Cooper GF, Miller RA. An experiment comparing lexical and statistical methods for extracting MeSH terms from clinical free text. J Am Med Inform Assoc 1998;5:62–75. [PMC free article] [PubMed]

90. Hersh WR, Donohoe LC. SAPHIRE International: a tool for cross-language information retrieval. Proc AMIA Symp 1998:673–7. [PMC free article] [PubMed]

91. Denny JC, Smithers JD, Armstrong B, et al. ‘Where do we teach what?’ Finding broad concepts in the medical school curriculum. J Gen Intern Med 2005;20:943–6. [PMC free article] [PubMed]

92. Elkin PL, Brown SH, Bauer BA, et al. A controlled trial of automated classification of negation from clinical notes. BMC Med Inform Decis Mak 2005;5:13. [PMC free article] [PubMed]

93. Elkin PL, Brown SH, Husser CS, et al. Evaluation of the content coverage of SNOMED CT: ability of SNOMED clinical terms to represent clinical problem lists. Mayo Clin Proc 2006;81:741–8. [PubMed]

94. Nadkarni P, Chen R, Brandt C. UMLS concept indexing for production databases: a feasibility study. J Am Med Inform Assoc 2001;8:80–91. [PMC free article] [PubMed]

95. Friedman C, Shagina L, Lussier Y, et al. Automated encoding of clinical documents based on natural language processing. J Am Med Inform Assoc 2004;11:392–402. [PMC free article] [PubMed]

96. Duda S, Aliferis C, Miller R, et al. Extracting drug–drug interaction articles from MEDLINE to improve the content of drug databases. AMIA Annu Symp Proc 2005:216–20. [PMC free article] [PubMed]

97. Chapman WW, Cooper GF, Hanbury P, et al. Creating a text classifier to detect radiology reports describing mediastinal findings associated with inhalational anthrax and other disorders. J Am Med Inform Assoc 2003;10:494–503. [PMC free article] [PubMed]

98. Chapman WW, Fizman M, Chapman BE, et al. A comparison of classification algorithms to automatically identify chest X-ray reports that support pneumonia. J Biomed Inform 2001;34:4–14. [PubMed]

99. Hripcsak G, Kuperman GJ, Friedman C, et al. A reliability study for evaluating information extraction from radiology reports. J Am Med Inform Assoc 1999;6:143–50. [PMC free article] [PubMed]

100. Denny JC, Spickard A, 3rd, Johnson KB, et al. Evaluation of a method to identify and categorize section headers in clinical documents. J Am Med Inform Assoc 2009;16:806–15. [PMC free article] [PubMed]

101. Johnson SB, Bakken S, Dine D, et al. An electronic health record based on structured narrative. J Am Med Inform Assoc 2008;15:54–64. [PMC free article] [PubMed]

102. Ruch P, Baud R, Geissbuhler A. Using lexical disambiguation and named-entity recognition to improve spelling correction in the electronic patient record. Artif Intell Med 2003;29:169–84. [PubMed]

103. Xu H, Stetson PD, Friedman C. A study of abbreviations in clinical notes. AMIA Annu Symp Proc 2007:821–5. [PMC free article] [PubMed]

104. Savova GK, Coden AR, Sominsky IL, et al. Word sense disambiguation across two domains: biomedical literature and clinical notes. J Biomed Inform 2008;41:1088–100. [PubMed]

105. Savova GK, Masanz JJ, Ogren PV, et al. Mayo clinical Text Analysis and Knowledge Extraction System (cTAKES): architecture, component evaluation and applications. J Am Med Inform Assoc 2010;17:507–13. [PMC free article] [PubMed]

106. Committee on Quality of Health Care in America, Using Information Technology Crossing the Quality Chasm: A New Health System for the 21st Century. Washington, DC: IOM, 2001.

107. Lowe HJ, Antipov I, Hersh W, et al. Automated semantic indexing of imaging reports to support retrieval of medical images in the multimedia electronic medical record. Methods Inf Med 1999;38:303–7. [PubMed]

108. Denny JC, Spickard A, 3rd, Miller RA, et al. Identifying UMLS concepts from ECG Impressions using KnowledgeMap. AMIA Annu Symp Proc 2005:196–200. [PubMed]

109. Meystre S, Haug PJ. Natural language processing to extract medical problems from electronic clinical documents: performance evaluation. J Biomed Inform 2006;39:589–99. [PubMed]

110. Ritchie MD, Denny JC, Crawford DC, et al. Robust replication of genotype–phenotype associations across multiple diseases in an electronic medical record. Am J Hum Genet 2010;86:560–72. [PubMed]

111. Chen ES, Hripcsak G, Xu H, et al. Automated acquisition of disease drug knowledge from biomedical and clinical documents: an initial study. J Am Med Inform Assoc 2008;15:87–98. [PMC free article] [PubMed]

112. Melton GB, Hripcsak G. Automated detection of adverse events using natural language processing of discharge summaries. J Am Med Inform Assoc 2005;12:448–57. [PMC free article] [PubMed]

113. Chuang JH, Friedman C, Hripcsak G. A comparison of the Charlson comorbidities derived from medical language processing and administrative data. Proc AMIA Symp 2002:160–4. [PMC free article] [PubMed]

114. Denny JC, Irani PR, Wehbe FH, et al. The KnowledgeMap project: development of a concept-based medical school curriculum database. AMIA Annu Symp Proc 2003:195–9. [PMC free article] [PubMed]

115. Denny JC, Bastarache L, Sastre EA, et al. Tracking medical students' clinical experiences using natural language processing. J Biomed Inform 2009;42:781–9. [PubMed]

116. Baud R. A natural language based search engine for ICD10 diagnosis encoding. Med Arh 2004;58(1 Suppl 2):79–80. [PubMed]

117. Xu H, Anderson K, Grann VR, et al. Facilitating cancer research using natural language processing of pathology reports. Stud Health Technol Inform 2004;107:565–72. [PubMed]

118. Pakhomov S, Weston SA, Jacobsen SJ, et al. Electronic medical records for clinical research: application to the identification of heart failure. Am J Manag Care 2007;13:281–8. [PubMed]

119. Brown SH, Elkin PL, Rosenbloom ST, et al. eQuality for all: Extending automated quality measurement of free text clinical narratives. AMIA Annu Symp Proc 2008:71–5. [PMC free article] [PubMed]

120. Cimino JJ, Bright TJ, Li J. Medication reconciliation using natural language processing and controlled terminologies. Stud Health Technol Inform 2007;129:679–83. [PubMed]

121. Day S, Christensen LM, Dalto J, et al. Identification of trauma patients at a level 1 trauma center utilizing natural language processing. J Trauma Nurs 2007;14:79–83. [PubMed]

122. Jain NL, Knirsch CA, Friedman C, et al. Identification of suspected tuberculosis patients based on natural language processing of chest radiograph reports. Proc AMIA Annu Fall Symp 1996:542–6. [PMC free article] [PubMed]

123. Friedman C, Alderson PO, Austin JH, et al. A general natural-language text processor for clinical radiology. J Am Med Inform Assoc 1994;1:161–74. [PMC free article] [PubMed]

124. Kashyap V, Turchin A, Morin L, et al. Creation of structured documentation templates using Natural Language Processing techniques. AMIA Annu Symp Proc 2006:977. [PMC free article] [PubMed]

Articles from Journal of the American Medical Informatics Association : JAMIA are provided here courtesy of
American Medical Informatics Association