PMCCPMCCPMCC

Search tips
Search criteria 

Advanced

 
Logo of procamiaLink to Publisher's site
 
AMIA Annu Symp Proc. 2011; 2011: 520–528.
Published online 2011 October 22.
PMCID: PMC3243287

GEM at 10: A Decade’s Experience with the Guideline Elements Model

Abstract

The Guideline Elements Model (GEM) was developed in 2000 to organize the information contained in clinical practice guidelines using XML and to represent guideline content in a form that can be understood by human readers and processed by computers. In this work, we systematically reviewed the literature to better understand how GEM was being used, potential barriers to its use, and suggestions for improvement. Fifty external and twelve internally produced publications were identified and analyzed. GEM was used most commonly for modeling and ontology creation. Other investigators applied GEM for knowledge extraction and data mining, for clinical decision support for guideline generation. The GEM Cutter software—used to markup guidelines for translation into XML— has been downloaded 563 times since 2000. Although many investigators found GEM to be valuable, others critiqued its failure to clarify guideline semantics, difficulties in markup, and the fact that GEM files are not usually executable.

BACKGROUND

The Guideline Elements Model (GEM) was first developed in 2000 to store and organize the heterogeneous knowledge and information contained in clinical practice guidelines. GEM provides an intermediate knowledge representation that permits natural language guidelines to be translated into a format that can be processed by computers. GEM uses XML to describe a comprehensive set of pertinent concepts, relationships between concepts, and attributes. The resulting guideline representation can be used for multiple purposes including incorporation into decision support systems, electronic guideline distribution, and guideline querying. The GEM Cutter editor was designed to facilitate markup of guideline text and to facilitate its translation into XML. It was intended for an array of guidelines users including developers, disseminators, implementers, quality appraisers, and end users.

GEM’s XML representation has been updated to accommodate evolving needs. GEM was initially described by an XML Document Type Definition in 2002. In 2006, the model was augmented and published as an XML Schema. Both representations underwent standardization by ASTM International.

Ten years after its introduction (1) we sought to systematically review the literature to identify how GEM was being used outside the laboratory where it was created and to identify barriers to its use and suggestions for how it might be improved.

METHODS

In June 2010, with the assistance of a medical librarian, we conducted a literature search using three databases: PubMed, SCOPUS, and ISI Web of Knowledge. Search terms used were: ‘GEM AND guideline elements model’ which resulted in 54 articles (see Figure 1). To further expand our search we used the original publication in which GEM was described (1), and searched for articles with related citations, or articles in which the original paper was referenced. Non-English journals were excluded. This expanded search resulted in 322 articles. After eliminating duplicates, 159 articles remained. Because of our interest in diffusion of the model, the 12 articles that were authored by Yale Center for Medical Informatics (YCMI) collaborators were set aside to be analyzed separately. Of the 147 remaining articles 59 could not be accessed (i.e., articles that were not electronically retrievable as full texts or available locally) and 38 articles were not about GEM, i.e. GEM was not mentioned in the text or referenced in the bibliography (e.g., geology articles). This resulted in a final list of 50 articles which were analyzed by all the authors (Figure 1). Articles were categorized according to subject matter and critiques of GEM were extracted for further analysis (Table 2). We reviewed a sample of 20 papers to help define categories of how GEM was being used. These subject categories were iteratively refined dependent upon major categories identified in the 20 papers, and continuously evalulated as the remaining articles were reviewed.

Figure 1.
Diagram of literature review
Table 2.
Manuscripts divided by how GEM was used and by the subject of the manuscript. 2a. Non-YCMI authored manuscripts; 2b. YCMI authored manuscripts. The uses of GEM are depicted in the columns and include articles in which Guideline Elements Model was applied ...

In addition, we queried the GEM website registry to characterize the incident downloads of the GEM Cutter software over the past 5 years. GEM Cutter has been freely available after a simple registration process.

RESULTS

The Guideline Elements Model was used or referenced in 50 manuscripts arising from 8 countries: Italy, Israel, Canada, Australia, France, Sweden, Austria and the United States. Review of the website registry of GEM Cutter downloads revealed 563 incident downloads since 2006 (see Table 1), suggesting that GEM has been applied by many users beyond those who have published their work.

Table 1.
Number of GEM Cutter downloads per year.

Categories of GEM use

We identified 4 categories of use of GEM: (1) Ontology/Modeling refers to papers primarily concerned with the development of an ontology or a model; (2) Knowledge extraction/retrieval/NLP/data mining refers to articles that are concerned with the extraction of information from guidelines using natural language processing or data mining; (3) Clinical Decision support/guideline application includes articles concerned with the application of guidelines in a clinical decision support system; and (4) Guideline generation refers to articles concerned with the authoring of clinical guidelines. The categories were found to be inclusive for all the articles reviewed, but were not mutually exclusive.

The most common application of GEM was for modeling or ontology creation (14/50 articles, 13 in which the model was applied and 1 in which the tool was used). GEM was applied in six articles for knowledge extraction/data mining, and in two articles for clinical decision support/guideline knowledge application. Two papers described application of GEM-based tools for guideline generation. The remaining articles did not use GEM, (either considered GEM but chose not to use it, or only made reference to GEM).

Articles in which GEM was considered but not used

We found 12 papers in which use of GEM was considered and rejected, 10 of which were considering its use for modeling/ontology creation. In most cases, the authors indicated the reason why GEM was not selected (see below). GEM was referenced without discussion in 15 review articles about modeling/ontology creation, guideline generation, or clinical decision support.

YCMI publications

Publications from the collaborators at the Yale Center for Medical Informatics included 4 articles in which both the GEM model and GEM related tools were applied for knowledge extraction/data mining, and/or clinical decision support/guideline knowledge application (Table 2b). The GEM model alone was applied in 5 articles and GEM-related tools were applied in 4 articles. These original research articles explored GEM’s used in modeling, knowledge extraction, clinical decision support and guideline generation. One review paper used GEM as an example (2).

CRITIQUES

GEM has been praised as “the leader” in the document-centered approach to guideline knowledge representation (3). In contrast to the model-centered approach to guideline formalization—a conceptual model in which the relationship between the original document and the model is only indirect—the document-centered approach takes the original guideline text as a starting point and constrains the translation process toward congruence with the source knowledge.

Several publications praised the detailed specification of the processes used for authoring guidelines: “GEM is considerably richer than other systems in specifying details of the guideline development process (e.g. authors, purpose, intended audience, versions, and the consensus process)” (4); and GEM’s explicit modeling of the evidence level of each part of the guideline (5). Users also appreciated GEM’s support of knowledge components “The Guideline Elements Model (GEM) is chosen… since it supports a broad range of knowledge elements, is easily extendable (it is based on XML), and its mark-up can be easily read and understood by domain professionals” (6). In addition, guideline developers recognized GEM as useful for maximizing the validity and utility of their products. “Yale Center for Medical Informatics used specialized software (GEM-COGS transforms, GLIA) to appraise adherence of the draft guideline to methodologic standards, to improve clarity of recommendations, and to predict potential obstacles to implementation. These steps helped to further maximize the validity and utility of the resulting guideline”(7).

GEM-Cutter was identified as useful for preventing errors when transitioning to newer versions of guidelines “(GEM Cutter) might limit errors that result from forgetting to represent part of the narrative guideline or to copy part of the clinical algorithm or sidebars to the next version. A tool like GEM-Cutter could be used to mark-up narrative guidelines using GEM elements. The tool could be used to view unmarked parts of the guideline, thus aiding in limiting omission errors”(8).

Several publications offered specific criticisms of the model. Four themes emerged from these critiques.

Critiques about GEM’s Failure to Clarify Guideline Semantics

Several authors pointed to a need for GEM to promote semantic clarification of narrative guideline text. One criticism was that the representation did not fulfill terminology needs (910), specifically, that “GEM does little to resolve ambiguities … the model is simply an abstraction of the guideline document” (10). Other authors expressed a desire that GEM include diagnosis and procedure codes, “GEM doesn’t provide adequate facilities for the coding of diagnoses, procedures etc.”(12), and suggested that GEM should allow for terminology standards (13). An additional criticism about GEM output was that the organization of the original text was not retained in the XML output.

Critiques about the GEM Markup process

The GEM Cutter XML editor applies a copy and paste metaphor to markup and parse original guideline text into GEM elements and create an XML document. Some authors noted that GEM’s markup process was time-consuming (14) and that there was substantial variation in the results (10)(15). The quantity of effort required for this markup process is directly related to the comprehensiveness desired of the final XML document.

Critiques about GEM Execution

Several authors critiqued GEM for not incorporating a formal computational model aimed at guideline execution. Critics noted the absence of control structures and temporal elements, which would allow the logic of a guideline to unfold over time: “[GEM] does not seem to support extended care over significant time periods, due to the lack of persistent-memory mechanisms, interaction with an electronic patient record, and complex control structures” (16); “[GEM] misses the logic of a guideline that unfolds over time” (17). In addition GEM was critiqued for not having workflow awareness: “…for electronic guidelines to be useful, the knowledge and logic components must be structured so that the semantic relationships between the knowledge components and the execution logic/workflow are clearly defined” (18).

Miscellaneous Critiques

Some authors claimed that GEM did not use attributes (“although GEM also tries to differentiate between “explicit” and “inferred” guideline text, it doesn’t consistently use XML concepts, especially XML attributes”) (12); was not hierarchical (19); and did not include the level of scientific evidence (“GEM doesn’t provide adequate facilities for the coding of diagnoses, procedures etc. or for adding essential information, such as the level of scientific evidence of a recommendation”) (12).

DISCUSSION

As GEM approaches the end of its first decade of use, we note that the Guideline Elements Model has been applied for a wide variety of purposes and achieved mention in a substantial number of research reports. We presume that there have been additional uses that have not resulted in publication, as suggested by the 921 downloads of GEM Cutter.

In reviewing the literature, we found a number of concerns that have been put forth regarding the model. Recognizing these critiques as potential limitations of the model we sought to categorize and understand them.

With respect to concerns about GEM’s inability to clarify guideline semantics, we note that a goal of the model was to improve transparency of the knowledge transformation process. GEM was intended to provide a standardized intermediate representation that categorizes and labels the original guideline content, but not interpret that text. Retaining the original text allows users to judge the accuracy of the transformed representation. Improving the clarity of the guideline content was seen as a separate process that is aided by a number of XSL transforms (collectively named EXTRACTOR transforms), which help to highlight vagueness and under-specification and provide a substrate to which standardized terminology and concept codes may be attached. In addition, each GEM element includes an attribute that indicates whether it is original text (“explicit”) or modified text (“inferred”).

As a declarative model, documents generated using GEM were not designed to be directly executable. Many guidelines do not define a sequence under which their recommendations are to be performed. Although control structures are not well developed in the model, the <algorithm> element contains <action>, <conditional>, <branch>, and <synchronization> subelements (20). In addition, the <link> element can be used to define a sequence of actions. Gershkovitch and colleagues demonstrated that a GEM-processed guideline can indeed be directly executed using an execution engine devised for that purpose, although they agree that the process is inefficient and ‘artificial’ (21).

The markup process is indeed time-consuming, particularly for those unfamiliar with the model. However, it has been shown that the time to perform markup diminished substantially with experience(22). To reduce the variation in markup, GEM incorporates a set of clear definitions for its elements (displayed in GEM Cutter) and the authors have worked with users to define a set of markup conventions that have increased consistency.

Finally, with respect to the comments characterized as “miscellaneous,” we note that GEM is a hierarchical model that incorporates several attributes (including source, unique identifier,and language) in every element definition. In addition, the model includes multiple elements that define the level of scientific evidence including evidence.quality and recommendation.strength at the level of individual recommendations and Description.evidence.collection, Method.evidence.collection, Number.source.documents, Evidence.time.period, Method.evidence.grading; Rating.scheme; Method.evidence.combination; Specification.harm.benefit; Quantification.harm.benefit; Role.value.judgment ; and Role.patient.preference (1).

This study was limited in that we were unable to access electronically a substantial number of papers that were identified by our queries. It is likely that several of these papers were not relevant to this review, but it is possible that additional themes might have been uncovered and we may have had a degree of sampling bias

In summary, our review of published studies and GEM cutter downloads suggests that in its first decade of use, the Guideline Elements Model is being applied internationally. Understanding limitations of the model may help to set expectations about its use at a reasonable level. Overall GEM continues to lead the field of document-centered approaches to guideline modeling and knowledge transformation.

Acknowledgments

We are grateful for the support of the Agency for Healthcare Quality and Research for its support through contract HHSA 290200810011 and the National Library of Medicine for its support through 2R01LM007199 and Training grant T-15LM07065.

References

(1) Shiffman RN, Karras BT, Agrawal A, Chen R, Marenco L, Nath S. GEM: a proposal for a more comprehensive guideline document model using XML. JAmMedInformAssoc. 2000 Sep-Oct;7(5):488–498. [PMC free article] [PubMed]
(2) Hussain T, Michel G, Shiffman RN. The Yale Guideline Recommendation Corpus: A representative sample of the knowledge content of guidelines. Int.J.Med.Informatics. 2009;78(5):354–363. [PubMed]
(3) Svatek V, Ruzicka M. Step-by-step mark-up of medical guideline documents. Stud.Health Technol.Inform. 2002;90:591–595. [PubMed]
(4) Sonnenberg FA, Hagerty CG. Computer-interpretable clinical practice guidelines. Where are we and where are we going. YearbMedInform. 2006:145–158. [PubMed]
(5) Seyfang A, Miksch S, Polo-Conde C, Wittenberg J, Marcos M, Rosenbrand K. MHB - A many-headed bridge between informal and formal guideline representations. Lect Notes Comput Sci. 2005;3581:146–150. 23 July 2005 through 27 July 2005; LNAI:
(6) Medintel: Decision support for general practitioners: A case study. 22nd International Conference on Medical Informatics Europe, MIE 2009; 30 August 2009 through 2 September 2009; 2009.
(7) Rosenfeld RM. Commentary: Guidelines and otolaryngology. OtolaryngolHeadNeckSurg. 2006;134(4 SUPPL.) [PubMed]
(8) Peleg M, Patel VL, Snow V, Tu S, Mottur-Pilson C, Shortliffe EH, et al. Support for guideline development through error classification and constraint checking. ProcAMIASymp; 2002. pp. 607–611. [PMC free article] [PubMed]
(9) Georg G, Seroussi B, Bouaud J. Does GEM-encoding clinical practice guidelines improve the quality of knowledge bases? A study with the rule-based formalism. AMIAAnnuSympProc; 2003. pp. 254–258. [PMC free article] [PubMed]
(10) Georg G, Seroussi B, Bouaud J. Interpretative framework of chronic disease management to guide textual guideline GEM-encoding. Stud.Health Technol.Inform. 2003;95:531–536. [PubMed]
(11) Seroussi B, Bouaud J, Chatellier G. Guideline-based modeling of therapeutic strategies in the special case of chronic diseases. Int.J.Med.Informatics. 2005;74(2–4):89–99. [PubMed]
(12) Hoelzer S, Schweiger R, Dudeck J. Representation of practice guidelines with XML--modeling with XML schema. Methods Inf.Med. 2002;41(4):305–312. [PubMed]
(13) Tu SW, Campbell JR, Glasgow J, Nyman MA, McClure R, McClay J, et al. The SAGE Guideline Model: Achievements and Overview. J.Am.Med.Informatics Assoc. 2007;14(5):589–598. [PMC free article] [PubMed]
(14) A document engineering environment for clinical guidelines. DocEng’07: 2007 ACM Symposium on Document Engineering; 28 August 2007 through 31 August 2007; 2007.
(15) Kaiser K, Miksch S. Modeling treatment processes using information extraction. Stud. Comput. Intell. 2007;48:189–224.
(16) Shahar Y, Young O, Shalom E, Galperin M, Mayaffit A, Moskovitch R, et al. A framework for a distributed, hybrid, multiple-ontology clinical-guideline library, and automated guideline-support tools. J.Biomed.Informatics. 2004;37(5):325–344. [PubMed]
(17) Mulyar N, van der Aalst WMP, Peleg M. A Pattern-based Analysis of Clinical Computer-interpretable Guideline Modeling Languages. J.Am.Med.Informatics Assoc. 2007;14(6):781–787. [PMC free article] [PubMed]
(18) Chu S. Guideline representation formalism and electronic decision support systems: Addressing the guideline - Implementation gap - Part 2. HealthcRevOnline. 2005;9(1)
(19) Kumar A, Quaglini S, Stefanelli M, Ciccarese P, Caffi E. Modular representation of the guideline text: An approach for maintaning and updating the content of medical education. Med.Informatics Internet Med. 2003;28(2):99–115. [PubMed]
(20) Boxwala AA, Peleg M, Tu S, Ogunyemi O, Zeng QT, Wang D, et al. GLIF3: a representation format for sharable computer-interpretable clinical practice guidelines. JBiomedInform. 2004 Jun;37(3):147–161. [PubMed]
(21) Gershkovich P, Shiffman RN. An implementation framework for GEM encoded guidelines. ProcAMIASymp. 2001:204–208. [PMC free article] [PubMed]
(22) Haskell Lisa T, Monteforte Mark. Applying the Guideline Elements Model (GEM) Cutter II Tool to Guidelines Represented in the National Guidelines Clearinghouse. Jul 30, 2010. pp. 1–25.
(23) An extensible framework for sharing clinical guidelines and services. Conference Proceedings - 26th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, EMBC 2004; 1 September 2004 through 5 September 2004; 2004.
(24) Del Fiol G, Rocha RA, Bradshaw RL, Hulse NC, Roemer LK. An XML model that enables the development of complex order sets by clinical experts. IEEE Trans.Inf.Technol.Biomed. 2005;9(2):216–228. [PubMed]
(25) Hagerty CG, Pickens DS, Chang J, Kulikowski CA, Sonnenberg FA. Prediction in annotation based guideline encoding. AMIAAnnuSympProc; 2006; pp. 314–318. [PMC free article] [PubMed]
(26) Eduljee A, Rocha RA. Practical evaluation of clinical guideline document models. AMIAAnnuSympProc; 2003. p. 836. [PMC free article] [PubMed]
(27) Evidence based knowledge retrieval from CPG knowledge base by context and statistical analysis techniques. 2nd International Conference on the Applications of Digital Information and Web Technologies, ICADIWT 2009; 4 August 2009 through 6 August 2009; 2009.
(28) Toussi M, Lamy J-, Le Toumelin P, Venot A. Using data mining techniques to explore physicians’ therapeutic decisions when clinical guidelines do not provide recommendations: Methods and example for type 2 diabetes. BMC MedInformatics DecisMak. 2009;9(1) [PMC free article] [PubMed]
(29) Abidi SR, Shepherd M. A clinical practice guideline-based ontology driven clinical decision support system for breast cancer follow-up. Journal on Information Technology in Healthcare. 2008 Fri, 24 Sep.
(30) Hussain S, Abidi SSR. An ontology-based framework for authoring and executing clinical practice guidelines for clinical decision support systems. J.Inf.Technol.Healthc. 2008;6(1):8–22.
(31) Semantic Web framework for knowledge-centric clinical decision support systems. 11th Conference on Artificial Intelligence in Medicine, AIME 2007; 7 July 2007 through 11 July 2007; 2007.
(32) Raza Abidi SS, Kershaw M, Milios E. Augmenting GEM-encoded clinical practice guidelines with relevant best evidence autonomously retrieved from MEDLINE. Health Informatics J. 2005;11(2):95–110.
(33) Integrating document-based and knowledge-based models for clinical guidelines analysis. 11th Conference on Artificial Intelligence in Medicine, AIME 2007; 7 July 2007 through 11 July 2007; 2007.
(34) Chittaro L, Ranon R, Carchietti E, Zampa A, Biasutti E, De Marco L, et al. A knowledge-based system to support emergency medical services for disabled patients. Lect Notes Comput Sci. 2009;5651:176–180. 18 July 2009 through 22 July 2009. LNAI:
(35) Ciccarese P, Caffi E, Quaglini S, Stefanelli M. Architectures and tools for innovative Health Information Systems: The guide project. Int.J.Med.Informatics. 2005;74(7–8):553–562. [PubMed]
(36) Ganslandt T, Mueller ML, Krieglstein CF, Senninger N, Prokosch HU. An XML-based system for the flexible classification and retrieval of clinical practice guidelines. ProcAMIASymp; 2002. pp. 280–284. [PMC free article] [PubMed]
(37) Schwartz SR, Cohen SM, Dailey SH, Rosenfeld RM, Deutsch ES, Gillespie MB, et al. Clinical practice guideline: Hoarseness (Dysphonia) OtolaryngolHeadNeckSurg. 2009;141(3 SUPPL. 2) [PubMed]
(38) Roland PS, Smith TL, Schwartz SR, Rosenfeld RM, Ballachanda B, Earll JM, et al. Clinical practice guideline: Cerumen impaction. OtolaryngolHeadNeckSurg. 2008;139(3 SUPPL. 2) [PubMed]
(39) Anselma L, Terenziani P, Montani S, Bottrighi A. Towards a comprehensive treatment of repetitions, periodicity and temporal constraints in clinical guidelines. Artif.Intell.Med. 2006;38(2):171–195. [PubMed]
(40) Boxwala AA, Tu S, Peleg M, Zeng Q, Ogunyemi O, Greenes RA, et al. Toward a representation format for sharable clinical guidelines. J.Biomed.Informatics. 2001;34(3):157–169. [PubMed]
(41) Svátek V, Růžička M. Step-by-step formalisation of medical guideline content. Lek.Tech. 2003;34(4):145–153.
(42) Choi J, Currie LM, Wang D, Bakken S. Encoding a clinical practice guideline using guideline interchange format: A case study of a depression screening and management guideline. IntJMedInformatics. 2007;76(SUPPL. 2) [PubMed]
(43) Peleg M, Boxwala AA, Tu S, Zeng Q, Ogunyemi O, Wang D, et al. The InterMed Approach to Sharable Computer-interpretable Guidelines: A Review. J.Am.Med.Informatics Assoc. 2004;11(1):1–10. [PMC free article] [PubMed]
(44) Kaiser K, Akkaya C, Miksch S. How can information extraction ease formalizing treatment processes in clinical practice guidelines?. A method and its evaluation. Artif.Intell.Med. 2007;39(2):151–163. [PMC free article] [PubMed]
(45) Toward the validation of patient data for clinical practice guidelines. 2009 6th International Conference on Electrical Engineering, Computing Science and Automatic Control, CCE 2009; 10 November 2009 through 13 November 2009; 2009.
(46) Ruzicka M, Svatek V. Mark-up based analysis of narrative guidelines with the Stepper tool. Stud.Health Technol.Inform. 2004;101:132–136. [PubMed]
(47) Formalizing ’living guidelines’ using LASSIE: A multi-step information extraction method. 11th Conference on Artificial Intelligence in Medicine, AIME 2007; 7 July 2007 through 11 July 2007; 2007.
(48) “On-demand” Access to a multi-purpose collection of best practice standards. Conference Proceedings - 26th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, EMBC 2004; 1 September 2004 through 5 September 2004; 2004. [PubMed]
(49) Terenziani P, Molino G, Torchio M. A modular approach for representing and executing clinical guidelines. Artif.Intell.Med. 2001;23(3):249–276. [PubMed]
(50) From rhetorical structures to document structure: Shallow pragmatic analysis for document engineering. 9th ACM Symposium on Document Engineering, DocEng’09; 15 September 2009 through 18 September 2009; 2009.
(51) Schweiger R, Hoelzer S, Altmann U, Rieger J, Dudeck J. Plug-and-play XML: a health care perspective. JAmMedInformAssoc. 2002 Jan-Feb;9(1):37–48. [PMC free article] [PubMed]
(52) Montani S, Terenziani P. Exploiting decision theory concepts within clinical guideline systems: Toward a general approach. Int J Intell Syst. 2006;21(6):585–599.
(53) Eriksson H, Tu SW, Musen M. Semantic clinical guideline documents. AMIAAnnuSympProc; 2005. pp. 236–240. [PMC free article] [PubMed]
(54) Kendall E, Sunderland N, Muenchberger H, Armstrong K. When guidelines need guidance: Considerations and strategies for improving the adoption of chronic disease evidence by general practitioners. J.Eval.Clin.Pract. 2009;15(6):1082–1090. [PubMed]
(55) Montani S. Case-based reasoning for managing noncompliance with clinical guidelines. Comput.Intell. 2009;25(3):196–213.
(56) 2008. Visualization methods to support guideline-based care management.
(57) Goud R, Hasman A, Strijbis AM, Peek N. A parallel guideline development and formalization strategy to improve the quality of clinical practice guidelines. IntJMedInform. 2009 Aug;78(8):513–520. [PubMed]
(58) Tran N, Michel G, Krauthammer M, Shiffman RN. Embedding the guideline elements model in web ontology language. AMIAAnnuSympProc; 2009 Nov 14; 2009. pp. 645–649. [PMC free article] [PubMed]
(59) Essaihi A, Michel G, Shiffman RN. Comprehensive categorization of guideline recommendations: creating an action palette for implementers. AMIAAnnuSympProc; 2003. pp. 220–224. [PMC free article] [PubMed]
(60) Agrawal A, Shiffman RN. Using GEM-encoded guidelines to generate medical logic modules. ProcAMIASymp; 2001. pp. 7–11. [PMC free article] [PubMed]
(61) Shiffman RN, Michel G, Essaihi A, Thornquist E. Bridging the guideline implementation gap: A systematic, document-centered approach to guideline implementation. J.Am.Med.Informatics Assoc. 2004;11(5):418–426. [PMC free article] [PubMed]
(62) Shiffman RN, Michel G. Toward improved guideline quality: using the COGS statement with GEM. Stud.Health Technol.Inform. 2004;107(Pt 1):159–163. [PubMed]
(63) Shiffman RN, Michel G, Essaihi A, Marcy TW. Using a guideline-centered approach for the design of a clinical decision support system to promote smoking cessation. Stud.Health Technol.Inform. 2004;101:152–156. [PubMed]
(64) Karras BT, Nath SD, Shiffman RN. A preliminary evaluation of guideline content mark-up using GEM--an XML guideline elements model. ProcAMIASymp; 2000. pp. 413–417. [PMC free article] [PubMed]
(65) Agrawal A, Shiffman RN. Evaluation of guideline quality using GEM-Q. Stud.Health Technol.Inform. 2001;84(Pt 2):1097–1101. [PubMed]
(66) Rosenfeld RM, Andes D, Bhattacharyya N, Cheung D, Eisenberg S, Ganiats TG, et al. Clinical practice guideline: Adult sinusitis. OtolaryngolHeadNeckSurg. 2007;137(3 SUPPL.) [PubMed]
(67) Rosenfeld RM, Brown L, Cannon CR, Dolor RJ, Ganiats TG, Hannley M, et al. Clinical practice guideline: Acute otitis externa. OtolaryngolHeadNeckSurg. 2006;134(4 SUPPL.) [PubMed]

Articles from AMIA Annual Symposium Proceedings are provided here courtesy of American Medical Informatics Association