AsthmaCritic: Issues in Designing a Noninquisitive Critiquing System for Daily Practice

doi:10.1197/jamia.M1273

J Am Med Inform Assoc. 2003 Sep-Oct; 10(5): 419–424.

PMCID: PMC212778

AsthmaCritic: Issues in Designing a Noninquisitive Critiquing System for Daily Practice

Manon M. Kuilboer, MD, MSc, Marc A. M. van Wijk, MD, PhD, Mees Mosseveld, MSc, and Johan van der Lei, MD, PhD

Affiliation of the authors: Department of Medical Informatics, Erasmus MC, University Medical Centre Rotterdam, The Netherlands.

Correspondence and reprints: Marc A. M. van Wijk, MD, PhD, Department of Medical Informatics, Erasmus MC, University Medical Centre Rotterdam, P.O. Box 1738, 3000 DR Rotterdam, The Netherlands; e-mail: <m.vanwijk/at/erasmusmc.nl>.

Received October 15, 2002; Accepted May 15, 2003.

This article has been cited by other articles in PMC.

Abstract

The noninquisitive critiquing system, AsthmaCritic, uses routinely recorded electronic patient data to select and analyze records of patients with asthma or chronic obstructive pulmonary disease (COPD). The system generates critiquing comments and adds these comments to the patient record. The system was developed by using and expanding an existing generic critiquing system. After a brief overview of the system, this report focuses on the authors' design choices in light of existing literature and the issues that underlie these design choices. Currently, AsthmaCritic is used by primary care physicians and is undergoing clinical evaluation.

Asthma and chronic obstructive pulmonary disease (COPD) are chronic diseases with a high prevalence and account for significant health care expenditures.¹ Professional health care organizations disseminate paper-based guidelines, reflecting the “state of the art” in medical care.²^,³ However, paper-based guidelines have had disappointingly little impact on physicians' behavior.⁴^,⁵ To change daily practice, dissemination of guidelines needs to be combined with an appropriate implementation strategy.⁶ One possible implementation strategy is to implement guidelines in computer-based decision-support systems (CDSSs)⁷^,⁸ that help the practitioner to manage patients with a particular disease by using the appropriate guidelines and protocols. Although researchers have shown that computed decision support is able to change the delivery of health care, the number of systems in daily use is limited.⁷^,⁹^,¹⁰^,¹¹^,¹²^,¹³^,¹⁴ Some investigators argue that the use of electronic patient records will provide new opportunities for decision support¹¹^,¹⁵^,¹⁶ by providing a natural way to integrate decision support in everyday practice.⁷^,¹⁴^,¹⁷

We designed and built a critiquing system, AsthmaCritic, for use in the domains of asthma and COPD, taking “integration into daily practice” as a precondition. In this report, we first describe the design and implementation of AsthmaCritic. We subsequently justify our design choices in light of the available literature.

AsthmaCritic

In The Netherlands, most general practitioners have replaced their paper medical records with electronic patient records.¹⁸ The general practitioner records data during consultation. Based on that data, AsthmaCritic provides feedback by generating a critique of the physicians' diagnosis and treatment plan for patients with asthma or COPD. AsthmaCritic is a noninquisitive critiquing system that solely relies on the data it receives from the electronic patient record and that does not ask for additional data entry. AsthmaCritic runs autonomously; that is, the system is triggered when the physician sees an asthma or COPD patient,^* starts analyzing the record when the consultation is finished, and then presents the critique. From the physician's viewpoint, AsthmaCritic is part of his or her medical-record system; when AsthmaCritic reports its critique, the interface is identical to the interface of the medical-record system (that is, screen- and data-manipulation is handled in the same fashion).

AsthmaCritic distinguishes between two different kinds of feedback: critique and transformed clinical measurement. A critique is presented to the physician as a patient-specific comment based on the current clinical situation. AsthmaCritic first presents a brief overview of all comments, ranked by clinical urgency and potential impact.

gives an example of a patient with COPD and a decreasing forced expiratory volume in 1 second (FEV₁) measurement. AsthmaCritic recognizes the decrease (in

: “Recent FEV₁ guideline for COPD patients”) and, because the patient is not yet receiving corticosteroids, suggests the possibility of treatment with inhaled corticosteroids (in

: “Option: inh. Corticos”). In

, the physician selects “Option: inh. Corticos,” and AsthmaCritic responds with a brief explanation. When the physician requests additional information (in

, the option “<A> Add. Info”), the system provides, based on the Dutch guidelines, the reasoning underlying this recommendation (

Figure 1.

For each of the short statements in the overview the physician can request the complete comment.

Figure 2.

For each comment the physician can request additional information.

Transformed clinical measurements constitute AsthmaCritic's second kind of comment; calculations on measurements are performed and presented to the user. Peak-flow values, for example, are presented to the physician in an overview that includes the original values, the expected value for this patient based on gender, age, and height, and the original values expressed as a percentage of the expected value.

The general practitioner controls the behavior of AsthmaCritic. The physician decides when to review the feedback: during the patient encounter, at some later time, or at the patient's next visit. When AsthmaCritic is running, it may, depending on the locally available hardware, slow down the electronic patient-record system. Therefore, the physician can interrupt the current analysis; interrupting the analysis initiates a background activity that continues processing the record. If the analysis of a record is done in background mode, a patient-specific message presenting the system's critique is attached to the medical record that is displayed the next time the physician opens the record. The general practitioner's electronic patient-record system keeps a log of the records that still have to be analyzed by AsthmaCritic and a log of critiques that have not yet been shown to the physician.

A description of AsthmaCritic's functionality and knowledge base is available to the physicians on paper. For quick reference, a small laminated yellow reference card shows on one side an overview of structured patient data relevant for asthma and COPD and, on the other side, the system's main control functions and helpdesk phone numbers.

AsthmaCritic Implementation

AsthmaCritic's implementation is based on the generic critiquing model of HyperCritic.²¹ This generic critiquing model supports the integration of a critiquing system with an electronic patient record. HyperCritic's ability to process time-stamped data was limited. In addition, HyperCritic presented its output as text, often several pages long, with no opportunity for the physician to control the behavior of the system. Therefore, HyperCritic lacked structures that enabled integration of the system in the physician's working environment and was never tested in daily practice.

AsthmaCritic's implementation differs from HyperCritic's by (1) increasing the use of time-stamped data, (2) allowing the physician to control the output, and (3) supporting functions that allow a better integration with the electronic patient record (for example, running the system in background mode, attaching messages to a patient's records, adding the results of the analysis to the medical record, or monitoring what feedback still has to be shown to the physician).

The generic critiquing model distinguishes between critiquing knowledge (knowledge that initiates and guides the critiquing process) and medical knowledge (the medical base for critiquing). The analysis of patient data and medical knowledge is controlled by critiquing knowledge. Critiquing knowledge is described as a hierarchical set of critiquing tasks. Each individual critiquing task describes a specific procedure that defines for which clinical situations a critiquing statement should be generated.

Critiquing tasks are triggered by event descriptions that identify the criteria that have to be met by the patient data for the task to be executed. Executing the critiquing task may result in a critiquing statement.

, for example, shows a critiquing task that is executed whenever a drug is prescribed. This task searches for interactions between drugs. If such an interaction is found, the critiquing statement “possible interaction” is generated.²¹

Figure 3.

Example of a critiquing task's specifications. If a clinical situation, as described by its event descriptions, matches a critiquing task's specifications, feedback is generated.

AsthmaCritic's critiquing knowledge is divided into four categories of critiquing tasks: diagnostic tasks, therapy-related tasks, referral-related tasks, and follow-up–related tasks. Each category is further subdivided into more specific tasks. For example, a therapy-related task contains nine different subtasks (such as checking for contraindications, checking for drug interactions, verifying dosing schedules). In total, AsthmaCritic contains 131 specific tasks.

Critiquing tasks, however, are solely procedural specifications. Based on the existing medical knowledge, the total number of different clinical situations that can be distinguished is much higher. For example, screening for contraindications is a single task, but the number of clinical situations that will be detected depends on the number of drugs and contraindications known in present-day medicine. As such, critiquing tasks only specify the process of a critique, not the content. For the critiquing statement to be generated, medical knowledge needs to be available (for example, dosing schedules, side effects, interactions).

AsthmaCritic's medical knowledge base has been built from national guidelines,²²^,²³^,²⁴ pharmaceutical reference books,²⁵ guides on interactions and side effects,²⁶ and the existing consensus among a group of specialists in asthma and COPD. Building the knowledge base was a three-year iterative process under the guidance of a medical content board consisting of four local specialists (two general practitioners and two pulmonologists) and seven national specialists. Members of the medical-content board reviewed each new version of the knowledge base.^†^‡

Design Choices

In building a decision-support system, designers have to make numerous decisions. The available literature includes little data to guide those decisions. System builders have to select, without prior verification of the consequences or feasibility, between the available options. In the case of AsthmaCritic, only one design decision was tested prior to building the system. That decision involved whether the system could be made noninquisitive or, phrased differently, whether the electronic patient record of the general practitioner contained sufficient medical data to enable the generation of critique without asking additional information. In a simulation study, we showed that the medical record did contain enough information,³⁵ based on that simulation study, we decided to build a noninquisitive system. When building the system, numerous other design decisions were made based on personal judgment and intuition. Throughout the design process, we were guided by three principles: (1) strive for a consistent information-technology environment, (2) respect the user as a professional, and (3) respect the limitations of a busy practice.

Consistent Environment

Physicians use a number of systems in their work—the decision-support system is only one of them. Researchers have argued that a computerized decision-support system should be integrated with an electronic patient record system to avoid reentering medical data³⁴^,³⁵ and reduce the number of workflow interruptions.¹¹^,²⁷^,²⁸^,²⁹^,³⁰ Others have emphasized that data entry itself should be minimal, that forced data entry should be avoided,³¹^,³²^,³³ and that the decision-support system should not be solely dependent on the physician's initiative to activate the system.⁹^,³³ From the physician's viewpoint, the different systems should, ideally, be integrated and present the user with a consistent environment.

In designing AsthmaCritic, we made sure that both the electronic patient records and the decision-support system could be used from the same physical location (in

, “Location Singularity”) without the user's having to switch applications (“Application Consistency”). In addition, we decided that the interfaces to the electronic record system and the decision-support system should be, as far as possible, identical (in

, “Interface Consistency”). AsthmaCritic is a noninquisitive system that relies only on routinely recorded data and does not ask the user for additional data (in

, “Data-Entry Effort”). Finally, AsthmaCritic continuously monitors the electronic patient record for data that may warrant detailed analysis of the record (in

, “Active Case Finding”).

Table 1

Issues and Design Choices in the Phases “Generating Output” and “Using Output”

The User as Professional

As early as 1989, Shortliffe³⁶ described “loss of control” as one of the barriers to the introduction of decision-support systems. Perhaps this sense of loss of control was partly due to the fear physicians had about being replaced by “expert systems,” which, in turn, leads to a rejection of the decision-support concept altogether.³⁶^,³⁷ Taylor³⁸ described another form of loss of control that involves the psychology of being denied the reward of using one's own skills. Treating the user as a professional also means that the decision-support system is able to explain its recommendations. Previous experience has shown that users' trust in generated advice increases if the system is able to justify its recommendations.⁴³^,⁴⁴

We designed AsthmaCritic as a critiquing system that provides a physician with feedback based on the physician's treatment plan as recorded in the patient's electronic patient record. He or she makes the decision. The system thus preserves the physician's professional autonomy by leaving him or her in control of the decision-making process (in

, “Professional Autonomy”). In addition, AsthmaCritic's feedback explains why it was generated and gives the underlying reasoning, including references to the source material (e.g., references to the appropriate sections of the guidelines) or the members of the medical content board responsible for reviewing that section of the knowledge base (in

, “Information Justification”).

Limitations of a Busy Practice

In daily practice, physicians often do not have the time to read all available information and to select the specific information that they need.⁴¹ Medical practice is characterized by (relative) information overload, and the available time and information requirements vary for each person.³⁹^,⁴⁰ If physicians are expected to read and process information when time is limited, the amount of information has to be limited as well.⁴² Lobach and Hammond,⁹ for example, reports that physicians prefer a telegraphic style when guideline information is presented. Timing and freedom are important issues; physicians should be able to read or interrupt reading the information provided at any moment.⁴⁵ Forcing physicians into a specific structure may lead to rejection.³¹

In designing AsthmaCritic, we gave the user control over timing. That is, the user can at any time interrupt AsthmaCritic. However, AsthmaCritic records that interruption and allows the user to continue at a later moment (

, “Timing”). The feedback is presented initially in a short telegraphic style (in

, “Information Conciseness”), and the user controls what additional information she or he wants to read at any given moment (in

, “Personal Autonomy”).

Discussion

AsthmaCritic reuses a previously published generic critiquing model that supports integration with an electronic patient record.²¹ However, the prototype of this generic model, HyperCritic, was never tested in daily practice. AsthmaCritic develops the critiquing model further by supporting additional functions to accommodate requirements for use in daily practice.

Of all the design decisions made in the development of AsthmaCritic, only one single decision was verified experimentally prior to building the system: the decision to build a noninquisitive system. We wanted to be able to integrate AsthmaCritic with the general practitioner's information system; AsthmaCritic should not interrupt the user to request specific or additional data. However, the availability of medical data depends on the ability and willingness of the physician to record data in a structured fashion.⁴⁶ These recording habits vary a great deal. The question was whether, given this variability, the choice for noninquisitiveness was feasible. Therefore, in a previous study, we investigated the feasibility of using data routinely recorded in electronic patient records for the generation of patient-specific feedback.³⁵ We concluded that enough structured data were available to generate relevant feedback for general practitioners. In the study, we also investigated the need to ask physicians for missing data.⁴⁷ Our study showed that information that was missed by reviewers often was available elsewhere but did not make much difference in the generation of comments by AsthmaCritic.⁴⁷

The critiquing process can be divided into two stages: first, the generation of critique followed by using that critique. With design issues playing a role, we minimized user dependency during the generation of critique, and we maximized user dependency during the phase of using the critique. That is, during the phase of generating output, control by the physician is undesirable, but during the phase of using the output, control by the physician is mandatory.⁴⁸

summarizes the issues and our choices for each phase.

The decisions all have their pros and cons. For example, the decision to tightly integrate the decision-support system with a medical-record system requires tailoring to that system. Using AsthmaCritic together with another electronic record system requires an effort that includes developing a new interface consistent with that record system. Ideally, a system developer knows how each choice influences system operation. If he or she knows these relationships, design choices can be optimized, considering the constraints and options of the system's intended working environment. That knowledge, however, often is not available. We believe that further research into the relationship between CDSSs and their operating environments is needed.

Currently, AsthmaCritic is the subject of a randomized clinical trial in which we measure the impact of the system in actual use. Results of that study will show whether the system has an impact on physician performance. It will not measure the impact, if any, of specific design decisions. We believe that trials comparing alternative methods of building comparable system functionality are required to begin to understand the consequences of each individual design decision.

Notes

Supported by The Netherlands Asthma Foundation (#92.62). The authors thank their local medical content board: B. Ponsioen, MD, Prof. E. van der Does, MD, PhD, Prof. J. C. de Jongste, MD, PhD, and S. E. Overbeek, MD, PhD, and their national medical content board: B. Bottema, MD, PhD, P. N. R. Dekhuyzen, MD, PhD, E. J. Duiverman, MD, PhD, E. E. M. van Essen, MD, PhD, Th. B. Voorn, MD, PhD, M. H. J. Vaessen, MD, and A. van der Kuy, MD, PhD, for their part in the development of AsthmaCritic knowledge base. The authors also thank M. Kroneman, MSc, PhD, for her valuable contributions to this report.

Footnotes

^*AsthmaCritic is triggered by the ICPC codes (International Classification of Primary Care¹⁹) for asthma (R96), chronic bronchitis (R91), emphysema (R95), other chronic pulmonary diseases (R83.4), and the ATC codes (Anatomical, Therapeutic and Chemical coding system of the World Health Organization²⁰) for prescriptions used in the treatment of asthma or COPD (R03).

^†Dr. B. Ponsioen, Professor Dr. E. van der Does, Dr. J. C. de Jongste, Dr. S. E. Overbeek.

^‡Dr. B. Bottema, Dr. P. N. R. Dekhuyzen, Dr. E. J. Duiverman, Dr. E. E. M. van Essen, Dr. Th. B. Voorn, Dr. M. H. J. Vaessen, Dr. A. van der Kuy.

References

1. Rutten-van Molken MP, Postma MJ, Joore MA, et al. Current and future medical costs of asthma and chronic obstructive pulmonary disease in The Netherlands. Respir Med. 1999;93:779–87. [PubMed]

2. Podell RN. National guidelines for the management of asthma in adults. Am Fam Physician. 1992;46:1189–96. [PubMed]

3. Geijer RMM, Thiadens HA, Smeele IJM, et al. NHG-Standaard COPD en astma bij volwassenen: diagnostiek [Guidelines of the Dutch College of General Practitioners: Chronic Obstructive Respiratory Diseases and Asthma in Adults: Diagnostics]. Huisarts Wetenschap. 1997;40:415–28 .

4. Crim C. Clinical practice guidelines vs actual clinical practice. The asthma paradigm. Chest. 2000;118(suppl):62S–4S. [PubMed]

5. Smeele IJ, Van Schayk CP, Van Den Bosch WJ, et al. [Discrepancy between the guidelines and practice by family physicians in treating adults with an exacerbation of asthma or chronic obstructive pulmonary disease]. Ned Tijdschr Geneeskd. 1998;142:2304–8. [PubMed]

6. Forrest D, Hoskins A, Hussey R. Clinical guidelines and their implementation. Postgrad Med J. 1996;72:19–22. [PMC free article] [PubMed]

7. Hunt DL, Haynes RB, Hanna SE, Smith K. Effects of computer-based clinical decision support systems on physician performance and patient outcomes: a systematic review. JAMA. 1998;280:1339–46. [PubMed]

8. van Wijk M, Mosseveld M, van der Lei J. Design of a decision support system for test ordering in general practice: choices and decisions to make. Methods Inf Med. 1999;38(4–5):355–61. [PubMed]

9. Lobach DF, Hammond WE. Computerized decision support based on a clinical practice guideline improves compliance with care standards. Am J Med. 1997;102(1):89–98. [PubMed]

10. Haynes RB, Sackett DL, Gray JM, Cook DJ, Guyatt GH. Transferring evidence from research into practice: 1. The role of clinical care research evidence in clinical decisions [editorial]. ACP J Club. 1996;125(3):A1416. [PubMed]

11. Elson RB, Connelly DP. Computerized decision-support systems in primary care. Prim Care. 1995;22:365–84. [PubMed]

12. Mitchell E, Sullivan F. A descriptive feast but an evaluative famine: systematic review of published articles on primary care computing during 1980–97. Br Med J. 2001;322:279–82. [PMC free article] [PubMed]

13. Zielstorff RD. Online practice guidelines: issues, obstacles, and future prospects. J Am Med Inform Assoc. 1998;5:227–36. [PMC free article] [PubMed]

14. Johnston ME, Langton KB, Haynes RB, Mathieu A. Effects of computer-based clinical decision-support systems on clinician performance and patient outcome: a critical appraisal of research. Ann Intern Med. 1994;120:135–42. [PubMed]

15. Tierney WM, Overhage JM, Takesue BY, et al. Computerizing guidelines to improve care and patient outcomes: the example of heart failure. J Am Med Inform Assoc. 1995;2:316–22. [PMC free article] [PubMed]

16. Henry SB, Douglas K, Galzagorry G, Lahey A, Holzemer WL. A template-based approach to support utilization of clinical practice guidelines within an electronic health record. J Am Med Inform Assoc. 1998;5:237–44. [PMC free article] [PubMed]

17. Smith BJ, McNeely MD. The influence of an expert system for test ordering and interpretation on laboratory investigations. Clin Chem. 1999;45(8 pt 1):1168–75. [PubMed]

18. van der Lei J, Duisterhout JS, Westerhof HP, et al. The introduction of computer-based patient records in The Netherlands. Ann Intern Med. 1993;119:1036–41. [PubMed]

19. Boersma JJ. ICPC: International Classification of Primary Care: Short Titles and Dutch Subtitles. The Netherlands: Huisartsen Genootschap, 1995.

20. WHO International Working Group for Drug Statistics Methodology. Anatomical therapeutic chemical (ATC) classification and defined daily dose (DDD) for pharmaceuticals and vitamins. 1996; <www.whocc.nmd.no/> Accessed: June 13, 2002.

21. van der Lei J, Musen MA. A model for critiquing based on automated medical records. Comput Biomed Res. 1991;24:344–78. [PubMed]

22. Dirksen WJ, Geyer RMM, De Haan M, et al. NHG Standaard Astma bij Kinderen. [Guidelines of the Dutch College of General Practitioners: Asthma in Children]. Huisarts Wetenschap. 1992;3535562.

23. Bottema BJAM, Fabels EJ, Van Grunsven PM, et al. NHG Standaard CARA bij Volwassenen: Diagnostiek [Guidelines of the Dutch College of General Practitioners: Chronic Respiratory Diseases in Adults: Diagnostics]. Huisarts Wetenschap. 1992;35:430–6.

24. Waart van der MAC, Dekker FW, Nijhoff S, et al. NHG Standaard CARA bij Volwassenen: Behandeling [Guidelines of the Dutch College of General Practitioners: Chronic Respiratory Diseases in Adults: Therapy]. Huisarts Wetenschap. 1992;35:437–43.

25. Kuy van der A (ed). Farmacotherapeutisch Kompas [Pharmaceutical Reference Book], Utrecht: Boekhoven-Bosch, 1997..

26. van Akkerveen HM, Gierde JJ, Hagenzieker MA, et al. Commentaren Medicatiebewaking Pharmacom en Medicom [Comments in Medication Alerts Pharmacom and Medicom]. Houten: Stichting Health Base, 1994.

27. Lobach DF, Underwood HR. Computer-based decision support systems for implementing clinical practice guidelines. Drug Benefit Trends. 1998;10(10):48–53.

28. Leader WG, Pestotnik SL, Chandler MH. Integrating pharmacokinetics into point-of-care information systems. Clin Pharmacokinet. 1996;31:165–73. [PubMed]

29. Gadd CS, Baskaran P, Lobach DF. Identification of design features to enhance utilization and acceptance of systems for Internet-based decision support at the point of care. Proc AMIA Symp. 1998:91–5. [PMC free article] [PubMed]

30. Shiffman RN. Towards effective implementation of a pediatric asthma guideline: integration of decision support and clinical workflow support. Proc Annu Symp Comput Appl Med Care. 1994:797–801. [PMC free article] [PubMed]

31. Wiederholt G, Perreault LE. Hospital information systems. In: Shortliffe EHS, Perreault LE (eds.). Medical Informatics. Computer Applications in Health Care. Menlo Park, CA: Addison-Wesley Publishing Company, 1990, p. 237.

32. Porcelli PJ, Lobach DF. Integration of clinical decision support with on-line encounter documentation for well child care at the point of care [in process citation]. Proc AMIA Symp. 1999:599–603. [PMC free article] [PubMed]

33. Aronsky D, Haug PJ. An integrated decision support system for diagnosing and managing patients with community-acquired pneumonia [in process citation]. Proc AMIA Symp. 1999:197–201. [PMC free article] [PubMed]

34. Ebell MH, Gaspar DL, Khurana S. Family physicians' preferences for computerized decision-support hardware and software [see comments]. J Fam Pract. 1997;45:137–41. [PubMed]

35. Kuilboer MM, van der Lei J, de Jongste J, Overbeek S, Ponsioen B, van Bemmel JH. Exploring the role of an integrated critiquing system: a simulation. J Am Med Inform Assoc. 1998;5:194–202. [PMC free article] [PubMed]

36. Shortliffe EH. Testing reality: the introduction of decision-support technologies for physicians. Methods Inf Med. 1989;28(1):1–5. [PubMed]

37. Miller RA, Masarie FE, Jr. The demise of the “Greek Oracle” model for medical diagnostic systems. Methods Inf Med. 1990;29:1–2. [PubMed]

38. Taylor TR. The computer and clinical decision-support systems in primary care. J Fam Pract. 1990;30:137–40. [PubMed]

39. Tang PC, Jaworski MA, Fellencer CA, et al. Methods for assessing information needs of clinicians in ambulatory care. Proc Annu Symp Comput Appl Med Care. 1995:630–4. [PMC free article] [PubMed]

40. Fafchamps D, Young CY, Tang PC. Modelling work practices: Input to the design of a physician's workstation. Proc Annu Symp Comput Appl Med Care. 1991:788–92. [PMC free article] [PubMed]

41. Noone J, Warren J, Brittain M. Information overload: opportunities and challenges for the general practitioner's desktop. Medinfo. 1998;9(pt 2):1287–91. [PubMed]

42. McDonald CJ. Protocol-based computer reminders, the quality of care and the non-perfectibility of man. N Engl J Med. 1976;295:1351–5. [PubMed]

43. Teach RL, Shortliffe EH. An analysis of physician attitudes regarding computer-based clinical consultation systems. Comput Biomed Res. 1981;14:542–58. [PubMed]

44. Morris AH. Developing and implementing computerized protocols for standardization of clinical decisions. Ann Intern Med. 2000;132:373–83. [PubMed]

45. Tierney WM, Hui SL, McDonald CJ. Delayed feedback of physician performance versus immediate reminders to perform preventive care. Effects on physician compliance. Med Care. 1986;24:659–66. [PubMed]

46. Grol R, Thomas S, Roberts R. Development and implementation of guidelines for family practice: lessons from The Netherlands [editorial]. J Fam Pract. 1995;40:435–439. [PubMed]

47. Kuilboer MM, van der Lei J, Bohnen AM, van Bemmel J. The availability of unavailable information. AMIA Annu Fall Symp. 1997:749–53. [PMC free article] [PubMed]

48. Vlug A. MEDEUR homepage. <http://www.eur.nl/fgg/mi/medeur/> Accessed July 14, 2002.

Articles from Journal of the American Medical Informatics Association : JAMIA are provided here courtesy of
American Medical Informatics Association