PMCCPMCCPMCC

Search tips
Search criteria 

Advanced

 
Logo of procamiaLink to Publisher's site
 
AMIA Annu Symp Proc. 2011; 2011: 878–887.
Published online 2011 October 22.
PMCID: PMC3243117

Medication Reconciliation: Work Domain Ontology, Prototype Development, and a Predictive Model

Eliz Markowitz, MS,1,4 Elmer V. Bernstam, MD, MSE,1,4 Jorge Herskovic, MD, PhD,1,4 Jiajie Zhang, PhD,1,4 Ben Shneiderman, PhD,2,4 Catherine Plaisant, PhD,2,4 and Todd R. Johnson, PhD3,4

Abstract

Medication errors can result from administration inaccuracies at any point of care and are a major cause for concern. To develop a successful Medication Reconciliation (MR) tool, we believe it necessary to build a Work Domain Ontology (WDO) for the MR process. A WDO defines the explicit, abstract, implementation-independent description of the task by separating the task from work context, application technology, and cognitive architecture. We developed a prototype based upon the WDO and designed to adhere to standard principles of interface design. The prototype was compared to Legacy Health System’s and Pre-Admission Medication List Builder MR tools via a Keystroke-Level Model analysis for three MR tasks. The analysis found the prototype requires the fewest mental operations, completes tasks in the fewest steps, and completes tasks in the least amount of time. Accordingly, we believe that developing a MR tool, based upon the WDO and user interface guidelines, improves user efficiency and reduces cognitive load.

1. Introduction

Medication errors occur during transitions in care, such as when patients are admitted to, or discharged from, hospitals and from the use of multiple medications to treat one or more disorders1,2,3,4,5. Medication errors can result from administration inaccuracies at any point of care and remain a major cause for concern; one study identified medication issues in 93.3% of patients4. Some medication errors can be prevented by carefully maintaining the list of currently prescribed medications3,4. One study showed that utilizing a medication reconciliation process reduced medication errors by 80%6.

While there are a number of standalone and embedded Medication Reconciliation (MR) tools available, there are few studies demonstrating the success of these tools. Furthermore, despite the fact that MR has been identified as an important process for reducing medication errors, there is no consensus on how best to create a MR tool. We hypothesize that a systematic user interface design process, accomplished by first identifying the essential task requirements using a Work Domain Ontology (WDO) and then incorporating user-centered interface design principles, will dramatically improve the efficiency and quality of the MR process. A WDO defines the explicit, abstract, implementation-independent description of the task by separating the task from work context, application technology, and cognitive architecture. In this paper, we show how this process results in an MR tool that is more efficient than two existing tools.

1.1. What is Medication Reconciliation?

Although The Joint Commission (TJC) designated medication reconciliation a National Patient Safety Goal (NPSG) in 2009, there is no precise definition of medication reconciliation (MR) to guide the design and assessment of MR processes and tools. The Joint Commission defines MR as, “the process of comparing a patient’s medication orders to all of the medications that a patient has been taking5.” In contrast, The National Institute of Standards and Technology (NIST), as part of the use cases for Stage 1, Meaningful Use Criteria, defines MR as the process of electronically comparing two or more medication lists for a single patient7. There is also little consensus on the relationship between medication order entry, the identification of medication allergies and interactions, and medication reconciliation. As a result, both the input to the MR task and the output or goal of the task are vaguely defined, providing almost no objective criteria for improving MR. Despite this lack of guidance in the national standards, there is a growing consensus among clinicians that the reconciliation process should address therapeutic duplication, medication allergies and interactions, medication exclusion, dosage form, dosage regiment, differences in medication strength, and an up-to-date list of medications8.

1.2. Issues with Medication Reconciliation

One study of primary care practices found that 25% of patients suffered from an adverse drug event, 11% of which were preventable and 28% ameliorable. In many cases, adverse drug events are the result of errors that occur in the prescribing stage; one study found this to be true in 246 out of 421 cases8. Many of these errors occur due to the complexity of the MR process.

TJC has faced numerous challenges since it deemed MR a NPSG. Despite allocating funds, healthcare organizations have been unable to develop a tool that fully meets the needs of the MR process911. The MR process has also proven to be resource intensive. Current systems have proven to be complex and to be implemented successfully, the current MR process demands trained personnel and may require a full-time, multi-disciplinary team79,1215.

The duration of the MR process has also proven itself to be unpredictable. Indeed, time taken to complete the MR process varies widely amongst healthcare professionals16. In particularly labor-intensive cases, the MR task has been shown to take up to a full hour17. We believe that creating a tool based upon a Work Domain Ontology, that defines the MR task in terms of its goals, operators, objects, relations among objects, and constraints, will provide better guidance for improved MR processes and tools.

1.3. Establishing the Work Domain Ontology for the Medication Reconciliation process

A WDO defines the basic structure of work that a system, in conjunction with human users, will perform18. A WDO defines the explicit, abstract, implementation-independent description of the task by separating the task from work context, environment, application technology, and the human cognitive architecture18. By separating a given task from artifacts used to perform the work, proper attention is placed on the necessary functionality of a task. In other words, the WDO separates the hard constraints and demands of the task from those imposed by the particular tools and workflow used to do the task. Accordingly, rather than focusing on technical details and innovative features, the WDO highlights the inherent nature of the work.

The primary step in creating a WDO is to establish the goals, objects, relations among objects, constraints, and operations of the task. Once the aforementioned have been defined, it is necessary to identify the design space in which the task will be performed. Next, the variety of possible constraints, and their respective relationships, amongst the objects must be determined. Finally, it is necessary to identify the possible implementation-independent operations for the previously defined relevant relationships.

Once defined, analysts use a WDO to determine how best to distribute tasks among humans and tools. This distribution defines the basic functionality of a tool and the required functional interactions between the tool and its human user(s). The detailed design of the user interface requires more traditional human factors user interface design guidelines.

1.4. Design Guidelines for an Electronic Medication Reconciliation Tool

Graphical user interface (GUI) guidelines have been developed to reduce problems and prevent negative consequences. For example, the following “Eight Golden Rules of Interface Design” have been shown to improve usability and ensure error reduction19:

  1. Strive for consistency
  2. Enable frequent users to use shortcuts
  3. Offer informative feedback
  4. Design dialog to yield closure
  5. Offer simple error handling
  6. Permit easy reversal of actions
  7. Support internal locus of control
  8. Reduce short-term memory load

These guidelines are used in both the development and evaluation stages of GUI design. By taking such design issues into consideration, one can reduce interface usability issues and decrease time spent in the development process. Such guidelines overcome challenges of designing systems that have an intuitive interface and incorporate essential principles of human-computer interaction20.

1.5. Prototyping

Effective prototyping allows detrimental issues to be identified and addressed prior to product release. In short, a prototype is a draft version of an application that represents the range of the product. Prototypes can be high-fidelity, medium-fidelity, or low-fidelity, ranging from those that are fully functional to paper mock-ups21. Despite the lack of functionality associated with low-fidelity, paper mock-ups, such prototypes are often valuable in visualizing design solutions that lead to innovation and improvement. While some question the validity of utilizing low-fidelity prototypes to determine usability, Catani and Biers, in a landmark study, used low- and high-fidelity prototypes to perform library search tasks and found that there was no reliable difference in the usability issues discovered20. In fact, there was a high commonality in the usability issues uncovered using the different prototypes. Further studies demonstrated no significant difference in the number of usability issues identified when analyzing low- and high-fidelity prototypes2224. We developed a low-fidelity MR prototype, based on the WDO and the GUI design guidelines.

2. Methods

2.1. Determining the Work Domain Ontology for the Medication Reconciliation Process

To develop the WDO for the MR process, we started with the NIST meaningful use case for MR7 and the literature review of MR discussed above. We defined the goal of the MR process to be the creation of a list of medications that the patient should be taking when presented with two different lists of medications for a patient: one older list and one newer. In addition, the final list should be therapeutically appropriate and free of undesired side-effects and medication interactions.

Consider a patient who visits his primary care physician (PCP) after discharge from the hospital. The patient presents his PCP with his post-discharge medication list that may or may not differ from the PCP’s list of current patient medications. Neither list may accurately reflect what the patient is currently taking. Furthermore, the PCP may acquire new information during the visit that would render one, or both, of the lists ineffectual for optimal patient treatment. Additional problems may also arise if the independent lists contain medications that cause drug-drug interactions, adverse drug reactions, or errors in dosage or frequency.

Regardless of the context of care, the objects in the MR process are two lists: one older and one newer. Each list consists of one, or more, medications and its associated name (brand or generic), dosage, frequency, route of administration, active ingredients, start date, and end date.

The basic operations of MR allow a medication to be continued, discontinued, or changed. We chose not to include adding a new medication, because this is best handled as part of physician order entry, which is a separate task that requires its own analysis and user interface. One may assume that the need to compare lists adds additional information operations, such as determining possible medication interactions. At the level of the WDO, however, such needs are not operations, but rather artifacts of the tools used to do the task and the distribution of effort between the tools and user(s). In the WDO, these considerations are specified as a combination of goals and relations among objects. The goals of the final list are that the prescribed medications are therapeutically appropriate and free of undesired side-effects and interactions. How the aforementioned goals are distributed between the tool and the user is part of the application design process. The relations among objects specify correspondences between medications on the two lists. We established that the following relationships could occur between medications on separate lists:

  1. Strict Equivalence—The medications are identical in all respects.
  2. Form Equivalence—The medications are identical in form; i.e. brand versus generic medications.
  3. Functional Equivalence—The medications are different in form, but have equivalent functions.
  4. Partial Equivalence—Medications may possess form or functional equivalence, but differ in dosage, frequency, or route. In compound medications, partial equivalence correspondences occur if an ingredient in one compound medication matches the ingredient in a separate medication.
  5. No Equivalence—The medication is unique in both form and function.

Creating a thorough WDO for the MR task allowed us to determine both the necessary medication information and the possible inter-medication relationships that could occur.

2.2. Prototyping

Prior to full-scale implementation of an application, it is important to develop a prototype in order to gain insight about the design prior to committing the additional resources required for a final working application. Accordingly, for the primary stage of this study, a low-fidelity, electronic prototype was created to simulate an actual MR task. In order to maximize usability, the constructed prototypes were designed with “The Eight Golden Rules of Interface Design” in mind19.

In an effort to ensure consistency and reduce short-term memory load, we represent the correspondences found in the MR WDO using spatial proximity and a series of colors throughout all stages of the MR process. The main reconciliation interface, showing the spatial and color-coding of correspondences, is shown in Figure 2. Medications that are equivalent in some way are grouped vertically with faint horizontal separating lines, so that they appear as part of the same visual group. In addition, medications with strict equivalence are green; medications with form, functional, or partial equivalence are yellow; and medications with no equivalence are red. In a further effort to reduce short-term memory load, the relationship between medications and the origin of each medication is explicitly stated. Placing all medication information on a single screen prevents the need to commit medications, dates, and possible interactions to memory.

Figure 2.
Prototype of the initial reconciled record.

To support the internal locus of control, the interface allows direct interaction. Rather than having the user respond to a series of prompts, she can traverse through the MR process in any order. The user is provided with feedback after each decision. The color-coding, relation, and alerts column all dynamically adapt to the actions of the user. Accordingly, the user can view the progress of the MR task step-by-step on a single screen.

The MR task is designed in an effort to allow easy reversal of actions. Primarily, the interface allows the user to switch views without losing any data. For example, if the user is in the midst of the MR process and wants to view the two original lists, she may simply click on the “Original Lists” tab without losing any progress in the “Reconcile Lists” tab. Furthermore, the drag-and-drop interface allows the user to reverse her decision without having to traverse through past actions or start the MR task again.

In order to address the issues that arise in the MR process, a low-fidelity prototype was created using the principles of user interface design (Figures 1, ,22 and and3).3). The prototype went through over 20 iterations in an attempt to bolster usability, harness efficiency, and reduce the probability of error.

Figure 1.
Two, original, unreconciled medication lists.
Figure 3.
Prototype of the final reconciled record.

2.3. GOMS/KLM Analysis

Evaluating the interface can be done in a number of ways after prototype development. One technique used in usability testing is GOMS (Goals, Operations, Methods, and Selection rules) analysis. The purpose of GOMS is to determine the functionality of the system and provide estimates of task performance time25. Proposed by Card and Moran, the Keystroke-Level Model (KLM) is a simplified version of a GOMS analysis in which the overall execution time of a task is estimated by assigning a time to each operation. The KLM model has six operators: K for pressing a key or mouse button, P for pointing to a location on screen with the mouse, H for moving hands to home position on the keyboard or to the mouse, B for a button press, M for mentally preparing to perform an action, T(n) for typing a string of characters (n * K seconds)26. While the exact time associated with each action in the KLM model varies, David Kieras at The University of Michigan has produced estimated execution times for each action: K is 0.20 seconds, P is 1.10 seconds, H is 0.40 seconds, B is 0.10 seconds, M is 1.20 seconds, and T(n) is 0.20 *n seconds27.

Three tasks were analyzed via the KLM analysis: reconciling two identical medications, removing a medication from the record, and editing the dosage of a medication. We compared our prototype to two other systems that are currently in use and claim MR capabilities: Legacy Health System (LHS) with RxPad and the Pre-Admission Medication List (PAML) Builder. Formed in 1989, LHS is an Oregon-based corporation that offers an integrated network of healthcare services through the northwestern United States. According to LHS, RxPad is a, “self-learning module designed to instruct the pharmacist in the process of medication reconciliation and its implementation at Legacy Health System and to define the role of the pharmacist in this interdisciplinary process28.” Information regarding LHS with RxPad can be found at http://www.legacyhealth.org. Similarly, spurred by the growing interest in MR, Partners Information Systems began developing the PAML Builder application in 2004 to assists clinicians with the process. PAML Builder was designed to build the PAML, store information collected through the PAML Builder, provide services to mediate access to the collected information, and support the analytical and supporting needs surrounding the MR process29. Between late 2005 and early 2006, the PAML Builder tool underwent promising pilot studies at both Massachusetts General Hospital and Brigham and Women’s Hospital29. Further information about PAML Builder can be found in EG Poon’s paper, “Design and implementation of an application and associated services to support interdisciplinary medication reconciliation efforts at an integrated healthcare delivery network,” and in the presentation, “It is Not Just a List,” located on the website for the American Society of Health-System Pharmacists (http://www.ashp.org/).

3. Results

3.1. KLM Analysis

Using available screen shots, training manuals, and walkthroughs for both the LHS and PAML Builder systems, a KLM analysis was conducted for three common MR tasks: reconciling two identical medications (Table 1), removing a medication from the record, and editing the dosage of a medication. For each of the three tasks examined, we recreated the necessary operations for each task and, thus, were able to determine the number of required actions, mental operations, and completion times. In each instance, the prototype was found to be the most efficient.

Table 1:
KLM analysis for reconciling two identical medications

The number of actions, the number of required mental operations, and overall task completion time were lower for the prototype than either LHS RxPad or PAML (Table 2). Specifically, the prototype required an average of seven operations per task compared to the 12.6 and 9.3 operations required for the electronic MR tools of LHS RxPad and PAML Builder, respectively. Furthermore, the average task completion time for each tool was 5.46 seconds for the prototype, 9.5 seconds for LHS and RxPad, and 6.9 seconds for the PAML Builder. Additionally, the prototype demands equal or fewer mental operations from the user than required by either the LHS RxPad or the PAML Builder systems.

Table 2:
Comparative Results from the KLM analysis

3.2. Comparative Analysis

The prototype required fewer operations, fewer mental operations, and less time to complete a task. We believe there are two main reasons why neither the LHS or PAML Builder tools were able to achieve the efficiency of the prototype: the operations defined by the WDO are not clear in the two alternative interfaces and neither interface adheres to the aforementioned principles of GUI design.

One use of a WDO is to provide important requirements for guiding GUI design.30 Research into the psychology of human problem-solving has shown that different implementations of a single problem can result in a drastically different efficiencies and cognitive loads. Since the WDO places functionality as the top priority in the design process, it lends itself to promote the creation of highly efficient and usable interfaces. When viewing both the LHS and PAML Builder MR tools, a number of issues arise due to a lack of consideration for both the objects and their respective constraints and relationships.

Consider the task of medication dosage modification. While both tools allow the user to modify the dosage of a medication, the user is given no information as to what, or why, she should modify a medication. Indeed, neither the LHS nor the PAML Builder tool provides visual indication of form equivalence between medications. Furthermore, in the LHS tool, the user must open a new screen to modify the medication. By guiding the user to an alternative screen, both the task completion time and cognitive load of the user is increased; the user must not only remember what other medications the patient is taking, but also the possible interactions. In PAML Builder, the user can modify a medication when adding it to the reconciled list, but is given no information to the possible constraints and relationships amongst the medications.

The WDO in conjunction with the GUI and cognitive engineering guidelines, promoted the creation of an efficient MR interface. The main reasons why the efficiency of both the LHS and PAML Builder tools was lower than the prototype is due to the prototype’s support for an internal locus of control, user feedback, consistency, and ability to reduce cognitive load. In the prototype, the entirety of the MR process is supported through direct interaction with three screens (Figures 1, ,2,2, and and3).3). Throughout all stages of the reconciliation process, we depict the possible constraints and relationships through a standard set of colors that change when a medication is acted upon. The three main screens are dynamically updated throughout the MR process to provide the user with constant feedback. Accordingly, the user can view the progress of the MR task step-by-step on a single screen. In contrast, the LHS and PAML Builder tools require the user to traverse numerous screens, pop-ups, and prompts. Furthermore, the lack of color-coding and dynamically updated relationships fails to provide the user with feedback on her progress. This leads to an increase in the number of physical and mental operations required to do the analyzed tasks in LHS and PAML.

4. Discussion

We hypothesized that a systematic user interface design process, accomplished by first identifying the essential task requirements and then incorporating user interface design principles in the design process, would dramatically improve the efficiency of the MR process. First, the WDO for the MR task was developed and used as the framework for a prototype design. The prototype was developed according to the WDO and the user interface design guidelines with an overall goal to improve efficiency and reduce the user’s cognitive load. When the prototype was compared to the LHS and PAML electronic MR tools by means of a KLM analysis, we found that the prototype required the fewest number of mental operations, the fewest number of physical steps, and the least amount of time.

The strengths of this study revolve around the utilization of a WDO, in conjunction with design guidelines, to inform prototype creation. Creating a WDO ensures that the essential goals, operations, objects, relations and constraints of a health-related information task are supported by the resulting tool. However, this study has several limitations. Primarily, while the predictive nature of a KLM analysis gives us insight into the tool’s efficiency, we did not conduct an actual usability study. As a result, we are only able to predict task times if we assume that user actions are optimal. In order to get a greater sense of the prototype’s value, we need to construct a higher-fidelity prototype and put it through rigorous usability and learnability testing.

Other studies have created MR tools and tested them in a variety of healthcare settings. Unlike our prototype, however, these tools were either paper-based or combined a number of different interventions such as home-visits, surveillance, computer prescription order entry, and intervention programs11,13,3032. Other studies were focused on clinical improvement, rather than efficiency, medication adherence, or went beyond the scope of MR as defined by NIST.1415 Generally, the aforementioned studies have identified MR as necessary, but do not have a definitive solution to current issues. Unfortunately, the variety of solutions means that our outcomes cannot be easily compared to other MR tools. The strength of our study lies in the fact that we have developed a design process for creating a tool that matches the fundamental requirements of MR and then embeds these requirements in a user-centered tool.

While the prototype predicts improved user efficiency, considerable work remains. In addition to creating a fully functional prototype, it is imperative to conduct usability testing. Doing so will allow for both interface improvement and reduced likelihood of inefficiency and errors. While the MR tool predicts increased efficiency and reduced cognitive load, there are still unanswered questions regarding its effectiveness at reducing the number of medication related adverse events. Accordingly, future research will resolve around creating a high-fidelity prototype, integrating the tool into a working system, and performing a clinical study documenting the number of medication related adverse events that occur. The outcomes from the study will then be compared to the number of medication related adverse events occurring via traditional means of MR. This will allow us to understand if a tool created with a WDO and modeled on design guidelines not only improves efficiency and reduces cognitive load, but also decreases the number of medication related adverse events.

5. Conclusion

This study demonstrates the importance of using a WDO plus GUI design guidelines for designing health information technology that supports fundamental task requirements while simultaneously improving user efficiency.

6. Acknowledgements

This project is supported by a training fellowship to E. Markowitz from the AHRQ Training Program of the W. M. Keck Center for Interdisciplinary Bioscience Training of the Gulf Coast Consortia (AHRQ Grant No. T32 HS017586) and by Grant No. 10510592 for Patient-Centered Cognitive Support under the Strategic Health IT Advanced Research Projects Program (SHARP) from the Office of the National Coordinator for Health Information Technology.

References

1. Roughead EE, Semple SJ. Medication safety in acute care in Australia: where are we now? Part 1: a review of the extend and causes of medication problems. Aust New Zealand Health Policy. 2009;6:18. [PMC free article] [PubMed]
2. De Koning JS, Klazinga NS, Koudstaal PJ, Prins A, Dippel DW, Heerings J, et al. Quality of care in stroke prevention: results of an audit study among general practitioners. Prev Med. 2004;38:129–36. [PubMed]
3. Poon EG. Medication reconciliation: whose job is it? AORN J. 2009 Jun;89(6):1180, 1122. [PubMed]
4. Ellitt GR, Engblom E, Aslani P, Westerlund T, Chen TF. Drug related problems after discharge from an Australian teaching hospital. Pharm World Sci. 2010 [PubMed]
5. Joint Commission on Accreditation of Healthcare Oganizations’ Using medication reconciliation to prevent errors. Sentinel Event Alert. 2006 Jan 25;(35):1–4. [PubMed]
6. Rozich JD, Howard RJ, Justeson JM, Macken PD, Lindsay ME, Resar RK. Standardization as a mechanism to improve safety in health care. Jt Comm J Qual Saf. 2004 Jan;30(1):5–14. [PubMed]
7. The National Institute for Standards and Technology. Test procedure for §170.302.j Medication Reconciliation. August 13, 2010.
8. Johnson CM, Marcy TR, Harrison DL, Young RE, Stevens EL, Shadid J. Medication reconciliation in a community pharmacy setting. J Am Pharm Assoc. 2010 Jul-Aug;50(4):523–6. [PubMed]
9. Royal S, Smeaton L, Avery AJ, Hurwitz B, Sheikh A. Interventions in primary care to reduce medication related adverse events and hospital admissions: a systematic review and meta-analysis. Qual Saf Health Care. 2006;15:23–31. [PMC free article] [PubMed]
10. Bayoumi I, Howard M, Holbrook AM, Schabort I. Interventions to improve medication reconciliation in primary care. Ann Pharmacother. 2009 Oct;43(10):1667–75. [PubMed]
11. Green CF, Burgul K, Armstrong DJ. A study of the use of medicine lists in medicines reconciliation: please remember this, a list is just a list. Int J Pharm Pract. 2010 Apr;18(2):116–21. [PubMed]
12. Gandhi TK, Weingart SN, Borus J, et al. Adverse drug events in ambulatory care. N Engl J Med. 2003 Apr 17;348(16):1556–64. [PubMed]
13. Peyton L, Ramser K, Hamann G, Patel D, Kuhl D, Sprabery L, Steinhauer B. Evaluation of medication reconciliation in an ambulatory setting before and after pharmacist intervention. J Am Pharm Assoc. 2010 Jul-Aug;50(4):490–5. [PubMed]
14. Steurbaut S, Leemans L, Leysen T, De Baere E, Cornu P, Mets T, Dupont AG. Medication history reconciliation by clinical pharmacists in elderly inpatients admitted from home or a nursing home. Ann Pharmacother. 2010 Oct;44(10):1596–603. [PubMed]
15. St Peter WL. Improving medication safety in chronic kidney disease patients on dialysis through medication reconciliation. Adv Chronic Kidney Dis. 2010 Sep;17(5):413–9. [PubMed]
16. Karnon J, Campbell F, Czoski-Murray C. Model-based cost-effectiveness analysis of interventions aimed at preventing medication error at hospital admission. J Eval Clin Pract. 2009 Apr;15(2):299–306. [PubMed]
17. Barnsteiner JH. Patient Safety and Quality: An Evidence-Based Handbook for Nurses. Rockville, MD: Agency for Healthcare Research and Quality (US); 2008. Medication Reconciliation.
18. Zhang J, Butler K. UFuRT: A work-centered framework and process for design and evaluation of information systems. Proceedings of HCI International; 2007.
19. Shneiderman B, Plaisant C. Designing the User Interface. 5th ed. Boston, MA: Pearson Addison-Wesley; 2010.
20. Rubin J, Chisnell D. Handbook of usability testing. 2nd Edition. New York: John Wiley & Sons; 2008.
21. Sauer J, Selbel K, Ruttinger B. The influence of user expertise and prototype fidelity in usability tests. Applied Ergonomics. 2010;41(1):130–140. [PubMed]
22. Catani MB, Biers DW. Usability evaluation and prototype fidelity: users and usability professionals. Proceedings of the Human Factors and Ergonomics Society 42nd Annual Meeting; Chicago, IL: 1998. Oct 5–9, pp. 1331–1336.
23. Virzi RA, Karis D, Sokolov JL. Usability problem identification using both low and high-fidelity prototypes. Proceedings of CHI’96; Vancouver, BC, Can: 1996. pp. 236–243.
24. Walker M, Takayama L, Landay JA. High-fidelity or low-fidelity, paper or computer? Choosing attributes when testing web prototypes. Proceedings of the Human Factors and Ergonomics Society 46th Annual Meeting; 2002. pp. 661–665.
25. John BE, Kieras DE. Using GOMS for user interface design and evaluation: which technique? ACM Transactions of Computer-Human Interaction (TOCHI) 1996;3(4):287–319.
26. Card SK, Moran TP, Newell A. The keystroke-level model for user performance time with interactive systems. Communication of the ACM. 1980;23(7):396–410.
27. Kieras D. Using the keystroke-level model to estimate execution times. Ann Arbor: Department of Psychology, University of Michigan; 2001. Unpublished report.
28. Legacy Health Collateral Materials for Medication Reconciliation. 2010. http://www.legacyhealth.org/body.cfm?id=757. Updated January 4th, 2010. Accessed December 9th,
29. Thomley S, Russel J. It is Not Just a List. American Society of Health-System Pharmacists. http://www.ashp.org/DocLibrary/MemberCenter/Webinars/MedReconciliation-SOPIT.aspx May 10, 2010. Accessed July 6, 2011.
30. Poon EG, Blumenfeld B, Hamann C, Turchin A, Graydon-Baker E, McCarthy PC, et al. Design and implementation of an application and associated services to support interdisciplinary medication reconciliation efforts at an integrated healthcare delivery network. J Am Med Inform Assoc. 2006 Nov-Dec;13(6):581–92. [PMC free article] [PubMed]
31. Zhang J. A representational analysis of relational informational displays. Int J Human-Computer Studies. 1996;45:59–7.
32. Pronovost P, Weast B, Schwarz M, et al. Medication reconciliation: a practical tool to reduce the risk of medication errors. J Crit Care. 2003 Dec;18(4):201–5. [PubMed]
33. Keohane CA, Bates DW. Medication safety. Obstet Gynelo Clin North Am. 2008 Mar;35(1):37–52. [PubMed]
34. Kane-Gill S, Weber RJ. Principles and practices of medication safety in the ICU. Crit Care Clin. 2006;22:273–290. [PubMed]

Articles from AMIA Annual Symposium Proceedings are provided here courtesy of American Medical Informatics Association