PMCCPMCCPMCC

Search tips
Search criteria 

Advanced

 
Logo of jamiaJAMIA - The Journal of the American Medical Informatics AssociationInstructions for authorsCurrent TOC
 
J Am Med Inform Assoc. 2011 Nov-Dec; 18(6): 799–804.
Published online Aug 11, 2011. doi:  10.1136/amiajnl-2011-000191
PMCID: PMC3197996
A nationwide medication incidents reporting system in The Netherlands
Ka-Chun Cheung,corresponding author1 Patricia M L A van den Bemt,2 Marcel L Bouvy,3 Michel Wensing,1 and Peter A G M De Smet4
1IQ Scientific Institute for Quality of Healthcare, Radboud University Nijmegen Medical Centre, Nijmegen, The Netherlands
2Department of Hospital Pharmacy, Erasmus MC, Rotterdam, The Netherlands
3Department of Pharmacoepidemiology and Clinical Pharmacology, Utrecht Institute for Pharmaceutical Sciences (UIPS), Utrecht University, Utrecht, The Netherlands
4Department of Clinical Pharmacy, Radboud University Nijmegen Medical Centre, Nijmegen, The Netherlands
corresponding authorCorresponding author.
Correspondence to Ka-Chun Cheung, Alexanderstraat 11, 2514 JL, Den Haag, The Netherlands; k.c.cheung/at/knmp.nl
Received February 16, 2011; Accepted July 11, 2011.
Objective
Many Dutch hospitals have established internal systems for reporting incidents. However, such internal systems do not allow learning from incidents that occur in other hospitals. Therefore a multicenter, information technology (IT) supported reporting system named central medication incidents registration (CMR) was developed. This article describes the architecture, implementation and current status of the CMR in The Netherlands and compare it with similar systems in other countries.
System Description
Adequate IT is required to sufficiently support a multicenter reporting system. The CMR system consists of a website, a database, a web-based reporting form, an application to import reports generated in other reporting systems, an application to generate an overview of reported medication incidents, and a national warning system for healthcare providers.
Current Status
From the start of CMR 90 of all 93 (96.8%) hospitals and 872 of 1948 (44.8%) community pharmacies participated. Between March 2006 and March 2010 the CMR comprised 15 694 reports of incidents. In the period from March 2010 to March 2011, 1642 reports were submitted by community pharmacies in CMR and the hospitals submitted 2517 reports. CMR is similar to various systems in other countries, but it seems to use more IT applications.
Discussion
The CMR is developing into a nationwide reporting system of medication incidents in The Netherlands, in which hospitals, community pharmacies, mental healthcare organizations and general practitioners participate.
Conclusion
The architecture of the system met the requirements of a nationwide reporting system across different healthcare providers.
Keywords: Incident reporting, medication errors, medication safety, patient safety, pharmacy, safety management
In 1999, the Institute of Medicine published the report ‘To err is human: building a safer health system’. This report placed patient safety high on the agenda and encouraged healthcare providers to participate in incident reporting systems.1 Reporting of incidents helps healthcare providers to learn from these incidents and improve patient safety. A well functioning system for the reporting of medication incidents is therefore a must.2 Reporting systems can provide information to healthcare providers and other stakeholders about types of errors, causes and risks, and preventive actions.3–8 To facilitate large-scale trend analyses multicenter reporting systems are necessary. In The Netherlands, the nationwide central medication incidents registration (CMR) was set up for hospitals in 2006 and adapted for additional settings in 2010. The system uses information technology (IT) to facilitate both implementation in daily practice, and trend analysis and feedback to healthcare providers. The purpose of this paper is to outline its basic structure and performance and to compare these briefly with other nationwide incident reporting systems (in the USA, Canada, the UK and Denmark).
The CMR was developed as a multicenter reporting system for medication incidents. The objectives of the CMR are to support risk management of medication processes by:
  • Sending out alerts and newsletters to prevent the reoccurrence of specific high-risk medication incidents.
  • Generally informing healthcare providers and policymakers about risks, based on trend analyses within the CMR database.
The CMR should fulfill the following requirements to be able to function as a multicenter reporting system:
  • The reporting system should be adequately supported by IT.
  • The system should be easily accessible and easy to use.
  • The system should be fit for nationwide implementation across different healthcare sectors.
  • The responsibility for reporting should remain with the practising healthcare providers.
  • Reporting should be safe for healthcare professionals (confidential and not punitive).
  • The reporting system should demonstrably contribute to medication safety.
In the pilot phase hospital pharmacists reported medication incidents derived from their internal reporting systems through a web-based CMR reporting form. After a successful pilot the CMR became available for all Dutch hospitals.9 Between March 2006 and March 2010 (phase I) CMR was only implemented in all Dutch hospitals. The CMR extended rather than replaced existing internal reporting systems in hospitals. From January 2009 to March 2010 the CMR was technically adapted and tested in 79 community pharmacies of a pharmacy franchise company. Since March 2010 the CMR has been available for all community pharmacies (phase II). Currently, the CMR is further expanded to primary care. In January 2011, 20 general practitioners started a pilot to incorporate the CMR into their daily practice. Mental healthcare institutions have agreed to start implementing CMR.
The CMR system (in phase II) consists of a website (http://www.medicatieveiligheid.info), a database, a web-based reporting form, an application to import reports generated in other reporting systems (including a real-time interface), an application to generate an overview of reported medication incidents (including trend analyses), and a national warning system for healthcare providers (alerts and newsletters by email, which are also made available through the website).
Web-based reporting form
Users can access the reporting form on a secure part of the CMR website. The reporting form consists of four sections: administrative information; patient data; information about the medication incident; and questions concerning the need to issue an alert.
In the administrative information section the user needs to fill in the reporting date, the date of the medication incident and the identification number of the healthcare organization. Personal patient data are limited to gender and year of birth of the patient (when applicable). Based on the experience of the US Institute for Safe Medication Practices (ISMP) the description of the medication incident starts with an open question to describe the medication incident. The remaining questions are multiple-choice questions with predefined answers in drop-down menus. The three most important questions are: What type of medication incident is it? What were the underlying cause(s)? What has been the harm to the patient? The fourth and final section of the reporting form consists of questions about the risk of recurrence, the educational potential for other healthcare providers and the perceived need for an alert (see supplementary appendix I, available online only).
Classifications in reporting form
The CMR reporting form has three important classifications: a medication error classification; a classification of causes; and a classification of harm to the patient. For the CMR in phase II we adapted the initial classification system of medication errors (based on work by Van den Bemt and Egberts).10 For this revision we also used the WHO international classification for patient safety, earlier experiences of hospitals, and suggestions from a panel of eight community pharmacists.11 The revised classification distinguishes eight steps in the medication distribution process and each step contains several subcategories (see supplementary appendix II, available online only).
The classification of causes was based on the Eindhoven classification method, which was originally developed for the chemical industry.12 The Eindhoven classification method is also useful to identify failure factors of medication incidents.13 This classification discriminates between technical, organizational, human, communication, and patient-related failure factors.
The CMR uses the Dutch coding system for patient safety, The Netherlands technical agreement 8009, to classify harm. The Netherlands technical agreement divides the harm into five classes: none, minimal/mild harm, serious temporary harm, serious permanent harm, and death.14 In the case of a near miss the healthcare provider can estimate the potential harm (what if the patient would have been exposed to the error) on a five-point scale.
Reporting routes
One of the routes for reporting a medication incident is the web-based reporting form. Most Dutch hospitals have their own internal system to register all kinds of reported events including medication incidents. If the hospital does not use the web-based reporting form then the hospital can use one of the two computerized ways to send these reports to the CMR database. The first way is to extract these reports manually from the internal reporting system and the hospital manually uploads these reports to the CMR database through the CMR website. Since 2007, hospitals can also use a direct real-time interface between their internal reporting systems and the CMR database for submitting their internal reports about medication incidents directly. Some community pharmacy chains are now also using internal reporting systems with a direct interface to the CMR. Both the manual upload function and the real-time interface prevent double reporting activity for the healthcare provider (reporting to two separate internal and multicenter reporting systems). For both functions the obligatory questions of the CMR have to be built into the internal reporting system. In the literature we have found that a state-wide reporting system in the USA (the Pennsylvania patient safety reporting system) is helping facilities to construct such an interface between existing reporting systems in hospitals and the Pennsylvania patient safety reporting system because of complaints that reporting to two separate systems (the internal and multicenter system) required extra work.3
Besides these formal ways healthcare providers may also contact the CMR team (currently consisting of a clinical pharmacologist, two pharmacists, one nurse, and two pharmacy technicians) informally by telephone or email.
Analysis and feedback
The CMR team screens the submitted reports every week by hand to sort out which medication incidents are potentially interesting. This is primarily done on the basis of three predefined general criteria: (1) risk of recurrence; (2) educational potential for other healthcare providers; and (3) actual or potential risk of serious harm to the patient. Reports may also be selected for further scrutiny when they concern a predefined topic of special interest (such as an accidental interchange of patients or of sound-alike and look-alike medicines). The CMR team decides which reports potentially qualify for an alert or as an item for the CMR newsletter, and which ones should be marked for further analysis of a special interest topic. The CMR team can also perform additional analyses of the entire database to track and define similar earlier cases.
Users can analyze their own reports and compare these with all the reported medication incidents within a sector (hospitals, community pharmacies, mental care institutions).
National warning system
Alerts consist of reported medication incidents with a high risk of recurrence, high educational potential for other healthcare providers, and/or actual or potential risk of serious harm to the patient. The healthcare providers can notify on the report form whether the medication incident meets the requirements of an alert, but the CMR organization forms its own opinion during the screening process. The CMR organization is submitting the selected reports for further evaluation to a multidisciplinary expert panel (consisting of an experienced general practitioner, internal medicine physician, psychiatrist, hospital pharmacist, clinical pharmacologist, pharmacist in mental care, community pharmacist, nursing home physician, nurse and patient representative). If the panel decides that an alert is warranted, a CMR alert is prepared in accordance with a prespecified format (a brief summary of the medication incident, general background information and comments, and specific recommendations to reduce the risk of recurrence). The CMR organization sends the alerts out to healthcare professionals by email and they are also made available through the public part of the CMR website.
Less urgent but relevant matters are communicated through a periodical electronic newsletter on the website and incidental publications in the Dutch Pharmaceutical Journal. The newsletter is sent out every 3 months by email and may be consulted through the public part of the CMR website.
All practising pharmacists in The Netherlands receive (for free) the alerts and newsletters. To receive the alerts and newsletters it is not necessary for the pharmacists to participate or to report actively to the CMR. Other healthcare providers only receive the newsletters when they have actively subscribed to them (also for free). If the alert is relevant for specific groups of other healthcare providers, the CMR organization informs their scientific and professional associations. The CMR has chosen distribution by email because of the quick delivery and because all pharmacists can be readily reached by email.
Security and confidentiality
The hosting and IT security comply with the latest Dutch ICT standard (NEN 7510), which is based on the international standard ISO/IEC 17799.15 Healthcare providers always submit their report over a secure Internet connection.
Each member of the CMR team has signed a contract of confidentiality. The CMR cannot publish any report without formal approval of the healthcare provider, even when the publication does not contain retraceable information. The database only records the ID number of the reporting healthcare practice. The analyst does not have information that is directly retraceable to the healthcare organization or person who reported the medication incident or was involved in it.
According to Dutch law, the CMR team is not obliged to hand over the content of the CMR database to public bodies like the Healthcare Inspectorate, Ministry of Health, etc. The healthcare provider always remains the legal owner of the submitted reports.
Database structure
The CMR database is a relational database that is maintained in a Microsoft SQL server. The applications use ColdFusion for data driving and the operating system is a Microsoft Windows server. The applications and data storage communicate use XML. The CMR database and the applications have been developed and are maintained by a software development firm (Ritense BV, Amsterdam—http://www.ritense.com).
Participants
In phase I, 90 of all 93 (96.8%) hospitals in The Netherlands applied for participation. Most of the hospitals used the web-based reporting form or the manual upload function to submit reports of medication incidents. Thirteen participants reported more than 100 medication incidents, 11 participants reported between one and 50 incidents and 67 participants did not report in the whole of phase I. In successive years in phase I, there was only a minimal shift between reporting and not-reporting participants.
From the start of phase II until March 2011, 872 of 1948 (44.8%) community pharmacies requested a username and password. Two community pharmacy chains (331 pharmacies in total) are using a real-time interface to submit reports. The other participating community pharmacies submit their reports through the web-based reporting form. Hospitals that were already participating in the CMR (phase I) are expected to switch over to CMR (phase II) in the period from March 2010 to March 2011 (intermediate stage).
Reported medication incidents
On March 1 2010 (end of phase I), the CMR database comprised 15 694 reports of incidents (including 651 reports collected in the pilot period from July 2004 to February 2006). When only these reports of phase I are considered, 44.2% are related to the administration of medication. Incidents in the prescribing phase (21.0%) are the second most prevalent type of incidents (table 1). The most commonly reported causes were classified as human performance failures (73.7%) (table 2). In the majority of cases (69.7%), the incident reached the patients and the medication was administered to the patient; 12.3% of all reported incidents required monitoring or another intervention and 6.1% were directly associated with harm to the patient including 0.1% (n=19) of deaths (table 3).
Table 1
Table 1
Types of reported medication incidents in CMR (phase I) and CMR (phase II)
Table 2
Table 2
Reported causes of the medication incidents in phase I and phase II
Table 3
Table 3
Reported patient harm in the medication incidents in CMR (phase I) and CMR (phase II)
In the period from March 2010 to March 2011, 1642 reports were submitted by community pharmacies. The reported incidents most often arose in the processing of prescriptions and medication surveillance phase (42.5%). Incidents in the dispensing phase (27.5%) were the second most prevalent type of incidents (table 1). Healthcare providers could select more than one cause per reported case. Behavioral factors (1642/1904, 86.2%) caused most of the medication incidents. The rest of the selected causes spread over technical factors (5.3%), organizational factors (2.5%), communication factors (4.9%) and patient-related factors (1.1%) (table 2). Less than half of the medication incidents (744/1642, 45.3%) reached the patient. The healthcare providers indicated that 80.6% of these 744 medication incidents were harmless for the patient. There were no cases of serious permanent harm or fatal harm (table 3).
In the same period, the hospitals submitted 2517 reports to CMR (phase II). Tables 1 and and22 summarize the frequencies of error classification and causes. A few of these incidents led to serious permanent harm (five, 0.2%) or fatal harm (eight, 0.3%) (table 3).
CMR alerts
Since the start of the CMR (including the pilot phase), 15 nationwide alerts with specific recommendations to prevent recurrence of the medication incident have been sent out (see supplementary appendix III, available online only).
In this section we briefly compare the CMR with other nationwide reporting systems and discuss the implications for practice and research.
Comparison with other nationwide reporting systems
For comparison we identified four nationwide reporting systems that collect medication incidents in the USA, Canada, the UK and Denmark. State-wide reporting systems such as the one in Pennsylvania (USA)3 and the one in Australia16 17 were not included in the comparison. In Australia the advanced incident management system has been in use since 1998 and four of the eight states use the advanced incident management system.17 Little detailed information about the architecture and performance of most other nationwide reporting systems is available in the scientific literature or on the internet. Only the national reporting and learning system (NRLS) in the UK offers extensive documentation on its website.18 To collect detailed information about the architecture and performance we interviewed (by telephone) the organizations maintaining the nationwide reporting systems and during the interview we used a pre-formatted questionnaire (which was also mailed to the contact person if requested). Table 4 summarizes the comparison with the following organizations: ISMP in the USA, ISMP–Canada in Canada, the Patientombuddet in Denmark and the National Patient Safety Agency in the UK.
Table 4
Table 4
Comparison of nationwide reporting systems
The earliest reporting system was set up in the USA in 1975. The other reporting systems were developed in the past 10 years. Between the nationwide reporting systems the cumulative numbers of reported medication incidents per 1 000 000 inhabitants differed from one to 6301 cases. The CMR and the Canadian medication incident reporting and prevention system only collect medication incidents, whereas the systems in the UK and Denmark register all kind of incidents concerning patient accidents, treatment/procedure, access/admission/transfer/discharge, and infrastructure. Most reporting systems are voluntary reporting systems except for the system in Denmark, where healthcare providers are legally obliged to report.
Runciman et al17 described the desirable attributes of an integrated system and the CMR meets some of these requirements. To meet the requirement of easy access, the CMR offers four reporting routes: a web-based reporting form; manual upload function; interface and the informal way by telephone and email. In 1975 the internet was not yet widely used so that it took substantially more effort to report an incident to the US ISMP medication errors reporting program. All of the reporting systems now have an internet form to receive medication incident reports. We believe that offering and maintaining this wide range of reporting routes, especially the automatic interface and upload function, have enhanced its utility. The NRLS offers a comparable interface between local reporting systems and the nationwide reporting system.
In the period from 2005 to May 2010 the CMR has sent out 15 national alerts and three newsletters. The nature of this output is more or less comparable to that of other national reporting systems. There appears to be rather a substantial variation, however, in the frequency with which other reporting systems are distributing alerts. Since 31 October 2002 the NRLS has sent out 71 alerts about medical incidents and the ISMP sent out 106 safety alerts (related to drugs and therapeutic biological products) in the period from March 2005 to December 2010.18 19 There is also a large variation in the frequency of distributing newsletters and other information. The ISMP medication errors reporting program issues a biweekly newsletter for hospitals; a monthly edition for community pharmacists, pharmacy technicians, nurses, physicians and other community health professionals; a monthly newsletter for nurses; and a monthly consumer health education newsletter.3 19 The NRLS publishes newsletters on its website with unknown frequency.3 18 Beside alerts and newsletters the USA, UK and Denmark also publish reports about annual aggregate analyses, specific issues or care settings, and individual incidents. The CMR is capable of producing similar reports. At this moment the CMR only publishes annual reports for individual participating hospitals about their own reported medication incidents (including a benchmark). To provide further guidance on specific medication safety issues the CMR has planned an analysis of large numbers of reported incidents for the near future.
Implications for practice
With the expansion of the CMR to community pharmacies, the collaboration with mental healthcare organizations and the pilot with general practitioners, the CMR is turning into a nationwide reporting system of medication incidents for different healthcare settings. A likely implication is that the focus may shift from medication-related incidents to healthcare-related incidents in general. Plausibly, the healthcare providers prefer one reporting system for all kinds of patient safety incidents. This is in accordance with Runciman et al,17 who recommended one integrated framework for the management of safety, quality and risk.
With the expansion of CMR it will become more difficult for the CMR team to screen every report in detail. One potential way forward is the development of prestructured methods to select relevant reports on the basis of the predefined classifications and to facilitate large-scale trend analyses to gain more insight into the risks of medication processes. Data mining techniques that have been developed for databases that collect spontaneous reports of adverse events might help to select relevant reports for further analysis.20 21
Implications for research
In phase I of the CMR the incident reporting rate of hospitals showed high variability. This is in line with a US study about a reporting system in 23 intensive care units, in which five hospitals submitted 58% of the reports.22 In another US study the rate of reports per 1000 inpatients also varied substantially among hospitals. The rates did not correlate with hospital size or the duration of reporting system use, although there was a trend towards less variation among hospitals that had used the reporting system for two or more years.23 In phase I of the CMR the number of reports per hospital did not clearly increase over time.
Besides the large variability of reporting rates, our data and other studies suggest that there may be substantial underreporting.22–26 Further research into underreporting and the variability in reporting is needed to identify underlying factors. For instance, to what extent is the reporting of incidents related to the safety culture or what characteristics of nationwide reporting systems (eg, automatic upload function, obligatory reporting or not, etc.) may stimulate reporting.
Although the data on phase II should be interpreted with appropriate caution due to the low number of reports, our first results suggest interesting differences in the characteristics of reports originating from hospitals and community pharmacies. The preliminary picture is that reports from community pharmacies are more often related to dispensing and order entry of the prescription and medication surveillance. Furthermore, more than half of the medication incidents in community pharmacies did not reach the patient. Such reports may still be valuable because they draw attention to a potentially poor aspect of performance in the medication process. In hospitals 22% (phase I) to 31% (phase II) of the medication incidents did not reach the patient and only 59% did not harm the patient.
The ultimate goal of CMR is to provide healthcare providers and other stakeholders with guidance on how to improve patient safety. Alerts are regularly sent out but their actual effects on practice and patient safety still have to be evaluated.
Last but not least there may potentially be an important role for the patient in reporting medication incidents. The current CMR does not include reports from patients, but experiences with patient reporting of adverse drug reactions suggest that they may well become a valuable source of information.27 More research can help to explore how patients can easily report incidents that are professionally useful.
This paper is the first to describe the architecture, implementation, and results of a nationwide reporting system (CMR) in The Netherlands. The architecture of the revised CMR (phase II) has been implemented for use in hospitals and community pharmacies. Dutch hospitals were the first sector to start reporting incidents followed by community pharmacies and mental care institutions. The strategy to expand the CMR to community pharmacists has been successful, and this approach will now be used to expand the CMR to the rest of primary care. In the near future the CMR also aims to attract general practitioners and residential care homes. The CMR is gradually turning into a nationwide reporting system that will be available for all healthcare providers. The next step for the CMR will be to gain insight into the risk of medication processes by large-scale trend analyses of the large numbers of reports in the CMR database.
Supplementary Material
Supplementary Data
Acknowledgments
The authors would like to thank Hayo Graatsma and Brigit van Soest-Segers for their contribution in the CMR. Thanks also go to the NVZ Dutch Hospitals Association and the Dutch Hospital Pharmacists Association (NVZA) for the support to develop the CMR. The authors also thank Ritense BV for their advice during the development and revision of the CMR and Alliance Apotheek BV for testing the revised CMR (phase II). Thanks go to the US Institute for Safe Medication Practices for their advice about the reporting form, and thanks to the US Institute for Safe Medication Practices, the Institute for Safe Medication Practice Canada, the Patientombuddet in Denmark and the National Patient Safety Agency in the UK for providing information about the architecture and performance of their nationwide reporting systems.
Footnotes
Competing interests: None.
Provenance and peer review: Not commissioned; externally peer reviewed.
1. Corrigan JM, Donaldson MS, Kohn LT, et al. To Err is Human Building a Safer Health System. Washington, DC: National Academy Press, 2000.
2. Leape LL. Reporting of adverse events. N Engl J Med 2002;347:1633–8. [PubMed]
3. Cohen MR., American Pharmacists Association Medication Errors, 2nd edn Washington, DC: American Pharmacists Association, 2006.
4. Montesi G, Lechi A. Prevention of medication errors: detection and audit. Br J Clin Pharmacol 2009;67:651–5. [PubMed]
5. Ashcroft DM, Cooke J. Retrospective analysis of medication incidents reported using an on-line reporting system. Pharm World Sci 2006;28:359–65. [PubMed]
6. Dovey SM, Phillips RL. What should we report to medical error reporting systems? Qual Saf Health Care 2004;13:322–3. [PMC free article] [PubMed]
7. Williams SD, Ashcroft DM. Medication errors: how reliable are the severity ratings reported to the national reporting and learning system? Int J Qual Health Care 2009;21:316–20. [PubMed]
8. Spigelman AD, Swan J. Review of the Australian incident monitoring system. ANZ J Surg 2005;75:657–61. [PubMed]
9. NVZA project group Medication Safety Projectplan CMR Landelijk meldpunt voor medicatiegerelateerde incidenten in ziekenhuizen. [In Dutch]. NVZA Project Group Medication Safety, 2004.
10. van den Bemt PM, Egberts AC. Drug-related problem: definitions and classification. EJHP Practice 2007;13:62–4.
11. World Health Organization The Conceptual Framework for the International Classification for Patient Safety. Version 1.1 Final Technical Report. Geneva: World Health Organization, 2009.
12. Aspden P, Corrigan JM, Wolcot J, et al. Patient Safety Achieving a New Standard for Care. Washington, DC: The National Academies Press, 2004.
13. Habraken MM. Better Care for Incidents in Health Care. [MSc thesis]. The Netherlands: Eindhoven University of Technology, 2005.
14. Netherlands Standardization Institute Netherlands Technical Agreement NTA 8009 (nl) Safety Manangement System for Hospitals and Institutions Which Deliver Hospital Care. [In Dutch]. Delft, The Netherlands: Netherlands Standardization Institute, 2007.
15. Netherlands Standardization Institute Handboek NEN 7510. Delft, The Netherlands: Netherlands: Standardization Institute, 2005.
16. Magrabi F, Ong MS, Runciman W, et al. An analysis of computer-related patient safety incidents to inform the development of a classification. J Am Med Inform Assoc 2010;17:663–70. [PMC free article] [PubMed]
17. Runciman WB, Williamson JA, Deakin A, et al. An integrated framework for safety, quality and risk management: an information and incident management system based on a universal patient safety classification. Qual Saf Health Care 2006;15(Suppl 1):i82–90. [PMC free article] [PubMed]
18. National Reporting and Learning Service. 2010. http://www.nrls.npsa.nhs.uk/ (accessed 2 Nov 2010).
19. ISMP Medication Safety Alert! Newsletters. 2010. http://www.ismp.org/newsletters/default.asp (accessed 29 Dec 2010).
20. Hochberg AM, Hauben M, Pearson RK, et al. An evaluation of three signal-detection algorithms using a highly inclusive reference event database. Drug Saf 2009;32:509–25. [PubMed]
21. Meyboom RH, Lindquist M, Egberts AC, et al. Signal selection and follow-up in pharmacovigilance. Drug Saf 2002;25:459–65. [PubMed]
22. Pronovost PJ, Thompson DA, Holzmueller CG, et al. Toward learning from patient safety reporting systems. J Crit Care 2006;21:305–15. [PubMed]
23. Milch CE, Salem DN, Pauker SG, et al. Voluntary electronic reporting of medical errors and adverse events. An analysis of 92,547 reports from 26 acute care hospitals. J Gen Intern Med 2006;21:165–70. [PMC free article] [PubMed]
24. Zhan C, Smith SR, Keyes MA, et al. How useful are voluntary medication error reports? The case of warfarin-related medication errors. Jt Comm J Qual Patient Saf 2008;34:36–45. [PubMed]
25. Kennedy AG, Littenberg B, Senders JW. Using nurses and office staff to report prescribing errors in primary care. Int J Qual Health Care 2008;20:238–45. [PubMed]
26. Flynn EA, Barker KN, Pepper GA, et al. Comparison of methods for detecting medication errors in 36 hospitals and skilled-nursing facilities. Am J Health Syst Pharm 2002;59:436–46. [PubMed]
27. de Langen J, van Hunsel F, Passier A, et al. Adverse drug reaction reporting by patients in the Netherlands: three years of experience. Drug Saf 2008;31:515–24. [PubMed]
Articles from Journal of the American Medical Informatics Association : JAMIA are provided here courtesy of
American Medical Informatics Association