Effect of an interactive voice response system on oral anticoagulant management

doi:10.1503/cmaj.081659

CMAJ. 2009 April 28; 180(9): 927.

PMCID: PMC2670905

Effect of an interactive voice response system on oral anticoagulant management

Natalie Oake, MSc, Carl van Walraven, MD MSc, Marc A. Rodger, MD MSc, and Alan J. Forster, MD MSc

From the Clinical Epidemiology Program (Oake, van Walraven, Rodger, Forster), Ottawa Health Research Institute, The Ottawa Hospital; the Department of Medicine (van Walraven, Rodger, Forster), University of Ottawa, Ottawa, Ont.; and the Institute for Clinical Evaluative Sciences (van Walraven), Toronto, Ont

Correspondence to: Dr. Alan Forster, Clinical Epidemiology Program, Ottawa Health Research Institute, 1053 Carling Ave., Administrative Services Building 1-008, Ottawa ON K1Y 4E9; Email: aforster/at/ohri.ca

See commentary "Can anticoagulation care be improved with technology?" on page 909.

This article has been cited by other articles in PMC.

Abstract

Background

Monitoring oral anticoagulants is logistically challenging for both patients and medical staff. We evaluated the effect of adding an interactive voice response system to computerized decision support for oral anticoagulant management.

Methods

We developed an interactive voice response system to communicate to patients the results of international normalized ratio testing and their dosage schedules for anticoagulation therapy. The system also reminded patients of upcoming and missed appointments for blood tests. We recruited patients whose anticoagulation control was stable after at least 3 months of warfarin therapy. We prospectively examined clinical data and outcomes for these patients for an intervention period of at least 3 months. We also collected retrospective data for each patient for the 3 months before study enrolment.

Results

We recruited 226 patients between Nov. 23, 2006, and Aug. 1, 2007. The mean duration of the intervention period (prospective data collection) was 4.2 months. Anticoagulation control was similar for the periods during and preceding the intervention (mean time within the therapeutic range 80.3%, 95% confidence interval [CI] 77.5% to 83.1% v. 79.9%, 95% CI 77.3% to 82.6%). The interactive voice response system delivered 1211 (77.8%) of 1557 scheduled dosage messages, with no further input required from clinic staff. The most common reason for clinic staff having to deliver the remaining messages (accounting for 143 [9.2%] of all messages) was an international normalized ratio that was excessively high or low, (i.e., 0.5 or more outside the therapeutic range). When given the option, 76.6% of patients (164/214) chose to continue with the interactive voice response system for management of their anticoagulation after the study was completed. The system reduced staff workload for monitoring anticoagulation therapy by 48 min/wk, a 33% reduction from the baseline of 2.4 hours.

Interpretation

Interactive voice response systems have a potential role in improving the monitoring of patients taking oral anticoagulants. Further work is required to determine the generalizability and cost-effectiveness of these results.

Health care institutions are increasingly using interactive voice response systems. Applications of this technology include screening for disease (e.g., depression),¹ monitoring disease symptoms,² monitoring behaviour (e.g., substance abuse),³ conducting behavioural counselling,⁴ assessing adherence to medication regimens⁵ and increasing attendance at appointments. In a recent systematic review, our group comprehensively described the populations, interventions and outcomes of clinical trials involving interactive voice response systems.⁶ Most of these trials examined low-risk interventions and low-risk patient populations. None of the studies examined an intervention in which patients received instructions about their medication therapy without human intervention.

Previous authors have reported a combined thromboembolic and hemorrhagic event rate of 15% per year and a fatal hemorrhagic event rate of 1% per year among patients taking oral anticoagulants.⁷^–⁹ Clinical event rates are typically minimal when patients’ international normalized ratio are within the therapeutic range,¹⁰ but more than half of adverse events related to oral anticoagulation are directly attributable to international normalized ratio values outside the therapeutic range.¹¹

More than half of the adverse drug events that affect ambulatory patients occur because of inadequate monitoring,¹²^,¹³ making this aspect of care an attractive target for safety strategies. Several interventions, including the use of automated telephone messages¹⁴ and academic detailing,¹⁵ appear to improve physician and patient compliance with laboratory monitoring. Strategies to enhance monitoring could be improved by targeting patients who take high-risk medications, such as oral anticoagulants.¹³^,¹⁶

Many anticoagulation clinics already use computerized decision support systems, because these systems are effective in improving anticoagulation control.¹⁷^–²⁰ However, even with the help of a computerized decision support system, monitoring patients is labour-intensive, because of the staff time required to call patients to communicate medication instructions. Additional staff time is needed to follow up with patients who forget to go for scheduled blood tests. Therefore, we developed an information technology-based solution to facilitate the monitoring of patients receiving oral anticoagulants. We were uncertain whether our system could relay complex instructions to patients in a manner that would be acceptable to the patients. In addition, the potential risk associated with using the system was high, since patients would be receiving dosage instructions without human intervention for a medication with a narrow therapeutic window. Therefore, we used a “proof-of-concept” approach to evaluate the feasibility of adding an interactive voice response system to a computerized decision support system under optimal conditions.

Methods

Study design and setting

We conducted an observational study using a 1-group, pretest–posttest design.²¹ Data collection was retrospective for the pre-intervention period and prospective for the intervention period. The pre-intervention period (retrospective) was the 3-month period before a patient’s enrolment in the study. The intervention period (prospective) was the 3-month period after study enrolment. We extended the prospective observation period for any patient who chose to continue with the intervention beyond 3 months.

We conducted this study within the Oral Anticoagulation Management Service of The Ottawa Hospital Thrombosis Program in Ottawa, Canada. At the time of the study, this service was monitoring about 1200 patients from eastern Ontario who were receiving oral anticoagulation therapy. Program staff used a computerized decision support system (DawnAC, 4S Information Systems Ltd., Milnthorpe, England) to assist in monitoring patients.

The study was approved by the Ottawa Hospital Research Ethics Board.

Eligibility criteria

Patients were potentially eligible for the study if they had completed 3 months of oral anticoagulation therapy with warfarin and their anticoagulation control was stable. We defined “stable control” as 2 consecutive international normalized ratio values within the therapeutic range during the month before recruitment. We used stability of control as an eligibility criterion because we wanted to evaluate the effectiveness of the interactive voice response system under optimal conditions.

We did not exclude patients who had experienced previous hemorrhagic or thromboembolic events. We did exclude patients whose anticoagulation control was unstable and those who did not speak English, were receiving an oral anticoagulant other than warfarin, had hearing problems, received calls at a telephone number with an extension, self-managed their warfarin dosage or had plans to stop being monitored by the clinic. We recruited consecutive patients from the clinic who met the inclusion criteria during the recruitment period.

Intervention

We linked an interactive voice response system (Call-AssureCDM, Vocantas Inc., Ottawa, Canada) to a computerized decision support system and a telephone network to facilitate monitoring of oral anticoagulation therapy. Interactive voice response systems allow a computer database to communicate with people via the telephone,²² for example, by automatically calling to deliver and retrieve information. In our case, health care professionals used the computerized decision support system to determine the new oral anticoagulant dosage and timing of the next international normalized ratio test.

The interactive voice response system then communicated this information to patients automatically. The interactive voice response system communicated with patients using 3 types of messages: dosage, reminder and missed. The “dosage” message reported the patient’s latest international normalized ratio, the weekly dosage schedule and the date of the next appointment for testing of international normalized ratio. This message also asked the patient if he or she wanted to speak with someone from the clinic and if he or she had started any new medications. If the patient responded “yes” to either question, the interactive voice response system notified a health care professional via email to follow-up with the patient. The “reminder” message notified patients of upcoming appointments for international normalized ratio testing. We programmed the interactive voice response system to deliver this message 2 days before the patient’s appointment. The “missed” message notified patients who had missed an appointment for testing of international normalized ratio and asked them to go for testing the next day.

The interactive voice response system documented the details (e.g., date and time) of all calls made in a report that could be accessed via a web-based interface. A health care professional reviewed this report daily and contacted any patients for whom delivery of a dosage message had been unsuccessful. We did not attempt to contact patients for whom delivery of a reminder or missed message was unsuccessful.

Protocol for individual patients

We recruited eligible patients by telephone using a standardized script.

The intervention period for a particular patient started with his or her first international normalized ratio test after provision of consent. We followed each patient for a minimum of 3 months, during which time the patient had regular appointments for testing of international normalized ratio (Appendix 1, available at www.cmaj.ca/cgi/content/full/180/9/927/DC2).

The laboratories forwarded international normalized ratio results to the clinic, as usual. The pharmacist manually entered each test result into the patient’s profile in the computerized decision support system. The support system then recommended a new oral dosage of anticoagulant and the date of the next international normalized ratio test. The pharmacist reviewed and approved or changed these recommendations. The interactive voice response system then called the patient to deliver a dosage message. If the system reached the patient, a caregiver or an answering machine, the message was delivered. Otherwise, the system disconnected and attempted to contact the patient later. We programmed the interactive voice response system to make up to 3 attempts to contact each patient. At the end of the study, we contacted patients and used a semistructured interview to elicit feedback about the interactive voice response system. We gave patients the option of continuing with the interactive voice response system or returning to the clinic’s standard monitoring system. We extended the study observation period for those patients who chose to continue with the interactive voice response system.

Study outcomes

The primary outcome of the study was anticoagulation control.²³ We collected patients’ international normalized ratio data from the clinic’s computerized decision support system for both the pre-intervention and intervention periods.

We also evaluated the interactive voice response system using a health technology assessment framework that incorporated outcome, process and structure indicators.²⁴ Within this framework, we measured 2 outcome indicators: the rate of hemorrhagic and thromboembolic events during the 2 study periods, and patients’ satisfaction with the interactive voice response system, defined as the proportion of eligible patients who continued with the interactive voice response system after the study.

We used the clinic’s computerized decision support system to identify hemorrhagic and thromboembolic events that occurred during the pre-intervention and intervention periods. We used semistructured interviews to assess patient satisfaction. Also within the health technology assessment framework, we measured 2 process indicators: the utility of the interactive voice response system, defined as the proportion of scheduled dosage messages that were successfully delivered by the interactive voice response system and that did not require further input from clinic staff, and the change in workload of the clinic staff. We calculated the utility of the interactive voice response system using data stored in the Web-accessible report generated by the system. For the second process indicator, we manually recorded and compared the time required to monitor the interactive voice response system and the time required to communicate with patients using the standard method. To generate the structure indicator, we described the setting and resources required to implement the interactive voice response system.

Data analyses and sample size calculation

We used 3 steps to determine the extent of anticoagulation control. First, we used linear interpolation²³ to calculate international normalized ratio values for the days between actual measurements. Second, for each patient, we calculated the proportion of days, for his or her total observation period, on which the international normalized ratio was within the therapeutic range. Third, we calculated the overall mean (i.e., the mean of individual patient proportions) and 95% confidence intervals (CIs). We applied this 3-step process to data for both the pre-intervention and intervention periods. We then performed a noninferiority test²⁵^,²⁶ to compare anticoagulation control for the 2 study periods. We selected a margin of non-inferiority of 5%, expressed as an absolute difference, based on the smallest minimal important difference²⁵ reported by published randomized controlled trials¹⁸^,²⁷^–³⁰ in which anticoagulation control was the primary outcome. We also conducted subgroup analyses to investigate if anticoagulation control during the intervention period differed according to patients’ indication for oral anticoagulant use, sex, age, duration of oral anticoagulant use and satisfaction with the interactive voice response system.

We included 2 regression models in our analyses. We used logistic regression³¹ to investigate if patient factors were associated with whether a patient continued with the interactive voice response system. In addition, we used Poisson regression³² to investigate whether the usefulness of the interactive voice response system was related to age.

To estimate the time required to monitor patients using the interactive voice response system, we measured the time required to perform each monitoring task over a 1-week period.

The monitoring tasks included identifying patients whose dosage messages were unsuccessful, contacting and receiving calls from patients, and relaying information about patients to the pharmacist. To estimate the time it would take staff to deliver the scheduled dosage messages, we determined the average time required to deliver 1 message, as follows. For 4 different periods in 1 week, the clerk at the clinic recorded the exact time required to successfully deliver information to 20 patients.

Because we had a single study group, we calculated the sample size using the desired final CI, rather than a predetermined effect size.³³ We selected a desired 95% CI around the intervention estimate of anticoagulation control of 8%. We also used data from a population-based study³⁴ to select an expected standard deviation estimate of anticoagulation control in a population (30.1%). We determined that 226 patients would be required for our study.

Results

Patient characteristics and follow-up

We recruited patients between Nov. 23, 2006, and Aug. 1, 2007. We approached a total of 355 patients (Figure 1). We excluded 41 patients because they did not meet the eligibility criteria at the start of the study. Of the 314 eligible patients, 88 patients declined to participate, 15 (17%) of these because they preferred not to participate in research. Therefore, we enrolled a total of 226 eligible patients in the study during the 9-month recruitment period. About half of the study participants were female (Table 1), and the median age was 58 (interquartile range 48–68, range 21–88) years. By far, the most common indication for warfarin was venous thromboembolism (179 patients [79.2%]). In total, 181 patients (80.0%) had been taking warfarin for longer than 1 year.

Figure 1

Flow diagram of patient recruitment, follow-up and analysis. Note: IVRS = interactive voice response system.

Table 1

Characteristics of the study population

The overall intervention period began on May 7, 2007, and ended on Dec. 14, 2007. We followed patients prospectively for a total of 942.2 months (78.5 years), with a mean follow-up period of 4.2 months (standard deviation 1.8 months, range 1 day to 7.2 months) (Table 1). We collected 3 months’ worth of retrospective data for each patient, for a total of 56.5 years. A total of 193 patients (85.4%) completed the 3-month intervention period (Figure 1). Twenty-one patients (9.3%) withdrew from the study because they found the automated instructions confusing or too fast (n = 19) or they missed the personal contact with clinic staff (n = 2). We excluded an additional 12 patients (5.3%) because they discontinued warfarin (n = 8) or stopped being monitored by the clinic (n = 4).

Outcome indicators

The primary outcome indicator was anticoagulation control, expressed as proportion of time within the therapeutic range. Anticoagulation control during the intervention period was similar to that during the pre-intervention period. With the interactive voice response system, international normalized ratio values were within the therapeutic range a mean of 80.3% of the time (95% CI 77.5% to 83.1%). In the pre-intervention period, values were within the therapeutic range a mean of 79.9% of the time (95% CI 77.3% to 82.6%). The mean difference in anticoagulation control between the 2 periods was 0.36% (95% CI –2.95% to 3.67%). This difference was non-inferior because the 95% CI of the mean difference included zero and excluded the 5% margin of noninferiority.

According to our subgroup analyses, anticoagulation control during the intervention period did not differ significantly by indication for oral anticoagulant use, sex, age, duration of oral anticoagulant use or patients’ satisfaction with the interactive voice response system.

No hemorrhagic or thromboembolic events occurred during the study period.

Most of the patients were satisfied with the interactive voice response system. A total of 164 patients continued with the interactive voice response system after the study (Table 2), representing 76.6% of the 214 patients who were eligible to do so. The most common reason for continuing with the interactive voice response system, cited by 86 patients, was its clear and timely delivery of information. Twenty-nine patients (13.6%) did not continue with the system after the study. The most common reasons for not continuing were missing the personal contact with staff (n = 12) and finding the automated instructions confusing or too fast (n = 12). Age was significantly associated with the decision to continue using the system. The likelihood of continuing use decreased with greater age (odds ratio 0.96, 95% CI 0.93 to 0.99).

Table 2

Patients’ satisfaction with the interactive voice response system

Process indicators

The interactive voice response system was useful for communicating information to patients, as indicated by the outcomes for dosage messages (Table 3). During the intervention period, we needed to deliver a total of 1557 dosage messages. The interactive voice response system successfully delivered 1211 (77.8%) of these without further input from clinic staff. The remaining 346 messages (22.2%) required input from clinic staff, most often because the patient’s international normalized ratio was 0.5 or more outside the therapeutic range (n = 143). The pharmacist contacted these patients to identify possible explanations for the out-of-range values. Overall, the system was unable to deliver 155 (10.0%) of the dosage messages (Table 3). Our Poisson regression model revealed a trend toward decreasing effectiveness of the interactive voice response system with increasing age of the patients.

Table 3

Overall utility of the interactive voice response system^*

Clinic staff spent a total of 1.6 hours per week monitoring the interactive voice response system. We found that it took 2 minutes and 50 seconds to deliver a single message in the absence of the interactive voice response system and estimated that it would have taken staff about 2.4 hours per week to deliver the scheduled messages. Therefore, we estimated that the system reduced overall staff workload by about 48 minutes per week (33%).

Structure indicators

We implemented the interactive voice response system in an anticoagulation clinic. Physicians, nurses and pharmacists work in the clinic and had experience using the computerized decision support system before our study began.

The bulk of the work after implementation involved monitoring the system’s functioning. This required clerical training and basic computing skills. We periodically required technical support from the manufacturer to trouble-shoot problems. However, none of the problems necessitated turning off the interactive voice response system.

Interpretation

In this proof-of-concept study, we evaluated the addition of an interactive voice response system to a computerized decision support system to facilitate the management of oral anticoagulation therapy in patients whose anticoagulation control was already stable. Anticoagulation control was similar with and without the interactive voice response system. Most of the patients were satisfied with the system. The system was effective in communicating complex information, as indicated by the high rate of successful delivery of messages (77.8%) without input from staff. Importantly, the interactive voice response system reduced the workload of clinic staff by 33%. However, these results may have limited generalizability to unselected patients.

Anticoagulation control in our patient population was exceptionally good at baseline. In a systematic review³⁵ of studies evaluating anticoagulation control, we found that the average time within the therapeutic range was 56.7% in community settings and 65.6% in anticoagulation clinics. In contrast, the patients in this study spent 79.9% of the time within the therapeutic range before the intervention, which made it very unlikely that the intervention would result in a significant improvement. Nonetheless, it was reassuring that anticoagulation control did not decrease during the intervention. In addition, the interactive voice response system required less work from staff members.

The absence of any hemorrhagic or thromboembolic events in our cohort might be considered notable. However, we feel that this result was unsurprising, for 3 reasons. First, we followed a relatively small number of patients for a short period of time (a total of 135.0 patient-years of observation). Given the overall risk of adverse events of 15% per year, it is possible that very few, if any, events would have occurred in our cohort simply by chance. Second, patients were within the therapeutic range close to 80% of the time during the study. Because the risk of adverse events is minimized when the international normalized ratio is within therapeutic range, the expected risk of events was low. Third, the patients in our cohort were younger and healthier than patients evaluated in previous studies. As age and comorbidity are strong predictors of the risk of bleeding, our study might have been biased toward a low rate of adverse events.

A previous evaluation of an interactive voice response system used in managing oral anticoagulation therapy³⁶ was limited because it did not measure anticoagulation control or structural indicators. Also, the system in the earlier study was not integrated with a computerized decision support system and was therefore more cumbersome to use. The interactive voice response system in our study was easy to use and could be implemented by large health management organizations. This decision support tool would be even more efficient if the computerized decision support system were integrated with a laboratory information system, so as to make manual data entry unnecessary.

Strengths and limitations

Our study had important strengths. First, we had perfect follow-up of our study population. All patients completed the post-study interview. In addition, once the study was complete, most chose to continue using the interactive voice response system rather than reverting to the standard monitoring system. Furthermore, the follow-up period for each patient was adequate to accurately assess anticoagulation control. Although the total observation time for the intervention period was 22 years longer than that for the pre-intervention period, we believe that the 2 periods were sufficiently long to justify our comparisons. Second, we selected our study population using clinically relevant, transparent eligibility criteria. These criteria yielded a clear inception cohort. In addition, a high proportion of the patients whom we approached agreed to participate, which indicates that the study population was highly representative of the clinic’s population. Third, our range of relevant, objective indicators provided a comprehensive assessment of the interactive voice response system.

Our study also had limitations. First, the study population consisted of a highly selected group of users of oral anticoagulants. Most of the patients monitored by the anticoagulation clinic had stable anticoagulation control and were therefore not representative of most community-based patients.³⁵ Second, we were unable to determine the true utility of the interactive voice response system because our study lacked a concurrent control. The results of this pretest–posttest study suggested that care with the interactive voice response system may be noninferior. However, the limited statistical power of the study and the lack of a randomized design prevented us from conclusively demonstrating noninferiority. A randomized controlled trial would be required to obtain more robust evidence.

Conclusions

We have demonstrated that interactive voice response systems have a potential role in improving the monitoring of patients who are taking oral anticoagulants. Future randomized studies will be required to conclusively demonstrate the effectiveness of this technology. Such studies should also be based on a more representative sample of users of oral anticoagulants. Given our experiences, we also recommend that this type of decision support tool be considered for management of other high-risk medication therapies for which laboratory monitoring is required.

Supplementary Material

[Résumé]

Click here to view.

[Online Appendix]

Click here to view.

Acknowledgements

We thank the staff of the Oral Anticoagulation Management Service, The Ottawa Hospital Thrombosis Program, especially Shemina Kherani, Lesley Yeung and Geoff Lewis.

Footnotes

Competing interests: There are no conflicts of interest between any of the authors and the company that manufactured the interactive voice response system, Vocantas Inc. The data were collected and analyzed independent of Vocantas Inc. Natalie Oake had full access to all of the study data and takes responsibility for the integrity of the data and the accuracy of the analyses.

Marc Rodger has received honoraria for speaking engagements and consulting from makers of oral anticoagulants, including Bayer, Pfizer and Boehringer Ingelheim. These funds were placed in the research trust funds of the Ottawa Health Research Institute.

None declared for Natalie Oake, Carl van Walraven and Alan Forster.

This article has been peer reviewed.

Contributors: All of the authors were involved in implementing the interactive voice response system at the study clinic. Natalie Oake recruited the study participants and collected the data. All of the authors were involved in the analyses and interpretation of the data, the writing and revision of the manuscript and gave approval of the final version for publication.

Funding: Marc Rodger is the recipient of a Career Investigator Award from the Heart and Stroke Foundation of Canada. Alan Forster holds a Career Scientist Award from the Ontario Ministry of Health and Long-Term Care and an Early Research Award from the Ontario Ministry of Research and Innovation.

REFERENCES

1. Kim H, Bracha Y, Tipnis A. Automated depression screening in disadvantaged pregnant women in an urban obstetric clinic. Arch Womens Ment Health. 2007;10:163–9. [PubMed]

2. Shaw WS, Verma SK. Data equivalency of an interactive voice response system for home assessment of back pain and function. Pain Res Manag. 2007;12:23–30. [PMC free article] [PubMed]

3. Gruenewald PJ, Searles J, Helzer J, et al. Exploring drinking dynamics using interactive voice response technology. J Stud Alcohol. 2005;66:571–6. [PubMed]

4. Friedman RH. Automated telephone conversations to assess health behavior and deliver behavioral interventions. J Med Syst. 1998;22:95–102. [PubMed]

5. Stuart GW, Laraia MT, Ornstein SM, et al. An interactive voice response system to enhance antidepressant medication compliance. Top Health Inf Manage. 2003;24:15–20. [PubMed]

6. Oake N, Jennings A, van Walraven C, et al. The use and effectiveness of interactive voice response systems for improving the delivery of ambulatory care: a systematic review and meta-analysis. Am J Manag Care

7. Ansell J, Jacobson A, Levy J, et al. Guidelines for implementation of patient self-testing and patient self-management of oral anticoagulation. Int J Cardiol. 2005;99:37–45. [PubMed]

8. Gitter MJ, Jaeger TM, Petterson TM, et al. Bleeding and thromboembolism during anticoagulant therapy: a population-based study in Rochester, Minnesota. Mayo Clin Proc. 1995;70:725–33. [PubMed]

9. Steffensen FH, Kristensen K, Ejlersen E, et al. Major haemorrhagic complications during oral anticoagulant therapy in a Danish population-based cohort. J Intern Med. 1997;242:497–503. [PubMed]

10. Oake N, Jennings A, Forster AJ, et al. Anticoagulation intensity and outcomes among patients prescribed oral anticoagulant therapy: a systematic review and meta-analysis. CMAJ. 2008;179:235–44. [PMC free article] [PubMed]

11. Oake N, Fergusson DA, Forster AJ, et al. Frequency of adverse events in patients with poor anticoagulation: a meta-analysis. CMAJ. 2007;176:1589–94. [PMC free article] [PubMed]

12. Gurwitz JH, Field TS, Harrold LR, et al. Incidence and preventability of adverse drug events among older persons in the ambulatory setting. JAMA. 2003;289:1107–16. [PubMed]

13. Gurwitz JH, Field TS, Judge J, et al. The incidence of adverse drug events in two large academic long-term care facilities. Am J Med. 2005;118:251–8. [PubMed]

14. Mundt JC. Interactive voice response systems in clinical research and treatment. Psychiatr Serv. 1997;48:611–2. [PubMed]

15. Lafata JE, Gunter MJ, Hsu J, et al. Academic detailing to improve laboratory testing among outpatient medication users. Med Care. 2007;45:966–72. [PubMed]

16. Forster AJ, Murff HJ, Peterson JF, et al. Adverse drug events occurring following hospital discharge. J Gen Intern Med. 2005;20:317–23. [PMC free article] [PubMed]

17. Fitzmaurice DA, Hobbs FD, Murray ET, et al. Evaluation of computerized decision support for oral anticoagulation management based in primary care. Br J Gen Pract. 1996;46:533–5. [PMC free article] [PubMed]

18. Fitzmaurice DA, Hobbs FD, Murray ET, et al. Oral anticoagulation management in primary care with the use of computerized decision support and near-patient testing: a randomized, controlled trial. Arch Intern Med. 2000;160:2343–8. [PubMed]

19. Poller L, Shiach CR, MacCallum PK, et al. Multicentre randomised study of com-puterised anticoagulant dosage. European Concerted Action on Anticoagulation. Lancet. 1998;352:1505–9. [PubMed]

20. Manotti C, Moia M, Palareti G, et al. Effect of computer-aided management on the quality of treatment in anticoagulated patients: a prospective, randomized, multi-center trial of APROAT (Automated PRogram for Oral Anticoagulant Treatment) Haematologica. 2001;86:1060–70. [PubMed]

21. Harris AD, McGregor JC, Perencevich EN, et al. The use and interpretation of quasi-experimental studies in medical informatics. J Am Med Inform Assoc. 2006;13:16–23. [PMC free article] [PubMed]

22. Lee H, Friedman ME, Cukor P, et al. Interactive voice response system (IVRS) in health care services. Nurs Outlook. 2003;51:277–83. [PubMed]

23. Rosendaal FR, Cannegieter SC, van der Meer FJ, et al. A method to determine the optimal intensity of oral anticoagulant therapy. Thromb Haemost. 1993;69:236–9. [PubMed]

24. Steuten L, Vrijhoef B, Severens H, et al. Are we measuring what matters in health technology assessment of disease management? Systematic literature review. Int J Technol Assess Health Care. 2006;22:47–57. [PubMed]

25. Jones B, Jarvis P, Lewis JA, et al. Trials to assess equivalence: the importance of rigorous methods. BMJ. 1996;313:36–9. [PMC free article] [PubMed]

26. Piaggio G, Elbourne DR, Altman DG, et al. Reporting of noninferiority and equivalence randomized trials: an extension of the CONSORT statement. JAMA. 2006;295:1152–60. [PubMed]

27. Fitzmaurice DA, Murray ET, McCahon D, et al. Self management of oral anticoagulation: randomised trial. BMJ. 2005;331:1057. [PMC free article] [PubMed]

28. Chan FWH, Wong RSM, Lau WH, et al. Management of Chinese patients on warfarin therapy in two models of anticoagulation service — a prospective randomized trial. Br J Clin Pharmacol. 2006;62:601–9. [PubMed]

29. Menendez-Jandula B, Souto JC. Comparing self-management of oral anticoagulant therapy with clinic management: a randomized trial. Ann Intern Med. 2005;142:1–10. [PubMed]

30. Wilson SJ, Wells PS, Kovacs MJ, et al. Comparing the quality of oral anticoagulant management by anticoagulation clinics and by family physicians: a randomized controlled trial [published erratum appears in CMAJ 2004;170:451] CMAJ. 2003;169:293–8. [PMC free article] [PubMed]

31. Kleinbaum DG, Gail M, editors. Modeling strategy guidelines: logistic regression — a self learning text. New York (NY): Springer-Verlag; 1994.

32. Stokes ME, Davis CS, Koch GG. Categorical data analysis using the SAS system. 2nd ed. Cary (NC): SAS Institute Inc; 2000.

33. Lewis RJ. Power analysis and sample size determination: concepts and software tools. Proceedings of the 2000 Annual Meeting of the Society for Academic Emergency Medicine; 2000 May 22–25; San Francisco (CA). [(accessed 2009 Mar. 30)]. Available: www.saem.org/download/lewis4.pdf.

34. van Walraven C, Austin PC, Oake N, et al. The effect of hospitalization on oral anticoagulation control: a population-based study. Thromb Res. 2007;119:705–14. [PubMed]

35. van Walraven C, Jennings A, Oake N, et al. Effect of study setting on anticoagulation control: a systematic review and metaregression. Chest. 2006;129:1155–66. [PubMed]

36. Cervi PL, Everitt AS. Automatic voice mail for delivering computer-generated anticoagulant dose advice to patients. J Telemed Telecare. 2002;8:259–63. [PubMed]

Articles from CMAJ : Canadian Medical Association Journal are provided here courtesy of
Canadian Medical Association