Search tips
Search criteria 


Logo of jgimedspringer.comThis journalToc AlertsSubmit OnlineOpen Choice
J Gen Intern Med. 2009 April; 24(4): 520–525.
Published online 2009 January 21. doi:  10.1007/s11606-009-0910-3
PMCID: PMC2659152

Identifying Patients with Post-discharge Care Problems Using an Interactive Voice Response System

Alan J. Forster, MD, FRCPC, MSc,corresponding author1,2,6 Loree Boyle, MD,1 Kaveh G. Shojania, MD,3 Thomas E. Feasby, MD, FRCPC,4 and Carl van Walraven, MD, FRCPC, MSc1,2,5



Adverse events (AEs) are poor outcomes caused by medical care. They occur in 20% of medical patients following hospital discharge. We designed an interactive voice response system (IVRS) with the intent of identifying patients who might be experiencing an AE following discharge or were at risk of developing one.


We determined the proportion of post-discharge patients requiring an intervention after identifying potential problems using the IVRS, the relationship between IVRS responses and AE occurrence, and patients’ opinions of the IVRS call.


We studied patients discharged from the general medical service of an academic hospital. The IVRS called patients 2 days post-discharge and asked three questions to determine the need for nurse follow-up. We contacted patients 30 days later to elicit AE status and perceptions of the IVRS.


Our cohort consisted of 270 elderly patients [median 64 years (IQR 50-76)] with multiple co-morbidities. Responses to the IVRS identified 57 patients (21%, 95% CI 17%-27%) for follow-up. When contacted by a nurse, 25 patients (9%, 95% CI 6%-13%) actually required an intervention. At 30-day follow-up, AEs occurred in 33 patients (12%, 95% CI 8%-17%). Only three AEs (9%) were identified by the IVRS; the remainder occurred before or after the IVRS call. Patients remembering the IVRS call found it easy to use (97%), and a minority would prefer a person to call (8%).


An IVRS-based method of monitoring was acceptable to patients and identified a significant proportion requiring changes in management. However, the method identified only a minority of AEs. To have a significant improvement in care, this method will need to be combined with other interventions.

Electronic supplementary material

The online version of this article (doi:10.1007/s11606-009-0910-3) contains supplementary material, which is available to authorized users.

Key Words: interactive voice response system, adverse events, hospital discharge


The transition from hospital to home poses considerable risk to patients.1-9 Adverse events, defined as poor outcomes due to medical care, occur in one in five medical patients in the first month following discharge.5,6 A significant proportion of adverse events have important clinical consequences, including death, and at least half of them are preventable or ameliorable.5,6 To improve the safety and effectiveness of hospital care, it is important to develop strategies to minimize risk during the transition home.

One factor contributing to post-discharge adverse events is inadequate monitoring.1,5,6 In hospital, physicians and other health professionals intensely monitor patients and their treatments. If a patient’s condition deteriorates, then it is usually recognized and acted upon quickly. After discharge, this is not the case. In many instances, patients experience warning symptoms yet do not act in a timely manner. Several studies suggest a simple telephone intervention may improve identification of these early symptoms and in turn reduce the risk of them progressing.2,3,8,10 Although this intervention is promising, resource limitations impede uptake.

We have devised an efficient method of monitoring patients after discharge using an interactive voice response system (IVRS). The IVRS performs an automated survey shortly after discharge to screen patients for those who might be having a problem. For screen-positive patients, the IVRS electronically notifies a nurse who is responsible for personally following up. In a pilot study, this monitoring method was simple to implement, acceptable to patients, and had potential for reducing adverse events.11

Our initial study of the IVRS was focused primarily on testing its feasibility rather than its effectiveness. As a result, we did not follow patients after their IVRS call to determine adverse event status or opinions pertaining to the IVRS. To address these limitations, we planned the current study. Our primary objective was to determine the proportion of study patients in whom the IVRS helped to identify a need for nurse intervention. Secondarily, for patients experiencing adverse events, we wanted to describe the timing and results of the IVRS call in relationship to their adverse event. Thirdly, we determined patients’ opinions of the call dialogue used. By meeting these objectives, we felt we would determine the potential effectiveness of an IVRS solution for reducing adverse events. Furthermore, we would identify how the IVRS solution could be modified to maximize its potential benefit.


Design, Setting and Patients

We used a prospective cohort design to study patients discharged to home from the General Internal Medicine Service at the Ottawa Hospital, Civic Campus. The service has three medical teams, each staffed by an attending physician, a senior medical resident, two to three junior residents, and varying numbers of medical students. These services function with a hospitalist model. With few exceptions, patients admitted to the service are unknown to the attending physician and very few are followed by the attending physician after discharge.

Consecutive patients admitted to this service between 5 May 2006 and 10 January 2007 were screened for eligibility. Patients were ineligible if they did not have a telephone, did not speak English, had cognitive impairment, or did not provide informed consent. We enrolled patients during their inpatient stay when their discharge disposition became apparent. The study was approved by the Ottawa Hospital’s Research Ethics Board.


After obtaining patient consent, we collected demographic and medical data. On the day of the patient’s discharge, we entered the patient’s telephone number, study ID number, and discharge date into the IVRS. The IVRS called patients 2 days after their discharge starting at 9:00 am. If there was no response to the call, the IVRS re-attempted the call every 2 h until a response occurred. If no response occurred before 9:00 pm that day, the system temporarily stopped calling the patient until the next morning. The system continued this pattern until a call was answered or 2 days had passed.

When a call was answered, the IVRS asked a series of questions that required a yes/no answer. The IVRS used telephone key pad entry for patients to update the database and prompt further actions. The IVRS provided instructions for the patient to press ‘1’ for ‘yes' and ‘2’ for ‘no.’ The system first asked: ‘Are you the patient recently discharged from the Ottawa Hospital?’ If the person responded ‘no' then the IVRS thanked the individual and disconnected. If the person responded 'yes' then IVRS asked three questions: ‘Since discharge, have you had any new or worsening symptoms?’ ‘Since your discharge, have you had any problems related to your medications?’ ‘Would you like to speak to a nurse?’ The responses to these questions were stored by the IVRS.

If the patient responded ‘yes’ to any question, the IVRS system sent an e-mail to the nurse coordinator of the medical team who discharged the patient. The nurse coordinator is a registered nurse who works with the medical team to facilitate various processes of care, including coordination of post-discharge tests and appointments.12 The e-mail identified the patient by the ID number and indicated the patient’s response to the questions. The nurse coordinator then telephoned the patient as soon as possible after receiving the e-mail. She recorded the reason why the patient required follow-up and how she dealt with the problem using a standard form. We used this information to determine the proportion of study patients requiring a nurse intervention.

The IVRS we used was CallAssure™. CallAssure can be run on any modern personal computer equipped with a telephony card and at least two analog lines. In addition, our system required access to an e-mail server (for notification messages).

Follow-up and Outcomes

Thirty days after hospital discharge, we manually contacted all patients by telephone to administer a telephone survey. Patients were asked if they remembered the automated call and to provide their opinion about some aspects of the IVRS, including perceived utility. They were asked: “Was the call useful to you?” “Did you understand the questions?” “Did you find it easy to use?” “Would you prefer a person to call?”

To determine adverse event status, we replicated methods previously developed and used by our group. At the 30-day interview, we determined whether the patient had any new or worsening health problems following discharge and whether they required any visits to health providers. For patients who had an encounter at our hospital, we supplemented data from the telephone survey with patient information collected from the hospital’s electronic health record, emergency department records of treatment, and hospital paper charts. This information was summarized into a case summary.

Two physicians (AF and LB) independently reviewed case material and determined whether or not each person had an adverse event. If cases were deemed adverse events, they were further classified according to preventability (adverse events were preventable if two reviewers judged that it could have been avoided), ameliorability (adverse events were ameliorable if two reviewers judged that its severity could have been significantly reduced if there was an appropriate response), severity, and type. Reviewers were blinded to role of IVRS in patients’ care.

We used the IVRS call statistics to determine the potential impact on reducing nursing workload with respect to making follow-up calls. We determined patients’ opinions regarding the automated call by asking them four questions.

Statistics and Data Management

We used SASv9.1 (Cary, NC) for all data analysis. Descriptive statistics were reported using median and inter-quartile range (IQR) for continuous variables and frequency distributions for categorical variables. For outcomes, we present the proportion of patients with the outcome of interest with 95% confidence intervals calculated using the Wilson score method.13

Role of the Sponsors

The sponsors did not have any role in the design, conduct, analysis, or manuscript preparation for the study.


Figure Figure11 illustrates the study flow. During the study period, 1,442 patients were discharged alive from the internal medicine service, 789 of whom went to their own home directly. Of the patients, 464 went home on days when our research assistant was available. Of these, 191 patients were not enrolled into the study because we were unable to approach them before discharge, they were cognitively impaired, they did not speak English, or they refused consent. Patients refused consent because of an unwillingness to participate in a research project and because they felt uncomfortable with the telephone technology. We did not collect information on patients not enrolled in the study so cannot provide the number of patients refusing consent. Thus, we enrolled 273 patients during the study period. Three patients were lost to follow-up, leaving 270 patients for whom we have 30-day follow-up data.

Figure 1
Study flow.

Table 1 describes the 270 patients in our cohort. The median age was 64 years (IQR = 50-76). Most patients had at least one chronic illness, with 43% reporting some form of heart disease and 28% reporting diabetes mellitus. Ninety-seven percent of patients had been admitted to hospital at least once in the preceding year; 21 percent of patients reported at least one limitation in activity of daily living. The median length of stay was 5 days (inter-quartile range 3-10 days).

Table 1
Patient Characteristics

Call statistics and survey responses are provided in Table 2. The IVRS made 912 calls to the 270 patients in the study. Fifty-four percent of patients did not answer the first call attempt. A considerable proportion received several calls, including 40 patients who were called more than six times before they answered. Of the automated telephone calls to the 270 patients, 239 calls (89%) were answered by a person rather than an answering machine. Of these 239 calls, 173 were answered by the patient. Thus, the IVRS connected to 173 of 270 patients (64%). Patients who did not answer the IVRS were similar in terms of their baseline characteristics to those who did.

Table 2
Feasibility and Usability of Interactive Voice Response System

Overall, 57 of 270 patients (21% of the study cohort, 95% CI 17%-26%) indicated that they were having a problem or that they needed to speak to a nurse (Table 2). Twenty-two patients (8%) reported having new or worsening symptoms, 22 (8%) reported having problems related to medications, and 45 (17%) reported a desire to speak with a nurse. Of these patients, 20 responded ‘yes’ to the question regarding nurse contact only.

Actions taken by the nurse in response to patients’ requests are described in Table 2. Of the 57 patients requiring help, the nurse successfully contacted 54 (95%). Of these, 25 had a problem that required a change in management, with the other 29 wishing confirmation that they were fine. Thus, our system led to interventions in 25/270 patients (risk = 9% 95% CI 6%-13%). Of the 25 patients in whom the nurse modified therapy, medication changes were required for 15. These included ensuring the patient took the medication, phoning in a prescription, and advising the patient to discontinue a specific medication. Other changes in therapy that were not related to medications included advice regarding follow-up and facilitating contact with the patient’s primary care provider. The nurse responses are described in Appendix 1 (available online).

Thirty-three of 270 patients (12%, 95% CI 8%-17%) experienced an adverse event in the 30 days following hospital discharge. Of the adverse events, 20 were considered preventable (7%, 95% CI 5%-11%), and four were considered ameliorable (1%, 95% CI 1%-4%). Of all adverse events, the types included: therapeutic errors (n = 13, 5%, 95% CI 3%-8%), adverse drug events (n = 12, 4%, 95% CI 3%-8%), system errors (n = 4, 1%, 95% CI 1%-4%), and miscellaneous other types. Ninety-one percent of all adverse events required a visit to a hospital (including emergency department visits), whereas the remainder were managed in doctors offices. A description of the adverse events is given in Appendix 2 (available online).

Adverse event status in relationship to the call responses is illustrated in Figure Figure2.2. For patients not responding to the IVRS, the adverse event risk was 5/31 (16%). For patients in whom the IVRS call was received but an incorrect person picked up the phone, the adverse event risk was 10/66 (15%). For the patients who correctly identified themselves to the IVRS but did not answer any further questions, the adverse event risk was 1/8 (13%). For patients indicating to the IVRS no need for nurse follow-up, the adverse event risk was 9/108 (8%). For patients who indicated a need for follow-up but the nurse was unable to contact, the adverse event risk was 1/3 (33%). For patients who indicated a need for follow-up but in whom no changes in care were made, the adverse event risk was 4/29 (14%). For patients who indicated a need for follow-up and in whom changes in care were made, the adverse event risk was 3/25 (12%).

Figure 2
Impact of IVRS-based monitoring system.

Thus, the risk of adverse events was similar irrespective of responses to the IVRS. The one group in whom there appeared to be a reduction in adverse event risk was patients who indicated they were doing fine when they interacted with the IVRS (risk = 9/108 or 8%, 95% CI 4%-15%)) versus all other patients (risk = 24/162 or 15%, 95% CI 10%-21%). This translates to a relative risk of 0.6, which is not statistically significant (95% CI 0.3-1.2).

Fifteen of 33 patients with adverse events were not contacted by the IVRS. The majority of these adverse events became manifest early after discharge as they occurred due to treatment while the patient was still in the hospital. For example, one patient developed symptoms as a result of a viral gastroenteritis acquired in hospital. Another patient required urgent cardiac revascularization despite a decision to the contrary while an inpatient. Ten of 33 patients with adverse events were contacted by the IVRS but indicated no need for nurse intervention. In nine of these cases, the patient was stable at the time of the call, and the adverse event occurred suddenly several days to weeks afterwards. For example, one patient had a sudden pulmonary embolism several weeks after discharge; another patient had sudden rectal bleeding as a result of anticoagulant therapy. Eight of 33 adverse event patients indicated a need to be contacted by a nurse by way of the IVRS. During five of the nurse follow-up calls, the patients indicated they were doing fine. However, the patients subsequently developed problems. Three of the patients were experiencing symptoms related to the adverse event at the time of the call. In these cases, the nurse helped the patient by arranging care related to the adverse event.

Patient perceptions of the IVRS are also presented in Table 2. During the follow-up call at 30 days, only 132 of the 173 patients (76%) actually remembered receiving an automated call from the IVRS. Of these patients, 119/132 (96%) stated that they understood the system, 120/132 (97%) stated that they found it easy to use, and 120/132 (97%) found the automated call useful. Only 10/132 patients (8%) indicated that they preferred a person to call them rather than the IVRS.


Our IVRS-based system led to health-care interventions in almost 10% of patients discharged from the hospital. The IVRS identified these patients even though it failed to make contact with over one-third of the cohort. Despite the interventions by nurses, one in eight patients experienced an adverse event within 30 days of discharge, almost two-thirds of which were considered preventable. Of the 33 adverse events, the IVRS intervention did not identify 30. This was because approximately half of the patients with adverse events did not respond to the IVRS. In addition, the remaining adverse events occurred either before or after the IVRS call, making it very challenging to capture them using a single telephone contact.

The current study design does not allow us to test a hypothesis regarding the effect of the IVRS intervention on adverse event risk. However, our intervention led to changes in management to 9% of patients. This finding alone might justify implementing an IVRS-based method of patient follow-up, except it is unclear whether anything would have happened to these patients if they had not been contacted by the IVRS. We do not know whether the inpatient physicians had anticipated the problems identified. We also do not know whether patients might have sought care elsewhere before the problem influenced their health, in the absence of the IVRS call. In order to better understand the impact of the IVRS, a randomized trial of the intervention versus routine care is required.

Even though the IVRS intervention resulted in care modifications for some patients, there are three reasons why it alone was not sufficient to prevent adverse events. First, approximately one-third of patients were not contacted by the IVRS. Thirty-one of the patients did not pick up the IVRS call after repeated attempts. It is unknown why they did not pick up the phone. It could be a reluctance to answer telephone calls from unknown numbers or it could be that they were not at home when called due to a return to normal activities or a return to hospital. Sixty-six patients did not answer affirmatively when asked by the IVRS if they were the correct patient. This could have occurred because it was not the correct person, a failure to understand how to correctly enter information into the IVRS, or a desire to provide purposefully misleading responses. Regardless of why the patient did not answer the phone, the IVRS cannot prevent adverse events if the patient does not respond to it. Second, many adverse events occurred before the IVRS call. These adverse events were often due to inappropriate treatment decisions in the hospital. In these patients, the best the IVRS intervention can do is flag the patient for more intensive follow-up. Third, in other instances the patient was doing fine at the time of the IVRS call, but had a sudden unpredictable deterioration, such as a gastrointestinal bleed, several days to weeks later. For these types of problems, it is unlikely the IVRS will be of any value at all.

These results suggest a need to modify the IVRS intervention to make it more effective. If there were better education at the time of discharge regarding a pending automated call to assess symptoms, then we might increase the proportion of patients responding. If the IVRS had incorporated a ‘call-in’ function to allow patients to notify us when they were having a problem rather than waiting to receive a call, then we might have identified more situations in which care required modification. Finally, if we had programmed the IVRS to perform a second follow-up call at 2 weeks, then we might have identified more impending adverse events.

Based on our data, the IVRS intervention could, at best, play an adjunctive role to other interventions designed to improve post-discharge care. In these cases, the IVRS might enhance the efficiency and effectiveness of providing so-called ‘after-care.’ For instance, Coleman et al.’s ‘transition coach’ concept, in which a nurse performs home visits following discharge and teaches patients to enhance their knowledge of their disease and its management,3 has been shown to reduce poor outcomes following discharge. The transition coach intervention might be less expensive if an IVRS identified high-risk patients for more intensive follow-up.

We believe the IVRS intervention implemented in this study is feasible in most settings. The information technology infrastructure required is not sophisticated. It does not require an electronic health record, although one could be leveraged to make the process even easier to implement. We manually entered patient information into our IVRS. We also created meaningless unique identifiers to enhance the security related to e-mail communications. Finally, we manually documented the nurse’s follow-up of IVRS-related calls. Integration with a hospital’s existing information system infrastructure would make many of these steps unnecessary or simpler. The human resources required by the IVRS are similarly non-intensive.

Our study was designed to establish the potential utility and effectiveness of the technology in order to plan for future controlled experiments. It is not possible to make inferences about the overall effectiveness of the intervention in reducing adverse events following discharge because the study was performed in a single center, had a relatively small sample size, and did not have a contemporaneous control group. However, from our experience at this institution, we believe that patients accept the technology, find it easy to use, and prefer it to personal follow-up calls. Other investigators have found similar patient opinions regarding IVRSs.14-19

We conclude that the IVRS-based intervention holds some promise in reducing adverse events post discharge. Some organizations might be tempted to implement such a program. We do not support such an approach given our uncertainty about the system’s effectiveness. Furthermore, costs for the system are not inconsequential. These include the IVRS itself plus annual license and support agreements, implementation costs for system integration, and nursing costs to monitor and follow-up calls. For these reasons, we recommend the completion of a multi-site randomized control trial to establish the net benefit of an IVRS.

Electronic supplementary material

Below is the link to the electronic supplementary material

ESM 1(110K, doc)

(DOC 109 kb)


Dr. Forster is supported by an Ontario Ministry of Health Career Scientist Award, and Dr. Shojania is supported by a Government of Canada Research Chair in Patient Safety and Quality Improvement.

This research protocol received funding from: The Canadian Patient Safety Institute, The Ottawa Hospital Center for Patient Safety, and Capital Health, Edmonton and area. Vocantas Inc. provided technical support for the project.

Conflict of Interest None disclosed.


Electronic supplementary material

The online version of this article (doi:10.1007/s11606-009-0910-3) contains supplementary material, which is available to authorized users.


1. Coleman EA, Berenson RA. Lost in transition: Challenges and opportunities for improving the quality of transitional care. Ann Intern Med. 2004;1417533–36. [PubMed]
2. Coleman EA, Boult C, American Geriatrics Society Health Care Systems Committee. Improving the quality of transitional care for persons with complex care needs. Journal of the American Geriatrics Society. 2003;514556–57. [PubMed]
3. Coleman EA, Parry C, Chalmers S, Min SJ. The Care Transitions Intervention: Results of a randomized controlled trial. Arch Intern Med. 2006;166171822–28. [PubMed]
4. Coleman EA, Mahoney E, Parry C. Assessing the quality of preparation for posthospital care from the patient’s perspective: the care transitions measure. Medical Care. 2005;433246–55. [PubMed]
5. Forster AJ, Clark HD, Menard A, et al. Adverse events affecting medical patients following discharge from hospital. CMAJ. 2004;1703345–49. [PMC free article] [PubMed]
6. Forster AJ, Murff HJ, Peterson JF, Gandhi TK, Bates DW. The incidence and severity of adverse events affecting patients after discharge from the hospital.[see comment]. Annals of Internal Medicine. 2003;1383161–7. [PubMed]
7. Roy CL, Poon EG, Karson AS, et al. Patient safety concerns arising from test results that return after hospital discharge. Ann Intern Med. 2005;1432121–8. [PubMed]
8. Schnipper JL, Kirwin JL, Cotugno MC, et al. Role of pharmacist counseling in preventing adverse drug events after hospitalization. Archives of Internal Medicine. 2006;1665565–71. [PubMed]
9. Forster AJ, Murff HJ, Peterson JF, Gandhi TK, Bates DW. Adverse drug events occurring following hospital discharge. Journal of General Internal Medicine. 2005;204317–23. [PMC free article] [PubMed]
10. Dudas V, Bookwalter T, Kerr KM, Pantilat SZ. The impact of follow-up telephone calls to patients after hospitalization. Disease-A-Month. 2002;484239–48. [PubMed]
11. Forster A, van Walraven C. Using an interactive voice response system to improve patient safety following hospital discharge. Journal of Evaluation in Clinical Practice. 2007;133346–51. [PubMed]
12. Forster AJ, Clark HD, Menard A, et al. Effect of a nurse team coordinator on outcomes for hospitalized medicine patients. The American Journal of Medicine. 2005;118101148–53. [PubMed]
13. Newcombe RG. Two-sided confidence intervals for the single proportion: comparison of seven methods. Statistics in Medicine. 1998;178857–72. [PubMed]
14. DeMolles DA, Sparrow D, Gottlieb DJ, Friedman R. A pilot trial of a telecommunications system in sleep apnea management. Medical Care. 2004;428764–69. [PubMed]
15. Weiler K, Christ AM, Woodworth GG, Weiler RL, Weiler JM. Quality of patient-reported outcome data captured using paper and interactive voice response diaries in an allergic rhinitis study: is electronic data capture really better? Annals of Allergy, Asthma, & Immunology. 2004;923335–9. [PubMed]
16. Mahoney D, Tennstedt S, Friedman R, Heeren T. An automated telephone system for monitoring the functional status of community-residing elders. Gerontologist. 1999;392229–34. [PubMed]
17. Mahoney DF, Tarlow BJ, Jones RN. Effects of an automated telephone support system on caregiver burden and anxiety: findings from the REACH for TLC intervention study. Gerontologist. 2003;434556–67. [PubMed]
18. Mahoney DF, Tarlow B, Sandaire J. A computer-mediated intervention for Alzheimer’s caregivers. Computers in Nursing. 1998;164208–16. [PubMed]
19. Stuart GW, Laraia MT, Ornstein SM, Nietert PJ. An interactive voice response system to enhance antidepressant medication compliance. Topics in Health Information Management. 2003;24115–20. [PubMed]

Articles from Journal of General Internal Medicine are provided here courtesy of Society of General Internal Medicine