PMCCPMCCPMCC

Search tips
Search criteria 

Advanced

 
Logo of jgimedspringer.comThis journalToc AlertsSubmit OnlineOpen Choice
 
J Gen Intern Med. Apr 2013; 28(4): 513–521.
Published online Nov 15, 2012. doi:  10.1007/s11606-012-2271-6
PMCID: PMC3599033

Patient Completion of Laboratory Tests to Monitor Medication Therapy: A Mixed-Methods Study

ABSTRACT

BACKGROUND

Little is known about the contribution of patient behavior to incomplete laboratory monitoring, and the reasons for patient non-completion of ordered laboratory tests remain unclear.

OBJECTIVE

To describe factors, including patient-reported reasons, associated with non-completion of ordered laboratory tests.

DESIGN

Mixed-Methods study including a quantitative assessment of the frequency of patient completion of ordered monitoring tests combined with qualitative, semi-structured, patient interviews.

PARTICIPANTS

Quantitative assessment included patients 18 years or older from a large multispecialty group practice, who were prescribed a medication requiring monitoring. Qualitative interviews included a subset of show and no-show patients prescribed a cardiovascular, anticonvulsant, or thyroid replacement medication.

MAIN MEASURES

Proportion of recommended monitoring tests for each medication not completed, factors associated with patient non-completion, and patient-reported reasons for non-completion.

KEY RESULTS

Of 27,802 patients who were prescribed one of 34 medications, patient non-completion of ordered tests varied (range: 0–24 %, by drug-test pair). Factors associated with higher odds of test non-completion included: younger patient age (< 40 years vs. ≥ 80 years, adjusted odds ratio [AOR] 1.52, 95 % confidence interval [95 % CI] 1.27–1.83); lower medication burden (one medication vs. more than one drug, AOR for non-completion 1.26, 95 % CI 1.15–1.37), and lower visit frequency (0–5 visits/year vs. ≥ 19 visits/year, AOR 1.41, 95 % CI 1.25 to 1.59). Drug-test pairs with black box warning status were associated with greater odds of non-completion, compared to drugs without a black box warning or other guideline for testing (AOR 1.91, 95 % CI 1.66–2.19). Qualitative interviews, with 16 no-show and seven show patients, identified forgetting as the main cause of non-completion of ordered tests.

CONCLUSIONS

Patient non-completion contributed to missed opportunities to monitor medications, and was associated with younger patient age, lower medication burden and black box warning status. Interventions to improve laboratory monitoring should target patients as well as physicians.

KEY WORDS: laboratory monitoring, patient completion, drug research

INTRODUCTION

Laboratory monitoring of many prescription medications is recommended to minimize the risk of drug-associated injury.1,2 While failure to monitor medications is a leading contributor to adverse drug events (ADEs),2 and laboratory monitoring is suboptimal for many drugs,3,4 patient factors that contribute to under-monitoring remain unclear.

Prior studies report inadequate laboratory monitoring of medications, but these studies do not examine patient completion of ordered testing. Some studies report clinician ordering only,5,6 while others report overall test completion rates.711 In addition, little information is available about why patients fail to complete ordered laboratory tests. Patient understanding of the reason for testing may affect test adherence, as suggested by one study of warfarin.12 Forgetting is known to be a common reason for missing physician appointments,13,14 but its role in performing laboratory tests is unknown. Health system factors have also been identified as reasons why patients miss appointments.15,16 Related work on abandoned prescriptions has suggested that patients’ relationships with physicians, wait times in the pharmacy, condition of the facility, and co-payments are associated with increased medication abandonment.17 Evidence shows poor rates of patient medication adherence18 and appointment attendance,19 but the data are sparse regarding laboratory testing completion, especially with regard to ordering. Taken together, these knowledge gaps make it difficult to determine how to improve monitoring rates.

We conducted a mixed-methods study to characterize patient completion of laboratory test monitoring for chronic disease medications. Using a sequential approach, we used quantitative methods to first describe patient completion of laboratory monitoring tests for medications in the ambulatory setting and to characterize factors associated with test non-completion. This was followed by qualitative interviews with patients to explore explanations for test non-completion and to solicit recommendations for interventions.

METHODS

Study Design and Population

This study was conducted in a large multispecialty group practice that provides most of the medical care for members of a New England-based health plan. In 2010, the group practice employed 330 outpatient clinicians, including 250 physicians at 23 ambulatory clinic sites covering 30 specialties. The practice uses the EpicCare Ambulatory electronic medical record (EMR) system (Epic, Verona, WI, Spring 2007 IU3) and provides care to approximately 180,000 individuals. The age and gender characteristics of the study population are similar to those of the general population of the United States, and include 36 % aged 65 years and older. While the health plan does not systematically measure race, the plan’s market research indicates a racial mix consistent with the plan’s catchment area, which includes whites 79 %, Hispanics 12 %, African Americans 5 %, and other races 4 %.

Quantitative Methods

Inclusion Criteria

For the quantitative study, patients were included if they received care from the multispecialty group, were age 18 or older, and had insurance from the associated health plan during the period of January 1, 2007 to July 31, 2008. Patients were excluded if not continuously enrolled during this period or were residing in a long-term care facility. Patients had to be prescribed a medication from a list of drugs with laboratory monitoring guidelines that were developed for a clinical decision support system of the multispecialty group’s EMR (see Table 1), and have a monitoring test order in the EMR within the recommended time frame of the guidelines. Patients are not scheduled for testing appointments, but orders can be found in the EMR, with a start date and an expiration date during which the patient can present to any laboratory in the system on a drop-in basis.

Table 1
Study Medications and Recommended Tests. Persons Prescribed the Following High-Risk Medications (or Classes) During 2008 Were Included in the Analysis

The development of the monitoring guidelines is described in detail elsewhere, but consisted of a multi-step process that included review of existing guidelines, recommendations in the Physicians’ Desk Reference (PDR), clinical guidelines, and black box warnings (BBW), consensus panel review by a national committee, and final review by a local expert panel.20 While monitoring of international normalized ratio (INR) for patients on warfarin and glucose for patients on diabetes medications were included in recommendations by the panel, those medication-test pairs are not included in this study because of the different way they are monitored—often in dedicated clinics or at home—and the existing infrastructure to remind patients and providers to conduct related testing.

Key Outcome Variable

The key outcome variable was non-completion (of an ordered test). Since each medication had one or more recommended laboratory monitoring test (Table 1), the unit of analysis was the drug-test pair (e.g., colchicine has two drug-test pairs: colchicine-CBC and colchicine-creatinine). Tests ordered outside of the practice were not captured. For drug exposure, we used claims data to identify the first dispensing of one of the study medications prescribed in 2008. When a patient had more than one new start of the same drug during the study time frame, we used the first prescription for that drug only. Chronic medication use was defined as a dispensing with a previous dispensing within 6 months.

Completion of each ordered test was determined by matching the test order with test results, based on a unique order identifier. Test ordering was defined as having occurred if there was at least one recommended test for the drug-test pair ordered up to 365 days before the index dispensing in 2008, through 14 days after the dispensing if the test was indicated annually (or 180 days before to 14 days after index dispensing if the test was indicated every 6 months). For each drug-laboratory test combination, the proportion of tests completed of tests ordered by clinicians was determined for index dispensings in the observation period.

Key Predictor Variables

Patient characteristics include age, gender, number of prescriptions for study medications (one vs. > 1), health status using a comorbidity score, and visit frequency (0–5 visits, 6–10 visits, 11–18 visits, ≥ 19 visits). Comorbidity was measured using the Charlson score, based on ICD-9 codes from encounter data in the EMR and its Romano variation.21,22 The Charlson score, correlated with 1-year mortality,23 is the most widely used comorbidity index.24

Other Variables

Provider characteristics, including gender, age, and specialist versus primary care status, were included in the model. Prescription characteristics included the drug, a hierarchical indication for monitoring (defined as inclusion in a black box warning [BBW], in clinical guidelines, or in the PDR-recommended testing for narrow therapeutic window), whether the drug had single or multiple recommended monitoring tests, and recommended testing frequency. The BBW category was identified by checking whether a given test addresses a warning via online databases (labeling only relevant drug-test combinations), with the caveat that even BBW status is reported differently in different locations.25

Quantitative Data Analysis

Prescriptions, laboratory orders, and test completion were extracted from the EMR for the study period. We linked each prescription to scrambled IDs for both prescriber and patient, and then linked those entries to laboratory orders and completion data, as well as demographic and medical information for provider and patient.

To determine which factors were independently associated with laboratory test non-completion, we used unadjusted and adjusted logistic regression models with a robust covariance estimator (sandwich estimator) to adjust standard errors for clustering. This approach provides conservative nonparametric estimates.2629 We first calculated robust standard errors based on clustering of medications within patients and separately performed calculations based on clustering within providers, ultimately clustering by provider for more conservative estimates. Using this model, odds ratios (ORs) of factors associated with test non-completion were calculated.

Our modeling approach aimed to develop an explanatory model to identify factors that could be changed through intervention, rather than simply to obtain a best predictive model.30 Therefore, unadjusted models examined relationships between each predictor and confounding variables. Final models included factors hypothesized to be associated with test ordering a priori, and factors associated with test ordering at the p < 0.20 level in the unadjusted analyses. The final model (Table 2) included patient gender, age, visit frequency, medication burden, and new user status, as well as evidence level for testing, recommended testing frequency, number of recommended tests, and provider specialty.

Table 2
Factors Associated With Test Non-Completion

We also calculated the c statistic to compare the multivariable logistic regression models. Analyses were conducted in SAS 9.2 (SAS Institute, Cary, NC) and StataSE (Stata Statistical Software: Release 11.1, Stata Corporation, College Station, TX, USA).

Qualitative Methods

Using a semi-structured interview format, we interviewed patients to explore their perceptions on non-completion of laboratory tests.

Qualitative Interview Guide Development

We developed an interview guide, based on factors and domains associated with patient-non-completion of health services identified from a review of the literature.1316,19,3135 Questions addressed the following topics: whether the patient had completed or missed any laboratory tests in the past year; underlying reason for laboratory tests; views on the importance of understanding the reason for the test; provider’s explanation of the reason for the test; experience at lab; presence of reminder systems, from either the patient or the clinic; burden of and other barriers to laboratory testing; and speculation about reasons patients might miss tests. The initial interview guide was pilot tested with two patients for comprehension and flow, and was revised accordingly.

Patient Recruitment and Data Collection

We used a purposive sampling approach to select patients from those included in the quantitative study who completed or did not complete a laboratory test ordered for one of the study medications. This sampling strategy aimed to include both men and women, as well as young and older (65 years old) adults. Patients were eligible if they had a prescription for one of a subset of study medications prescribed in the outpatient setting for a chronic condition (cardiovascular [ACE inhibitors and angiotensin receptor blockers {ARBs}, statins, digoxin, diuretics, fibrates, niacin, and potassium supplements], anticonvulsant [phenytoin, valproic acid, carbamazepine and phenobarbital], or thyroid replacement medications), had an order for a related lab test, received care at the multispecialty group practice, were aged 18 or older, and spoke English. We excluded patients unable to provide informed consent due to a history of cognitive impairment or severe mood disorder.

To identify potential interviewees, a member of the research team reviewed a list of patients who missed an ordered laboratory test between July 2008 and October 2010. Missed tests (as well as completed tests) were identified by reviewing the expected completion date for the test and determining whether the date had passed without a completed test registered in the system.

After reviewing the EMR to confirm each patient’s eligibility, a research nurse sent letters inviting eligible patients to participate. If patients did not respond to the letter after 1 week, the research nurse contacted patients via telephone and invited them to an in-person interview in our research office or the option of participating in a telephone interview if there was an indication of transportation difficulties in the medical record, or if the patient suggested travel would limit participation.

We sent study invitation letters to 102 patients; 36 patients declined to participate, 35 weren’t reached, and 31 expressed interest in participating; of the latter, we interviewed 23 (16 non-adherent/‘no-show’ and seven adherent/‘show’ patients). We contacted and interviewed the non-declining patients until theme saturation was achieved, the point in the interviews where we no longer heard new ideas from participants,3638 which prior research suggests can be reached in as few as 12 interviews.36 We conducted 23 interviews (16 no-show and seven show patients). Interviews took about 45 min and patients were given a $25 stipend.

Qualitative Data Analysis

Each interview was audio-recorded and transcribed. Using a grounded theory approach,39 two researchers (SF, SG) developed codes based on four transcripts (17 % of total sample). They conducted a transcript-by-transcript iterative process of independently coding and then meeting to reconcile differences. By the fourth transcript, the research team was confident of coding consistency, and SG moved forward with coding the remaining transcripts. The codebook developed by the two researchers was used to code the remaining transcripts and helped us determine when theme saturation was reached.

Analyses were conducted using NVivo qualitative data analysis software (QSR International Pty Ltd. Version 8, 2008, Victoria, Australia). Patients provided informed consent prior to participating in the interview. This study was approved by the institutional review boards of the University of Massachusetts Medical School and the multispecialty group practice.

RESULTS

Patient Population

The final study sample included 27,802 patients prescribed one of 34 medications or medication classes, some of which had multiple recommended tests, resulting in a total of 55,592 drug-test pairs with ordered laboratory monitoring tests. Patients had a mean age of 67, were prescribed on average two study medications each (ranging from 1–9), and received care from one of 251 providers.

Quantitative Assessment of Medication-Monitoring Non-Completion

Table 3 shows patient non-completion of ordered tests for the most prescribed study medications. Overall, patient non-completion was responsible for a lower proportion of the non-completion for each drug-test pair than clinicians’ lack of ordering (for the entire data set, 85 % ordered vs. 92 % completion of ordered tests), but patient non-completion still accounted for a portion of overall non-completion. The rates varied based on drug-test pair. Patient non-completion rates, including only those drug-test pairs with more than 100 orders, varied from 2.3 % (Digoxin-creatinine and Digoxin-potassium) to 24 % (loop diuretics-BMP). In general, non-completion of different tests for the same drug (e.g. carbamazepine-AST, carbamazapine-CBC) was more similar than non-completion rates of the same test for different medications (e.g., AST tests for carbamazepine vs. for azole antifungals).

Table 3
Non-Completion Rates of Ordered Tests (> 100 Users)

Factors associated with higher odds of test non-completion included younger patient age (< 40 years vs. ≥ 80 years, adjusted odds ratio [AOR] 1.52, 95 % confidence interval [95 % CI] 1.27–1.83); lower medication burden (one medication vs. more than one drug, AOR for non-completing 1.26, 95 % CI 1.15–1.37); and lower visit frequency (0–5 visits/year vs. ≥ 19 visits/year, AOR 1.41, 95 % CI 1.25 to 1.59). Drug-test pairs with black box warning status were associated with greater odds of non-completion compared to drugs included only in the PDR [AOR 1.91, 95 % CI 1.66–2.19)] (Table 2). Patient gender was not associated with test completion, and provider factors examined besides prescribing frequency were not associated with non-completion. Patient comorbidity measured by Charlson index was not associated with non-completion in the adjusted model, though visit number was, and in the unadjusted model healthier patients were less likely to complete ordered tests. The final model had a c statistic of 0.69.

Qualitative Results

Of the 23 patients interviewed, the mean age was 63 years, 73.9 % were female, 100 % were white (Table 4), and 78 % were prescribed a cardiovascular medication. Two main domains were discussed during the interviews: the effect of patient factors and the effect of provider factors on lab attendance.

Table 4
Qualitative Interview Patient Characteristics

Patient memory played the largest role in contributing to non-completion. Of the 16 patients who did not complete an ordered test, seven patients reported they did not remember their lab test order, while four others were unaware that they did not complete a lab test order. Three other patients reported that competing demands prevented them from reaching the lab. One patient missed his test due to transportation issues, and another came to the lab, but decided to leave before his blood was drawn because the phlebotomist on duty had never drawn his blood before.

Patient knowledge and beliefs did not appear to affect non-completion. Most patients (17; 12 no-show, five show) were able to explain the reason for their lab test. Patients most commonly spoke about tests as necessary for preventing side effects (eight;, six no-show, three show), adjusting medication dose (ten; nine no-show, one show), and checking organ function (14; eight no-show, five show). The majority (18; 15 no-show, three show) expressed understanding of the connection between the test and their medication. Nearly all (20; 14 no-show, six show) said that it was important to understand the purpose of the test. None reported they missed a lab test due to not understanding the reason for the test. Discussions explored other patient-related issues, including medication adherence, presence of personal reminder system, and relationship with provider, but did not establish any other connections with test completion.

Interviews did not identify a relationship between provider-related processes and non-completion. Most patients (18; 12 no-show, six show) received an explanation from their provider about the reason for the lab test and expressed satisfaction with that explanation (16; ten no-show, six show). No patient attributed a missed lab test to not receiving an explanation from his/her provider. In addition, the majority of participants (17; 11 no-show, six show) noted that their provider follows up with results after a lab test. Three participants (two no-show, one show) were discontented with follow-up, but it did not influence them to miss a test. Nine no-show patients stated that they did not have a problem with the frequency of lab test orders; no patient complained of having too many tests.

Patients highlighted certain facility operations as facilitating completion. Some (nine; six no-show, 3three show) noted the convenience of the facility hours or ability to perform a lab test order at a non-fixed time. Others (seven; five no-show, two show) appreciated the option to choose from different locations. When asked if the multispecialty group practice should send a particular type of reminder, some participants (nine; seven no-show, two show) expressed preference for a telephone call, while others did not have a preference (six; four no-show, two show) or thought a reminder would be unnecessary (four; two no-show, two show).

DISCUSSION

This mixed-methods study adds to the literature on laboratory monitoring of medications, by quantifying the contribution of patient test completion of overall monitoring and providing patient perspectives about test completion. While prior literature either evaluates only physician test ordering5,6 or overall test completion,8,9 rarely are both reported.20,40 Most studies have reported only completion rates, usually determined from claims data,41 while we have separated outcomes into test ordering and patient completion of ordered tests. Our study further builds on prior work by using qualitative techniques to study patient non-completion.

The highest prevalence of patient non-completion of an ordered test for a prescribed medication with over 100 users was 23.6 %, a metabolic panel (or creatinine and potassium) test for patients taking a loop diuretic. Overall, we found levels of non-completion of ordered tests lower than those found in studies focusing on completion of monitoring at medication initiation.3

Quantitative analysis found non-completion of laboratory monitoring associated with patient age and patient visits, with younger age and fewer visits associated with less completion. Being on fewer medications or having fewer tests recommended for a given medication also increased the odds of non-completion. Older, sicker patients may have more contact with the health care system, and consequently have more opportunities for testing. Interestingly, however, black box warning status, a proxy for the seriousness of the potential adverse event caused by a drug, was associated with decreased completion, although associated with increased ordering. It is concerning that patients are less likely to complete testing for these medications, particularly since we have shown that providers order such tests at higher rates. Interventions should target tests ordered for patients taking these medications via the patient, the prescriber, or the system.

We also found that new users of a medication were much less likely to complete ordered tests, perhaps due to lack of familiarity with the test process or the reasons behind the test. Test frequency was not associated with non-completion, perhaps because this factor reflects the recommended frequency of testing, not the actual ordered frequency, which is more likely to affect patient behavior. However, we only included the first incidence of each type of test for each prescription.

Our qualitative study found that patients were generally satisfied with the current clinical processes for lab tests. Patients largely attributed non-completion to forgetting or a lack of awareness to the order instead of to the practice-factors, such as logistical access or laboratory facility hours, which previous studies have reported as important for patients missing clinician appointments.15,16 Most patients did not advocate for a particular change within the clinical operations. Patient views and insufficient knowledge did not appear to lead to non-completion; most patients were able to explain, and believed it was important to understand, the reason for the lab test.

Although patients did not fault the current system, the system may contribute to forgetting. At the study practice, as is not unusual in ambulatory settings, providers enter an expected completion date for lab tests rather than a specific date; as a result, patients can perform the test at any time through this completion date, rather than having a specific pre-scheduled appointment for the laboratory test. This flexibility generally works, with relatively high overall completion rates, but without a concrete appointment, human factors make it easy to forget to complete an open order. Responses from patients in our study suggest that telephone reminders may be a well-received method of addressing this issue.

This study did not reveal a single overriding explanation for incomplete laboratory monitoring. Interventions will have to target both test ordering and completion, and investigators should report the two rates separately.42 Strengths of the current study are the identification of factors associated with test non-completion using both quantitative and qualitative methods. This work was facilitated by the integrated electronic health and claims system used for our study. As EMRs become more prevalent, health care systems and investigators may benefit from further disentangling provider and patient contributions to similar quality-of-care metrics. This will allow better targeting of quality improvement initiatives for both patients and providers.

This work is particularly relevant today, as more and more providers adopt electronic records that allow them to track the testing process in more detail, and as reporting requirements increase the level of monitoring and quality management. EMRs are expected to increase quality of care in general and medication management in particular,43,44 but it’s not clear that they are yet fulfilling this goal.45 A recent study on Healthcare Effectiveness Data and Information Set (HEDIS) guidelines for monitoring gives an example of how the shift of the source of data from claims data to EMR data can significantly alter results: when using claims data as the source, the reported item is effectively test completion, while using EMR data gives test ordering, which will necessarily be as high or higher. Thus, using claims data for quality reporting compared to EMRs will underestimate quality of care.46 As quality ratings increasingly become a factor in reimbursement and as Accountable Care Organizations (ACOs) expand as the Affordable Care Act is further implemented, the distinction between these sources of data and their implications will be even more central.

Limitations of our study should be noted. First, laboratory tests may have been ordered for another reason (i.e., not for monitoring), so that we may have overestimated the prevalence of recommended testing. We may have missed monitoring that was done outside of our system, but prior research suggests that this is unlikely.47 Third, the interviewed patients may not be representative of the whole population (all white and educated); the same barriers that prevent completing testing may also impede appearing for an interview. We attempted to overcome that issue with telephone interviews to reach more patients, but we cannot rule out that we missed patients who had other major barriers to both testing and interviewing. Because we selected interview participants from a list of patients who missed a test in the past year, the elapsed time between the missed test and scheduled interview could affect accuracy of recall. Lastly, our subjects were all continuously enrolled members of a single health plan, and we only studied the ambulatory setting, limiting the generalizability of our findings.

Our study demonstrates that patient non-completion contributes to under-monitoring of prescribed medications. Younger, healthier patients may be more likely to not complete ordered laboratory tests, along with patients with medications that have black box warnings, which are presumed to pose higher risk. Reminding providers to order tests will not be sufficient, as patients do not reliably complete ordered tests. Our results further suggest that provider factors were not major reasons for test non-completion. Similarly, patient understanding did not seem to be a large factor in the decision to complete a test. Patients reported that they simply did not remember to get tests. Patients, particularly those at high risk of not completing tests, should be targeted directly for reminders or other interventions, ideally through a modality of their choice, in order to maximize completion.

Acknowledgements

Contributors

The authors would like to acknowledge the contributions of Yanfang Zhao, MA.

Funders

This study was funding by grant R18 HS 017906 from the Agency for Healthcare Research and Quality (AHRQ).

Prior Presentations

We presented the manuscript abstract at the Clinical and Translational Science Research Retreat at the University of Massachusetts Medical School on May 22, 2012 and the qualitative results only on May 20, 2011. We presented the results from the qualitative interviews at the HMO Research Network Conference in Boston, MA, on March 24, 2011. We presented a portion of the quantitative study findings as a poster at the AHRQ Health Information Technology conference on September 27, 2010.

Conflict of Interest

The authors declare that they do not have a conflict of interest.

REFERENCES

1. Gandhi TK, Weingart SN, Borus J, et al. Adverse drug events in ambulatory care. N Engl J Med. 2003;348(16):1556–64. doi: 10.1056/NEJMsa020703. [PubMed] [Cross Ref]
2. Gurwitz JH, Field TS, Harrold LR, et al. Incidence and preventability of adverse drug events among older persons in the ambulatory setting. JAMA. 2003;289(9):1107–16. doi: 10.1001/jama.289.9.1107. [PubMed] [Cross Ref]
3. Raebel MA, Lyons EE, Andrade SE, et al. Laboratory monitoring of drugs at initiation of therapy in ambulatory care. J Gen Intern Med. 2005;20(12):1120–6. doi: 10.1111/j.1525-1497.2005.0257.x. [PMC free article] [PubMed] [Cross Ref]
4. Simon SR, Andrade SE, Ellis JL, et al. Baseline laboratory monitoring of cardiovascular medications in elderly health maintenance organization enrollees. J Am Geriatr Soc. 2005;53(12):2165–9. doi: 10.1111/j.1532-5415.2005.00498.x. [PubMed] [Cross Ref]
5. Lo HG, Matheny ME, Seger DL, Bates DW, Gandhi TK. Impact of non-interruptive medication laboratory monitoring alerts in ambulatory care. J Am Med Inform Assoc. 2009;16(1):66–71. doi: 10.1197/jamia.M2687. [PMC free article] [PubMed] [Cross Ref]
6. Steele AW, Eisert S, Witter J, et al. The effect of automated alerts on provider ordering behavior in an outpatient setting. PLoS Med. 2005;2(9):e255. doi: 10.1371/journal.pmed.0020255. [PMC free article] [PubMed] [Cross Ref]
7. Feldstein AC, Smith DH, Perrin N, et al. Improved therapeutic monitoring with several interventions: a randomized trial. Arch Intern Med. 2006;166(17):1848–54. doi: 10.1001/archinte.166.17.1848. [PubMed] [Cross Ref]
8. Hoch I, Heymann AD, Kurman I, Valinsky LJ, Chodick G, Shalev V. Countrywide computer alerts to community physicians improve potassium testing in patients receiving diuretics. J Am Med Inform Assoc. 2003;10(6):541–6. doi: 10.1197/jamia.M1353. [PMC free article] [PubMed] [Cross Ref]
9. Matheny ME, Sequist TD, Seger AC, et al. A randomized trial of electronic clinical reminders to improve medication laboratory monitoring. J Am Med Inform Assoc. 2008;15(4):424–9. doi: 10.1197/jamia.M2602. [PMC free article] [PubMed] [Cross Ref]
10. Palen TE, Raebel M, Lyons E, Magid DM. Evaluation of laboratory monitoring alerts within a computerized physician order entry system for medication orders. Am J Manag Care. 2006;12(7):389–95. [PubMed]
11. Raebel MA, Carroll NM, Andrade SE, et al. Monitoring of drugs with a narrow therapeutic range in ambulatory care. Am J Manag Care. 2006;12(5):268–74. [PubMed]
12. Tang EO, Lai CS, Lee KK, Wong RS, Cheng G, Chan TY. Relationship between patients’ warfarin knowledge and anticoagulation control. Ann Pharmacother. 2003;37(1):34–9. doi: 10.1345/aph.1A198. [PubMed] [Cross Ref]
13. Canizares MJ, Penneys NS. The incidence of nonattendance at an urgent care dermatology clinic. J Am Acad Dermatol. 2002;46(3):457–9. doi: 10.1067/mjd.2002.118340. [PubMed] [Cross Ref]
14. Pal B, Taberner DA, Readman LP, Jones P. Why do outpatients fail to keep their clinic appointments? Results from a survey and recommended remedial actions. Int J Clin Pract. 1998;52(6):436–7. [PubMed]
15. Martin C, Perfect T, Mantle G. Non-attendance in primary care: the views of patients and practices on its causes, impact and solutions. Fam Pract. 2005;22(6):638–43. doi: 10.1093/fampra/cmi076. [PubMed] [Cross Ref]
16. Lacy NL, Paulman A, Reuter MD, Lovejoy B. Why we don’t come: patient perceptions on no-shows. Ann Fam Med. 2004;2(6):541–5. doi: 10.1370/afm.123. [PubMed] [Cross Ref]
17. Lash S, Harding J. “Abandoned prescriptions” a quantitative assessment of their cause. J Manag Care Pharm. 1995;1:193–9.
18. Briesacher BA, Andrade SE, Fouayzi H, Chan KA. Comparison of drug adherence rates among patients with seven different medical conditions. Pharmacotherapy. 2008;28(4):437–43. doi: 10.1592/phco.28.4.437. [PMC free article] [PubMed] [Cross Ref]
19. George A, Rubin G. Non-attendance in general practice: a systematic review and its implications for access to primary health care. Fam Pract. 2003;20(2):178–84. doi: 10.1093/fampra/20.2.178. [PubMed] [Cross Ref]
20. Tjia J, Field TS, Garber LD, et al. Development and pilot testing of guidelines to monitor high-risk medications in the ambulatory setting. Am J Manag Care. 2010;16(7):489–96. [PubMed]
21. Romano PS, Roos LL, Jollis JG. Adapting a clinical comorbidity index for use with ICD-9-CM administrative data: differing perspectives. J Clin Epidemiol. 1993;46(10):1075–9. doi: 10.1016/0895-4356(93)90103-8. [PubMed] [Cross Ref]
22. Deyo RA, Cherkin DC, Ciol MA. Adapting a clinical comorbidity index for use with ICD-9-CM administrative databases. J Clin Epidemiol. 1992;45(6):613–9. doi: 10.1016/0895-4356(92)90133-8. [PubMed] [Cross Ref]
23. Charlson ME, Pompei P, Ales KL, MacKenzie CR. A new method of classifying prognostic comorbidity in longitudinal studies: development and validation. J Chronic Dis. 1987;40(5):373–83. doi: 10.1016/0021-9681(87)90171-8. [PubMed] [Cross Ref]
24. Chu YT, Ng YY, Wu SC. Comparison of different comorbidity measures for use with administrative data in predicting short- and long-term mortality. BMC Health Serv Res. 2010;10:140. doi: 10.1186/1472-6963-10-140. [PMC free article] [PubMed] [Cross Ref]
25. Wang LM, Wong M, Lightwood JM, Cheng CM. Black box warning contraindicated comedications: concordance among three major drug interaction screening programs. Ann Pharmacother. 2010;44(1):28–34. doi: 10.1345/aph.1M475. [PubMed] [Cross Ref]
26. Rogers WH. Regression standard errors in clustered samples. Stata Tech Bull. 1993;13:19–23.
27. Williams RL. A note on robust variance estimation for cluster-correlated data. Biometrics. 2000;56(2):645–6. doi: 10.1111/j.0006-341X.2000.00645.x. [PubMed] [Cross Ref]
28. Wooldridge JM. Econometric Analysis of Cross Section and Panel Data. Cambridge: MIT Press; 2002.
29. Froot KA. Consistent covariance matrix estimation with cross-sectional dependence and heteroskedasticity in financial data. J Financ Quant Anal. 1989;24:333–55. doi: 10.2307/2330815. [Cross Ref]
30. Harrell FE.Regression Modeling Strategies: with Applications to Linear Models, Logistic Regression, and Survival Analysis. Springer Verlag; 2001.
31. Oppenheim GL, Bergman JJ, English EC. Failed appointments: a review. J Fam Pract. 1979;8(4):789–96. [PubMed]
32. Rodriguez Pacheco R, Negro Alvarez JM, Campuzano Lopez FJ, et al. Non-compliance with appointments amongst patients attending an Allergology Clinic, after implementation of an improvement plan. Allergol Immunopathol (Madr) 2007;35(4):136–44. doi: 10.1157/13108224. [PubMed] [Cross Ref]
33. Zailinawati AH, Ng CJ, Nik-Sherina H. Why do patients with chronic illnesses fail to keep their appointments? A telephone interview. Asia Pac J Public Health. 2006;18(1):10–5. doi: 10.1177/10105395060180010301. [PubMed] [Cross Ref]
34. Simmons AV, Atkinson K, Atkinson P, Crosse B. Failure of patients to attend a medical outpatient clinic. J R Coll Physicians Lond. 1997;31(1):70–3. [PubMed]
35. Collins J, Santamaria N, Clayton L. Why outpatients fail to attend their scheduled appointments: a prospective comparison of differences between attenders and non-attenders. Aust Health Rev. 2003;26(1):52–63. doi: 10.1071/AH030052. [PubMed] [Cross Ref]
36. Guest G, Bunce A, Johnson L. How many interviews are enough?: An experiment with data saturation and variability. Field Methods. 2006;18(1):59–82. doi: 10.1177/1525822X05279903. [Cross Ref]
37. Higginbottom GM. Sampling issues in qualitative research. Nurse Res. 2004;12(1):7–19. [PubMed]
38. Marshall MN. Sampling for qualitative research. Fam Pract. 1996;13(6):522–5. doi: 10.1093/fampra/13.6.522. [PubMed] [Cross Ref]
39. Glaser B, Strauss A. Grounded Theory: the Discovery of Grounded Theory. New York: de Gruyter; 1967.
40. Raebel MA, Lyons EE, Chester EA, et al. Improving laboratory monitoring at initiation of drug therapy in ambulatory care: a randomized trial. Arch Intern Med. 2005;165(20):2395–401. doi: 10.1001/archinte.165.20.2395. [PubMed] [Cross Ref]
41. Tang PC, Ralston M, Arrigotti MF, Qureshi L, Graham J. Comparison of methodologies for calculating quality measures based on administrative data versus clinical data from an electronic health record system: implications for performance measures. J Am Med Inform Assoc. 2007;14(1):10–5. doi: 10.1197/jamia.M2198. [PMC free article] [PubMed] [Cross Ref]
42. Fischer SH, Tjia J, Field TS. Impact of health information technology interventions to improve medication laboratory monitoring for ambulatory patients: a systematic review. J Am Med Inform Assoc. 2010;17(6):631–6. doi: 10.1136/jamia.2009.000794. [PMC free article] [PubMed] [Cross Ref]
43. Chaudhry B, Wang J, Wu S, et al. Systematic review: impact of health information technology on quality, efficiency, and costs of medical care. Ann Intern Med. 2006;144(10):742–52. [PubMed]
44. McMahan R. Operationalizing MTM, through the use of health information technology. J Manag Care Pharm. 2008;14(2 Suppl):S18–21. [PubMed]
45. Black AD, Car J, Pagliari C, et al. The impact of eHealth on the quality and safety of health care: a systematic overview. PLoS Med. 2011;8(1):e1000387. doi: 10.1371/journal.pmed.1000387. [PMC free article] [PubMed] [Cross Ref]
46. Tjia J, Field TS, Fischer SH, et al. Quality measurement of medication monitoring in the “meaningful use” era. Am J Manag Care. 2011;17(9):633–7. [PubMed]
47. Raebel MA, Carroll NM, Simon SR, et al. Liver and thyroid monitoring in ambulatory patients prescribed amiodarone in 10 HMOs. J Manag Care Pharm. 2006;12(8):656–64. [PubMed]

Articles from Journal of General Internal Medicine are provided here courtesy of Society of General Internal Medicine