|Home | About | Journals | Submit | Contact Us | Français|
Sudden cardiac death (SD) and ventricular arrhythmias (VAs) caused by medications have arisen as an important public health concern in recent years. The validity of diagnostic codes in identifying SD/VA events originating in the ambulatory setting is not well known. This study examined the positive predictive value (PPV) of hospitalization and emergency department encounter diagnoses in identifying SD/VA events originating in the outpatient setting.
We selected random samples of hospitalizations and emergency department claims with principal or first-listed discharge diagnosis codes indicative of SD/VA in individuals contributing at least 6 months of baseline time within 1999–2002 Medicaid and Medicare data from five large states. We then obtained and reviewed medical records corresponding to these events to serve as the reference standard.
We identified 5239 inpatient and 29 135 emergency department events, randomly selected 100 of each, and obtained 119 medical records, 116 of which were for the requested courses of care. The PPVs for an outpatient-originating SD/VA precipitating hospitalization or emergency department treatment were 85.3% (95% confidence interval [CI]=77.6–91.2) overall, 79.7% (95%CI=68.3–88.4) for hospitalization claims, and 93.6% (95%CI=82.5–98.7) for emergency department claims.
First-listed SD/VA diagnostic codes identified in inpatient or emergency department encounters had very good agreement with clinical diagnoses and functioned well to identify outpatient-originating events. Researchers using such codes can be confident of the PPV when conducting studies of SD/VA originating in the outpatient setting.
Sudden cardiac death (SD) is defined as the sudden, abrupt loss of heart function.1 It is responsible for about 12–19% of all deaths in the US.2,3 Because ventricular fibrillation is present in three-quarters of outpatient cases of SD,4 some authors prefer the term ‘presumed arrhythmic death’ to SD.5
SD and ventricular arrhythmias (VAs) caused by medications have arisen as an important public health concern in recent years. The University of Arizona Center for Education and Research on Therapeutics (http://azcert.org) has identified over 130 drugs that prolong the electrocardiographic QT interval (a potential marker of drug arrhythmogenicity) and/or cause torsade de pointes (a commonly drug-induced arrhythmia that can progress to ventricular fibrillation). Regulatory action has been taken on a number of potentially arrhythmogenic drugs, including astemizole, cisapride, terfenadine, terolidine, and thioridazine.
Because the baseline incidence of SD and VA combined is only about 0.5 to two events per thousand person-years,6–10 very large studies are needed to characterize risks associated with specific drugs. Medicaid data have been used to study SD and VA associated with the use of prokinetics,11 antipsychotics,10,12,13 antidepressants,14 antibiotics,15 urinary antispasmodics,16 and non-sedating antihistamines.8,17 However, the validity of diagnostic codes for SD and VA has not been well studied. Previous studies have shown that death certificate causes of death have a positive predictive value (PPV) for SD of only 19–32%.18,19 In a study not relying on death certificates, Staffa et al.17 obtained medical records for 11 patients receiving antihistamines, each of whom had a Medicaid hospital discharge encounter record indicating either SD or VA as a principal or non-principal discharge diagnosis. Based on chart review, the PPV for this composite outcome was 73% (eight of 11). This study, however, did not distinguish between events that led to admission and those that occurred during hospitalization. This distinction is important when evaluating if outpatient-dispensed medications increase the risk for SD/VA events originating in the ambulatory setting.
Recently, using a United Kingdom electronic medical record database, our group found that diagnostic codes within that database functioned poorly to identify instances of SD/VA in the outpatient setting.20 Using Medicaid and Medicare data, our group also obtained 128 inpatient medical records of patients receiving a prokinetic drug or a proton pump inhibitor who were admitted with either a principal or non-principal hospital discharge diagnosis of a composite outcome of SD/VA.11 While the validation criterion was met in 92% of subjects, only 19% of the validated events originated in the outpatient setting, with the remainder originating during the hospital course. In a post hoc analysis of records for which the diagnosis of interest was the principal diagnosis (i.e., ostensibly the diagnosis chiefly responsible for the admission), seven of seven subjects met the validation criterion, and all of these events originated prior to hospitalization.11 Thus, the PPV for a principal inpatient diagnostic code of SD/VA was 100%, although the lower bound of the exact binomial 95%CI was 59%.
Therefore, we sought to examine the PPV of a principal hospitalization diagnosis of SD or VA as a marker for an event originating in the outpatient setting in a larger, prospectively identified sample of Medicaid enrollees. Further, because many cases of SD or VA treated in the emergency department (ED) may not result in admission (e.g., if the patient dies before admission), we sought to examine the PPV for first-listed ED diagnoses for such encounters that did not lead to hospital admission.
We evaluated in Medicaid beneficiaries the PPV of hospitalization and ED encounter records with principal or first-listed diagnosis codes indicative of the composite outcome of SD/VA, using primary medical records as the reference standard. We used 1999–2002 data from the Medicaid programs of California, Florida, New York, Ohio, and Pennsylvania, which we obtained from the Centers for Medicare and Medicaid Services (CMS) in 2006. Because approximately 14% of Medicaid beneficiaries are co-enrolled in Medicare,21 we also included Medicare data obtained from CMS on dually eligible persons in these states.
This study was approved by the University of Pennsylvania's Committee on Subjects Involving Human Beings, which waived the requirement for informed consent and subject authorization to use and disclose protected health information under the Privacy Rule of the Health Insurance Portability and Accountability Act of 1996 (HIPAA). Except for suggestions made by reviewers during peer review of the grant proposal, the funding agency (the National Institutes of Health) had no role in the study's design, conduct, or interpretation.
The study population consisted of enrollees of the Medicaid programs of our study states with no prior encounter of any type with an any-position (principal or non-principal, first-listed or later-listed) diagnosis of SD/VA (Table 1) during database-contributed time, and no putative event (defined below) in the first 6 months of observed eligibility. The rationale for excluding persons with a prior SD/VA diagnosis was to identify incident events. The rationale for excluding persons with less than 6 months of baseline observation was to mirror the common practice in epidemiologic studies of requiring a baseline period of some length in order to restrict the study to apparently new users of the drug of interest.22
Putative events resulting in hospitalization were identified by the presence of an International Classification of Diseases, 9th Revision, (ICD-9) diagnostic code of interest (Table 1) listed as the principal discharge diagnosis within either the Medicaid Analytic eXtract (MAX) inpatient file, the Medicare Provider Analysis and Review file, or both. Putative events resulting in only ED treatment were identified based on a first-listed diagnosis of SD/VA from encounters within the MAX other therapy file, the Medicare Carrier standard analytic file (SAF), the Medicare outpatient SAF, or more than one of these files, using the algorithm described in the Appendix. We included only the first putative event per subject.
From among all the putative outcomes, we selected a random sample of 100 admission and 100 ED events for which to request medical records. Simple random sampling without replacement was conducted by utilizing a computer program to assign a random number to each outcome, ordering the outcomes numerically, and selecting the top 100 of each event type. The rationale for validating 100 outcomes for each event type was based on a lower bound of 95% for the CI of each overall PPV. Given the retrieval rate from our previous validation experience,23 we anticipated receiving 78 records for every 100 requested. Therefore, if the PPV was 100%, the lower bound of the one-sided 97.5%CI would be 95.4% based on 78 records.
In collaboration with a subcontractor (Information Collection Enterprises, LLC [ICE]), we contacted hospitals via mail to request medical records to be used for research purposes. Hospitals then sent photocopies of the medical records to ICE, which hand-redacted direct personal identifiers, scanned the redacted records, and provided them to us on electronic storage media. ICE re-contacted non-responding hospitals via mail and/or telephone to re-request records and ascertain reasons for non-shipment of records.
Medical records were reviewed independently by a doctorally trained clinical pharmacist (C.E.L.) and by a specially trained public health researcher (C.P.F.), both of whom had prior experience examining the validity of this outcome.11,20 Reviewers used a structured electronic abstraction form to record: (a) clinical documentation of SD, cardiac arrest, or cardio-respiratory arrest; (b) clinical documentation of VA; (c) documentation of the event on electrocardiogram (ECG) reports or mention of ECG evidence; (d) evidence of an outpatient witnessed sudden collapse or person found dead in the field; (e) initial presentation with signs or symptoms suggestive of VA (e.g., near-syncope, palpitations, dizziness, sweating); (f) inpatient telemetry readings and electrophysiology study (EPS) reports; (g) the location of the event's onset (i.e., inpatient vs. ED vs. prior to ED); and (h) documentation of non-physiologic, extrinsic causes precipitating the event (e.g., motor vehicle accident, blunt trauma).
The validation criterion, adapted from one used by Ray et al.,13 was a clinician-diagnosed SD, cardiac arrest, cardiorespiratory arrest, or VA as evidenced by a verbatim statement within the medical record, excluding the coder's face sheet. In addition, documentation of an outpatient witnessed sudden collapse, or a person found dead or unconscious in the field with evidence that the individual had been alive in the prior 24 hours, met the validation criterion. Events identified by clinicians after admission did not meet the criterion unless the person presented to the hospital with signs or symptoms suggestive of VA (e.g., near-syncope, palpitations, dizziness, sweating), plus either (1) ventricular fibrillation or sustained ventricular tachycardia was evidenced on telemetry or (2) an EPS was reported as either positive or demonstrating a sustained or sustainable ventricular tachycardia or fibrillation. Events precipitated by a non-physiologic, extrinsic cause (e.g., blunt trauma) did not meet the outcome criterion regardless of elements met above.
Initial inter-rater agreement was assessed by the per cent initial agreement and Cohen's κ regarding whether the record met the validation criterion. When reviewers disagreed initially, they then attempted to resolve disagreements by consensus. When consensus could not be reached, the record was referred to a third reviewer (S.H.), a doctorally trained clinical pharmacist, to break the tie.
PPVs were calculated as the number of events meeting the validation criterion divided by the number of eligible records. Exact binomial 95%CIs, one-sided 97.5%CIs (when dictated by the proportion), and κ statistics were calculated using Stata 9.2 (StataCorp LP, College Station, Texas, 2007).
Our study database consisted of nearly 26 million Medicaid enrollees contributing approximately 57 million person-years of observation. We identified 34 374 putative events, of which 5239 constituted hospitalizations and 29 135 ED encounters only. Of the 200 putative events randomly selected (100 hospitalization and 100 ED), 199 were linkable to data allowing the identification of the hospital of interest. We randomly selected one additional event to replace the non-linked event. The distribution of principal (for hospitalization) or first-listed (for ED only) ICD-9 codes for sampled events, stratified by hospitalization versus ED, is listed in Table 1.
We contacted hospitals in ten states and received 119 of 200 (60%) records requested: 71 of 100 (71%) inpatient records and 48 of 100 (48%) ED records. The reasons for non-retrieval were: no response from the hospital (N = 32), hospital's inability to identify the correct patient based on the information provided (N = 25), hospital required patient authorization (N = 12), records purged or removed from site (N = 9), and no reason provided by the hospital (N = 3). Three of the records that we received were for an encounter other than the one requested.
Hospitalization events had a principal diagnosis of paroxysmal ventricular tachycardia (N = 39, 56%), cardiac arrest (N = 22, 32%), or ventricular fibrillation (N = 8, 12%). In addition, of the 39 hospitalization events with a principal diagnosis of paroxysmal ventricular tachycardia, two had a secondary diagnosis of ventricular fibrillation and one had a secondary diagnosis of cardiac arrest. Of the 22 hospitalization events with a principal diagnosis of cardiac arrest, two had a secondary diagnosis of paroxysmal ventricular tachycardia, and one had a secondary diagnosis of ventricular fibrillation. Of the eight hospitalization events with a principal diagnosis of ventricular fibrillation, four had a secondary diagnosis of cardiac arrest, and one had a secondary diagnosis of paroxysmal ventricular tachycardia.
The first-listed diagnoses of the 47 ED events were cardiac arrest (N = 45, 96%), paroxysmal ventricular tachycardia (N = 1, 2%), and death occurring in less than 24 hours from onset of symptoms (N = 1, 2%). No ED events reviewed had both a first- and later-listed diagnosis of interest.
The two independent reviewers reached identical conclusions on whether the event met the outcome criterion in 112 of 116 records, for an initial inter-rater agreement of 97% (κ = 0.85). Three of the four initial disagreements were resolved by consensus between the two abstractors, while the fourth was settled by referral to the third reviewer.
Table 2 presents the validation results, both overall and separately for hospitalization and ED encounters. Table 2 also presents results stratified by components of the validation criterion and whether the principal or first-listed diagnosis was SD or VA. Furthermore, it presents whether there was a secondary diagnosis of interest. The rationale for presenting PPVs by subset of encounter diagnoses is so that readers can examine the performance of subsets of claims codes. The rationale for presenting results stratified by components of the validation criterion is so that readers can examine the performance of alternative validation criteria.
Of the 116 putative hospitalization and ED claims combined, 99 met the composite outcome criterion, for an overall PPV of 85.3% (95%CI = 77.6–91.2) (Table 2). When the principal or first-listed diagnosis was VA and there was no secondary diagnosis of interest, the PPV was 74.4% (95%CI = 58.8–86.5). When the principal or first-listed diagnosis was SD and there was no secondary diagnosis of interest, the PPV was 92.3% (95%CI = 83.0–97.5).
The PPV for subsets of diagnostic codes for hospitalization and ED claims combined based on the composite validation criterion ranged from 48.3 to 66.4% (Table 2). The PPV for subsets of codes varied by component of the validation criterion. For example, SD claims codes had a high PPV for verbatim statement of SD or cardio-respiratory arrest (96.9%), but a low PPV for a verbatim statement of VA (32.3%). Similarly, VA claims codes had a high PPV for verbatim statement of VA (90.7%), but a low PPV for verbatim statement of SD or cardiorespiratory arrest (16.3%).
The overall PPV for hospitalization claims was 79.7% (95%CI = 68.3–88.4). The overall PPV for ED claims was 93.6% (95%CI = 82.5–98.7). The pattern of PPVs for subsets of claims codes by specific component of the validation criterion was similar for hospital and ED claims.
It is widely recognized that researchers using administrative data to study clinical outcomes should examine the validity of claims diagnoses by comparing them to clinical records.24 Despite the challenge of obtaining medical records, exacerbated by the implementation of the HIPAA Privacy Rule,25,26 we were able to obtain 71% of inpatient records and 48% of ED records requested. It is not surprising that we were less successful obtaining ED records, since hospitals may be less accustomed to responding to requests for ED records than inpatient records. Nevertheless, despite the existence of waivers of the requirements for informed consent and HIPAA authorization, 12 hospitals specifically cited lack of patient authorization in refusing our record request. Further, we speculate that some of the 30 non-responses and three refusals without reason may also have been due in part to lack of patient authorization. Thus, while HIPAA is not making it impossible for researchers to obtain clinical records, it may be reducing the yield.
Overall, principal inpatient and first-listed ED diagnoses for SD/VA identified in CMS encounters had very good agreement with clinical diagnoses. In particular, the PPV for a principal hospitalization or first-listed ED diagnosis of SD/VA was 85.3%. The PPV was 79.7% for hospitalization claims and 93.6% for ED claims. VA diagnoses predominated in hospitalization claims, while SD diagnoses predominated within ED claims. Not surprisingly, VA diagnoses had a high PPV versus VA components of the validation criterion, and SD diagnoses had a high PPV versus SD components of the validation criterion.
Our rationale for studying only encounters for which the diagnosis of interest appeared in the principal position (for hospitalization) and first-listed position (for ED) was to attempt to identify events that originated in the outpatient setting. This is important because, in studies in which the goal is to study cardiac effects of outpatient prescription drugs, one would expect these effects to originate in the outpatient setting. Furthermore, our previous research of the validity of any-position, inpatient SD/VA diagnoses in identifying outpatient-originating events in CMS claims found a PPV of 19%.11
Outcome validity can be described in two levels.24 The first is whether the encounter diagnosis accurately reflects the clinical diagnosis. The second is whether the clinical diagnosis reflects the patient's truey condition. Validation studies such as this typically focus on the first level, as it is often difficult to retrospectively reconstruct the clinical scenario in enough detail to examine the accuracy of the clinical diagnosis.24 Because SD is a non-specific clinical diagnosis, our validation criterion for SD may well have included events that were non-cardiac in origin, and thus not inducible by drugs. On the other hand, 83.7% of events with a principal or first-listed VA diagnosis had an ECG reading that indicated VA.
It is important to delineate this study's limitations. First, we attempted to validate only a small proportion of events (0.58%) identified in our dataset. Yet, the process by which these events were selected was random and we were sufficiently powered to obtain relatively narrow CIs. Nonetheless, it is possible that sampling error may have occurred, and therefore our sampled events may not be representative of the entirety of identified events. Second, we made no attempt to assess the completeness (i.e., sensitivity) of our approach in identifying SD/VA, and therefore do not know what proportion of serious outpatient-originating events were missed, for example, because the patient died without ever being taken to a hospital. Although some researchers have used death certificate causes of death to identify such events, death certificate diagnoses have a PPV of only 19–32%.18,19
In conclusion, principal inpatient and first-listed ED diagnostic codes have a very good PPV in identifying the occurrence of the SD/VA that originates in the outpatient setting.
This study was supported by the National Heart, Lung, and Blood Institute (R01HL076697) of the National Institutes of Health. The authors thank the staff of Information Collection Enterprises LLC for their role in this project.
For Medicaid, a claim was flagged as emergency department only if (criterion 1, 2, or 3) AND criterion 4 were met.
For Medicare, a claim was flagged as emergency department only if (criterion 1, 2, 3, or 4) AND criterion 5 were met.
† The authors declare no conflict of interest.