|Home | About | Journals | Submit | Contact Us | Français|
To assess the accuracy of International Classification of Diseases, 9th Revision, Clinical Modification (ICD-9-CM), codes in identifying cases of child physical abuse in 4 children’s hospitals.
We included all children evaluated by a child abuse pediatrician (CAP) for suspicion of abuse at 4 children’s hospitals from January 1, 2007, to December 31, 2010. Subjects included both patients judged to have injuries from abuse and those judged to have injuries from accidents or to have medical problems. The ICD-9-CM codes entered in the hospital discharge database for each child were compared to the decisions made by the CAPs on the likelihood of abuse. Sensitivity and specificity were calculated. Medical records for discordant cases were abstracted and reviewed to assess factors contributing to coding discrepancies.
Of 936 cases of suspected physical abuse, 65.8% occurred in children <1 year of age. CAPs rated 32.7% as abuse, 18.2% as unknown cause, and 49.1% as accident/medical cause. Sensitivity and specificity of ICD-9-CM codes for abuse were 73.5% (95% confidence interval 68.2, 78.4), and 92.4% (95% confidence interval 90.0, 94.0), respectively. Among hospitals, sensitivity ranged from 53.8% to 83.8% and specificity from 85.4% to 100%. Analysis of discordant cases revealed variations in coding practices and physicians’ notations among hospitals that contributed to differences in sensitivity and specificity of ICD-9-CM codes in child physical abuse.
Overall, the sensitivity and specificity of ICD-9-CM codes in identifying cases of child physical abuse were relatively low, suggesting both an under- and overcounting of abuse cases.
Researchers and clinicians have recently been using hospital discharge databases (HDDs) to identify and track injuries due to child physical abuse. Data from traditional sources, such as Child Protective Services agencies, lack detailed information on the types and severities of abusive injuries.1 In contrast HDDs document the types of injuries diagnosed by medical professionals during hospitalizations for abuse. In the United States, these databases use International Classification of Diseases, 9th Revision, Clinical Modification (ICD-9-CM) codes to document the diagnoses and external cause of injury (E-codes). Studies have used HDDs, such as the Kids’ Inpatient Database (KID),2–4 the Nationwide Inpatient Sample,5 or the Pediatric Hospital Information System (PHIS),6 to assess the occurrence and characteristics of abusive injuries.
Despite the recent use of HDDs to examine the incidence of abusive injuries, there is little information available on the accuracy of the ICD-9-CM codes to identify cases of physical abuse correctly. One of the early studies examining the accuracy of coding of injuries in children who were hospitalized or died in California in 1991–1992 showed that only 75% of injuries due to violence had the correct ICD-9-CM codes; the authors noted that of the incorrectly coded cases, 75% were due to physical abuse. In contrast, of the cases with injuries that were not due to violence (eg, falls), 99.7% had the correct ICD-9-CM codes.7 More recent studies have focused specifically on the coding of physical abuse cases but have been limited because they use data from single children’s hospitals.8,9 For example, Hooft et al8 examined the accuracy of ICD-9-CM codes at one children’s hospital and found that the sensitivity of ICD-9-CM codes for abuse was only 77%, but the specificity was 100%; thus, over 20% of abuse cases documented by a physician in the medical record were not coded as abuse in the hospital discharge data. Because coding practices can vary among hospitals, additional studies are needed to assess whether such findings are generalizable to other hospitals.
Therefore, the study’s purpose was to determine how accurately ICD-9-CM codes reflect the decisions made by child abuse pediatricians (CAPs) about the likelihood of abuse in 3 different children’s hospitals compared to the previously published results from Yale–New Haven Children’s Hospital.8 We also examined whether incorrectly coded cases were due to errors of physicians or coders.
To determine the accuracy of ICD-9-CM codes for abuse at 4 hospitals, we identified inpatient cases for which a CAP consultation was obtained. On the basis of the CAP’s consultation, cases were classified as abuse or not abuse. We then determined whether the ICD-9-CM codes for these cases correctly identified cases of abuse and nonabuse.
Hospitals chosen for participation were required to have: (1) a child abuse consultation service with at least 1 CAP, (2) an electronic registry of all inpatient cases for which CAP consultations were provided for 2007–2010, (3) availability of ICD-9-CM diagnosis codes for patients in the registry; and (4) for each case in the registry, the CAP’s recorded decision on the likelihood of abuse and the ability to translate this decision into a 3-point clinical rating scale (abuse, unknown/questionable cause, and accident/medical cause).
On the basis of the above criteria, we invited 2 child abuse teams at stand-alone children’s hospitals (Connecticut Children’s Medical Center [CCMC] in Hartford, Connecticut, and Children’s Hospital of Philadelphia [CHOP] in Philadelphia, Pennsylvania) and one team at a children’s hospital within a general hospital (Children’s Hospital at Montefiore [CHAM] in Bronx, New York) to participate. The data from these 3 hospitals were combined with the original data8 from Yale–New Haven Children’s Hospital (YNHCH) in New Haven, Connecticut, a children’s hospital within a general hospital.
At each hospital, subjects were identified using electronic registries kept by the CAPs. Subjects admitted with suspicion for physical abuse and evaluated by a consultation from a CAP from January 1, 2007, to December 31, 2010, were included. On the basis of the determination of the CAP, cases were classified using a 3-point scale on the likelihood that the child’s injuries were due to physical abuse: 1) abuse, 2) unknown/questionable cause, and 3) accidental/medical cause. Cases of suspected medical child abuse, sexual abuse, or neglect were excluded from the analyses unless there was also a suspicion of physical abuse.
Because of the retrospective nature of the data collection for this study, there were variations in the process of categorizing cases as abuse or not. These differences are described by institution.
CAPs at CCMC used a 6-point scale to rate the likelihood of abuse: 1) definite/diagnostic of abuse, 2) highly suspicious of abuse, 3) suspicious of abuse, 4) indeterminate, 5) probable accident, and 6) other illness. The cases were reclassified to fit the 3-point scale in the following manner: cases rated as 1 or 2 were considered abuse; cases rated as 3 or 4 were considered unknown/questionable cause; and cases rated as 5 or 6 were considered accidental/medical cause.
The registry did not contain ratings of the likelihood of abuse. Therefore, the CAP who evaluated the case reviewed the information in the database and medical record and retrospectively assigned a rating on the 7-point scale used at YNHCH.8 This rating was based on the information known during the child’s admission and was independent of the assignment of ICD codes by the hospital coders. Cases were then reclassified to fit the 3-point scale as done at YNHCH.8
The CHOP registry contained CAPs’ ratings of the likelihood of abuse for each case, and these were reclassified to fit the 3-point scale in the following manner: cases rated as abuse or likely/probably abuse were considered abuse; cases rated as possibly abuse, concerning for abuse, or as unknown/uncertain were considered unknown/questionable cause; and cases rated as not abuse or likely/probably not abuse were considered accidental/medical cause. There were 29 cases with no rating by a CAP that were excluded. To ensure that the ratings for the remaining cases were correct, one author (JNW) reviewed the case descriptions and comments entered in the database.
Cases evaluated for suspected physical abuse were classified using a 7-point scale: 1) definite/diagnostic of abuse; 2) highly suspicious of abuse, 3) suspicious of abuse, 4) indeterminate, 5) possible accident, 6) likely accident, and 7) definite accident or medical cause. The cases were reclassified to fit the 3-point scale in the following manner: cases rated as 1 or 2 were considered abuse; cases rated as 3, 4, or 5 were considered unknown/questionable cause; and cases rated as 6 or 7 were considered accidental/medical cause.8
Patients’ injuries were grouped into the following categories using ICD-9-CM codes: traumatic brain injury (TBI), fracture, retinal hemorrhage, abdominal injury, skin (superficial) injury, and other injuries.2 A child categorized as having an injury, such as a TBI, may have had additional types of injuries. In addition, we grouped all cases into those who had a single type of injury and those who had at least 2 types of injuries (eg, TBI and fracture, or TBI and retinal hemorrhage). Demographic information, such as patient age and year of admission, was obtained from the HDDs.
ICD-9-CM codes and E-codes for cause of injury were obtained from the HDD at each hospital and classified into the 4 categories used in the previous study: 1) abuse, 2) unknown/undetermined cause, 3) accidental/medical cause, and 4) no injury.8 Cases included in the last category had no ICD-9-CM diagnosis code for an injury and no E-code for cause of injury.
When examining the sensitivity and specificity of the ICD-9-CM codes for physical abuse, the CAPs’ ratings were considered the reference standard. To calculate sensitivity and specificity, we reduced the 3-point scale used to rate the likelihood of abuse into 2 categories: abuse and not abuse. The category of abuse included only the cases rated as abuse by the CAPs, and the category of not abuse included cases rated as unknown/questionable cause and accidental/medical cause.
The ICD-9-CM codes assigned to the patients by the hospital coders were grouped in a similar fashion: abuse codes were included in the abuse category, while any codes for unknown/uncertain cause, accidental/medical cause, and cases that received no injury code were considered coded as not abuse. If a case had an ICD-9-CM code for abuse and accident, the case was classified as abuse.
We then calculated the sensitivity and specificity of the ICD-9-CM codes to identify cases of abuse using the CAP ratings as the reference standard. These calculations were done for all hospitals in combination and for each hospital. We also calculated the sensitivity and specificity for different types of injuries and for children who had at least 2 types of injuries.
All cases in which there was disagreement in the 2 × 2 table of Abuse versus Not abuse were reviewed except at CHOP, where one third of the discordant cases were reviewed because of the large number of cases. At each site, one of the authors reviewed the key notes from the medical record, including the assessments by the CAP or other physicians, and prepared a case summary. Two of the authors (AMH and JML) independently reviewed each case summary and categorized the reason for the discordance as due to an error by the coder or an error by the CAP in not providing clear documentation of the likelihood of abuse. Discrepancies between the 2 reviewers were resolved through discussion. An example of a coder error occurred if the CAP’s note stated clearly that the cause of the injury was abuse, but there was no ICD-9-CM or E-code code listed for abuse. An error by the physician occurred if the case was classified in the abuse registry as definite abuse but the CAP’s note in the medical record said that abuse was being considered and the ICD code had no code for abuse.
The percentages of discordant cases that were due to coder or CAP errors were calculated accounting for the incomplete sampling of the discordant cases from CHOP.
Characteristics of the samples at each hospital were compared by chi-square test or analysis of variance. Sensitivity and specificity and 95% binomial confidence intervals were calculated; chi-square was used for comparing sensitivities and specificities. All statistical analysis was performed by SAS software, version 9.3 (SAS Institute, Cary, NC).
This study was approved by the institutional review boards at each hospital.
There were 936 cases of suspected physical abuse among the 4 hospitals. Table 1 lists the demographic and clinical characteristics of the sample. The most common types of injuries were fractures (51.4%), TBI (23.5%), and skin (21.8%).
There were statistically significant differences among the hospitals in the distribution of patient age and the occurrence of TBI (data not shown). CHAM had the lowest percentage of children who were <1 year of age (57.1%) and the highest percentage of those >3 years of age (19.4%). The hospital with the highest percentage of children with TBIs was YNHCH (30.1%), while CHAM had the lowest (15.2%).
Table 2 shows the CAPs’ ratings of the likelihood of abuse overall and at each hospital: 32.7% of the cases were rated as abuse, 18.2% as unknown cause, and 49.1% as accidents/medical cause. There was a statistically significant difference in the ratings among the 4 hospitals (P < .008), with CHAM having the lowest percentage of abuse (22.4%) and the highest percentage of unknown (26.1%) and accidents (51.5%).
Table 3 compares the CAPs’ ratings with the ICD-9-CM codes for the overall sample. Of the 306 cases rated as abuse by the CAPs, 73.5% had an ICD-9-CM code for abuse. In contrast, of 170 cases rated as unknown by the CAPs, 10.0% had an ICD-9-CM code for undetermined cause and 55.3% for accident, while 16.5% had no injury code; of the 460 cases rated as accident/medical causes by the CAPs, 71.1% had an ICD-9-CM code for accident, and 23.5% had no injury code.
Coding varied by hospital. For example, of the 65 cases classified as abuse by the CAPs at CCMC, 29.2% had an ICD-9-CM code for undetermined cause. In contrast, at the other hospitals, less than 5% of abuse cases had an ICD-9-CM code for undetermined. At CHOP, of the 66 cases rated as unknown cause by the CAPs, 42.4% had an ICD-9-CM code for abuse, as opposed to 7.0% at CHAM and 0% at the other 2 hospitals.
The overall sensitivity and specificity of the ICD-9-CM codes for abuse were 73.5% and 92.4%, respectively (Table 4). There was significant variability among hospitals, with sensitivity ranging from 53.8% to 83.9%, and specificity from 85.4% to 100%. Sensitivity and specificity also varied by the child’s injuries (Table 5). In cases where retinal hemorrhages were present, the sensitivity was 88.2%; this was significantly higher than when retinal hemorrhages were not present (70.6%) (P <.05). Similar differences in sensitivity were noted with the presence versus absence of a TBI, fracture, and the occurrence of >1 type of injury. In contrast, specificity was less affected by the presence or absence of specific types of injuries: only retinal hemorrhages showed a statistically significant difference.
Of the 936 cases, there were 129 (13.8%) with discordant coding for Abuse versus Not abuse. Of the 123 cases with available data, 63.4% were discordant as a result of errors by hospital coders and 36.6% as a result of errors by CAPs.
On the basis of data from 4 children’s hospitals, we have shown that the use of ICD-9-CM codes to identify physically abused children results in both under- and overidentification of abuse cases. The overall sensitivity was 74% and specificity was 92%, but there was marked variation among the hospitals, suggesting that that the coding of abuse differs across the United States. The low sensitivity indicates that about 25% of abuse cases identified by child abuse physicians were not identified on the basis of ICD codes. The specificity of 92% means that about 8% of cases rated as nonabuse by CAPs had ICD codes for abuse.
The sensitivity of the ICD coding for abuse was higher for certain types of injuries. Children with retinal hemorrhages or TBI or those with at least 2 types of injuries had higher sensitivities than those without the specific finding. The presence of retinal hemorrhages (usually in a child with a TBI) or the presence of 2 types of injuries may have led to more conviction and clearer documentation about the likelihood of abuse by the CAPs or better recognition by the coders, thus leading to higher sensitivities.
A recent study to examine the accuracy of ICD-9-CM codes for abusive injuries examined children with suspected abusive head trauma admitted to a single hospital.9 In that study, which used a similar methodology to ours, the sensitivity and specificity of ICD-9-CM codes for abusive head trauma were 92% and 96%, respectively, similar to our results for TBI of 85% and 92%.
Although the sensitivity and specificity of our study were relatively low, our results are similar to those from studies examining injuries in general and other medical diagnoses. For example, a review of published studies on the use of E (external cause of injury) codes, found that the accuracy varied from 64% when exact code agreement was examined (eg, fall from a specific object) to 85% when broader groups of codes were examined (eg, falls). In general, these studies compared the codes in the hospital record in selected cases with coding that was done by an independent coder after reviewing the hospital record.10
Using ICD-9-CM codes to identify medical diagnoses, such as pneumonia, has resulted in both under- and overidentification of children with pneumonia. In a recent study conducted at 4 children’s hospitals, the sensitivity of ICD-9-CM codes for the diagnosis of definite pneumonia varied from 66% to 100%, and the specificity varied from 69% to 93% depending on the diagnostic codes used.11
ICD-9-CM coding practices are not uniform and are based on practices at individual hospitals. For example, the hospital coders at CCMC had a much higher tendency to use the undetermined cause code (22.5%) compared to coders at the other hospitals (2% or less). At CHOP, the ICD-9-CM code for abuse was used frequently in cases categorized by the CAPs as unknown or accidental (42.4% of the cases rated as unknown by the CAPs) versus 7%, 0%, and 0% at the other 3 hospitals.
Documentation by the CAPs also varied by hospital, which may have influenced coding. For example, at CCMC, the CAPs used consistent terminology from note to note to indicate their judgments on the likelihood of abuse, but this was not the practice at all of the sites. One factor discussed when examining coding accuracy is the use of electronic medical records (EMR). During the study period (2007–2010), only one institution (YNHCH) began using EMRs for note writing full time. The other 3 hospitals were using paper medical records during this time period. The CAPs at CCMC typed notes in a word-processing software program and then placed a printed copy in the medical record. The CAPs at CHAM also had the capability of using this approach. On the basis of our results, it does not appear that the method of documentation was associated with improved sensitivity or specificity, but this conclusion is based on limited data.
The major strength of this study is that it was conducted at 4 children’s hospitals, in contrast to the 2 recent studies that were each conducted at a single children’s hospital.8,9 We also have shown that over one third of the coding errors were due to unclear documentation by the CAPs, a result that highlights the importance of clearly documenting the likelihood of abuse.
The study has several limitations. First, we collected data from 4 children’s hospitals comprising 2 standalone children’s hospitals and 2 children’s hospitals in general hospitals. US data on hospitalized children are based on information from over 4000 hospitals,12 most of which are community hospitals, so our sampling frame is limited, and this may affect the generalizability of our results. Second, we relied on retrospective data in the child abuse registries to classify the likelihood of physical abuse as determined by the CAP. The rating scales used by the CAP physicians for the clinical data were not standardized; they varied across site and may have varied across time. Furthermore, a previous study has shown that CAPs can disagree on the likelihood of abuse.13 In our study, however, we were not focused on whether the CAP’s classification of the specific case was correct but whether the ICD code accurately reflected the CAP’s classification. Third, we did not use specific criteria to determine whether the error in the discordant cases was due to the physician’s poor documentation or due to the coder’s error. We did, however, require agreement between 2 authors in making this judgment. Fourth, it is a possible that some cases labeled as discordant were misclassified. For example, because we focused on the CAP’s classification of the likelihood of physical abuse, a case in which the CAP determined that physical abuse was unlikely but that there was severe neglect would be classified as not abuse. Thus, the assignment by the hospital coder of a code for “other abuse or neglect” may have been appropriate in such a case, but it would have been considered a coding error based on our methodology. Finally, we did not examine the ICD-9-CM codes for children who were admitted to the hospital for injuries but had no CAP consultation. In this large group, most of whom would have had accidental injuries, we do not know if there were abuse cases that were missed by our study because they did not receive a CAP consultation or whether children with accidental injuries were sometimes incorrectly coded as cases of abuse. In the previous study by Hooft et al,8 these scenarios were rare; of the almost 29,000 hospitalizations of children during a 4-year period, there were only 2 cases identified with abuse codes in children who did not have a CAP consultation.
These results, if replicated at other hospitals, have important implications for the use of hospital data to count the occurrence of physical abuse. A sensitivity of 74% means that approximately one quarter of abuse cases are not being identified based on ICD-9-CM codes. For example, if this sensitivity were used to correct the number of cases of abuse identified in 200914 using the KID, the number of abuse cases would be increased from 4782 to 6462.
A specificity of 92% means that 8% of cases in which a CAP provided a consultation and believed that abuse had not occurred received an ICD code for abuse. This type of error would result in a falsely high number of abuse cases and thus would offset to some degree the undercounting due to the low sensitivity. It is important to note that only a few children hospitalized with injuries receive a child abuse consultation, so the specificity that we calculated does not apply when considering all children with accidental injuries in hospitals.
Coding of physical abuse is influenced by both physician documentation and coder interpretation. Physician documentation could be improved with increasing awareness of the need for a clear statement on the likelihood of abuse. If CAPs are more informed about the coding process and understand that coders require clear wording of an abuse diagnosis in order to assign an ICD code, they may be more likely to write follow-up notes communicating a clear diagnosis.
Coding of physical abuse is also dependent on the coder. There are minimal guidelines for coding abuse, and much is left to the individual coder as to the comfort level in assigning an ICD code for abuse. It may be difficult for the coder to interpret the documentation and determine the cause of injury. In addition, the social implications of coding abuse may influence coding (eg, fear of misassigning an abuse code or legal implications).
There are 2 important changes occurring in the United States that may influence the accuracy of coding of abuse. First, the widespread use of EMRs may have a substantial impact on the coding of physical abuse because some EMRs require physicians to provide their own diagnostic codes. Although hospital coders review physicians’ coding for accuracy and completeness (eg, adding appropriate E codes), it is unclear how this shift of responsibility to the physician will affect the accuracy of coding of child abuse cases.
Second, the United States will soon be using ICD-10-CM coding, which provides a more detailed approach to specify the type of maltreatment, the level of certainty (confirmed or suspected), the cause of the injury, and the suspected perpetrator.15 Because other countries have been using ICD-10-CM coding for many years, there has been interest in the use of these codes to identify child maltreatment. For example, a recent Canadian study used ICD-10-CM codes to estimate the incidence of abusive head trauma in Canada,16 and an Australian study showed that coding of child maltreatment in Queensland hospitals was accurately done with high sensitivity and specificity.17 Although such studies are encouraging as the United States converts to ICD-10-CM starting in October 2015, it will be important to monitor whether the transition affects the coding of children hospitalized for suspected physical abuse.
In summary, in 4 children’s hospitals, we found marked variability among hospitals in the sensitivity and specificity of ICD coding of abuse compared to the CAP’s likelihood of abuse. Overall, the sensitivity was 74% and the specificity 92%, suggesting both under- and over identification of abuse by ICD codes.
At 4 children’s hospitals, the sensitivity and specificity of International Classification of Diseases, 9th Revision, Clinical Modification (ICD-9-CM), coding for physical abuse compared to the classification by child abuse pediatricians were 74% and 92%, respectively, suggesting both under- and overcounting of abuse cases when relying on ICD-9-CM coding.
Funding was provided by the Child Abuse Funds, Department of Pediatrics and the Office of Student Research, Yale School of Medicine, and a grant from the Doris Duke Foundation for Clinical Research for Child Abuse Prevention. Dr Wood’s effort was supported in part by grant 1K23HD071967 from the US National Institute of Child Health and Human Development. We would like to acknowledge Valerie Mondestin for her assistance with data collection at CHOP.