|Home | About | Journals | Submit | Contact Us | Français|
Death registration systems in rural China are in a developmental stage. The Disease Surveillance Points (DSP) system provides the only nationally representative information on causes of death. In this system, there are no standard procedures or instruments for ascertaining causes of death; hence available statistics require careful evaluation before use.
To assess the reliability of data from the DSP.
14 DSP sites were selected through stratified sampling, enrolling 2482 deaths registered during June–November 2002. Defined verbal autopsy (VA) procedures were used to derive underlying causes of death. κ Measures of agreement between VA and registered diagnoses were computed. VA diagnoses were used as references to compute sensitivity and positive predictive values. Finally, patterns of cause‐specific discordance between the two data sources were studied.
κ Scores indicate only moderate agreement for chronic obstructive pulmonary disease (COPD) and ischaemic heart disease (IHD). The results also suggest that COPD is undercounted in registration data. A significant degree of cross attribution of cause was found between the two data sources. Overall, the VA was successful in limiting deaths with ill‐defined causes to <10%.
Diagnoses based on structured symptom questionnaires in the VA approach seem plausible and reliable as compared with lay‐reported diagnoses in registration data. Concerns with attribution of cause of death due to COPD, IHD and tuberculosis in registration data suggest caution in their use for research and health programme purposes. The VA methods tested in this study offer promise for implementation in the routine registration system.
Reliable information on the levels and causes of mortality is critical for the assessment of population health status. Although complete vital registration systems with medical certification of the cause of death are the “gold” standard source for such data, they are difficult to establish and expensive to maintain in large populations such as that of China. Medical certification of the cause of death is dependent on the availability of a doctor at death. In rural China, most deaths occur at home, in the absence of medical attention and with limited contact with health services during the illness preceding death. Given this background, statistics on registered causes of death in rural China require careful evaluation if they are to be useful for epidemiological research and public health planning.
Mortality registration and cause of death reporting systems in China are in a developmental stage. A detailed description of the systems in place for the registration and compilation of cause of death statistics in China is available elsewhere.1 In brief, although close to a billion people reside in rural China, cause of death statistics are compiled for only 0.6% of this population through the nationally representative Disease Surveillance Points (DSP) system. This system operates in 78 townships (Xiangs) in rural areas, with populations ranging from 30000 to 100000 in each site. There are no standard procedures or instruments for the collection and reporting of causes of death. A total of about 35000 deaths are registered annually in these rural sites.2,3 Causes of death are investigated and registered through the Disease Prevention Unit in the local township hospitals, on the basis of unstructured interviews of the family of the deceased. In this paper, we report on the findings of an evaluation of the quality of cause of death attribution in rural China on the basis of a sample of 2482 deaths in 14 DSP sites. Clearly, understanding the reliability of cause of death data for one sixth of the world's population has important implications for global burden of disease assessments as well as for health sector planning in China.
A critical element in any evaluation of data quality is the assessment of content validity of the reported causes of death, by comparing them with gold standard reference diagnoses for the same deaths,4 preferably for a representative sample of deaths occurring in the population. In rural China, few deaths occur in a health facility, which excludes the possibility of deriving reference diagnoses for most deaths in any representative sample. Hence, it is not possible to conduct a classical validation study in rural China. The next best option is to assess the reliability of cause of death reporting in the routine registration system by comparing registered diagnoses with diagnoses derived through an independent, standardised cause‐of‐death ascertainment strategy. In this study, we used a standardised verbal autopsy (VA) procedure for the cause‐of‐death ascertainment concurrently implemented and validated in urban sites in China,5 to derive independent diagnoses for each death in the study sample.
In principle, the VA method is more reliable than the procedures used in the routine registration system since a structured, detailed set of questions about symptoms should lead to better cause of death certification and coding of the underlying cause of death. The validation study in urban areas of China used detailed hospital records to derive reference diagnoses. Although admittedly a blunt method, the VA was found to have good validity for stroke, cancers and perinatal conditions, and although less satisfactory for other leading causes such as chronic obstructive pulmonary disease (COPD), ischaemic heart disease (IHD) and tuberculosis (TB), the misclassification by VA was compensatory, resulting in minimal distortion of the overall cause‐specific mortality patterns in the study sample. With this knowledge of its intrinsic operational characteristics from the urban validation study, we have applied the VA method as a proxy gold standard to estimate the reliability of the routine registration system in rural areas. Although formal validation is not possible in rural China, it is likely that cause of death patterns based on the uniform application of a symptom‐based questionnaire will be more reliable than those based on informal enquiry of families about their opinion as to the cause of death of family members.
For those causes for which the VA is likely to predict correctly the underlying cause of death, the results of the VA re‐investigation might be used to correct the cause of death pattern reported by the routine registration system, for which statistical procedures have been proposed.6 Moreover, the re‐investigation is likely to reduce substantially the proportion of deaths coded to ill‐defined conditions and thus can be directly used in burden of disease estimation exercises, as has been done in India7 and Thailand,8 or for epidemiological research.9
The DSP system operates in a stratified random sample of 78 counties across four rural socioeconomic strata, ranging from rural 1 (best off) to rural 4 (worst off). For this project, 14 sites were randomly selected, including two from rural 1, and four each from the other three strata (fig 11).). The poorer rural strata were better represented in the study since it was anticipated that the quality of cause of death data would be more questionable in these counties.
The reference period for the study was 1 June 2002 to 30 November 2002. All deaths registered in each project site during this period were recruited into the study. In each site, the local Chinese Centre for Disease Control field office (responsible for the routine compilation of data) submitted a complete list of deaths registered during the reference period to the Project Office at the Chinese Academy of Medical Sciences (CAMS) in Beijing. The list, arranged in chronological order of date of death, included the name and address of the deceased, informant details, place of death and the registered cause‐of‐death. This resulted in a total of 2482 cases across the four strata (table 11).). The Project Office subsequently abstracted a list omitting the cause of death as the set of cases for field investigation.
For each selected site, a medical doctor from the corresponding township hospital was trained in the use of standard VA field instruments, and in VA interviewing skills at a central training programme conducted in Beijing. Each trained investigator was provided with the list of cases for field investigation, and the interviews were conducted during January–December 2003. Senior research management staff from the CAMS visited all field sites twice to provide onsite refresher training, supervision and feedback on interviewing skills and data recording. For the two sites in western China (Hetian and Menyaun), the study reference period was extended to March 2003 and the data collection to March 2004, owing to smaller population size and greater under‐reporting of deaths. The interval between death and household interviews ranged between 3 months and 1 year in most cases. After household interviews, each completed VA instrument was submitted to the CAMS for further processing. First, a panel of three medical practitioners (internists) from the Military Hospital in Beijing with similar training reviewed the VA responses, and assigned a cause of death for each case on the basis of the standard international form of medical certificate of cause of death.10 Trained medical coders at the World Health Organization international classification of diseases (ICD) Collaborating Centre in Beijing then selected and coded the underlying cause of death from each certificate, according to the rules and specifications of the ICDandhealth related problems, 10th revision (ICD‐10).10 In comparison, doctors at township hospitals certify causes of death in the routine registration data, and underlying causes are selected and coded by a team at the CAMS. Summary tabulations of deaths by age, sex and underlying cause were developed for each data source (registration and VA investigation), aggregated by cause according to the Mortality Tabulation List 1.11 Diagnoses from registration data were compared with VA diagnoses on the basis of the κ statistic of chance corrected agreement between the two data sources for the leading causes of death in the study sample.12 Using the VA diagnoses as a proxy gold standard, the inferred “sensitivity” and “positive predictive values” of the registration data for each cause were computed. Under the same assumption, a comparative analysis of cause‐specific mortality proportions for each cause category from the two data sources was used to determine the extent to which the routine registration system over‐ or under‐reported specific causes. Finally, matrices were constructed to describe details of the cause‐specific disagreement between the two data sources. Since the study sample is a close approximation of the registered cause of death profile in rural China (fig 22),), the findings of such descriptive analyses provide evidence for correcting routine registration data in rural areas and estimating overall cause‐specific mortality in rural China.
Table 11 shows the distribution of the study sample across the four socioeconomic strata according to leading causes as suggested by the routine registration system in these project sites. It would appear that cause attribution by the routine system in the worst‐off socioeconomic stratum (rural 4) is problematic, with a high proportion of deaths assigned to “other cardiac diseases” (25.7%) and “ill‐defined conditions” (12.5%), compared with other strata. On the other hand, the higher proportions of deaths due to perinatal causes, drowning, TB and pneumonia in this stratum are not inconsistent with lower socioeconomic status. Overall, a comparison of the proportionate distributions by cause from the aggregated study sample and the latest published DSP data from rural areas during 1997–9 (fig 22)) suggests that the study sample is reasonably representative of the overall DSP system in rural areas, the most notable exceptions being the lower proportionate representation of deaths due to COPD and pneumonia in the study sites.
Table 22 gives the results of comparative analyses between the registration diagnoses and the VA‐based re‐investigation of causes‐of‐death in the study sample. κ measures of chance corrected agreement >0.4 represent fair to excellent agreement according to the rating scale proposed by Landis and Koch,13 and are observed for almost all the leading causes‐of‐death in the table. Moderate or poor agreement (κ<0.4) was found only for TB, hypertensive diseases and poisonings, among the classified causes. Interestingly, κ was only marginally greater than 0.4 for COPD (κ=0.41) and IHD (κ=0.42) suggesting significant concerns with the coding of these conditions in rural China. The low κ values observed for “ill‐defined causes” and “other cardiac diseases” are reassuring since one would expect the VA procedures to be more likely to assign classifiable cause‐of‐death diagnoses for these cases.
Cause‐specific mortality fractions from the registered data and the VA procedures differ by >25% only for falls, TB and poisonings. This implies that if the VA diagnoses are a reasonable reflection of the true cause of death pattern, then the routine registration system is probably performing well for some causes (eg, stomach cancer and diabetes), but is yielding cause‐specific mortality fractions for many of the leading causes‐of‐death that are only within 25% of their true value. Although this may be a reasonable basis to ascertain broad policy priorities, it suggests caution if the data are to be used for epidemiological end points.14,15
Table 33 shows the major patterns of disease misclassification, at least when compared with the VA‐based diagnoses. Importantly, although the number of deaths due to stroke in the routine system (482) is similar to that suggested by the verbal autopsy enquiry (497), only 366 or 75% of the VA‐based deaths due to stroke were identified as such by the routine system. Significant numbers of deaths due to stroke were coded to COPD (33), other cardiac diseases (18), IHD (17) or hypertensive disease (15). Thus, although most of the misclassified deaths due to stroke fall within the broad category of cardiovascular diseases, the implications for research or planning that requires more specific epidemiological detail are substantial. Interestingly, the table suggests that COPD may be undercounted as a cause of death in rural China, with the routine system identifying 224 or 75% of probable cases (297), as identified by VA. On the other hand, IHD does not seem to be undercounted in this sample of rural deaths and, if anything, may be 30% or so over‐represented in rural mortality data.
A more stringent measure of reliability is the assessment of sensitivity of the registration system in accurately diagnosing causes of death compared with the cause suggested by the VA procedure. High sensitivity and positive predictive value scores (>75%) are observed only for some external causes, cancers, diabetes, cerebrovascular disease and perinatal conditions. Consistent with the findings from the κ analyses, leading causes of death such as IHD and COPD have low “validity” scores against this standard.
One of the most striking observations from table 33 is the poor agreement between the two data sources for deaths coded to “other cardiac diseases” (ICD codes I26–I51), with only 14 of the 224 (6.2%) events reported in the registration data being confirmed by the VA. Rather, the VA classified most of these deaths (45.5%) to COPD. Further investigation revealed that for these deaths, the cause of death registered in the routine system was labelled as “pulmonary heart disease”, the code for which is I27.9 (which falls in the “other cardiac diseases” cause grouping), whereas the VA identified the cause as “heart disease secondary to chronic obstructive pulmonary disease”, which under the ICD rules is coded to J44.9, the code for COPD.10 This difference in semantics seems to be responsible for the generalised under‐reporting of COPD by the routine registration system in rural areas. This has important implications for the rank order and magnitude of COPD as a cause of death in rural China.
Another condition for which agreement is low is TB (44%). Although the disagreement between TB and COPD seems compensatory, 11 deaths classified by the routine system as due to “other cardiac diseases” were diagnosed as TB by the VA. This suggests a possible under‐reporting of TB by the registration system of almost 50% in this study sample, although caution is required in making population level inferences because the number of deaths due to TB is small.
Disagreement between the two data sources for other leading cardiovascular, respiratory and neoplastic causes of death seems to be largely compensatory, resulting in minimal changes in the overall cause‐specific mortality profile. Interestingly, the proportion of ill‐defined diagnoses suggested by the VA (9%) is almost twice that in the routine system, (5.5% (137/2482)), and is a more plausible fraction given the procedures involved.
Table 44 shows the principal differences in diagnoses between the two datasets for external causes of death. If the VA diagnoses were likely to be a more reasonable assessment of the true cause of death in the study sample, then the registration system seems to under‐report deaths due to external causes by about 1%. This discrepancy mainly arises from the VA re‐classifying 16 deaths reported as due to “natural causes” by the registration system as due to falls, and a similar number to external causes other than those specified in the table. Thus, if the VA causes are correct, the implication is that the registration system under‐reports deaths due to falls by 57% (table 22).). The VA also re‐classified 10 deaths for which the registered cause was given as “suicide” to poisoning, which accounts for much of the substantial undercount of poisonings in the rural registration data. For the remaining external causes, the patterns of disagreement seem largely compensatory, and do not distort the rank order and magnitude of the reported cause‐specific mortality.
A recent evaluation of cause of death statistics in China4; and their uses,16 suggested an urgent need to evaluate critically the content validity of available mortality data. We provide such an assessment of routine statistics from urban China in a companion paper.17 The availability of reference diagnoses (ie, from clinical case histories) for validation in urban areas makes such an assessment more meaningful. Yet, since the vast majority of deaths in China occur in rural areas,18 it is of obvious importance to assess the validity of rural mortality data using similar procedures. However, the lack of reliable, detailed clinical information on deaths in rural areas precludes such an assessment.
This study has attempted to evaluate the quality of cause‐of‐death data in rural China using standard VA procedures, arguably the next best option. Overall, the registered cause‐specific mortality statistics seem to be reasonably reliable for several leading causes‐of‐death, but substantially incorrect for several other major conditions including COPD and TB, both of which are significantly under‐reported in the routine statistics, and IHD, which seems to be significantly overcoded. The probable misclassification of COPD is particularly worrying given that recent global estimates of mortality by cause, which have utilised the Chinese data, suggest that COPD kills about one million people a year in China, or 50% of the global total.19 The discrepancy in diagnosing deaths due to COPD may well be an artefact of the methods for certification and coding of death certificates in the routine registration system, and suggests the need for better training of certifying doctors and coders in the correct ICD procedures.
Inferences about the true underlying pattern of causes‐of‐death in rural China from our study need to be drawn with extreme caution. In the absence of a true gold standard diagnosis based on autopsies, or a good approximation to a gold standard based on detailed clinical evidence, it is unclear how reliable VA procedures will be in identifying the true cause of death for comparative purposes. Validation of the VA in urban areas of China suggests that the instrument is only approximately valid for several leading causes and this is a very questionable basis to assess reliability. Rather, our findings should be interpreted more as an alternative set of diagnoses based on standardised procedures than an attempt at validation of the rural cause of death data. Even where the operational characteristics of the VA procedures are known, and hence can be adjusted to suggest true cause‐specific mortality fractions, the problem of transferability of validation characteristics from one (urban) population to another (rural) creates additional uncertainty about the reliability of the VA diagnoses as a comparator. Additionally, the small sample sizes for some diseases such as TB make it difficult to draw conclusions about possible misclassification patterns for the large rural population in China.
Having said this, the performance of the VA in limiting the proportion of deaths classified to ill‐defined conditions to <10% is impressive, and consistent with conventional norms for cause of death statistics under this criterion.20 Where the VA investigation reveals no significant evidence, the procedures recommend the assignment of a symptom‐based or ill‐defined cause, and the fact that this was relatively uncommon in rural China suggests that the approach may have promise as a cost‐effective means of data collection for almost one billion people. However, more research is needed to ascertain the reliability of specific diagnoses; at present, there is no evidence to support or refute the diagnoses reported in the routine registration data, whether classified adequately or not.
Although the instruments and procedures for VA have only been formalised and termed as such for the past 20 years or so,21,22,23 similar methods have been used to derive causes of death in earlier times,24 and widespread adoption of VA procedures could very well hasten the development of useable cause‐of‐death statistics in much of the developing world.
An evaluation of data quality such as in this study serves additional purposes. The use of formal procedures for cause‐of‐death ascertainment in the study enables one to test their applicability and acceptance in rural China. The methods could therefore be applied more widely to improve the overall quality of cause of death data in rural China. Although data from the registration system have already been widely used for descriptive analyses of burden of disease19 and for evaluating health policy,25 the reliability and validity of the data are critical for these functions, as well as to monitor the health status of rural China, given the current differentials in access to healthcare26 and socioeconomic status27 in this vast country. In this situation, mortality indices, although crude, could be adequate to assess the magnitude of these differentials.28,29
Also, having greater confidence in the cause of death data emanating from rural China is essential if Chinese health planning is to be more reliably informed and global disease burden estimates are to be more confidently used. Although the methods used in this study do not allow formal validation of data from rural China, they are nevertheless suggestive of major misclassification errors for specific causes of scientific and public health interest. Certainly, epidemiological research based on these data that requires information on disease‐specific end points should proceed with great caution, although cause‐of‐death patterns for broader cause groups are likely to be more reliable since misclassification is often internal to broad disease categories. The research does suggest significant changes to population‐level cause‐specific patterns being estimated from these data, particularly for COPD and TB, and, together with the findings from validation research in urban China, could be used to re‐evaluate the pattern and stage of epidemiological transition in China.
We thank the US National Institute on Aging for providing financial support for this project (research grant number PO1 AG17625). The funding agency had no role in the study design and data collection, data analysis or in the preparation of this manuscript.
CAMS - Chinese Academy of Medical Sciences
COPD - chronic obstructive pulmonary disease
DSP - Disease Surveillance Points
ICD - international classification of disease
IHD - ischaemic heart disease
TB - tuberculosis
VA - verbal autopsy
Competing interests: None.