|Home | About | Journals | Submit | Contact Us | Français|
Background Data on mortality rates are crucial to guide health interventions in crisis-affected and resource-poor settings. The methods currently available to collect mortality data in such settings feature important methodological limitations. We developed and validated a new method to provide near real-time mortality estimates in such settings.
Methods We selected four study sites: Kabul, Afghanistan; Mae La refugee camp, Thailand; Chiradzulu District, Malawi; and Lugufu and Mtabila refugee camps, Tanzania. We recorded information about all deaths in a 60-day period by asking key community informants and decedents’ next of kin to refer interviewers to bereaved households. We used the total number of deaths and population estimates to calculate mortality rates for 60- and 30-day periods. For validation we compared these rates with a best estimate of mortality using capture–recapture analysis with two further independent lists of deaths.
Results The population covered by the new method was 76476 persons in Kabul, 43794 in Mae La camp, 54418 in Chiradzulu District and 80136 in the Tanzania camps. The informant method showed moderate sensitivity (55.0% in Kabul, 64.0% in Mae La, 72.5% in Chiradzulu and 67.7% in Tanzania), but performed better than the active surveillance system in the Tanzania refugee camps.
Conclusions The informant method currently features moderate sensitivity for accurately assessing mortality, but warrants further development, particularly considering its advantages over current options (ease of implementation and analysis and near-real estimates of mortality rates). Strategies should be tested to improve the performance of the informant method.
Data on the levels and causes of population mortality are critical for assessing a population’s health status and informing public health interventions in crisis-affected and resource-poor settings.1,2 However, in such settings, routine vital registration systems are often unreliable, cease to function or do not exist. The two main alternative approaches currently used in such situations to measure population mortality are prospective demographic surveillance (regular exhaustive enumeration by home visitors of individual births and deaths) and retrospective mortality surveys (estimation of mortality over a specified recall period based on interviews with a representative sample of households). Both approaches have serious limitations. Surveillance requires a large number of closely supervised staff to collect data; it can be difficult to implement in geographically dispersed populations;2 and the sensitivity of the surveillance system may be low, even in long-standing refugee camps.3 Retrospective surveys are poorly suited to generating essential operational data on current levels and causes of mortality, because, over very short recall periods (e.g. <1 month), they require unfeasibly large sample sizes to avoid meaninglessly imprecise estimates (with imprecision further compounded by the design effect inherent in cluster sampling).4–6 Retrospective mortality surveys are also subject to various selection and reporting biases, stemming mainly from inadequate sampling frames, unequal probability of selection of households (especially in cluster designs) and deliberate or involuntary inaccuracies in questionnaire responses.7 Statistical analysis of retrospective survey data is also more difficult than for surveillance data. Partly as a result of these limitations, the quality of mortality surveys done in the humanitarian sector is generally low.5
In this article, we report on a multi-site evaluation of an alternative method (which we refer to as the informant method) designed to furnish precise and unbiased estimates of mortality over a very short recall period (i.e. almost on a real-time basis) and be simple enough so that practitioners without statistical training can implement it to rapidly measure mortality in their communities of intervention. We evaluated the validity of the informant method against a best estimate of mortality provided by capture–recapture analysis.
The study was approved by the Ethics Committee of the London School of Hygiene and Tropical Medicine; the Institutional Review Board of the Ministry of Public Health of Afghanistan; the Malawi National Health Sciences Research Committee; and the National Institute for Medical Research and also the Commission for Science and Technology in Tanzania. In Thailand, no local institutional review board has jurisdiction over the refugee camps. Household respondents and focus group discussion (FGD) members were asked to provide oral consent to participate in this study. In Thailand, written consent was sought from all study participants.
Recognizing that validity may be affected by local culture and types of human settlement, we aimed to document the informant method's implementation, performance and limitations in a variety of settings. Specifically, although we mainly conceived of the method as a useful option in refugee or internally displaced persons’ camps, we also wanted to document its validity and feasibility in at least one chaotic, dense urban setting and one rural, scattered community. Furthermore, we wanted to ensure that the cultural settings of the study sites would be as unrelated as possible, though we specified that half should be in Sub-Saharan Africa. Accordingly, we selected the following four sites: (i) District (nahia) 1 of Kabul city, Afghanistan, a poor urban community with chaotic layout; (ii) Mae La camp for Karen refugees, on the Thai-Burma border; (iii) Chiradzulu District in Malawi, a rural, scattered community with a high HIV/AIDS burden; and (iv) Lugufu and Mtabila camps (hereafter considered as one site) in western Tanzania, which host refugees from the Great Lakes region. The Kabul site was chosen as it featured an existing London School of Hygiene and Tropical Medicine research programme. The broad regions of the other three sites (Thai-Burma border camps; Malawi rural; East Africa camps) were determined a priori, and the actual sites were identified through investigators’ field contacts.
In each site we enquired about a recall period of 2 months (60 days), but restricted the main analysis to the most recent 1 month (30 days). Results for the full 60-day period are also presented, as they provide useful information about recall of increasingly remote dates.
We surveyed the populations of District 1, Kabul; Mae La camp; and the Tanzania camps exhaustively. Assuming a minimum crude mortality rate (CMR) of 0.2 deaths per 10000 person-days within each of the above sites, and given that each site had an estimated population of more than 40000, we calculated that the method's sensitivity (i.e. proportion of all deaths that are detected by the method) for a 30-day recall period could be estimated among 26 or more deaths in each site. This number of deaths would have provided a precision within ±20% for any sensitivity point estimate >80%, which we considered an acceptable performance of the method.
In Chiradzulu district, it was considered inefficient to visit all 757 villages. Instead, we surveyed a fraction, selected using a modified centric systematic area sampling design. We divided the district into 32 5km×5km quadrats, with area mostly falling within the district boundaries. Using high-resolution maps, we then selected the three villages closest to the centre of each quadrat, thus yielding a non-self-weighting sample of 96 villages. Within each sampled village the search for recent deaths was exhaustive. Assuming an average population per village of 500, nearly 50000 people would be sampled, yielding a minimum precision comparable with that of the other sites.
The informant method consists of an exhaustive, active search for all deaths occurring in the surveyed community over a defined recall period. Key community informants, selected after rapid qualitative work, recall households that have experienced a recent death and refer data collectors to these households; decedents’ next of kin also refer data collectors to other recently bereaved households. The number of deaths recorded from the method is then combined with population estimates to calculate a mortality rate for the recall period (see below).
We held FGDs in each study site before data collection. Each FGD obtained information for the whole study site and sought to: (i) identify which key community informants would be most knowledgeable about recent deaths; (ii) explore information sharing about recent deaths in the community; and (iii) identify alternative sources of mortality information that would provide additional lists for capture–recapture analysis, later used to validate the method (see below). Local study team members and collaborators helped to identify participants, who were selected based on their knowledge of the community (Table 1). Although elements of this method have been used previously, such as obtaining information on deaths from key-informants such as persons in charge of cemeteries8 and traditional birth attendants (for maternal deaths),9 or using snowballing techniques for referrals to households that have experienced a maternal death,10,11 to the best of our knowledge this is the first method to combine the use of formative FGDs, information from key-informants, household referrals and population estimation to compute mortality rates.
To carry out the exhaustive search for deaths, we divided each community into sectors, corresponding to pre-existing administrative areas: guzar in Kabul (n=24); sections in Mae La camp (n=22); villages in Chiradzulu District (n=96); and zones in the Tanzania camps (n=23). The primary and secondary informants selected based on the FGD (see list 1 in Table 2) were located for each sector. The data collection then proceeded sector by sector. Sector leaders were informed in advance about the study team's upcoming visit. We asked key informants to separately list all deaths occurring among community residents during the previous 60 days and then visited the households in which those deaths had occurred.
We administered a questionnaire to next of kin aged ≥18 years to record the date of death, which we established with the aid of a detailed calendar of local events, age and sex of the decedent and the cause and place of death. We included deaths among neonates if they were breathing at birth. In Chiradzulu only, we also administered verbal autopsy questionnaires, using the latest World Health Organization standards.12 To uniquely identify decedents for the purpose of validation only (see below), we also collected the decedent's name; in Kabul and Mae La we collected the name of the father, and in Chiradzulu and Tanzania camps, the name of the household.
After administering the questionnaire, we asked respondents to list other deaths in their household or anywhere in the community within the previous 60 days. Once we had visited all households with deaths identified by the key informants and households of decedents, we considered the sector exhausted and moved on to the next sector. We did not visit deaths among households outside the community.
To measure the method's sensitivity, we obtained a best estimate of the true number of deaths in the recall period through capture–recapture analysis, used extensively in epidemiology to evaluate the completeness of reporting and/or the true burden of various diseases,13,14 and recently applied to war-related killings.15–17 This technique analyses the overlap among different lists of the events in question (in our case, recent deaths) to estimate the number of deaths that do not appear on any lists, which is then added to the ‘known’ or ascertained deaths (i.e. those that are found on one or more lists) to provide the best-estimate total.
In each site we generated three different lists of deaths, the first being that obtained from implementation of the informant method itself. We selected sources for the two additional lists (Table 2), based on criteria including sensitivity as judged by FGD participants, feasibility (i.e. how easy it would be to contact informants) and independence from the key informants already consulted (i.e. we avoided sources that would themselves obtain their knowledge from the informant method key informants, or vice versa). We collected these two additional lists only after the implementation of the informant method, to minimize investigator bias. We approached sources for the two additional lists in the same way as the informants used for our method, and asked about the same recall period. We considered deaths recorded in different sources as the same if they matched the sex, age, place of residence and name of the deceased in Kabul and Mae La, or father or household name in Chiradzulu and Tanzania camps.
Because the probability of a death appearing on a list is usually not independent from the probability of appearing on a different list (e.g. prominent individuals may be remembered by most sources; a death mentioned by one religious group might be unlikely to appear in the list of another religious group), possible interactions among lists need to be explicitly factored into the capture–recapture estimation. To do this, we fitted log-linear Poisson models18 to the three-way contingency table formed by each xabc category, where a, b and c denote the presence (1) or absence (0) of a death within lists 1, 2 and 3, respectively (e.g. x110 is the number of deaths on lists 1, 2 but not 3). Model terms represented the probabilities of inclusion in lists 1, 2 and 3; additional interaction terms represented possible dependencies among lists (i.e. inclusion in one list being a function of inclusion in another). Eight alternative candidate models were thus fitted (no interactions; one interaction among any two lists; two two-way interactions and three two-way interactions, i.e. saturated). Each model predicts mabc, i.e. the quantity inside each cell of the three-list contingency table, where m is the estimator of x (the true value in the cell). From these predictions,
and N, the estimated total number of deaths, is the number of deaths appearing on one or more lists plus m000.13
Instead of selecting one of the models as the most likely, we used Bayesian Model Averaging19 with uninformative priors to derive average estimates from all the models combined (in this method, the prediction is a weighted average of predictions from each model, with the weight provided by each model's posterior probability, a Bayesian statistic that indicates how likely it is that the model is the correct representation of reality). However, we first excluded any models that did not converge to a solution or had a Chi-square goodness of fit P-value >0.60 (we considered this evidence of overfitting; in practice, all these models also predicted implausibly high m000 values). Using the remaining models, we computed
where K is the total number of models averaged over, i is one of these models and Pr is its posterior probability. Ninety-five per cent confidence intervals (CIs) were derived from the models’ likelihood profiles.
When there were a sufficient number of deaths recorded, we estimated the total number of deaths among children <5 years of age, as above; however, this was not always feasible due to small numbers.
For each site and recall period (30 and 60 days), we expressed the sensitivity of the informant method or any other source as the number of deaths captured by the source, divided by the capture–recapture estimate of the total number of deaths (N).
Existing population estimates for all ages and children aged <5 years in Mae La and the Tanzanian camps were judged reliable and adopted. In Kabul and Chiradzulu, we performed our own population estimation (see Supplementary File 1 available as supplementary data at IJE Online). We expressed mortality rates as deaths per 10000 person-days. We estimated three different mortality rates using the number of deaths (i) captured by the informant method, (ii) captured by all sources combined and (iii) estimated by the capture–recapture analysis. We computed point estimates and 95% CIs through bootstrapping programmes specific to each field site, taking into account observed design effects and sampling weights where needed (see Supplementary File 1 available as supplementary data at IJE Online).
We double-entered and cleaned all data using EpiData version 3.0 software (EpiData Association, Odense, Denmark). All analysis was done using R software.
Data collection was conducted between July and October 2008, taking 11, 5, 16 and 6 working days in Kabul, Mae La, Chiradzulu and the Tanzania camps, respectively (Table 3); this includes time for collection of additional lists for capture–recapture analysis. The size of the four study populations ranged from 43794 to 80136 people. All community informants who were found, agreed to provide information. There were no instances in which a household was empty. One household in Tanzania did not consent.
In all four sites, primary and secondary informants provided the majority of reports (Table 4). Few reports came from respondent households; of these 3/11 in Kabul, 3/9 in Mae La, 3/6 in Chiradzulu and 12/14 were also reported by community informants, considering a 60-day recall period. Several deaths were excluded from the analysis, for the following reasons: (i) stillbirth (n=5); (ii) death outside the 60-day recall period (n=16); and (iii) not residing in the community (n=8).
Of the three different lists collected in each site (Table 5), the informant method identified the largest number of deaths, except in Mae La, where the existing surveillance system identified more deaths (the overlap of the deaths recorded among the three lists is shown as Venn diagrams in Supplementary File 2 available as supplementary data at IJE Online). In each site, the age and sex profile of deaths was similar for all lists, except in Kabul, where hospital-registered deaths were all male and mostly children, and surveillance records in Mae La which had a higher proportion of children. None of the lists showed notable differences in the age and sex profile comparing the 60- and 30-day time periods.
Sensitivity of the informant method ranged from 45 to 65% over a 60-day period, according to the site, and from 55 to 73% over a 30-day period (list 1, Table 6). The sensitivity of the informant method was higher than that of the other lists, except in Mae La and in Kabul over a 30-day period. The sensitivity of any source was higher over a 30-day recall period than a 60-day recall period, with the exception of the informant method in Kabul. The combination of all three sources appeared to achieve good sensitivity in all sites.
Among children <5 years of age, the method's sensitivity was 52.6% (95% CI 17.1–71.4) in Kabul (60-days period), 66.7% (95% CI 27.1–81.2) in Chiradzulu (60 days) and in Tanzania it was 53.7% (95% CI 36.1–61.1) over 60 days and 47.1% (95% CI 15.7–61.5) over 30 days. Other sensitivity estimates could not be computed due to the low number of child deaths, precluding capture–recapture analysis. However, considering ascertained deaths only, the informant method detected no more than 1/5 in Kabul (30 days), 2/16 and 0/4 in Mae La (60 and 30 days, respectively) and 12/15 in Chiradzulu (30 days).
The mortality rates estimated by the informant method, all the lists combined, and capture–recapture analysis are given in Table 7. The mortality rates estimated by the informant method and any other single method are substantially lower than those estimated through capture–recapture analysis over the 30- and 60-day periods. Mortality rates among children aged <5 years of age were approximately double the all-age CMR, as typically observed in developing country settings. Comparisons of our estimated mortality rates over a 60-day recall period with recent survey-based estimates of mortality from broadly comparable regions and also national-level estimates are provided in Table 8, and these show our capture–recapture estimates are broadly consistent with these other estimates.
To our knowledge this is the only recent study with the aim of developing and validating a new mortality estimation method based on primary data collection. The informant method potentially offers a more rapid means of estimating mortality than retrospective surveys and surveillance systems. The results show that the informant method performed better than the active surveillance system in the Tanzania refugee camps, but achieved moderate sensitivity compared with a best estimate of mortality, and was not good at identifying deaths in children <5 years of age. The informant method also cannot be used to record data on exposure to mortality risk factors, whereas retrospective surveys can potentially record such data.
There are different reasons why the informant method did not identify some deaths. The choice of key informants may have been sub-optimal and inappropriate for childhood deaths. This may have arisen because of dominance of community leaders among FGD participants, which may have masked the diversity of opinions in the community and discouraged other participants from expressing their views, resulting in over-direction by community leaders about which key informants we used. Furthermore, in both Kabul and Chiradzulu, key informants derived their information about deaths from similar sources. A greater number and diversity of key informants may have been warranted.
It may have been inherently difficult for key informants to know about all deaths in their sectors, due to: (i) large sector populations (about 3000 people in Kabul, 2000 in Mae La and 4500 in Tanzania); (ii) lack of trust and thus information sharing between key informants and community members (e.g. in Kabul many residents were short-term renters without kinship ties to the wakils); and (iii) inexperience of key informants (e.g. in Mae La many section leaders were newly appointed as part of a repatriation scheme).
Household respondents provided a minority of all referrals; given the relatively low mortality in the study sites, this may have been because households did not know about other deaths. Additionally, households may have been reticent to share such information with strangers, wary of upsetting community leaders or the bereaved families themselves or acting based on other cultural and/or religious beliefs and practices.
Both key informants and households may also have deliberately withheld information on deaths due to sensitive causes, such as suicide or drug addiction. This issue would, however, affect alternative mortality measurement methods and hence is not a particular weakness of this method. Furthermore, the main mortality rate estimate would not be affected if these deaths were mentioned, but falsely attributed to a more socially acceptable cause.
Theoretically, the informant method presents considerable advantages over both surveillance and surveys for the purposes of real-time mortality measurement. It is a one-off activity that can be repeated on a regular basis by staff with limited research skills. The small number of interviews conducted (i.e. only in households with deaths) greatly reduces data entry requirements; a simple analysis can be done with pen and paper, as there is no need for weighting and design effect adjustment; and the questionnaire can be expanded to explore the timing/location of the death (e.g. if the death occurred when residing in the study site or beforehand) and the causes and circumstances of death through methods such as verbal autopsy, since considerably more interview time can be devoted to each household than for other methods. The findings on issues of feasibility of the new method (e.g. time, financial, use of verbal autopsy questionnaires and ethical implications) will be presented elsewhere. A potential disadvantage is the need for accurate population estimation. However, population estimation is frequently required by humanitarian agencies for operational response in crisis-affected and resource-poor settings, and so is itself a useful measurement activity. Like retrospective surveys (but not surveillance), the informant method is also affected by potential survival bias (households disintegrating during the course of the recall period), though the potential effect of this bias may be small in our case due to the short recall period.
The method's sensitivity did not reach 80%, the level we aimed for when developing this method. Based upon our experience of developing and testing the method, we speculate that it may be possible to increase sensitivity by: (i) ensuring that the FGD participants represent a mix of men and women, ages, occupations and economic and social hierarchies; (ii) using multiple rounds of FGDs; (iii) using other formative research methods alongside FGDs, such as in-depth individual interviews or informal discussions; (iv) using more than two informants: in this study, we easily collected additional lists of deaths and; (v) using less formal informants (e.g. groups of people at gathering points such as water sources, shops).
Capture–recapture analysis may not have provided an accurate estimate of mortality. The method is highly sensitive to accuracy in matching individuals who appear in more than one list. In this study we matched on the basis of place of residence, sex, age and name: the latter two variables are often problematic in settings without vital registration. While some over- or under-matching may have occurred (resulting in under or overestimation of the true number of deaths, respectively), we did not note instances of equivocal combinations of the above characteristics, probably because the chance of repetition was very low (there were almost always less than five deaths reported on any list, for a given sector). Other problems with establishing the lists occurred, however. In Kabul, hospital records were often incomplete, leaving us with a very small list that decreased the precision of the capture–recapture estimate. In Mae La and Chiradzulu, data were collected through religious leaders and Health Surveillance Assistants, respectively, and we were not able to closely supervise data quality: if some of the deaths reported by these sources did not in fact meet our inclusion criteria, the total number of deaths would have been overestimated and sensitivity underestimated.
Establishing the correct date of death was challenging. It was apparent that many household respondents and informants had difficulties pinpointing the actual date of death. We attempted to minimize error through a series of questions about each death and a detailed calendar containing local salient events. The sensitivity of the informant method was lower for 60 days compared with 30 days, which suggests that respondents’ recall was worse over time: a well-known problem for retrospective epidemiological studies. Non-systematic misclassification of date of death, particularly around the 60-day mark, would generally have led to less overlap among lists, and thus underestimation of sensitivity and overestimation of the true number of deaths: this may explain the curious finding that, in all four sites, more deaths were estimated to have occurred during the period 60–30 days before the survey, than in the last 30 days (Table 6, last column). This finding, if not spurious, may also be due to a simultaneous seasonal effect across all four sites, though this is less plausible.
Capture–recapture analysis also entails arbitrary decisions in the choice of statistical models, though we attempted to minimize these through Bayesian averaging. Alternative choices of models did not significantly alter the main sensitivity findings (data not shown).
Despite these potential biases, the mortality rates we estimated through capture–recapture reflect expected patterns given the underlying epidemiological and demographic profile in the study sites, and were broadly consistent with recent survey data from neighbouring areas within the study countries20–22 (Morgan, O, personal communication) and also national-level estimates.23–25 In Afghanistan, we investigated an urban neighbourhood of Kabul that is well served by health facilities and reasonably well covered by preventive services and where mortality would be expected to be lower than the nationwide average, which includes under-served rural areas. The camp-based populations we investigated in Thailand and Tanzania enjoyed long-standing humanitarian assistance and comparatively good health services, which helps explain why their mortality rates are similar to or lower than the local and national mortality rates. In Chiradzulu district of Malawi, the CMR recorded by the informant method falls within the range of survey data from the local region and is slightly higher than the national rate, which is to be expected, as the national rate includes urban areas, which generally have lower mortality rates.
Alternative best-estimate options included prospective surveillance systems, but these either did not exist in the study sites or were suspected to have poor sensitivity, as suggested by a review of camp-based surveillance.3 We did not compare our estimates with retrospective household surveys as to our knowledge these have not been validated sufficiently to provide a reliable best-estimate comparator.2,4,7,26 Moreover, retrospective mortality surveys typically estimate mortality rates averaged over several months and are therefore not directly comparable with mortality estimates from the informant method, whose main objective is to use very short recall periods.
Lastly, we were unable to test the method in settings of high mortality, and further studies are required to understand how well it would perform in such settings.
This study suggests that a new method to estimate recent mortality, based on information provided by community informants, while performing at least as well as other available methods in crisis-affected populations, still missed a substantial proportion of all deaths, particularly among children. Overall, the method's sensitivity did not reach 80%, the level we aimed for when developing this method.
However, we believe the method shows sufficient promise to warrant further development given the paramount importance of real-time mortality measurement, particularly in crisis-affected populations. Variants of our method are already being used to identify cases of maternal mortality and severe acute malnutrition in the community,9–11,27 and further applications may be envisaged. These could cover supplementing or evaluating existing mortality surveillance systems (including cause-specific mortality such as maternal mortality, as recommended elsewhere);28 monitoring cause-specific proportional mortality within the context of vertical disease programmes; rapidly identifying persons living with disabilities or investigating outbreaks of diseases with very recognizable symptoms. Improving sensitivity through strengthened methods of selecting appropriate informants and eliciting their knowledge should be the focus of future research and development of this method.
Supplementary data are available at IJE online.
This study was funded by the generous support of the American people through the support of the Office of Health, Infectious Disease, and Nutrition, Bureau for Global Health, United States Agency for International Development (USAID), through both the FANTA-2 Project under the terms of Cooperative Agreement Number GHN-A-00-08-00001-00, and the FANTA Project (1998–2008) under the terms of Cooperative Agreement Number HRN-A-00-98-00046-00, managed by the Academy for Educational Development (AED). The contents do not necessarily reflect the views of USAID or the United States Government.
The authors thank the field staff and residents in the four sites. They are grateful to Megan Deitchler at FANTA-2; the Ministry of Public Health in Afghanistan; Annemarie ter Veen, Aurore Debellut and colleagues at Aide Médicale Internationale in Thailand; Médecins Sans Frontières-France in Malawi; the United Nations High Commissioner for Refugees; the Tanzania Red Cross Society, and Patrick Ball and colleagues at the Benetech Initiative.
Conflict of interest: None declared.