PMCCPMCCPMCC

Search tips
Search criteria 

Advanced

 
Logo of intjepidLink to Publisher's site
 
Int J Epidemiol. 2010 April; 39(Suppl 1): i32–i39.
Published online 2010 March 23. doi:  10.1093/ije/dyq019
PMCID: PMC2845858

Comparing modelled to measured mortality reductions: applying the Lives Saved Tool to evaluation data from the Accelerated Child Survival Programme in West Africa

Abstract

Background The Lives Saved Tool (LiST) projects the magnitude of mortality reduction based on baseline coverage, demographic characteristics and coverage targets. As a validation exercise, we compared neonatal, post-neonatal, infant, child and under-5 mortality reductions as projected by LiST to changes in mortality measured through demographic surveys in Ghana and Mali as part of a recently completed, retrospective evaluation of a child survival programme.

Methods Using coverage and other information collected during the evaluation, we modelled the predicted mortality reduction, using logical assumptions to fill gaps if no data were available. We performed a sensitivity analysis on several indicators for which we used a proxy, using the results to examine model sensitivity and readdress our assumptions.

Results In Ghana, the modelled mortality reductions were within the 95% confidence boundaries of the measured reduction. In Mali LiST significantly underestimated the reduction. Several coverage indicators were found to influence the projection, specifically case management of serious neonatal illness in both countries and pneumonia treatment, vitamin A measles treatment and breastfeeding promotion in Mali.

Conclusions We consider LiST to be a useful tool given the limitations of the available data. Although the model was a good match in Ghana, we identified several limiting factors with the input data in the Mali projection. This exercise highlights the importance of continually improving the availability of sound demographic, epidemiological and intervention coverage data at district and national levels. More comparative studies are needed to fully assess the strengths and weaknesses of LiST.

Keywords: Child health, lives saved tool, mortality, evaluation, Ghana, Mali, accelerated child survival and development

Introduction

The Lives Saved Tool (LiST) can use baseline coverage data, programme targets and a demographic projection to model the magnitude of mortality reduction expected from planned programme activities, as described earlier in this issue1. As a validation exercise, we modelled data from a recently completed, retrospective evaluation of a child survival programme in West Africa and compared the results with estimates of mortality measured through demographic surveys as part of the evaluation. Using this large-scale, ‘real life’ evaluation to validate LiST will help current and future users understand the strengths and limitations of mortality-modelling tools.

The United Nations Children’s Fund (UNICEF) implemented the Accelerated Child Survival and Development (ACSD) programme in 11 countries from 2001 to 2005, supported by the Canadian International Development Agency (CIDA). UNICEF focused efforts on several ‘high-impact’ countries, including Benin, Ghana, Mali and Senegal, working closely with country governments and other partners to implement ACSD with the ultimate goal of reducing under-5 mortality. Focus districts within these countries implemented three intervention packages. The ‘EPI+’ package included vaccinations, vitamin A supplementation and distribution of insecticide-treated nets (ITN). The ‘ANC+’ package targeted pregnant women and included improved access to antenatal care and skilled delivery along with tetanus vaccination, intermittent preventive treatment of malaria (IPTp) and supplementation with iron/folate and postpartum vitamin A supplementation. The ‘IMCI+’ component included promotion of appropriate infant-feeding practices, iodized salt and improved sick-child care at facility and community levels, specifically treatment of diarrhoea, malaria and pneumonia.2

UNICEF commissioned the Institute for International Programs at the Johns Hopkins University (IIP-JHU) to carry out an external, retrospective evaluation of the ACSD programme in the four ‘high-impact’ countries of Benin, Ghana, Mali and Senegal. The evaluation design was a non-randomized, comparative study of changes in coverage, nutritional status and mortality in the focus districts and national comparison areas, defined as the entire country excluding the focus districts and the major metropolitan areas. The evaluation team, working with UNICEF and in-country partners, documented ACSD implementation in the focus districts and contextual factors in both the focus districts and nationally. Data on coverage of ACSD interventions, nutritional status and mortality were collected in both the ACSD focus districts and the national comparison areas through demographic surveys. The full methodology and results of the evaluation are available elsewhere.3

Methods

Data sources

We concentrate on the ACSD focus districts in Ghana and Mali for this exercise, as more robust information pertaining to intervention implementation and contextual factors was available for these districts. In Benin, we observed few significant changes in intervention coverage over time;3 thus, Benin was excluded from this examination of LiST-projected mortality reductions based on scale-up of intervention coverage. The endline survey in Senegal did not yield mortality estimates of adequate quality and the results were not used to evaluate the ACSD programme.

We used baseline and endline coverage and mortality measured through Demographic Health Surveys (DHS) in both countries and a Multiple Indicator Survey (MICS) in Ghana. The baseline and endline survey coverage indicators were inputted into the LiST and the projected mortality rates were compared to the survey endline mortality rates. Table 1 identifies the ACSD focus districts and provides information on the timing of the household surveys and the pre- and post-ACSD mortality estimation periods and rates for Ghana and Mali. Pre-existing survey data were reanalysed for baseline (pre-ACSD) estimates of coverage. We worked with on-going DHS or MICS to develop supplemental surveys that over-sampled households in the ACSD focus districts to collect and analyse endline (post-ACSD) coverage and two period estimates (pre- and post-ACSD) of mortality. Periods for estimation of mortality were based on country-specific timelines of ACSD implementation, defined through documentation; the periods differed in each country. Both pre- and post-ACSD period mortality rates were retrospectively calculated from the endline surveys using direct methods from the full birth history. A detailed explanation of methods used in the ACSD evaluation is available elsewhere.3

Table 1
Summary of the ACSD focus districts, the timing of the household surveys and mortality periods for Ghana and Mali

Inputs and assumptions

We used version 3.44 of LiST.4 Because the demographic projections and baseline data pre-loaded into LiST are national, we first adjusted the projection to represent the ACSD focus districts using available data from the 2000 census in Ghana and the 1998 census in Mali. National demographic data were used where no information was available for the focus districts. For Ghana, we adjusted the baseline sex and age population distribution and estimated migration to represent the Upper East region (UER) using 2000 census data. This assumes that the proportion of the population living in the UER relative to the rest of the country has remained stable. The total fertility rate was adjusted using regional estimates from the DHS 1988, 1993, 1998 and 2003. For Mali, we adjusted the baseline age and sex population distribution and migration to represent the focus districts using projected data from the 1998 census and adjusting for the proportional population of these districts. District-level total fertility rates (TFR) were not available, so we used the national-level estimates, which were similar to the regional TFR (containing the focus districts) estimates reported in the 2006 Mali DHS.

The national under-5 cause-of-death profiles loaded into LiST are from 2000 to 2003 for Ghana and Mali based on data from the Child Health Epidemiology Reference Group (CHERG) estimates.5 Cause-of-death data are not readily available sub-nationally for Mali, thus, we used the national profiles with the assumption that the cause-of-death patterns in the focus districts are similar to the national. In Ghana, we used the national cause-of-death profiles, which were similar to those reported from the Navrongo demographic surveillance site (DSS) located in the UER that conducts verbal autopsies to determine cause of death. The greatest causes of under-5 mortality for communicable diseases at the Navrongo DSS were malaria, followed by respiratory infections and diarrhoea.6

We entered baseline (pre-ACSD period) neonatal, infant and under-5 mortality rates, as measured retrospectively through the endline surveys, into the model for both countries (Table 1). Baseline, mid-line (for Ghana) and endline survey coverage estimates were also input into the model. Combined, Tables 2 and and33 present the intervention coverage indicators that are available for input in LiST. Table 2 shows the interventions for which we had available survey data, the indicator definition we used in the projection and the pre- and post-ACSD coverage values. Some coverage indicators were available in the surveys of both countries such as vaccination, antenatal care, oral rehydration salts solution for treatment of diarrhoea and others. Some indicators were only available in one of the country surveys. For instance, the Mali DHSs did not collect information on antibiotic treatment of suspected pneumonia so we used a proxy indicator, estimating that one-half of the children taken to a health facility would receive antibiotics. Other coverage indicators are not routinely collected in household surveys such as treatment of measles with vitamin A or case management of serious neonatal illnesses. In these instances, we used the survey data to estimate coverage of these interventions and the assumptions are shown on Table 2 as well. Table 3 shows the indicators that could be inputted in LiST for which no data were available from the surveys and we either excluded from the analysis or estimated using LiST formulas.

Table 2
LiST Indicators used in the country projections: indicator definition and the baseline and endline values for the Ghana and Mali projections
Table 3
LiST indicators not available from the survey data and not used in the country projections: either estimated using LiST algorithms or not included in the model

Sensitivity analysis

We conducted sensitivity analyses on the indicators listed in Table 2 for which we did not have the exact indicator from the survey and instead used a proxy. These include: complementary feeding—education only; complementary feeding—education and supplementation; full supportive care: case management of serious neonatal illness; vitamin A for measles treatment; breastfeeding promotion and case management of pneumonia (oral antibiotics). We did not perform a sensitivity analysis on indicators not routinely collected through household surveys (Table 3) such as emergency obstetric practices and abortion services, or for interventions that were not yet implemented in these two countries, e.g. zinc treatment and rotavirus vaccination.

Two scenarios were run for each indicator: a ‘low’ scenario in which coverage did not change from baseline to endline and a ‘high’ scenario in which coverage increased to 100%. For each scenario we changed only the indicator in question. We considered the model as being sensitive to a particular indicator if the range between the low and high scenario differed by five or more deaths per 1000 births relative to the original modelled under-5 mortality rate. If we found that the projection was sensitive to a particular indictor, we then examined why there was model sensitivity and critically assessed our assumptions.

Results

Table 4 shows the modelled results compared to the measured reduction in mortality rates. Because we measured period rather than yearly mortality rates in the ACSD evaluation, we used the 2005 LiST-projected mortality rates to compare to the midpoint from the post-ACSD endline periods. In Ghana LiST underestimates all mortality categories except post-neonatal. However, all LiST projections are within the 95% confidence interval of the measured mortality reduction and hence are not significantly different. In Mali, using the 95% confidence intervals, all mortality categories are significantly underestimated by LiST, except for post-neonatal mortality. However, the latter is not a significant difference when assessed against the 95% confidence interval.

Table 4
A comparison of the projection and the measured reduction for neonatal, post-neonatal, infant, child and under-5 mortality rates in Ghana and Mali

Table 5 shows the results of the sensitivity analysis. For each indicator we show the difference in the number of deaths for the under-5 mortality rate from the original projection for both the ‘low’ and ‘high’ scenarios and the total range. In both Ghana and Mali, we found the projection to be sensitive to changes in case management of serious neonatal illnesses. According to the national cause-of-death profiles, sepsis pneumonia causes ~30% of neonatal deaths in both Ghana and Mali, and no data were available on coverage for correct management of these infections.

Table 5
Sensitivity analysis on indicators estimated from survey data: the number of deaths different compared to the LiST projection for a ‘low’ scenario in which coverage did not change from baseline to endline and a ‘high’ scenario ...

Case management of pneumonia in Mali is shown to be the most sensitive across all the indicators tested for Mali and Ghana. However, the 100% coverage used in the sensitivity for the ‘high’ scenario is far from what is reported by surveys—care-seeking of 28% (baseline) and 31% (endline) in Mali, with one-half of these values used for pneumonia treatment with antibiotics. To further test this assumption we increased the endline coverage from one-half to three-fourth of the pneumonia care-seeking indicator which resulted in two fewer under-5 deaths—a minor difference. Hence, the difference between the LiST estimate of under-5 mortality change (21 per 1000 live births) and the lower bound of the measured mortality confidence interval (43 per 1000 live births) appears too large to be explained only by pneumonia treatment impact. At the same time recent evidence suggests that antibiotics are widely available in the informal private sector of Mali,7 so the impact of pneumonia treatment with antibiotics may be under-reported for Mali.

In Mali, we also found the projection to be sensitive to changes in coverage of vitamin A for measles treatment, breastfeeding promotion and case management of pneumonia (Table 5). Vitamin A for treatment of measles was also found to be marginally sensitive with the range between the ‘low’ and ‘high’ scenarios differing by five deaths per 1000 from the original projection. However, since the ACSD programme did not focus on treatment of measles with vitamin A supplementation, we do not have any additional information on this intervention in Mali.

Breastfeeding promotion was also found to be a sensitive component of the model in Mail, and we examined the prevalence of exclusive breastfeeding that was used as a proxy indicator. As part of the ACSD evaluation in Mali, we measured a statistically significant reduction of exclusive breastfeeding coverage from 44% in 2001 to 28% in 2007 (P < 0.001) in the HIDs.3 This reduction in breastfeeding is counter-intuitive and may have resulted from an imprecise measurement at baseline. However, having no strong argument to remove the indicator, we left it in the model.

Discussion

LiST projections of mortality reduction based on changes in coverage for proven interventions were within the 95% confidence intervals of the measured mortality reduction in Ghana, and significantly underestimated mortality in Mali. Given the limitations of the available input data, we consider the Ghana projection to be an adequate match to the measured mortality reduction. This does not hold true for Mali and there are important limitations with the input indicators for the Mali projection.

Our assumptions for breastfeeding improvement, neonatal infection case management, treatment of measles with vitamin A and antibiotic treatment for pneumonia in Mali may be underestimating the actual coverage of these interventions. Pneumonia treatment had the largest influence on the model and there is supplemental evidence that antibiotics are likely more widely available and prescribed than indicated in the survey-reported care seeking for pneumonia in Mali. Our documentation in the two countries also indicates that there was a rapid expansion of first-level health facilities in the focus districts in Mali during this period, and no similar expansion in Ghana. Another explanation for this finding may therefore be that increased access to health facilities in Mali increased coverage of unmeasured indictors, relative to Ghana. In the future, LiST users should consider using health facility surveys along with household surveys to capture the full spectrum of child health data that impacts mortality.

We have highlighted the sensitivity of LiST mortality projections in the case of treatment of pneumonia and serious neonatal illness, but it also applies to other indicators, such as diarrhoea and malaria treatment and nutritional interventions. The primary reason is that these interventions target illnesses that represent a high proportion of deaths in Africa. There is also the issue of accuracy of measurement in household surveys used to input data into the model. Our results indicate that the LiST projections were highly sensitive to changes in coverage levels for several interventions for which coverage was not, and in some cases is not, measured well in the household surveys that serve as the major, and sometimes only source of coverage data in most African countries. For example, accurate reports of the treatment received by a child depends on the caretaker’s recognition of the illness and correct reporting of the medicines provided at home or by a health care provider. The CHERG is currently working on improving the measurement of coverage for these and other proven child survival interventions.

A further dimension of accuracy is sample size. As can be seen from the baseline and midline household surveys for Ghana (Table 1), the sample sizes are small. Furthermore, the target group of children for which to assess treatment of some of the most sensitive LiST indicators, such as those on treatment of pneumonia and diarrhoea, is only a small proportion of all children. This is because the measurement of treatment can only be done on those children under-5 who had this illness in the last 2 weeks, a period required to reduce the impact of memory recall errors. Such situations require larger sample sizes in order to adequately assess changes in coverage between two points in time for such critical treatment indicators.

Our findings highlight the importance of continuing efforts to improve the availability of sound demographic, epidemiological and intervention coverage data at district and national levels. With respect to demographic data, LiST currently uses national-level projections. We have no reason to expect large differences between the ACSD focus districts and the country for fertility or migration patterns. Potentially more serious are differences in cause-of-death profiles for children <5 years of age between the focus districts and the country as a whole and changes in the profiles over time, especially with respect to levels of exposure to Plasmodium falciparum malaria. We attempted to assess the possible effect of using national estimates of the proportion of deaths due to malaria in the two countries by reducing the percentage of under-5 deaths due to malaria by half and examining the impact on the projected mortality reduction. We found negligible impact on the projected mortality (less than five deaths per 1000 live births different from the under-5 projected mortality rate). Given rapid changes in epidemiology in these and other countries in Africa due to the scale-up of malaria prevention,8,9 it will be important to periodically update the cause-of-death profiles.

Given this and other validation exercises10, LiST does appear to be a useful tool for projecting reductions in mortality based on changes in coverage. However, more comparison studies are needed to fully assess the strengths and weakness of the model. The quality of LiST projections is determined by the validity, quality and completeness of the input data—a limitation shared by all modelling tools. LiST users must examine and report on all assumptions made, and work together with countries and programmes to make informed decisions where high-quality data are not available.

As countries scale-up activities to reach Millennium Goal 4 to reduce under-5 mortality, indicators for tracking progress on high impact interventions need to be strengthened. Child mortality measures in many developing countries depend on household surveys, and these can only report on past mortality levels. More current data on high impact interventions can be obtained from the same surveys and these should greatly facilitate monitoring both changes in such interventions as well as likely changes in under-5 mortality using LiST.

Supplementary data

Supplementary data are available at IJE online.

Funding

US Fund for the United Nations Children's Fund from the Bill & Melinda Gates Foundation (grant 43386 partial) to “Promote evidence-based decision making in designing maternal, neonatal and child health interventions in low- and middle-income countries”; United Nations Children's Fund through the Canadian International Development Agency (to The Accelerated Child Survival and Development programme evaluation).

Conflict of interest: None declared.

KEY MESSAGES

  • LiST is a useful modelling tool, however, the limitations of available input data must be considered.
  • Modelled mortality reduction is sensitive to certain indicators, such as management of pneumonia and neonatal sepsis.
  • Continual improvements in demographic, epidemiological and intervention coverage data at national and district levels are necessary.

References

1. Boschi-Pinto C, Young M, Black RE. The child health epidemiology reference group reviews of the effectiveness of interventions to reduce maternal, neonatal and child mortality. Int J Epidemiol. 2010;39(Suppl 1):i3–6. [PMC free article] [PubMed]
2. United Nations Children's Fund. Accelerating Early Child Survival and Development in High Under-five Mortality Areas in the Context of Health Reform and Poverty Reduction: A Results-based Approach. New York: 2002.
3. Bryce J, Gilroy K, Jones G, Hazel E, Black RE, Victora CG. A retrospective evaluation of the accelerated child survival and development project in West Africa. Lancet. 2010;375 : [Epub 12 January 2010] [PubMed]
4. Lives Saved Tool, verision 3.44. [(25 November 2009, date last accessed)]. Available from: http://www.healthpolicyinitiative.com/index.cfm?id=software&get=Spectrum.
5. Bryce J, Boschi-Pinto C, Shibuya K, Black RE. WHO estimates of the causes of death in children. Lancet. 2005;365:1147–52. [PubMed]
6. Adjuik M, Smith T, Clark S, et al. Cause-specific mortality rates in sub-Saharan Africa and Bangladesh. Bull WHO. 2006;84:7. [PubMed]
7. Winch PJ, Gilroy KE, Doumbia S, et al. Operational issues and trends associated with the pilot introduction of zinc for childhood diarrhoea in Bougouni district, Mali. J Health Popul Nutr. 2008;26:151–62. [PMC free article] [PubMed]
8. Monasch R, Reinisch A, Steketee Rw, Korenromp El, Alnwick D, Bergevin Y. Child coverage with mosquito nets and malaria treatment from population-based surveys in African countries: a baseline for monitoring progress in roll back malaria. Am J Trop Med Hyg. 2004;72(2 Suppl):7. [PubMed]
9. United Nations Children's Fund. Roll Back Malaria, The Global Fund to fight AIDS, Tuberculosis and Malaria. Malaria & Children. Progress in Intervention Coverage, Summary Update 2009. New York: United Nations Children's Fund; 2009.
10. Friberg IK, Bhutta ZA, Darmstadt GL, et al. Comparing modelled predictions of neonatal mortality impacts using LiST with observed results of community-based interventions trials in South Asia. Int J Epidemiol. 2010;39(Suppl 1):i11–20. [PMC free article] [PubMed]

Articles from International Journal of Epidemiology are provided here courtesy of Oxford University Press