PMCCPMCCPMCC

Search tips
Search criteria 

Advanced

 
Logo of intjepidLink to Publisher's site
 
Int J Epidemiol. 2010 April; 39(2): 613–629.
Published online 2010 March 3. doi:  10.1093/ije/dyp389
PMCID: PMC2846447

An adequacy evaluation of a 10-year, four-country nutrition and health programme

Abstract

Background Evaluations of large-scale health and nutrition programmes in developing countries are needed for determining the effectiveness of interventions. This article critically analyses a non-governmental organization (NGO)-led large-scale, multi-country, 10-year micronutrient and health (MICAH) programme with an ‘adequacy evaluation’, that is, a documentation of time trends in the expected direction.

Methods MICAH was implemented from 1996 to 2005 in selected areas of Ethiopia, Ghana, Malawi and Tanzania, reaching >6 million people with numerous health and nutrition interventions. Coverage and impact were monitored through surveys at baseline, midpoint and end of funding. The data were subjected to post-hoc methods of quality determination, and, if of suitable quality, included in the adequacy evaluation.

Results Most collected data were of moderate or high quality and therefore included in the adequacy evaluation. There were moderate to large improvements in vitamin A status in Ethiopian school-age children, children <5 years of age in Tanzania and Ghana and mothers in Ghana. Iodine status improved in Malawi and Tanzania. Anaemia rates and malaria prevalence decreased in women, pregnant women and pre-school children in Ghana, Malawi and Tanzania, but anaemia increased in Ethiopian women. Large increases were reported for rates of exclusive breastfeeding and immunization. Child growth improved to the maximum that would be predicted with the given interventions.

Conclusions Numerous nutrition and health impacts were observed in the intervention areas, often of a magnitude equal to or larger than observed in controlled interventions or trials. These results show the value of integrated long-term interventions.

Keywords: program evaluation, nutrition, public health, Africa, iron, vitamin A, iodine, anaemia

Introduction

A recent series of publications in the Lancet has described the scale of malnutrition in developing countries,1 the possible effects of successful nutrition interventions on malnutrition2 and some of the financial, bureaucratic and institutional hurdles preventing implementation of these interventions.3 One of the key hurdles is the limited number of evaluations of the effectiveness of large-scale multiple-intervention nutrition programmes.2 The lack of effectiveness evaluations discourages investment in nutrition programmes and precludes well-informed decision making.

The efficacy of some single nutrition interventions has been frequently and thoroughly evaluated (e.g. iron supplementation,4 vitamin A supplementation,5 salt iodization6). However, the effectiveness (i.e. the efficacy in ‘real world’ settings) of large-scale integrated health and nutrition programmes has not been thoroughly evaluated.2,7 Victora et al.8 and Habicht et al.9 argue that evaluations of large-scale programmes are necessary for determining the effectiveness of interventions, and thus for deciding upon resource allocation and public health programming: ‘In summary, there are important restrictions to the external validity of RCTs [Randomized Controlled Trials] for complex public health interventions. The likelihood of effect modification implies that one cannot take for granted that interventions that are proven efficacious in controlled trials can be generalized to other settings’ (p. 403), although such assumptions are in fact made.2 Thus, there is a need for high-quality evaluation of large-scale programmes. However, such programmes are not usually led by capable researchers with the budget and mandate to evaluate programme effectiveness, but rather by programme implementers without the budget, mandate or skills to conduct the evaluation. As a result there is a paucity of literature documenting the nutrition outcomes of large-scale integrated nutrition and health programmes.2

The objective of this article is to provide evidence of the effectiveness of a large-scale nutrition and health programme in Africa. For this, we evaluated a non-government organization (NGO)-led large-scale, multi-country, 10-year comprehensive micronutrient and health (MICAH) programme, assessed with an ‘adequacy evaluation’. Victora et al. define an adequacy evaluation as, ‘documentation of time trends in the expected direction, following introduction of an intervention’ (p. 404).8 Adequacy evaluations have less rigorous designs than ‘plausibility evaluations’ and ‘probability evaluations’, which provide the evaluator with a more confident assessment of the programme’s impact. Included within adequacy evaluation are ‘performance evaluations’, which encompass evaluations of provision, utilization and coverage, and ‘impact evaluations’, which evaluate health outcomes and behaviours. Most of the evaluations within this article are ‘adequacy impact evaluations’, although immunization coverage should be considered an ‘adequacy performance evaluation’.

Prior to the evaluation, it was necessary to determine the quality of the evaluation data. Two specific questions were considered: (i) are programme evaluation data, collected by trained surveyors but with less rigour than would be expected in a randomized control trial, of sufficient quality to monitor programme impacts? And if so, (ii) were the interventions effective in improving the nutrition and health of beneficiaries?

The MICAH programme was launched in 1995 by World Vision Canada. The goal of the programme was to improve the nutrition and health status of women and children through the most cost-effective and sustainable interventions, in line with international targets set at the 1990 World Summit for Children for the virtual elimination of vitamin A and iodine deficiencies, and the reduction of iron deficiency anaemia in women by one-third.10

MICAH was implemented from 1996 to 2005 in five African countries, four of which are reported here (Ethiopia, Ghana, Malawi and Tanzania), reaching 4 million direct beneficiaries, and more than 6 million indirect beneficiaries.11,12 The fifth country, Senegal, was of different structure and duration, consisting of two 4-year interventions conducted in different areas, rather than one longer intervention, and is therefore excluded from this evaluation.

In this article, we evaluate the impact of MICAH through a detailed assessment of programme evaluation data, and we consider the implications for future large-scale nutrition and health interventions.

Description of the MICAH programme interventions

Considering baseline assessment of the vitamin A, iron and iodine deficiencies epidemiology, and following national policies, context-specific programme plans were developed for each MICAH country within a programme-wide framework of objectives and strategies. Multiple interventions, ranging from community-based supplement distribution to fortifying and diversifying foods, and to national-level advocacy for national policy change, were conducted to address the deficiencies and target groups identified. Interventions were integrated into existing systems, structures and services, wherever possible, to increase potential for sustainability. The interventions not only had a shared purpose, but also had many gross as well as nuanced differences based on country context. Table 1 summarizes the MICAH activities: all countries used multi-pronged integrated approaches to address various aspects of nutrition and health. Not all countries had the same target groups and activities, but all had multiple targets and activities. Programme areas within countries were poor rural communities, where few or no other major development organizations were operational. In Malawi and Ethiopia, proposals were invited from other NGOs, government organizations and local World Vision units (‘Area Development Programmes’). In both countries, programme areas were selected based on quality of the proposals, evidence of community need and applicants’ organizational capacity. In Ghana, the Kwahu South district was selected as it was known to have a high goitre rate, and World Vision was working in the area on a water and sanitation project. In Tanzania, the eastern zone was selected by World Vision’s National Office, with the agreement of the Ministry of Health, as an area in need of micronutrient interventions. Further details of each country programme are available in the Supplementary data available at IJE online (see Appendix 1).

Table 1
Summary table of MICAH interventionsa

A comparison of MICAH data with the Demographic and Health Survey (DHS) data for national rural samples (see Table 5) suggests that the selected areas were similar to or worse off than the rural average in each country at baseline. Ghana’s MICAH baseline in 1997 was worse than the 1998 DHS rural national numbers for immunization coverage, access to protected water source (but not access to latrines) and exclusive breast-feeding rates, with mixed results for child growth status (better growth in height, worse growth in weight). Tanzania’s baseline in 1997 was worse than the 1996 DHS rural national numbers for access to protected water source and latrines, and for child growth status. In Ethiopia and Malawi the DHS data were available for the follow-up year (2000) but not for the baseline year. If the DHS data are extrapolated back to the baseline year, assuming a linear trend, then 1997 MICAH indicators in Ethiopia were worse than DHS for vitamin A capsule (VAC) coverage in children <5 years of age, and better for measles coverage and latrine access. In Malawi, 1996 (extrapolated) DHS indicators were better than MICAH for all available indicators, except child growth, for which DHS and MICAH were similar.

Table 5
Programme coverage rates and outcome indicator prevalences at baseline (1996 or 1997), follow-up (2000) and endline (2004), and comparable available DHS data

Methods

Sampling and data collection

Details of the methods for sampling, data collection and data management are provided in the project documents.13 In brief, cross-sectional surveys were conducted in each programme area, at baseline (1996/97), the end of phase I (2000) and the end of phase II (2004). The surveys were conducted in the same month of the year in each country. Two-stage cluster sampling was employed, in which clusters were randomly selected using probability proportional to population size. Key indicators were assessed through structured interviews with a standardized questionnaire, and collection of biochemical, clinical and anthropometric data.14

The sample sizes per country were between 900 and 4801 randomly selected households per survey. Sub-samples for clinical and biochemical indicators were randomly selected from the parent sample. When there were multiple individuals of a given age group in a household, one was chosen using standard criteria, except for the immunization and anthropometric indicators in which case all children <5 years of age were included. Data were collected by trained enumerators and clinical staff (varying by survey, but including laboratory technicians, clinicians, nurses, anthropometrists). More than two-thirds of immunization data were from child health cards, otherwise as reported by the child’s parent. Child growth was measured using calibrated Salter scales and locally made length boards. Questionnaires were filled out and anthropometrics taken at the subject’s house. In some cases, the biological samples were collected from the selected children when they were assembled at local clinics or schools rather than at their homes.

Haemoglobin was measured using a portable haemoglobin photometer: HemoCue B-Haemoglobin photometer (HemoCue AB, Angelholm, Sweden)15 in Ethiopia (adjusted for altitude), Ghana and Malawi; and Hemo-Control (EKF-diagnostic GmBH, Barleben, Germany)16 in Tanzania. Urine was collected from school-age children for measuring urinary iodine. Samples were transported to the laboratory, where they were frozen until analysis with the Sandall–Kolthoff reaction following digestion with ammonium persulfate.17 For breast-milk retinol, breast milk was expressed from both breasts and a 20-ml sample was collected in a foil-wrapped container, transported with ice packs and then stored at –25°C until laboratory analysis by high-performance liquid chromatography (HPLC).18 Goitre was assessed by palpation19 by trained team members or clinical officers, and Bitot’s spots were observed20 by trained nurses, physicians or technicians. For malaria parasitemia, capillary blood samples were smeared and stained according to standard methods21 and read by trained technicians, who assigned a malaria parasites score of ‘present’ or ‘absent’.

Children’s heights and weights were converted to standardized height-for-age Z-score (HAZ) and weight-for-age Z-score [WAZ; the number of standard deviations (SDs) the child’s height or weight is from the reference population mean for children of the same age],22 using EpiInfo 6.04 (Centers for Disease Control and Prevention, Atlanta, Georgia).

Data were entered by data clerks using standard software (e.g. EpiInfo, MS Access, SPSS or CSPro); 10% of the data were double entered and checked for accuracy. For key indicators with >10% data entry error, data entry was manually checked. Initial analyses were performed by statisticians using EpiInfo or other statistical software. Additional cleaning and analyses were done by World Vision Canada using MS Access (various versions, Microsoft Corporation) and SPSS (versions 10 and 11, Chicago: SPSS Inc.). The results presented in this article are from this additional cleaning and analyses.

In all countries, the MICAH programme received ethics clearance from the Ministry of Health. In addition, local authorities, community leaders and respondents were informed of the purpose of the evaluation and consent was requested from each individual respondent.

Quality control

MICAH evaluation survey data were collected by trained enumerators, with field supervision by trained programme staff and overall supervision and guidance by an international team of epidemiologists, nutritionists and statisticians. Accepted practices for field-level quality control were followed.14 However, systematic repeat measures, repeat sampling and inter-lab sampling were not available for quality control of the MICAH data. Therefore alternative, post-hoc methods were used for evaluating the quality of data collected. Some of these methods have been used previously,23 whereas others were developed for the purpose of this evaluation. The methods were adapted for each specific indicator, but are outlined in general in Table 2. Not all variables were amenable to all types of quality control checks.

Table 2
Methods used for quality control assessmenta

Adequacy evaluation

The magnitude of the change for each key indicator from baseline to follow-up to final survey was compared with that observed in published controlled trials and reports of other large-scale programmes (Table 4). If improvement was of comparable magnitude to the high end observed in controlled trials, the impact was considered ‘high’; if the improvement was of a range common in other programmes, the impact was considered ‘moderate’; if the change was smaller than other programmes but greater than zero the impact was considered ‘low’. Testing of differences from baseline to follow-up was done by chi-square for categorical variables and t-tests for continuous variables.

Table 4
The maximum efficacy found in the literature for clinical, biochemical and biological indicators tracked in MICAH

Results

Quality control

Comparison of SDs

In most cases for which SDs could be calculated, the MICAH values were within range of observed SDs in published trials. Furthermore, the SDs were usually relatively constant between survey years (Table 3). Unusually high SDs (20-fold of that reported elsewhere) were observed for breast-milk retinol in Tanzania. The SDs for haemoglobin in pregnant women in Ghana in 1997, Tanzania in 2000 and Ethiopia in 2000 as well as for children <5 years of age in Ghana in 1997 were outside the range of SDs reported elsewhere by –40 to +15%.

Table 3
SDs of continuous variables in MICAH surveys in baseline (1996 or 1997), follow-up (2000) and endline (2004) compared with examples from the literature, for quality control purposes

Magnitude of change compared with other programmes and trials

The magnitude of change in the MICAH indicators was compared with what has been achieved in trials and large-scale nutrition or health interventions (Table 4). The Table 4 data were drawn largely from a recent review2 and the references therein. The objective was to document the maximum impacts observed, not necessarily the expected impacts, and so only the highest efficacies are reported. No data were found for nightblindness and Bitot’s spots in school-aged children, or for malaria parasitemia in non-pregnant women. Data were available for all other indicators.

The maximum impacts observed in efficacy trials were then compared with the changes observed in MICAH (Table 5). Changes in MICAH, equal to or greater than those observed in reviewed efficacy trials, are shown in bold. Changes as large as those observed in efficacy trials may initially be considered ‘questionable’—were there sampling biases in either the efficacy trials or the MICAH sample? And if not, can impacts observed in effectiveness studies be as large as those observed in efficacy trials? There were questionable results in all four countries but there were no indicators for which the results were questionable in more than 3 countries. Within countries, there were as few as three and as many as six questionable results. Whether these are false results due to poor assessment or excellent results due to strong programming is considered in the ‘Discussion’ Section.

Relationship between coverage and outcome

Changes in coverage data for vitamin A and iron supplementation and iodine fortification correspond to expected changes in outcomes in most cases (Table 5, cells with non-concordance between programme coverage and outcome indicator are shaded). Increased coverage with VACs from one survey to the next corresponded to decreased nightblindness in Ethiopia and Tanzania and decreased prevalence of low breast-milk retinol in Ghana. An increase in the percentage of households with iodized salt corresponded to decreased prevalence of low urinary iodine in children in Malawi and Tanzania but not Ghana. In Ethiopia, goitre rates dropped despite the decrease in iodized salt in the households and in Ghana goitre rates did not decrease despite the increase in households with iodized salt. An increase in iron tablet supplementation corresponded with a decrease in anaemia prevalence in seven of eight cases (that is, across countries for pregnant women, non-pregnant women and children <5 years of age, where supplementation and anaemia data are available for ≥2 years). In all five cases where malaria parasitemia data were available, an increase in iron tablet supplementation and a decrease in parasitemia corresponded to a decrease in anaemia. In all four countries, an increase in access to a protected water source and latrines corresponded to decreased intestinal parasite infection.

Pattern of cross-sectional growth curves

For each country survey, average HAZ and WAZ and sample size at 1-month intervals were plotted against child age, and the plot was examined for the characteristics described in Table 2. The data were then scored as low quality (do not use), medium quality (use with caution) or high quality. The Ethiopian data were considered medium quality as many of the child ages were reported by mothers, and they appeared to be very often rounded to the half-year and year. The 2004 data for age > 48 months were low quality (Z-scores higher than considered feasible). The data from Ghana and Malawi were low quality at age <6 months as there was a gain of ~1 Z-score from birth to 4 months of age that is considered unlikely; otherwise the data were of high quality in all three surveys. Tanzanian data were high quality in all three surveys.

Immunization coverage

In all cases, except Ethiopia 2004, the MICAH immunization coverage rates followed the pattern of BCG > DPT3 ≈ OPV3 ≥ measles. Furthermore, the ratio of coverage of BCG:measles was between 1.0 and 1.3 in all cases. Overall, the MICAH immunization data were considered reliable and useable for evaluation purposes.

Adequacy evaluation

The MICAH programme impact in each country is summarized in Table 6, where the change in each indicator over the course of the intervention (appearing in Table 5) is graded as high, moderate, low, none or negative, and the quality of the data supporting the claim is rated as high, medium or low.

Table 6
Summary of impact of MICAH, by indicator and country

There were moderate improvements in vitamin A status in Ethiopian school-age children, and high improvement in vitamin A status of Ethiopian children <5 years of age and Ghanaian mothers. Nightblindness dropped to zero in Tanzanian children <5 years of age (P > 0.05). Iodine status improved in Ethiopia, Malawi and Tanzania. Anaemia rates decreased in women, pregnant women and pre-school children in Ghana, Malawi and Tanzania, but increased in Ethiopian women. Malaria prevalence dropped in the same groups. Large increases were reported for rates of EBF and immunization rates and child growth status also improved in all countries.

Discussion

Integrated nutrition programmes are commonly recommended for developing countries, but they are not commonly implemented and even more rarely evaluated. A recent review identified 30 large-scale nutrition programmes, most of which did not have strong designs for monitoring and evaluation.2

The MICAH programme appears to be unique in that it was an NGO-led multi-year, multi-country integrated nutrition and health intervention with monitoring of numerous clinical, biochemical and behavioural indicators. The data were collected for results-based management purposes, and as part of the contractual requirements of the donor, and have been reported in detail to the donor. In this study, we considered whether the evaluation data were of sufficient quality to evaluate programme impacts, and, if so, the direction and magnitude of the impact—an adequacy evaluation.

Post-hoc assessment of quality of data collected

Most of the collected data were considered of good quality, according to the methods used in this article (Table 2). The important exceptions included anthropometric data in Ethiopia, anthropometric data for children <6 months of age in Ghana and Malawi, breast-milk retinol analysis and urinary iodine in Tanzania and EBF data in all countries. There may have been poor laboratory practices in Tanzania for breast-milk retinol and urinary iodine, and other published evaluations of the quality of retrospective EBF data24–26 cast doubt on all retrospective exclusive breastfeeding data. In addition, breast milk retinol data and urinary iodine data in Tanzania were of very low quality and were excluded from the adequacy evaluation.

Based on the quality checks carried out in this study, it appears that the following micronutrient indicators are of sufficient quality when collected by programme-based organizations (such as NGOs) if adequate training and quality control measures are taken during the data collection: for vitamin A status—clinical indicators (night blindness, Bitot’s spots); for iodine status—goitre; and the biochemical indicator of anaemia as a proxy for iron status. In addition, anthropometric indicators can also be reliably collected for children between 6 and 59 months of age. Health indicators such as malaria and immunizations coverage also appeared to be of adequate quality.

Although the quality checks employed here (Table 2) cannot replace standard methods of repeat measures, inter-laboratory comparisons, inter-observer reliability tests and others, they do offer a relatively inexpensive and easily applied means of post-hoc data quality assessment. These quality check analyses allow the assessment of the impact of MICAH to be conducted with more confidence. Furthermore, the analyses give confidence that future surveys carried out by World Vision, or by other similarly staffed and resourced NGOs, can be done with sufficient accuracy and reliability for an adequacy evaluation.

However, these post-hoc methods are limited by their subjectivity, such as choosing the cut-offs used for magnitude of SDs, deciding what constitutes a consistent relationship between coverage and outcome, the maximum feasible magnitude of change for each indicator and the definition of an acceptable level of fluctuation in average Z-score between months. These were all determined after the analyses were completed and there are no existing guidelines. Further research is needed to provide standards and guidelines on such post-hoc methods. Having standards and guidelines could increase the reliability of data collected by non-research and/or programme institutions without adding significantly to the cost of data collection.

Adequacy evaluation

Magnitude of impact

Based on the finding that most of the indicators were of good quality, an evaluation of the change in indicators over time (adequacy evaluation) revealed many positive results in the MICAH programme (Tables 5 and and66).

MICAH programme staff reported five aspects of the programme that they believe uniquely contributed to the positive results: (i) results-based management identified programme activities that were not producing positive outcomes and allowed for mid-stream corrections, such as the decision in Malawi (following the 2000 evaluation) to reduce the geographic spread and intensify the intervention in a smaller area to ensure all participants received all interventions; (ii) provision of regular, intensive technical support in the form of regular email correspondence, monitoring and support visits by World Vision technical staff and expert consultants, and annual training workshops; (iii) use of a broad-based integrated package of interventions rather than relying on a single ‘magic bullet’; (iv) community participation in programme design, implementation and monitoring and evaluation, such that some interventions could be tailored to suit community preferences, especially regarding animal husbandry where existing practices differed from community to community; and (v) regular supervision of staff in communities. Other aspects of the intervention, such as having competent staff, effective supply chains and proper financial controls are always prerequisites for success, though not unique to MICAH’s success.

In Malawi there was a positive impact on all indicators, perhaps a result of higher investment per direct beneficiary compared with the other countries, as well as MICAH’s support for the National Micronutrient Coordinator position housed within the Ministry of Health and Population that led to concurrent positive developments in national nutrition policy and action. In Ghana and Tanzania, improvements were observed in vitamin A, iron, malaria, EBF, immunization and child growth status. There were fewer positive results in Ethiopia, which may be because the intervention efforts were ‘diluted’ over a larger number of beneficiaries. However, there was positive impact on vitamin A status in school-age children—an uncommon target and an uncommon success.

It has been estimated that ~US $5−10/head/year is a workable level of expenditure in nutrition programmes,27 and Malawi, the one MICAH country with costs that fell within this range, had the broadest range of positive results and overall impact appeared correlated with per capita expenditures.

Vitamin A

Decreases in clinical symptoms of vitamin A are similar to those found in other large-scale programmes. The xerophthalmia results in Ethiopia were better than observed previously (Table 4), but considering the 10-year duration of the intervention the large effect reported is feasible.

The logistics of breast-milk collection and storage were difficult, as was measuring retinol in the samples. The breast-milk retinol data collected were considered of moderate quality (Ghana) or too poor quality to use (Tanzania). Breast-milk retinol data should perhaps be reserved for research/controlled trials, with other indicators being used for programme monitoring.

Coverage of children <5 years of age with vitamin A supplements increased greatly in Ethiopia, Ghana and Tanzania (from 9 to > 70%), although it dropped in Ghana in 2004 to 26% when a novel method of distribution was tested. Coverage with supplements to new mothers increased in Ethiopia, Ghana and Tanzania from ~10 to 40–83%, an effect likely related to the introduction by MICAH of community-based distribution alongside facility-based distribution. There was a concomitant improvement in vitamin A status in Ethiopian children <5 years of age (P < 0.001) and of school age (P < 0.001), and in Ghanaian post-partum women (P < 0.001), but not in children <5 years of age in Ghana (P = 0.53) and Tanzania (P < 0.001) (although observed xerophthalmia dropped to zero).

Iodine

There were improvements in iodine status in Malawi, consistent with improvements observed elsewhere when successful long-term salt iodization programmes are in place.6,28,29 In Malawi, the MICAH programme had a particularly strong focus on increasing usage of iodized salt (increased household coverage from 59% in 1996 to 88% in 2004) through efforts at multiple levels (i.e. national, regional and community), as well as ensuring provision of iodized oil capsules in areas where iodine deficiency disorder was a severe public health problem. National-level advocacy facilitated full implementation of salt iodization legislation. District-level customs and border control staff and port authority health inspectors implemented testing of salt at border crossings. Intensive community education and regular monitoring of salt at the vendor and household levels increased consumer awareness and demand for iodized salt. In contrast, although iodine deficiency was a major concern of MICAH Ethiopia, there was decreasing iodized salt available at household level (i.e. 48% in 1997 decreased to 2% in 2004) due to conflict with neighbouring Eritrea, their traditional supplier of iodized salt, and lack of identification of adequate alternate sources of iodized salt. Neither Ghana nor Tanzania had major public health problems with iodine deficiency at the beginning of the programme, based on goitre rates (i.e. 2 and 9%, respectively). In Ghana, a reliable iodized salt supply was difficult to secure, as there are many small-scale producers in the country and existing salt iodization legislation is difficult to enforce, although household supply increased from 33 to 63% during the programme. This was due to strong programme advocacy efforts at the national level and collaboration with one of the major iodized salt suppliers in the country to provide increased supply to the targeted district.

In Tanzania, the prevalence of goitre was reduced to 0%, perhaps due to the programme emphasis on ensuring communities’ access to iodized salt (i.e. household salt coverage increased from 41% in 2000 to 72% in 2004).

Iron

In project areas of Ghana, Malawi and Tanzania, where anaemia was identified as a severe public health problem, programme resources were focused on decreasing anaemia and subsequent decreases in anaemia prevalence were observed. This is a particularly important finding, as anaemia is often very difficult to address—although this magnitude of change is not unprecedented (Table 4), it is uncommon. The improvements observed in MICAH are likely a result of the use of multiple strategies (supplements, animal source foods, fortification, malaria control, water quality improvement, sanitation, deworming) over a long intervention period. Furthermore, treating not only pregnant women but also non-pregnant adult women with iron supplementation allows for improved iron status before pregnancy, making it easier to prevent anaemia in pregnancy30 and may explain the unusually large improvement in anaemia rates in pregnant women of Ghana and Malawi despite high prevalence of HIV/AIDS in the latter country.

EBF

The large increases reported in EBF are relatively high compared with other interventions (of much shorter duration).25,31 Self-reports on duration of exclusive breastfeeding have been shown to have low32 to moderate24 accuracy (whereas duration of any breastfeeding has higher accuracy).24,26 Although over-reporting of EBF duration may be expected as promotional messages sensitize mothers to the ‘right’ answers, this does not necessarily occur.33 Although acknowledging the potential weakness of EBF data, MICAH’s focus on breastfeeding communication through community-based interventions for behaviour change [e.g. through women’s groups, community health workers (CHWs) and volunteers, traditional birth attendants, agricultural extension workers] in all countries and partnerships in Ghana and Malawi with the Baby Friendly Hospital Initiative—in which baby friendly practices were promoted not only in the hospitals but also in health centres that provided outreach services by trained nurses to pregnant women and new mothers in rural communities—is likely to have contributed to improved infant feeding practices.

Immunization

MICAH supported the delivery system (cold chain equipment, monitoring, per diems), and put extensive efforts into social mobilization and community education. This likely contributed to improvements in immunization coverage in all four countries (high in Ethiopia, moderate in Malawi, low in Ghana and Tanzania).

Anthropometry

Perhaps the most important indicator of all is child growth status, as it integrates the health effects of all the interventions. Stunting and underweight were reduced in all countries by ≥10%, except Ethiopia. Improved dietary diversification was promoted for children <2 years of age, utilizing food demonstrations and nutrition education to caregivers to promote improved infant and complementary feeding practices, including a revolving fund to increase household access to animal source foods in Ghana, Malawi and Tanzania. Infant and young child feeding was also an emphasis of the programme in Ghana, Malawi and Tanzania, through women’s groups, education through CHWs and volunteers, and agricultural extension workers (Malawi), as well as through supporting the Baby Friendly Hospital Initiative (Ghana and Malawi), and forming mother support groups in communities (Ghana and Tanzania). The smaller impact on child growth in Ethiopia is consistent with less emphasis placed on infant and young child feeding. Reduced illness frequency and parasitic infection also likely contributed to the improvement in growth, given the programme’s emphasis on water, sanitation and deworming.

Bhutta et al.2 developed a model predicting the reduction in stunting with a large-scale programme comprised of eight types of interventions: balanced energy–protein supplementation, intermittent preventive treatment of malaria, multiple micronutrient supplementation in pregnancy, breastfeeding promotion, complementary feeding and other supportive feeding strategies, vitamin A and zinc supplementation and hygiene intervention. If coverage of all eight interventions was 99 or 70% then the relative reduction in prevalence of stunting at 12 months would be 33 and 23%, at 24 months 36 and 24% and at 36 months 36 and 24%, respectively.2 Over the 10-year MICAH programme (which included four of the eight interventions in the Bhutta model, plus other components) this theoretical maximum reduction in prevalence of stunting of 24–36% after 3 years was approximated in Malawi (15%), Ghana (28%), and Tanzania (36%). This is most encouraging and points again to the potential impact of multiple, integrated strategies.

Attribution

The question of ‘attribution’ of improvements is not easy to answer. An adequacy evaluation is not designed to enable attribution, as other confounding factors have not been controlled for: a plausibility evaluation would be required. In MICAH’s case, for example, concurrent activities and inputs by other international (e.g. UNICEF) and national (e.g. Ministry of Health) actors would also have contributed to improvements; also, there were some improvements in average socio-economic status of beneficiaries over time, which, independent of MICAH, could have contributed to improved water and latrine access, ITN usage and child growth status. However, the MICAH programme was implemented in predominantly rural areas where existing government nutrition and health services were generally weak. In an attempt to estimate the impact of MICAH independent of other local and global influences, comparisons were made with DHS data for the national, rural samples. The change in those indicators for which there were both MICAH and DHS data at baseline or follow-up and endline is shown in Table 7 as the difference of the differences (the difference between the MICAH difference between endline and baseline and the DHS difference between endline and baseline). For most indicators in all four countries, MICAH areas outperformed rural areas of the country as a whole. Although the comparison groups are not perfectly suited as control groups (different years, different baseline conditions, other differences not related to MICAH), given the MICAH performance compared with the DHS along with the general concordance between coverage and outcome indicators in the MICAH samples, it does support the interpretation that the improvements in programme beneficiaries are greater than the general trends, and greater than would have occurred if MICAH was not implemented.

Table 7
Difference in differences of coverage rates, EBF and prevalence of stunting, wasting and underweight between MICAH surveys and DHS surveysa

Was this an adequacy evaluation?

Although this evaluation has documented the trends over time, it still perhaps falls short of a full adequacy evaluation according to the prerequisites proposed by Victora et al.: ‘(i) the causal pathway must be relatively short and simple, (ii) the expected impact must be large and (iii) confounding must be unlikely’.8 These prerequisites are vague and open to interpretation.

The first criteria proposed by Victora et al. would not be met, as in the MICAH programme the causal pathways are variable, ranging from the very short and direct (e.g. MICAH-funded staff supervising the distribution of MICAH-purchased VAC to school-aged children in MICAH communities to decrease clinical symptoms of vitamin A defieciency among school-aged children), to long and indirect (e.g. MICAH partners using MICAH-financed training materials to advocate for salt iodization to increase availability of iodized salt at the household level to decrease goitre rate), where the health impact, if any, cannot clearly be attributed to the availability of training materials. In addition, the use of multiple, integrated strategies precludes the evaluation of any one intervention’s causal pathway.

The second criterion, that expected impacts be large, is also not consistent, as expected impacts are variable in the MICAH programme. It is not uncommon for salt iodization programmes to virtually eliminate low urinary iodine.34 On the other hand, improvements in growth status in previously published research are, on average, ~0.3 Z-scores, which may be about one-third to one-sixth of the population standard deviation. However, the sample sizes chosen provided adequate power in most cases such that statistical tests of programmatically important changes had results with P < 0.05.

The third prerequisite, that confounders are unlikely, is most certainly not met, as confounding (i.e. effects from multiple sources) is likely for many of the indicators. However, it is interesting that programme success was observed in those indicators most open to confounding (growth status, anaemia) as well as those least open to confounding (e.g. nightblindness, urinary iodine).

Thus, perhaps not a ‘perfect’ adequacy evaluation, the analyses and interpretation presented here are useful in assessing the quality of MICAH monitoring and the level of impact, and strongly suggest important positive changes of micronutrient status during the MICAH programme.

Conclusion

In conducting an adequacy evaluation we documented trends in health indicators following introduction of the MICAH programme. In doing so we have established World Vision’s capacity for undertaking such evaluations and we expect other similarly positioned NGOs to be able to implement similar evaluations. The challenge in future assessments will be to determine the most appropriate key indicators to measure impact, to improve the quality of the data collection and management so that all of the data are of high quality and to strengthen the study design to enable ‘plausibility’ and ‘probability’ evaluations.8

The rating of the magnitude of changes as high, moderate or low was, of course, subjective. Considering the specific characteristics of MICAH, it is a challenge in the interpretation to determine if the ‘questionable’ changes (i.e. improvements greater than observed in efficacy trials) were real results due to excellent programming or false positives due to limitations in the data collection methods. The duration of MICAH was longer than almost all the efficacy trials reviewed, and MICAH simultaneously addressed numerous aspects of nutrition and health, thus results larger than those reported in the published literature on single interventions are feasible. Therefore, unless there were specific, known problems, the results are considered to be valid, whereas it is acknowledged that there may be false positives. Further research on broad-based, long-term interventions is required to quantify maximum magnitude of impacts that may be feasible through long-term interventions. Presumably with a very successful, broad-based long-term intervention, the end points can be similar to the nutrition and health situation in developed countries (e.g. no goitre, anaemia prevalence <5%).

In conclusion, we have identified three lessons from this work. First, post-hoc methods appear reasonable and feasible for quality control assessment, but further research is required for standardization of the cut-offs and limits for the methods described in Table 2. Secondly, NGOs that are capable of executing large-scale health interventions can also be capable of conducting adequacy evaluations, using similar levels of support as MICAH, if planned for from the beginning. Thirdly, for programme implementation, integrated nutrition and health interventions can produce excellent results, perhaps even greater than randomized controlled trials. Further operational research quantifying the magnitude of the effectiveness of integrated programmes is required.

Supplementary Data

Supplementary data are available at IJE online.

Funding

Canadian International Development Agency (CIDA) (694/M3544, M-011372) and World Vision Canada.

KEY MESSAGES

  • This article complements the sparse literature on evaluations of large-scale nutrition and health programmes, through the ‘adequacy evaluation’ of an NGO-led, large-scale, multi-country, 10-year nutrition and health programme.
  • Post-hoc assessment of data quality indicated most data were of moderate or high quality and therefore suitable for an adequacy evaluation.
  • Results varied by country, with one or more countries achieving improvements in vitamin A and iodine status, decreases in anaemia rates and malaria prevalence, higher rates of EBF and immunization and improved child growth.
  • The magnitude of the nutrition and health impact was often larger than observed in controlled interventions or trials.

Supplementary Material

[Supplementary Data]

Acknowledgements

We acknowledge the programme managers who led implementation of the field programmes: Haile Meskel Balcha, Bernard Kindoli, Gideon Muganda, Rose Namarika and Michael Neequaye. Technical assistance was provided by Dr Zewdie Wolde-Gabriel. The contributions of many other individuals and institutions in supporting the implementation and evaluation of the MICAH programme are also gratefully acknowledged. Sian FitzGerald provided valuable comments on an earlier draft, and Philip Harvey’s insightful review contributed to the completion of the final manuscript.

References

1. Black RE, Allen LH, Bhutta ZA, et al. Maternal and child undernutrition: global and regional exposures and health consequences. Lancet. 2008;371:243–60. [PubMed]
2. Bhutta ZA, Ahmed T, Black RE, et al. What works? Interventions for maternal and child undernutrition and survival. Lancet. 2008;371:417–40. [PubMed]
3. Morris SS, Cogill B, Uauy R. For the Maternal and Child Undernutrition Study Group. Effective international action against undernutrition: why has it proven so difficult and what can be done to accelerate progress? Lancet. 2008;371:608–21. [PubMed]
4. Beaton GH, McCabe GP. Efficacy of Intermittent Iron Supplementation in the Control of Iron Deficiency Anaemia: An Analysis of Experience in Developing Countries. Ottawa: The Micronutrient Initiative; 1999.
5. Beaton GH, Martorell R, Aronson KJ, et al. Effectiveness of Vitamin A Supplementation in the Control of Young Child Morbidity and Mortality in Developing Countries. New York: United Nations; 1993.
6. Hetzel BS, Pandav CS. S.O.S. for a Billion: The Conquest of Iodine Deficiency Disorders. 2nd. Delhi, India: Oxford University Press; 1996.
7. Allen LH, Gillespie SR. What Works? A Review of the Efficacy and Effectiveness of Nutrition Interventions. ACC/SCN, Geneva, in collaboration with the Asian Development Bank, Manila, 2001.
8. Victora CG, Habicht JP, Bryce J. Evidence-based public health: moving beyond randomized trials. Am J Public Health. 2004;94:400–5. [PubMed]
9. Habicht JP, Victora CG, Vaughan JP. Evaluation designs for adequacy, plausibility and probability of public health programme performance and impact. Int J Epidemiol. 1999;28:10–8. [PubMed]
10. UNICEF. Goals for Children and Development in the 1990s. [5 May 2009, date last accessed]. http://www.unicef.org/wsc/goals.htm#Nutrition.
11. World Vision Canada. Mississauga: World Vision Canada; 2002. MICAH Phase I Results: 1995–2001.
12. World Vision Canada. Final Program Report 2006. Mississauga: World Vision Canada; 2006. Improving Nutrition of Women and Children: The MICAH Program.
13. World Vision Canada. Mississauga: World Vision Canada; 2006. MICAH Program Final Evaluation Reports: World Vision Ethiopia, Ghana, Malawi, Tanzania.
14. World Vision Canada. Mississauga: World Vision Canada; 1996. The MICAH Guide: a practical handbook for micronutrient and health programmes.
15. HemoCue® Angelholm, Sweden: [5 May 2009, date last accessed]. Blood Hemoglobin Photometer Operating Manual. http://www.hemocue.com/files/900138_GB.pdf.
16. Hemo-Control (EKF-diagnostic GmBH, Barleben Germany) [5 May 2009, date last accessed]. [cited] http://www.ekf-diagnostic.de/index.php?id=75&L=1&MP=12-273.
17. Dunn JT, Crutchfield HE, Gutekunst R, et al. Methods for Measuring Iodine in Urine. The Netherlands: ICCIDD; 1993.
18. WHO. Geneva: World Health Organization; 1996. Indicators for Assessing Vitamin A Deficiency and their Application in Monitoring and Evaluating Intervention Programmes. WHO/NUT/96.10.
19. World Health Organization. Report of a Joint WHO/UNICEF/ICCIDD Consultation. Geneva: WHO; 1992. Indicators for Assessing Iodine Deficiency Disorders and their Control Programmes. Review version September 1993. WHO/NUT/93.1. editor.
20. McLaren DS, Frigg M. Sight and Life Manual on Vitamin A Deficiency Disorders (VADD). Task Force SIGHT AND LIFE, Basel, Switzerland. 1997.
21. Reilly JT, Bain BJ, Amos R, et al. The laboratory diagnosis of malaria. Clin Lab Haematol. 1997;19:165–70. [PubMed]
22. WHO. WHO Technical Report Series No. 854. Geneva: World Health Organization; 1995. Physical status: the use and interpretation of anthropometry. [PubMed]
23. Mei Z, Grummer-Strawn LM. Standard deviation of anthropometric Z-scores as a data quality assessment tool using the 2006 WHO growth standards: a cross country analysis. Bull World Health Organ. 2007;85:441–48. [PubMed]
24. Cupul-Uicab LA, Gladen BC, Hernandez-Avila M, Longnecker MP. Reliability of reported breastfeeding duration among reproductive-aged women from Mexico. Matern Child Nutr. 2009;5:125–37. [PMC free article] [PubMed]
25. Kramer MS, Chalmers B, Hodnett ED, et al. Promotion of Breastfeeding Intervention Trial (PROBIT): a randomized trial in the Republic of Belarus. JAMA. 2001;285:413–20. [PubMed]
26. Li R, Scanlon KS, Serdula MK. The validity and reliability of maternal recall of breastfeeding practice. Nutr Rev. 2005;63:103–10. [PubMed]
27. Gillespie S, Mason J, Martorell R. How Nutrition Improves. ACC/SCN Nutrition Policy Discussion Paper 15. Geneva: United Nations Administrative Committee on Coordination/Sub-Committee on Nutrition; 1996.
28. Yusuf HK, Rahman AM, Chowdhury FP, et al. Iodine deficiency disorders in Bangladesh, 2004–05: ten years of iodized salt intervention brings remarkable achievement in lowering goitre and iodine deficiency among children and women. Asia Pac J Clin Nutr. 2008;17:620–28. [PubMed]
29. Zhao J, van der Haar F. Progress in salt iodization and improved iodine nutrition in China, 1995–99. Food Nutr Bull. 2004;25:337–43. [PubMed]
30. Viteri FE, Berger J. Importance of pre-pregnancy and pregnancy iron status: can long-term weekly preventive iron and folic acid supplementation achieve desirable and safe status? Nutr Rev. 2005;63:S65–76. [PubMed]
31. Quinn VJ, Guyon AB, Schubert JW, Stone-Jimenez M, Hainsworth MD, Martin LH. Improving breastfeeding practices on a broad scale at the community level: success stories from Africa and Latin America. J Hum Lact. 2005;21:345–54. [PubMed]
32. Bland RM, Rollins NC, Solarsh G, Van den Broeck J, Coovadia HM. Maternal recall of exclusive breast feeding duration. Arch Dis Child. 2003;88:778–83. [PMC free article] [PubMed]
33. Kramer MS, Chalmers B, Hodnett ED, et al. Promotion of breastfeeding intervention trial (PROBIT): a cluster-randomized trial in the Republic of Belarus. Design, follow-up, and data validation. Adv Exp Med Biol. 2000;478:327–45. [PubMed]
34. Jooste PL, Weight MJ, Lombard CJ. Short-term effectiveness of mandatory iodization of table salt, at an elevated iodine concentration, on the iodine and goiter status of schoolchildren with endemic goiter. Am J Clin Nutr. 2000;71:75–80. [PubMed]
35. Rice AL, Stoltzfus RJ, de Francisco A, Kjolhede CL. Evaluation of serum retinol, the modified-relative-dose-response ratio, and breast-milk vitamin A as indicators of response to postpartum maternal vitamin A supplementation. Am J Clin Nutr. 2000;71:799–806. [PubMed]
36. Stoltzfus RJ, Hakimi M, Miller KW, et al. High dose vitamin A supplementation of breast-feeding Indonesian mothers: effects on the vitamin A status of mother and infant. J Nutr. 1993;123:666–75. [PubMed]
37. Assey VD, Greiner T, Mzee RK, et al. Iodine deficiency persists in the Zanzibar Islands of Tanzania. Food Nutr Bull. 2006;27:292–99. [PubMed]
38. Kulwa KB, Kamuzora K, Leo G. Urinary iodine concentration and availability of iodated salt in school children in a goitre-endemic district of Tanzania. East Afr Med J. 2006;83:79–84. [PubMed]
39. Cherinet A, Kelbessa U. Determinants of iodine deficiency in school children in different regions of Ethiopia. East Afr Med J. 2000;77:133–37. [PubMed]
40. Egbuta J, Onyezili F, Vanormelingen K. Impact evaluation of efforts to eliminate iodine deficiency disorders in Nigeria. Public Health Nutr. 2003;6:169–73. [PubMed]
41. Tatala S, Svanberg U, Mduma B. Low dietary iron availability is a major cause of anemia: a nutrition survey in the Lindi District of Tanzania. Am J Clin Nutr. 1998;68:171–78. [PubMed]
42. van den Broek NR, Letsky EA. Etiology of anemia in pregnancy in south Malawi. Am J Clin Nutr. 2000;72:247S–56S. [PubMed]
43. Rogerson SJ, van den Broek NR, Chaluluka E, Qongwane C, Mhango CG, Molyneux ME. Malaria and anemia in antenatal women in Blantyre, Malawi: a twelve-month survey. Am J Trop Med Hyg. 2000;62:335–40. [PubMed]
44. Dreyfuss ML, Stoltzfus RJ, Shrestha JB, et al. Hookworms, malaria and vitamin A deficiency contribute to anemia and iron deficiency among pregnant women in the plains of Nepal. J Nutr. 2000;130:2527–36. [PubMed]
45. Katz J, West KP, Jr, Khatry SK, et al. Impact of vitamin A supplementation on prevalence and incidence of xerophthalmia in Nepal. Invest Ophthalmol Vis Sci. 1995;36:2577–83. [PubMed]
46. Vijayaraghavan K, Radhaiah G, Prakasam BS, Sarma KV, Reddy V. Effect of massive dose vitamin A on morbidity and mortality in Indian children. Lancet. 1990;336:1342–45. [PubMed]
47. Klemm RDW, Villate EE, Tuazon-Lopez C, Ramos AC. Coverage and Impact of Adding Vitamin A Capsule (VAC) Distribution to Annual National Immunisation Day in the Philippines. Manila: Philippines Department of Health and Helen Keller International; 1996.
48. Vinutha B, Mehta MN, Shanbag P. Vitamin a status of pregnant women and effect of post partum vitamin a supplementation. Indian Pediatr. 2000;37:1188–93. [PubMed]
49. Bahl R, Bhandari N, Wahed MA, Kumar GT, Bhan MK. Vitamin A supplementation of women postpartum and of their infants at immunisation alters breast milk retinol and infant vitamin A status. J Nutr. 2002;132:3243–48. [PubMed]
50. Ayaha RA, Mwanikia DL, Magnussena P, et al. The effects of maternal and infant vitamin A supplementation on vitamin A status: a randomised trial in Kenya. Br J Nutr. 2007 ; [E-pub ahead of print: 29 March 2007] [PubMed]
51. Delange F, Burgi H, Chen ZP, Dunn JT. World status of monitoring iodine deficiency disorders control programs. Thyroid. 2002;12:915–24. [PubMed]
52. Untoro J, Schultink W, West CE, Gross R, Hautvast JG. Efficacy of oral iodized peanut oil is greater than that of iodized poppy seed oil among Indonesian schoolchildren. Am J Clin Nutr. 2006;84:1208–14. [PubMed]
53. Berger J, Thanh HT, Cavalli-Sforza T, et al. Community mobilization and social marketing to promote weekly iron-folic acid supplementation in women of reproductive age in Vietnam: impact on anemia and iron status. Nutr Rev. 2005;63(12 Pt 2):S95–108. [PubMed]
54. Vir SC, Singh N, Nigam AK, Jain R. Weekly iron and folic acid supplementation with counseling reduces anemia in adolescent girls: a large-scale effectiveness study in Uttar Pradesh, India. Food Nutr Bull. 2008;29:186–94. [PubMed]
55. Gadallah M, Rady M, Salem B, Aly EM, Anwer W. The effect of nutritional intervention program on the prevalence of anemia among pregnant women in rural areas of Belbis district-Sharkia Governorate-Egypt. J Egypt Public Health Assoc. 2002;77:261–73. [PubMed]
56. Menon P, Ruel MT, Loechl CU, et al. Micronutrient Sprinkles reduce anemia among 9- to 24-mo-old children when delivered through an integrated health and nutrition program in rural Haiti. J Nutr. 2007;137:1023–30. [PubMed]
57. Gamble C, Ekwaru JP, ter Kuile FO. Insecticide-treated nets for preventing malaria in pregnancy. Cochrane Database Syst Rev. 2006 (2):CD003755. [PubMed]
58. Lengeler C. Insecticide-treated bed nets and curtains for preventing malaria. Cochrane Database Syst Rev. 2004 (2):CD000363. [PubMed]
59. Brown KH, Peerson JM, Rivera J, Allen LH. Effect of supplemental zinc on the growth and serum zinc concentrations of prepubertal children: a meta-analysis of randomized controlled trials. Am J Clin Nutr. 2002;75:1062–71. [PubMed]
60. Rivera JA, Sotres-Alvarez D, Habicht JP, Shamah T, Villalpando S. Impact of the Mexican program for education, health, and nutrition (Progresa) on rates of growth and anemia in infants and young children: a randomized effectiveness study. JAMA. 2004;291:2563–70. [PubMed]
61. Ramakrishnan U, Aburto N, McCabe G, Martorell R. Multimicronutrient interventions but not vitamin a or iron interventions alone improve child growth: results of 3 meta-analyses. J Nutr. 2004;134:2592–602. [PubMed]

Articles from International Journal of Epidemiology are provided here courtesy of Oxford University Press