Worldwide, administrative data tend to be the primary source used by health information systems to estimate immunization coverage and to guide decision-makers and Expanded Programme on Immunization (EPI) managers in planning and monitoring immunization programmes and assessing their performance. They are also the primary source used by national authorities to fill in the WHO/UNICEF joint reporting forms (JRF) on vaccine-preventable diseases, a major source of information on global coverage and immunization systems performance (WHO 2006
Because resource allocation processes tend to be more intimately linked to systems performance, and because intervention requires estimates of coverage levels, administrative data play a key role in determining the allocation of government and donor-supported initiatives (such as the GAVI Alliance) aimed at strengthening countries’ immunization activities. However, the validity of administrative data has been repeatedly questioned (Crabb 2003
; Murray et al. 2003
; Lim et al. 2008
). The immunization Data Quality Audit (DQA) was developed and implemented with the support of the World Health Organization (WHO) and GAVI to help overcome these limitations and improve the usefulness of information systems for immunization data (WHO 2003
). DQAs help diagnose specific weaknesses in health information systems through systematic process analysis at the national, district and health facility levels. The quality, accuracy, timeliness and completeness of immunization data is audited, and the DQA provides information about which level of the reporting system contributes most to inaccuracies (Ronveaux et al. 2005
). Recommendations for change are made and the quantitative indications of reporting consistency and accuracy provided by DQAs can be further used to monitor the impacts of strategies implemented to improve the quality of health information systems.
While it provides a comprehensive assessment of process quality, as well as evidence on current practices to support quality assurance strategies, the DQA does not provide indications of the validity of existing immunization coverage estimates per se
. For this, administrative-based estimates have to be compared with population-based estimates, as Murray et al.
) did, for example, in 45 countries using the household Demographic and Health Survey (DHS), and Lim et al.
) did more recently using DHSs, Multiple Indicator Cluster Surveys (MICSs) and country-specific surveys. Both studies concluded that administrative data tend to overstate the number of immunized children. Overestimation of immunization coverage by administrative data has also been reported in Burkina Faso (Zuber et al. 2003
) and Cameroon (Guyer and Atangana 1977
), as well as in a sample of case study countries including Uganda, India and the Philippines (UNICEF 1996
), while administrative data have underestimated coverage rates in surveys done in Panama (Huezo et al. 1982
) and Zimbabwe (Borgdorff et al. 1988
The gap thus measured between population-based and administrative sources provides valuable information on the outcomes of processes of information production and management. It is important to realize, however, that the gap observed in any given country is an aggregate that can be represented as a mean estimate of variations observed within regions and health districts. The validity of coverage estimates can, in fact, vary considerably from one district to another. This heterogeneity is well reflected by DQA indicators, which tend to vary substantially between districts, notably in poorly performing countries like Burkina Faso (Bosch-Capblanch et al. 2009
), and national estimates of the verification factors have such wide confidence intervals (CIs) that this alone compromises their usefulness at the subnational level. For example, nothing prevents national estimatesshowing concordance between administrative- and population-based sources from actually masking heterogeneity at the district level, with administrative data sometimes overestimating and sometimes underestimating real coverage.
Because the district is the first level of data aggregation in countries’ national information systems, and because decentralization has given districts a lead role in health planning and management, it is at least as relevant to get evidence on the accuracy of administrative-based estimates at the district level and on inter-district variations, as it is to obtain evidence of overall health information systems performance.
Using an EPI cluster survey conducted in Burkina Faso in 2003, the authors previously showed that the national average hides wide variations between districts in immunization coverage (Bicaba et al. 2009
). A consultation undertaken with EPI officials and district health managers in Burkina Faso revealed substantial underutilization of existing administrative data for planning and monitoring immunization activities, ascribable to shared apprehensions about their quality. District heterogeneity also emerges from a recent comparative study across African countries which demonstrated, among other things, that one-third of the districts had not achieved 50% DTP3 coverage by the end of 2004 (Arevshatian et al. 2007
). Significant heterogeneity in immunization outcomes between states and municipalities has also been reported, advocating convincingly for the need to look beyond the aggregated national or state pictures (Pande and Yazbeck 2003
; Gaudin and Yazbeck 2006
However, neither the research we have been able to consult to date nor the available literature allows us to document variations in the quality of administrative-based estimates of immunization coverage across districts. This can be explained by the limitations of available surveys. The DHSs used by Murray et al.
) or Zuber et al.
), for example, only make it possible to provide regional estimates, at best. Another probable explanation is the intense attention focused by national decision-makers and cooperation agencies on national aggregates, which are more directly applicable and more immediately convincing than are measurements that take into account territorial distribution.
This article aims to provide evidence on the validity of administrative-based estimates of immunization coverage across 52 health districts in Burkina Faso and their usability for planning and monitoring immunization activities at the district level. The analysis is followed by a preliminary exploration of factors that might account for inter-district differences in the significance and size of inconsistencies observed between administrative-based and population-based estimates. The article takes advantage of the existence of a cluster survey carried out nationwide in 2003 using the standard EPI methodology. Contrary to the DHS, whose sampling design only allows regional-level estimates, the EPI cluster survey (ECS) provides district-level representative estimates, thus offering a unique opportunity for assessing the quality of administrative coverage data beyond the national average.