This study was approved by the Institutional Review Board (IRB) of the Mozambique National Institute of Health. Co-investigators also discussed the study with the IRB at the UW, which determined that the UW IRB review was not necessary because the primary intent of the project was for program evaluation purposes, and therefore not considered human subjects research under United States federal regulations.
Study setting and sites
Sofala Province, with a population of approximately 1.6 million, is similar to Mozambique as a whole in terms of high disease burden (23% HIV prevalence) and limited availability of health workers (three medical doctors and 21 nurses per 100,000 inhabitants) [10
]. Sofala Province is traversed by major highways connecting its provincial capital city, Beira, with Manica Province and Zimbabwe to the west and smaller roads connecting to Zambezia Province in the north and to Inhambane Province to the south. Sofala was heavily affected during the 20-year civil war, which decimated the health infrastructure. HAI, a nongovernmental organization (NGO), is a main external support organization for the health sector in the Province. HAI has over 20 years of experience working with the Mozambican MOH and began working in Sofala Province in 1995.
In Sofala Province, there are a total of 137 primary-level health facilities (12 urban health centers, 10 rural health centers type I, 89 rural health centers type II, and 26 health posts), four secondary-level health facilities (rural hospitals), and one quaternary health facility (central hospital). There is only one notable private clinic in Sofala Province, located in the provincial capital (Beira), which provides a nonconsequential level of formal health services (estimated to be less than 1% of facility-assisted births and outpatient consults in the entire province, as reported by the HIS).
For this study, we assessed HIS data quality in nine government-run health facilities within three districts (Beira City, Dondo, and Caia) of Sofala Province in central Mozambique. The researchers employed a purposive sampling approach, selecting three focus districts based on their representativeness of the three main types of districts in Sofala Province--urban, peri-urban, and rural--and because of their accessibility to the researchers. In Beira City, three large urban health centers were selected as demonstrative of sites where most people receive PHC services. The largest hospital in the city (Beira Central Hospital) was excluded because it does not provide core PHC services, such as immunization and antenatal care services. The health facilities chosen from the other two districts were selected to represent a broad range of PHC facilities in those locales, and included from each district: 1) a large district hospital/health center, 2) a medium-sized health center outside the district capital, and 3) a small rural health center/health post.
Mozambique's National Health Service (NHS) is the key organizational unit through which PHC services are managed, coordinated, and brought to scale. Since its inception in 1975, the NHS has been designed to provide integrated primary health care services through a widespread network of health facilities distributed across the country. Chronic resource shortages, vertical funding from donors, and management challenges all limit service coverage and quality. The Mozambique health system is currently undergoing a decentralization process, which will further devolve important management and planning tasks from the provincial to the district level. Unfortunately, most district health directorates remain under-resourced with limited technical, managerial, and workforce capacity to assume these devolved responsibilities. District management is further hampered by a combination of relatively weak data collection systems and low district capacity to analyze data for district-level decision-making and planning.
The Mozambique HIS is similar to the data systems of other countries in the region, incorporating both paper and electronic elements, depending on the health system level. Patient-level information is collected daily in facility paper-based clinical registries, typically by nurses within health facilities (Figure ). On a monthly basis, key indicators are culled from the facility clinical registries and aggregated into monthly facility reports, which are sent on to the district level. In most cases, the district planning and statistics department enters the monthly facility reports into the MOH electronic database, known as the "Basic Module," and the electronic files are sent to the provincial level, where they are aggregated and forwarded to the national level via email or on flash drives.
Mozambique health information system flow map.
Data collection and statistics
For the purposes of this activity, we used a methodology similar to the Global Fund to Fight AIDS, Tuberculosis and Malaria (GFATM) on-site data verification bottom-up audit tool [11
] to assess the primary health care HIS, which included the following components (see Table ):
Population and patient data sources of information
1. Verification of the availability of monthly facility reports at the health facility and district health departments
2. Evaluation of the reliability (concordance) of monthly statistics obtained from facility clinical registries, monthly facility reports, and the MOH electronic database
3. Examination of the validity of the HIS data by comparison with population-level surveys over time
The GFATM data verification tool includes the first and second components but not the third, which was added by the researchers as a means to assess the validity of the routine HIS data compared to population-level surveys that are powered for provincial-level analysis.
Another important difference between the GFATM data verification tool and our methodology was the selection of indicators representative of PHC services, rather than indicators from the GFATM focus diseases (HIV, tuberculosis, and malaria). However, we used the GFATM data verification rating system to classify the concordance of the reliability indicators according to the following scale:
A - less than 10% error margin
B1 - between 10% to 20% error margin
B2 - above 20% error margin
C - no systems in place
All data were collected by provincial level health managers and NGO counterparts based at the MOH's Beira Operations Research Center (BORC). This applied research center is in no way involved with the management of the health system, nor in the delivery of health services, and thus its employees are uniquely poised to evaluate how the health system operates. In addition, the training provided to the data collection staff also mitigated potential bias in data collection. It should be noted that district and facility staff did help in providing the data registries and monthly reports but were not involved in collecting data.
To assess the availability of monthly health facility and district reports for immunization and maternity services, we noted the monthly presence or absence of these two reports over a 12-month period (November 1, 2007 through October 31, 2008) at both health facilities and district health departments (Table ). Ideally, paper copies of the monthly facility reports should be organized chronologically in binders at the health facility, and original reports should be sent to the district health department offices, where paper copies are kept and filed after they are entered into the MOH electronic database. Provincial-level health managers and NGO counterparts from the BORC worked together to collect this information over a one-week period.
To assess data reliability, we recorded monthly figures from facility clinical registries and monthly facility reports for five key indicators (first antenatal care [ANC1], institutional birth, DPT3, HIV testing, and outpatient consults) for the six-month period from June through December 2008, which was deemed adequate to gain a sufficient picture of data reliability over time (Table ). We then calculated the proportion of facility-months in which the data were not identical and calculated the percentage difference for those months where data were not identical.
We also compared recorded monthly figures from aggregated district reports and the MOH electronic database (obtained at the provincial level) for immunization and maternity services for a 12-month period from November 2007 to October 2008. Seven indicators from the immunization report were included: target group, first tetanus dose, second to fifth tetanus dose, total tetanus doses, coverage rate/goal, doses administered, and wastage rate. Eight indicators from the maternity reports were included: institutional births, live births, low-weight births, stillbirths, stillbirths with heartbeat upon hospital admission, maternal deaths, discharges, and total inpatient days. We calculated the proportion of district months where data were not identical between the district reports and the MOH electronic database and the percentage difference for those months when data were not identical. Data collection for assessing data reliability was carried out by MOH provincial level health managers and NGO counterparts over a period of two weeks.
To assess data validity, we compared statistics supplied from the provincial health department's annual reports (which use data from the MOH electronic database) with those obtained from the 1997 and 2003 Mozambique Demographic Health Surveys (DHS) and the 2008 Multiple Indicator Cluster Survey (MICS). The DHS and MICS surveys were national-level community-based surveys, carried out in Mozambique following standardized and internationally-recognized methodology (Table ).[12
For this analysis, we focused on three provincial-level indicators that were representative of PHC and obtainable from all three aforementioned sources: ANC coverage, institutional birth coverage, and DPT3 coverage. For the surveys, ANC coverage was defined as births that received ANC from a trained health professional during pregnancy, institutional birth coverage was defined as births that occurred in a health facility, and DPT3 coverage as the proportion of children 12 to 23 months old who had received three DPT immunizations (from documentation on immunization cards or mothers' reports) at the time of the survey. Due to data constraints for the variable indicating site of delivery in the MICS dataset, we estimated institutional birth coverage from this dataset using a different question ascertaining whether births were attended by a doctor, nurse, or midwife. In the DHS, the concordance between these two items was very good (33.9% vs. 34.0% in the 1997 DHS and 57.4% vs. 56.1% in the 2003 DHS, for births in a health facility versus births attended by a doctor, nurse, or midwife, respectively), justifying its use in the 2007 MICS dataset.
For ANC and institutional birth coverage, we harmonized the DHS and MICS by including responses from women who had given birth in the previous two years before the survey (1995-1996 for the 1997 DHS, 2001-2002 for the 2003 DHS, and 2006-2007 for the 2008 MICS). We estimated DPT3 coverage up to two (1997 DHS) and four (2003 DHS and 2008 MICS) years prior to the survey. We utilized the responses recorded for children aged 12-23 months, 24-35 months, 36-47 months, and 48-59 months to estimate coverage for 1 year, 2 years, 3 years, and 4 years prior to the survey, respectively.
MOH statistics were collected from the Sofala provincial health department annual reports, which were derived from the MOH electronic database for the numerators, and census data for the denominators. DHS and MICS metrics were obtained from the original DHS and MICS datasets from http://www.measuredhs.com
and the Mozambican National Institute of Statistics, respectively. For all DHS and MICS analyses, we used only data from Sofala Province to obtain provincial-level estimates.
Secular trends of HIS data derived from the MOH electronic database and the community level surveys (DHS and MICS) with 95% confidence intervals were plotted. Note that HIS data do not have calculable confidence intervals, as they are not estimates derived from sampling but rather represent the entirety of provincial-level data. To evaluate the degree of correlation between the two data sources, we first determined the proportion of HIS annual estimates that were within the 95% confidence intervals of the corresponding DHS/MICS estimate. Next, we determined the degree of correlation between the annual HIS and corresponding DHS/MICS figures, using a weighted linear regression model with the survey estimate as the dependent variable and the HIS data as the independent variable. The weights were inversely proportional to the variance of the survey estimates. The correlation coefficient was determined by taking the square root of the R-squared estimate. For ANC coverage and institutional birth delivery, we compared the 1995-1996, 2001-2002, and 2006-2007 survey estimates to the mean of the administrative data over these two years, while the yearly DPT3 estimates were compared to the administrative estimate for the corresponding year.
All simple descriptive statistics were carried out using Stata version 11.1 (College Station, Texas, USA).