To illustrate the effective use of LQAS to assess performance on malaria outcome indicators based on data collected as part of the MIS, we applied this method to an existing dataset to show that one can assess sub-national performance. Analysing MIS data from Mozambique (2007) with the LQAS method, we determine which EAs of the MIS sample are performing adequately based on 70% bednet and ITN coverage targets. We find variation in the performance of EAs that is masked by a single point estimate for the province. Because of the limited resources available to the Ministry of Health for malaria prevention, this information on local-level performance could be used to target areas with severely inadequate coverage for intensive interventions. For example, based on the LQAS classifications, we identified the 266 of the 345 lowest performing EAs. This result suggests that substantial behaviour change and communication interventions may be required to promote the use of bednets, or more intensive distribution campaigns may be needed. Classifying an area on multiple indicators refines the analysis and suggests types of interventions needed. Of the 79 EAs classified as high with respect to bednet ownership, 73 of them were classified as low with respect to ITN ownership. These areas that exhibit high bednet ownership but low ITN ownership could be targeted for specialized interventions, such as those focusing on ITN impregnation and retreatment, or distribution of long-lasting insecticide-impregnated nets (LLIN). The LQAS classification did not isolate many areas in the high category for the ITN coverage indicator because the performance in all provinces for this indicator was quite low. However, as malaria programmes strengthen and go to scale, more areas will achieve this standard, increasing the utility of LQAS to identify and focus programmes on priority actions.
Beyond this specific application, this article illustrates the feasibility of incorporating LQAS into multistage cluster sample surveys by emulating an LQAS analysis of local-level data collected for a Mozambican MIS. Previous implementation of LQAS for local programme monitoring has demonstrated that this tool can be used frequently, rapidly and cost-effectively to provide information for allocating resources.7,9,11,22
Since LQAS is used primarily to classify areas as ‘high’ or ‘low’ (rather than estimating the exact proportion of the population covered for each local area), the method does not require large samples to maintain excellent accuracy and its results lead directly to public health action. Additionally, the simplicity of the LQAS tool empowers local programme managers to implement the classification schemes with little additional training. These favourable attributes encourage the use of LQAS for local malaria programme M&E. Despite these benefits, to our knowledge, most examples of LQAS applications only aggregate LQAS data if information is collected in ‘all’ lots or areas. Such aggregation is also feasible when a sample of the lots is selected, so long as they are sampled in a probabilistic way.23
Recent M&E activities have created precedent for taking samples of lots rather than requiring that all be sampled.23–25
Although the regional and national point estimates reported in the Mozambique MIS are extremely valuable for country level and international planning and monitoring, the report does not provide means to assess the variability among the EAs by reporting design effects or confidence intervals.17
Failure to report these measures of variability is a common lapse in final reports from many large surveys. Application of LQAS classifications to the data collected in multistage surveys provides a means for understanding the level of local variation around regional estimates. Furthermore, these classifications not only link directly to action, so that managers realize when substantial variability exists and can strategize effective and localized responses, but they can also be combined, as shown, to provide the same national- and regional-level information provided by the multistage surveys.
Since this analysis was conducted using a pre-existing dataset, certain limitations are inherent in our study. For example, EAs included in the MIS sample were not aligned with specific health districts in Mozambique, and, therefore, the local-level results may not directly relate to operations because the results do not correspond to a specific local manager’s supervision area. Although the EA may represent a village within a supervision area, and may be akin to a sentinel site, the results do not reflect the performance of the entire supervision area as a whole. However, this is a solvable problem and suggests how future MIS and other macro surveys can be designed to more effectively inform about district or sub-district variation. Furthermore, the target sample size in the rural areas (15 households per EA) was too small to select decision rules with a total classification error < 0.20.
In the future, all these limitations can be addressed in the design, such as described by the Large Country Lot Quality Assurance Sampling (LCLQAS) protocol.23
Specifically, implementation of multistage cluster sampling and LQAS should consider the following in the design: first, the clusters (or EA) should be aligned with programmatically relevant supervision areas. This ensures that recommended responses based on the LQAS classifications link directly to a programme area. Secondly, the within-cluster (or EA) sample sizes should be fixed across clusters, which allows for one set of decision rules to ease training of programme managers in applying LQAS to their supervision areas. Thirdly, these fixed sample sizes must be large enough to meet the constraints for determining the decision rules—for example, we were unable to meet our target of overall classification error <20% for sample sizes less than 17. A sample size of 19 at the EA level is sufficient to maintain the error level <20% when looking at a 30% difference in programme performance (pU
= 0.30); a smaller difference or smaller error rate requires a larger sample size.
Clearly, the design of the Mozambique MIS did not sample all EAs, and, as a result, we only have information on 345 EAs. Therefore, not all areas can be classified and prioritized for intervention according to their classification. Methods of intervention in these areas without data are too numerous to cover here. Ultimately, programmes should aim to implement these surveys on an ongoing, rolling basis, so that each area has an opportunity for a localized assessment, strengthening programme response.