Benefits of the design
This study design has four major benefits. First, the careful selection of a range of qualitative case studies allowed for systematic comparison across the studies. Second, in the qualitative work, the use of participant observation yielded important insights that interview and document review data alone might have missed. Third, the complementarity of qualitative and quantitative methods was very fruitful, as results from our quantitative analysis could be explained by results from qualitative work. Finally, this research provides useful information on how to implement vertical programs in the future in ways that will benefit health systems in the long run.
First, the use of multiple sites, carefully selected to represent a range of experiences with the GPEI, allowed us to distinguish between contextual and universal factors in the relationship between polio eradication, RI, and PHC. Our parallel case studies also allowed us to follow cross-cutting themes, such as cold chain, surveillance, and motivation of workers, across the different field sites, and reach conclusions regarding the role of these factors globally. For example, in a number of case studies—though not all of them—qualitative data supports the hypothesis that polio contributed to the development of cold chain infrastructure in the mid-90s. This was a subtle effect we could not pick up in analysis of the quantitative data available. The fact that it existed across such widely disparate case studies gives us strong grounds to suggest that the effect was widespread though not ubiquitous.
Other factors varied across the case studies, often in patterned ways. For example, in the very weak health systems in our study, polio eradication activities were often the only health activities occurring according to plan—in one case, the gound-level workers were on strike, and yet continued to work on polio vaccination campaigns. In such cases, polio eradication’s supervisory and incentive structure was one of the few motivations strong enough to get staff to work at all. In contrast, in many of the stronger health systems in our study, busy staff across all cadres juggled polio activities along with a variety of other responsibilities. A full understanding of the extent to which polio campaigns might or might not interrupt services must take such factors into account. The important patterned dynamics here—observable across widely disparate case studies in South Asia and Africa—were detected because of our systematic case study selection. This study design, then, allowed us to address general trends, as well as identify best practices in specific contexts that might be adapted for use in other places.
Second, in the qualitative portion of this research, participant observation yielded important insights that interview data alone might have missed. For example, when it came to topics like service interruption during campaigns, direct observation and experience with the dynamics described in the previous paragraph were often more illuminating than interviews, where respondents might be reluctant to speak frankly. By visiting local health posts both during campaign days and outside of campaigns, we were able to gather information on what constituted ‘normal’ health post functioning, and learn how that might or might not change during the campaign.
The third major benefit of the study design is that results from the quantitative analysis could be explained by results from qualitative work. In many cases, our quantitative analysis revealed no significant impact, either positive or negative, of the GPEI on RI or PHC. In these cases, the qualitative case studies often showed a complex mix of positive and negative effects, mediated through different factors, leading to an overall situation that could not be expected to have statistical significance in its impacts.
As an example of just one of many small effects, our qualitative data showed numerous examples of small-scale impact where supervision and monitoring and evaluation were performed at higher levels during the campaigns, and provided models for the health system that could be used for other services. (In many cases, however, they were not.) While too subtle to have a statistically observable impact on, say, DTP3 coverage, these areas of small impact visible through qualitative analysis are very important, and in some cases form the basis for recommendations for improvement.
Thus this research design has utility beyond simply assessing the impact of vertical programs on health systems in the past. Because the qualitative work revealed examples of best practices that could be more widely implemented, it also provides insight on ways that vertical programs like the GPEI could better strengthen health systems in the future. In addition, the rich description of on-the-ground strategies that comparative ethnography provides could be used to better understand how a given vertical program is being carried out in a variety of contexts, and could inform the development of new and improved program design. Further, in addition to providing currently actionable recommendations, the research described here could serve as groundwork for future research and strategic evaluation. For example, the quantitative and qualitative data that was collected in this research project could be used as baseline information to design more complex system dynamics models to test how various strategies might play out in the future [33
As might be expected in carrying out parallel qualitative and quantitative analyses in eight case studies across the developing world, we experienced some challenges. However, we did identify ways to mitigate some of these challenges in the course of doing our research.
One of the major challenges confronted by every large-scale study in the developing world is the accessibility of data. In the quantitative arm of the study, we faced difficulties accessing reliable data for our analyses, particularly on where and when polio campaigns had occurred. In some countries such data did not exist, while in others government officials were reluctant to share them. Some officials may fear that analysis of quantitative data has the potential to reveal inadequacies in health services or otherwise reflect negatively on the health system; as long as this potential exists, officials may quite rationally decide that declining to share data is the safer course. In addition, discrepancies exist on immunization coverage estimates in particular and DHS, while robust, does not triangulate administrative data in proposing estimates. The short timelines allocated for the study, as well as the lack of political will and the need to locate the right partners in each of the study sites, made quantitative data collection extremely challenging—ultimately, despite our best efforts, impossible in a few cases (see Table ).
Availability of key data for quantitative analysis in study countries
The quantitative analyses were further complicated by the nature of polio vaccination campaigns themselves. The targeting of campaigns has changed over time. While early efforts were almost exclusively national in scope, over time, targeting evolved to focus on first level administrative units (states/provinces) and then to second level administrative units (districts). As targeting became more sophisticated, regions with relatively low levels of RI and PHC coverage had the most frequent polio eradication campaigns. This creates a selection bias that makes it very difficult to establish causality between the frequency of polio eradication campaigns and changes in RI and PHC coverage.
To avoid confounders and address selection bias, a potential way forward is for a study to collect new quantitative data on key variable(s) of interest—depending on the vertical program being studied, possibilities include functioning and reach of the cold chain, number of people serviced per day for a specific service (e.g., routine immunization) by type of worker, or availability of key supplies like syringes. Such a study design might also allow the quantitative analysis to pick up small impacts on cold chain or worker time availability that might not be strong enough to affect a downstream indicator like overall DTP3 coverage. Creating such datasets would, of course, require significant time and resources. Ideally, these efforts would be designed into and funded as part of the evaluation process for vertical campaigns.
The collection of qualitative data was challenged by accessibility as well. In-country approval procedures including Institutional Review Boards (IRBs) are continually evolving and becoming more complex, and the steps required are not always clear. While we congratulate the formation of institutional and national IRBs, this change in procedure requires specific attention. In addition, it is increasingly difficult for research projects to compete with donors and programs that bring direct funding and benefits to government officials. Methodological advancement on this front is not simple, and ultimately requires improving relations between country officials and researchers. Before providing approval for a research project that required time on the part of government officials and health workers, we found that government officials reasonably wanted to know how they and their programs would benefit from the research. Therefore, researchers need to be very clear about how their research may be used to the country’s benefit, and transparent about how and with whom the results of the study will be shared.
We recommend assigning a local partner to the position of a full-time in-country point-person. Such a point-person would not be engaged in actual data collection, but would be responsible for nurturing political will, creating partnerships, and leading the study’s interactions with local IRBs. While such tasks were in the past do-able in a short time span, or even from afar, the situation on the ground in most countries today necessitates a full-time staff member to liase with local institutions and policies.
Further, any multi-sited study involving a variety of countries has potential to encounter problems of security and political instability; for example, we faced political unrest in our case studies in Kano, Nigeria and Karachi, Pakistan. Our study’s qualitative data collection team struggled with a constant need to adapt to changing circumstances in order to ensure the personal safety of our staff. We accepted that the standardized research protocol would sometimes have to be modified to meet the exigencies of fieldwork. While ideally we would have collected identical data in all eight case studies, security issues and other extenuating circumstances limited our ability to follow exactly the same guidelines in all of the sites. For example, when our researcher in Karachi, Pakistan received a text message warning her not to go to the study site as she might be targeted, she scaled back her participant observation activities from those outlined in the guide. Such deviations are inevitable in a project committed to gathering data in a range of contexts, not just settings where research is convenient.
Beyond accessibility, in a large research project there is likely to be variability (including among researchers working at the same site) in the experience and skill level of qualitative field researchers. In the course of this research we learned that our more experienced fieldworkers found it easier to adapt our research protocol to local situations. These experienced staff appreciated having the leeway to make their own adaptations to the research tasks, and those of us running the study valued their thoughtful, context-appropriate modifications.
Those research staff with less experience found it more challenging to make necessary modifications to the research guide to tailor it to local conditions. To support their work, we recommend (1) providing samples of appropriate adaptations in the research guide; (2) pairing less experienced researchers with more experienced researchers when possible; and (3) holding a series of orientation sessions for all research staff prior to conducting the research, to discuss questions of appropriate modifications.
We did not hold such orientation sessions, but would do the following in the future: The first orientation session would include the lead researcher for each case study and would be dedicated to refining the research guide and discussing questions regarding the methods to be used in the study. This orientation should be held after a pilot, testing interview questions in the field, in at least one study site. A second pre-fieldwork workshop would be held in-country, and would include the full team of fieldworkers for that country. The in-country workshop would be used to test interview questions in a short country-specific pilot and make the appropriate country-specific adjustments to the study’s methods and tasks prior to fieldwork. Periodic debriefs among field staff over the course of fieldwork, concentrated in the early stages, would ensure that staff had the opportunity to discuss challenges and provide suggestions for appropriate solutions in modifying the guide. We held a workshop of fieldworkers post-fieldwork and found this to be an extremely useful exercise in discussing cross-cutting themes and results.
Finally, in developing a research protocol, we would encourage the oversampling of historically knowledgeable key informants. In many of our case studies, there were too few interviewees with deep historical memory. We found that frequent transfers limited the number of respondents with institutional memory, including locating documents and quantitative data from early years. Those who did have such knowledge were generally the most useful interviewees. As detailed quantitative data from the early years of the GPEI was often unavailable, rich qualitative data illuminating past trends is especially important. Interview protocols should also pay special attention to the need to learn about events in the past.
Flexibility in research across sites includes the ability to make adjustments to the timeline and budget to adapt to changing circumstances. While most public health research is held to short timelines because of the need to make data-driven programmatic decisions, more time allocated to research projects will mean better quality research. Flexibility in budget planning is important as well. The challenges of data accessibility, the need to realign to meet new IRB procedures, and the need to address changes in political circumstances, are easiest to productively address in the context of a flexible timeline and a flexible budget.