None of the identified studies explicitly stated that the studies were originally designed to capture or to assess health system effects (positive or negative). Only seemingly anecdotal evidence could be extracted from the included studies. Scientifically sound, high-quality research must be conducted before generalizations can be made on the negative (or positive) health system impacts of Global Fund investments.
In view of the absence of experimental studies directly assessing health system effects, the strength of our approach was that we were not limited to a particular study design. Our search strategy was sensitive for detecting the ‘Global Fund’ regardless of the actual projects and interventions, but was limited to studies making formal reference to the Global Fund in the published articles. Some potentially relevant studies may not have been identified, if the published articles did not make a reference to the Global Fund. As we were unable to estimate studies that might have been missed due to lack of referencing to the Global Fund, the representativeness of our sample in relation to all interventions remains unknown.
Our search strategy, however, enabled us to identify all studies that explicitly contribute to the debate on the health systems effects of Global Fund financing of disease-targeted programmes in low- and middle-income countries. Given that none of the included studies were explicitly designed to study health system effects and that there are no uniform guidelines for reporting health system effects, some authors of the original papers may have omitted reporting relevant health system effects alongside their results.
The assessment of study eligibility was often complicated because the authors of the identified articles did not use consistent approaches when referring to the Global Fund. For example, the Global Fund was often indicated to support national programmes, but the link between Global Fund-supported national programmes and the interventions described in the study was not always clearly established. In some cases, the reference to the Global Fund could have easily been omitted or replaced with some other donor organization. Some authors referred to the Global Fund financing of the interventions studied in the acknowledgements section, but not in other parts of the article such as in the introduction or methods, with many studies making the connection between the Global Fund, the interventions described in the study and the relevance to health systems in the discussion sections of the studies. Several discussions had to take place at this stage to clarify decisions to reach a transparent agreement between the review authors – a process, which undeniably involved a certain level of subjectivity by the review authors when determining eligibility. Assessment of eligibility was also significantly affected by the generally suboptimal quality of reporting in the screened studies.
Several studies, both quantitative and qualitative, omitted significant parts of describing data and methods that would have facilitated the assessment of eligibility. Given the methodological challenges faced and the certain level of subjectivity involved in assessing eligibility, it is worth considering potential effects of reviewer bias. The field of evidence synthesis addressing complex adaptive systems, such as health systems, is still in its infancy, and therefore reviewers are forced to make subjective decisions. We aimed to control this current methodological shortcoming by transparently describing each step of the review process and stating our rationale for all decisions so that potential sources of bias would be visible to the reader. Furthermore, the purpose of this review was to assess the current evidence base specifically in relation to type and quality of evidence. Our main results and conclusions are therefore related to general principles of scientific quality, and are thus less affected by subjectivity.
Studies addressing health system effects of the Global Fund investments have been published after the literature search of this review was conducted in 2009.47–51
Due to financial resource restrictions, we were not able to extend the analysis to cover years after 2009. Including more recent evidence into this review would undeniably add to the overall picture provided by this review, particularly in relation to observed health system effects, but it would not change the findings and conclusions on the evidence generation during the period studied.
Evidence on negative health system effects
None of the identified studies published between 2002 and 2009 explicitly and rigorously assessed effects of funding by the Global Fund on health systems. The evidence on effects of funding by the Global Fund currently arises from study designs with higher levels of uncertainty in relation to causality and potential sources of bias. Current discourses around GHIs, including the Global Fund, seem to form a significant part in generating the evidence on the potential negative effects of disease-specific programmes. In line with the previous major reviews,5,9
much of the current debate also specifically around the Global Fund was found to be based on anecdotal evidence and assumptions of perceived negative effects of disease-specific programmes in general.
The review shows the considerable gap between the optimal study designs and the actual study methods used to analyse health system effects of Global Fund investments. The use of anecdotal evidence is undeniably important in some situations, for example when drawing attention to potential adverse effects. But the persistent use and generation of anecdotal evidence when evaluating health system impacts is not scientifically justifiable. More importantly, in situations where anecdotal evidence is the only evidence, it should always be accompanied with careful and critical break down and assessment of attribution. This was not found to be the case in the studies we reviewed.
Compared with the evidence-base for effective health interventions, the current evidence-base for effective implementation of inherently complex health system interventions is very weak,8,52,53
requiring high-quality monitoring and evaluation as well as rigorously designed and executed studies to address the evidence gap – given the quantum of investment by the Global Fund which is essentially funded by tax payers of donor countries.
Limited theoretical understanding of models of causality at health system level further handicaps efforts to establish plausible or probable relationships between interventions targeted to individual health system components and system-level impacts. The lack of rigorous scientific evidence, however, complicates the assessment of observed health system impact and restricts conclusions that could be drawn on system level performance from information derived from lower levels (e.g. individual health system building blocks).
A recent comprehensive assessment on Global Fund's health impact (Global Fund 5-year evaluation) showed that evaluating health system effects at country level faces significant methodological challenges and problems, e.g. in terms of data availability and quality.54
Strengthening country health information systems is therefore a prerequisite in improving evidence base through high-quality research.