|Home | About | Journals | Submit | Contact Us | Français|
The escalating HIV/AIDS epidemic worldwide demands that on-going prevention efforts be strengthened, disseminated, and scaled-up. System-level interventions refer to programs aiming to improve the functioning of an agency as well as the delivery of its services to the community. System-level interventions are a promising approach to HIV/AIDS prevention because they focus on (a) improving the agency’s ability to adopt evidence-based HIV prevention and care programs; (b) develop and establish policies and procedures that maximize the sustainability of on-going prevention and care efforts; and (c) improve decision-making processes such as incorporating the needs of communities into their tailored services. We reviewed studies focusing on system-level interventions by searching multiple electronic abstracting indices, including PsycInfo, PubMed, and ProQuest. Twenty-three studies out of 624 peer-reviewed studies (published from January 1985 to February 2007) met study criteria. Most of the studies focused on strengthening agency infrastructure, while other studies included collaborative partnerships and technical assistance programs. Our findings suggest that system-level interventions are promising in strengthening HIV/AIDS prevention and treatment efforts. Based on our findings, we propose recommendations for future work in developing and evaluating system-level interventions.
Researchers’ understanding of how HIV/AIDS is transmitted, prevented, and managed has increased substantially over the past twenty years. Researchers and practitioners have developed clinical and behavioral strategies that seek to decrease HIV transmission and enhance HIV-related quality of life. Multiple literature reviews have found solid evidence examplifying effective prevention strategies, including community needle exchange programs (Bastos & Strathdee, 2000), community education and advocacy (Janz, Zimmerman, Wren, Israel, Freudenberg, & Carter, 1996), linkage to antiretroviral therapy to decrease mother-to-child transmission (Volmink, Siegfried, van der Merwe, & Brocklehurt, 2007), distribution and use of the female condom (Hoffman, Exner, Leu, Ehrhardt, & Stein, 2003), and behavioral interventions to modify sex risk behaviors (Johnson, Hedges & Díaz, 2007; Lyles, Kay, Crepaz, Herbst, Passin, Kim et al., 2007). These comprehensive literature reviews have evidenced that prevention is effective, yet these efforts are sometimes labor intensive and slow. To strengthen on-going prevention efforts through dissemination and scale-up, we must ensure that the capacity and resources of the agencies carrying out these efforts are maximized.
A social system refers to the network of institutional structures providing standardized and quality-assured services to large numbers of people and communities. These institutional networks may include federal agencies (e.g., CDC), local government agencies (e.g., city health departments), non-government organizations (NGOs) (e.g., AIDS service organizations [ASOs] and community-based organizations [CBOs]), and private health organizations (e.g., hospitals and healthcare providers). Delivery of HIV prevention through these institutions has been an efficient rollout mechanism because they may (a) reach and affect large numbers of people efficiently; (b) create and establish policies and procedures that maximize the diffusion of interventions; (c) increase program sustainability; and (d) incorporate the needs of specific communities into their tailored services. Nonetheless, we know very little in the HIV literature of how strengthening these social structures, individually and collectively, may ameliorate the provision of HIV prevention programs. System-level interventions are a promising approach to HIV/AIDS prevention because they focus on (a) improving the agency’s ability to adopt evidence-based HIV prevention and care programs; (b) develop and establish policies and procedures that maximize the sustainability of on-going prevention and care efforts; and (c) improve decision-making processes such as incorporating the needs of communities into their tailored services.
While most psychosocial interventions have focused on delivering individual and community programs, structural interventions have received little attention until recently (Blankenship, Bray, & Merson, 2000). We define system-level interventions as a set of strategies used to modify an agency’s culture, its internal process to preserve and maximize resources, and the efforts placed in strengthening external ties with other institutions for HIV prevention and care. An underlying assumption of this approach is that an agency’s capacity to deliver strong HIV prevention messages across different levels is tied intricately to the stability and effectiveness of its internal processes. Consequently, a system-level intervention focuses on strategies that promote institutional growth, capacity, and sustainability rather than on the mechanisms used to disseminate a program into the community.
The distinction between system-level strategies and dissemination mechanisms is vital in understanding the impact of system-level interventions. A system-level intervention may have as one of its outcomes the successful scale-up and dissemination of a program, yet it may not be the only outcomes of interest. For example, a CBO conducts a system-level intervention when it seeks to strengthen its staff skills through a technical assistance program or when it restructures its organizational hierarchy by including different community stakeholders in its Executive Board to increase diversity in its decision-making process.
A variety of theoretical frameworks and outcomes may inform system-level interventions. To this end, our goal is to review HIV/AIDS system-level interventions published in scientific journals from January 1, 1985 to February 1, 2007. In summarizing the findings by intervention strategy (i.e., technical assistance, infrastructure development, and inter-agency partnerships), we hope to illustrate different approaches and identify their strengths and weaknesses to inform future studies. We conclude with a summary discussion of intervention efforts demonstrating potential success or needing further replication and propose recommendations for future work
Theoretical frameworks should inform system-level interventions to facilitate program rationale and increase transferability of lessons learned when successful. Various theoretical approaches have been proposed to guide system-level intervention development, but their use has been limited in the HIV literature. We add to the basic assumption that, at a minimum, three components take place in creating and delivering services (Alter & Hage, 1993). These three components are: (1) inputs (internal and external resources that an agency can access), (2) process (an agency’s culture and its decision-making procedures), and (3) outputs (an agency’s aptitude to deliver services). We present an overview of common theoretical frameworks used to intervene in these three steps.
STOC proposes a staged framework through which organizations incorporate innovations. Similar to the Transtheoretical Model (Prochaska et al., 2002), an organization’s ability to move through the stages towards adoption, implementation, and maintenance of an innovation depends on the institution’s ability to overcome external and internal barriers within each stage. Beyer and Trice (1978) propose seven stages: (1) sensing of unsatisfied demands on the system, (2) search for possible responses, (3) evaluation of alternatives, (4) decision to adopt a course of action, (5) initiation of action within the system, (6) implementation of the change, and (7) institutionalization of the change. Goodman and Steckler (1989) have argued for an eighth stage, renewal, to account for the program’s ability to evolve as demands change.
Rooted in management theory and influenced by social ecology models, ODT posits that environmental influences (e.g., an organization’s norms and values) can shape organizational performance by influencing the individual and collective attitudes and behaviors of an organization’s staff (Steckler, Goodman, & Kegler, 2002). Three inter-related constructs inform interventions based on this theory: (a) organizational climate, (b) organizational culture, and (c) organizational capacity. Organizational climate can be defined as the institution’s personality and is caused and continuously shaped by the interaction of members’ collective beliefs, perceptions, and attitudes (Hoy & Miskell, 1987). Organizational culture refers to the underlying assumptions, values and norms that grow and persist as an institution evolves (Bolman & Deal, 2003). Finally, organizational capacity is characterized by an institution’s ability to create deliverables, to maintain the organization in operation, to respond to and receive support from the social and political demands in the environment, to adapt to changes within and outside of the institution, and to synchronize these competing demands appropriately (Katz & Kahn, 1978; Dill, 1994).
IOR focuses on how multiple organizations work together to maximize deliverables within their communities (Steckler et al., 2002). IOR posits that partnerships across organizations occur in order to tackle comprehensively the social, political, and economic demands affecting their communities. Interorganizational linkages facilitate access to new information, skills and resources, while sharing costs and responsibilities. Conversely, interorganizational linkages may delay action steps due to slow consensus development, may cause conflict if a policy or position taken by the whole is incongruent with the goals or mission of one or more of the organizations, and may divert much needed organizational resources to the collective mission of the coalition.
Based on these frameworks, several strategies may be proposed as system-level intervention activities. We catalogue these strategies into three domains: (a) technical assistance, (b) infrastructure development, and (c) inter-agency partnerships and coalition building.
System-level interventions may use technical assistance strategies to improve the delivery of services by intervening on an agency’s input component. Technical assistance may include agency staff training in the program planning process, consultation with experts in program planning and implementation, or external support in monitoring and evaluation procedures. Within HIV/AIDS, researchers and practitioners have identified evidence-based prevention strategies and programs (National Institute of Mental Health Multisite HIV Prevention Trial Group, 1998), efforts to prepare system-wide dissemination of these efforts have relied on STOC and ODT strategies. Examples of STOC and ODT strategies include the training in conducting a problem diagnosis like the one carried out in HIV Prevention Community Planning Groups (CPGs; Johnson-Masotti et al., 2000). Similarly, another example is the provision of technical assistance to agencies as they select, translate, and implement a Diffusion of Effective Behavioral Intervention (DEBI) to address community needs.
System-level interventions may seek to build an institution’s capacity by increasing an agency’s resource availability and improving its inputs. However, infrastructure development may also focus on an agency’s process component by reorganizing its internal functioning. Agency reorganization may include a shift in the agency’s mission and goals, an increase in fund-raising activities to increase personnel retention as well as resources for new and ongoing services, and/or the reorganization of an agency’s leadership and decision-making process. Within HIV/AIDS, an example of a system-level strategy that may be informed by STOC and ODT would be the development and refinement of a clinic’s goals and objectives for a high-risk group, and the monitoring of quality of care standards to ensure optimal treatment of HIV-positive patients. Another strategy might include the rearranging of an ASO’s organizational structure to diversify staff tasks, increasing volunteer manpower and ability to secure funding, or the training of staff to strengthen their ability to translate evidence-based interventions into community-relevant HIV prevention programs.
System-level HIV/AIDS strategies informed by IOR rely on the development of new or existing collaborations across a social system as a way of strengthening an agency’s outputs. These strategies can include partnerships of a particular kind of institution (e.g., a coalition formed exclusively by ASOs in a defined region) or a variety of stakeholders from various kinds of institutions seeking to minimize duplicate efforts in service provision across agencies (e.g., representatives from ASOs, CBOs, religious leaders, policy-makers, hospital managers). Potential activities include the development of coalition of ASOs and CBOs to increase outreach and testing efforts in their communities, the formation of a service bundle to provide greater comprehensive care to high-risk populations such as injection drug users, and the expanded access to resources and lessons learned, locally and at a distance, through electronic networks such as listservs and newsgroups.
Markers of “success” can vary according to the strategies proposed as part of a system-level intervention and can range across different levels of analysis (e.g., individual, agency, and city/region) and units of analysis (e.g., patient, care provider, geospatial unit). We present briefly an overview of potential outcome indicators for system-level interventions and categorize them as (a) institutional-based outcomes and (b) HIV-related markers.
System-level evaluation plans include an agency’s internal (“inputs”) markers, external (“outputs”) markers, or both, to monitor change in organizational efficiency and productivity. A capacity building initiative may focus, for example, on improving access to care by optimizing the organizational efficiency of a hospital. Internal agency markers include indicators such as changes in team composition and responsibilities within a medical team; creation and buy-in of group rules and values among staff; ensuring transparency in agency decision-making procedures by recording the number of staff meetings and the quality of feedback from clients and personnel; developing systems to monitor quality of care; and increasing agency funding and revenue.
Similarly, system-level programs may focus on improving organizational productivity. DEBIs, for example, seek to maximize the reach and impact of prevention programs across ASOs, CBOs, and health departments by identifying evidence-based behavioral strategies that are transferable to different communities. External agency outputs may include indicators such as the number of people reached within a catchment area; number of cross-agency collaborations; the quality of services, workshops and trainings offered; the proportion of evidence-based programs and services successfully meeting the needs of the community; and, the number of counseling and outreach services sponsored by the agency, among others.
As part of a system-level intervention, changes can occur at the individual level of analysis. System-level evaluation plans may include provider-specific measures to assess changes in provider attitudes and behaviors. A technical assistance program to facilitate the scale-up of voluntary counseling and testing (VCT) in a health clinic, for example, may include intervention components seeking to increase providers’ knowledge and attitudes regarding HIV counseling and testing, maximize their perceived job satisfaction, their perceived comfort towards service delivery, and enhance their skills to deliver counseling to clients and provide social support, among others. Similarly, individual-level changes can be evaluated among clients receiving services before and after a system-level intervention. Increases in the number of services offered in health clinic, for example, may increase clients’ intention to seek medical care, facilitate centralization of multiple health services within a single setting to optimize adherence, offer a greater number of opportunities for health education, and improve the quality of life of the patient and their social support networks.
System-level interventions can modify the epidemiologic characteristics of a disease at the regional and population levels. An agency strengthened by a systems-level initiative such as its capacity building to disseminate a DEBI may implement a primary prevention intervention that may decrease the incidence of HIV infection across a region. Similarly, restructuring a hospital’s initiative to increase the time spent in quality patient-provider interactions may increase the access and adherence to medical care that may result in the decrease of a county’s AIDS-related mortality rate and improve the quality of life of people living with HIV/AIDS.
Measurement of population health markers such as HIV/AIDS incidence and prevalence may be important to test the reach of a system-level intervention at a population-level. Nonetheless, these estimates may change slowly, additional epidemiologic markers such as quality of life indicators (QALYs/DALYs) and adherence rates should be included into system-level evaluations.
We identified an inclusive list of search terms for studies that would be added to an electronic database for cataloguing: community, network, social, system, structural, intervention, program(me), and HIV/AIDS. To maximize the search, each term was assessed individually (i.e., structural & HIV) or in conjunction with other terms (i.e., structural intervention & AIDS) using Boolean expressions.
We approached the review process from two directions: (1) an inductive systematic collection of all potentially relevant articles found through computerized searches and (2) a deductive approach by reviewing the references of identified publications. The literature search included iterative searches of MEDLINE, Pub Med, PsycInfo, and The Cochrane Library. In addition, retrieved articles’ bibliographies were examined for any additional relevant intervention studies. To maximize the search, the research team placed no initial restrictions on publication year, population, or explicit mention of psychosocial factors associated with HIV prevention. Any publication written in English and related to HIV prevention, related HIV-risk behaviors, or potential predictors of these behaviors in the social sciences, public health, medicine, social work and human organization fields were eligible for inclusion.
We had a three-step filtering of the identified citations. First, we identified 4,129 citations published in over 100 journals between January 1, 1985 and February 1, 2007 using the generated list of keywords. From the citation abstracts, we excluded 3,339 studies because they were not written in English, solely detailed medical procedures, focused solely on HIV/AIDS prevalence or incidence surveillance statistics, and/or presented the results of predictor studies. An additional 166 entries were excluded because they were duplicates.
Second, we refined the search by reading each citation’s abstract and kept citations that offered any description of a potential system-level effect (n= 624 citations). Where relevant, we also included studies describing the implementation of an intervention. To be conservative, we included a citation if the abstract did not include sufficient information to determine if there a systems-level effect (i.e., did not include an agency-level or provider-level outcome). After filtering all non-relevant citations, we identified 104 potential citations for review. We were unable to find the full-text of sixteen of these studies.
Finally, we developed additional criteria for inclusion in the review. We included in this review interventions focusing on reducing HIV risk through technical assistance, scale-up and roll-out programs, or through optimizing agency capacity of hospitals, community-based organizations (CBOs) and AIDS service organizations (ASOs), and government institutions. To be considered for review, studies had to have been published in the scientific literature between January 1985 and February 2007. Only studies between 1995 and 2007 met review criteria. This period also reflects the onset of ARV scale-up and rollout. Following Exner et al. (1997), we focused on studies designed specifically to address HIV/AIDS and with available process and outcome evaluation data. We did not restrict our review to a subset of intervention outcomes to capture the scope and breadth of eligible systems-level intervention outcomes.
We acknowledge that many systems-level interventions may be missing from our review because they are implemented by agencies (i.e., UNAIDS, FHI, WHO, World Bank, Global Fund) that disseminate their program findings through non-peer-reviewed journals (for a recent review of these initiatives, see Druce et al., 2006). Furthermore, while the reach and importance of health policies as system-level interventions should not be undermined, we chose to exclude policy interventions from our review because they require different analytic considerations (Cason et al., 2002; Lazzarini & Klitzman, 2002; Weimer & Vining, 1999). Policy analysis requires an analytic approach that takes into account the political process, the social action movements lobbying for changes in government regulations. While influencing system-level initiatives, policy interventions are beyond (more “upstream”) the scope of system-level interventions.
Out of 624 citations, we reviewed 26 reports meeting study criteria (i.e., interventions focusing on reducing HIV risk through technical assistance , infrastructure development, scale-up and roll-out programs, or through optimizing partnerships between hospitals, community-based organizations (CBOs) and AIDS service organizations (ASOs), and government institutions). While most studies reported more than one intervention strategy, the most common strategies were infrastructure development (17 studies; 74%), followed by collaborative partnerships across agencies to provide bundled services (12 studies; 53%), and technical assistance for agencies trying to develop and implement new interventions (9 studies; 43%). We found, however, that the choice of a particular strategy varied by location (see Table 1). Among studies based in the United States, infrastructure development was the strategy cited most often, followed by technical assistance programs and external partnerships. Among international studies, infrastructure development was the strategy cited most often, followed by external partnerships and technical assistance programs.
Four research designs repeatedly emerged in the studies reviewed: a pretest-posttest design with one group (11 studies; 48%) or two groups (2 studies; 9%), a one-shot case study (5 studies; 22%), a randomized control trial (4 studies; 17%), and a one-group time series design (1 study; 4%). Below, we summarize our findings by cataloguing studies across intervention strategies: (a) technical assistance, (b) infrastructure development, or (c) external partnerships. Within each strategy, we identify common intervention components and comment on selection of outcomes to test intervention effects.
Nine studies reported technical assistance as a component of their intervention (see Table 2). Over 75% of all technical assistance strategies included a component aiming to improve skills in program planning (7 studies) and intervention implementation (7 studies), respectively. Two studies included technical assistance in community outreach and service delivery. Technical assistance components were found more often in the U.S. than in international settings, particularly within NGO and government institutions.
Interventions using technical assistance components presented internal (e.g., improved in agency performance) and external (e.g., proportion of agencies adopting a new program) agency indicators among their primary outcomes. Only one study using technical assistance reported a positive increase in agency staff behavior change. Three studies reported improved outcomes among community members; two studies reported increased HIV prevention behavior change among clients while another reported decreased HIV/STI prevalence. All studies reported positive effects from baseline to follow-up in their primary outcomes.
Taken together, these studies suggest that technical assistance is a promising system-level strategy. Nonetheless, the impact of these interventions was variable. In particular, technical assistance components were given to a large number of agencies yet a smaller proportion adopted these interventions. Harshbarger et al. (2006), for example, reported that staff members from 229 agencies were training in the VOICES/VOCES intervention. At the three-month follow-up, more than a third of the agencies had decided to forgo implementation. Another third decided to implement the intervention with modifications, with no consideration as to how it would affect the intervention’s success. On the other hand, the studies by Kelly et al. (2000, 2006) deserve particular mention because they identify critical intervention components when delivering technical assistance to NGOs. In their randomized control trial comparing different teaching components within a national technical assistance program in the United States, Kelly et al. (2000) found that technical assistance efforts may have the strongest effect when multiple teaching components (i.e., intervention manuals, staff training, and follow-up consultation calls) are delivered as a bundle. In a multi-site international study, Kelly et al. (2004, 2006) found NGOs in the treatment condition (e.g., receiving distance education and technology transfer through the telephone, emails, or instant messaging) were more likely to implement a program than control NGOs. These findings are promising and highlight that system-level interventions including technical assistance components should carefully consider the inclusion of different types of learning materials as well as the integration of multi-communication modalities.
Seventeen studies included infrastructure development components (see Table 2). The most common intervention strategies included skill building activities (8 studies; 47%) such as increasing patient-provider competency and efficient HIV prevention outreach strategies, followed by organizational restructuring (7 studies; 41%) such as the creation of institutional decision-making groups, and access to material resources (6 studies; 35%) such as computers and condoms. Only one study by Takahashi and colleagues (2007) discussed the access to seed-money for trainings and workshops. Infrastructure development components were found more often in the U.S. than in international settings (see Table 1).
Many of the studies including infrastructure development reported agency-level outcomes. Five studies reported internal agency-level outcomes. Four of these studies reported increased internal agency functioning; one study reported decreases in internal agency functioning due to a decrease in number of volunteers as greater number of paid positions were created in the agency. Three studies reported increased external agency outcomes such as an increase in number of programs implemented by the agency. Infrastructure development programs also included positive increases in agency staff’s skills and knowledge (n=2). Five studies found increases in client health behavior (e.g., integrating HIV care to antenatal care). Two studies found decreases in STI/HIV prevalence and incidence markers.
Most studies reported positive effects across their primary outcome measures. Overall, these studies suggest that system-level initiatives including infrastructure development components can help increase agency functioning and HIV prevention outcomes. The study by Morrill et al. (2005) deserves particular mention as an interesting case study on how the Massachusetts Prevention Planning Group restructured their internal processes to improve its decision-making process. Equally important were the findings on provider-communication skill building activities as an infrastructure development initiative to improve client outputs. Bluespruce et al. (2001), for example, found that training health care providers increased their self-efficacy to counsel patients on risky sexual practices as well as their ability to integrate sexual health questions into routine check-ups. Furthermore, Grosskurth et al. (1995) found that training health care providers in clinical examinations for STD-related symptoms, combined with community mobilization, decreased the incidence of HIV among clinic attendees. Taken together, these studies suggest that system-level interventions using infrastructure development components may benefit by activities that aid refine an institution’s internal processes (e.g., organization stability and community-input in the decision-making process) while also paying attention to the training needs of agency staff.
Twelve studies included external partnerships (see Table 2). The most common intervention strategies included coalition building across community stakeholders (5 studies; 42%), followed by community mobilization (4 studies; 33%) and service bundling (4 studies; 33%). External partnership programs were based in the United States (see Table 1).
Studies including external partnerships reported increases in agency-level internal (n=4) and external (n=2) outcomes, respectively. Six studies measured changes in clients’ behaviors. Five of these studies found positive increases in clients’ behaviors. One study with an external partnership component informed decreases in HIV/STI prevalence and incidence markers.
The bundling of services, per se, is variable and may not always lead to clients’ use of services. Blank et al. (2005), for example, found that only 27% of all MSM attending an event in a non-traditional health venue accessed one or more services. This suggests that the availability of bundled health services may result in poor client uptake if the program setting is not considered carefully. On the other hand, given the increasing needs in bundling services and maximizing HIV prevention and care efforts, it is encouraging to find most system-level interventions with external partnership components reported positive effects.
Overall, our findings suggest that system-level initiatives are a promising modality if we are to maintain or increase institutional capacity to provide HIV prevention services and rollout evidence-based interventions. From our review, we found three overarching themes that may strengthen system-level strategies. Within technical assistance, Kelly et al.’s (2000, 2004, 2006) work suggests that multiple highly-active education components in technical stategy are most effective. Particularly in a global world, using new technologies for distance learning and consultation seems promising and increases iterative consultations. Within infrastructure development, restructuring agency procedures to enhance internal staff communication and feedback seems vital. Furthermore, staff-training initiatives in interpersonal communication and public health skills seem promising as a mechanism to strengthen agency productivity and stability. External partnership strategies linking existing resources into bundles were found to be effective. In an era of scare funds for new services, the impact of service linkage is particularly heartening.
Reported intervention efforts varied by geographic locale, highlighting different epidemiologic priorities in the HIV epidemic. Interventions in the U.S., for example, commonly focused on strengthening the capacity of agencies working with subgroups at greatest risk for HIV infection (e.g., men who have sex with men and racial/ethnic minority women) in the U.S. System-level efforts in the U.S. focused on infrastructure development (e.g., organizational restructuring of community planning groups, or the creation of a database to monitor quality of care in a hospital) and external partnerships (e.g., coalition building among community stakeholders interested in a particular population group). These strategies reflect the need to strengthen the individual and collective resources of U.S.-based agencies in order to deliver, implement and tailor evidence-based HIV prevention education and treatment.
International locations with high HIV prevalence and incidence, on the other hand, sought to deliver HIV prevention and treatment services to the population in a defined catchment area. Most system-level interventions focused on infrastructure development (e.g., access to material resources for day-to-day functioning such as computers and funding) and external partnerships (e.g., the bundling of reproductive and HIV prevention services within a clinic). It is important to note that the selection of infrastructure development and external partnerships may reflect the need for prompt HIV prevention and treatment rollout in regions with high infection rates and mitigating social and economic vulnerabilities. Strikingly, very few of the studies meeting eligibility criteria focused on international settings.
Effective programs, both U.S.-based and international, commonly incorporated multiple intervention strategies. The presence of multiple strategies in an intervention limits our ability to understand the efficacy of the intervention components independently. While this is a common critique of public health programs (commonly referred to as an ‘intervention black box’), our findings suggest that, similar to individual and community prevention programs, system-level interventions require multiple strategies in order to support an agency’s ‘behavior change’ during service delivery, optimization and maintenance. Below, we propose several recommendations for future work in HIV/AIDS system-level programs.
Efforts to curtail the spread of HIV/AIDS, locally and globally, have highlighted the importance of identifying contexts and populations and tailoring theoretical constructs and strategies to improve program effectiveness. Unfortunately, although perhaps implicit, none of the reports reviewed mentioned explicitly the theoretical framework informing their system-level program. Moreover, the ecological nature of system-level interventions requires that these be matched with evidence-based community-level and individual-level health education programs. Consequently, peer-reviewed journals should allocate publication space so authors may include richer descriptions of the system-level constructs informing their program. These descriptions will facilitate program replication, translation, and adaptation, and, avoid wasting material and human resources, especially in communities where these resources are severely strained. Similarly, formative work is required prior to the implementation of a system-level intervention and the importance of this should not be trivialized (Jenkins & Carey, 2005). Researchers and practitioners’ experiences in developing and implementing system-level interventions are needed in the scientific literature. Without an understanding of the needs and resources required during the formative phase, replication of a system-level intervention may be difficult.
Stronger evaluation components are required in system-level intervention programs. The acknowledgement that system-level programs are context-specific highlights the importance of developing strong causal linkages between the intervention components and the desired outcomes (internal validity) and undermines the usefulness of randomization for generalizability (external validity). While some would argue that a randomized control trial design is the “gold standard” within system-level interventions, these designs may not be feasible because of pragmatic considerations such as increased costs, ethical concerns in randomizing systems, identifying and selecting characteristics for randomization and comparison of trials arms, among others (Rapkin & Trickett, 2005). Borrowing from lessons learned from community-level interventions (Shinn, 1990), system-level interventions should include elements that, when possible, strengthen the system-level intervention’s internal validity (i.e., having multiple groups, having different operationalizations of the outcome, alternating or reverting treatment conditions across groups, offering delayed treatment to a comparison group, and including multiple observations before and after treatment).
Quantitative and qualititative system-level process and outcome indicators deserve greater attention in the literature (Lipsey & Cordray, 2000). Careful programmatic and data monitoring procedures should be in place to detect potential biases such as measurement corruptibility (Rossi, 1997). When possible, success indicators should encompass multiple levels of analysis (i.e., agency outcomes, staff outcomes, and HIV markers).
Data linking system, community, and individual-level outcomes should be explored further in intervention evaluation. Nested data allow the testing of cross-level effects that, until recently, were disheartening because they provided biased estimates and violated the assumption of independence in procedures such as ANCOVA and linear regression (Raudenbush & Bryk, 2002). New statistical advances in this area, however, have overcome these issues and allowed for computation of nested models through packages such as Mplus and HLM. While a detailed review of this analytic technique is beyond the scope of this paper (see Bingenheimer & Raudenbush (2004) for an introduction to multilevel modeling in health), we provide several examples to illustrate the potential power of this technique. Examples of multi-level models within system-level programs may include: (1) exploring the effects of an agency capacity-building program (i.e., increases in funding) on the characteristics of the communities (i.e., community capacity) these intend to serve; (2) assessing the effects of a training for physicians (i.e., provision of social support toward HIV+ clients) on patients’ behavior (i.e., antiretroviral adherence); or (3) appraising how external partnerships increase the diffusion of an evidence-based intervention across multiple agencies and impact a region’s HIV/AIDS prevalence and incidence.
We acknowledge that our review has several limitations. First, our findings may be subject to publication bias given that unpublished studies offering weak or null effects are not reported as often in the scientific literature. Second, our criteria for peer-reviewed journal publications excluded system-level interventions implemented by international agencies (i.e., UNAIDS, FHI, WHO, World Bank) or health policies that are system-level interventions (i.e., needle exchange programs). While the bulk of system-level interventions tend to be carried out by these international agencies, we chose to exclude these interventions from our analysis as a recent paper by Druce et al. (2006) addressed these iniatives. Nonetheless, our review does include multiple system-level interventions within non-government agency settings. Furthermore, as the demand for peer-reviewed evidence-based findings continues to increase, we present the evidence in peer-reviewed scholarly journals in hopes that their evidence informs on-going programmatic efforts. Nonetheless, future studies reviewing system-level programs implemented by international agencies is useful. Finally, the variety in outcome operationalization and intervention strategies did not allow us to perform a meta-analysis. As more HIV system-level interventions are published with similar outcomes, however, we hope that measuring effect sizes across different system-level intervention strategies will be possible.
These limitations notwithstanding, our literature review of the peer-reviewed evidence on system-level interventions suggests that they are effective and varied. Future research focusing on expanding/refining system-level theoretical frameworks and sharing lessons learned from implementing these initiatives are needed. On-going prevention and treatment efforts can incorporate system-level interventions to strengthen their efforts from a top-down approach that may lead to strengthening evidence-based community, behavioral, and clinical HIV prevention and care efforts.
Special thanks to Drs. Alex Carballo-Diéguez, Jenny Higgins, Shari Dworkin, Robert H. Remien, Theresa Exner, and Ray Smith for their suggestions and contributions to earlier versions of this manuscript. This research was supported by a center grant from the National Institute of Mental Health to the HIV Center for Clinical and Behavioral Studies at New York State Psychiatric Institute and Columbia University (P30-MH43520; Principal Investigator: Anke A. Ehrhardt, Ph.D.) and a training grant from the National Institute of Mental Health (T32 MH19139 Behavioral Sciences Research in HIV Infection; Principal Investigator: Anke A. Ehrhardt, Ph.D.).