|Home | About | Journals | Submit | Contact Us | Français|
Communities face challenges achieving positive outcomes using evidence-based prevention programs. The knowledge, attitudes, and skills needed to address these challenges constitutes community-level capacity for prevention. Policy makers are exploring how to build this type of capacity for thousands of organizations or communities implementing prevention across the United States. This article reports on a formative evaluation of an internet-based system designed to build capacity on a large scale called interactive Getting To Outcomes (iGTO). We assessed iGTO within the substance use prevention systems in two states. In Tennessee, there were 15 matched pairs of county coalitions that were randomly assigned to receive either the iGTO system or no intervention. In Missouri, we compared 18 coalitions receiving iGTO to a comparison group of 8 like coalitions. The primary outcome of the the study was iGTO's impact on the performance of prevention by the programs in participating coalitions, assessed through interviews using a structured rating scale at baseline and after a year of iGTO implementation. Analyses suggest that programs using iGTO demonstrated an increase in the quality of performance of key prevention practices over non-iGTO programs. We also assessed barriers and facilitators to the diffusion of iGTO with semi-structured interviews of iGTO users and state level stakeholders. Results show that iGTO was diffused into most iGTO-assigned coalitions, although it was mostly being used in an elementary fashion. iGTO users' experience with and perceptions of the iGTO system were mixed. The findings suggest that to achieve more comprehensive and long-term integration into regular operations likely requires that state leadership also use iGTO themselves and provide more support for its use at the local level.
Communities often face enormous challenges achieving positive outcomes using evidence-based prevention programs (Chinman et al., 2005; Wandersman & Florin, 2003). The knowledge, attitudes, and skills needed to address these challenges constitutes community-level capacity for prevention. This capacity then influences how well programs are implemented, which, in turn, influences the degree to which positive outcomes are achieved (Chinman et al., 2005). A major challenge for policy makers is how to build capacity of thousands of organizations or communities implementing prevention across the United States. This article reports on the formative evaluation of an internet-based system that has the potential to advance the field of prevention by building this kind of capacity on a large scale.
To date, many capacity-building efforts have focused on information dissemination through written materials and websites. Also, the federal government and national non-profits (e.g., Community Anti-Drug Coalitions of America) offer some training and technical assistance. However, these efforts have not had the desired impacts nationwide (Chinman et al., 2005). More recently, researchers have developed “Prevention Support System” models (Wandersman et al., 2008) in which communities are provided a great deal of in-person assistance and some funding (over and above what is usually available) to improve their capacity to implement and monitor evidence-based prevention programming. Two such models—PROSPER (Spoth et al., 2004) and Communities That Care (Feinberg et al., 2007)—involve expert staff helping communities to form coalitions and start evidence-based programs. Another model, Getting To Outcomes™ (GTO, Chinman, Imm, & Wandersman, 2004), aims to build organizations' capacity to provide effective prevention programs using a 10-step planning, implementation, evaluation, and quality improvement model. Early evaluations of these models show impacts on drug use (Spoth et al., 2007), risk factors for drug use (Feinberg et al., 2007), and the quality of prevention performance (Chinman et al., 2008).
For several reasons, these models could extend their reach dramatically through the use of the internet. First, written material could be delivered to users only when it was needed. This information could easily be modified without needing reprinting. Second, some assistance offered could be automated—e.g., how to write a realistic and measurable program objective. Third, users could be guided through steps that follow the logic of how programs ought to be planned and implemented (e.g., develop goals before planning an evaluation). Fourth, information generated from completing certain tasks (e.g., goals, objectives, plans) could be saved, shared among multiple users, and used for both local and funder reports. Fifth, this system could use information entered into one part of the system (e.g., objectives) in another (e.g., evaluation that uses the objectives), making work more efficient. Finally, such a system would be able to track, store, and report a great deal of information about program implementation (attendance, fidelity, satisfaction, activities delivered) and participant outcomes. Current examples of this type of system include the Community ToolBox (http://ctb.ku.edu/en/Default.htm) and SAMHSA's Prevention Platform (https://preventionplatform.samhsa.gov).
Started in 1994, the Community Tool Box (CTB) is a free web site provided by the University of Kansas that provides skill-building information on community development and program implementation organized into 46 chapters across 13 categories. “The Tool Box provides over 7,000 pages of practical information to support work in promoting community health and development (Community Tool Box, 2009).” In addition to its text pages, key features of the website include (1) downloadable Powerpoint slide sets and a series of summary checklists; (2) a section that contains outlines, examples, and how-tos across many content domains; (3) use of their 12 part logic model of community change, which provides links to competencies and an evidence base for each part of the model; (4) a troubleshooting guide that offers solutions to common problems; and (5) an online tool that allows users to ask questions and get responses from CTB advisors. Evaluations of the Community Tool Box have tracked web site usage (e.g., over 1,068,878 user sessions in 2006) and conducted surveys and focus groups of its users, which has been used to improve the web site.
Organized around SAMHSA's five-step Strategic Prevention Framework, the SAMHSA Prevention Platform Web site provides interactive tools for a variety of prevention professionals and community groups. The tools guide users through the five SPF steps—assessment, capacity, planning, implementation, evaluation—allowing users to create and save data about prevention projects. The Platform also has a library of measurements and instruments that can be searched and/or browsed, a Service and Activity Tracking Tool which allows users to enter data about the prevention services performed and produce reports, and a Readiness Tool which allows users to assess and improve their readiness to implement prevention programs. Currently, there are few users of the SAMHSA Prevention Platform, and its status as a viable system that receives ongoing updates and support is unclear.
Both systems have helpful features. Neither have been evaluated to assess ther impact on the quality with which communities perform various tasks associated with high quality prevention—what we call the performance of prevention.
This article addresses the need to better understand how internet-based systems can improve the quality of prevention on a large scale. As such, this article discusses the formative evaluation of an internet system that was developed based on Getting To Outcomes, called interactive Getting To Outcomes or iGTO. iGTO has significant potential because it uses a web platform, is based on the GTO logic model that guides users to engage in effective prevention practices (Chinman et al., 2004; Wandersman, Imm, Chinman, & Kaftarian, 2000) and has an emerging evidence base (Chinman et al., 2008).
Getting To Outcomes (GTO) was developed to build prevention capacity to improve the performance of key prevention practices (e.g., choosing evidence-based practices; and planning, implementing, evaluating, and sustaining those practices). GTO does this by posing ten questions (see Table 1) that must be addressed in order to obtain positive results and then provides practitioners with the guidance necessary to answer those questions with quality. Each question is linked to a specific step in the GTO process, six for planning, including the use of evidence-based strategies (steps 1-6), two for process and outcome evaluation (steps 7-8), and two steps on the use of data to improve and sustain programs (steps 9-10). This GTO process is facilitated by the GTO intervention, which traditionally has three components: the GTO manual of text and tools published by the RAND Corporation, Getting to Outcomes 2004: Promoting Accountability Through Methods and Tools for Planning, Implementation, and Evaluation (Chinman et al., 2004), face-to-face training, and on-site technical assistance (TA).
The theory of GTO is detailed in Wandersman et al. (2000). In brief, GTO is an operationalization of Empowerment Evaluation theory, which states there will be a greater probability of achieving positive results when evaluators collaborate with program implementers and provide them with both the tools and the opportunities to engage in accountability activities that are associated with quality prevention, such as conducting process and outcome evaluation and using the information to inform decision making (Fetterman & Wandersman, 2005). iGTO is designed to provide additional support and guidance to communities to perform those accountability activities with greater quality.
We had combined goals of (a) evaluating the impact of iGTO on the programs' prevention performance over a year's time (defined by how well the programs were rated to perform key prevention activities such as needs assessment, planning and evaluation), and (b) documenting the use of the iGTO system among coalitions conducting substance abuse programs in two state prevention systems (Missouri and Tennessee), including the extent to which iGTO was diffused into the operations of these programs. In this project, “program” is used broadly to encompass the staff of traditional substance abuse prevention programs and environmental strategies—policies and practices that aim to prevent underage drinking mostly by focusing on restricting access to alcohol. As in many states, the community-based coalition is the operating infrastructure that houses the staff of these substance abuse prevention program in these states.
We employed randomized and quasi-experimental designs to assess the effects of iGTO and elements of the Continuous Quality Improvement Design (CQI-Design; Rapkin & Trickett, 2005) to modify and improve the system within the participating sites. The CQI-Design is one of a group of designs called “comprehensive dynamic trials”, which have been suggested as alternatives to Randomized Control Trials (RCTs). Rapkin and Trickett point out that RCTs have several limitations including the discounting of individual choice in choosing experimental conditions and the inability to utilize participant input and feedback to make intervention improvements midstream. Strict adherence to a RCT or even quasi-experimental design could greatly limit ecological validity (given that participants would likely choose to participate in iGTO in the “real world”) and discourage local input and midcourse improvement, a central feature of how iGTO is implemented. The power of the CQI-Design is that it utilizes multiple sources of data to indicate performance quality, has local participants and researchers review those data to collaboratively make improvements, and then repeats that cycle with the improved intervention. This hybrid design was used to answer two questions: (1) can an internet system such as iGTO improve the level of performance of coalitions' prevention programs and (2) what lessons from the implementation process and midcourse corrections can be applied to future demonstrations of systems like iGTO?
Figure 1 shows the straightforward study logic model. All programs enter the study at a certain level of prevention performance. We expected that, over time, programs assigned to iGTO would show greater improvement in prevention performance than those that were not assigned to receive iGTO. The Figure also shows the various measures we used to document the level of prevention performance and iGTO utilization, all of which are described below.
The study sites were community-based substance abuse prevention coalitions located in Missouri and Tennessee that were part of those states' Strategic Prevention Framework State Incentive Grant (SPF-SIG), funded by the Substance Abuse and Mental Health Services Administration's Center for Substance Abuse Prevention (CSAP). CSAP gives SPF-SIG grants to states for delivering and sustaining services that prevent substance abuse and underage and risky drinking. CSAP funds empower state officials to grant the funds to community based prevention coalitions, while playing a supportive and monitoring role.
In MO, it was not possible to use random assignment because state officials wanted all funded coalitions to receive iGTO. Therefore, there were 18 iGTO coalitions and 8 comparison coalitions. All iGTO coalitions had successfully applied for SPF-SIG funding. All comparison coalitions either did not apply for funding or failed to receive funding and were recruited by state substance abuse prevention officials. iGTO coalitions operated 36 individual programs, 13 (36%) of which were listed on the National Registry of Effective Prevention Practices (NREPP). Types of evidenced-based programs or strategies were Alcohol Awareness and Enforcement, Drug Free Communities, and Those who Host Lose the Most. Programs not listed on NREPP were similar to the evidence-based programs in their aims and scope, but had not been formally evaluated. All programs from SPF-SIG-funded coalitions were expected to use iGTO. The 8 comparison coalitions operated 9 individual programs and none of those were listed on NREPP.
In MO, iGTO coalitions had an average budget of $34,869, (SD=$63,132), comparison coalitions had an average budget of $18,931 (SD=$17,193). Number of Full-time employees averaged 1.12 (SD=1.50) for the iGTO group and 1.66 (SD=.50) for the comparison group. The iGTO and comparison coalitions were not statistically significantly different in terms of budget [t(42)=1.33, p=.19] and employees [t(43)=-1.06, p=.30], but these comparisons may have been hampered by a large amount of variation in the budgets. Demographic data was only available for the iGTO coalitions. For MO, 17% of coalition staff who used iGTO were Male and 83% were Female; 59% were a coalition project director; 10% were coalition evaluator; 14% were prevention staff; 14 % were other. Approximately 43% had a graduate degree; 21% had some college; 29% had a college degree, and 7% had some graduate education. About 90% were white (non-Hispanic), 7% were Asian-Pacific Islander, and 3% were Black. The average duration of use of iGTO for this group was 11 months. The iGTO group had about six years experience in substance abuse prevention (SD= 6.74).
In TN, there was a total of 30 community-based prevention coalitions—15 iGTO and 15 comparison—all of which were SPF-SIG-funded. These 30 were paired based on a number of characteristics—baseline levels of alcohol consequences in the county served by the coalition, consumption of alcohol and other drugs in the county, capacity, coalition staffing, and county population size (details of the matching and all the variables are available upon request). The pairing of the TN coalitions was achieved by using Mahalanobis Distance metric matching (Rubin, 1980), in which Mahalanobis distance was calculated from the scores from the various data sources discussed above. The 30 counties were sorted by their Mahalanobis distance scores, two-by-two pairs were selected, and then one member of each pair was randomly assigned to the intervention (Cohort 1) or comparison group (Cohort 2).
At baseline, Cohort 1 coalitions contributed 33 individual programs, 16 of which (48%) were listed on NREPP. At baseline, Cohort 2 coalitions contributed 21 individual programs and two were listed on NREPP. Examples of NREPP programs were Life Skills, Communities Mobilizing for Change, and Too Good for Drugs. Similar to MO, the non evidence-based programs were similar in scope and aim but had not been formally evaluated.
The iGTO coalitions had an average budget of $15,714 (SD=20,920), comparison coalitions had an average budget of $23,430 (SD=$55,759). Number of Full-time employees averaged 0.28 (SD=.33) for iGTO group and 1.32 (SD=2.34) for the comparison group. The iGTO and comparison coalitions were not significantly different in terms of budget [t(26)=-.35, p=.72] or employees [t(26)=-1.15, p=.26). Demographic data was only available for the iGTO coalitions. For TN, 40% of coalition staff who used iGTO were Male and 60% were Female; 40% were a coalition project director; and 33% were coalition evaluator. Approximately 47% had a graduate degree, 27% had a college degree and 20% had some college; about 87% were white (non-Hispanic), while 13% were Black. The average duration of use of iGTO for this group was 10 months. The iGTO group reported about six years (SD=5.64) of experience in substance abuse prevention.
iGTO is a web application that allows users to accomplish many tasks online based on the logic of the GTO model. On the website, each of GTO's 10 steps has its own set of pages, beginning with a landing page that provides an overview of that step. Depending on the step, users are prompted to enter various types of information into the system (e.g., goals and objectives in Step 2, expected outputs in Step 6). Reports and data extracts allow users to organize the information entered and share it with other stakeholders. Online help is available from all pages and is tailored to the particular step being used. This help is a mix of technical guidance on how to use iGTO and content help based on two GTO manuals: Preventing Underage Drinking: Using the SAMHSA Strategic Prevention Framework and the Getting to Outcomes Model to Achieve Results (Imm et al., 2007) and Getting to Outcomes 2004: Promoting Accountability Through Methods and Tools for Planning, and Implementation, and Evaluation (Chinman, Imm, & Wandersman, 2004). iGTO also includes coalition management tools that help users track membership, subcommittees, and meetings. iGTO balances data security with data accessibility by allowing state level users to view data for the coalitions they fund, while providing coalition level users access only to information regarding their own organization. To support the use of iGTO, training and ongoing technical assistance was provided to users in both states and is described in detail below.
In both MO and TN, two full-days of training was provided to all iGTO coalitions at the start of the project. In day one, staff from the iGTO coalitions were trained in the use of the GTO model. This training was based on the training curriculum developed for the two GTO manuals mentioned above. The training provided opportunities to learn more about the 10-step GTO model, to gain practice with GTO tools, and to work interactively with examples in relevant content areas (e.g., prevention, coalition building, etc). The training was critical to ensuring that staff understood the logic and theory underlying the GTO model.
The second full day of training was on the use of the GTO web-based application, and was provided by Kit Solutions, LLC staff. All participants had access to a computer during the training. Kit staff provided an overview of the layout, logic, navigation, and functionality of iGTO and then guided participants through each module of iGTO using a prepared training scenario. In that scenario, participants were taught to add, edit, delete and organize records in each module and review the online help and knowledge base information. Also, multiple online trainings were held throughout the field testing with various subgroups of iGTO users. These trainings varied, focusing on specific skills on which different groups of coalitions requested assistance. Coalitions participated in the trainings from their computer and communicated with Kit staff through a conference phone line.
As part of the iGTO system, technical assistance (TA) was provided by designated staff members in each state. Each state's method is described below. In MO, the TA team consisted of two SPF-SIG staff members from the state and a TA provider from the iGTO Team. All were PhD level individuals with strong prevention and evaluation backgrounds. Each coalition met with the three-member TA team at least once a month for an hour using the same protocol: TA providers would query the coalition representatives about their progress on their plans, their implementation, and their use of iGTO. They were also asked about their plans for the following month. The TA provided was based on answers to these questions. Most of the meeting focused on prevention and SPF-SIG implementation issues. TA for iGTO was given about 5 to 10 minutes during this session. Any follow-up questions about iGTO were handled by the iGTO TA provider by phone or in person, as the TA provider also visited each coalition at least once. As the SPF-SIG and iGTO project continued, the protocol was updated to reflect current deliverables of the grantees.
TA in TN began in April 2007, three months after the iGTO group training in Nashville conducted by Kit Solutions, LLC staff. TA in TN consisted of a PhD level individual with a strong prevention and evaluation background discussing with coalition staff how to use all of the various modules of iGTO. The TA provider began with the set of management modules (e.g., Community Mobilization, Administration) rather than with the GTO steps themselves. For example, in the Community Mobilization module, the TA provider went over how to use four sub-modules: “Coalition/Partnership Member,” “Coalition/Partnership Subcommittee,” “Coalition/Partnership & Staff Meetings,” and “Training and Technical Assistance.” If coalitions had not entered any names and contact information in the first module, coalition staff were instructed to do so during a TA session. After those modules, the TA provider then discussed the “Data Sources” and “Assessment Summary” under the “Needs/Assessment” iGTO modules, then to “Goals” and “Evidence-Based Practices” modules, and then the “Planning” and “Implementation” modules. During May, June and July, TA was provided on-site to personnel in 12 of the 15 coalitions in Cohort 1. By January 2008, the entire cohort had completed SPF step products approved by the state and the TA provider had visited all 15 coalitions at least once and responded to telephone and email questions about iGTO.
Consistent with the CQI-Design, throughout the study we gathered feedback from the coalitions through site visits, technical assistance calls, e-mails and phone calls about modifications to iGTO that were needed. This information was sent to KIT Solutions, LLC. To date the following modifications were made: (a) fixed bugs causing technical difficulties during use, (b) added a Quarterly Report function, (c) added a Frequently Asked Questions page within the Knowledge Base (Library), (d) added a custom program name field for programs not included in the evidence based strategies, and (e) added a statement on the data entry page about how iGTO is compliant with HIPAA regulations.
This study's outcome—practitioners' performance of key planning, implementation, and evaluation activities at the program level—was assessed using the Innovation Configuration Map or IC Map (Hall & Hord, 1987; Hall & Hord, 2006). Although programs consist of individual people with varying levels of ability, we made performance ratings at the program level since the program operates as a gestalt. Based on the idea that innovations are often implemented differently than intended, “IC Maps” are a framework that can be tailored to evaluate the quality of use of any innovation (Hall & Hord 1987; 2006). In a previous project (Chinman et al., 2008), we adapted the IC Map to create the GTO-IC Map. The GTO-IC Map has 10 items (called “components”) tied to each of the ten steps of the GTO model. Components are rated based on information collected through a semi-structured interview. Each component had seven possible response choices that include the ideal performance of prevention practices targeted by GTO and six other possible variations ranging from “highly faithful to” to “highly divergent from” what is specified in the GTO model (see Table 2). Each component's seven response choices are also accompanied by descriptions of observable behaviors specific to GTO. For example, for Outcome Evaluation, the highly faithful description was: “Synthesized review and making strategic decisions based upon all outcomes evaluation results for outcome improvement” and the highly divergent description was: “Actions not influenced by empirical data (e.g., use of personal anecdotes for making strategic decisions)”. The steps of the GTO model (e.g., conducting outcome evaluation) are good prevention practices in general; thus, it is possible that the comparison programs may demonstrate varying levels of performance, making the GTO-IC Map applicable for assessment of both intervention and comparison programs. The GTO-IC Map has demonstrated excellent concurrent validity, inter-rater reliability, and sensitivity to change (Chinman et al, 2008).
All programs were asked to participate in the GTO-IC Map interview. Interview questions were added to the LoU interviews described below for the iGTO programs and done as a stand alone interview with comparison programs. The follow-up rate for the programs assigned to use iGTO (i.e., intervention) in both states is described in the LoU section. In MO, all nine comparison group programs were interviewed at both time points. In TN, the same 16 programs out of 21 were interviewed at both time points. Again, technical assistance staff in each state conducted the GTO-IC Map interviews with the program directors of each of the programs within the iGTO coalitions (or with whoever was the most knowledgeable about the program's operations) at baseline. At follow-up, due to staffing changes, the MO technical assistance staff person conducted the GTO-IC Map interviews in both states.
At baseline, the two TA staff members made GTO-IC Map ratings from digital recordings of the same interview for 2 programs (TN programs). The inter-rater reliability (Pearson r) and % agreement across all 10 components was .65 and 82% for program 1 and .91 and .96% for program 2. Also, a third research staff member who is an expert GTO-IC Map rater (trained and certified by Dr. Hall) rated two MO and two TN programs at baseline. The Inter-rater reliability and % agreement between the expert and MO raters, across these two programs were perfect. The Inter-rater reliability and % agreement between the expert and TN raters, across all 10 components was .73 was 85% for program 1 and .71 and .86% for program 2.
The use of iGTO was measured in several ways. First, we recorded monthly activity every coalition exhibited in each of iGTO's 10 steps (iGTO Monitoring). Second, we conducted interviews with staff who led programs within each coalition assigned to iGTO to assess their overall use of the iGTO system (Levels of Use). Third, near the end of the study, we administered a survey to all coalition members who used iGTO to assess their perceptions of the iGTO's usability (Computer Usability Survey). Finally, near the end of the project, we conducted qualitative interviews with the leaders of each iGTO coalition and with state level officials leading the SPF-SIG in the two states to assess satisfaction with iGTO and the barriers and facilitators to its diffusion within both states (Diffusion Interviews).
The IT developers of iGTO, KIT Solutions, LLC, developed data tables that displayed, in real time, all of the text that had been entered for each of the 10 steps within iGTO for all of the coalitions (e.g., a Step 1 table, a Step 2 table, etc). Then, the TA staff accessed these tables, and assessed whether each coalition had attempted to complete the tasks specified by each step. For example, iGTO Step 2 asks coalitions to enter goals and objectives. At the end of each month, the technical assistance providers would look at the data tables for Step 2 and check to see if goals and objectives had been entered for each coalition over the course of the previous month. If a reasonable attempt had been made to enter in goals and objectives, the technical assistance provider entered a 1 for that coalition under Step 2 for that month in the iGTO Monitoring database. If not, a 0 was entered. This process was repeated for each Step, for all coalitions, for several consecutive months during iGTO implementation (10 in MO, 14 in TN). Data was cumulative, so that Steps in iGTO that received a 1 during a certain month, remained a one for the duration of the monitoring. Monthly monitoring on use of iGTO was conducted from June 2007 to March 2008 for MO coalitions and from June 2007 to July 2008 for TN coalitions.
The LoU measure was chosen because it provides a generic framework for evaluating the utilization of innovations—e.g., iGTO—(Hall & Hord, 2006; Hall et al., 1975; Hall & Loucks, 1977). As shown in Table 3, a single score is assigned, classifying individual programs assigned to iGTO as one of eight types: three types of nonusers (LoU 0 - II) and five types of innovation users (LoU III - VI). Hall and colleagues (Hall & Hord, 2006; Hall et al., 1975; Hall & Loucks, 1977) created these levels based on 25 years of research that showed most innovators progress through similar levels regardless of the innovation. We have applied the LoU categories specifically to the use of the iGTO, identifying behaviors that characterize each level (in consultation with Hall). The LoU has demonstrated strong concurrent validity and inter-rater reliability (Hall & Loucks, 1977), and sensitivity to change and predictive validity (George, Hall, & Uchiyama, 2000).
To determine the LoU score for each iGTO program, TA staff in each state conducted a 20-minute semi-structured interview using a branching format with specific questions and follow-up probes (Foster & Nixon, 1975). The interviews were conducted with the program directors of each of the programs within the iGTO coalitions—or whoever was the most knowledgeable about the program's operations—at baseline. A year later at follow-up, due to staffing changes, the MO technical assistance staff person conducted the LoU interviews in both states. It should be noted that the programs at baseline and follow-up were not always the same. Many of the coalitions were in existence and running programs prior to receiving SPF-SIG funding from their respective states. When these coalitions accepted the SPF-SIG funds, some discontinued some of their current programming and then used the new funds to start new programs. In this study, we assumed the “use” of iGTO would be unaffected by a change in program given that the same staff, with the same Level of Use, were involved in the new programs. Therefore, we compared old programs to new programs across time on the LoU.
At baseline, we interviewed and rated 36 programs across 18 coalitions in MO. At follow-up, we interviewed and rated 36 programs across 18 coalitions in MO, 69% (n=25) of which were the same programs as at baseline, and the rest were new programs. In MO, no programs were lost to follow-up. At baseline in TN, we interviewed and rated 33 programs across 15 coalitions. At follow-up, we interviewed and rated 12 programs across 12 coalitions. In TN, 19 programs were not interviewed because they were delayed and would not have had at least 10 months of iGTO implementation; 2 programs were lost to follow-up. Of the remaining 12 programs at follow-up, 83% (n=10) were the same program. See Figure 2 for the flow of programs being interviewed.
At baseline, the two TA staff members made LoU ratings in their respective states. TA staff in each state also made LoU ratings, from digital recordings, of two programs from the other state. The inter-rater reliability (Pearson r) across these programs was perfect (1.0). At follow-up, the MO TA staff person conducted all the LoU interviews and made all the ratings; therefore inter-rater reliability was not conducted.
All users of iGTO were asked to complete an online Computer Usability Survey after about one year of use. Questions included demographic information, the degree to which the iGTO system helped users complete the steps in the GTO model better, and the likelihood of continuing using the iGTO system after the period of free use ended. These questions were used in an earlier evaluation of a prototype of iGTO (Zhang, 2003).
The survey sample in both states was created by asking program leaders who in their coalition used iGTO at least once. After contact information was obtained, an email was sent to all iGTO users linking them to an internet-based survey site. After an average of 15 months of using iGTO, 36 iGTO users in MO were sent an email, and 28 completed the survey (78% response rate; some coalitions had more than one user). After an average of 13 months of using iGTO, 15 iGTO users in TN were sent an email, and 15 completed the survey (100% response rate).
Interviews were conducted at two levels to better understand how iGTO was diffused: coalition level and the state level. At the coalition level, most directors of coalitions assigned to use iGTO in MO (n=16, 89%) and TN (n=14, 93%) were interviewed about half way through the intervention period. The interview was constructed to assess characteristics of the innovation specified in Rogers' (1995) Diffusion of Innovations theory that would influence diffusion, namely relative advantage over previous tools, complexity, and compatibility with current systems and structures. At the state level, officials responsible for substance abuse prevention activities statewide in both states (3 in MO, 6 in TN) were interviewed at the end of the intervention and asked similar questions, tailored for a statewide perspective.
The analyses of the GTO-IC Map data were a series of repeated measures, Time (pre/post) by Group (iGTO/comparison) ANCOVAs. The effect of interest was the Time by Group interaction, answering the question of whether programs assigned to use iGTO experienced differential improvement over time compared to those who did not use iGTO. The dependent variables were each of the GTO 10 steps as assessed on the GTO-IC Map. At baseline, we conducted independent t-tests comparing the iGTO and comparison groups on all the GTO 10 steps dependent variables. In MO, significant differences at baseline were found for the step of Capacity [t(43)=2.22, p=.03]. In TN, significant differences at baseline were found for the steps of Best Practices [t(26)=2.15, p=.04], Fit [t(26)=3.82, p=.00], Capacity [t(26)=2.55, p=.02], and Plan [t(26)=2.17, p=.04]. Therefore to be conservative, we included the baseline version of these dependent variables as covariates to adjust for any baseline differences. In addition, we conducted dependent t-tests on all of the GTO ten steps by group to assess what was driving the significant Time by Group interactions. In TN, we also compared those who were not available to be interviewed at follow-up to those who remained on all the baseline GTO-IC Map variables and found no differences.
Data on iGTO use across the 10 GTO Steps was averaged to create a single iGTO use score for each month. These monthly iGTO use scores were then plotted on a graph to show the level of implementation of the iGTO system.
Change in LoU scores between the start and end of the iGTO implementation period were assessed using two dependent t-tests (one for MO, one for TN). Also, mean follow-up LoU scores were compared to the established categories of use established by Hall (see Table 3).
Items assessing how well iGTO helped coalition members do tasks better compared to how they were done before were analyzed descriptively.
To better understand the factors that facilitated or inhibited how iGTO was diffused at the coalition and state levels, we used classic content analysis on the interview data (Krippendorf, 1980). To create a final list of themes from both sets of diffusion interviews, we (a) reviewed list of responses for all questions; (b) identified the key concept in each statement and placed it in a second list; (c) removed redundant statements in the second list, yielding a third list; (d) reviewed the third list for parallel themes (not redundant, but part of the same topic) and combined them into a final list with key themes. To place these themes within a theoretical context, we then overlaid the resulting themes onto Rogers' (2005) Diffusion of Innovations theory's three characteristics of innovations that influence adoption, namely (a) the degree to which an innovation has a relative advantage over current practice), (b) compatibility with current systems, and (c) complexity.
The Time × Group interaction for the total GTO-IC Map score from the repeated measures ANCOVAs was significant in both MO and TN. In both states, those in the iGTO group improved more over time on the GTO-IC Map total score than those in the comparison group. Coalitions in both states improved about the same overall, but in MO, the comparison coalitions worsened significantly over time whereas the comparison coalitions in TN remained flat. Looking at the repeated measures ANCOVAs for the individual GTO steps, in MO, the Time × Group interaction was significant in eight out of the 10 steps (Fit and Plan were not significant). For seven of the eight significant steps in MO, individual dependent t-tests showed that the iGTO group improved significantly. Seven out of the eight steps for the comparison group worsened over time on individual t-tests (Capacity was flat). The Time × Group interaction of the step of Goal was significant because the control group got much worse while the iGTO group improved only slightly. Looking at the repeated measures ANCOVAs for the individual GTO steps in TN, three steps (Best Practice, Outcome Evaluation, and Continuous Quality Improvement) had significant interactions in which the iGTO coalitions improved and the comparisons worsened. Capacity worsened for the iGTO group and improved for the comparison group over time in TN.
Across all the GTO steps and both states, there were 66 significance tests conducted (22 dependent t-tests for the iGTO group, 22 dependent t-tests for the comparison group, and 22 Time × Group interaction effects), potentially inflating the Type I error rate. Adopting a 5 or 10% error rate, one could expect three to six analyses to be significant by chance respectively. However, in the case of this study, 36 out of the 66 analyses were significant. Of those 36 significant analyses, 32 were in the expected direction (i.e., iGTO group improved, or comparison worsened).
As shown in Figure 3, use of the iGTO system increased about 10 to 15% a month in MO (n=18 coalitions) and TN (n=16 coalitions) for five months. After that, implementation in TN remained flat for the remaining iGTO implementation period. In MO, the use of iGTO did increase more, albeit at a slow rate, 13% over six months, with the increase coming in one month (January, 2008).
All iGTO programs began at baseline with a LoU score of 2, indicating the Preparation Level of Use. In both states, a t-test was used to assess the degree to which iGTO programs progressed to higher Levels of Use. In MO, there was a significant improvement in LoU [t(36)=4.90, p=.00] after the end of the iGTO implementation period to 2.92 (SD=1.14), about the Mechanical Level of Use. This suggests that MO programs were using iGTO, but in a mostly basic manner, still trying to become comfortable with the system. In TN, the average LoU score at follow up was 3.58 (SD=0.33), which was a significant improvement in use [t(11)=4.71, p=.00]. This suggests that TN programs were on average between the basic usage of the Mechanical level (=3) and the more stabile use that characterizes the Routine level (=4).
The Computer Usability survey included questions about the degree to which the iGTO system helped users complete the steps in the GTO model any better than prior to using the iGTO system (See Table 5). In MO, while iGTO helped users understand the 10 step GTO conceptual model somewhat better (M=4.04), the other items asking about individual steps ranged from 3.33 to 2.67, indicating that iGTO helped to a modest degree. In TN, users rated iGTO as helpful in assisting users find evidence-based programs compared to what they were doing before (M=5.07). On the remaining items, TN users rated iGTO as moderately more helpful with all items ranging from 3.73 to 4.40. Users were also asked questions about their potential use of iGTO after the end of the study when the coalitions would be expected to pay for the iGTO service on their own ($175 per month). In MO, 93% of the respondents said they were not likely to subscribe to iGTO after the iGTO demonstration was over, while 2% were not sure. In TN, 60% were not likely to subscribe to iGTO when the SPF-SIG was over. Twenty percent were likely or very likely to use it and 20% were not sure.
To better understand the factors that facilitated or inhibited how iGTO was diffused within the coalitions, we interviewed both coalition level users of iGTO and state level prevention officials in terms of their perceptions of relative advantage, complexity, and compatibility of iGTO.
In terms of relative advantage in MO, the state level stakeholders viewed iGTO as having some drawbacks for the coalitions they were funding such as duplication of work and conflict with other online systems. The local iGTO users perceived some advantages to iGTO. For example, some users stated they appreciated the resources within the iGTO library and links to other information. However, several stated that iGTO was a “record keeper” as opposed to a more integrated system. In TN, the state level officials as a group agreed that the use of iGTO by the coalitions helped link their activities with their goals and activities, support the coalitions reporting of needs assessment data, and facilitate the choice of evidence-based programs and strategies. They also did state, however, that iGTO had a limited number of evidenced-based programs and strategies available and that it was not clear how to extract information out of the iGTO for their own use. The local iGTO users reported some benefits including guidance on use of different prevention programs and increased organization among coalitions who were new to prevention. However, TN users also reported significant drawbacks including the aesthetics of the reports, design issues, the layout and logic, and that the full list of environmental strategies were not well represented.
In terms of complexity, MO state officials all felt that the coalitions' lack of process evaluation skills was a major inhibitor to use of iGTO. Echoing this sentiment, many local iGTO users in MO reported that iGTO called for actions beyond what some coalitions normally did in their practice and that iGTO requires different skills sets than many possessed. In TN, state officials said that the coalitions they fund exhibit a resistance to using data to make decisions, which could have made the use of iGTO appear more complex. iGTO users in TN agreed, stating that their coalition members were not normally as “data minded”, requiring a cultural change in order to fully appreciate and engage with iGTO. In addition, local users stated that there was a widespread lack of understanding about the iGTO's full capabilities.
In terms of compatibility, MO state officials stated that there were many barriers to making iGTO compatible with the state's existing procedures and structures. For example, a lack of funding, a resistance to new technologies and innovations at the state level, and a perceived disconnect between iGTO and the needs of the SPF-SIG negatively affected diffusion. Local MO users talked about similar issues. Most reported that there was a “competition” between iGTO and the SPF-SIG model. The two have different starting points, which caused frustration and confusion for the coalition members. Because the link between iGTO and SPF-SIG was “not easily observed”, the coalitions reported that they had to use two different systems. As a result, iGTO became “something else to do”, prioritized behind “doing the work” of the coalitions. Some local coalitions did report that iGTO helped them to become more organized in their coalition work and provided a history and information for future action. In TN, state officials reported that it was problematic that iGTO contained fewer programs and strategies than the local coalitions were using. Local TN coalitions reported that their lack of familiarity of using data made it more difficult to incorporate iGTO into their routine operations.
This study assessed the impact, use, and diffusion of an interactive, web-based system—Interactive Getting to Outcomes or iGTO—designed to assist community-based practitioners and state level officials with the performance of substance abuse and underage drinking prevention activities. The repeated measures ANCOVAs assessing prevention performance showed that iGTO programs demonstrated an increase in performance on the GTO-IC Map over time, while comparison programs were mostly either unchanged or worse. This impact was more pronounced in MO, where the activities of eight out of the 10 GTO steps (and the GTO total score) were found to have improved over time. In TN, the impact was somewhat less, where the activities of only three out of the 10 GTO steps (and the GTO total score) were found to have improved over time. The analyses suggest that the use of iGTO had a positive impact on how well programs were able to carry out key prevention tasks such as planning and evaluation.
In terms of iGTO diffusion, monitoring data shows that, overall, iGTO was diffused by most of the iGTO-assigned coalitions. Further, the level of iGTO diffusion—defined as how much information was recorded in each step—did appear to increase according to a predictable and appropriate schedule, namely that programs recorded information within each step of iGTO in a sequential order as their programs progressed. The fact that 70 programs across 33 coalitions in two states began using a new system such as iGTO was noteworthy. The quality with which the coalitions used iGTO remains unclear however. The Level of Use (LoU) data showed that most iGTO programs reached a Mechanical level of use, indicating that iGTO was being used in a rudimentary fashion. This type of utilization is actually typical for use of a new system such as iGTO (Hall & Hord, 2006).
In addition, users' experience with and perceptions of the iGTO system were mixed. On the one hand, several iGTO users mentioned that the system helped them become more organized and provided useful links to resources (e.g., best practice information). On the other hand, iGTO users in both states reported experiencing technical difficulties that made using iGTO challenging and frustrating. iGTO users in both states reported that iGTO provided a moderate amount of assistance in completing various key prevention tasks better than they had done before using iGTO. Based on these responses, it is not surprising that a majority from both states reported they would not use iGTO after the intervention period ended (i.e., when they would have to pay for it). Several months after the end of the intervention period, none of the Missouri or Tennessee coalitions asked to continue using iGTO.
Applying Diffusion of Innovation theory to our observations about the diffusion and use of iGTO—including the intention by the coalition users not to pay for iGTO after the free intervention period had ended—provides explanations for what was observed as well as guidance for how to improve such roll outs of computer systems like iGTO in the future. Both local coalition users and state level officials in both states, reported on three key dimensions of the innovation's characteristics that Diffusion of Innovation theory posits is related to diffusion: relative advantage, compatibility, and complexity. Although, in interviews and open-ended comments, several reported that iGTO provided useful links to various resources (e.g., to best practice program information), generally these stakeholders reported that iGTO offered modest relative advantage over what they were doing before. Also, according to the usability survey and stakeholder interviews, most of the coalitions found iGTO to be complex to use because of the design of iGTO, technical bugs, and the lack of planning and evaluation capacity on the part of the coalitions. Finally, most stakeholders in both states commented that iGTO was not compatible with other processes and tools the coalitions were using as result of their state funding, namely the SPF-SIG. Having difficulty integrating iGTO into the required SPF-SIG led coalitions to have to do extra work to complete tasks under both models. The fact that iGTO and the associated training were not able to demonstrate its relative advantage, ease of use, or compatibility are important factors according to Diffusion of Innovation theory in the lack of intended and eventual adoption by the coalitions after the end of the intervention period.
Certain limitations should be noted. First, the impact of iGTO on prevention performance was evaluated with a quasi-experimental design in MO. Coalitions and programs in MO that received iGTO were supported by SPG-SIG funds and the comparison programs were not. Thus, these two groups differed in terms of the size of their budgets (although not statistically significant because of the large amount of variation, iGTO coalitions still had more funding). Given these baseline differences, it is possible that unmeasured characteristics biased the results and these results should be interpreted with caution. In addition, the GTO-IC Map analyses in both states may have had some bias since variations across coalitions were not taken into consideration given the small sample sizes. We did attempt to mitigate this somewhat by adjusting for baseline differences in the outcome of interest, the GTO-IC Map score. This design was appropriate given the early stage of development of iGTO and the opportunity to evaluate the use of iGTO in real prevention settings outweighed the design challenges inherent in such an approach. A next step should be studies that use random assignment with larger numbers of similar types of prevention programs when possible to determine the differential effects of iGTO on prevention performance.
Even with the noted limitations, these results suggest certain implications. iGTO did enjoy a certain level of support from state level officials, who required that all local coalitions and programs use iGTO in their work. However, even with this level of support, iGTO was never integrated into how both states, and therefore all the local coalitions, were running their state level prevention systems. One possible reason for this is that while state officials said that they believed the iGTO system would be helpful for coalitions and programs at the local level, they did not rely on iGTO for anything at the state level. The lack of connection between the state and local levels reduced iGTO's relative advantage for stakeholders at both levels. Therefore, it is clear that much greater support and use of a system such as iGTO by the funder (the state) is needed to incorporate it into the operations of prevention at the state and local levels.
The second implication is that when rolling out a technology innovation, such as iGTO, to actual users, the innovation itself ought to be beyond the beta level of readiness if the goal is to gain adoption of the innovation among these same users. The project was conceptualized and funded as a pilot and as such, that local users would participate in the refinement of the iGTO system. The model of the CQI-Design, in which knowledge is gained in an iterative cycle of deployment-feedback-refinement-deployment, remains an excellent one for getting critical feedback about an innovation. However, the actual users who participated in this process appeared not to favor such an approach in terms of their own work. The local coalitions' interest was in accomplishing their prevention work as efficiently as possible. While they participated in the demonstration of iGTO, they were clearly frustrated by the technical difficulties of the iGTO system. Coupled with a lack of a connection to the state level mentioned above, the local coalitions had less motivation to continue with iGTO after the intervention period. A better approach for states interested in systems such as iGTO may be to work with state level officials more closely at the beginning to design a system that would meet state level needs, test it with a small group of coalitions to ensure it met local coalition needs, and to only roll it out broadly once these tasks were completed satisfactorily. Such an approach would likely improve the user-friendliness of such a system.
A third implication is that like other innovations, outside support is often needed to assist with diffusion and successful implementation of innovations such as iGTO. Most implementation models and theories—e.g., REAIM (Glascow et al., 1999), PRECEDE/PROCEDE (Green et al., 1980), Simpson Transfer Model (Simpson, 2002), Concerns Based Adoption Model (Hall & Hord, 2006), Diffusion of Innovations (Rogers, 1995), and the Interactive Systems Framework for Dissemination and Implementation (Wandersman et al., 2008)—are all similar in their call for this type of outside facilitation. We believe that the level of training and technical assistance provided in this project was not enough to support the incorporation of iGTO into routine coalition operations in the short-term or the continued diffusion in the long-term. Related to this point, is the fact that iGTO may simply require a higher level of prevention capacity than many current community-based prevention coalitions typically possess. Several stakeholders both at the state and local levels mentioned this lack of capacity as a barrier to use iGTO and long-term use of iGTO. Therefore, this study shows that those interested in developing large scale capacity-building systems ought to provide sufficient technical assistance.
iGTO was designed to help community-based substance abuse prevention coalitions better plan, implement, and self-evaluate their own programming as a way to improve outcomes. Combining the GTO model and tools (manuals, training, TA) with the flexibility of the internet has the potential to assist communities in conducting various prevention tasks much more efficiently. This formative evaluation of its deployment across the state prevention systems in Missouri and Tennessee suggests that it is possible to roll out iGTO across a large number of users and have it be helpful. However, to achieve more comprehensive and long-term integration into regular operations likely requires a much greater amount of support from state level leadership. State leaders interested in large scale capacity building using technology such as iGTO need to use and rely upon the system themselves, not just view it as a tool that would only benefit local coalitions they fund. Also, systems such as iGTO have to be well developed before widely released and local coalitions need support for using new tools such as iGTO. Despite challenges faced in the roll-out, programs whose staff were using iGTO demonstrated better prevention performance than those who were not; this suggests that iGTO has great potential to aid community-based prevention practitioners.
The National Institute on Alcohol Abuse and Alcoholism provided funding for this paper through support of A web-based IT Solution for Outcome Based Prevention (5R42AA015529-03). Work on this paper was also supported by The Department of Veterans Affairs Desert Pacific Mental Illness Research, Education, and Clinical Center (MIRECC). We would like to thank the technical assistance provider in Tennessee, Jack Willis; the two state agencies of substance abuse prevention in Missouri and Tennessee (DADAS); Knowlton Johnson, David Collins, and Melissa Harris of Pacific Institute for Research and Evaluation (the Tennessee State Incentive Grant evaluator); Mary Dugan, Christine Owens, and Susan Depue, Missouri State Incentive Grant evaluator, project director, and assistant project director, respectively; Pam White and Community Anti-Drug Coalitions Across Tennessee; and Xiaoyan Zhang, Jennifer Caputo, and Matt Puskar of Kit Solutions, LLC, the IT developer of iGTO. We would also like to thank all of those in Missouri and Tennessee, whose participation in this project made it possible. Any opinions expressed are only the authors', and do not necessarily represent the views of any affiliated institutions.
Publisher's Disclaimer: The following manuscript is the final accepted manuscript. It has not been subjected to the final copyediting, fact-checking, and proofreading required for formal publication. It is not the definitive, publisher-authenticated version. The American Psychological Association and its Council of Editors disclaim any responsibility or liabilities for errors or omissions of this manuscript version, any version derived from this manuscript by NIH, or other third parties. The published version is available at www.apa.org/pubs/journals/ort.