Search tips
Search criteria 


Logo of nihpaAbout Author manuscriptsSubmit a manuscriptHHS Public Access; Author Manuscript; Accepted for publication in peer reviewed journal;
J Subst Abuse Treat. Author manuscript; available in PMC 2013 June 1.
Published in final edited form as:
PMCID: PMC3345887

Adolescent treatment initiation and engagement in an evidence-based practice initiative

Margaret T. Lee, Ph.D.,a,* Deborah W. Garnick, Sc.D.,a Peggy L. O'Brien, M.A., J.D.,a Lee Panas, M.S.,a Grant A. Ritter, Ph.D.,a Andrea Acevedo, M.S., M.A.,a Bryan R. Garner, Ph.D.,b Rodney R. Funk, B.S.,b and Mark D. Godley, Ph.D.b


This study examined client and program factors predicting initiation and engagement for 2,191 adolescents at 28 outpatient substance abuse treatment sites implementing evidence-based treatments. Using Washington Circle criteria for treatment initiation and engagement, 76% of the sample initiated, with 59% engaging in treatment. Analyses used a 2-stage Heckman probit regression, accounting for within-site clustering, to identify factors predictive of initiation and engagement. Adolescents treated in a pay-for-performance (P4P) group were more likely to initiate, whereas adolescents in the race/ethnicity category labeled other (Native American, Asian, Pacific Islander, Native Alaskan, Native Hawaiian, mixed race/ethnicity), or who reported high truancy, were less likely to initiate. Race/ethnicity groups other than Latinos were equally likely to engage. Among White adolescents, each additional day from first treatment to next treatment reduced likelihood of engagement. Although relatively high initiation and engagement rates were achieved, the results suggest that attention to program and client factors may further improve compliance with these performance indicators.

Keywords: Adolescent substance abuse treatment, Performance measures, Treatment engagement, Evidence-based practices, Pay-for-performance

1. Introduction

Although treatment is recognized as effective for recovery from a substance use disorder, substantial numbers of adolescents referred to outpatient care do not return for more treatment after the first visit (Szapocznik, Lopez, Prado, Schwartz, & Pantin, 2006; Szapocznik et al., 1988; Waldron, Kern-Jones, Turner, Peterson, & Ozechowski, 2007). Increasing the likelihood of retaining an adolescent in substance abuse treatment is recognized as challenging (Campbell, Weisner, & Sterling, 2006; Dakof, Tejeda, & Liddle, 2001; Henggeler, Pickrel, Brondino, & Crouch, 1996; Simpson, 2001; Waldron et al., 2007). However, because the steepest dropout occurs during the first few sessions (Dakof et al., 2001), focusing on treatment processes at the beginning of a new episode of care could make a difference.

1.1. Adolescent treatment initiation and engagement

Treatment initiation and engagement are recognized as important benchmarks on the path to recovery from a substance use disorder (Garnick et al., 2007; Godley, Hedges, & Hunter, 2011; Harris, Kivlahan, Bowe, Finney, & Humphreys, 2009; Harris, Humphreys, Bowe, Tiet, & Finney, 2008; Lee et al., 2007). Performance measures that measure provision of timely substance abuse services at the start of a new outpatient treatment episode were developed by the Washington Circle (WC), a group focused on developing performance measures for substance abuse treatment in both the private (Garnick et al., 2002) and public sectors (Garnick, Lee, Horgan, & Acevedo, 2009). Clients who initiate receive another treatment service within 2 weeks after beginning a new episode, and those who engage receive two additional services within the month after initiation. These measures are already being used extensively by, among others, the National Committee for Quality Assurance (2010), the Department of Veterans Affairs (Harris, Humphreys, Bowe, Tiet, & Finney, 2010; Harris, Humphreys, & Finney, 2007), and several states (Garnick et al., 2011), and have been endorsed by the National Quality Forum (2009). Performance measures that assess the extent that clients initiate and engage in treatment can be useful tools for providers to monitor the delivery of quality care.

The WC initiation and engagement measures were developed for an adult population, but the WC specification of the number and timing of treatment sessions may be applicable for adolescents as well. The WC measures specify a minimum floor of services that is consistent with previous research focused on adolescent treatment. Dakof et al. (2001) report that the period when we would most likely see high dropout for adolescents is during the first few sessions and that participation in less than four sessions is not enough to engage adolescents in treatment. Waldron et al. (2001) considered adolescents as enrolled with one therapy session and engaged in treatment with four or more sessions. Thus, although the WC measures were developed for an adult population, prior research with adolescents suggests that they should be tested for use with adolescents as indicators of timely service delivery at the beginning of a new treatment episode.

1.2. Influences on adolescent treatment initiation and engagement

There are both client and program factors that influence adolescent initiation and engagement in treatment. Research focused on client-level factors has examined a variety of client characteristics including the nature of the individual's substance use, demographic characteristics, homelessness, and indicators of problem severity. One study by Lee et al. (2007) found that type of substance used and frequency of use were associated with treatment initiation. They found adolescents who used alcohol frequently (three or more times) in the past week were more likely to initiate treatment than those who used less frequently, whereas those who used marijuana frequently in the past week were less likely to initiate treatment. Another recent study examining the impact of gender and race/ethnicity on engagement in outpatient adolescent treatment found no significant difference by gender or race/ethnicity (Godley, Hedges, et al., 2011).

Factors like victimization or homelessness that render adolescents more vulnerable also may add to the challenges of engaging them in treatment. For example, adolescents who have been victimized tend to have more substance-related problems and thus may be more difficult to treat and keep in treatment (Shane, Diamond, Mensinger, Shera, & Wintersteen, 2006). Homeless youth also tend to present with greater problems and intensified severity (Bantchevska, Bartle Haring, Dashora, Glebova, & Slesnick, 2008) and are more likely to have trouble utilizing care for many reasons, including limited transportation, lack of insurance, and social barriers (Ensign & Bell, 2004; Hudson et al., 2010; Slesnick, Kang, & Aukward, 2008), although one study found that homeless adolescents with a history of sexual abuse and suicide attempts demonstrated higher substance abuse treatment attendance (Slesnick et al., 2008). The higher attendance by homeless youth in this study was accomplished through an open door policy, which made it easier for the adolescent to meet with the therapist. When adolescents have other problems, it may be more difficult for them to stay in treatment, and they may require supportive services to keep them in treatment.

In addition to client factors, programs also could adopt new approaches and/or make changes in how they conduct their business to bring about improvements that increase the chances of adolescents entering and returning for more treatment (Slesnick et al., 2008). Two program factors that research has found to be related to initiation and engagement are included in this study: time to initiation and incentives.

First, we include the time that lapses from first treatment service to the subsequent service because it has been shown that a shorter window of time improves the chances of a client keeping an appointment and returning for more services (Carr et al., 2008; Claus & Kindleberger, 2002). Second, many programs are beginning to use incentives to improve quality of care, especially since the Institute of Medicine (2001, 2007) recommended pay-for-performance (P4P) as an approach for enhancing adherence to treatment processes and interventions that are associated with more effective treatment. Providers are paid based on their performance on a specified set of measures with a goal of bringing about increased adherence to recommended practices or clinical or business outcomes. P4P is increasingly becoming more prevalent in health care organizations, and more recently, the approach has been adopted for behavioral health care (Bremer, Scholle, Keyser, Houtsinger, & Pincus, 2008). Experience with P4P in substance abuse/mental health agencies in Connecticut, Maine, and Delaware have shown the importance of timely payments of incentives linked to performance (Commons, McGuire, & Riordan, 1997; McLellan, Kemp, Brooks, & Carise, 2008; Stewart, 2009), including utilization targets (McLellan et al., 2008; Stewart, 2009), and considering programs’ abilities to influence continuity of care outside their own organization (Daley et al., 2010).

1.3. Study goals and hypotheses

Because past research shows that client and program factors both contribute to treatment initiation and engagement, this study examines the impact of each on the likelihood of initiation and engagement among adolescents using WC measures. In view of a current focus on therapist-level incentives to improve treatment (Garner, Godley, & Bair, 2011; Garner, Godley, Dennis, Godley, & Shepard, 2010; Shepard et al., 2006; Vandrey, Stitzer, Acquavita, & Quinn-Stabile, 2011), we examine the impact of a P4P initiative on adolescents’ treatment initiation and engagement. We also study the effect of timely provision of services at the beginning of a new treatment episode on engagement.

To increase an adolescent's likelihood of initiation, engagement, and recovery outcomes and in agreement with the recommended use of evidence-based practices (EBPs) in substance abuse treatment (Garner, 2009), the Center for Substance Abuse Treatment (CSAT) launched a national dissemination and implementation initiative called Assertive Adolescent and Family Treatment (AAFT). AAFT focused primarily on the dissemination and implementation of the Adolescent Community Reinforcement Approach and Assertive Continuing Care (A-CRA/ACC; Godley, Godley, Karvinen, Slown, & Wright, 2006; Godley et al., 2001). These EBPs focus on the interaction between youth and their environments and are individual and family centered. AAFT grantees serve as the sites for this study, and more details of these approaches are presented in the Methods section.

Based upon the forgoing literature, the following hypotheses were tested:

  1. Adolescents who are more vulnerable are less likely to initiate or engage in treatment. Those who report greater severity on Global Appraisal of Individual Needs (GAIN) subscales such as victimization, being homeless, or having behavioral problems (e.g., conduct disorder or truancy) are less likely to initiate or engage in treatment.
  2. Adolescents’ demographic characteristics, such as race and gender, will not be associated with their initiating and engaging in treatment. We will test the null hypothesis because past literature found no difference between groups (Dakof et al., 2001; Godley, Hedges, et al., 2011).
  3. Implementation of P4P in treatment programs will increase the likelihood of clients initiating and engaging in treatment.
  4. Treatment process, such as shortening the time to initiation, will increase the likelihood of clients engaging in treatment.

2. Methods

2.1. Study context

This study uses secondary data (collected between January 2007 and May 2010) from 28 treatment sites that were part of a large-scale initiative funded by the CSAT to implement A-CRA/ACC. The treatment sites were spread across 12 states in different regions of the country (Northeast [5], Southeast/South Atlantic [8], Midwest [2], Southwest [4], and West [9]) and served a diverse range of urban and rural communities, including “colonias” on the Texas–Mexican border and Native American communities. To take into account the impact of start-up and close-out activities at each site, data from the first quarter and final two quarters of each site's program under the initiative were excluded from the current analyses. All procedures for the study were in compliance with the standards of the appropriate institutional review boards.

2.1.1. Intervention

The treatment sites each received financial resources of approximately $300,000 per year for up to 3 years to implement A-CRA and ACC. The CRA has been widely studied for adult substance use disorders (Roozen et al., 2004) and was adapted for adolescents and their caregivers (Godley et al., 2001). The change theory underlying A-CRA is to help youth and their caregivers find rewarding activities in the community, improve communication, and develop other skills to effectively compete with substance use. A-CRA involves a menu of 17 behavioral assessment and intervention procedures that are used in weekly outpatient sessions to address specific client issues. The average A-CRA/ACC session length is approximately 60 minutes. Over a 90-day period, the goal is to conduct 10 individual sessions, 2 caregiver sessions, and 2 combined caregiver and adolescent sessions. The ACC (Godley, Godley, Dennis, Funk, & Passetti, 2002; Godley, Godley, Dennis, Funk, & Passetti, 2007) extends the use of A-CRA procedures and case management activities for another 90 days to help youth and caregivers generalize new skills across situations and settings and help youth comply with juvenile justice, education, and other community systems within which they may be involved.

Therapists for this project (a) attended a 3.5-day A-CRA/ACC standardized training with the model developers; (b) audio-recorded sessions and received feedback from model experts until they demonstrated competency; and (c) participated in biweekly coaching calls with model developers for 1 year. After competency was achieved, randomly selected therapy sessions were checked for model fidelity bimonthly, with more than two-thirds demonstrating competency in either the first or second attempt (Godley, Garner, Smith, Meyers, & Godley, 2011).

As part of an effort to improve A-CRA/ACC implementation within this CSAT initiative and also to test the effectiveness and cost-effectiveness of P4P approaches in general, Garner et al. (2010) implemented a cluster randomized experiment called Reinforcing Therapist Performance (RTP). The A-CRA/ACC sites and therapists within sites were recruited to participate in the study. Those who agreed to participate returned signed copies of informed consent. Sites were randomized into either implementation as usual or the P4P condition, and an equal number of sites were in each condition.

Site recruitment for the RTP experiment occurred between November 2008 and February 2009. Therapist recruitment for the experiment began 1 month after site recruitment. Both groups received the same training and technical assistance common to all sites in the AAFT initiative. The P4P group had the opportunity to earn monetary bonuses for two sets of behaviors related to quality implementation of the treatment model: (a) delivering target A-CRA and (b) demonstrating monthly A-CRA competency. For each adolescent who received a threshold exposure of A-CRA (10 of 12 procedures) within 14 weeks of AAFT treatment and in no fewer than seven sessions, the therapist in the P4P condition received a $200 bonus. Incentives also were given for quality of A-CRA procedures delivered. Specifically, P4P therapists also received a $50 bonus for each month that a randomly selected session recording had at least one core procedure rated at or above the minimum level of competence that was required for certification.

2.2. Sample characteristics

The study sample was composed of 2,191 adolescents who were predominantly male (73.5%, 1,610) and ranged in age from 12 to 18 years. Approximately one-third were White (33.8%, 740), another third were Latino (31.5%, 689), 14.9% (326) were Black, and 19.9% (436) were of “other” race/ethnicities. Those included in the Other category were 18.8% (82) Native American, 71.3% (311) mixed race/ethnicity, and 9.9% (43) other race/ethnicities including but not limited to Asian, Pacific Islander, Native Alaskan, and Native Hawaiian. Approximately one-third (33.9%, 742) of the sample subjects had been in school less than 31 days during the 90 days prior to intake, and 7.8% (170) were homeless. Significant numbers reported symptoms of attention deficit hyperactivity disorder (ADHD; 44.5%, 975) or of conduct disorder (48.3%, 1,058) in the year before intake, with ADHD requiring clients to endorse six or more symptoms related to (a) inattention, (b) hyperactivity/impulsivity, or (c) both inattentive and hyperactive type, and conduct disorder requiring endorsement of three or more conduct disorder symptoms. Furthermore, 63.4% (1,389) of the sample scored at either moderate or high levels on GAIN's General Victimization Scale (GVS; Titus, Dennis, White, Scott, & Funk, 2003), and 61.4% (1,345) were identified as having either medium or high levels of substance abuse-related problems in the past month (SPSM; Dennis, Chan, & Funk, 2006).

2.3. Dependent and independent variables

The two dependent variables used in our study models were treatment initiation and engagement, two process measures developed by the WC (Garnick et al., 2002; Garnick et al., 2009). Specifically, initiation was defined as receiving a second substance abuse treatment service within 14 days of the index service. The index service marks the start of a new outpatient treatment episode and is preceded by a 60-day service-free period. The rationale for the initiation measure is that after the index, or first treatment session, the client should return for another session within a short window of time. The index visit includes treatment, but it often also may involve intake and assessment services; therefore, the individual needs to return for an additional service to initiate treatment. The definition of engagement is receiving two additional substance abuse treatment services within 30 days of initiation (Garnick et al., 2009). Service dates abstracted from electronic records collected as part of the AAFT project were used to determine initiation and engagement rates for each site and whether each adolescent met the criteria for the measures. The low threshold used to define engagement represents a minimum level applicable for all clients starting a new episode of substance abuse treatment. The WC concept of early treatment engagement is different from that of retention, which would require a separate measure with criteria requiring a longer stay.

Independent variables in our study were derived from the GAIN, which is a standardized biopsychosocial assessment instrument that has undergone extensive psychometric testing and has been normed on both adolescents and adults (Dennis et al., 2002; Dennis, White, Titus, & Unsicker, 2008). The GAIN assessment was administered at intake, and variables considered for inclusion in our models were those that previous research suggested would be related to initiation and engagement. General domains considered were client demographics, clinical factors, behavioral problems, hospitalizations, and medical history; the latter two were excluded from our final models. Within each of the domains that were included, we selected the best variables from the GAIN data set to measure the domain, since there are multiple ways to measure a specific domain. Criteria for selection took into consideration data quality and completeness, excluding those with too many missing values; inadequate cell size for analysis; and avoidance of multicollinearity. We also examined predictive value using preliminary logistic models of initiation and engagement with each independent variable separately. Those variables that satisfied the significance level of p < .25 were retained for our final models, specifically client demographics, including living situation; clinical factors, including history of victimization and symptoms of conduct disorder or ADHD; and behavioral problems, such as truancy and substance use problems (Chan, Dennis, & Funk, 2008; Funk, McDermeit, Godley, & Adams, 2003; Hussey, Drinkard, Falletta, & Flannery, 2008; Tims et al., 2002; Titus, et al., 2003). To determine whether monetary incentives to therapists influence initiation or engagement, our models included variables indicating whether the treatment site was a P4P site, whether the client was admitted after the start of the RTP study, and the interaction of these two variables. To examine how treatment processes may impact engagement, time from the index service to the next service (i.e., treatment initiation, as defined above by the WC measures) also was included in the model predicting engagement.

2.4. Analysis plan

A two-stage Heckman probit procedure, accounting for clustering within sites, was used to investigate predictors of initiation and engagement. The Heckman probit procedure (StataCorp, 2009) models initiation at a first stage and then models engagement at the second stage, based on the portion of the sample who initiated. The second stage of the Heckman procedure included a variable for number of days from index service to next treatment service (or “days to initiation”). In addition, interactions between days to initiation and the race/ethnicity variables were included in the second stage as those interactions satisfied the significance test of p < .25. In both the first and second stages, we included variables related to the P4P inquiry, as discussed above.

Before including the days to initiation variable, we considered its potential endogeneity. This question focuses on, among those who initiate, whether the site or the client was more influential on the number of days from index to initiation. If the sites have the greatest influence, then there is no problem with its inclusion in our model, and if the time variable is found to be significant, then sites might choose to address this in their provision of early treatment in hopes of improving engagement rates. On the other hand, if the time to initiation variable is influenced by client characteristics such as gender, age, race/ethnicity, client motivation, substance use history, mental health status, or behavioral problems, then it is said to be endogenous. If so, its inclusion in the model could bias estimation of the effects of these other characteristics, and the other demographic and historical variables in the model may appear to be less significant than they are. Based on these considerations, we conducted preliminary analyses to determine whether client characteristics such as those mentioned above were associated with days to initiation. Results did not show evidence of endogeneity, and thus, the days to initiation variable was included in the second step of our Heckman procedure that predicts engagement.

3. Results

3.1. Initiation and engagement rates

Among the 2,191 adolescent subjects enrolled in the study, 76% (1,668) initiated treatment and 59% (1,287) engaged. The number who engaged in treatment represents 77.2% of adolescents who initiated. The mean number of days from index to initiation among these subjects who initiated was 7.4 (SD = 3.25). As Fig. 1 shows, however, there was considerable variation between sites as to their rates of initiation and engagement.

Fig. 1
Rates of initiation and engagement by site.

3.2. Predictors of initiation

As indicated in Table 1, adolescents who were admitted after the RTP experiment started, and to a site that was randomized to the RTP experiment's P4P condition, were significantly more likely to initiate treatment. In addition, those in school less than 31 days during the 90 days before intake, or who were from a race/ethnicity other than White, Black, or Latino, were significantly less likely to initiate treatment. Other client variables, including homelessness, victimization, substance use problems, and conduct disorder, were not significant predictors of initiation.

Table 1
Results of two-stage Heckman probit regression of treatment, demographic, and clinical factors associated with treatment initiation and engagement (stage 1: factors associated with treatment initiation, N = 2,191)

3.3. Predictors of engagement

Table 2 shows that adolescents who were Latino were significantly less likely to engage in treatment. Other client factors including age, gender, homelessness, truancy, substance use disorders, ADHD, and conduct disorder were not significant predictors of engagement.

Table 2
Results of two-stage Heckman probit regression of treatment, demographic, and clinical factors associated with treatment initiation and engagement (stage 2: factors associated with treatment engagement, N = 1,668)

In addition, the variable for the number of days between an adolescent's first (“index”) treatment visit and second (“initiation”) visit significantly relates to likelihood of engagement. It is important to note, however, that because interactions between this variable and race/ethnicity indicators are also included in the model, the main effect of the variable applies without further equivocation for White adolescents only. The net results for other groups need to be calculated using the interaction terms as well. The interaction term between Latino and days to initiation is statistically significant and offsets the main effect. This suggests that, in total, days to initiation had virtually no net impact for Latinos on their likelihood of engaging in treatment. The interaction terms for Black adolescents and adolescents in the Other race category played a similar role for these groups. Although the interaction terms for these two groups were not significant, they do signal that the best estimates of the net effects of days to initiation for these two groups will be small and no longer significantly different from zero. Involvement in the other program factor, P4P, did not significantly impact treatment engagement.

3.4. Variance partitioning

Because site of treatment could impact initiation and engagement, our regression analyses also calculated the partitions in total variances that come from variation at the site level versus variation at the individual level (Goldstein, Browne, & Rasbash, 2002). These calculations show that for our initiation model, the portion attributable to the site was 7% with variation among individual clients accounting for the remaining 93%. For the engagement model, 4% of variance was attributable to the site, and the remaining 96% was attributable to individual clients.

4. Discussion

In this study, 76% of adolescents initiated, with 59% engaging. Although little comparable work has been done using the WC measures of initiation and engagement in adolescent samples, a similar study found comparable rates of initiation (75%) but slightly lower rates of engagement (50%; Godley, Hedges, et al., 2011). Another study of adolescents in Oklahoma that used the WC measures found an overall initiation rate of 64%, with an engagement rate of 48% (Lee et al., 2007). A recent study of adolescents participating in a four-group randomized clinical trial did not report initiation rates but found engagement rates between 65% and 76% (Godley et al., 2010), as compared with the rates in this study of 24% to 82%. As the number of studies grow using the WC measures of initiation and engagement, it will become increasingly clear that there are considerable differences in initiation and engagement rates among treatment sites, as is evident in this study. Despite the fact that the overall initiation and engagement rates were relatively high, there is room for improvement as the variation in site rates demonstrates. Such variation underscores the need to understand the reasons for differences, whether they are brought about by client characteristics, program attributes, or both.

4.1. Program factors associated with initiation and engagement

This study found two program factors that significantly influenced initiation or engagement. First, assignment to a P4P condition that provided monetary bonuses to therapists for superior treatment implementation significantly increased the number of adolescents who initiated treatment. This finding is particularly interesting given that the RTP experiment did not specifically target improvements in initiation.

Days to initiation, the other program factor of interest, was a significant main effect in the second stage of the Heckman procedure predicting engagement. Of note, the interaction of days to initiation with Latino also was significant, at almost the same magnitude but in the opposite direction. The significant main effect indicates that, among White adolescents, fewer days to treatment initiation was statistically associated with likelihood of engagement. The significant interaction term in the opposite direction indicates, however, that among Latino adolescents, no equivalent benefit was noted. The net effect of days to initiation for Black adolescents also needs further comment. The interaction between days to initiation and Black was not significant and only half as large as for Latinos (.031 vs. .064). When combined with the main effect, however, the net effect of number of days to initiation among Blacks, although still in a direction indicating an increased likelihood of engagement, is no longer significant.

Our findings confirm research in the realm of adult treatment indicating that decreasing time from first contact to intake/first service coincides with improved treatment retention (Hoffman, Ford, Choi, Gustafson, & McCarty, 2008) and that shorter wait time from assessment to first treatment appointment increases the likelihood of the client keeping the appointment (Claus & Kindleberger, 2002). Wait time can be experienced at two points when seeking treatment: wait time from first contact with the facility to assessment, and wait time from assessment to first treatment service (Carr et al., 2008). Longer wait time at either point in time may negatively impact treatment engagement. These findings render meaningful the disparity between sites in this study for days to initiation, which ranged at site level from a mean of 4.99 days (SD = 3.80) to 9.67 days (SD = 3.58).

Keeping in mind that this is a cross-sectional study, nonetheless, the results suggest some potentially fruitful directions for providers. Providers might prioritize efforts to decrease time between the client's index visit and the next service. They could, for instance, use treatment approaches such as P4P (Garner et al., 2010) or contingency management (Lott & Jencius, 2009) or change how they conduct business and deliver services, for example, by calling the client, conducting home visits, easing intake procedures, or expanding office hours. These changes are all under the control of the provider, and the results of this study indicate that such efforts would have a positive effect on likelihood of initiation and/or engagement. Such intervention and fine-tuning are well within the appropriate function of a substance abuse treatment provider.

4.2. Client factors associated with initiation and engagement

Disparities by client characteristics also were present. Our study found that some demographic characteristics, such as race/ethnicity, and behavioral characteristics, such as level of school truancy, were predictive of initiation and/or engagement. These results contrasted with another recent study of the AAFT initiative examining a variety of treatment process measures, including rates of initiation and engagement among adolescents by gender and race/ethnicity (Godley, Hedges, et al., 2011), which did not find significant differences based on either characteristic. Variation in findings of this study are likely because of differences in (a) sampling time frame, (b) study inclusion criteria (e.g., eligible organizations; controlling for startup and wind down of grants), (c) statistical methods, and (d) different covariates, including the number of days to initiation and P4P participation as a primary explanatory variable; as a result, this study found some significant differences by race/ethnicity for both initiation and engagement, but not by gender. Although programs tend to have more control over treatment processes than client characteristics, such as client motivation levels and other such barriers to successful treatment, programs need to understand the likelihood of specific population groups initiating or engaging in treatment as they plan on resources and interventions. Adolescents from a group identified as less likely to initiate into or engage in treatment may benefit from additional resources directed at keeping them in treatment longer.

Adolescents who made up the Other race/ethnicity category were less likely to initiate substance abuse treatment. Furthermore, adolescents with behavioral problems such as truancy were less likely to initiate into treatment. In addition, as a main effect, Latinos were less likely to engage in treatment, and, as discussed above, the interaction of Latino with days to initiation indicates that reducing the time between index service and next service does not benefit all adolescents in engaging in treatment. By being aware that certain groups are not likely to return for further treatment, programs can take steps to try to understand the reason for this and implement changes that may increase their engagement in treatment. Possible barriers for adolescent clients may include travel distance between home and treatment site, insurance or payment issues, psychological discomfort and intimidation adolescents may perceive from having to attend a clinic, transportation issues (monetary and/or safety of public transportation in gang-involved urban areas), and parent work schedules or family crises that compete with transporting their adolescent to clinic appointments. Some of these may be targeted by programs and could lead to improved engagement rates.

4.3. Study limitations

The WC measures have not yet undergone much testing with adolescent populations; thus, this study furthers the work in this area. Limitations of this study pertain to data availability, generalizability of findings, sample and data collection methodology, and limited program variables. We selected variables to include in our models based on what is conceptually related to the dependent variables with consideration for data quality and dropped those that were highly correlated with another variable in the model. Furthermore, because our study relies on secondary data analysis, this limits our investigation to what is available in the data set. With all these considerations, there is always a chance of not including a variable that may be predictive of initiation or engagement. For example, socioeconomic status was not included; however, all the adolescents in the study were treated in public sector programs and were therefore from similar socioeconomic groups. This may limit generalizability to adolescents in other groups.

Other limitations relate to the sample and data collection methodology. Only the top-scoring grant applicants (13% of applicant sites out of 265 applications) received AAFT funding. It is possible that participant sites were relatively high performing so that variation in performance on initiation and engagement was more limited than if the entire range of AAFT applicants were included in the study or if nonapplicants had been included. Furthermore, participant sites were trained in procedures to keep the adolescent in treatment for more than 90 days. Despite the relatively high overall rates of initiation and engagement, performance rates at many of these sites still showed significant room for improvement. In addition, we note that the GAIN data were self-reported and thus relied on adolescent recall.

Moreover, although we feel that program factors are important to treatment quality, the data set that was used had only a limited number of program variables to examine. Thus, we were only able to examine the effects of program incentives and time to initiation.

5. Conclusions

The literature shows that retention in treatment is associated with better outcomes (Claus, Kindleberger, & Dugan, 2002; Hser et al., 2001; Simpson, 2001), and treatment initiation and engagement are important first steps that an adolescent needs to take before achieving longer retention. Both programmatic and client characteristics can influence initiation and engagement.

Programmatic characteristics explored here included days to initiation and use of P4P incentives to improve treatment implementation. This study demonstrates that how quickly adolescents are connected to treatment services upon entering substance abuse treatment can be critical to their engagement in treatment, although, as evidenced by the interaction of Latino and days to initiation, not all adolescents benefit from fewer days to initiation. In addition, monetary bonuses to therapists (i.e., P4P) can be effective in increasing implementation of proven treatment procedures, which, in turn, may increase the likelihood of initiation.

Investigation of predictors that affect the likelihood of initiation and engagement in treatment also are beneficial to providers because it helps them to identify characteristics of adolescents who are at risk for dropping out of treatment early. Client characteristics such as high truancy and being from certain race/ethnicity subgroups were associated with lower likelihood of initiation or engagement. Although providers cannot influence client characteristics of the adolescents they serve, these findings suggest the importance of providers identifying high-risk adolescents and focusing additional initiation and engagement efforts on them.

In light of both the programmatic and client influences on treatment initiation and engagement, treatment providers can make changes in how they deliver services that would impact treatment processes such as initiation and engagement. For example, providers might take more assertive steps (e.g., meeting adolescent clients at school or their home) to increase adolescents’ likelihood of engagement and to decrease the time to the first treatment session after entering treatment. Providers also could implement approaches such as P4P.

Treatment programs can take steps to improve their service delivery, which will in turn improve the numbers of adolescents who initiate and engage in treatment, which is critical to longer retention. Performance measures are useful tools in tracking whether a specified level of service is delivered at the start of a new outpatient treatment episode and may contribute to improved delivery of care to adolescents with a substance use disorder.


This research was supported by the Robert Wood Johnson Foundation Grant 65078, the National Institute on Alcohol Abuse and Alcoholism (NIAAA) Grant R01-AA017625, and the Substance Abuse and Mental Health Services Administration's CSAT (TI17589, TI17604, TI17605, TI17638, TI17646, TI17673, TI17702, TI17719, TI17724, TI17728, TI17742, TI17751, TI17755, TI17761, TI17763, TI17765, TI17769, TI17775, TI17779, TI17786, TI17788, TI17812, TI17817, TI17830, TI17847, TI17864, TI19313, TI19323, and contract no. 270-07-0191). In addition, partial support for this work came from the NIAAA training Grant 2T32AA007567-16. Preliminary findings from this study were presented at the Addiction Health Services Research Conference, Lexington, KY, October 26, 2010, and at From Disparities Research to Disparities Interventions: Lessons Learned and Opportunities for the Future of Behavioral Health Services, Arlington, VA, April 7, 2011. Finally, the authors would like to thank Randy Muck and Jutta Butler of the CSAT (SAMHSA) for their promotion of EBPs for treatment of adolescent substance use; Dr. Susan Godley and the EBT Coordinating Center team for the A-CRA/ACC training and support provided; and Dr. Michael Dennis and the GAIN Coordinating Center team for the GAIN training and support provided.


  • Bantchevska D, Bartle Haring S, Dashora P, Glebova T, Slesnick N. Problem behaviors of homeless youth: A social capital perspective. Journal of Human Ecology. 2008;23:285–293. [PMC free article] [PubMed]
  • Bremer RW, Scholle SH, Keyser D, Houtsinger JV, Pincus HA. Pay for performance in behavioral health. Psychiatric Services. 2008;59:1419–1429. [PubMed]
  • Campbell CI, Weisner C, Sterling S. Adolescents entering chemical dependency treatment in private managed care: Ethnic differences in treatment initiation and retention. J Adolesc Health. 2006;38:343–350. [PubMed]
  • Carr CJ, Xu J, Redko C, Lane DT, Rapp RC, Goris J, et al. Individual and system influences on waiting time for substance abuse treatment. Journal of Substance Abuse Treatment. 2008;34:192–201. [PMC free article] [PubMed]
  • Chan YF, Dennis ML, Funk RR. Prevalence and comorbidity of major internalizing and externalizing problems among adolescents and adults presenting to substance abuse treatment. Journal of Substance Abuse Treatment. 2008;34:14–24. [PMC free article] [PubMed]
  • Claus RE, Kindleberger LR. Engaging substance abusers after centralized assessment: Predictors of treatment entry and dropout. Journal of Psychoactive Drugs. 2002;34:25–31. [PubMed]
  • Claus RE, Kindleberger LR, Dugan MC. Predictors of attrition in a longitudinal study of substance abusers. Journal of Psychoactive Drugs. 2002;34:69–74. [PubMed]
  • Commons M, McGuire TG, Riordan MH. Performance contracting for substance abuse treatment. Journal of Psychoactive Drugs. 1997;32:631–650. [PMC free article] [PubMed]
  • Dakof GA, Tejeda M, Liddle HA. Predictors of engagement in adolescent drug abuse treatment. J Am Acad Child Adolesc Psychiatry. 2001;40:274–281. [PubMed]
  • Daley M, Shepard DS, Tompkins C, Dunigan R, Reif S, Perloff J, et al. Randomized trial of enhanced profiling in substance abuse treatment. Administration and policy in mental health, (July) 2010 Retrieved from doi:10.1007/s10488-010-0306-z. [PMC free article] [PubMed]
  • Dennis M, Titus JC, Diamond G, Donaldson J, Godley SH, Tims FM, et al. The Cannabis Youth Treatment (CYT) experiment: Rationale, study design and analysis plans. Addiction. 2002;97(Suppl 1):16–34. [PubMed]
  • Dennis ML, Chan YF, Funk RR. Development and validation of the GAIN Short Screener (GSS) for internalizing, externalizing and substance use disorders and crime/violence problems among adolescents and adults. American Journal on Addictions. 2006;15(Suppl 1):80–91. [PubMed]
  • Dennis ML, White M, Titus JC, Unsicker J. Global Appraisal of Individual Needs (GAIN): Administration Guide for the GAIN and Related Measures (Version 5) 2008 Available from
  • Ensign J, Bell M. Illness experiences of homeless youth. Qualitative Health Research. 2004;14:1239–1254. [PubMed]
  • Funk RR, McDermeit M, Godley SH, Adams L. Maltreatment issues by level of adolescent substance abuse treatment: The extent of the problem at intake and relationship to early outcomes. Child Maltreatment. 2003;8:36–45. [PubMed]
  • Garner BR. Research on the diffusion of evidence-based treatments within substance abuse treatment: A systematic review. Journal of Substance Abuse Treatment. 2009;36:376–399. [PMC free article] [PubMed]
  • Garner BR, Godley SH, Bair CM. The impact of pay-for-performance on therapists’ intentions to deliver high-quality treatment. Journal of Substance Abuse Treatment. 2011;41:97–103. [PMC free article] [PubMed]
  • Garner BR, Godley SH, Dennis ML, Godley MD, Shepard DS. The Reinforcing Therapist Performance (RTP) experiment: Study protocol for a cluster randomized trial. Implementation Science. 2010;5:1–12. Retrieved from 10.1186/1748-5908-5-5. [PMC free article] [PubMed]
  • Garnick D, Lee M, Horgan C, Acevedo A, Botticelli M, Clark S, et al. Lessons from five states: Public sector use of the Washington Circle performance measures. Journal of Substance Abuse Treatment. 2011;40:241–254. [PMC free article] [PubMed]
  • Garnick DW, Horgan CM, Lee MT, Panas L, Ritter GA, Davis S, et al. Are Washington Circle performance measures associated with decreased criminal activity following treatment? Journal of Substance Abuse Treatment. 2007;33:341–352. [PMC free article] [PubMed]
  • Garnick DW, Lee MT, Chalk M, Gastfriend D, Horgan CM, McCorry F, et al. Establishing the feasibility of performance measures for alcohol and other drugs. Journal of Substance Abuse Treatment. 2002;23:375–385. [PubMed]
  • Garnick DW, Lee MT, Horgan CM, Acevedo A. Adapting Washington Circle performance measures for public sector substance abuse treatment systems. Journal of Substance Abuse Treatment. 2009;36:265–277. [PMC free article] [PubMed]
  • Godley MD, Godley SH, Dennis ML, Funk R, Passetti LL. Preliminary outcomes from the Assertive Continuing Care experiment for adolescents discharged from residential treatment. Journal of Substance Abuse Treatment. 2002;23:21–32. [PubMed]
  • Godley MD, Godley SH, Dennis ML, Funk RR, Passetti LL. The effect of Assertive Continuing Care on continuing care linkage, adherence and abstinence following residential treatment for adolescents with substance use disorders. Addiction. 2007;102:81–93. [PubMed]
  • Godley SH, Garner BR, Passetti LL, Funk RR, Dennis ML, Godley MD. Adolescent outpatient treatment and continuing care: Main findings from a randomized clinical trial. Drug and Alcohol Dependence. 2010;110:44–54. [PMC free article] [PubMed]
  • Godley SH, Garner BR, Smith JE, Meyers RJ, Godley MD. A large-scale dissemination and implementation model for evidence-based treatment and continuing care. Journal of Clinical Psychology (New York) 2011;18:67–83. [PMC free article] [PubMed]
  • Godley SH, Godley MD, Karvinen T, Slown LL, Wright KL. The Assertive Continuing Care protocol: A clinician's manual for working with adolescents after residential treatment for alcohol and other substance use disorders. 2nd ed. Chestnut Health Systems; Bloomington, IL: 2006.
  • Godley SH, Hedges K, Hunter B. Gender and racial differences in treatment process and outcome among participants in the adolescent Community Reinforcement Approach. Psychology of Addictive Behaviors. 2011;25:143–154. [PubMed]
  • Godley SH, Meyers RJ, Smith JE, Godley MD, Titus JM, Karvinen T, Dent G, Passetti L, Kelberg P. The Adolescent Community Reinforcement Approach (ACRA) for adolescent cannabis users (DHHS Publication (SMA) 01-3489, Cannabis Youth Treatment (CYT) Manual Series, Volume 4) Center for Substance Abuse Treatment, Substance Abuse and Mental Health Services Administration; Rockville, MD: 2001.
  • Goldstein H, Browne W, Rasbash J. Partitioning variation in multilevel models. Understanding Statistics. 2002;1:223–231.
  • Harris A, Humphreys K, Bowe T, Tiet Q, Finney J. Does meeting the HEDIS substance abuse treatment engagement criterion predict patient outcomes? Journal of Behavioral Health Services & Research. 2010;37:25–39. [PubMed]
  • Harris A, Humphreys K, Finney J. Veterans Affairs facility performance on Washington Circle indicators and casemix-adjusted effectiveness. Journal of Substance Abuse Treatment. 2007;33:333–339. [PubMed]
  • Harris AH, Kivlahan DR, Bowe T, Finney JW, Humphreys K. Developing and validating process measures of health care quality: An application to alcohol use disorder treatment. Medical Care. 2009;47:1244–1250. [PubMed]
  • Harris AHS, Humphreys K, Bowe T, Tiet Q, Finney JW. Does meeting the HEDIS substance abuse treatment engagement criterion predict patient outcomes? Journal of Behavioral Health Services & Research. 2008;37:25–39. [PubMed]
  • Henggeler SW, Pickrel SG, Brondino MJ, Crouch JL. Eliminating (almost) treatment dropout of substance abusing or dependent delinquents through home-based multisystemic therapy. American Journal of Psychiatry. 1996;153:427–428. [PubMed]
  • Hoffman KA, Ford JH, III, Choi D, Gustafson DH, McCarty D. Replication and sustainability of improved access and retention within the Network for the Improvement of Addiction Treatment. Drug and Alcohol Dependence. 2008;98:63–69. [PMC free article] [PubMed]
  • Hser YI, Grella CE, Hubbard RL, Hsieh SC, Fletcher BW, Brown BS, et al. An evaluation of drug treatments for adolescents in 4 US cities. Archives of General Psychiatry. 2001;58:689–695. [PubMed]
  • Hudson A, Nyamathi A, Greengold B, Slagle A, Koniak Griffin D, Khalilifard F, et al. Health-seeking challenges among homeless youth. Nursing Research. 2010;59:212–218. [PMC free article] [PubMed]
  • Hussey DL, Drinkard AM, Falletta L, Flannery DJ. Understanding clinical complexity in delinquent youth: Comorbidities, service utilization, cost, and outcomes. Journal of Psychoactive Drugs. 2008;40:85–95. [PubMed]
  • Institute of Medicine . Crossing the quality chasm: A new health system for the 21st century. National Academy Press; Washington, DC.: 2001. [PubMed]
  • Institute of Medicine . Rewarding provider performance: Aligning incentives in Medicare. National Academies Press; Washington, DC.: 2007.
  • Lee MT, Garnick DW, Horgan CM, Panas L, Ritter GA, Davis S, et al. Adolescent substance abuse treatment initiation and engagement.. Paper presented at the Joint Meeting on Adolescent Treatment Effectiveness; Washington, D.C.. April 27.2007.
  • Lott DC, Jencius S. Effectiveness of very low-cost contingency management in a community adolescent treatment program. Drug and Alcohol Dependence. 2009;102:162–165. [PubMed]
  • McLellan AT, Kemp J, Brooks A, Carise D. Improving public addiction treatment through performance contracting: The Delaware experiment. Health Policy. 2008;87:296–308. [PMC free article] [PubMed]
  • National Committee for Quality Assurance [12/8/10];NCQA HEDIS 2011. 2010 from
  • National Quality Forum [12/8/10];NQF Endorsed Standards. 2009 from
  • Roozen HG, Boulogne JJ, van Tulder MW, van den Brink W, De Jong CA, Kerkhof AJ. A systematic review of the effectiveness of the Community Reinforcement Approach in alcohol, cocaine and opioid addiction. Drug and Alcohol Dependence. 2004;74:1–13. [PubMed]
  • Shane P, Diamond GS, Mensinger JL, Shera D, Wintersteen MB. Impact of victimization on substance abuse treatment outcomes for adolescents in outpatient and residential substance abuse treatment. American Journal on Addictions. 2006;15(Suppl 1):34–42. [PubMed]
  • Shepard DS, Calabro JA, Love CT, McKay JR, Tetreault J, Yeom HS. Counselor incentives to improve client retention in an outpatient substance abuse aftercare program. Administration and Policy in Mental Health. 2006;33:629–635. [PubMed]
  • Simpson DD. Modeling treatment process and outcomes. Addiction. 2001;96:207–211. [PubMed]
  • Slesnick N, Kang MJ, Aukward E. Treatment attendance among homeless youth: The impact of childhood abuse and prior suicide attempts. Substance Abuse. 2008;29:43–52. [PMC free article] [PubMed]
  • StataCorp . Stata Statistical Software: Release 11. StataCorp LP; College Station, TX: 2009. Heckman two-stage procedure.
  • Stewart M. Unpublished Dissertation. Heller School of Social Policy and Management, Brandeis University; Waltham, MA: 2009. Use of performance-based contracts in outpatient alcohol and drug abuse treatment.
  • Szapocznik J, Lopez B, Prado G, Schwartz SJ, Pantin H. Outpatient drug abuse treatment for Hispanic adolescents. Drug and Alcohol Dependence. 2006;84(Supplement 1):S54–S63. [PubMed]
  • Szapocznik J, Perez-Vidal A, Brickman AL, Foote FH, Santisteban D, Olga H, et al. Engaging adolescent drug abusers and their families in treatment: A strategic structural systems approach. J Consult Journal of Clinical Psychology. 1988;56:552–557. [PubMed]
  • Tims FM, Dennis ML, Hamilton N, Buchan JB, Diamond G, Funk R, et al. Characteristics and problems of 600 adolescent cannabis abusers in outpatient treatment. Addiction. 2002;97(Suppl 1):46–57. [PubMed]
  • Titus JC, Dennis ML, White WL, Scott CK, Funk RR. Gender differences in victimization severity and outcomes among adolescents treated for substance abuse. Child Maltreatment. 2003;8:19–35. [PubMed]
  • Vandrey R, Stitzer ML, Acquavita SP, Quinn-Stabile P. Pay-for-performance in a community substance abuse clinic. Journal of Substance Abuse Treatment. 2011;41:193–200. [PMC free article] [PubMed]
  • Waldron HB, Kern-Jones S, Turner CW, Peterson TR, Ozechowski TJ. Engaging resistant adolescents in drug abuse treatment. Journal of Substance Abuse Treatment. 2007;32:133–142. [PMC free article] [PubMed]
  • Waldron HB, Slesnick N, Brody JL, Turner CW, Peterson TR. Treatment outcomes for adolescent substance abuse at 4- and 7-month assessments. J Consult Journal of Clinical Psychology. 2001;69:802–813. [PubMed]