Search tips
Search criteria 


Logo of nihpaAbout Author manuscriptsSubmit a manuscriptHHS Public Access; Author Manuscript; Accepted for publication in peer reviewed journal;
J Subst Abuse Treat. Author manuscript; available in PMC 2011 January 1.
Published in final edited form as:
PMCID: PMC2789887

The Washington Circle continuity of care performance measure: Predictive validity with adolescents discharged from residential treatment


The current study examined the predictive validity of the Washington Circle (WC) continuity of care after long-term residential treatment performance measure, as well as the impact of assertive continuing care interventions on achieving continuity of care. This measure is a process measure that focuses on timely delivery of a minimal floor of services that are necessary to provide sufficient quality of treatment but should not be construed to be the optimal continuity of care after residential treatment for any specific adolescent. Participants included 342 adolescents who were admitted to long-term residential treatment and randomly assigned to either standard continuing care or an assertive continuing care condition. Overall, results provide initial support for the WC continuity of care after residential treatment performance measure as a useful predictor of 3-month recovery status. Additionally, assignment to an assertive continuing care condition was found to significantly increase the likelihood of achieving continuity of care.

Keywords: adolescent, continuing care, substance use treatment, performance measures, Washington Circle

1. Introduction

There is increasing recognition of the utility of process measures as tools to monitor and improve the quality of care in treatment systems providing care for individuals with substance use and mental health disorders (Garnick et al., 2002; 2007; 2009; Harris, Humphreys, Bowe, Kivlahan, & Finney, 2009; Harris, Humphreys, Bowe, Tiet, & Finney, 2008; Harris, Humphreys, & Finney, 2007; Herman & Palmer, 2002; Horgan & Garnick, 2005, McCarty et al., 2007; McCorry, Garnick, Bartlett, Cotter, & Chalk, 2000; McLellan, Chalk, & Bartlett, 2007). Two major advantages of using process measures to monitor quality of care are: 1) they are typically less expensive to collect relative to other types of quality indicators (e.g., post treatment assessment of improvement) and 2) treatment providers have a greater ability to influence such treatment-related process measures relative to post-treatment outcomes that generally are determined by multiple factors outside of a treatment provider's control. Despite these advantages, however, without knowing the extent to which such process measures are associated with client outcome measures, the ability to balance validity and feasibility in choosing a quality indicator is not possible (Harris, Humphreys, Bowe, Kivlahan et al, 2009; Harris, Humphreys, Bowe, Tiet et al., 2008; Harris et al., 2007).

In 1998, with funding from the Center for Substance Abuse Treatment (CSAT), a group called the Washington Circle (WC) was formed and charged with the task of developing and disseminating several performance measures (McCorry et al., 2000). Performance measures are methods or instruments used to evaluate the extent to which health care practitioners' actions conform to practice guidelines, medical review criteria, or standards of quality (AcademyHealth 2004). Initial WC efforts focused on developing measures for commercial health plans (e.g., identification, initiation, and engagement) and these measures have been adoption by the National Committee for Quality Assurance (NCQA) as part of their Healthcare Effectiveness Data and Information Set (HEDIS). Other organizations that have adopted the WC measures and have explored whether the achievement of the WC measures is related to meaningful improvements in client outcomes include the Veterans Administration (VA) and the Oklahoma Department of Mental Health and Substance Abuse Services (ODMHSAS) (Garnick et al., 2007; Harris, Humphreys, & Finney, 2007). However, as described below, validation studies of the WC private sector measures have produced mixed support.

Using ODMHSAS administrative data, Garnick and colleagues (2007) examined patient-level relationships between initiation and engagement in treatment with subsequent arrests/incarcerations among 5,169 adults treated in outpatient. Although initiation alone was not significantly associated with being arrested or incarcerated, individuals who initiated and engaged were significantly less likely to be arrested or incarcerated in the following year. Using a similarly large data set of 5,723 adults who had received substance abuse treatment from one of 110 VA programs, Harris and colleagues (2007) examined the relationship between case-mix adjusted facility-level improvements in subsequent alcohol and drug use and the WC performance measures. They found higher rates of initiation were significantly related to reductions in drug use (but not alcohol use) and that neither identification nor engagement was significantly related to reductions in alcohol or drug use. Sub-setting to the 2,789 (49%) individuals who had initiated (thus eligible to engage in treatment), Harris et al. (2008) conducted patient-level analyses to examine relationships between engagement and improvement in substance use and legal status. Consistent with findings by Garnick et al. (2007), individuals who met the criteria for engagement reported significantly greater improvements relative to those who did not engage. Although Harris et al. (2007) concluded that the WC engagement measure does appear to be related to important treatment outcomes when analyzed at the patient-level, they noted future research is still needed to validate the WC performance measures for discriminating quality at the facility-level.

In an effort to adapt the WC performance measures for use in the public sector, the WC Public Sector Workgroup was formed in 2004. In addition to adapting the existing WC measures (e.g., initiation and engagement) for use in public sector outpatient and inpatient settings, the group addressed the gap between the acknowledged importance of continuing care and the lack of any related performance measures. In the field of addiction treatment, continuing care has been used to indicate the stage of treatment that follows an initial episode of intensive care, including the ongoing active treatment after residential care (McKay, 2009). Closely related to continuing care, the term continuity of care refers to a treatment philosophy where the substance abuse treatment system is held accountable for the entire process of care for individuals with substance use problems (McCorry et al., 2000). As with other chronic disorders, long-term recovery depends on services that continue after discharge and how a treatment system delivers services to support clients post-treatment is a key indicator of performance. As described by Garnick et al. (2009), 12 states were involved in the specification of nine WC public sector performance measures. Four of these measures focused on initiation and engagement with respect to either outpatient or intensive outpatient, while the other five focused on continuity of care following: assessment, detoxification, short-term residential, long-term residential, and inpatient care. Pilot testing with six states demonstrated that many of these new public sector performance measures can be calculated from information routinely maintained by states' administrative data. The current study, however, is the first to examine whether achievement of the WC's continuity of care after long-term residential treatment performance measure (hereafter referred to as continuity of care) is significantly related to better treatment outcomes. Additionally, the current study is the first to examine the usefulness of the WC performance measures with a sample of adolescents and because of its randomized design, is the first to experimentally test the ability to directly impact achievement of the a WC performance measure. This latter point is particularly significant because it tests whether the performance measure is changeable and whether such change is related to clinical outcomes.

2. Method

2.1 Participants

Participants were recruited between January, 2004 and September, 2006 from Chestnut Health Systems (CHS), which is a publically funded treatment organization located in the Midwest. To be included in the study, adolescents had to 1) meet the Diagnostic and Statistical Manual of Mental Health Disorders (4th ed) (DSM-IV; American Psychiatric Association, 1994) diagnostic criteria for dependence on alcohol, marijuana and/or another drug in the past year; 2) be age 12-17 at intake; 3) stay at least seven days in residential treatment; and 4) be discharged to live in the community (vs. jail or prison) in one of the 10 catchment counties. Adolescents were excluded from study participation if there was: 1) evidence of a psychotic or an organic state of sufficient severity to interfere with the understanding of study instruments, continuing care procedures, or the informed consent process; 2) the adolescent was deemed dangerous to self or others during residential treatment; 3) the adolescent provided evidence of a pathological gambling disorder; or 4) the adolescent was already participating in another research study.

During the first week of admission to residential treatment, adolescents and an accompanying parent/caregiver were approached for voluntary participation in the study, including an explanation of the nature and conditions of the study as part of the informed consent process, which was approved by the CHS Institutional Review Board. As shown in Figure 1, 86% (342/396) of eligible participants signed an informed consent agreement to participate in the study. Additionally, follow-up rates ranged from 94% to 98%.

Figure 1
Participant flowchart

2.2 Study Design

Adolescents who agreed to participate in the research study and stayed at least one week in residential treatment were randomly assigned to either a control condition (Standard Care [SC]) or one of three assertive conditions (Assertive Care [AC]). The three assertive conditions included assertive continuing care (ACC; see Godley, Godley, Dennis, Funk, & Passetti, 2002, 2007), contingency management (CM; see Petry, Martin, Cooney, & Kranzler, 2000), or both ACC+CM. Randomization was conducted by the research coordinator using a computer random number generator to select one of 16 possible treatment-blocking sequences for blocks of four study participants. Although it was not possible to blind staff to condition, research staff with no connection to the treatment conditions administered all interviews.

2.2.1 Study Conditions Standard Condition

Adolescents who have planned discharge from residential treatment, which is determined via discharge criteria under the ASAM (1996) guidelines, typically are given recommendations for continuing care treatment in an intensive outpatient or regular outpatient program and for other services (e.g. attendance at self-help groups, medication monitoring). In general, an initial appointment for the continuing care service is made for adolescents discharged from residential treatment successfully. However, when adolescents are discharged against staff advice (e.g., running away; request early discharge) or at staff request (e.g., fighting; other disruptive behavior), only written recommendations are generally made to the family suggesting they make an appointment for continuing care treatment. Thus, continuing care under SC procedures are often passive and do not generally include follow-up to see whether adolescents actually initiate continuing care services. Assertive Condition

In addition to receiving the same procedures as adolescents in the SC, adolescents assigned to one of three assertive conditions also were assigned a clinician for 90 days following discharge from residential treatment regardless of their discharge status (i.e., planned or unplanned). Regardless of which assertive condition adolescents were randomized to, clinicians were responsible for initiating continuing care services by meeting with the adolescent in their home, school, or other convenient setting in the community as soon as possible following discharge from residential treatment. Transferring the responsibility for initiating continuing care from the adolescent to the clinicians was a major defining characteristic of assertive interventions. As relapse following residential treatment has been found to exceed 60% within the first 90 days (Brown, Vik, & Creamer, 1989; Godley et al., 2002), these assertive interventions were designed to provide rapid initiation of continuing care in the community in order to help reduce the incidence or consequences of relapse.

Adolescents assigned to the ACC condition received community reinforcement approach (CRA) procedures (Azrin, Sisson, Meyers, & Godley, 1982; Meyers & Smith, 1995) adapted for adolescents (A-CRA; Dennis et al., 2004) and case management procedures (Godley, Godley, Karvinen, Sloan, & Wright, 2006; Godley, Godley, Pratt, & Wallace, 1994). A-CRA is a behavioral therapy that focuses on rearranging environmental contingencies by attempting to make non-using (alcohol and other drug) activities and behaviors rewarding and thus compete with substance using behavior. It integrates an operant conditioning model with social skills training to teach adolescents new ways of coping without using alcohol or drugs and increasing pro-social activities (see Godley et al., 2001 for more details).

Adolescents assigned to the CM condition earned draws from a “fish bowl” of 510 slips (30% were smiley faces (no value), 62% were for small prizes worth approximately $1, 7% were for large prizes worth approximately $25), and 1% were jumbo prizes worth up to $150). Draws could be earned for providing alcohol/drug free urine specimens and for completing verifiable pro-social activities (Godley, Godley, Wright, Funk, & Petry, 2008). In order to promote continuous alcohol/drug abstinence and continuous completion of the pro-social activities, adolescents earned bonus drawings based upon the number of weeks in a row that they provided negative samples and completed activities. Adolescents who failed to attend a scheduled session (e.g., they are not home at the agreed upon time for the appointment) or attended the session but failed to complete (or verify) their activities or if they provided a sample positive for alcohol/drugs, the number of bonus draws earned reset to one until alcohol and drug-free specimens and/or both activities were completed two weeks in a row. Information related to the use of this contingency management approach has previously been provided by Petry and colleagues (e.g., Petry, 2000; Petry et al., 2000; Petry, Tedford, & Martin, 2001).

Adolescents assigned to the ACC+CM condition received both of the above procedures. Since CM has proven useful for increasing prosocial behaviors in adult studies (e.g., Morral, Iguchi, Belding, & Lamb, 1997; Petry et al., 2001), it was believed to be complementary to ACC, and the combination of ACC+CM might be more effective than either one alone.

2.3 Measures

Baseline and follow-up assessments were collected using the Global Appraisal of Individual Needs (GAIN; Dennis, Titus, White, Unsicker, & Hodgkins, 2003), which is widely used in adolescent substance abuse treatment evaluation. The GAIN's main scales have been shown to demonstrate good internal consistency (with alphas over .90 on main scales, .70 on subscales), test-retest reliability (Rhos over .70 on days/problem counts, kappas over .60 on categorical measures), and to be highly correlated with measures of use based on timeline follow-back methods, urine tests, collateral reports, treatment records, and blind psychiatric diagnosis (Rho of .70 or more, kappa of .60 or more; Dennis, Chan, & Funk 2006; Dennis, Dawud-Noursi et al., 2003; Dennis, Funk et al., 2004; Dennis, Scott, & Funk 2003; Dennis, Titus et al., 2002, 2003; Godley et al., 2002, 2004; Lennox et al., 2006a&b; Shane, Jasiukaitis, & Green, 2003). Copies of the instruments, manual, publications, and documentation on the 100 plus scales in the GAIN are available at Baseline assessments were administered during the first week of residential treatment, while follow-up interviews (including urine tests) were administered three months after the residential treatment discharge date. Adolescents were compensated $40 for the time and effort required to complete follow-up measurement interviews, as well as an additional $10 for completing interviews within one week of its due date.

2.3.1 Substance-related problems

In order to control for adolescents' baseline problem severity, the GAIN's substance problem scale (SPS) was used. This measure assesses problems related to substance use or substance using behavior and is based upon recency ratings (e.g., past month, 2-12 months ago, more than 12 months ago, never) for 16 symptoms: seven corresponding to DSM-IV criteria for dependence, four for abuse, two for substance-induced health and psychological problems, and three on lower severity symptoms of use (hiding use, people complaining about use, weekly use). The past month SPS symptom count has been shown to have good test-retest reliability (r =.70; Dennis, Babor, Roebuck, & Donaldson, 2002; Dennis, Dawud-Noursi, Muck, & McDermeit, 2003; Godley et al., 2002) and demonstrated good internal consistency with the current sample (coefficient alpha = .88).

2.3.2 Days in residential treatment

Adolescents' length of stay in residential treatment prior to discharge (measured in days) was controlled for. Based upon treatment records, it is calculated by taking the difference in days between the admission and discharge date.

2.3.3 Assertive Condition

In order to examine the impact of being in one of the three experimental assertive conditions relative to the standard condition, a dummy variable called assertive was created. The standard condition served as the referent, thus standard care condition was coded 0 and the three assertive care conditions were coded 1. The rationale for collapsing across the three assertive conditions was based upon preliminary analyses that indicated there were no significant differences between the three assertive conditions on meeting the WC residential continuity measure.

2.3.4 Continuity of care

Based upon the WC definition, continuity of care is a dichotomous measure indicating whether or not an adolescent had another service within 14 days after discharge from long-term residential treatment (0 = no and 1 = yes). This measure is a process measure that focuses on timely delivery of a minimal floor of services that are necessary to provide sufficient quality of treatment but should not be construed to be the optimal continuity of care after residential treatment for any specific adolescent.

2.3.5 Recovery status

Based upon one of the primary clinical outcomes used in the Cannabis Youth Treatment (CYT) study (see Dennis et al., 2004), recovery status is a dichotomous measure (0 = no and 1 = yes). In order to be considered being currently in recovery, there must have been no alcohol or other drug use, abuse, or dependence symptoms while living in the community (i.e., not incarcerated or in residential treatment) during the past 30 days at follow-up. In an effort to increase the validity of adolescent self-reported use, adolescents were asked prior to completing the substance use section of the GAIN if they had used any substances other than alcohol during the past 7 days). If the adolescent acknowledged use of any substance other than alcohol, a urine sample was deemed not necessary. If the adolescent denied use of any substance other than alcohol, a urine sample was tested on site and the results were provided to the participant before beginning the substance use section of the GAIN. Use of this procedure resulted in a false-negative rate of 7% for the study, which was not significantly different by condition and was lower than the 12.5% reported in our prior study (Godley et al., 2002). For the current study, if the adolescent had self-reported abstinence, but had a positive urine screen for drug use, their recovery status was adjusted from yes to no.

2.4 Analytic Plan

Similar to other studies focused on WC measures (e.g., Garnick et al., 2007; Harris et al., 2008) the current study focused on client-level analyses as opposed to facility-level analyses. A major reason for distinguishing between client- and facility-level analyses is that relationships observed at the individual level do not necessarily hold when aggregated to higher levels or vice versa (Robinson, 1950). For example, in a study by Harris, Humphreys, Bowe, Kivlahan, and Finney (2009), client-level analyses revealed that having at least two outpatient visits every 30 days for 90 consecutive days (i.e., retention) was not significantly associated with reduced substance use, however, facility-level analyses revealed VA programs with higher rates of retention had significantly less improvements in their average substance use outcomes. All client-level analyses were conducted following an intent-to-treat approach using SPSS version 15.0. Baseline differences between adolescents assigned to the two conditions of interest were examined using chi-square tests. Multivariate logistic regression analysis (Hosmer & Lemeshow, 1989) was used to predict 1) continuity of care and 2) recovery status 3-months post residential discharge. In addition to reporting at what p value each predictor was significant, we also report the corresponding odds ratio and the 95% confidence interval (CI).

3. Results

3.1 Participant characteristics

As shown in Table 1, the majority of adolescents were male (63%), Caucasian (70%), and between the ages of 15 and 16 (61%). Although four out of five (80%) reported having been in school during the past 90 days prior to residential treatment, even more (86%) reported having current involvement with the criminal justice system. Consistent with expectations for adolescents admitted to residential treatment, the current sample reported relatively high levels of dependence (68% reported marijuana dependence and 26% reported alcohol dependence), abuse (24% reported marijuana abuse and 32% reported alcohol abuse), and co-occurring psychological problems (80% reported at least one psychological problem in the past year with 63% reporting prior mental health treatment). Approximately half of adolescents reported involvement in risky behaviors such as having multiple sexual partners (44%) and having unprotected sex (46%).

Table 1
Baseline background/clinical characteristics and rates of continuity of care.

3.2 Predictors of continuity of care

As shown in Table 1, continuity of care was significantly higher among adolescents assigned AC (78%) compared to those assigned to SC (56%) and represents a moderate effect size (d = .60). Moreover, results of multivariate logistic regression (see Table 2) indicated the impact of being assigned to an assertive care condition remained significant (OR = 3.15, p < .001) even after controlling for substance-related problems (OR = 1.06, p < .10) and the number of days in residential treatment (OR = 1.01, p < .001).

Table 2
Predictors of continuity of care and recovery status at 3-follow-up.

3.3 Predictors of recovery status at 3-month follow-up

Overall, 34% of adolescents were classified as being in recovery at 3-months post residential discharge. As shown in Table 2, continuity of care was a significant predictor of 3-month recovery status (OR = 1.92, p < .05). This finding indicates adolescents who received another service within 14 days of their residential discharge had approximately a 92% higher likelihood of being in recovery at the end of the three month follow-up than adolescents who did not receive another service within this time frame. Additionally, adolescents' baseline substance-related problems were found to be a significant predictor of 3-month recovery status (OR = 0.93, p < .05), such that greater substance problem severity at baseline significantly decreased the odds of being in recovery at 3-months. Days in residential treatment and being randomized to one of the assertive conditions were not found to be significant in the model predicting 3-month recovery status.

4. Discussion

Overall, results provide support for the predictive validity of the WC long-term residential continuity measure. Specifically, multivariate logistic regression analysis revealed even after controlling for baseline substance-related problems, length of stay in index residential treatment, and assigned continuing care condition that adolescents achieving continuity of care criteria had approximately a 92% higher odds of being in recovery at the 3-month follow-up compared to adolescents who did not achieve this criteria. As one might expect, results also indicated adolescents with greater substance-related problems at intake were significantly less likely to be in recovery at the 3-month follow-up. Neither being assigned to the assertive conditions nor index residential treatment length of stay were found to be a significant predictor of 3-month recovery status in the multivariate model; though consistent with prior research on treatment retention (e.g., Simpson, Joe, & Brown, 1997; Simpson, Joe, & Rowan-Szal, 1997), the latter did have a significant positive zero-order relationship with 3-month recovery status.

In addition to providing support for the predictive validity of the continuity measure, the current study also provided experimental support for the hypothesized positive impact assertive continuing care conditions can have on continuity of care. Specifically, multivariate logistic regression analysis revealed adolescents assigned to the assertive continuing care conditions were much more likely to achieve continuity of care relative to adolescents assigned to the standard continuing care condition. What makes this finding more impressive was the relatively high rate of continuity of care achieved by adolescents in the standard condition (56%), which while very similar to the standard continuing care rate of 54% reported by Godley et al. (2007), is considerably higher than the 36% rate observed several years earlier as part of an evaluation of the same treatment system (Godley, Godley, & Dennis, 2001).

4.1 Limitations

It is important to acknowledge the limitations of the current study. First, outcome data are based upon self-report, however, urine test results were used to reduce false negative self-reporting and the procedure employed in this study reduced false-negative reports from an earlier trial (Godley et al., 2002). Nevertheless, it is possible that some false negative reports were not detected by urine tests. Second, because the current sample is composed mostly of Caucasian male adolescents from the Midwest, it is unknown to what extent the findings generalize to other samples. Third, many, perhaps most of the participants in the SC condition who initiated continuing care self-selected and to the extent that this is true, internal validity is compromised. This concern is at least partially mitigated because assertive procedures in the AC condition caused significantly more participants to achieve continuity of care and because results for continuity of care still significantly predicted adolescent recovery status at three months.

4.2 Clinical Implications

Findings of the current study have important implications for practice, research, and policy-making. First, results provide initial support for using the WC's residential continuity measure as a minimum practice standard for treatment providers to work towards. Contingent upon further predictive validation of its relationship to important client treatment outcomes (e.g., reduced substance use and criminal activity), achievement of continuity of care could potentially serve as an important target criterion upon which providers are not only held accountable for, but also rewarded for achieving (i.e., pay-for-performance [P4P]). Indeed, P4P is one approach that has been recommended by the Institute of Medicine (2007) and is has been applied in both general medicine (Rosenthal & Dudley, 2007; Rosenthal, Landon, Normand, Frank, & Epstein, 2006) and the substance abuse treatment field (Garner et al., 2009; McLellan, Kemp, Brooks, & Carise, 2008; Shepard et al., 2006). Second, as approximately 8 out of 10 AC participants met the WC criterion for continuing care initiation it suggests continuity of care can be achieved with significantly more adolescents than only those who are sufficiently motivated, organized, and resourced to attend clinic appointments. Although a growing body of research has demonstrated the ability to positively impact this important transition and treatment outcomes (Garner et al., 2007; Godley et al., 2002; 2007; McKay, 1999, 2001, 2005, 2009), the costs associated with such interventions may not be practical for treatment providers with limited resources. Cost-effectiveness studies are clearly needed to elucidate this issue. However, it does seem possible the incremental cost associated with assertive linkage could be offset by the cost savings associated with decreases in substance use and substance-related problems such as criminal activity, which has been shown to provide the greatest economic benefit of substance abuse treatment (McCollister & French, 2003).

4.3 Conclusions and Additional Directions for Future Research

The current study's findings indicated that assertive continuing care interventions significantly increased the odds of achieving the WC's residential continuity measure and that achieving continuity of care increased the odds of adolescents being in recovery 3-months post residential discharge. In addition to the directions already noted, future research is needed to conduct additional examinations of the predictive validity of the other WC continuity of care performance measures (i.e., continuity of care after assessment, detox, short-term residential, and inpatient) with other samples of adolescents and adults and to compare and contrast these findings. Should future studies provide additional support for the importance of continuity of care, research focused on identifying the mechanisms of change would be warranted. For example, future research should seek to identify the specific components of continuing care responsible for improved outcomes, as well as identify what the optimal dosage of continuing care is (i.e., both effective and cost-effective). Related to these recommendations is examining the role discharge status from the index level of care has on outcomes. For example, do continuity of care rates differ for individuals who successfully complete residential treatment and those who do not? Does discharge status moderate the relationship between continuity of care and treatment outcome, and if so to what extent can assertive or other continuing care approaches overcome these differences? While these recommendations listed above have been written specifically for measures of continuity of care, many of these recommendations also apply to the other WC performance measures. In the meantime, results of the present study provide initial support for monitoring the WC residential continuity for adolescents. More importantly, this study offers further support that all of the WC performance measures can be used for performance monitoring by treatment systems that maintain administrative data containing the necessary data elements (e.g., unique identifier, dates of service, level of care, diagnoses) for calculating the measures.


This work was supported by the National Institute on Alcohol and Alcoholism Abuse (NIAAA grant R01 AA10368), the National Institute on Drug Abuse (NIDA; Grant R01 DA018183), the Substance Abuse and Mental Health Services Administration through a supplement to the Brandeis Harvard NIDA Center on Managed care and Drug Abuse Treatment (Grant P50 DA 010233, and the Robert Wood Johnson Foundation (Grant 65078). The authors also acknowledge Christopher Roberts and Stephanie Merkle for their assistance in preparing this article. The opinions are those of the authors and do not represent official positions of the government.


Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

Contributor Information

Bryan R. Garner, Research Scientist, Chestnut Health Systems, 448 Wylie Drive, Normal, IL 61761; phone: 309-451-7809, fax: 309-451-7761; gro.tuntsehc@renraGRB.

Mark D. Godley, Director of Research & Development, Chestnut Health Systems, 448 Wylie Drive, Normal, IL 61761.

Rodney R. Funk, Research Associate, Chestnut Health Systems, 448 Wylie Drive, Normal, IL 61761.

Margaret T. Lee, Research Scientist, Institute for Behavioral Health, Schneider Institutes for Health Policy, Heller School for Social Policy and Management, Brandeis University, 415 South St., Waltham, MA 02454.

Deborah W. Garnick, Professor, Institute for Behavioral Health, Schneider Institutes for Health Policy, Heller School for Social Policy and Management, Brandeis University, 415 South St., Waltham, MA 02454.


  • American Psychiatric Association (APA) American Psychiatric Association Diagnostic and Statistical Manual of Mental Disorders. 4th. Washington, DC: APA; 1994.
  • American Society of Addiction Medicine (ASAM) Patient Placement Criteria for the Treatment for Substance-Related Disorders. 2nd. Chevy Chase, MD: ASAM; 2001.
  • Azrin NH, Sisson RW, Meyers R, Godley M. Alcoholism treatment by disulfiram and community reinforcement therapy. Journal of Behavioral Therapy and Experimental Psychiatry. 1982;13(2):105–112. [PubMed]
  • Brown SA, Vik PW, Creamer VA. Characteristics of relapse following adolescent substance abuse treatment. Addictive Behaviors. 1989;14:291–300. [PubMed]
  • Dennis ML, Babor T, Roebuck MC, Donaldson J. Changing the focus: The case for recognizing and treating marijuana use disorders. Addiction. 2002;97:S4–S15. [PubMed]
  • Dennis ML, Chan YF, Funk R. Development and validation of the GAIN Short Screener (GSS) for internalizing, externalizing, and substance use disorders and crime/violence problems among adolescents and adults. American Journal on Addictions. 2006;15:80–91. [PubMed]
  • Dennis ML, Dawud-Noursi S, Muck RD, McDermeit (Ives) M. The need for developing and evaluating adolescent treatment models. In: Stevens SJ, Morral AR, editors. Adolescent substance abuse treatment in the United States: Exemplary models from a National Evaluation Study. Binghamton, NY: Haworth Press; 2003. pp. 3–34.
  • Dennis ML, Funk R, Godley SH, Godley MD, Waldron H. Cross-validation of the alcohol and cannabis use measures in the Global Appraisal of Individual Needs (GAIN) and Timeline Followback (TLFB; Form 90) among adolescents in substance abuse treatment. Addiction. 2004;99:120–128. [PubMed]
  • Dennis ML, Scott CK, Funk R. An experimental evaluation of recovery management checkups (RMC) for people with chronic substance use disorders. Evaluation and Program Planning. 2003;26:339–352.
  • Dennis ML, Titus JC, Diamond G, Donaldson J, Godley SH, Tims FM, Webb C, Kaminer Y, Babor T, Roebuck MC, Godley MD, Hamilton N, Liddle H, Scott CK, The C.Y.T. Steering Committee The Cannabis Youth Treatment (CYT) experiment: rationale, study design and analysis plans. Addiction. 2002;97(Suppl 1):16–34. [PubMed]
  • Dennis ML, Titus JC, White M, Unsicker J, Hodgkins D. Global Appraisal of Individual Needs (GAIN): Administration guide for the GAIN and related measures. Bloomington, IL: Chestnut Health Systems; 2003.
  • Garner BR, Godley MD, Funk RR, Dennis ML, Godley SH. The impact of continuing care adherence on environmental risks, substance use and problems following adolescent residential treatment. Psychology of Addictive Behaviors. 2007;21(4):488–497. [PubMed]
  • Garner BR, Godley SH, Dennis ML, Funk RR, Godley MD, Shepard DS. The Reinforcing Therapist Performance (RTP) Experiment: An Overview. Presented at the 2009 NIH Conference on the Science of Dissemination and Implementation; Bethesda, MD. 2009. Jan,
  • Garnick DW, Horgan CM, Lee MT, Panas L, Ritter GA, Davis S, Leeper T, Moore R, Reynolds M. Are Washington Circle performance measures associated with decreased criminal activity following treatment? Journal of Substance Abuse Treatment. 2007;33:341–352. [PMC free article] [PubMed]
  • Garnick DW, Lee MT, Chalk M, Gastfriend D, Horgan CM, McCorry F, McLellan AT, Merrick EL. Establishing the feasibility of performance measures for alcohol and other drugs. Journal of Substance Abuse Treatment. 2002;23:375–385. [PubMed]
  • Garnick DW, Lee MT, Horgan CM, Acevedo A. Adapting Washington Circle performance measures for public sector substance abuse treatment systems. Journal of Substance Abuse Treatment. 2009;36:265–277. [PMC free article] [PubMed]
  • Godley MD, Godley SH, Dennis ML, Funk R, Passetti L. Preliminary outcomes from the assertive continuing care experiment for adolescents discharged from residential treatment. Journal of Substance Abuse Treatment. 2002;23:21–32. [PubMed]
  • Godley MD, Godley SH, Dennis ML, Funk RR, Passetti LL. The effect of Assertive Continuing Care (ACC) on continuing care linkage, adherence and abstinence following residential treatment for adolescents. Psychology of Addictive Behaviors. 2007;102:81–93. [PubMed]
  • Godley SH, Godley MD, Dennis ML. The Assertive Aftercare Protocol for adolescent substance abusers. In: Wagner E, Waldron H, editors. Innovations in adolescent substance abuse interventions. New York: Elsevier Science; 2001. pp. 311–329.
  • Godley SH, Godley MD, Karvinen T, Slown LL, Wright KL. The Assertive Continuing Care Protocol: A clinician's manual for working with adolescents after residential treatment of alcohol and other substance use disorders. Bloomington, IL: Lighthouse Institute; 2006.
  • Godley SH, Godley MD, Pratt A, Wallace JL. Case management services for adolescent substance abusers: A program description. Journal of Substance Abuse Treatment. 1994;11:309–317. [PubMed]
  • Godley SH, Godley MD, Wright K, Funk RR, Petry N. Contingent reinforcement of personal goal activities for adolescents with substance use disorders during post-residential continuing care. The American Journal on Addictions. 2008;17:278–286. [PubMed]
  • Godley SH, Jones N, Funk R, Ives M, Passetti MS. Comparing outcomes of best-practice and research-based outpatient treatment protocols for adolescents. Journal of Psychoactive Drugs. 2004;36(1):35–48. [PubMed]
  • Harris AH, Humphreys K, Bowe T, Kivlahan DR, Finney JW. Measuring the quality of substance use disorder treatment: Evaluating the validity of the Department of Veterans Affairs continuity of care performance measure. Journal of Substance Abuse Treatment. 2009;36:294–305. [PubMed]
  • Harris AH, Humphreys K, Bowe T, Tiet Q, Finney JW. Does meeting the HEDIS substance abuse treatment engagement criterion predict patient outcomes? Journal of Behavioral Health Services and Research. 2008 doi: 10.1007/s11414-008-9142-2. [PubMed] [Cross Ref]
  • Harris AH, Humphreys K, Finney JW. Veterans Affairs facility performance on Washington Circle indicators and casemix-adjusted effectiveness. Journal of Substance Abuse Treatment. 2007;33:333–339. [PubMed]
  • Horgan C, Garnick D. Background paper for the institute of medicine institute for behavioral health, Schneider institute for Health Policy, Heller School for Social Policy and Management, Brandeis University. 2005. The quality of care for adults with metal and addictive disorders: issues in performance measurement.
  • Hosmer DW, Lemeshow S. Applied logistic regression. New York, NY: Wiley; 1989.
  • Institute of Medicine. Improving the quality of health care for mental and substance-use conditions: Quality chasm series. Washington, DC: National Academy Press; 2006. [PubMed]
  • Rewarding provider performance: Aligning incentives in Medicare: Pathways to Quality Health Care Series. Institute of Medicine; Washington, D.C.: The National Academies Press; 2007.
  • Lennox R, Dennis ML, Ives M, White MK. The construct and predictive validity of different approaches to combining urine and self-reported drug use measures among older adolescents after substance abuse treatment. The American Journal on Addictions. 2006;15:92–101. [PubMed]
  • Lennox RD, Dennis ML, Scott CK, Funk RR. Combining psychometric and biometric measures of substance use. Drug and Alcohol Dependence. 2006;83:95–103. [PubMed]
  • McCarty D, Gustafson DH, Wisdom JP, Ford J, Choi D, Molfenter T, Capoccia V, Cotter F. The Network for the Improvement of Addiction Treatment (NIATx): Enhancing access and retention. Drug and Alcohol Dependence. 2007;88:138–145. [PMC free article] [PubMed]
  • McCollister KE, French MT. The relative contribution of outcome domains in the total economic benefit of addiction interventions: A review of first findings. Addiction. 2003;98:1647–1659. [PubMed]
  • McCorry F, Garnick DW, Bartlett J, Cotter F, Chalk M. Developing performance measures for alcohol and other drug services in managed care plans. Journal of Quality Improvement. 2000;26:633–643. [PubMed]
  • McKay JR. Studies of factors in relapse to alcohol, drug and nicotine use: A critical review of methodologies and findings. Journal Studies on Alcohol. 1999;60:566–576. [PubMed]
  • McKay JR. Effectiveness of continuing care interventions for substance abusers: Implications for the study of long-term treatment effects. Evaluation Review. 2001;25(2):211–232. [PubMed]
  • McKay JR. Is there a case for extended interventions for alcohol and drug use disorders? Addiction. 2005;100:1594–1610. [PubMed]
  • McKay JR. Continuing care research: What we have learned and where we are going. Journal of Substance Abuse Treatment. 2009;36:131–145. [PMC free article] [PubMed]
  • McLellan AT, Chalk M, Bartlett J. Outcomes, performance, and quality-what's the difference? Journal of Substance Abuse Treatment. 2007;32:321–330. [PubMed]
  • McLellan AT, Kemp J, Brooks A, Carise D. Improving public addiction treatment through performance contracting: The Delaware experiment. Health Policy. 2008;87(3):296–308. [PMC free article] [PubMed]
  • Meyers RJ, Smith JE. Clinical guide to alcohol treatment: The Community Reinforcement Approach. New York: Guilford Press; 1995.
  • Morral AR, Iguchi MY, Belding MA, Lamb RJ. Natural classes of treatment response. Journal of Consulting and Clinical Psychology. 1997;65:673–685. [PubMed]
  • National Committee on Quality Assurance. HEDIS 2008 technical specifications. Washington, D.C.: 2008. [February 2, 2008].
  • Petry NM. A comprehensive guide to the application of contingency management procedures in clinical settings. Drug and Alcohol Depend. 2000;58:9–25. [PubMed]
  • Petry NM, Martin B, Cooney JL, Kranzler HR. Give them prizes and they will come: Contingency management for treatment of alcohol dependence. Journal of Consulting and Clinical Psychology. 2000;68:250–257. [PubMed]
  • Petry NM, Tedford J, Martin B. Reinforcing compliance with non-drug-related activities. Journal of Substance Abuse Treatment. 2001;20:33–44. [PubMed]
  • Rosenthal MB, Dudley RA. Pay-for-Performance: Will the Latest Payment Trend Improve Care? The Journal of the American Medical Association. 2007;297:740–744. [PubMed]
  • Rosenthal MB, Landon BE, Normand ST, Frank RG, Epstein AM. Pay for performance in commercial HMOs. The New England Journal of Medicine. 2006;355:1895–1902. [PubMed]
  • Robinson WS. Ecological correlations and behavior of individuals. American Sociological Review. 1950;15(3):351–357.
  • Shane P, Jasiukaitis P, Green RS. Treatment outcomes among adolescents with substance abuse problems: The relationship between comorbidities and post-treatment substance involvement. Evaluation and Program Planning. 2003;26:393–402.
  • Shepard D, Calabro J, Love C, McKay J, Tetreault J, Yeom H. Counselor incentives to improve client retention in an outpatient substance abuse aftercare program. Administration and Policy in Mental Health. 2006;33:629–635. [PubMed]
  • Simpson DD, Joe GW, Brown BS. Treatment retention and follow-up outcomes in the Drug Abuse Treatment Outcome Study (DATOS) Psychology of Addictive Behaviors. 1997;11:294–307.
  • Simpson DD, Joe GW, Rowan-Szal GA. Drug abuse treatment retention and process effects of follow-up outcomes. Drug and Alcohol Dependence. 1997;47:227–235. [PubMed]