|Home | About | Journals | Submit | Contact Us | Français|
The Washington Circle (WC), a group focused on developing and disseminating performance measures for substance abuse services, developed three such measures for private health plans. In this article, we explore whether these measures are appropriate for meeting measurement goals in the public sector and feasible to calculate in the public sector using data collected for administrative purposes by state and local substance abuse and/or mental health agencies. Working collaboratively, twelve states specified revised measures and six states pilot tested them. Two measures were retained from the original specifications: initiation of treatment and treatment engagement. Additional measures were focused on continuity of care after assessment, detoxification, residential or inpatient care. These data demonstrate that state agencies can calculate performance measures from routinely available information and that there is wide variability in these indicators. Ongoing research is needed to examine the reasons for these results which might include lack of patient interest or commitment, need for quality improvement efforts, or financial issues.
The Washington Circle (WC) was formed in 1998 with the goals of developing and disseminating performance measures for substance abuse services. These efforts were explicitly endorsed by the Institute of Medicine which recommended the continued development and use of substance abuse performance measures in its recent report, Improving the Quality of Health Care for Mental and Substance Use Conditions (Horgan & Garnick, 2005; Institute of Medicine, 2006). Initially, the WC proposed a continuum of care for substance abuse services and then specified and tested three performance measures - identification of adults with alcohol or other drug diagnoses, treatment initiation and treatment engagement (Garnick et al., 2002; McCorry, Garnick, Bartlett, Cotter, & Chalk, 2000). These WC performance measures have been widely used by private health plans, for which they were designed, and also by the Veterans Health Administration and several state agencies (www.washingtoncircle.org).
Working collaboratively with 12 states, the WC turned more recently to exploring the suitability in terms of matching measurement goals and feasibility of adapting these core measures for use by state and local substance abuse and/or mental health agencies. Focusing on performance measures is particularly important for substance abuse treatment funded by the public sector because this type of funding covers the largest share of specialty treatment for substance use disorders. In 2003, 77 percent of substance abuse treatment was financed by public sources, leaving only 23 percent of expenditures covered by private insurers, philanthropy, or out-of-pocket payments by clients or their families (Mark et al., 2007).
In this paper, we present the results of a two-year process of adapting the original three measures from their original application in private health plans for use by state and local behavioral health agencies. We describe the expanded set of measures, summarize key decisions in expanding the original measures, and show pilot testing results from six states.
The Washington Circle (WC), convened by the Substance Abuse and Mental Health Services Administration (SAMHSA), consists of a multidisciplinary group of providers, researchers, health plan representatives and public policymakers. The WC developed performance measures that are focused on the quantity and timing of substance abuse treatment services. First, the WC developed a model of the process of care in four domains representing a continuum of substance abuse services (prevention/education, recognition, treatment and maintenance) based on a chronic disease perspective applied to substance abuse (McCorry et al., 2000). Then the WC selected private health plans for the initial pilot testing because these plans have standardized datasets that make the testing more straightforward (Garnick et al., 2002).
Current users of these performance measures include the National Committee for Quality Assurance (NCQA), an organization whose mission is to improve the quality of health care and whose activities include accrediting health plans and publishing statistics that track the quality of care delivered by the nation’s health plans. NCQA adopted the WC identification, initiation and engagement measures to be included in its 2004 Healthcare Effectiveness Data and Information Set (HEDIS) (National Committee for Quality Assurance, 2007; National Committee on Quality Assurance, 2008). The Veterans Health Administration currently uses WC-like measures to monitor delivery of care (Harris, 2006). The three performance measures are:
The Washington Circle measures focus only on a segment of treatment for substance use conditions, so it is useful to consider them in the context of broader conceptual frameworks that encompass the dynamic stages of treatment and recovery as well as patient attributes, organizational characteristics of treatment programs, and the larger social context. For example, the Washington Circle measures correspond to the Texas Christian University (TCU) Treatment Model’s stage of “early engagement”(Simpson, 2004). Also, the Substance Abuse and Mental Health Services Administration has developed a model of Recovery Oriented Systems of Care (ROSC) in which treatment for substance use conditions is only one element in which systems (e.g., treatment/recovery, family, medical, housing/homeless, child welfare, criminal justice, educational) are integrated to offer a fully coordinated menu of services and supports to maximize outcomes (Addiction Technology Transfer Center Network, 2008).
The Washington Circle measures also can provide system markers for measuring the impact of evidence-based practices (EBPs) in increasing clients’ engagement in treatment for substance use conditions. Starting in the 1980s, documented quality deficiencies in general health care and rising health costs created a movement to promote the use of EBPs. In Crossing the Quality Chasm, a report on health care quality, the Institute of Medicine defines EBPs, in general, as “the integration of best research evidence with clinical expertise and patient values.”(Institute of Medicine, 2001) Building on the EBP movement in general health care during the 1990s, many organizations began to focus on EBPs in behavioral health. For example, the National Registry of Evidence-Based Programs and Practices (NREPP) is a repository designed to help both professionals in the field and the public to become better consumers by identifying scientifically tested programs and practices (Center for Substance Abuse Treatment). Other organizations focusing on EBPs include the National Institute on Drug Abuse, with a research-based guide outlining principles of effective treatment (National Institute on Drug Abuse, 1999), the National Institute on Alcohol Abuse and Alcoholism with a range of professional education materials, the American Society of Addiction Medicine (American Society of Addiction Medicine, 2008), SAMHSA’s Treatment Improvement Protocols (TIPs) (Substance Abuse and Mental Health Services Administration, 2008), and the American Psychiatric Association’s guidelines for treatment of substance abuse disorders (American Psychiatric Association, 2008).
In addition, in 2007, the National Quality Forum, a private, not-for-profit membership organization created to develop and implement a national strategy for healthcare quality measurement and reporting, developed a set of recommended treatment practices for substance use disorders that are supported by evidence of effectiveness. Among the eleven recommended practices for substance use conditions, the National Quality Forum included initiation and engagement in substance treatment (National Quality Forum, 2007).
In considering the WC process measures, it is important to recognize the differences between measures of process and of outcomes. The strengths and weaknesses of each of these types of performance indicators continue to be debated. A consensus seems to be emerging in the medical sector, however, that process and outcome measures are both important and that they complement each other in monitoring the delivery of quality of care (Horgan & Garnick, 2005; Krumholz, Normand, Spertus, Shahlan, & Bradley, 2007; Mant, 2001; McLellan, Chalk, & Bartlett, 2007).
These measures describe client status. In evaluating substance abuse treatment, outcomes that are frequently used include criminal justice activity, employment, substance use, and reconnection with family and community. In general, outcome measures are focused on goals, make intuitive sense and reflect accumulative aspects of care including providers’ special skills and expertise (Mant, 2001). There may be a time lag before such outcomes are apparent and follow-up with clients to collect outcomes, often expensive, requires much time and effort. Furthermore, improved outcomes often depend on patient compliance and other factors that are not under control of the provider. Outcome measures may suggest specific areas of care that require quality improvement, but further investigation often is necessary to determine how to influence outcomes.
These measures specify the treatment services that patients receive. Such a measure would be expressed as the percent of suitable candidates for a particular treatment who have that treatment in the appropriate timeframe. Process measures are used to assess adherence to clinical practices based on evidence or consensus. They offer two advantages: they are more immediately actionable; and they are often useful for identifying specific areas of care that may require improvement. Process measures can be relevant because they may be important for some proximal or in-treatment outcomes.
Some substance abuse research on the association of process and outcomes exists. Treatment completion is a commonly used process measure that has been applied to data on clients in several state-funded treatment systems. This research shows that treatment completion is associated with improved outcomes, such as decreased criminal justice involvement, higher wages, and lower readmissions (Alterman, Langenbucher, & Morrison, 2001; Arria & TOPPS-II Interstate Cooperative Study Group, 2003; Evans, Longshore, Prendergast, & Urada, 2006; Luchansky, Brown, Longhi, Stark, & Krupski, 2000; Luchansky, He, Krupski, & Stark, 2000; Luchansky, He, & Longhi, 2002; Luchansky, Krupski, & Stark, 2007; Luchansky et al., 2006; Wickizer, Campbell, Krupski, & Stark, 2000).
The exploration of the WC measures’ association with outcomes was built on the foundation of the earlier studies cited above. New projects that are aimed specifically at validating the WC measures are in the planning stages, but several studies already have been focused on the association of the WC process measures with outcomes of substance abuse treatment. Using data from Oklahoma, researchers found that clients who initiated a new episode of outpatient treatment and who engaged in treatment were significantly less likely to be arrested or incarcerated in the following year (Garnick et al., 2007). These results were replicated using data from Washington State (Campbell, forthcoming 2008). Also, in a study focused on adolescents in residential treatment, those who achieved continuity of care after their treatment had a significantly increased likelihood of being abstinent at the 3-month follow-up interview (Garner, Godley, Funk, Lee, & Garnick, 2008).
In 2004 the Washington Circle Public Sector Workgroup was formed to determine if the three WC measures - identification, initiation and engagement -- could be useful tools for ultimately improving continuity of care in the public sector treatment system. This WC workgroup set four tasks: 1) evaluating suitability of these three measures for meeting measurement goals about public sector clients, 2) adapting and potentially expanding the set of measures, 3) developing common specifications and 4) conducting pilot testing. In this paper, we present the revised measure specifications and the findings of this pilot testing.
The Public Sector Workgroup includes members from substance abuse/mental health agencies in twelve states (AZ, CT, DE, KS, MA, NY, NV, NC, OK, TN, VT, and WA), some local jurisdictions, federal officials from the Center for Substance Abuse Treatment, academic researchers, and the Washington Circle Policy Group. Decisions reflected the variety of participants’ experience in: providing treatment to clients and examining service provision patterns and how data are collected and maintained on state databases. In the fall of 2006, six states that are members of the Workgroup pilot tested the expanded set of performance measures for adults.
In this section, we specify the data required for pilot testing, outline the set of performance measures, and discuss two key questions that the Workgroup grappled with in defining the performance measures.
Availability of data is a key consideration in exploring the transferability of the WC measures from entities with an enrolled population (e.g. private health plans) to public sector agencies without enrolled populations. All twelve states participated in the conceptual discussions and development of measures specifications, although only some states (CT, MA, NY, NC, OK, and WA) pilot tested the measures using their administrative data on adults aged 18 and over. Information for the other states is not reported here for several reasons: either they did not collect the required data (VT, DE, NV), did not yet have the data or resources to conduct the analyses (AZ and KS are now participating in ongoing measure development), or information was only available for adolescents (TN). One state also had a change in leadership during the measure development process that led to decreased involvement in the WC Public Sector Workgroup (AZ). Three types of administrative data are required to calculate all three of the original measures.
All states typically collect admissions and discharge data that have the core data elements required by the Treatment Episode Data Set (TEDS) (SAMHSA, 2006) including client ID, level of care, demographics, substance use problem and frequency of use, and employment. For the pilot testing, all states used routinely collected administrative data on publicly funded admissions and discharges using episodes in their Fiscal Year 2005.
Private health plans typically collect information on each treatment encounter with a client through claims that are submitted for payment or through encounter-based data sets. Some states also collect encounter level data that is similar to the claims data maintained at commercial health plans and typically include details regarding services provided and the dates of service linked to a client identification number. With the exception of New York (see below), five of the states that conducted the pilot routinely collect encounter data, including the date of treatment service and type of service, which are needed to calculate the Washington Circle measures. New York collects the number but not the dates of treatment services that a client receives during an episode, so for the pilot the process measures for outpatient and intensive outpatient services in New York were adapted and the results are presented separately.
Private health plans and the Veterans Health Administration serve a defined population and keep enrollment files. This information is key for calculating an identification rate where: the numerator is the number of clients who receive substance abuse services or diagnoses during the year; and the denominator is the total number of enrollees for the year.
Public sector agencies generally do not serve an enrolled population so determining who should be in the denominator is much more complex. Generally states serve individuals who are uninsured, medically underinsured, low income, and those who meet other criteria that vary from state to state. Some state agencies serve those below a specified percent below poverty level (e.g., below 100 percent of the poverty line), while others serve all who request services. Although we wanted to estimate the population that is eligible to be served using federal population and income estimates, we could not find a common approach to estimate a denominator across all states to calculate the identification measure. Therefore, only the results of the initiation and engagement measures are presented in this paper, along with results for new measures that were developed for use in the public sector as described below.
The Workgroup expanded the original WC measures for initiation and engagement into nine measures to better reflect the public sector client population and state behavioral health agencies reporting needs. These measures are described here and summarized in Table 1. Two of the three original WC measures for private health plans were retained with minor adaptations (initiation and engagement for outpatient and intensive outpatient services). Other measures were newly developed for public sector populations building on the original WC continuity of care approach. For these measures, the WC’s original focus on the timing of treatment services was adapted to fit the information obtained from state substance abuse agencies. (More detailed specifications of the measures can be found at http://www.washingtoncircle.org.)
Table 2, shows detailed examples of client scenarios and provides explanations of how these scenarios relate to the definitions of the measures. The first and most straightforward example deals with outpatient initiation and engagement measures. The rest of the examples show combinations of client services that reflect the complexities of actual treatment patterns. These examples show that clients can qualify for multiple measures simultaneously, during which time they may meet the criteria for some measures and not for others.
Initiation measures and engagement measures are defined for two levels of care -- outpatient and intensive outpatient.
Initiation is defined as the percent of individuals, who have an index outpatient (OP) or intensive outpatient (IOP) service with no other substance abuse services in the previous 60 days and received a second substance abuse service (other than detoxification or crisis care) within 14 days after the index service.
Engagement is defined as the percent of individuals who initiated OP or IOP substance abuse treatment and received two additional services within 30 days after initiation.
To be included in the OP and IOP initiation and engagement measures, a client must be starting a new episode of care. The index service should be preceded by a 60-day service-free period during which the client had not received substance abuse treatment, as evidenced by claims or encounters with any diagnoses or services related to alcohol or drug abuse or dependence treatment. Detoxification or assessment can occur within this 60-day “service-free” period because they are not considered to be treatment. For the calculation of initiation and engagement rates, each additional OP or IOP service or each day of residential care counts as one day of service. Also, two or more OP or IOP services that occur on the same day count as one service.
Continuity of care is defined for five levels of care.
This measure is defined as percent of individuals who have a positive assessment for substance abuse and received another substance abuse service (other than detoxification or crisis care) within 14 days.
This measure is defined as percent of individuals who receive a detoxification service and received another substance abuse service (other than detoxification or crisis care) within 14 days of discharge from detoxification. Based on how data are maintained on its database, each state can decide between two methods to calculate the detoxification measure when there is a string of multiple detoxification services:
This measure is defined as percent of individuals, who have a stay that is followed by another service (other than detoxification or crisis care) within 14 days after discharge. This percentage is calculated separately after Short-Term Residential (STR), Long-Term Residential (LTR) or Inpatient (IP) Care in most states. All stays are included in the denominator because there is not a requirement for a new episode of care for this level of care.
Residential stays are classified as short- or long-term according to the type of program and not how long the client stayed. Substance abuse services -- outpatient or intensive outpatient --that occur on the last day of the residential stay should be included in the numerator. The rationale is that the residential facility has made the connection with the next level of care after discharge. Also, either a step-down to a less intensive level of care, a readmission to the same level of care, or a step-up to a more intensive level of care qualifies for continuity of care because no assumptions are being made about the appropriateness of the continuing care for this measure.
The Workgroup members reviewed preliminary data and grappled with numerous questions before reaching consensus about how to define the expanded set of measures. Two key decisions related to: first, the types of services included in the measures; and second, differences in measure specification across different levels of substance abuse treatment.
Not all the services that states offer to substance abuse clients can be viewed as services that initiate or engage the client in treatment. Certainly therapeutic services such as individual counseling or group counseling are always considered to be substance abuse treatment services. Some ancillary services, such as employment counseling and housing assistance, were also included because they involve face-to-face contact between the provider and client that supports rehabilitation. Generally, services provided without the client in attendance, such as clinician consultation and laboratory analysis, did not qualify for initiation or engagement. Some services were determined to be appropriate to start a treatment episode but could not count as an additional service for any of the measures. For example, detoxification services can start treatment episodes, but the Workgroup decided that neither further detoxification services nor other crisis care services could count as a subsequent visit because these services are not considered substance abuse treatment.
We developed separate measures by level of care in order to make the measures most useful to states and providers who could use more disaggregated information to target quality improvement efforts. We also concluded that the idea of an episode of care needed to be considered differently for each level of care. We reasoned that the measures focused on outpatient (OP) and Intensive Outpatient (IOP) services should include only clients who are starting new treatment episodes. For these OP and IOP clients, it is important to determine that the index service, the first treatment, is the start of a new episode and not the middle or near the end of treatment when frequent services may not be clinically appropriate. For all other services (assessment, detoxification, residential, and inpatient services), regardless of whether there is a preceding service-free period that would indicate the start of a new episode of care, continuity of care is important for monitoring quality of care.
The goal of this project was to develop specifications and test feasibility. In addition, comparisons across the selected states may be of interest. Table 3 shows the pilot test results for the five states with client encounter data. Formal testing of the statistical significance of differences between states is a function of sample size and the design effect of clustered data. However, in general terms, differences under 8 percent are not significant at the .05 percent level, under the assumption that a particular design effect is not greater than 10 (a relatively high value).
Initiation and engagements rates varied widely across states. Outpatient initiation rates ranged from 42 to 73 percent. Based on the measure definitions, engagement rates are always lower than initiation rates, and for outpatient services they ranged from 24 to 67 percent. For Intensive Outpatient services, initiation rates ranged from 44 to 88 percent and engagement rates ranged from 34 to 76 percent.
We conducted further analysis to determine the sensitivity of 14-day requirement for a next substance abuse services to meet the outpatient and intensive outpatient initiation specification. To do this, four states calculated the number of days from the index Outpatient service to the next service for each Outpatient episode of care. Figure 1 shows the distribution of days to next service for episodes with an index Outpatient service for each state. As can be seen in Figure 1, changing the initiation cut-off timeline to a few days before or a few days after 14 days would not change the initiation rates substantially because only very small percentages of clients had services in those surrounding days. The results lead to the same conclusion for Intensive Outpatient index services (results not shown).
Continuity of care after detoxification ranged from 19 percent in Oklahoma to 59 percent in Connecticut. We found that using the different methods (individual detoxification vs. grouping contiguous detoxifications) would alter the results minimally (results not shown).
Only two states were able to calculate continuity of care after assessment, North Carolina (48 percent) and Washington (35 percent) because the other states either did not routinely collect assessment information in their administrative databases or did not record information about whether the assessment resulted in a positive substance abuse diagnosis.
When calculating continuity of care rates after residential treatment services, some states separated short-term from long-term residential services based on the type of residential treatment program, not on the client’s length of stay.
As noted above, New York collects admission and discharge dates for all substance abuse treatment services and the number of services received between admission and discharge, but not the actual dates of those services. Therefore we adapted the measures in the following way: outpatient/intensive outpatient initiation was measured as having two or more days of service during the episode, regardless of length of stay; and engagement was measured as a length of stay of 30 days or more and at least 4 treatment services during the episode. These specifications were designed to approximate as closely as possible the specifications using encounter data.
Based on New York’s definitions, the initiation rate for Outpatient treatment was 83 percent and the engagement rate was 66 percent (N = 91,875). New York’s initiation rate for Intensive Outpatient treatment was also 83%, and the engagement rate was 57percent (N = 7,140). Continuity of care rates were 16 percent for assessment (N = 42,444) and 28 percent for detoxification (N = 96,686). New York does not distinguish between short- and long-term residential treatment, and their overall residential treatment continuity of care rate was 26 percent (N = 24,366). The inpatient treatment continuity of care rate for this state was similar, at 28 percent (N = 39,923).
It is feasible for some state substance abuse agencies to use routinely collected administrative data to calculate performance measures, as shown by our Washington Circle Public Sector Workgroup pilot project. The use of a common approach and a shared set of specifications allows for comparisons across states. Such comparisons should be done with caution, however, because of variation among states in terms of the types of services offered and data collected, which can influence the results. It is important to reiterate that the main goal of this study is not to focus on state to state comparisons. We focus rather on feasibility testing and explaining methods that can be used across multiple states and on identifying patterns of care in the treatment systems that cut across states. For example, one consistent pattern across states is that two measures (continuity of care after detoxification and after positive assessment) need attention in all the states in the pilot to improve continuity of care.
Nevertheless, these measures, used as performance management tools, can serve as a simple indicator of areas that require further examination. Initiation, engagement and continuity rates vary across providers and across states for a range of reasons including client motivation, systems barriers (such as lack of transportation support for clients), differences in the resources directed towards treatment of substance use conditions and quality improvement initiatives, differences in completeness of recorded data, prevalence of referral to self-help rather than formal treatment, or differences in quality of care provided. In our analysis, we did not focus on the extent that such influences contribute to the rates reported here. These measures might best be used, therefore, for tracking over time as part of quality improvement efforts within providers or states where variations in data collection or local treatment patterns and availability of resources may be less problematic.
The results we report here are the same or higher than reported for different populations. Using the original WC specifications developed for private health plans, the WC pilot testing reported initiation rates from 26 percent to 46 percent and engagement rates from 14 percent to 29 percent (Garnick et al., 2002). In the Veterans Health Administration annual report for Fiscal Year 2006, the initiation rate is 27.7 percent while the engagement rates is 8.6 percent (Harris, Bowe, & Humphreys, 2007). For private health plans reporting to NCQA, the initiation rate in 2005 was 44.5 percent and the engagement rate was 14.1 percent (NCQA, 2007). It is key to note, however, that these results are not directly comparable to the pilot results reported here because all levels of care are reported together for both NCQA and the Veterans Health Administration. In addition, the results reported here for continuity after detoxification are in the same range as the 26.9 percent reported using earlier data from 1996-1997 for Delaware, Oklahoma, and Washington (Mark, Vandivort-Warren, & Montejano, 2006).
State and local behavioral health agencies face challenges in implementing substance abuse performance measures. For states with adequate data systems, assessments of data quality are critical as is the need to balance states’ desire to develop their own methods with the possibility of making cross-state comparisons by using common methods. Faced with resource constraints, states also must evaluate the costs of calculating and disseminating performance measures in terms of the benefits that can arise from using performance measures as a basis for starting a conversation about quality improvement with policy makers, providers, client advocates, contracting with providers, sharing program-specific reports with providers, or offering programs comparisons with others in the state. As the Federal government places more emphasis on performance measurement, the need to develop indicators that can help to target quality improvement efforts may become more important to states. Of course, states face common barriers including lack of encounter-level data on existing data systems to support calculating performance measures such as those developed here, leadership instability that might erode ongoing support, or changing resource constraints that might preclude ongoing programming support.
Although not reported here, our results show variation across programs in client engagement as also has been reported by large-scale field studies (Broome, Simpson, & Joe, 1999; Gerstein et al., 1997; Thune, 2000). This variation is part of the impetus behind the reporting efforts reported below in two states that have already incorporated all or several of the WC measures and activities noted above into their ongoing efforts.
Since 2003, a quarterly report has been produced and posted on the web (www.odmhsas.org) that reports the results of performance measures for the eight regions in the state. In order to track changes over time, measures are compared to the state average for the last eight quarters. The May 2006 quarterly report shows higher rates than in 2005 and results include the following: initiation after detoxification (24.2 percent); engagement after residential treatment (12.2 percent); initiation after outpatient treatment (80.7 percent); and engagement after outpatient treatment (69.0 percent) (Oklahoma Department of Mental Health and Substance Abuse Services, 2006). Currently, the Washington Circle measures, along with the NOMs and other Oklahoma-specific measures, are provided to contracted treatment facilities in four new formats: a comparison of the current time period to the previous time period: a trend analysis, providing 12 quarters of data; a comparison of performance with all other providers for a specific level of care; and a detailed report providing information about specific clients in each measure.
In 2006, the state started to report on performance indicators for mental health and developmental disabilities in addition to substance abuse (North Carolina Department of Health and Human Services, 2006). The impetus for the quarterly report is the belief that human service systems should be held accountable for provision of services and progress toward set goals. Regular reporting helps to identify areas that are doing well and those that need attention at local and state levels. One of the performance indicators in North Carolina is timely initiation and engagement in treatment. For the first quarter of SFY 2006-2007, 58 percent of those receiving substance abuse services initiated treatment using the WC definition with a range from a low of 31 percent in one local management entity (LME) to a high of 80 percent in another LME. Forty percent engaged in treatment with the range among LMEs from a low of 6 percent to a high of 65 percent.
In addition, New York uses substance abuse performance measures in its system for county stakeholders, Connecticut is offering monthly reports to providers, and Massachusetts provides bi-monthly reports to state leadership.
Two other initiatives that are directly relevant to the WC Public Sector measures share our focus on improving quality of care.
This initiative relates to the WC performance measures in two ways. Most importantly, it offers a tool for providers who seek to improve their performance rates. Through a philosophy based on effective, efficient and consumer-oriented care that focuses on continuous improvement models, the NIATx initiative shows that improved access and retention in substance abuse treatment can be accomplished through changes in management and business practices of treatment programs (McCarty et al., 2007). NIATx uses a Plan-Do-Study-Act cycle to identify problems, develop solutions, implement the new process, and measure the resulting outcomes (Capoccia et al., 2007; McCarty et al., 2007; Wisdom et al., 2006). Through the application of process improvement strategies that reduced the number of days between client’s first contact with the system and first treatment service, reduction in no-shows and increased admission and retention resulted. Also, the WC measures reported on here and NIATx core measures (e.g., timely services and engagement or retention) are similar in concept. Both the NIATx and WC measures capture clients’ experience in the early part of their treatment experience. Both groups measure the proportion of clients with an assessment that become admitted. Once admitted to treatment, NIATx and the WC have developed measures that focus on their actual engagement in treatment within a specified time period post admission. For example, the NIATx continuation measure focuses on four units of service in 30 days and the WC measures of outpatient initiation and engagement focus on four units of service in up to 45 days.
This initiative will provide SAMHSA and the States with information that may be used for quality improvement. The NOMs contain 10 distinct domains including the following outcomes for substance abuse treatment: abstinence, employment (or school), criminal justice, housing, and retention in treatment. The WC measures do not track directly to the NOMs, but there are important complementarities. By measuring and reporting on initiation and engagement, states can contribute to improving the NOM for increased retention in treatment by offering information that will allow them to target providers, populations, or levels of services. Many studies of treatment retention indicate that substance abuse clients who remain in treatment for a longer duration of time have better outcomes (Hubbard, Craddock, Flynn, Anderson, & Etheridge, 1997; Hubbard, Craddock, & Anderson, 2003; McLellan, 1997; Simpson, 1995). Improvements in outcomes after treatment have been found for employment (Arria & TOPPS-II Interstate Cooperative Study Group, 2003; Koenig et al., 2005; Luchansky, Brown et al., 2000; Wickizer et al., 2000; Zarkin, Dunlap, Bray, & Wechsberg, 2002), criminal justice involvement (Hubbard et al., 2003; Luchansky et al., 2002; Luchansky et al., 2007; Luchansky et al., 2006), level of functioning (Conners, Grant, Crone, & Whiteside-Mansell, 2006; McLellan, 1997), and reduced alcohol and other drug use (McLellan, 1997). States’ implementation of measurement and reporting of the WC measures has the potential, therefore to contribute to more clients to becoming engaged in the early stages of treatment. Certainly this early engagement is a prerequisite to achieving longer lengths of stay or treatment completion, which in turn are associated with improvement in these four key NOMs.
Developing and pilot testing substance abuse performance measures for the public sector is only a first step in making them useful tools for performance management and quality improvement. Currently, only about a third of the states collect the detailed encounter data that is required for the optimal calculation of these measures. The concepts we propose are simple but their application to data and the interpretation of the results can be complex. Even for states with complete data on encounters, taking into account the nuances of treatment patterns requires some intricate calculations. Furthermore, these efforts require data collection, analytic support and funding that some states may find difficult to obtain. Nevertheless, between 2004 and 2006 a dozen states have been able to participate in the development of the measures and half of them have also participated in the empirical testing.
Additional efforts related to the Washington Circle measures are needed in several areas. First, immediately important is how states display results and communicate with providers. While these measures can be powerful tools for targeting quality improvement efforts, ongoing work is needed to further develop approaches to offering information to providers on a timely basis and using clear formats. Second,, more attention now is being paid to leadership and staff relations in provider organizations, drawing on emerging evidence for how treatment systems’ and counselors’ attributes can influence quality of services and institutional openness to innovations (Broome, Flynn, Knight, & Simpson, 2007; Greener, Joe, Simpson, Rowan-Szal, & Lehman, 2007; Joe, Broome, Simpson, & Rowan-Szal, 2007; Simpson, Joe, & Rowan-Szal, 2007). Thus, considering how Washington Circle measures fit into providers’ organizational context also is a fruitful area for further investigation. Finally, examining costs in relation to the Washington Circle measures will reveal the business case for using these measures. If reporting initiation and engagement results leads to targeted quality improvement efforts and to subsequent improvements in client outcomes, then there is the potential for savings not only in the cost of reduced readmission to substance abuse treatment but also reduced criminal justice, unemployment or other related costs.
The substance abuse treatment field continues to move in the coming decade towards understanding of the need for a recovery oriented system of care, recognition of substance use as a chronic condition for some individuals, and introduction of new payment approaches and quality improvement methods. The Washington Circle measures described here offer system markers for monitoring the impact of these changes over time in reaching the goals of improving interventions and service quality for individuals in need.
This project was supported by the Substance Abuse and Mental Health Services Administration through a supplement to the Brandeis Harvard National Institute on Drug Abuse (NIDA) Center on Managed care and Drug Abuse Treatment, Grant # P50 DA 010233 and National Institute on Alcohol Abuse and Alcoholism grant # R21 AA014229.