|Home | About | Journals | Submit | Contact Us | Français|
To review literature published during the past year relevant to identifying the best measures for monitoring progress in the treatment child and adolescent clients and their families.
The current literature shows an increasing focus on clinical utility in measure development as demonstrated by the recent emphasis on evidence based assessment. However, there is very little research on how the inclusion of monitoring might enhance clinician practice and ultimately youth and family outcomes. There is great promise in expanding our field of thinking beyond mere outcome measurement to a Measurement Feedback System (MFS) that provides timely feedback that is comprehensive and concurrent with treatment.
Investment in the development of MFSs is needed to enhance clinical judgment and increase effectiveness of treatment. Clinical utility and consumer appeal need to be key considerations for measures intended to be used in everyday clinical practice. Most importantly, we must harness the power of technology and advances in measurement to provide clinicians with the tools to use effectively the systematic data provided through frequent measurement with MFSs.
Much has been written about the importance of using outcome data in clinical practice but such advocacy is outmoded and of limited clinical usefulness. We believe that what is necessary for effective treatment is a Measurement Feedback System (MFS) that is comprehensive, administered frequently concurrent with treatment, and provides timely feedback to clinicians [1**].
Real world mental health treatment requires attention to the complexities inherent in the interaction among treatments, settings, therapists, and clients. In order for clinicians to make appropriate decisions concerning treatment they must be able to evaluate client progress in meeting outcome goals but also have valid knowledge concerning key clinical processes. Yet the available evidence indicates that clinicians are not accurate in making these judgments based solely on their own observations and experience [1**, 2, 3*].
MFSs can transform practice not only by supporting clinical decision making but also in helping assure us that treatments labeled as evidence based treatments (EBTs), when brought into community settings, actually are effective [4*]. A practical MFS must contain measures that are short, psychometrically sound, and are useful in everyday practice by clinicians. Further, MFSs should assess several domains by multiple reporters that include treatment progress (e.g., youth and family outcomes) and treatment processes (e.g., therapeutic alliance, treatment activities).
In most settings information concerning client progress resides completely within the clinician’s purview. Even supervision, when provided, is based upon the therapist’s report, not an independent assessment. While supervision during training may include direct observation, outside of training it typically relies on the clinician’s self-report. Moreover, it is rare for graduate students to receive instruction in decision-making skills. This dependence on clinical judgment alone persists despite the plethora of research over the last 60 years that demonstrates critical flaws in the clinicians’ intuition and observations of the therapeutic process [2, 3*]. While we believe the clinician’s training, experience, and education are central to guiding the ongoing therapeutic process, they are not sufficient. Clinical decision-making is typically based on multiple imperfect cues where errors in judgment will always occur [5*]. The addition of a MFS will enhance clinical decision-making by providing systematic feedback the clinician could use to make incremental adjustments to the treatment plan.
A study of clinician- and research-based diagnostic agreement provides compelling evidence that agreement predicts better therapy engagement and treatment outcomes [6**]. We agree with the authors’ suggestion to directly measure therapeutic alliance and engagement in addition to outcomes. Incorporating standardized measures into clinical practice settings may improve youth outcomes, but the costs involved in instituting structured diagnostic interviewing into clinical practice settings may present a significant barrier. However, a recent study found the use of semi-structured diagnostic interviews both feasible and effective in identifying youth in need of mental health services in the school setting . An innovative approach to utilizing the principles of structured and semi-structured interviewing in real world mental health settings is the use of Bayesian logic to create a dynamic system using diagnostic base rate information to select interview content [8*]. In an initial feasibility study, these authors found that the dynamic system reduced administration time yet maintained accuracy.
Few mental health clinicians are accountable for the quality or outcomes of their services since payment for services is usually based on treatment type, length, and location. One of the main purposes of MFSs is to bring such accountability to the provision of mental health services. The “stick” of accountability can also be accompanied by the “carrot” of incentives for providing effective services. For example, the United States system of Medicare provides financial incentives for physicians to participate in a quality reporting initiative . Of note, there is need for continued discussion of how quality indicators, such as outcomes, should be used to rate performance and determine funding criteria. Client and agency characteristics that may affect outcomes should be considered in any ‘pay for performance’ system [10*].
With support from the United States Substance Abuse and Mental Health Services Administration, several states are implementing some form of a MFS such as Massachusetts, Utah, and Hawaii [1**] although states vary widely in their decisions about what data to collect, how to provide feedback, and how to make the feedback useful [11*, 12**]. A recent case study details the process for implementing evidence-based assessment for adolescent substance abuse treatment at the state level [13**], providing a thorough review of issues faced at the external, organizational, and individual staff levels. It has been suggested that both state and federal government should take a more active role in fostering accountability through the promotion of outcomes management systems to support clinical decision making .
It is unlikely that traditional EBT research will be able to identify the exceedingly complex combinations of variations in treatments that will be most advantageous for which clients in which settings. A modular approach to identifying common elements or specific strategies across EBTs is a relatively new focus in the field [15**, 16**] with significant implications for research and practice. Identifying effective strategies within and across EBTs can lead to more flexibility in individually tailoring treatment. MFSs offer an additional approach for individualizing treatments. Systematic and frequent measurement of treatment progress and processes is a key component to promoting a practice-based evidence in a continuous quality improvement framework [1**]. Kazdin [17**] echoes this argument in a recent review of the continuing gap between research and practice in which he notes that clinical work can contribute directly to the scientific knowledge through the large amount of information that can be gathered in a MFS. He argues strongly for the use of systematic measures of client progress as a way to promote high quality care by individualizing treatment, ongoing monitoring of treatment effects, and complementing clinical judgment through systematic evaluation.
Measures intended to be used in busy clinical settings must meet the standard scientific psychometric criteria as well as several practical considerations. In addition to being reliable and valid, measures should be sensitive to change and provide interpretable change indicators. They should be clinically useful and acceptable to consumers. To make routine assessment feasible, measures should be short [1**, 17–20].
For over a decade there has been an increasing emphasis on evidence based assessment (EBA), focusing on the clinical utility of measures to guide interventions rather than just upon their use in research [21**, 22**]. A recent special issue of the Journal of Pediatric Psychology (2008) reviewed the evidence base for assessment of medical treatment adherence, pain, coping and stress, psychosocial adjustment and psychopathology, and other problems related to pediatric psychology. While some measures were identified as meeting the criteria as well-established assessments , there were several suggestions for improvement in the areas of assessing cultural differences, multi-dimensional assessment, and integration into clinical practice (e.g., comprehensive treatment planning) [24–27].
While not incompatible, it is important to note that there are different criteria for determining the quality of EBA. For example, the approach used in the special issue mentioned above was similar to that used to evaluate empirically supported treatments (ranging from promising to well-established). Hunsley and Mash [21**, 28] advocate an approach that specifies levels of reliability, including norms, and several forms of validity, including treatment sensitivity and clinical utility. For example, clinical utility is rated not only by consideration of costs, ease of use, etc. but also published evidence of demonstrable clinical benefit.
In contrast to the outmoded typical pre-post outcomes measurement, monitoring client progress through frequent administration of measures necessitates the use of measures that are sensitive to change over time in addition to being valid and reliable. It is difficult for a practicing clinician to determine what is clinically meaningful by simply ‘eyeballing’ scores. Bost and colleagues [29*] propose four statistical strategies to assist clinicians in determining whether a change in scores is meaningful, including the Reliable Change Index (RCI), RCI adjusted for systematic bias (RCIadj), bivariate regression (BIV), and multivariate regression (MIV). The authors note that RCI is the simplest and most straightforward of the four approaches, assisting the practitioner in determining whether a change in client scores is due to chance. Because RCI is disadvantaged by not accounting for systematic biases such as client demographic characteristics or practice effects, the latter three strategies are proposed as additional indices.
While these are not new statistical techniques, their application is relatively new in clinical practice. Few measures are designed to be administered frequently, and even fewer published studies include consideration of sensitivity to change. Providing clinicians with reliable and valid indices of clinically meaningful change may well serve to promote perceived usefulness of a MFS. Practically, however, this would require the use of computerized technology to make such information rapidly available and feasible in a busy clinical setting. Further, sensitivity to change implies sensitivity to change caused by treatment. The most direct way to assess this is to compare scores from individuals who have received a known effective treatment in the real world to those who received no treatment or treatment as usual. Unfortunately, there are few treatments for children and adolescents that have shown be consistently effective . Finally, it is important to attend to the clinical significance of any change as opposed to just statistical significance [31**].
In addition to the above criteria, MFSs should measure broadly; in other words, assess several domains relevant to youth outcomes through a multidimensional battery with multiple reporters. Domains should include not only traditional standards of treatment progress (e.g., symptoms and functioning) but also indices of treatment process (e.g., therapeutic alliance, session engagement, therapeutic activities).
Recent studies have provided intriguing evidence that factors common to any treatment (as compared to those specific to an EBT) are associated with outcomes in both youth and adult psychotherapy. For example, the finding that a substantial amount of youth improvement occurs very early in treatment supports this common factors perspective, arguing that specific treatment effects had not had sufficient time to be effective . Similarly, studies using more sophisticated analytic techniques have confirmed the existence of a therapist effect . However, the field is deficient in the number of good studies that link process with outcomes.
Research on therapeutic alliance, the most popular construct with over 6000 (almost all with adults) contains less than a handful of experimental studies. However, it has been shown to be modestly but robustly related to youth outcomes in correlational studies despite continued lack of consensus on how to conceptualize and measure alliance [34*]. One study of therapeutic alliance in youth psychotherapy found that the link between alliance and youth outcomes varied by type of treatment, with a weak correlation in nondirective supportive therapy compared to a stronger correlation between alliance and youth outcomes in more structured cognitive behavioral treatment . A second study found low consistency among reporters of alliance , consistent with our own research as yet unpublished where we have found that clinicians are not accurate at estimating youth and parent alliance. These two studies (and our own research) highlight the idea that therapeutic alliance is important to include in any ongoing measurement given the discrepancies between reporters and the potential influence on alliance of different kinds of treatment.
While measures of session content are usually part of an EBT’s treatment adherence monitoring, typical treatment in community settings is a “black box” of unknown dimensions. Attempting to describe non-EBT is a daunting task but one of the few examples can be found in a study by Bearsley-Smith and colleagues [37**]. Using a very careful developmental approach they adapted a measure that describes techniques utilized in treatment that was reported to be a valuable tool for viewing themes and trends in individual treatment. Of particular note in the development process was the inclusion of mental health practitioners who would be typical users of the measure. Our approach to measuring session content differs, focusing on topics addressed during treatment rather than specific strategies or techniques [38*]. There is clearly a great deal of development and refinement necessary to measure what happens in the typical treatment. Inclusion of such a measure is an important part of a MFS, particularly in meeting the increasing demand for documentation and identifying aspects of treatment as usual as potentially promising interventions [39*].
Systematic feedback that uses valid, reliable, and standardized measures has been found to have substantial and replicable benefits in adult mental health treatment [17**, 40, 41], and is currently being tested in youth mental health practice settings [1**, 17**]. Typical availability of computers in practice settings makes the technology readily available to take advantage of the advances in psychometrics that make feedback feasible.
Although the use of a MFS potentially carries great clinical benefits, these benefits cannot be realized if it is not used. Currently, the field knows very little about how best to implement and sustain such interventions, particularly in how to include feedback on individual client progress and change over time. The success of outcomes management and feedback hinges on a complex mix of organizational- and clinician-level factors. Of the few studies to date, significant barriers to implementation have been identified that contribute not only to a lack of adherence to protocols (e.g. completion of measures) but to a general lack of utilization in supervision or clinical decision-making [1**]. Comprehensive and collaborative efforts are needed from policy makers, researchers, and clinicians to promote the widespread use of MFSs to improve mental health care for children and adolescents. Further research is needed to develop and refine measures of treatment process and outcomes for routine clinical use and learn how best to sustainably implement MFSs. Mental health policy reform should make resources available for harnessing technological advances to support the day-to-day use of MFSs and careful consideration of how to use them in determining indicators of quality and performance. Clinicians must be willing to change the way they practice by actively engaging in the development and testing of MFSs to ensure clinical utility. Stakeholders need to collaborate in an epistemological shift in what we consider appropriate approaches to assessing quality of mental health care—beyond outcomes monitoring—to the active use of systematic data to inform ongoing clinical decision-making.
Preparation of this article was partially supported by grants from NIMH (MH068589-01) and the Lowenstein Foundation. Drs. Bickman and Kelley report that they and Vanderbilt University have a financial agreement with Qualifacts Systems Inc. for the online development of CFIT.