In routine practice, different stakeholders in mental health may wish to monitor adherence for a variety of purposes. One set of purposes pertains to the accountability of service providers and organizations for client outcomes. The director of an organization contracted to deliver a particular treatment to a specific target population may wish to use adherence data to demonstrate to service contractors (typically one or more public agencies) the treatment was indeed delivered. A clinical supervisor may wish to monitor adherence to one or more treatments or elements of treatment for the purpose of identifying clinician professional development needs; that supervisor and therapist may wish to monitor adherence with each client with the goal of using adherence feedback to guide treatment planning and speed progress for that client; and, the client may wish to know only that the clinician delivered the treatment he or she requested, irrespective of the treatment provided by the same clinician or program to other clients. Accordingly, in routine care, a desirable characteristic of an adherence instrument or set of instruments may be the extent to which its use for different purposes by different stakeholders can be valid.
As noted earlier and depicted in , however, the many decisions entailed in developing a valid and reliable instrument, generally follow from the clear definition of a paramount, if not single, use of the instrument. An ongoing services trial illustrates one attempt to develop fidelity instruments to meet multiple purposes. The study, conducted by Atkins and colleagues (Cappella et al. 2008
), evaluates the viability and effectiveness of a school mental health service model for children with disruptive behavior disorders in high poverty urban communities. The service model includes a mental health services team composed of community mental health agency providers, family advocates, and key opinion leader teachers whose differentiated intervention activities all coalesce around the goal of children's learning. Through a combination of professional development activities and in vivo classroom support for teachers, and parent groups and home visits for families, the mental health team introduces parents and teachers to evidence-based intervention tools designed to influence the empirical predictors of children's learning.
The investigators aimed to create a set of adherence tools for this multi-component service model that simultaneously (1) enabled the investigative team to measure adherence with rigor and examine statistically relations among adherence indicators and outcomes, and (2) provided partner schools and mental health agencies with clinical tools they could use to sustain implementation of the service components once the research ended. Observational measures were deemed too intrusive and expensive to be sustained in clinics and schools. Instead, a fidelity checklist to be completed by the pertinent service participants was developed for each component of the service model: Teachers completed checklists on every professional development meeting; parents reported on every parent group they attended; and, teachers and parents also reported on (a) the frequency and content of classroom and home support provided by the mental health team, respectively, and (b) their own use of recommended intervention tools and strategies. The interventionists—mental health providers and family advocates—reported on the content and perceived quality of clinical supervision designed to support the service model. To determine data collection frequency, the investigators tried to balance evidence regarding the limitations of individual recall over time, practice effects, and the response burden for teachers, parents, and mental health professionals juggling multiple demands in high stress settings. Ultimately, parents reported in the fall and spring on the frequency and content of support provided by their mental health team during the prior month, and on their own use of specific intervention tools. Teachers reported in the fall and spring on services provided by their mental health team, and reported bimonthly (on average) on their implementation of recommended intervention strategies. Finally, mental health providers and family advocates reported monthly on the content and quality of their most recent supervision session.
As to the second purpose of the measure—to support the continued service delivery in schools and clinics after the research ended - agency directors and school principals considered utilizing adherence data to monitor the activities of their staff in decision making related to salary and advancement (a high stakes purpose). Clinical supervisors and school support staff utilized adherence data to identify professional development needs and to structure data-informed dialogue, feedback, and support for their staff related to work performance and skills acquisition (lower stakes purposes).
A critical caveat, however, is that the validity and reliability of these tools remains to be determined, pending the results of data analyses once the trial has ended. These analyses will reveal the extent to which instruments designed both to index fidelity of implementation and to sustain implementation after research has ended can achieve both goals. Quality improvement research suggests the most effective data sources are those that can be used in the feedback loop for improvement and for evaluation. With few exceptions, fidelity data obtained in psychotherapy research trials have not been used in this way. The goals of measuring and improving fidelity need not be mutually exclusive, however, and the data from this trial may illuminate the extent to which a fidelity measurement approach informed primarily by the feasibility of use in routine care (i.e., efficiency) can also be effective (i.e., validly and reliably measure the service being implemented).