PMCCPMCCPMCC

Search tips
Search criteria 

Advanced

 
Logo of nihpaAbout Author manuscriptsSubmit a manuscriptHHS Public Access; Author Manuscript; Accepted for publication in peer reviewed journal;
 
Psychiatr Serv. Author manuscript; available in PMC 2012 May 1.
Published in final edited form as:
PMCID: PMC3281556
NIHMSID: NIHMS353277

Clinician Factors Related to Outcome Differences Between Black and White Patients at CMHCs

Abstract

Objectives

Data are limited on how clinicians contribute to outcome differences between black patients and white patients. Because the clinician-patient relationship is the foundation of mental health services, understanding clinicians’ role in outcome differences may help identify evidence-based interventions that decrease disparities and capitalize on positive differences. Symptoms and functioning in a sample of black and white adults receiving outpatient services were examined to determine the effects of their primary clinician on those patterns.

Methods

The study included 551 patients (25% black) with serious mental illness and 62 mental health professionals (21% black) identified as the patients’ primary clinician. Treatment outcomes were measured at baseline and two follow-ups (two and four months) with the Behavior and Symptom Identification Scale, a measure of symptoms and functioning. Data were analyzed with hierarchical linear modeling. Clinicians’ levels of multi-cultural competence, burnout, and education were analyzed.

Results

Clinicians moderated the relationship between patient race and outcome differences. There was significant variability among clinicians: approximately 20% had black patients whose outcomes were worse than those of their white patients, and 40% had black patients with better outcomes than their white patients. The only clinician factor predicting these differences was clinician's general experiences and relationships with people from racial-ethnic and cultural groups other than their own.

Conclusions

The occurrence of outcome differences varied across clinicians, with some clinicians magnifying outcome differences between black and white patients and others minimizing them. Factors other than clinicians’ race, multicultural competence, education, and burnout may contribute to outcome differences between black and white patients.

The foundation for most mental health services is the relationship between the clinician providing services and the patient receiving services (1). There is an a priori theory that for black patients this re lationship is less than optimal, mirroring the racial tensions in society and contributing to inferior treatment outcomes for black patients compared with white patients (2-7). This is based on an implicit assumption that black patients differ from white clinicians in their life experiences, which create subtle and complex personal constructions of racial identity that diverge significantly from those constructed by white clinicians (2-6).

Two clinician factors are frequently posited to either produce or ameliorate outcome differences for black patients: clinicians’ level of multicultural counseling competency (MCC) and their race (3,8-10). In a content analysis covering 20 years of MCC research, Worthington and colleagues (11) found some evidence that such competence improves clinicians’ skills with patients from nonwhite racial-ethnic and cultural minority groups. Two significant weaknesses can be found in these studies. First, 70% of the studies were descriptive field surveys. Second, 80% of the studies relied on “pseudo-patients,” leaving some question about the impact of MCC in naturalistic settings (11,12). Shin and colleagues (13) conducted a meta-analysis of research on racial matching and found that matched and unmatched clinician-patient dyads were similar in service retention and total number of sessions. In general, matching appears to be a weak approach to addressing disparities in mental health services, but no research has examined the impact of matching on patients’ symptoms and functioning over time.

Levels of clinician burnout and education have been shown to affect treatment outcomes (14-16). For this study we hypothesized that these factors also influence outcome differences—higher levels of burnout by decreasing the overall functioning of clinicians and thereby affecting outcomes negatively for black patients, who may require an extra effort to ensure the appropriate provision of services, and higher levels of education through greater exposure to MCC via classroom and continuing education experiences (17). The few studies that have examined clinician factors that may affect treatment outcomes for black patients have neglected to answer the following basic questions that are addressed in this study. Are clinicians empirically linked to the occurrence of outcome differences between black and white patients? Does a clinician's race and levels of burnout, education, and MCC have an impact on the occur-rence of outcome differences? What proportion of clinicians produce outcome differences between black and white patients?

Methods

Participants were drawn from 13 community mental health centers (CMHCs) (18-21) located in the Upper Mississippi River Basin (UMRB). The southern region of the UMRB, comprising southern Illinois, western Kentucky, eastern Missouri, and northern Arkansas, has one of the highest concentrations of rural blacks outside the deep South (22). The UMRB provides a unique opportunity to study racial differences in a predominantly rural context. The CMHCs are located in micropolitan statistical areas (micro areas) or core-based statistical areas (CBSA). Micro areas contain at least one urban center of between 10,000 and 50,000 people as well as “adjacent outlying counties” that are socially and economically integrated with the urban center through commuting (23). The CBSA is a more expansive category, containing a metropolitan statistical area and one or more adjacent micro areas that are integrated by close proximity.

Data were collected between 2007 and 2009. CMHC staff introduced interested patients to a research team member located on site who provided an overview of the study. After consent was obtained, baseline data were collected and arrangements were made for follow-up data collection. Up to two follow-ups were conducted at two-month intervals. Clinicians were invited to participate via mail and one group meeting held at each CMHC. All patients were in treatment at the time of the baseline measure. (There was no random or purposeful nonrandom assignment of patients to clinicians.) All methods were approved by the University of Illinois at Urbana-Champaign Institutional Review Board.

Instruments

Clinicians

Clinicians completed three standardized instruments: the Universal Orientation Scale (UOS), Multicultural Counseling Inventory (MCI), and Maslach Burnout Inventory (MBI). The 20-item UOS (24) and the 43-item MCI (25) both capture aspects of MCC. The UOS was developed to measure nonprejudice and has moderate positive correlations with measures of empathy and adequate test-retest and internal consistency reliability coefficients (24). The MCI is used in a wide body of MCC research and consistently demonstrates adequate psychometric properties (25,26). The four MCI subscales measure multicultural skills, awareness, relationships, and knowledge. The MBI is a widely used measure of burnout that has been normed on mental health workers (27). It has three subscales: emotional exhaustion, depersonalization, and personal accomplishment. Clinicians’ race and ethnicity and other demographic variables, such as gender and education, were self-reported.

Patients

Treatment outcomes were measured with the Behavior and Symptom Identification Scale (BASIS- 24), a self-report instrument that measures difficulty in several major symptom and functional domains (28-30). It is used across the diagnostic spectrum in outpatient settings to measure the outcome of interventions over time. It has been normed with black and white patients (28-30). Respondents are asked to indicate the degree of difficulty they have experienced during the past week. Patients’ race and ethnicity were self-reported, along with gender, marital status, education, and employment status.

Administrative data

Patients’ diagnosis, utilization of medication, number of months in service at the CMHC at the time of baseline measure, receipt of psychosocial rehabilitation services, and hours of service received during the 90 days before the final follow-up measure were gathered from administrative data. Patients were provided with a variety of service packages tailored to their needs, which may have included individual counseling, group therapy, psychosocial rehabilitation, or medication. The possible range of services available to any patient was similar across CMHCs. A variety of therapeutic models shaped these services; the most common modalities were cognitive-behavioral therapy and insight-oriented psychotherapy. Each patient's primary clinician was identified by use of administrative records.

Statistical analysis

The analysis was similar to analyses utilized in the therapist effects literature (31-36) and was conducted with HLM software, version 6.06 (37). Full maximum likelihood estimation was used, which permits model comparison tests of nonnested models (38). The model contained three levels: repeated measures of the BASIS-24 at level 1, patient demographic and service variables at level 2, and clinician variables at level 3. Conceptually, level 1 represents treatment outcomes measured over time for each patient. At level 2, each patient is described by his or her baseline score (intercept), treatment outcomes (slope), and demographic and service predictors. Level 3 represents the influence of clinicians on outcome differences between white and black clients.

Results

Clinicians

Table 1 presents data on clinician characteristics. Sixty-two clinicians who were treating at least one white and one black patient in the study were included in the analysis. The median number of patient participants per clinician was eight (mean± SD=8.89±5.24). The median number of black patient participants per clinician was two (mean±SD=2.26± 2.06). Clinicians provided therapeutic sessions, coordinated care, made referrals, and monitored patient progress.

Table 1
Characteristics of 62 primary mental health clinicians of 551 patients, by race of clinician

Patients

Table 2 presents data on patient characteristics. Of the 551 patients in the study, most relied on Medicaid and had received CMHC services for several years. A total of 340 patients (62%) completed the baseline interview and the two follow-up interviews. During the study, nine patients (one black) declined to continue participation. Black and white patients differed on a number of characteristics, including hours of services received during the study period; black patients received more service hours than white patients. Differences in demographic characteristics and service hours between black and white patients were included as controls in the final model.

Table 2
Characteristics of 551 patients receiving outpatient mental health services at 13 community mental health centers (CMHCs), by race of patient

Model comparisons

An unconditional growth curve model was fit to the data in the first step of the model-building process (Table 3, model 1). [Additional data from this analysis are presented in supplemental Tables 6 and 7 in an online appendix to this article at ps.psychiatryon-line.org.] This model included no predictors of treatment outcomes other than time; however, it did control for the clustering of observations within patients as well as the clustering of patients within clinicians (38-40). Each subsequent model was compared with the unconditional model to determine which provided the better fit. This was accomplished using a chi square test of the deviance statistics with degrees of freedom equal to the difference in the number of parameters estimated (38). The variance component associated with treatment outcomes (Table 3) denotes how well each model fit the data, with higher numbers indicating greater amounts of random (unexplained) variance between the observed and predicted treatment outcomes (38). Subsequent models sought to further reduce the unexplained variance through the inclusion of patient and clinician variables.

Table 3
Information on the fit of three models to predict treatment outcomes of black and white patients receiving outpatient mental health services

The second model represents the effect of patient-level variables on treatment outcomes (Table 3). [Tables 8 and 9 in the appendix at psychyatryonline.org present additional data.] Although this model controlled for the clustering of patients within clinicians, no clinician predictors were included (37,38). Compared with the unconditional model, model 2 explained an additional 27% of variability in treatment outcomes (40). The deviance statistics showed that model 2 also fit significantly better than model 1 (χ2=19.61, df=8, one-tailed p<.01).

Model 3, the final model, included all of the patient-level predictors identified in model 2 plus the clinician-level predictors (Tables 4 and and5).5). Compared with the unconditional model (model 1), the final model explained an additional 37% of variability in treatment outcomes and provid ed a significantly better fit to the data (χ2=29.67, df=21, one-tailed p<.05). Model 3 did not fit the data significantly better than model 2.

Table 4
Fixed-effects model predicting change in outcomes over time for black and white patients receiving outpatient mental health services
Table 5
Random-effects model predicting change in outcomes over time for black and white patients receiving outpatient mental health services

Model specification

All patient-level variables (Table 2) were tested for inclusion in the final model. Months in service and receipt of psychosocial rehabilitation services were excluded because of multi-collinearity with hours of service. Level of education, marital status, and living situation were excluded because they were not essential to the stability of the model and were not significantly associated with outcomes (41).

Of all the patient-level variables tested, only the effect of race significantly varied across clinicians; therefore, it was the only predictor treated as a random effect (39). Subsequent model building was dedicated to describing the random effect of race.

Level 1 model (repeated measures)

Time was coded to represent the measurement occasion (baseline=1, first follow-up=2, and second follow-up=3). The log-time model was selected because it provided a better fit than a linear model of time. The log transformation was necessary to meet the linearity assumption and merely changed the scale of the time variable (38) without altering other properties of the model.

Level 2 model (patient level)

Predictors of treatment outcomes at the patient level included a diagnosis of schizophrenia, male gender, use of psychotropic medications, being black, working in a paid or volunteer capacity, having public insurance, age, and the square root of hours of services (42). The original variable for service hours in the prior 90 days was highly skewed. Most of these predictors were coded and interpreted as dummy variables reflecting the presence or absence of that predictor; however, both age and root-hours of treatment were centered on the grand mean. For centered variables, each increment in a positive direction meant that the patient was one year older or received one root-hour more service than the “mean patient” across the entire sample.

The average patient BASIS-24 score at baseline was estimated at 1.49 (Table 4). The average treatment outcome was -.003, which did not differ significantly from zero. The average treatment outcome represented the expected change over time for a patient with values of 0 on all dummy variables and who was of average age and received an average amount of services. Patient-level parameters represent increments or decrements to the average treatment outcomes.

On average, the effect of patient race was not statistically significant; however, this effect was moderated by clinicians, with substantial variability among clinicians in the size and direction of the effect. The random effect of race had an acceptable reliability (Table 5). This finding answered the first research question by indicating that outcome differences did exist between white and black patients and that those differences were linked to clinicians.

Level 3 model (clinician level)

To answer the second research question, clinicians’ level of education, MCC, and burnout as well as their race and gender were included at level 3 to determine whether these variables explained the observed difference between white and black patients’ outcomes. MCI and UOS scores (MCC) were centered at the grand mean such that each unit increase or decrease in the parameter indicated a clinician's deviation above or below the mean among all clinicians. MBI (burnout) subscale scores were dummy coded, with 1 indicating a score that signified clinician burnout (27). Other dummy variables included male gender, having a graduate degree, and whether the therapist was black.

The only variable that significantly predicted differences in treatment outcomes was the relationship sub-scale of the MCI. Higher scores on that subscale predicted slightly but significantly decreased differences in treatment outcomes (-.094; Table 4), although substantial variability among clinicians remained unexplained.

To answer the third research question concerning the degree of variability in outcome differences across clinicians, we used the residuals from level 3 to calculate the predicted treatment outcomes for a hypothetical white patient and black patient. Both patients were in the reference category (coded as 0) or at the grand mean for all predictors (other than patient race). For each clinician, the difference between the predicted treatment outcomes for white and black patients was calculated. Negative values indicated that black patients’ outcomes were worse than those of white patients (moderate disparities, -.25 to -.50 points on the BASIS-24 5-point scale; substantial disparities, -.51 or fewer points). Values near zero indicated the absence of differences between black and white patients (-.24 to .24 points), and positive values indicated outcome differences that favored black patients (moderate differences, .25 to .50 points; substantial differences, .51 or more points). The categories denote a difference of less than 5% in outcomes between black and white patients was considered clinically insignificant, between 5% and 10% moderately clinically significant, and more than 10% highly clinically significant.

Among the 62 clinicians, seven (11%) produced substantial outcome disparities for their black patients (between -.59 and -1.02 points on the BASIS-24 5-point scale), five (8%) produced moderate outcome disparities for their black patients (between -.26 and -.46 points), 26 (42%) had no or virtually no outcome differences between black and white patients (between -.24 and .24 points), nine (15%) produced moderately better outcomes for their black patients (between .31 and .43 points), and 15 (24%) produced substantially better outcomes for their black patients (between .82 and 1.34 points) (data not shown).

Discussion

This study found that treatment outcomes for black and white patients varied depending on the clinician providing the services. Within this sample of black and white clinicians providing services to both black and white patients, approximately 20% had black patients with worse treatment outcomes than their white patients, and 40% had black patients with better outcomes than their white patients. For the other 40% of these clinicians few or no differences in outcomes between black and white patients were found. The assumption that black patients treated by white clinicians receive less than optimal care does not reflect accurately this continuum of outcomes. This assumption oversimplifies the complex relationships between patients and clinicians and may hinder the development of evidence-based practices that decrease outcome disparities.

Of the clinician factors tested, only a single dimension of MCC (multicultural counseling competency) pre dicted outcome differences among black and white patients—the degree of positive experiences and relationships a clinician has with people from racial and ethnic groups other than his or her own. Outcome differences did not appear to be readily influenced by clinicians’ self-reported race, education, or job burnout or their MCC skills, knowledge, or empathy, as we had hypothesized. Substantial evidence indicates that racial bias is shaped by past experiences and influences behavior in ways beyond an individual's conscious awareness (43). Prejudices as well as positive feelings by whites and blacks about each other are therefore connected to unconscious perceptions of race and ethnicity (44). Our findings support this line of reasoning and suggest that the traditional approach to reducing disparities in mental health services—that is, having all clinicians participate in multicultural competency training—may be less than effective at decreasing the racial bias of the clinicians associated with outcome disparities.

The most significant limitation of the study was the naturalistic setting. Considerable disagreement exists regarding the effects that naturalistic settings may have on outcomes. To examine clinicians’ role in producing outcomes, naturalistic settings may be preferable because controlled settings purposely minimize clinician differences, such as those related to patient race (11). Some argue that naturalistic settings introduce serious confounds created by the lack of random assignment of patients to clinicians and the researchers’ lack of control over treatment regimens (31,33). In our study, the lack of random assignment meant that the distribution of black patients was uneven; some clinicians saw only one or two black patients. Clinicians whose black and white patients had different outcomes could have been seeing only one or two black patients, which leaves some question as to whether the observed differences were related to patients’ race. The race parameter had ample reliability and was statistically significant (Table 5), indicating that the relationship between clinicians and patient race was not an anomaly.

Our lack of control over treatment regimens meant that black patients received significantly more hours of services than white patients. The hours of services received did not vary systematically by clinician. Any differences in hours of services received by black patients were therefore not moderated by clinician factors and are irrelevant to the moderating role clinicians played in the outcome differences associated with patient race.

How variations in the services received and the number of black patients treated by a clinician are related to outcome differences warrants further study. However, insufficient knowledge of these relationships should not prevent further study of clinician functioning in settings such as CMHCs, where naturally occurring variations in services and patient mix are likely. In general, more investigation is needed on the implications of naturalistic settings and the role of modeling for validity (31,33).

Conclusions

Outcome differences experienced by black and white adults with serious mental illness who were receiving mental health services from CMHC clinicians did not consist exclusively of disparities in which black patients had worse outcomes than white patients. Instead, a continuum of differences was observed, from outcomes favoring white patients to those favoring black patients. Clinicians’ role in outcome differences and the reasons for that role should be reexamined. Levels of education, burnout, and MCC and clinicians’ race appear less important than general experiences and relationships with people from other racial and ethnic groups. Perhaps factors embedded in community dynamics or organizational functioning also play a role, along with experiences and relationships, in shaping clinicians’ ability to work effectively with black and white patients (10,45).

Acknowledgments and disclosures

This work was supported by grant R34 MH074640-01A2 from the National Institute of Mental Health.

Footnotes

The authors report no competing interests.

Contributor Information

Christopher R. Larrison, the School of Social Work, University of Illinois at Urbana-Champaign, 1010 West Nevada St., Urbana, IL 61801.

Susan L. Schoppelrey, Waldron College of Health and Human Services, Radford University, Radford, Virginia.

Samantha Hack-Ritzo, the School of Social Work, University of Illinois at Urbana-Champaign, 1010 West Nevada St., Urbana, IL 61801.

Wynne S. Korr, the School of Social Work, University of Illinois at Urbana-Champaign, 1010 West Nevada St., Urbana, IL 61801.

References

1. Elkin I. A major dilemma in psychotherapy outcome research: disentangling therapists from therapies. Clinical Psychology: Science and Practice. 1999;6:10–32.
2. Davis LE, Proctor EK. Race, Gender, and Class: Guidelines for Practice with Individuals, Families, and Groups. Prentice Hall; Englewood Cliffs, NJ: 1989.
3. Ortega AN, Rosenheck R. Hispanic patient–case manager matching: differences in outcomes and service use in a program for homeless persons with severe mental illness. Journal of Nervous and Mental Disease. 2002;190:315–325. [PubMed]
4. Ridley CR. Overcoming Unintentional Racism in Counseling and Therapy: A Practitioner's Guide to Intentional Intervention. Sage; Thousand Oaks, Calif: 2005.
5. Snowden LR. Bias in mental health assessment and intervention: theory and evidence. American Journal of Public Health. 2003;93:239–243. [PubMed]
6. Mental Health Culture, Race and Ethnicity: A Supplement to Mental Health: A Report of the Surgeon General. US Department of Health and Human Services, US Public Health Service; Rockville, Md: 2001.
7. Hölzer CE, Goldsmith HF, Ciarlo JA. In: Effects of rural urban county type on the availability of health and mental health care providers; in Mental Health, United States. Manderscheid RW, Henderson MJ, editors. Substance Abuse and Mental Health Services Administration, Center for Mental Health Services; Rockville, Md: 1998.
8. Bickman LA. Measurement Feedback System (MFS) is necessary to improve mental health outcomes. Journal of the American Academy of Child and Adolescent Psychiatry. 2008;47:1114–1119. [PMC free article] [PubMed]
9. Institute of Medicine Improving the Quality of Health Care for Mental and Substance-Use Conditions. National Academies Press; Washington, DC: 2005.
10. Hernandez M, Nesman T, Mowery D, et al. Cultural competence: a literature review and conceptual model for mental health services. Psychiatric Services. 2009;60:1046–1050. [PubMed]
11. Worthington RL, Soth-McNett AM, Moreno MV. Multicultural counseling competencies research: a 20-year content analysis. Journal of Counseling Psychology. 2007;54:351–361.
12. Arredondo P, Rosen DC, Rice T, et al. Multicultural counseling: a 10-year content analysis of the Journal of Counseling and Development. Journal of Counseling and Development. 2005;83:155–161.
13. Shin S-M, Chow C, Camacho-Gonsalves T, et al. A meta-analytic review of racial-ethnic matching for black and Caucasian American patients and clinicians. Journal of Counseling Psychology. 2005;52:45–56.
14. Bednar SG. Elements of satisfying organizational climates in child welfare agencies. Families in Society. 2003;84:7–12.
15. Schulz R, Greenley JR, Brown R. Organization, management, and client effects on staff burnout. Journal of Health and Social Behavior. 1995;36:333–345. [PubMed]
16. Glisson C, Landsverk J, Schoenwald S, et al. Assessing the Organizational Social Context (OSC) of mental health services: implications for research and practice. Journal of Administration and Policy in Mental Health and Mental Health Services Research. 2008;35:98–113. [PubMed]
17. Larrison CR, Schoppelrey SL, Hadley-Ives E, et al. Organizational climate and treatment outcomes for African American clients receiving services at community mental health agencies. Administration in Social Work. 2008;32:111–138.
18. Cutler DL, Bevilacqua J, McFarland BH. Four decades of community mental health: a symphony in four movements. Community Mental Health Journal. 2000;39:381–398. [PubMed]
19. Tevis C. Rural health: new help for rural mental health needs. Successful Farming. 2003;101:54.
20. Torrey EF. Surviving Schizophrenia: A Manual for Families, Patients, and Providers. 5th ed. HarperCollins; New York: 2006.
21. Mental Health A Report of the Surgeon General. US Department of Health and Human Services, US Public Health Service; Rockville, Md: 1999.
22. Percent of Persons Who Are Black or African American Alone US Census Bureau; Washington, DC: 2000. 2000.
23. Office of Management and Budget Standards for defining metropolitan and micropolitan statistical areas: notice. Federal Register. 2000;65:82228–82238.
24. Phillips ST, Ziller RC. Toward a theory and measure of the nature of nonprejudice. Journal of Personality and Social Psychology. 1997;73:420–434.
25. Sodowsky GR, Taffe RC, Gutkin TB, et al. Development of the Multicultural Counseling Inventory: a self-report measure of multicultural competencies. Journal of Counseling Psychology. 1994;41:137–148.
26. Leong FTL. Appendix C: Definitions and measurement of cross-cultural counseling competencies; in Community Support: Notes From a Roundtable on Conceptualizing and Measuring Cultural Competence. National Mental Health Information Center; Rockville, Md: 1998.
27. Maslach C, Jackson SE, Leiter MP. Maslach Burnout Inventory Manual. 3rd ed. Consulting Psychologists Press; Palo Alto, Calif: 1996.
28. Eisen S, Dill DL, Grob MC. Reliability and validity of a brief patient report instrument for psychiatric outcome evaluation. Hospital and Community Psychiatry. 1994;45:242–247. [PubMed]
29. Eisen SV, Gerena M, Ranganathan G, et al. Reliability and validity of the BASIS-24 Mental Health Survey for whites, African-Americans, and Latinos. Journal of Behavioral Health Services and Research. 2006;33:304–323. [PubMed]
30. Eisen SV, Normand SLT, Belanger AJ, et al. The revised Behavior and Symptom Identification Scale (BASIS-24): reliability and validity. Medical Care. 2004;42:1230–1241. [PubMed]
31. Elkin I, Falconnier L, Martinovich Z, et al. Clinician effects in the National Institute of Mental Health Treatment of Depression Collaborative Research Program. Psychotherapy Research. 2006;16:144–160.
32. Kim DM, Wampold BE, Bolt DM. Clinician effects in psychotherapy: a random-effects modeling of the National Institute of Mental Health Treatment of Depression Collaborative Research Program data. Psychotherapy Research. 2006;16:161–172.
33. Lutz W, Leon SC, Martinovich Z, et al. Clinician effects in outpatient psychotherapy: a three-level growth curve approach. Journal of Counseling Psychology. 2007;54:32–39.
34. Collins JF, Elkins I. Randomization in the NIMH Treatment of Depression Collaborative Research Program. New Directions for Program Evaluation. 1985;28:27–37.
34. Crits-Christoph P, Mintz J. Implications of clinician effects for the design and analysis of comparative studies of psychotherapies. Journal of Consulting and Clinical Psychology. 1991;59:20–26. [PubMed]
36. Crits-Christoph P, Gallop R. Clinician effects in the TDCRP and other psychotherapy studies. Psychotherapy Research. 2006;16:178–181.
37. HLM 6.06. Scientific Software International; Lincolnwood, Ill: 2004.
38. Singer JD, Willett JB. Applied Longitudinal Data Analysis: Modeling Change and Event Occurrence. Oxford University Press; New York: 2003.
39. Byrk AS, Raudenbush SW. Hierarchical Linear Models: Applications and Data Analysis Methods. Sage; Newbury Park, NJ: 2002.
40. Merlo J, Chaix B, Yang M, et al. A brief conceptual tutorial of multilevel analysis in social epidemiology: linking the statistical concept of clustering to the idea of contextual phenomenon. Journal of Epidemiology and Community Health. 2005;59:443–449. [PMC free article] [PubMed]
41. Little RJA. Modeling the dropout mechanism in repeated-measures studies. Journal of the American Statistical Association. 1995;90:1112–1121.
42. Bush PW, Drake RE, Xie H, et al. The long-term impact of employment on mental health service use and cost for persons with severe mental illness. Psychiatric Services. 2009;60:1024–1031. [PubMed]
43. Greenwald AG, Banaji MR. Implicit social cognition: attitudes, self-esteem, and stereotypes. Psychological Review. 1995;102:4–27. [PubMed]
44. Plant EA, Devine PG. The antecedents and implications of interracial anxiety. Personality and Social Psychology Bulletin. 2003;29:790–801. [PubMed]
45. Hohmann AA, Parron DL. How the new NIH guidelines on inclusion of women and minorities apply: efficacy trials, effectiveness trials, and validity. Journal of Consulting and Clinical Psychology. 1996;64:851–855. [PubMed]