Mental disorders (including serious mental illness, such as schizophrenia or bipolar disorder, depression, anxiety disorders, and substance use disorders) are leading causes of disability worldwide, are associated with substantial costs, and when left untreated, can lead to premature mortality.1
Despite the proliferation of evidence-based guidelines for the treatment of mental disorders, the quality of care and subsequent outcomes remain suboptimal for those suffering from these illnesses.2
A recent report by the U.S. Institute of Medicine (IOM) and others documented substantial gaps in evidence-based care for mental disorders, citing poor quality in detection, treatment, and follow-up care.2
The stigma of a mental disorder diagnosis may also hinder many from seeking care in the first place.
Within the past ten years, there has been a dramatic growth in the development of performance (or quality) measures to assess and redress gaps in evidence-based health care in general. Experts have recognized that quality measurement is a key driver in transforming the health care system, and routinely measuring quality using performance measures derived from evidence-based practice guidelines is an important step to this end.3
Notably, national and provincial governments as well as regulatory, accreditation and other non-governmental organizations around the globe have proposed and implemented performance measures to be used by different health plans and organizations for a broad range of services and conditions. In the U.S., these organizations have included the National Committee on Quality Assurance (NCQA) and the National Quality Forum (NQF). Performance measures have been increasingly used in health care to compare and benchmark processes of care in order to remediate gaps between evidence-based care and actual practice, and hold providers accountable for improving quality of care. Recently, in the U.S. and United Kingdom (UK), for example, pay-for-performance (P4P) initiatives and other financial incentives have been implemented as a means to promote quality improvement.4–5
Nonetheless, there is documented evidence (i.e., IOM) that worldwide, the mental health services sector lags behind in the development and implementation of performance measures and strategies for implementing them as tools to improve quality and outcomes.2
There are three key reasons for this lag: lack of a sufficient evidence base through which to develop specific, valid and clearly defined measures,2
inadequate infrastructure to develop and implement quality measures and capture elements of mental health services, and lack of a cohesive strategy to apply mental health quality measurement across different settings in the service of improving care. Measuring quality of mental health care is particularly challenging because a substantial amount of mental health services are delivered outside the health care sector (e.g., criminal justice, education, social services), and there is insufficient evidence for some mental health treatments as well.2
Data elements necessary to measure quality of mental health care are incomplete or even missing in many settings, and, even when data collection does occur, it tends to be inconsistent across different organizations. Moreover, mental health programs and providers have not fully embraced quality measurement due to infrastructure and policy barriers intrinsic to mental health, including providers’ concerns regarding patient privacy, “cookbook medicine,” and the silos across different provider types and credentialing requirements. Finally, in many countries, the mental health sector is far behind the rest of health care in the use of health information technology.
The purpose of this paper is to delineate the barriers to mental health quality measurement, and identify strategies to enhance the development and use of quality measures by mental health providers, programs, payers, and other stakeholders in the service of improving outcomes for individuals with mental health and substance use disorders.
Quality Measurement in Healthcare
Understanding the challenges of measuring quality of mental health care requires an understanding of the history of measuring quality. Over 30 years ago, Donabedian published a quality framework that is much used today.6
This framework incorporates three domains of quality measurement: structure, process, and outcomes. The rationale underlying the framework is that health care structures, including resources and policies, can inform processes of care provided by clinicians, which in turn can influence patient outcomes. The framework also provides a useful typology for distinguishing among different approaches to measurement. Specifically, health care structure measures evaluate characteristics of the treatment setting’s services, including program fidelity, staffing, and infrastructure (i.e. are quality services available?). Process measures examine interactions between consumers and the structural elements of the health care system (i.e. are consumers actually receiving high quality services in a way that conforms to the evidence base?). Outcome measures examine the results of these interactions for patients, including functioning, morbidity, mortality, quality of life,7
and patient satisfaction (i.e. is the care making a difference for individuals and society?).
Each type of quality measure has its strengths and limitations. Structure measures are relatively simple to ascertain through reports from program or clinic leaders, yet are subject to response bias (e.g., program leader tendency to over-report resources or over-idealize clinic operations) or vagueness in terminology or questions (e.g., defining program fidelity). Moreover, structure measures do not indicate whether good care happened, but only if the site has capacity to provide good care. Process measures have been widely used for performance measurement, and are more appealing to providers because they represent aspects of care over which providers have the most control. However, there are concerns that many process measures are overly dependent on a patient’s care-seeking behavior, which is often not reported in data sources (e.g., were outpatient visits not completed because they were not scheduled by the provider, or did the patient miss them?). Alternatively, outcome measures are appealing because they actually assess whether a patient’s status improves or not. However, use of these measures requires additional case mix adjustment in order to ensure that observed differences in outcomes are not due to clinical differences in severity of illness across patients. Currently, methods to adjust for case mix (risk adjustment) in mental health are limited, primarily due to incomplete data on psychiatric symptoms and other patient characteristics. Use of outcome measures to incentivize providers can also be problematic, as there is concern that providers may be reluctant to take on sicker patients in order to make patients panels look better overall.
In light of the strengths and limitations, some regulatory agencies have implemented criteria for evaluating and selecting quality measures, notably, the U.S. National Quality forum and National Committee on Quality Assurance. In the early 1990s, organizations such as the NCQA began to operationalize evidence-based care by establishing quality measures, notably process measures, through the Health Experience Data and Information Set (HEDIS), and hence, had to develop standards of performance measurement. These standards have their origins in the criteria developed by the RAND Corporation,8
which have been widely used to select and refine quality indicators, provide a good start for identifying and refining performance measures. These criteria include: 1) clinical importance (i.e., Does the measure represent a substantial deficit in mental health care? Can it result in actionable improvements in mental health care?), 2) validity (i.e., Is the measure scientifically sound? Is it associated with improved clinical care? Is it sensitive to risk adjustment and insensitive to being gamed?), and 3) feasibility (i.e., How easy is it to collect the information required for the quality measure?). The RAND method also recommends a panel of experts (usually 9) to rate each indicator based on these criteria using a scale (1–5), and dropping indicators that fall below a certain threshold (e.g., below 5).
The number of organizations using quality indicators to benchmark care across different practices has grown substantially since the development of quality indicators, with several countries are taking the lead to develop and implement mental health performance measures as part of their nationalized health systems.9–12
In a systematic review of governmental and non-governmental organizations in the U.S., Herbstman (2009) found 36 initiatives that applied quality indicators for mental disorders, ranging from federal and state governments, health plans, and non-governmental and professional organizations.13
Currently, one of the most extensive implementations of mental health quality measures is the International Initiative for Mental Health Leadership Clinical project on mental health quality measurement, described in detail in this issue.
In the U.S., an important example of quality measurement in mental health is the work of the Department of Veterans Affairs (VA) Veterans Health Administration. The VA provides a useful comparator organization in the international sphere given that it is single-payer, national health care program for U.S. veterans.14
Currently, the Altarum Institute and the RAND Corporation are conducting a national evaluation of the VA’s mental health services, using Donabedian’s structure-process-outcomes model of quality measurement. Structure is evaluated by describing the continuum of care available to veterans with five targeted mental health diagnoses (i.e., schizophrenia, bipolar disorder, post-traumatic stress disorder [PTSD], major depressive disorder, and substance use disorder) using facility survey and administrative data. Process is evaluated by looking at what services veterans with the five targeted mental health diagnoses actually receive using administrative and medical record data. Outcomes measures consist of services received that made a difference to veterans’ functioning and quality of life using client survey and medical record data. As part of this work, the Altarum-RAND team has developed and vetted over 200 quality indicators for the five targeted mental disorders. In addition, the VA Primary Care-Mental Health integration program has implemented performance measurement for depression, alcohol, and PTSD screening and management.15
Applying Quality Measures
Quality measures are vital for ensuring the uptake and delivery of evidence-based care. Quality of care cannot be improved without monitoring how such care is delivered, from its organization to processes and ultimately patient outcomes. Measuring quality using quality indicators derived from evidence-based practice guidelines is an important step towards implementation of evidence-based care and monitoring quality improvement efforts. Experts have strongly suggested that quality measurement is a key driver in transforming the health care system and clinicians at the front line need to be actively engaged in the process of improving quality at multiple levels (e.g., payment, licensure, organizational change), and payment systems should reward high quality of care). Standardized measures that allow results to be monitored and tracked uniformly over time are the foundation of performance improvement.
Moreover, the public reporting of performance is vital for holding health care organizations accountable for improving care. Despite the plethora of quality measures, some health care payers and organizations have realized that they would be ignored unless providers were held accountable for improving care. For example, in Scotland, there are national efforts to measure quality of mental health care through the use of “National Targets” on readmission rates, antidepressant prescribing, suicide prevention, and dementia screening, as well as benchmarking programs to compare healthcare providers on performance measures.12
In the U.S., the National Quality Forum applies quality indicators to rate different private health plans.
Quality measures are also used to evaluate whether differences in care exist across regions or healthcare providers, and in the U.S. and U.K, they can also be used to reward or reprimand providers financially. Some countries have gone a step further and established financial incentives, such as pay-for-performance (P4P), for achieving standards based on quality measures.5
P4P involves financial incentives to providers or health care systems when certain target quality of care measures are achieved within providers' patient panels. Currently, a growing number of financial incentives programs focus on behavioral health in the U.S.16
and in England.5
In particular, England established P4P for general practitioners in 2004, comprising of several chronic disease composite indicators, including mental health, notably the “percentage of patients with severe long-term mental health problems reviewed in the preceding 15 months, including a check on the accuracy of prescribed medication, a review of physical health, and a review of coordination arrangements with secondary care”.5
However, the use of performance-based financial incentives such as P4P has been criticized for failing to adequately address patient complexities. In mental health, current risk adjustment methods cannot fully account for these complexities, as these patients tend to be poorer and sicker. “Cherry picking” is a concern, as the sickest patients can be dropped from a panel in order to improve performance measures. Still, while this has been a concern with physical health performance measures, in England, there has been little evidence of cherry-picking (i.e., less than 1% of practices excluded large numbers of payments from the program who might not have scored well on the indicators).5
The cost of implementing P4P is also an issue. In the U.S., evidence suggests that financial incentives for improved performance need to be 10–15% of an individual provider’s salary to induce practice change.17
In some cases, P4P implementation has led to rising costs18
with little improvement in quality. In contrast to typical U.S. P4P programs, the UK’s program paid practitioners up to 25% of their salary, leading to rapidly rising healthcare costs. Hence, without a means to restructure existing payment systems, it is unclear whether P4P can be sustained over time.
Overall, while P4P holds some promise for incentivizing providers to achieve performance benchmarks, especially for single-payer health systems, cost is still a concern, and the metrics used to incentivize providers need to clearly represent evidence-based care and adequately account for patient case-mix. For P4P to be successfully implemented and accepted by mental health providers, the indicators that are used to benchmark care need to be refined and accepted by providers and other stakeholders. Moreover, for mental health services, improving the quality measures themselves is a vital step towards improving performance-based measurement and gaining widespread acceptance of programs designed to change provider behavior.
Limitations of Mental Health Quality Measures
The overall development and implementation of valid quality indicators and their application to promote accountability of care for mental disorders has not kept pace with physical health care. While several measures have been developed for mental disorders across different organizations, it is uncertain whether they have been validated, or at least vetted and rated by experts. Notably, only 16 of almost 600 NQF measures and only a fraction of U.S. HEDIS measures address mental health, and those that do are limited by too narrow a focus on clinical activity or a particular diagnosis, failing to encourage coordination, and not being linked to quality improvement activities.19
Important reasons for the lag in mental health performance measurement include:
- Lack of sufficient evidence regarding appropriate care. Mental health, unlike general health sectors, lacks sufficient evidence for many conditions. For example, there is limited information on evidence-based care for adolescents, including use of atypical antipsychotics20 or the use of combinations of antipsychotics in adults.21 The guidelines are also inconclusive in regards to the use of antidepressants in bipolar spectrum disorders, and medication treatment for PTSD.22
- Poorly defined parameters (e.g., definitions of treatment “engagement,” “psychotherapy,” “recovery,” or intervention “fidelity” are often not well-described)23. Existing data sources often lack sufficient information to establish accurate numerators and denominators for quality indicators (e.g., ICD-9 codes for mental health diagnoses), and some quality measures use strict exclusion criteria that do not apply to the majority of patients at risk (e.g., indicators for newly diagnosed depression only).
- Limited descriptions of mental health care based on available data sources preclude their utility in routine care. Data on appropriate pharmacotherapy and psychotherapy are only available when pulled from multiple sources (e.g., billing claims, outpatient, and pharmacy records). Most performance measures are based on medical chart review or electronic claims data. Manual chart review is slow, labor intensive, expensive, constrained to small samples, and prone to inter-reviewer variation. Administrative claims data, while appealing because the sample size is larger and data pulls can be done electronically, are less meaningful given the wide variability in diagnostic coding of mental health conditions and lack of complete data on types of visits or treatments (e.g., specific treatment modalities). Administrative data may also be incomplete within mental health programs, as many lack access to data from general medical providers, labs, or prescriptions. There are also multiple definitions used to define specific performance measures (e.g., post-hospitalization follow-up: number of days), resulting in growing burden and confusion over data capture and reporting processes as providers try to implement quality measurement program requirements.
- Lack of electronic health information to facilitate capture of the required health information. In a recent study, only 12% of U.S. hospitals had electronic medical records (EMRs).24 In other countries such as Canada, national electronic medical records exist25 but may lack full information on services needs of individuals with mental disorders.26 Moreover, a substantial amount of mental health services are delivered outside the health care sector (e.g., criminal justice, social services), making integration of electronic information across sites challenging.