|Home | About | Journals | Submit | Contact Us | Français|
Inconsistent performance measurement schemes hinder attempts to make international comparisons about mental health-care quality. This report describes a project undertaken by an international collaborative group that aims to develop a common framework of measures that will allow for international comparisons of mental health system performance.
Representatives from each country submitted reports of quality measurement initiatives in mental health. Indicators were reviewed, and all measurable indicators were compiled and organized.
Twenty-nine programs from 11 countries and two cross-national programs submitted reports.
Indicators were evaluated according to measurable inclusion criteria.
These methods yielded 656 total measures that were organized into 17 domains and 80 subdomains.
No single program contained indicators in all domains, highlighting the need for a comprehensive, shared scheme for international measurement. By collecting and organizing measures through an inductive compilation of existing programs, the present study has generated a maximally inclusive basis for the creation of a common framework of international mental health quality indicators.
A report from the US Institute of Medicine described a ‘chasm’ that divides potentially effective, evidence-based health-care practices from the actual care that consumers receive , and the need to improve the quality of mental health care is no less urgent, especially considering the large global burdens of mental illness  and the relatively high prevalence rates of diagnostic and statistical manual of mental disorders diagnoses (from 12% to over 45% of national populations) . There are significant challenges to improving the quality of medical care. Some commentators have expressed concern about the evidence base supporting quality measures and the existing infrastructure for developing such measures . Other challenges to quality improvement include the amount of mental health services delivered outside of formal medical settings (e.g. the criminal justice system) and the relatively slow adoption of information technology in the mental health sector .
Accordingly, many countries and international organizations have attempted to develop formal indicator schemes to measure quality in mental health care, with varying degrees of consensus. Researchers began developing conceptual frameworks for quality measurement as early as 30 years ago, and organizations including the US National Committee on Quality Assurance, the Institute of Medicine and the RAND Corporation have refined several iterations of criteria for selecting quality measures . Recent larger efforts have led to inconsistent results. An expert panel of the Organization for Economic Cooperation and Development (OECD) identified 12 indicators across four main domains: (i) continuity of care, (ii) coordination of care, (iii) treatment and (iv) patient outcomes  while a similar European Union initiative developed a different set of 32 indicators across four broad domains: (i) demographic and socioeconomic factors, (ii) health status, (iii) determinants of health and (iv) health systems . We  recently reviewed US initiatives, and while we identified several shared foci, we also found a lack of coordination that led to significant gaps in both indicator development and data reporting. Similarly, in another review of 55 literature reports from 12 countries , we found that no one program covered every measurement domain.
This inconsistency in performance measurement schemes points toward an overarching methodological need. Although individual quality improvement efforts continue to be refined, broader problems still remain, as indicators and their organizing structures are often too heterogeneous to be compared with other schemes. Sometimes, the proposed indicators themselves are not measurable in practice (i.e. the measure cannot be quantified at all).
Analogous efforts from other branches of medicine provide informative methodological examples for developing quality indicators sets. In particular, a 2006 OECD project collected existing quality indicators from around the world, judged them for importance, scientific soundness and feasibility, and mapped them to an organizing framework they developed to categorize those indicators .
This paper describes an initial component of a project to develop a set of mental health quality indicators that would be a reasonably comprehensive and compatible scheme for international comparisons. In 2008, clinical experts from 12 countries (Australia, Canada, England, Germany, Ireland, Japan, the Netherlands, New Zealand, Norway, Scotland, Taiwan and the USA), meeting as part of the Clinical Leaders group of the International Initiative for Mental Health Leadership (IIMHL), initiated this project to develop a consensus framework for reporting on mental health-care quality. In the recently concluded first phase of this project, representatives reported a wide range of mental health performance and outcome measures from their national performance measurement programs. We then collected and analyzed these measures in an effort to identify and organize the most commonly used indicators and domains. The ultimate aim of the IIMHL is to develop and implement a balanced, inclusive and common framework of measures that will allow for international comparisons of system performance, with a long-term goal of informing initiatives that will improve mental health services in these countries.
Representatives from each of the participating countries were contacted to identify peer-reviewed journal articles, government reports, white papers and other ‘gray literature’ on population-based quality or performance measurement initiatives in mental health being developed or implemented in each country at the national or other representative level (e.g. province and state, etc.). Initiatives were extracted into a standard document listing context, mental health indicators and original domains, process of indicator development, intended or actual use and related studies and reports. This initial phase of the overall IIMHL project aimed to provide an overview and assessment of the broader context of mental health quality measurement in each participating country .
In a second step, performance measures, performance indicators and outcome measures that met the following three main criteria were collated into a single document and determined for inclusion in the current review:
Indicators selected and included based on these criteria represent an inventory of measures currently used in participating countries without validating the measures' clinical importance, validity or feasibility. The measures were then organized and assigned to a list of 16 domains and 77 subdomains initially developed for an international survey. These domains and subdomains were adopted from the National Inventory of Mental Health Quality Measures established by the Center for Quality Assessment and Improvement in Mental Health and recommendations from the IIMHL country experts . Researchers at Columbia University reviewed the list of indicators and identified indicators that were unclear or were classified differently to develop consensus on the classification. Discrepancies were resolved by discussion among the three lead authors to develop consensus on the evaluation and to iteratively modify the framework of domains and subdomains to better reflect the range and purpose of the measures collected. Coding was continued until 100% consensus was achieved. This process reflects the ongoing iterative work to establish an overall indicator framework that will ultimately allow for international benchmarking and comparison of mental health quality indicators. Finally, based on this list of indicators and domains, we identified common (and differing) themes, methods and definitions that are discussed below.
Indicators from 31 programs in 11 countries and two cross-national programs were compiled, yielding 656 total measures. The final framework comprised 17 domains and 80 subdomains. Each indicator specified an objective numerator and denominator drawn from an identifiable dataset—as required by the selection criteria noted above.
Numbers of indicators by country or organization are given in Table 1. There was considerable variability in the numbers and types of indicators, programs and datasets. The numbers of total measurable indicators per country ranged from 3 to 211, which is explained only in part by the range of 1–7 programs per country. Supplementary Table S1 (online) gives lists of the programs from which the indicators were drawn, demonstrating the wide scope of the data sources used. Sources of data included, for example, governmental bureaus, state/territory administrations, surveys, protocols, censuses, standardized patient measures, budget reports, insurance records, patient charts and physician reports.
Counts of indicators by domain are reported in Table 2 that also lists the subdomains into which redundant or similar indicators were combined. After condensing the indicators in this way, 80 total subdomains were identified. Counts for individual subdomains are also listed in Table 2. The numbers of indicators per domain and subdomain were also highly variable, and we found broad types of measurable indicators within each domain. For example, both adequate medication dosage and medication reconciliation are found in the Evidenced-Based Pharmacotherapy domain; the former measures compliance to clinical standards, whereas the latter is a system-level practice. Similarly, the Outcome Assessment domain (which contains 72 total indicators, the greatest number) contains subdomains, including examples as diverse as mortality rates, employment status and the Global Assessment of Functioning scale. (Examples of each subdomain are listed in the online Supplementary Table S2 for illustrative purposes.)
Several aspects of this analysis provide important insights into the state of quality measurement worldwide, and the provisional organization of these data represents a meaningful step toward developing an international framework for mental health quality indicators. No single program contained indicators in all domains, further emphasizing the need for a comprehensive, shared scheme for international measurement. This finding echoes the results of two recent literature reviews arising from the IIMHL project, which reviewed numerous US  and international  mental health quality measurement programs, but still identified significant gaps in the scope of every program.
A significant methodological contribution of this effort is its inductive approach to organize mental health measures. By deriving categories and examples of quality indicator measures through a compilation of the existing programs, this stage of the IIMHL project has generated an organized list of measures that is maximally broad in scope, thus allowing future efforts to proceed from the most inclusive basis possible.
Quality indicators have complicated origins, as each guideline for measurement is a product of the goals, context and time from which it came. The priorities and influences of these performance measurement programs will vary under the influence of broader cultural factors, and the framers, providers and consumers coming from a given culture or health-care system may not be fully aware of how their context impacts the ways they measure quality. One important motivation of a structured framework approach such as this, derived from a comprehensive inventory of international measures, is to attempt to avoid these potential biases by providing an inclusive global perspective on the state of quality measurement.
The relative frequencies of indicators per domain may give a sense of how different measures have been prioritized or an indication of which measures are actually measurable. For example, despite the recent popularity of the recovery movement in mental health , the recovery domain in this sample has among the fewest measurable indicators. It is important to note, however, that no strong conclusions can be drawn from these counts. In many cases, these frequencies may simply reflect the feasibility of measurement: outcome assessment and symptom assessment are likely easier to quantify than culture issues or perceptions of care. It is also important to note that high numbers of indicators in particular domains do not necessarily indicate greater coverage of topics within those domains. In many cases, different indicators were capturing quite similar measurement concepts with different approaches or minor variants in numerator or denominator definition (e.g. measuring medication adherence or access/wait times to outpatient services). That said, on a more straightforward level, these frequencies do give an important sense of which areas are in need of further quality measure development.
There are several limitations to this study. The member countries of the IIMHL Clinical Leaders Group are necessarily a ‘convenience sample’, and participation is limited to countries that already have some infrastructure and resources in place to measure mental health quality indicators. Language issues may have hampered the collection of indicators from countries, for example, translations were done by the IIMHL partners who provided the indicators themselves, creating a somewhat onerous step for participating members. More broadly, the first step of obtaining measures from representatives in peer countries depended on those representatives for thoroughness and rigor, and while they were asked to submit a broad collection of measures, individuals may have gone about their selection processes differently. In terms of the organization of the indicators, the system of domains that was created is inherently subjective and rests in part on a priori assumptions about the organization of quality measures. That said, this provisional categorization was not the primary goal of this effort, and the study methodology allowed for flexible adjustments to these categories as the data were analyzed. In addition, it should be noted that these measures have not been subjected to a validation process. In fact, a key component of a research agenda for quality improvement in mental health is evaluation of the reliability, validity and utility of quality indicators. Obviously, evaluating 600+ measures of mental health quality is unrealistic. Ultimately, each country needs to determine its own priorities, amidst the reality of what is feasible, given available data and resources. It is likely that national programs of quality measurement will develop ways to create composite measures or dashboards to compress the information in ways that can be directly acted upon to improve quality.
Several of the next key steps of the Clinical Leaders project in Phase II will proceed from these data. Now that the performance indicators have been collected and grouped, the IIMHL Clinical Leaders Group will rate this sample for validity, importance and feasibility, with a goal to propose a consensus list of mental health quality indicators that could be collected by each of the participating countries. At the same time, country networks are forming to further discuss data infrastructures and explore implementation strategies that will allow for the comparison of performance measures across countries in the future. The ultimate goal is to enable quality measurement initiatives that will help to transform mental health services worldwide.
This work was supported by: the Center for Mental Health Services, Substance Abuse and Mental Health Services Administration, U.S. Department of Health and Human Services; the Irving Institute for Clinical and Translational Research, Columbia University [grant number UL1 RR024156]; the National Center for Research Resources, a component of the National Institutes of Health; the Mental Health Therapeutics CERT at Rutgers, the State University of New Jersey, funded by the Agency for Health Care Research and Quality [AHRQ, grant number 5 U18 HS016097]. Additional funding came from government and non-government organizations of the countries participating in the IIMHL project (Australia, Canada, England, Germany, Ireland, Japan, Netherlands, New Zealand, Norway, Scotland, Taiwan and the USA).
Thanks to the members of the IIMHL Clinical Leaders Project. This article represents the collaborative effort of a group of researchers, clinicians and government officials across the participating countries. The views expressed in this article are those of the authors.