|Home | About | Journals | Submit | Contact Us | Français|
In many occasions, routine mental health care does not correspond to the standards that the medical profession itself puts forward. Hope exists to improve the outcome of severe mental illness by improving the quality of mental health care and by implementing evidence-based consensus guidelines. Adherence to guideline recommendations should reduce costly complications and unnecessary procedures. To measure the quality of mental health care and disease outcome reliably and validly, quality indicators have to be available. These indicators of process and outcome quality should be easily measurable with routine data, should have a strong evidence base, and should be able to describe quality aspects across all sectors over the whole disease course. Measurement-based quality improvement will not be successful when it results in overwhelming documentation reducing the time for clinicians for active treatment interventions. To overcome difficulties in the implementation guidelines and to reduce guideline non-adherence, guideline implementation and quality assurance should be embedded in a complex programme consisting of multifaceted interventions using specific psychological methods for implementation, consultation by experts, and reimbursement of documentation efforts. There are a number of challenges to select appropriate quality indicators in order to allow a fair comparison across different approaches of care. Carefully used, the use of quality indicators and improved guideline adherence can address suboptimal clinical outcomes, reduce practice variations, and narrow the gap between optimal and routine care.
A growing body of evidence suggests that routine mental health care varies strongly between different regions and providers, and that in many occasions, it does not correspond to the standards that the medical profession itself puts forward [12, 25, 29]. Besides existing gaps between clinical practice and guideline recommendations, improving quality of care presents itself as an avenue to restrain the growth of medical expenditures by reducing costly complications and unnecessary procedures. These economic forces increase the desire for information evaluating the health benefits of investments in mental health care. In other words, better organisation and management of medical care would allow countries to spend their health budget more prudently. To improve care for their citizens and to realise these potential efficiency gains, policymakers are looking for methods to measure and benchmark the performance of their health care systems as a precondition for evidence-based health policy reforms. Five mental disorders are among the ten leading causes of disability: depression, bipolar disorder, schizophrenia, obsessive-compulsive disorder, and alcohol abuse . The direct and indirect costs for societies are high. For example, about 5–6% of National Health Service inpatient costs in England have been estimated to be attributable to schizophrenia . Efforts to reduce these enormous costs and the burden for patients and relatives are urgently needed.
With the development of treatment guidelines, there is growing hope that the quality of care will improve by diminishing inadequate care and increasing evidence-based practices. In mental health care, guidelines are intended for use by all physicians investigating, diagnosing and treating patients with mental illness, especially those with severe mental illness and a supposed unfavourable natural disease course. From the beginning, these practice guidelines have been used as standards against which routine care has been compared. Thus, they also served as tools to detect practice variations across settings and across geographical areas, and to evaluate over- and underuse of services and interventions . The reasons for guideline non-adherence and for the gaps in quality detected by a variety of studies are complex and include areas that are only partly or completely out of the control of physicians and other healthcare providers. However, the methods for mental health quality assessment and improvement have been refined in the last years supported by increasing degrees of computerisation. Although there seem to exist some structural measures of health care that have been shown to influence patient outcomes with sufficient reliability and validity , a paradigm shift has begun from developing and implementing measures of structural and process quality towards outcome quality. However, the challenging problem how to assess treatment quality now needs to be addressed in order to decide what type of measure will be best used for which specific purpose.
To document the disease course, treatment effects and provider performance, many measures have been developed. Physician organisations, healthcare agencies, governments and other payers, consumers and researchers have created and implemented process measures that are typically rate-based and indicate the percentage of persons among the eligible population receiving adequate care. These quality indicators are being increasingly used in all fields of medicine including psychiatry . Efforts to improve the quality of medical care must be measured with simple and reliable criteria. Our article summarises the efforts and problems to develop quality indicators assessing the treatment of severe mental illness, and critically discusses whether increased measure performance and guideline implementation is likely to lead to better treatment quality and disease outcome.
In general, quality is a theoretical construct. Psychiatric interventions take place in complex bio-psycho-social interactions and settings that are not very well understood . Various efforts have been undertaken to find valid parameters of the quality of care including the structural attributes of the settings in which care occurs, the processes of care, and the outcomes of care . Before assessing quality, one has to decide how quality of care should be defined and this depends on whether one assesses only provider performance or also the contributions of patients and of the health care system. Quality assessment may differ according to how broadly health and responsibility for health are defined; whether the maximum effective or average care is sought; and whether individual or social preferences define the optimum .
Today, psychiatric hospitals are, for example, often required to report their performance on standardised core measures, and to conduct both internal measurement-based as well as external quality improvement activities. Public and private health care providers, consumers and accreditors may use the results several ways to encourage hospitals to improve their quality. In many countries, hospitals are provided with feedback systems comparing their performance with peer organisations, disclosing results publicly in an effort to influence purchaser or consumer decisions, or linking financial incentives to improved performance . Their criteria, however, may vary according to the domains of care regarded as important, the available data sources, and the basis of comparison for the determination of quality.
To increase comparability, the members of the OECD mental health care panel, consisting of international experts in mental health care, suggested a quality indicator set covering the most relevant domains of mental health care (primary care) by selecting four key aspects (http://www.oecd.org/els/health/technicalpapers): treatment, continuity of care, coordination of care, and patient outcomes. Based on a framework outlined by Hermann et al. [20, 21], the panel decided that the indicators should meet the following screening criteria: the indicator measures the technical quality provided, not interpersonal or consumer perspectives; the indicator is focused on quality of care, not on cost or health care utilisation; the indicator is built on a single item, not on a multi-item scale; and the indicator is likely to be useful in quality assessment at the health care system level, rather than the provider level. In addition, Hermann et al. argued that an indicator should likely be constructed from administrative data using uniform coding systems (e.g., ICD or DSM codes), rather than requiring dedicated data collection or non-standardised data elements. The importance of a selected indicator was assumed to be made up of three different dimensions: the impact on health addressing the areas in which there is a clear gap between the actual and potential levels of health, the policy importance, and the susceptibility to be influenced by the health care system. For the latter, the question has to be addressed whether changes in the indicator will give information about the likely success or failure of policy changes.
The scientific soundness of the indicators can be broken down into two dimensions: face validity and content validity . The face validity indicates if the measure is meaningful in a logical and clinical sense. The face validity of an indicator should be based on its basic clinical rationale, and on its past usage in national or other quality reporting activities. The content validity of an indicator addresses if the measure captures meaningful aspects of the quality of care. In addition, the feasibility of an indicator reflects the question of data availability and the burden of reporting . This last dimension should in particular address the issue whether the value of the information contained in an indicator outweighs the costs of data collection and reporting. For mental healthcare, some of the measure attributes conflict with each other. Indicators with the best measurement properties such as utilisation data may not represent the breadth and diversity of the mental health care system in terms of processes, modalities, settings and interventions. For example, whereas psychopharmacological interventions have the broadest evidence base, other areas based on less rigorous studies may be similarly relevant for the long-term course of psychiatric disorders.
Specifications of quality indicators often require the use of approximations, and some measures are a proxy for a broader concept. For example, using hospital re-admission rates as a proxy for the quality of discharge planning assumes that hospital admissions are an unintended outcome. This builds on the ethics of a mental health care system that offers the least restrictive care which is effective. However, there may be research studies showing significant relations between re-admission rates and other measures of quality [1, 45]. Targeted short-term hospital readmissions may even be an indicator of good quality avoiding long-term hospitalisations. Thus, readmissions as quality indicators need to be evaluated in a more-differentiated manner. One of the indicators for provider performance selected by The Mental Health Panel for primary care is the timely ambulatory follow-up after mental health hospitalisation, because the continuity of care was seen as an important aspect of quality in mental health . This indicator can be measured by the number of persons hospitalised due to a primary mental health diagnosis with an ambulatory mental health encounter or with a mental health practitioner within (a) 7 days, or (b) 30 days of hospital discharge. The importance of the indicator is outlined by the fact that most patients with a psychiatric disorder treated in an inpatient setting require follow-up ambulatory care to promote further recovery and prevent relapse. Scheduling outpatient appointments proximally to discharge is generally recommended to address side effects that can result from inpatient medication changes, and to support compliance with the treatment plan. Data indicate that there is wide variability in the duration between hospital discharge and the first ambulatory follow-up visit, some of which is related to patient factors (e.g., severity of illness) and some to system factors (e.g., availability of outpatient appointments). Shorter gaps between discharge and aftercare may contribute to greater continuity of care and a decreased risk of relapse, although research evidence on this question is mixed . More continuity of care may be achieved by improvements in discharge planning interventions that have been shown to be effective in reducing rehospitalisation and in improving adherence to aftercare , making this indicator potentially useful in assessing the quality of an integrated care delivery system .
A further quality indicator is the hospital readmission rate for psychiatric patients measured by the quota of the total number of readmissions to psychiatric inpatient care that occurred within (a) 7 days or (b) 30 days, divided by the total number of discharges from psychiatric inpatient care during a 12-month reporting period. Hospital readmission rates are widely used as proxies for relapse or complications following an inpatient stay for psychiatric and substance use disorders . Since they indicate premature discharge or lack of coordination with outpatient care, high readmission rates have led some inpatient facilities to examine factors associated with readmissions, including patient characteristics, length of stay, discharge planning, and links with outpatient care . Given the high cost of institutional care, reducing readmission rates can have a substantial effect on mental health spending. However, the relation between readmission rates and other quality criteria are far from consistent. One study showed no association between readmission rates and clinical measures of treatment quality  suggesting that it is not the success of the hospital intervention per se which influences the likelihood of readmission. This measure is therefore not a useful measure for one institution but rather for a whole mental health care delivery system, and indicates the quality of post-discharge outpatient treatment as well as the quality of inpatient treatment and information management between mental healthcare sectors.
A further process measure proposed for provider performance measurement is the rate of persons with a mental illness receiving continuous medication treatment in the maintenance phase for disorders like major depression. In this case, the indicator may be specified as the number of persons treated with anti-depressant medication for a period of at least 180 days divided by the number of persons who are diagnosed with a new episode of depression. Depressive disorders can impair personal, social and family functioning, decrease work productivity, and increase the risk of suicide. They are highly prevalent and very disabling. The World Health Organisation has estimated that by the year 2020, major depression will be the second leading disorder in terms of the global burden of disease . Studies have consistently demonstrated that compared with their non-depressed counterparts, individuals with depression experienced impaired physical and role functioning, more workdays lost, and decreased productivity. Many studies show heavy utilisation of health services, with hospitalisations accounting for a high proportion of costs. A range of antidepressant medications have been shown to be effective in ameliorating symptoms, and in improving quality of life and social functioning. However, adherence remains a problem when the choice for a medication trial has been taken with patients who discontinue their antidepressant early having a high risk to experience relapse or recurrence . Randomised clinical trials have provided evidence that antidepressants need to be continued for 4–9 months after initiation to minimise the likelihood of relapse. The health system has considerable influence on this indicator of medication adherence with clinicians playing an important role in influencing patient attitudes to treatment by providing education, addressing concerns, and evaluating and treating side effects [6, 36]. The indicator has good face and content validity. It assesses the effectiveness of clinical management in achieving medication adherence as the basis of the effectiveness of an established dosage regimen by determining the percentage of adults who complete a period of continuation phase treatment adequate for defining a recovery according to the US Agency for Health Care Research and Quality (AHRQ) criteria. Adherence indicators can be constructed from pharmacy data, which may be easily useful to identify patients who need assistance with medication adherence . At the same time, there is no consistent evidence that adherence to antidepressant medication dosages and other guideline recommendations are sufficient to improve patient outcomes. In one study, the authors found no differences in mean endpoint depression scores between a depression guideline intervention group and the control group, so that depression scores were only marginally better in the intervention group . Other studies, however, showed considerable advantages of guideline-based intervention focusing on structured medication treatment and adherence compared to a group with treatment as usual in self-reported and physician-assessed depression scores after 12 months .
A possible outcome indicator on the population level in primary care could be the mortality of persons with severe psychiatric disorders specified as the standardised mortality rate for persons with particular psychiatric disorders. Individuals with schizophrenia and other severe mental illnesses have higher age- and sex-adjusted mortality rates than members of the general population . Studies in some countries have found medical conditions and co-morbidities to be under-detected and under-treated in individuals with psychiatric disorders . Such relative mortality rates, which are frequently used in cancer epidemiology studies, are well-accepted and plausible measures to indicate and evaluate the excess mortality in subgroups with certain diseases. They may also provide an estimate of the impact on longevity of these diseases. As there is no a priori biological reason why patients with mental health disorders should die prematurely, a large survival difference between different regions and different mental health care systems could point to shortfalls in the overall quality of medical care, not just mental health care, for this especially vulnerable group of patients. This may also provide a starting point for further investigation.
Aggregate-level measures such as mortality or suicide rates are reliant on external and aggregate sources of data for interpreting results. Inferences can only be made for a whole population and not for the individual patient. When there is no intrinsic standard, a process or outcome measure may be used to identify problems or outliers in the performance of a mental health care system, although such a measure cannot determine the appropriateness of single interventions. Measure performance is heavily influenced by patient characteristics, the case-mix. For example, when hospitals are to be compared, statistical case-mix adjustment is needed to remove the influence of patient characteristics on the results in order to avoid unfair comparisons . In other areas, adjustment of process measures is not required such as suicidality assessment in depression.
Quality measurement is limited by the availability of data. To assess patient outcome, documentation systems for primary care as well as documentation systems for secondary mental health care may be used. This assessment should include instruments to record the first consultation/admission, weekly consultations, and discharge . For specialised care, these could assess sociodemographic data, diagnosis, disease-specific history, treatment course, and outcome. For major depressive disorder, such outcome assessments have been suggested to include the Clinical Global Impressions (CGI) and the Global Assessment of Functioning Scale (GAF), the rating of depression scores by the self-rating Patient Health Questionnaire (brief PHQ-D), and the Beck Depression Inventory (BDI) as well as expert ratings using, i.e., the Hamilton Depression Scale (HAM-D). General aspects of patients’ satisfaction could be measured by the Client Satisfaction Questionnaire . However, a major problem has been identified in routine assessment of patient outcomes. A therapist who is responsible for delivery of a clinical intervention, and thus has a stake in the outcome of the intervention, is not in a position to objectively assess the outcome. Incentives will probably affect rating behaviour most strongly when the individual provider’s job performance is directly assessed . Thus, a rationale approach for quality measurement may combine the assessment of “hard” outcome parameters such as mortality, job integration, and long-term disease course including relapse and remission rates combined with an evaluation of process measures and treatment adherence. Data suggest that the assessment of treatment adherence might have a more positive effect on clinical practice than routine outcome measurements .
A cautionary note appears also necessary for payment for performance schemes in psychiatry using indicator systems. Research indicates that clinicians are influenced by the implications of a new quality and outcomes framework indicator when recording diagnoses, disease severity, or treatment processes . Non-incentivised activities and patients’ concerns may receive less clinical attention . Thus, pay-for-performance or prospective payment schemes based on quality indicator performance may hold their promise to improve outcome quality only when such measures are valid and feasible, adopt a longitudinal perspective on quality management, and are successful at ensuring that all providers who are responsible for a particular patient’s care are held accountable for the quality of care they provide .
Quality management measures are necessary to analyse weak points in routine care, detect opportunities for improvement of care, and check the implementation of guidelines. Because some studies show that adherence to evidence-based guidelines in psychiatry can improve outcomes , whereas others have failed to do so , we further discuss this point in the following section.
Throughout the 1990s, educational initiatives for implementing guidelines were begun, and evidence-based guidelines have been formulated in many countries. Several countries have initiated national clinical guideline programmes [16, 18, 36]. There is an ongoing debate whether evidence-based guidelines improve patient outcomes. Clinical guidelines aim to improve quality of care by advocating best-practice models and reducing treatment variation . Some authors have even claimed that the term evidence-based guideline should only be used if a positive impact of guideline implementation on patient outcomes following guideline implementation has been shown [26, 42]. However, there is little evidence that guideline dissemination alone affects the behaviour of mental health clinicians or general practitioners. Guideline implementation programmes using complicated and multifaceted procedures or participatory approaches appear to have an impact on professional behaviour .
In a systematic review of psychiatric guideline implementation studies, the observed effects on provider performance or patient outcome after implementation were moderate and temporary in most cases. The studies with positive outcomes used complex multifaceted interventions or specific psychological methods to implement guidelines . Interventions associated with better provider performance were multifaceted interventions with ongoing expert consultation, ongoing supervision, or interventions using marketing techniques and psychological theories to overcome guideline implementation obstacles. Only one study using an intensive and costly intervention strategy showed a consistent positive and substantial improvement in the self- and physician-rated psychopathology of depression . Most studies were statistically underpowered to show small intervention effects.
There is, however, some evidence from observational studies that guideline adherence is associated with better outcomes. In a German multisite hospital study, treatment processes and patient outcomes were compared across seven psychiatric hospitals using case-mix adjustment models . Patient structure and treatment processes showed a great variability between hospitals with mental state, chronicity of the disease, and other patient factors being the strongest predictors of clinical outcome. Benchmarking hospitals, a poorer average clinical outcome was associated with lower guideline conformity in a variety of treatment domains, although it is not clear whether better guideline adherence in the hospitals with the best results was a causal factor for their enhanced performance.
Many quality indicators are derived from clinical practice guidelines. Evidence-based guidelines incorporate research evidence and clinical consensus. They provide a useful foundation for quality improvement. The paucity of studies showing positive effects of guideline implementation on patient outcome in mental health care should not discourage quality improvement initiatives based on guideline recommendations. It should be noted that programmes, which are directed only towards increasing guideline adherence, are too simplistic. Measurement-based quality improvement needs organisational changes and may only be successful when positive incentives exist to continually improve treatment quality.
Quality indicators are only one method to measure treatment quality . They have the advantage of being databased and enabling scientific analyses. They can be used to address suboptimal clinical outcomes, reduce the variability of care, and close the gaps between evidence-based guidelines and routine care. However, most indicators in mental health care are not empirically validated themselves, but are rather based on recommendations for interventions that have been evaluated in efficacy studies. Further research may broaden the validation base for single quality indicators. Due to a variety of national and other professional efforts, there is an extensive set of indicators available in mental health care that can be adapted for multiple purposes [19, 21].
Among the basic setting cornerstones for quality indicators are inadequate variations in routine mental health care and a high degree of variability in guideline conformance rates. As an example, in severe mental disorders such as schizophrenia, there has been a trend towards polypharmacy in routine care not supported by evidence-based guidelines [2, 17]. Reducing polypharmacy may reduce complications and side effects of antipsychotic treatment and therefore improve patient outcome. Antipsychotic polypharmacy was suggested as an indicator of guideline adherence in the audits of the National Institute of Clinical Excellence , specified as the number of individuals receiving only one antipsychotic at a time. Based on reviews and meta-analyses, the research evidence to date is consistent with the goal of avoiding antipsychotic polypharmacy in patients who lack guideline-recommended indications for its use . Two of the measures implemented by the Joint Commission on Accreditation of Healthcare Organisations as a core measure set for Hospital-Based Inpatient Psychiatric Services address antipsychotic polypharmacy. The first measure assesses the overall rate. The second measure determines whether clinically appropriate justification has been documented supporting the use of more than one antipsychotic drug . This seems to be a pragmatically based measure of provider performance related to patient outcome.
Other quality indicators have to consider the regional mental healthcare system and need case-mix adjustment to avoid unfair comparisons. In general, quality indicators have to be meaningful, feasible and actionable , and address different dimensions of the mental health care system. Many indicators rely on psychiatric guideline recommendations. Studies show that there may be a differential effect of structured guideline implementation on the quality of provider performance and patient outcome, while the underlying causes for these differential effects are neither obvious nor easily explainable. For example, it is not clear why the Texas Medication Algorithm Project (TMAP) intervention resulted in sustained improvement of patient outcomes in the depression study , but not in the schizophrenia study  and not in the mania study . Based on the results of guideline implementation studies, three implementation components may be necessary to improve patient outcomes by guideline implementation and other measurement-based quality improvement efforts : (1) ongoing support or feedback with an option to use expert consultation, (2) the use of specific psychological models to overcome obstacles to guideline implementation, or (3) social marketing techniques. Rather than primarily relying on information, education and promotion of better quality of care, multifaceted guideline interventions should probably be specifically tailored to raising clinicians’ willingness to change, encouraging behaviour change through motivational techniques, reducing barriers through system reconfiguration, ensuring continued change, and establishing behavioural reinforcers. Behaviour-driven education appears to be more effective than strategies driven by knowledge. It should be noted that measures of rate-based processes and outcomes represent a subset of a broader range of approaches to quality assessment in mental health care .
Dr. Wobrock has accepted paid speaking engagements from AstraZeneca, Bristol-Myers Squibb, Janssen-Cilag, Eli Lilly, Novartis, Organon, Pfizer, and Sanofi-Synthelabo/Aventis, and travel or hospitality not related to a speaking engagement from Astra Zeneca, Eli Lilly, Janssen Cilag, and Sanofi-Synthelabo, and received a research grant from Astra Zeneca.
Dr. Gaebel has participated in speakers bureaus for AstraZeneca, Bristol-Myers Squibb, GlaxoSmithKline, Janssen-Cilag, Eli Lilly, Lundbeck, and Sanofi-Synthelabo/Aventis; has been a member of advisory boards for Eli Lilly, Lundbeck, Novartis, and Wyeth; and has received grant/research support from Bristol-Myers Squibb, Eli Lilly, Wyeth, Janssen-Cilag, and Lundbeck.
Dr. Falkai is a member of the speakers bureaus for AstraZeneca, Eli Lilly, Janssen-Cilag and Lundbeck, and has accepted paid speaking engagements in industry-sponsored symposia from AstraZeneca, Bristol-Myers-Squibb, Eli-Lilly, Janssen-Cilag, Lundbeck and Pfizer, and travel or hospitality not related to a speaking engagement from Astra Zeneca, Bristol-Myers-Squibb, Eli Lilly, Janssen Cilag, Lundbeck and Sanofi-Synthelabo, and received a research grant from Astra Zeneca.
Dr Weinmann reports no financial or other relationships relevant to the subject of this article.
T. Wobrock and S. Weinmann contributed equally to this paper.