|Home | About | Journals | Submit | Contact Us | Français|
Composite measures of performance are insufficient on their own
Why are high performing healthcare organisations so hard to find? In this week's BMJ, Wilson and colleagues report a study that evaluates 69 facilities in 30 US states that receive categorical funding for HIV services.1 The authors assessed performance using a bundle of eight clinical measures considered by a panel of experts to represent high quality of care for HIV. They found that few organisations scored highly across more than a handful of measures.
Interpreting the results at face value suggests that these facilities are not performing well, and that their organisations do not support strong systems of care for people living with HIV. We would expect all clinics to provide comprehensive elements of care that have been shown to improve patients' outcomes. However, closer scrutiny of the study raises methodological and theoretical questions about the selection and measurement of the indicators and, importantly, the association between overall performance and designation as a high performing healthcare organisation.
Composite measures are commonly used to monitor performance in healthcare systems. An overall score is computed by aggregating each component into a bundle of related measures. Bundled measures, however, are not all alike. Selecting measures appropriate to the system under review and defining those measures consistently is crucial to generating meaningful performance data.2 As Nolan and Berwick3 point out, some groups of measures are linked because they constitute a sequence of essential steps leading to one desired outcome, such as an infection control procedure, and omission of one measure compromises the outcome. Other bundles are related to disease; these bundles include appropriate monitoring, treatment, and preventive screening indicators that may need a broad range of strategies to implement within one system of care.
Although the bundle used in Wilson and colleagues' study comprised measures of comprehensive ambulatory care for HIV, they reflect a different type of complexity. Four measures—prescription of highly active antiretroviral therapy (HAART), prophylaxis against Pneumocystis carinii, screening for hepatitis C, and flu vaccination—require a provider to follow recommended guidelines. Three other measures—screening for cervical cancer, screening for tuberculosis, and suppression of viral load—are partly dependent on the patients' behaviour and might not yield a reliable picture of organisational quality.
Suppression of viral load is a particularly difficult outcome to interpret as several variables determine the likelihood of response. In some people, suppression never occurs because of resistance to antiretroviral agents; others recently started on HAART might not yet show suppression despite responding to treatment. More importantly, some people do not show suppression because they do not adhere to their regimen.
Although these measures form an ideal package, they are affected not only by the behaviour of the provider, but also by delivery of services, the structure of the organisation, and the behaviour of patients. Ideally, high quality care provided by a model system would consistently perform the activities associated with all these measures; in truth, this rarely occurs. Even if all measures are satisfied during one time period they are not likely to be sustained over time.4
High performing organisations are characterised by sustainable performance over time. As complex and dynamic units, organisations face staff turnover, changing leadership, and the effects of external factors. They have unique cultures that influence the quality and sustainability of performance.
Studies of high performing organisations5 6 7 8 9 find that a good infrastructure is crucial for sustained high performance. Infrastructure unifies important organisational elements, including meaningful strategy and inspired vision implemented by a consistent leadership, a commitment to meeting the expectations of consumers, a dedicated structure for quality, and constant feedback to staff.7 8 Moreover, attaining the highest levels of performance is not an overnight effort. Time is needed for whole system transformation that includes changing culture, redesigning processes, and crafting solid information systems that support useful and robust measurement, while keeping the vision of quality in sight at all times. Once this transformation is complete, appropriate measures of performance should consistently reflect improved outcomes.
A comprehensive package is necessary to measure system-wide performance, but a one time measurement is clearly not sufficient. Performance must be measured over time to identify whether quality, once achieved, is sustained. In addition, models of organisational clinical performance and frameworks for quality assessment must be united to help us understand the attributes of healthcare organisations that perform well. Frameworks such as the Malcolm Baldridge quality award criteria and the European Foundation quality management excellence model10 offer a starting point to link organisational variables to clinical outcomes. We have much to learn about how these models intersect, and we need a better understanding of the relation between structural elements and clinical performance. The paradigms of effectiveness research, the psychology of planned social change, organisational theory, and social anthropology may contribute to our understanding of dynamic and complex organisations.11 Identifying essential attributes of high performing organisations and disseminating strategies for improvement will help us to achieve consistent performance of the highest quality to benefit the general population.
What then is a high performing organisation? Even without looking at performance data, high performance is often apparent when visiting an organisation—performance data are openly displayed on the walls, staff are familiar with their performance, and they openly share ideas for improvement. Evidence of patient input and a commitment to meeting consumer expectations confirm that an organisation is performing well. When the organisational elements supporting sustainable high performance are in place, measurement across appropriate elements is bound to reflect improvement; while performance rates may not all be in the top quarter, they may well be when measured the next time.
This article was published on bmj.com on 29 October 2007
Competing interests: None declared.
Provenance and peer review: Commissioned; not externally peer reviewed.