PMCCPMCCPMCC

Search tips
Search criteria 

Advanced

 
Logo of jepicomhJournal of Epidemiology and Community HealthVisit this articleSubmit a manuscriptReceive email alertsContact usBMJ
 
J Epidemiol Community Health. 2007 September; 61(9): 791–796.
PMCID: PMC2660002

Measuring the performance of urban healthcare services: results of an international experience

Anna García‐Altés, Carme Borrell, Louis Coté, Aina Plaza, Josep Benet, and Alex Guarga, for the Montreal and Barcelona group on performance of healthcare services

Abstract

The objective of this paper is to apply a framework for country‐level performance assessment to the cities of Montreal, Canada, and Barcelona, Spain, and to use this framework to explore and understand the differences in their health systems. The UK National Health Service Performance Assessment Framework was chosen. Its indicators went through a process of selection, adaptation and prioritisation. Most of them were calculated for the period 2001–3, with data obtained from epidemiological, activity and economic registries. Montreal has a higher number of old people living alone and with limitations on performing one or more activities of daily life, as well as longer hospital stays for several conditions, especially in the case of elderly patients. This highlights a lack of mid‐term, long‐term and home care services. Diabetes‐avoidable hospitalisation rates are also significant in Montreal, and are likely to improve following reforms in primary care. Efficient health policies such as generic drug prescription and major ambulatory surgery are lower in Barcelona. Rates of caesarean deliveries are higher in Barcelona, owing to demographics and clinical practice. Waiting times for knee arthroplasty are longer in Barcelona, which has triggered a plan to reduce them. In both cities, avoidable mortality and the prevalence of smoking have been identified as areas for improvement through preventive services. In conclusion, performance assessment fits perfectly in an urban context, as it has been shown to be a useful tool in designing and monitoring the accomplishment of programmes in both cities, to assess the performance of the services delivered, and for use in policy development.

The objective of performance assessment is to provide governments, health authorities and populations with appropriate information about the state of their healthcare systems.1 Some of the aims of any relevant performance assessment are to build an evidence base on the relationship between a health‐system design and its performance, providing policy makers with crucial tools to develop effective, efficient and equitable systems, and to help determine priorities for healthcare interventions, contributing to their design and management.2

In recent years, an increasing amount of work and attention has been devoted to the measurement of performance within health systems at the country level, with an aim to improve them. The drivers of this trend are rooted in the current characteristics of the environment where healthcare is provided, and the concerns of governments and populations.3 Resource spending constraints, growing public expectations and concerns about safety, quality and equity are increasing the pressure on accountable healthcare systems.

Urban contexts have the elements to develop such an initiative. Big cities have several social problems, inherent to their development: low‐income families, unemployment, immigration, inequalities, an ageing population, small social networks and so on. In particular, the existing health problems in Montreal, Canada and Barcelona, Spain are among the consequences of these particular social conditions: higher HIV/AIDS rates, higher illegal drug consumption, lower birth rates, stronger social class inequalities in health status, higher prevalence of mental health problems and so on.4 Moreover, in many big cities, the available public health and healthcare services try to address these issues with specific policies and programmes, as local governments tend to have an active role in policy development and service delivery assurance as well as public health functions.5 As a result, urban contexts could be seen as a microenvironment in which to develop a performance assessment initiative, much more focused than a country‐level initiative, making it possible to measure the achievement of health goals and the effectiveness of the programmes in place.6

The objective of this paper was to apply a framework for country‐level performance assessment to the cities of Montreal and Barcelona, and to use this framework to explore and understand differences in their health systems. The present initiative is the first one at the city level, which is an indication of the uniqueness of this project, and could be an incentive for other cities interested in this kind of framework.

Methods

The project on performance assessment began in 2002, as one of the agreements between the Montreal Health Authority (Régie Régionale) and the Health Consortium of Barcelona, under the auspices of the declaration of cooperation between the governments of Quebec and Catalunya. Its objective was to develop a set of indicators that would allow Barcelona and Montreal to assess the performance of their healthcare services.

First, a systematic search in PubMed was carried out, looking for performance assessment initiatives that have actually been put in place in different countries. 7 Such initiatives were identified in the UK, Canada, Australia, New Zealand and Sweden.8,9,10,11,12,13 Afterwards, the applicability of each of the five initiatives to an urban context was assessed, valuing their capability to reflect the nuances of that setting, assessing the health status of the inhabitants of big cities, and the public health and healthcare services available. From the revised initiatives, the indicators contained in the UK initiative —UK National Health Service (NHS) Performance Assessment Framework—were seen as the most suitable for Montreal and Barcelona, as they take into account general dimensions of health, while also focusing on areas more specific to those cities (ie, HIV/AIDS, health of immigrants, concerns about community care, particular types of cancer and geographical inequities).8 Additionally, most of the indicators could be obtained from epidemiological, activity and economic information regularly available. Finally, the UK healthcare system is quite similar to the Canadian and Spanish systems, and hence one could expect that the UK NHS Performance Assessment Framework would be applicable.

The UK NHS Performance Assessment Framework, in its 2002 version, defines a set of 51 indicators (table 11)) grouped in dimensions.8 The application of the framework to an urban context would be as follows:

Table thumbnail
Table 1 UK National Health Service performance indicators, 2002*
  • Health improvement: To reflect the overarching aims of improving the general health of urban populations and reducing health inequalities within the cities.
  • Fair access: To assert that the city must offer fair access to health services in relation to people's needs, irrespective of neighbourhood or district of residence, socio‐economic group, ethnicity, age or sex.
  • Effective delivery of appropriate healthcare: To acknowledge that there must be effective, appropriate and timely services that comply with agreed‐upon standards.
  • Efficiency: To ensure that healthcare services are delivered with the minimum of waste, and that the city uses its resources to achieve value for money.
  • Patient/care giver experience: To assess the way in which patients and their care givers experience and view the quality of the care they receive.
  • Health outcomes: To assess the direct contribution of the city healthcare services to improvements in overall health.

The Barcelona city officials invited a multidisciplinary group of experts (10 people) to join the project. These experts were outstanding professionals working in the Barcelona healthcare system in the areas of primary care, long‐term care, mental health, public health, health policy and planning, hospital management and healthcare services research. The multidisciplinary group of experts held three meetings to review the UK NHS Performance Assessment Framework indicators. During these meetings, indicators were selected and adapted and new indicators were added, finally arriving at a set of 107 indicators. To prioritise the indicators obtained, the multidisciplinary group of experts, together with Montreal and Barcelona officials, then assessed four dimensions for each indicator—importance, reliability, information readily available and comparability—on a scale from 1 to 5. Indicators with worse scores, and those that could not be obtained at least in the short term, were dropped, leaving a final set of 76 indicators. Afterwards, three video conferences were carried out in which the Montreal and Barcelona officials discussed the specifications of the indicators, and developed and agreed upon specific technical definitions for each of them. Information for the last year available was collected.

This was an observational cross‐sectional study. For each indicator, counts, percentages or rates were calculated, in accordance with the definitions made. When comparing percentages and rates, they were age standardised using the world standard population 2000–25, so that the values of both cities would be directly comparable. Additionally, information was collected on a set of descriptive indicators related to basic demographic information of both cities and the main characteristics of their healthcare systems.

Results

Populations in both cities are comparable in terms of size, demographic structure and literacy level (table 22).). In Barcelona, there is a bigger proportion of people aged >64 years, especially women (24.9% vs 17.9%); however, in Montreal there is a higher number of people aged >64 years living alone (35.6% vs 25.8%). Montreal has a larger foreign‐born population, as massive immigration is a recent phenomenon in Barcelona; in 2003, foreigners represented 12.8% of the population of the city, while in 1991 it was 1.4%. Regarding resources, Montreal and Barcelona are quite similar in terms of the number of inhabitants per primary care doctor, per nurse and the number of public acute beds. It was not possible to compare long‐term care resources because the organisation and type of services available are different. Per capita public healthcare spending is higher in Montreal.

Table thumbnail
Table 2 Description of the population and healthcare services

In the health improvement dimension, Montreal has a larger proportion of people reporting very good health status (91.6% vs 79.5% in men and 84.7% vs 70.9% in women; table 33).). At the same time, Montreal has a larger proportion of men aged >65 years with limitations to perform one or more activities of daily living (35.3% vs 24.0%), whereas Barcelona has a larger proportion of women (42.3% vs 39.6%) in this situation. With regard to mortality, Montreal has higher rates of coronary diseases and suicide, especially for men. Barcelona, by contrast, has higher rates of cancer, cerebrovascular diseases and respiratory diseases (table 33).). Traffic injuries, mortality and prevalence of smoking are similar in both cities. The percentage of smoking population is high in both cities.

Table thumbnail
Table 3 Performance indicator values for Barcelona and Montreal

With regard to access, Montreal has higher rates of myocardial infarction, angioplasty and hospitalisation due to bypass; hospitalisation rates due to knee replacement are higher in Barcelona; the number of emergencies per inhabitant is similar in both cities (table 33).). In the effective delivery of appropriate services dimension, the number of patients aged >65 years with cerebrovascular accident or with femur fracture, with a hospital stay of >20 days, is higher in Montreal, as well as diabetes‐avoidable hospitalisation rates. The percentage of emergency admissions is also higher in Montreal (58.1% vs 41.2%). Barcelona has a higher percentage of caesarean deliveries (22.2% vs 20.4%).

In the efficiency dimension, Barcelona has a lower substitution rate for major ambulatory surgery (57.9% vs 91.9%), and lower generic drug prescription (14.5% vs 36.5%; table 33).). Gross length of stay of patients with cerebrovascular diseases is higher in Montreal. Regarding user experience, waiting times for knee arthroplasty are much longer in Barcelona (22.3 vs 7.8 months; table 33).

Finally, in the health outcomes dimension, Montreal has a higher inhospital mortality from femur fractures, whereas Barcelona has a higher inhospital mortality from bypass (table 33).). Regarding avoidable mortality, Montreal has higher lung cancer rates for both sexes; Barcelona has higher cerebrovascular disease rates, especially for men.

Discussion

The results of this project are an example of the applicability and usefulness of performance assessment initiatives to an urban context. This is the first effort in applying this kind of framework at the city level. The results made it possible to identify a few issues related to demographics, acute and preventive services available, clinical practice and management techniques in Montreal and Barcelona.

In terms of demographics, Montreal has a higher number of old people living alone and with limitations to perform one or more activities of daily living (table 22).). Related to this is the fact that Montreal has a longer length of stay for several conditions, especially in the case of elderly patients (table 33).). This highlights a lack of mid‐term and long‐term healthcare services, and a greater need for home‐care services in Montreal. Taking action, in early 2005, a plan to improve post‐acute care. Diabetes‐avoidable hospitalisation was initiated in Montreal rates are also significant in Montreal (table 33).). This is something that the Montreal local health authorities hope to see an improvement in for the coming years, following an important primary care reform that was started in mid‐2004.

What this paper adds

  • This paper presents the application of a framework for country‐level performance assessment to the cities of Montreal and Barcelona, and explains how this framework was used to explore and understand differences in their health systems.
  • The results show that performance assessment is applicable in an urban context and can be used as a tool for policy development, as it has allowed both cities to identify a few issues related to demographics, clinical practice and management techniques.
  • The application of a performance framework to two cities, the collaboration between both cities and the extensions of the project make this a unique and successful project in the international arena.

Policy implications

  • Performance assessment is a useful tool to design and monitor the accomplishment of policies and to assess the services delivered.
  • This project is allowing decision makers in Montreal and Barcelona to identify the strengths and weaknesses of the healthcare services of both cities, in terms of available resources and quality of care.

Generic drug prescription and major ambulatory surgery are lower in Barcelona, indicating the low expansion of efficient health policies. Rates of caesarean deliveries are higher in Barcelona, which is probably related to demographics and clinical practice. Waiting times for knee arthroplasty are longer and hospitalisation rates for this condition are higher in Barcelona, mainly because of the slow diffusion of this technique in the city hospitals. This fact led to the development of a plan to reduce them in 1999, mainly guaranteeing a certain annual number of interventions, and the development of clinical units for knee arthroplasty, among other measures. Both indicators should decrease in the future as the number of people waiting for this operation diminishes.

In both cities, avoidable mortality indicates an area for improvement: lung cancer and suicide in Montreal and cerebrovascular diseases in Barcelona give plenty of action to preventive services (table 33).). This is also the case for the prevalence of smoking.

All the performance assessment initiatives at the country level previously mentioned have actually been put in place, and are still being used and continuously improved, a testimony to the importance and usefulness of such initiatives for policy makers and populations. By drawing on other experiences, countries can adapt their frameworks to their own particular conditions, and learn from examples of good practice, thus avoiding errors made elsewhere.14,15,16,17 In fact, some specific studies have been carried out comparing the performance of healthcare services among countries, and within countries, looking for the relationship between health results, health determinants and resources.18,19,20,21,22,23,24 Additionally, this kind of approach could be used to assess health sector reforms and compare healthcare providers.25,26 However, the present initiative is the first at the city level, which confirms the uniqueness of this project, and could be an incentive for other cities interested in this kind of framework.

The comparison of two cities made it possible to confirm previously identified problem areas in each city (eg high mortality from suicide in Montreal, long waiting times for knee artroplasty in Barcelona), and, more interestingly, made it possible to identify other areas in which there is room for improvement (eg, caesarean deliveries and generic drug prescription in Barcelona, emergency admissions in Montreal). In some sense, instead of looking for “benchmarks” for each indicator, each city acts as a benchmark for the other. Continuing this project by collecting annual data would make it possible to monitor any improvements. The international comparison of cities makes the project especially attractive, but even without an intercity comparison, this project is an example of how the application of a performance assessment framework to an urban context could be useful in any city setting, where there is an interest in health system assessment and improvement.

As in any initiative, some limitations could be found: in this study, these are mainly related to the process followed and the availability of information. First, the selection of indicators was based on a pre‐existing set of indicators and carried out by a group of experts. The selection of the UK NHS Performance Assessment Framework was based on a careful assessment of the applicability of all the initiatives found to an urban context, and the similarities of the healthcare systems. The use of a group of experts to select, add and prioritise indicators has the limitations intrinsic to the participation of a particular group of people. Finally, some indicators that either the Barcelona or the Montreal officials thought would be interesting to include were dropped in the prioritisation process because they were not available in both cities. Although this is a limitation, the comparison of available data sources was useful to identify information gaps, and says a lot about the strengths and weaknesses of a system. Improving information systems on drug consumption, vaccines and primary care in Montreal, and on hospital activity information at the Barcelona district level would be a way to advance.

At present, we are working on the development of a strategy to disseminate the results of this project. It will include the presentation of the project in local, national, and international forums through scientific meetings and papers. We have also developed a web page for the project, based on geographical information systems, so that people can interactively see the performance of each indicator, together with its technical definition, and the comparison between cities (http://www.cmis.mtl.rtss.qc.ca/fr/performance/coop_mtl_barcelone/barcelone_presentation.html). We are already working on the continuation of this project, collecting annual data and extending the project to the metropolitan areas of both cities.

This project allows decision makers in Montreal and Barcelona to identify the strengths and weaknesses of the healthcare services of both cities, in terms of available resources and quality of care. The application of a performance assessment framework to an urban context, the collaboration between two cities and the extensions of the project make this a unique and successful project in the international arena.

Acknowledgements

We thank Agència de Salut Pública de Barcelona, Consorci Sanitari de Barcelona, Corporació Sanitària de Barcelona and Agence de la Santé et des Services Sociaux de Montréal, our partners in this project; and the Montreal and Barcelona group on performance of healthcare services: Josep Benet, Mike Benigeri, Carme Borrell, Marc Bourguignon, Xavier Castells, Louis Côté, Danièle Dorval, Jaume Estany, Gonçal Foz, Lluís Franch, Anna García‐Altés, Josep Ma Giralt, Àlex Guarga, José Manuel Pérez‐Castejón, Antoni Plasència, Alfonso Pozuelo, Aina Plaza, Elvira Torné, Corinne Zara and Lauriane Zonco.

Abbreviations

NHS - National Health Service

Footnotes

Funding: No extra funding apart from the authors' institutions.

Competing interests: None.

References

1. Murray C J L, Frenk J. A framework for assessing the performance of health systems. Bull World Health Organ 2000. 78811–820.820 [PubMed]
2. Pan American Health Organization Health system performance assessment and improvement in the region of the Americas. Washington, DC: PAHO, 2001
3. Hurst J. Performance measurement and performance management in OECD health systems: an overview of issues and challenges. “Measuring up” improving health systems performance in OECD countries. OECD Health Conference on Performance Measurement and Reporting. Ottawa, November 2001
4. Borrell C, Pasarín M. Inequalities in health and urban areas. Gac Sanit 2004. 18(Suppl 2)1–4.4 [PubMed]
5. Líndez P, Villalbí J R, Vaqué J. Public health functions, activities and structures: the role of large and middle‐sized municipalities. Gac Sanit 2001. 15164–171.171 [PubMed]
6. Derose S F, Schuster M A, Fielding J E. et al Public health quality measurement: concepts and challenges. Annu Rev Public Health 2002. 231–21.21 [PubMed]
7. García‐Altés A, Zonco L, Borrell C. et al Measuring the performance of health care services: a review of international experiences and their application to an urban context. Gac Sanit 2006. 20316–324.324 [PubMed]
8. National Health Service NHS performance indicators. National figures: February 2002. London: NHS, 2003
9. Canadian Institute for Health Information Health indicators. Vol 2003, Number 2. Ottawa: Canadian Institute for Health Information, 2003 (Catalogue Number 82‐221‐XIE),
10. National Health Performance Committee Fourth national report on health sector performance indicators. A report to the Australian Health Ministers' conference. Sydney: NSW Health Department, 2000
11. Or Z. Improving the performance of health care systems: from measures to action (a review of experiences in four OECD countries). Paris: OECD, 2002 (Labour Market and Social Policy, Occasional Papers Number 57),
12. Hurst J, Jee‐Hugues M. Performance measurement and performance management in OECD Health Systems. Paris: OECD, 2000 (Labour Market and Social Policy, Occasional Paper Number 47),
13. McIntyre D, Rogers L, Heier E J. Overview, history and objectives of performance measurement. Health Care Financ Rev 2001. 227–21.21 [PMC free article] [PubMed]
14. Evans D B, Edejer T T, Lauer J. et al Measuring quality: from the system to the provider. Int J Qual Health Care 2001. 13439–446.446 [PubMed]
15. Upshaw V. The National Public Health Performance Standards Program: would it strengthen governance of local public health? J Public Health Pract 2000. 61–18.18 [PubMed]
16. McPake B, Mills A. What can we learn from international comparisons of health systems and health system reform? Bull World Health Organ 2000. 78811–820.820 [PubMed]
17. Mannion R, Goddard M. Performance measurement and improvement in health care. Appl Health Econ Health Policy 2002. 113–23.23 [PubMed]
18. Feachem R G A, Sekhri N K, White K L. Getting more for their dollar: a comparison of the NHS with California's Kaiser Permanente. BMJ 2002. 324135–141.141 [PMC free article] [PubMed]
19. Evans D B, Tandon A, Murray C J L. et al Comparative efficiency of national health systems: cross national econometric analysis. BMJ 2001. 323307–310.310 [PMC free article] [PubMed]
20. The Conference Board of Canada Challenging health care system sustainability: understanding health system performance of leading countries. Ottawa: The Conference Board of Canada, 2004
21. Hussey P S, Anderson G F, Osborn R. et al How does the quality of care compare in five countries? Health Aff (Millwood) 2004. 2389–99.99 [PubMed]
22. Organisation for Economic Cooperation and Development Towards high‐performing health systems. Paris: OECD, 2004
23. Arah O A, Westert G P. Correlates of health and healthcare performance: applying the Canadian health indicators framework at the provincial‐territorial level. BMC Health Serv Res. 20051 576 [PMC free article] [PubMed]
24. Schoen C, Davis K, How S K. et al U.S. health system performance: a national scorecard, Health Aff (Millwood) 2006. 25w457–w475.w475 [PubMed]
25. Knowles J C, Leighton C, Stinson W. Measuring results of health sector reform for system performance: a handbook of indicators. Special Initiatives Report Number 1. Bethesda, MD: Partnerships for Health Reform, Abt Associates, 1997
26. Abdad P, Rouse A, Mohamemed A. et al Performance league tables: the NHS deserves better. BMJ 2002. 32495–98.98 [PMC free article] [PubMed]

Articles from Journal of Epidemiology and Community Health are provided here courtesy of BMJ Group