|Home | About | Journals | Submit | Contact Us | Français|
The objective of this paper is to apply a framework for country‐level performance assessment to the cities of Montreal, Canada, and Barcelona, Spain, and to use this framework to explore and understand the differences in their health systems. The UK National Health Service Performance Assessment Framework was chosen. Its indicators went through a process of selection, adaptation and prioritisation. Most of them were calculated for the period 2001–3, with data obtained from epidemiological, activity and economic registries. Montreal has a higher number of old people living alone and with limitations on performing one or more activities of daily life, as well as longer hospital stays for several conditions, especially in the case of elderly patients. This highlights a lack of mid‐term, long‐term and home care services. Diabetes‐avoidable hospitalisation rates are also significant in Montreal, and are likely to improve following reforms in primary care. Efficient health policies such as generic drug prescription and major ambulatory surgery are lower in Barcelona. Rates of caesarean deliveries are higher in Barcelona, owing to demographics and clinical practice. Waiting times for knee arthroplasty are longer in Barcelona, which has triggered a plan to reduce them. In both cities, avoidable mortality and the prevalence of smoking have been identified as areas for improvement through preventive services. In conclusion, performance assessment fits perfectly in an urban context, as it has been shown to be a useful tool in designing and monitoring the accomplishment of programmes in both cities, to assess the performance of the services delivered, and for use in policy development.
The objective of performance assessment is to provide governments, health authorities and populations with appropriate information about the state of their healthcare systems.1 Some of the aims of any relevant performance assessment are to build an evidence base on the relationship between a health‐system design and its performance, providing policy makers with crucial tools to develop effective, efficient and equitable systems, and to help determine priorities for healthcare interventions, contributing to their design and management.2
In recent years, an increasing amount of work and attention has been devoted to the measurement of performance within health systems at the country level, with an aim to improve them. The drivers of this trend are rooted in the current characteristics of the environment where healthcare is provided, and the concerns of governments and populations.3 Resource spending constraints, growing public expectations and concerns about safety, quality and equity are increasing the pressure on accountable healthcare systems.
Urban contexts have the elements to develop such an initiative. Big cities have several social problems, inherent to their development: low‐income families, unemployment, immigration, inequalities, an ageing population, small social networks and so on. In particular, the existing health problems in Montreal, Canada and Barcelona, Spain are among the consequences of these particular social conditions: higher HIV/AIDS rates, higher illegal drug consumption, lower birth rates, stronger social class inequalities in health status, higher prevalence of mental health problems and so on.4 Moreover, in many big cities, the available public health and healthcare services try to address these issues with specific policies and programmes, as local governments tend to have an active role in policy development and service delivery assurance as well as public health functions.5 As a result, urban contexts could be seen as a microenvironment in which to develop a performance assessment initiative, much more focused than a country‐level initiative, making it possible to measure the achievement of health goals and the effectiveness of the programmes in place.6
The objective of this paper was to apply a framework for country‐level performance assessment to the cities of Montreal and Barcelona, and to use this framework to explore and understand differences in their health systems. The present initiative is the first one at the city level, which is an indication of the uniqueness of this project, and could be an incentive for other cities interested in this kind of framework.
The project on performance assessment began in 2002, as one of the agreements between the Montreal Health Authority (Régie Régionale) and the Health Consortium of Barcelona, under the auspices of the declaration of cooperation between the governments of Quebec and Catalunya. Its objective was to develop a set of indicators that would allow Barcelona and Montreal to assess the performance of their healthcare services.
First, a systematic search in PubMed was carried out, looking for performance assessment initiatives that have actually been put in place in different countries. 7 Such initiatives were identified in the UK, Canada, Australia, New Zealand and Sweden.8,9,10,11,12,13 Afterwards, the applicability of each of the five initiatives to an urban context was assessed, valuing their capability to reflect the nuances of that setting, assessing the health status of the inhabitants of big cities, and the public health and healthcare services available. From the revised initiatives, the indicators contained in the UK initiative —UK National Health Service (NHS) Performance Assessment Framework—were seen as the most suitable for Montreal and Barcelona, as they take into account general dimensions of health, while also focusing on areas more specific to those cities (ie, HIV/AIDS, health of immigrants, concerns about community care, particular types of cancer and geographical inequities).8 Additionally, most of the indicators could be obtained from epidemiological, activity and economic information regularly available. Finally, the UK healthcare system is quite similar to the Canadian and Spanish systems, and hence one could expect that the UK NHS Performance Assessment Framework would be applicable.
The UK NHS Performance Assessment Framework, in its 2002 version, defines a set of 51 indicators (table 11)) grouped in dimensions.8 The application of the framework to an urban context would be as follows:
The Barcelona city officials invited a multidisciplinary group of experts (10 people) to join the project. These experts were outstanding professionals working in the Barcelona healthcare system in the areas of primary care, long‐term care, mental health, public health, health policy and planning, hospital management and healthcare services research. The multidisciplinary group of experts held three meetings to review the UK NHS Performance Assessment Framework indicators. During these meetings, indicators were selected and adapted and new indicators were added, finally arriving at a set of 107 indicators. To prioritise the indicators obtained, the multidisciplinary group of experts, together with Montreal and Barcelona officials, then assessed four dimensions for each indicator—importance, reliability, information readily available and comparability—on a scale from 1 to 5. Indicators with worse scores, and those that could not be obtained at least in the short term, were dropped, leaving a final set of 76 indicators. Afterwards, three video conferences were carried out in which the Montreal and Barcelona officials discussed the specifications of the indicators, and developed and agreed upon specific technical definitions for each of them. Information for the last year available was collected.
This was an observational cross‐sectional study. For each indicator, counts, percentages or rates were calculated, in accordance with the definitions made. When comparing percentages and rates, they were age standardised using the world standard population 2000–25, so that the values of both cities would be directly comparable. Additionally, information was collected on a set of descriptive indicators related to basic demographic information of both cities and the main characteristics of their healthcare systems.
Populations in both cities are comparable in terms of size, demographic structure and literacy level (table 22).). In Barcelona, there is a bigger proportion of people aged >64 years, especially women (24.9% vs 17.9%); however, in Montreal there is a higher number of people aged >64 years living alone (35.6% vs 25.8%). Montreal has a larger foreign‐born population, as massive immigration is a recent phenomenon in Barcelona; in 2003, foreigners represented 12.8% of the population of the city, while in 1991 it was 1.4%. Regarding resources, Montreal and Barcelona are quite similar in terms of the number of inhabitants per primary care doctor, per nurse and the number of public acute beds. It was not possible to compare long‐term care resources because the organisation and type of services available are different. Per capita public healthcare spending is higher in Montreal.
In the health improvement dimension, Montreal has a larger proportion of people reporting very good health status (91.6% vs 79.5% in men and 84.7% vs 70.9% in women; table 33).). At the same time, Montreal has a larger proportion of men aged >65 years with limitations to perform one or more activities of daily living (35.3% vs 24.0%), whereas Barcelona has a larger proportion of women (42.3% vs 39.6%) in this situation. With regard to mortality, Montreal has higher rates of coronary diseases and suicide, especially for men. Barcelona, by contrast, has higher rates of cancer, cerebrovascular diseases and respiratory diseases (table 33).). Traffic injuries, mortality and prevalence of smoking are similar in both cities. The percentage of smoking population is high in both cities.
With regard to access, Montreal has higher rates of myocardial infarction, angioplasty and hospitalisation due to bypass; hospitalisation rates due to knee replacement are higher in Barcelona; the number of emergencies per inhabitant is similar in both cities (table 33).). In the effective delivery of appropriate services dimension, the number of patients aged >65 years with cerebrovascular accident or with femur fracture, with a hospital stay of >20 days, is higher in Montreal, as well as diabetes‐avoidable hospitalisation rates. The percentage of emergency admissions is also higher in Montreal (58.1% vs 41.2%). Barcelona has a higher percentage of caesarean deliveries (22.2% vs 20.4%).
In the efficiency dimension, Barcelona has a lower substitution rate for major ambulatory surgery (57.9% vs 91.9%), and lower generic drug prescription (14.5% vs 36.5%; table 33).). Gross length of stay of patients with cerebrovascular diseases is higher in Montreal. Regarding user experience, waiting times for knee arthroplasty are much longer in Barcelona (22.3 vs 7.8 months; table 33).
Finally, in the health outcomes dimension, Montreal has a higher inhospital mortality from femur fractures, whereas Barcelona has a higher inhospital mortality from bypass (table 33).). Regarding avoidable mortality, Montreal has higher lung cancer rates for both sexes; Barcelona has higher cerebrovascular disease rates, especially for men.
The results of this project are an example of the applicability and usefulness of performance assessment initiatives to an urban context. This is the first effort in applying this kind of framework at the city level. The results made it possible to identify a few issues related to demographics, acute and preventive services available, clinical practice and management techniques in Montreal and Barcelona.
In terms of demographics, Montreal has a higher number of old people living alone and with limitations to perform one or more activities of daily living (table 22).). Related to this is the fact that Montreal has a longer length of stay for several conditions, especially in the case of elderly patients (table 33).). This highlights a lack of mid‐term and long‐term healthcare services, and a greater need for home‐care services in Montreal. Taking action, in early 2005, a plan to improve post‐acute care. Diabetes‐avoidable hospitalisation was initiated in Montreal rates are also significant in Montreal (table 33).). This is something that the Montreal local health authorities hope to see an improvement in for the coming years, following an important primary care reform that was started in mid‐2004.
Generic drug prescription and major ambulatory surgery are lower in Barcelona, indicating the low expansion of efficient health policies. Rates of caesarean deliveries are higher in Barcelona, which is probably related to demographics and clinical practice. Waiting times for knee arthroplasty are longer and hospitalisation rates for this condition are higher in Barcelona, mainly because of the slow diffusion of this technique in the city hospitals. This fact led to the development of a plan to reduce them in 1999, mainly guaranteeing a certain annual number of interventions, and the development of clinical units for knee arthroplasty, among other measures. Both indicators should decrease in the future as the number of people waiting for this operation diminishes.
In both cities, avoidable mortality indicates an area for improvement: lung cancer and suicide in Montreal and cerebrovascular diseases in Barcelona give plenty of action to preventive services (table 33).). This is also the case for the prevalence of smoking.
All the performance assessment initiatives at the country level previously mentioned have actually been put in place, and are still being used and continuously improved, a testimony to the importance and usefulness of such initiatives for policy makers and populations. By drawing on other experiences, countries can adapt their frameworks to their own particular conditions, and learn from examples of good practice, thus avoiding errors made elsewhere.14,15,16,17 In fact, some specific studies have been carried out comparing the performance of healthcare services among countries, and within countries, looking for the relationship between health results, health determinants and resources.18,19,20,21,22,23,24 Additionally, this kind of approach could be used to assess health sector reforms and compare healthcare providers.25,26 However, the present initiative is the first at the city level, which confirms the uniqueness of this project, and could be an incentive for other cities interested in this kind of framework.
The comparison of two cities made it possible to confirm previously identified problem areas in each city (eg high mortality from suicide in Montreal, long waiting times for knee artroplasty in Barcelona), and, more interestingly, made it possible to identify other areas in which there is room for improvement (eg, caesarean deliveries and generic drug prescription in Barcelona, emergency admissions in Montreal). In some sense, instead of looking for “benchmarks” for each indicator, each city acts as a benchmark for the other. Continuing this project by collecting annual data would make it possible to monitor any improvements. The international comparison of cities makes the project especially attractive, but even without an intercity comparison, this project is an example of how the application of a performance assessment framework to an urban context could be useful in any city setting, where there is an interest in health system assessment and improvement.
As in any initiative, some limitations could be found: in this study, these are mainly related to the process followed and the availability of information. First, the selection of indicators was based on a pre‐existing set of indicators and carried out by a group of experts. The selection of the UK NHS Performance Assessment Framework was based on a careful assessment of the applicability of all the initiatives found to an urban context, and the similarities of the healthcare systems. The use of a group of experts to select, add and prioritise indicators has the limitations intrinsic to the participation of a particular group of people. Finally, some indicators that either the Barcelona or the Montreal officials thought would be interesting to include were dropped in the prioritisation process because they were not available in both cities. Although this is a limitation, the comparison of available data sources was useful to identify information gaps, and says a lot about the strengths and weaknesses of a system. Improving information systems on drug consumption, vaccines and primary care in Montreal, and on hospital activity information at the Barcelona district level would be a way to advance.
At present, we are working on the development of a strategy to disseminate the results of this project. It will include the presentation of the project in local, national, and international forums through scientific meetings and papers. We have also developed a web page for the project, based on geographical information systems, so that people can interactively see the performance of each indicator, together with its technical definition, and the comparison between cities (http://www.cmis.mtl.rtss.qc.ca/fr/performance/coop_mtl_barcelone/barcelone_presentation.html). We are already working on the continuation of this project, collecting annual data and extending the project to the metropolitan areas of both cities.
This project allows decision makers in Montreal and Barcelona to identify the strengths and weaknesses of the healthcare services of both cities, in terms of available resources and quality of care. The application of a performance assessment framework to an urban context, the collaboration between two cities and the extensions of the project make this a unique and successful project in the international arena.
We thank Agència de Salut Pública de Barcelona, Consorci Sanitari de Barcelona, Corporació Sanitària de Barcelona and Agence de la Santé et des Services Sociaux de Montréal, our partners in this project; and the Montreal and Barcelona group on performance of healthcare services: Josep Benet, Mike Benigeri, Carme Borrell, Marc Bourguignon, Xavier Castells, Louis Côté, Danièle Dorval, Jaume Estany, Gonçal Foz, Lluís Franch, Anna García‐Altés, Josep Ma Giralt, Àlex Guarga, José Manuel Pérez‐Castejón, Antoni Plasència, Alfonso Pozuelo, Aina Plaza, Elvira Torné, Corinne Zara and Lauriane Zonco.
NHS - National Health Service
Funding: No extra funding apart from the authors' institutions.
Competing interests: None.