Process evaluations are recommended to open the ‘black box’ of complex interventions evaluated in trials, but there is limited guidance to help researchers design process evaluations. Much current literature on process evaluations of complex interventions focuses on qualitative methods, with less attention paid to quantitative methods. This discrepancy led us to develop our own framework for designing process evaluations of cluster-randomised controlled trials.
We reviewed recent theoretical and methodological literature and selected published process evaluations; these publications identified a need for structure to help design process evaluations. We drew upon this literature to develop a framework through iterative exchanges, and tested this against published evaluations.
The developed framework presents a range of candidate approaches to understanding trial delivery, intervention implementation and the responses of targeted participants. We believe this framework will be useful to others designing process evaluations of complex intervention trials. We also propose key information that process evaluations could report to facilitate their identification and enhance their usefulness.
There is no single best way to design and carry out a process evaluation. Researchers will be faced with choices about what questions to focus on and which methods to use. The most appropriate design depends on the purpose of the process evaluation; the framework aims to help researchers make explicit their choices of research questions and methods.
Process evaluation; Complex intervention; Cluster-randomised controlled trial; Qualitative; Quantitative; Reporting
High-risk prescribing in primary care is common and causes considerable harm. Feedback interventions to improve care are attractive because they are relatively cheap to widely implement. There is good evidence that feedback has small to moderate effects, but the most recent Cochrane review called for more high-quality, large trials that explicitly test different forms of feedback.
Methods and analysis
The study is a three-arm cluster-randomised trial with general practices being randomised and outcomes measured at patient level. 262 practices in three Scottish Health Board areas have been randomised (94% of all possible practices). The two active arms receive different forms of prescribing safety data feedback, with rates of high-risk prescribing compared with a ‘usual care’ arm. Sample size estimation used baseline data from participating practices. With 85 practices randomised to each arm, then there is 93% power to detect a 25% difference in the percentage of high-risk prescribing (from 6.1% to 4.5%) between the usual care arm and each intervention arm. The primary outcome is a composite of six high-risk prescribing measures (antipsychotic prescribing to people aged ≥75 years; non-steroidal anti-inflammatory drug (NSAID) prescribing to people aged ≥75 without gastroprotection; NSAID prescribing to people prescribed aspirin/clopidogrel without gastroprotection; NSAID prescribing to people prescribed an ACE inhibitor/angiotensin receptor blocker and a diuretic; NSAID prescription to people prescribed an oral anticoagulant without gastroprotection; aspirin/clopidogrel prescription to people prescribed an oral anticoagulant without gastroprotection). The primary analysis will use multilevel modelling to account for repeated measurement of outcomes in patients clustered within practices.
Ethics and dissemination
The study was reviewed and approved by the NHS Tayside Committee on Medical Research Ethics B (11/ES/0001). The study will be disseminated via a final report to the funder with a publicly available research summary, and peer reviewed publications.
ClinicalTrials.gov, dossier number NCT01602705.
Primary Care; Clinical Pharmacology; Randomised controlled trials; Audit and feedback; Medication review; Medication errors
Antidepressant prescribing is increasing worldwide, prompting policy interventions and targets to halt the rise.
To examine time trends in GP antidepressant prescribing using patient-level data.
Design and setting
Longitudinal population database of all community pharmacy dispensed prescriptions for all 325 000 residents of theTayside region of Scotland.
In each of 3 study years (1995/1996, 2000/2001 and 2006/2007), the volume of antidepressants prescribed was calculated, and numbers of patients prescribed antidepressants in each year, mean treatment duration, and mean dose per patient in that year examined using descriptive statistics.
Total drug volume increased threefold between 1995/1996 and 2006/2007, largely driven by increases in selective serotonin reuptake inhibitor (SSRI) prescribing, and laterally also in ‘other’ antidepressant prescribing. Tricyclic prescribing is static, but low-dose amitriptyline increasingly dominates this drug class. Increased drug volume was initially driven by increasing patient numbers (from 8.0% of the population prescribed at least once in 1995/1996 to 11.9% in 2000/2001) and increased treatment duration (from 170 days in the measurement year to 200). Latterly, drug volume increases are increasingly attributable to longer duration of treatment and higher mean daily dose.
The large rise in antidepressant volumes is caused by a complex mixture of more patients being prescribed SSRI and ‘other’ antidepressants, the use of higher doses, and longer durations of treatment, with the balance changing overtime. Tricyclic prescribing is now largely low dose, and probably for conditions otherthan depression. Interventions to improve the quality of antidepressant prescribing need to be more subtle than blanket targets to reduce the total volume of antidepressants prescribed.
antidepressant drugs; family practice; prescribing patterns; Scotland
Trials of complex interventions are criticized for being ‘black box’, so the UK Medical Research Council recommends carrying out a process evaluation to explain the trial findings. We believe it is good practice to pre-specify and publish process evaluation protocols to set standards and minimize bias. Unlike protocols for trials, little guidance or standards exist for the reporting of process evaluations. This paper presents the mixed-method process evaluation protocol of a cluster randomized trial, drawing on a framework designed by the authors.
This mixed-method evaluation is based on four research questions and maps data collection to a logic model of how the data-driven quality improvement in primary care (DQIP) intervention is expected to work. Data collection will be predominately by qualitative case studies in eight to ten of the trial practices, focus groups with patients affected by the intervention and quantitative analysis of routine practice data, trial outcome and questionnaire data and data from the DQIP intervention.
We believe that pre-specifying the intentions of a process evaluation can help to minimize bias arising from potentially misleading post-hoc analysis. We recognize it is also important to retain flexibility to examine the unexpected and the unintended. From that perspective, a mixed-methods evaluation allows the combination of exploratory and flexible qualitative work, and more pre-specified quantitative analysis, with each method contributing to the design, implementation and interpretation of the other.
As well as strengthening the study the authors hope to stimulate discussion among their academic colleagues about publishing protocols for evaluations of randomized trials of complex interventions.
Data-driven quality improvement in primary care trial registration
Complex intervention; Process evaluation; Protocol; Mixed methods; Randomized controlled trial
Influenza immunisation is recommended for all people aged ≥65 years and younger people with particular chronic diseases. The Quality and Outcomes Framework (QOF) has provided new financial incentives for influenza immunisation since 2004.
To determine the impact of the 2004 UK General Medical Services contract on the overall uptake of, and socioeconomic inequalities associated with, influenza immunisation.
Design and setting
Retrospective general-practice population database analysis in 15 general practices in Scotland, UK.
Changes in influenza-immunisation uptake for those in at-risk groups between 2003–2004 and 2006–2007 were measured, and variation in uptake examined using multilevel modelling.
Uptake rose from 67.9% in 2003–2004 to 71.4% in 2006–2007. The largest increases were seen in those aged <65 years with chronic disease, with uptake rising from 49.6% to 58.4%, but rates remained considerably lower than in those aged ≥65 years. Differences between practices narrowed (median odds ratio [OR] for two patients randomly selected from different practices: 2.13 (95% confidence interval [CI] = 2.00 to 2.26) in 2003–2004 versus 1.44 (95% CI = 1.40 to 1.49) in 2006–2007. However, inequalities in uptake by patient socioeconomic status did not change: adjusted OR for most deprived versus most affluent was 0.75 (95% CI = 0.70 to 0.80) in 2003–2004 versus 0.72 (95% CI = 0.68 to 0.76) in 2006–2007.
Overall uptake rose significantly and differences between practices narrowed considerably. However, socioeconomic and age inequalities in influenza immunisation persisted in the first 3 years of the QOF. This contrasts with other ecological analyses, which have concluded that the QOF has reduced inequalities. The impact of financial incentives on inequalities is likely to vary, and some kinds of care may require more targeted improvement activity and support.
contracts; disparities; family practice; influenza (human); pay for performance; socioeconomic factors
Medical revalidation decisions need to be reliable if they are to reassure on the quality and safety of professional practice. This study tested an innovative method in which general practitioners (GPs) were assessed on their reflection and response to a set of externally specified feedback.
Setting and participants
60 GPs and 12 GP appraisers in the Tayside region of Scotland, UK.
A feedback dataset was specified as (1) GP-specific data collected by GPs themselves (patient and colleague opinion; open book self-evaluated knowledge test; complaints) and (2) Externally collected practice-level data provided to GPs (clinical quality and prescribing safety). GPs' perceptions of whether the feedback covered UK General Medical Council specified attributes of a ‘good doctor’ were examined using a mapping exercise. GPs' professionalism was examined in terms of appraiser assessment of GPs' level of insightful practice, defined as: engagement with, insight into and appropriate action on feedback data. The reliability of assessment of insightful practice and subsequent recommendations on GPs' revalidation by face-to-face and anonymous assessors were investigated using Generalisability G-theory.
Main outcome measures
Coverage of General Medical Council attributes by specified feedback and reliability of assessor recommendations on doctors' suitability for revalidation.
Face-to-face assessment proved unreliable. Anonymous global assessment by three appraisers of insightful practice was highly reliable (G=0.85), as were revalidation decisions using four anonymous assessors (G=0.83).
Unlike face-to-face appraisal, anonymous assessment of insightful practice offers a valid and reliable method to decide GP revalidation. Further validity studies are needed.
Audit and feedback; continuous quality improvement; general practice; governance; medical education; evaluation methodology
High-risk prescribing of non-steroidal anti-inflammatory drugs (NSAIDs) and antiplatelet agents accounts for a significant proportion of hospital admissions due to preventable adverse drug events. The recently completed PINCER trial has demonstrated that a one-off pharmacist-led information technology (IT)-based intervention can significantly reduce high-risk prescribing in primary care, but there is evidence that effects decrease over time and employing additional pharmacists to facilitate change may not be sustainable.
We will conduct a cluster randomised controlled with a stepped wedge design in 40 volunteer general practices in two Scottish health boards. Eligible practices are those that are using the INPS Vision clinical IT system, and have agreed to have relevant medication-related data to be automatically extracted from their electronic medical records. All practices (clusters) that agree to take part will receive the data-driven quality improvement in primary care (DQIP) intervention, but will be randomised to one of 10 start dates. The DQIP intervention has three components: a web-based informatics tool that provides weekly updated feedback of targeted prescribing at practice level, prompts the review of individual patients affected, and summarises each patient's relevant risk factors and prescribing; an outreach visit providing education on targeted prescribing and training in the use of the informatics tool; and a fixed payment of 350 GBP (560 USD; 403 EUR) up front and a small payment of 15 GBP (24 USD; 17 EUR) for each patient reviewed in the 12 months of the intervention. We hypothesise that the DQIP intervention will reduce a composite of nine previously validated measures of high-risk prescribing. Due to the nature of the intervention, it is not possible to blind practices, the core research team, or the data analyst. However, outcome assessment is entirely objective and automated. There will additionally be a process and economic evaluation alongside the main trial.
The DQIP intervention is an example of a potentially sustainable safety improvement intervention that builds on the existing National Health Service IT-infrastructure to facilitate systematic management of high-risk prescribing by existing practice staff. Although the focus in this trial is on Non-steroidal anti-inflammatory drugs and antiplatelets, we anticipate that the tested intervention would be generalisable to other types of prescribing if shown to be effective.
ClinicalTrials.gov, dossier number: NCT01425502
Adverse drug event; Non-steroidal anti-inflammatory drug; Antiplatelet; Medication error; Medication review; Decision support systems; Clinical; Stepped wedge; Randomised controlled trial; Primary healthcare
Addressing the problem of preventable drug related morbidity (PDRM) in primary care is a challenge for health care systems internationally. The increasing implementation of clinical information systems in the UK and internationally provide new opportunities to systematically identify patients at risk of PDRM for targeted medication review. The objectives of this study were (1) to develop a set of explicit medication assessment criteria to identify patients with sub-optimally effective or high-risk medication use from electronic medical records and (2) to identify medication use topics that are perceived by UK primary care clinicians to be priorities for quality and safety improvement initiatives.
For objective (1), a 2-round consensus process based on the RAND/UCLA Appropriateness Method (RAM) was conducted, in which candidate criteria were identified from the literature and scored by a panel of 10 experts for 'appropriateness' and 'necessity'. A set of final criteria was generated from candidates accepted at each level. For objective (2), thematically related final criteria were clustered into 'topics', from which a panel of 26 UK primary care clinicians identified priorities for quality improvement in a 2-round Delphi exercise.
(1) The RAM process yielded a final set of 176 medication assessment criteria organised under the domains 'quality' and 'safety', each classified as targeting 'appropriate/necessary to do' (quality) or 'inappropriate/necessary to avoid' (safety) medication use. Fifty-two final 'quality' assessment criteria target patients with unmet indications, sub-optimal selection or intensity of beneficial drug treatments. A total of 124 'safety' assessment criteria target patients with unmet needs for risk-mitigating agents, high-risk drug selection, excessive dose or duration, inconsistent monitoring or dosing instructions. (2) The UK Delphi panel identified 11 (23%) of 47 scored topics as 'high priority' for quality improvement initiatives in primary care.
The developed criteria set complements existing medication assessment instruments in that it is not limited to the elderly, can be implemented in electronic data sets and focuses on drug groups and conditions implicated in common and/or severe PDRM in primary care. Identified priorities for quality and safety improvement can guide the selection of targets for initiatives to address the PDRM problem in primary care.
Medication error; quality indicator; primary health care; adverse drug events; preventable drug related morbidity
The authors aimed to determine US and UK doctors' professional values and reported behaviours, and the extent to which these vary with the context of care.
1891 US and 1078 UK doctors completed the survey (64.4% and 40.3% response rate respectively). Multivariate logistic regression was used to compare responses to identical questions in the two surveys.
UK doctors were more likely to have developed practice guidelines (82.8% UK vs 49.6% US, p<0.001) and to have taken part in a formal medical error-reduction programme (70.9% UK vs 55.7% US, p<0.001). US doctors were more likely to agree about the need for periodic recertification (completely agree 23.4% UK vs 53.9% US, p<0.001). Nearly a fifth of doctors had direct experience of an impaired or incompetent colleague in the previous 3 years. Where the doctor had not reported the colleague to relevant authorities, reasons included thinking that someone else was taking care of the problem, believing that nothing would happen as a result, or fear of retribution. UK doctors were more likely than US doctors to agree that significant medical errors should always be disclosed to patients. More US doctors reported that they had not disclosed an error to a patient because they were afraid of being sued.
The context of care may influence both how professional values are expressed and the extent to which behaviours are in line with stated values. Doctors have an important responsibility to develop their healthcare systems in ways which will support good professional behaviour.
Culture; health policy
To examine if the quality of primary medical care varies with remoteness from urban settlements.
Cross‐sectional analysis of publicly available data of 18 process and intermediate outcome measures for people with coronary heart disease (CHD), diabetes and stroke.
Setting and participants
Populations registered with 912 general practices in Scotland grouped into three categories by level of remoteness from urban settlements: not remote, remote and very remote.
Main outcome measures
Mean percentages achieving quality indicators and interquartile range scores.
Remote and very remote practices were more likely to have characteristics associated with low Quality and Outcomes Framework (QOF) total points score (smaller, higher capitation income, dispensing practice, and had lower statin prescribing despite higher prevalence of cardiovascular disease and diabetes). However, in contrast with previous research, there was little evidence that quality of care was lower in more remote areas for the 18 process and intermediate outcome measures examined. The exception was significantly lower cholesterol measurement and control in people with CHD, diabetes and stroke attending very remote practices (p<0.01) and β‐blocker prescription in CHD (p = 0.01).
Under QOF, there are few differences in the quality of care delivered to patients in practices with different degrees of remoteness. The differences in achievement for cholesterol were consistent with lower rates of statin prescribing relative to disease burden in very remote practices. No differences were found for complex process measures such as retinopathy screening, implying that differences under QOF are more likely to be due to slower adoption of evidence‐based practice than access problems. Examining this will require analysis of individual patient data.
Many patients with hypertension remain undertreated despite the apparent rosy picture given by doctors meeting current targets. Different measures are needed to overcome therapeutic inertia, argue Bruce Guthrie, Melanie Inkster, and Tom Fahey
Determining the effectiveness of complex interventions can be difficult and time consuming. Neil C Campbell and colleagues explain the importance of ground work in getting usable results
The Shipman Inquiry recommended mortality rate monitoring if it could be ‘shown to be workable’ in detecting a future mass murderer in general practice.
To examine the effectiveness of cumulative sum (CUSUM) charts, cross-sectional Shewhart charts, and exponentially-weighted, moving-average control charts in mortality monitoring at practice level.
Design of study
Analysis of Scottish routine general practice data combined with estimation of control chart effectiveness in detecting a ‘murderer’ in a simulated dataset.
Practice stability was calculated from routine data to determine feasible lengths of monitoring. A simulated dataset of 405 000 ‘patients’ was created, registered with 75 ‘practices’ whose underlying mortality rates varied with the same distribution as case-mix-adjusted mortality in all Scottish practices. The sensitivity of each chart to detect five and 10 excess deaths was examined in repeated simulations. The sensitivity of control charts to excess deaths in simulated data, and the number of alarm signals when control charts were applied to routine data were estimated.
Practice instability limited the length of monitoring and modelling was consequently restricted to a 3-year period. Monitoring mortality over 3 years, CUSUM charts were most sensitive but only reliably achieved >50% successful detection for 10 excess deaths per year and generated multiple false alarms (>15%).
At best, mortality monitoring can act as a backstop to detect a particularly prolific serial killer when other means of detection have failed. Policy should focus on changes likely to improve detection of individual murders, such as reform of death certification and the coroner system.
family practice; homicide; outcome and process assessment (health care); quality assurance, health care; regulation
Health professionals, policy-makers and researchers need to be able to explore potential associations between prevalence rates and quality of care with a range of possible determinants including socio-economic deprivation and morbidity levels to determine the impact of commissioning and service delivery. In the UK, data in England are only available nationally at practice postcode level. In Scotland, such data are available based on an aggregate of the practices population's postcodes. The use of data assigned to the practice postcode may underestimate the association between ill health and income deprivation. Here, we report on the impact of using data assigned to the practice population by comparing analyses using English and Scottish data.
Income deprivation based on data assigned to the practice postcode under-estimated deprivation compared to using income deprivation data assigned to the practice population for the five least deprived deciles, and over-estimated deprivation for the five most deprived deciles. The biggest differences were found for the most deprived decile. A similar trend was found for limiting long-term illness (LLTI). Differences between the QOF prevalence rates of the least and most deprived deciles using practice postcode data were similar (0.2% points or less) in England and Scotland for 8 out of 10 clinical domains. Using practice population assigned deprivation, differences in the prevalence rate between the least and most deprived deciles increase for all clinical domains. A similar trend was again found for LLTI. Using practice population assigned deprivation, differences for population achievement increase for all CHD quality indicators with the exception of beta-blockers (CHD10). With practice postcode assigned deprivation, significant differences between the least and most deprived deciles were found for 2 out 8 indicators, compared to 5 using practice population assigned deprivation. For LLTI differences between the lowest and most deprived deciles increased for all indicators when ill health assigned to the practice population was used.
We have found, through comparing deprivation and ill health data assigned to either the practice postcode or the practice population postcode in Scotland, that analyses based on practice postcode assigned data under-estimated the relationship between deprivation and ill health for both prevalence and quality care. Given the importance of understanding the effect of deprivation and ill health on a range of determinants related to health care, policy makers should ensure that practice population data are available and used at national level in England and elsewhere where possible.
The Quality and Outcomes Framework (QOF) of the 2004 UK General Medical Services (GMS) contract links up to 20% of practice income to performance measured against 146 quality indicators.
To examine the distribution of workload and payment in the clinical domains of the QOF, and to compare payment based on true prevalence to the implemented system applying an adjusted prevalence factor. We aimed also to assess the performance of the implemented payment system against its three stated objectives: to reduce variation in payment compared to a system based on true prevalence, to fairly link reward to workload, and finally, to help tackle health inequalities.
Design of study
Retrospective analysis of publicly available QOF data.
Nine hundred and three GMS general practices in Scotland.
Comparison of payment under the implemented Adjusted Disease Prevalence Factor, and under an alternative True Disease Prevalence Factor.
Variation in total clinical QOF payment per 1000 patients registered is significantly reduced compared to a payment system based on true prevalence. Payment is poorly related to workload in terms of the number of patients on the disease register, with up to 44 fold variation in payment per patient on the disease register for practices delivering the same quality of care. Practices serving deprived populations are systematically penalised under the implemented payment system, compared to one based on true prevalence.
The implemented adjustment for prevalence succeeds in its aim of reducing variation in practice income, but at the cost of making the relationship between workload and reward highly inequitable and perpetuating the inverse care law.
family practice; quality healthcare; physician incentive plans; workload
Health policy in the UK has rapidly diverged since devolution in 1999. However, there is relatively little comparative data available to examine the impact of this natural experiment in the four UK countries. The Quality and Outcomes Framework of the 2004 General Medical Services Contract provides a new and potentially rich source of comparable clinical quality data through which we compare quality of primary medical care for coronary heart disease (CHD), stroke, hypertension and diabetes across the four UK countries.
A cross-sectional analysis was undertaken involving 10,064 general practices in England, Scotland, Wales and Northern Ireland. The main outcome measures were prevalence rates for CHD, stroke, hypertension and diabetes. Achievement on 14 simple process, 3 complex process, 9 intermediate outcome and 5 treatment indicators for the four clinical areas.
Prevalence varies by up to 28% between the four UK countries, which is not reflected in resource distribution between countries, and penalises practices in the high prevalence countries (Wales and Scotland). Differences in simple process measures across countries are small. Larger differences are found for complex process, intermediate outcome and treatment measures, most notably for Wales, which has consistently lower quality of care. Scotland has generally higher quality than England and Northern Ireland is most consistently the highest quality.
Previously identified weaknesses in Wales related to waiting times appear to reflect a more general quality problem within NHS Wales. Identifying explanations for the observed differences is limited by the lack of comparable data on practice resources and organisation. Maximising the value of cross-jurisdictional comparisons of the ongoing natural experiment of health policy divergence within the UK requires more detailed examination of resource and organisational differences.
Personal continuity is a core value for family practice, but policy and performance targets emphasise other aspects of care, particularly waiting times for consultation. This study examined patient and general practitioner (GP) perceptions of the value of personal continuity and rapid access, and the relationship between them.
Qualitative analysis of semi-structured interviews with a purposive sample of 16 GPs and 32 patients in the Lothian region of Scotland, to identify whether, how, why and in which circumstances personal continuity and rapid access were valued.
From the patients' perspective, what mattered was 'access to appropriate care' depending on the problem to be dealt with. For a few patients, rapid access was the only priority. For most, rapid access was balanced against greater involvement in the consultation when seeing 'their' trusted doctor, which was particularly valued for chronic, complex and emotional problems. GPs focused on the value of personal continuity in the consultation for improving the diagnosis and management of the same kinds of problem. GPs did not perceive enabling access to be a core part of their work. There was little evidence that GPs routinely discussed with patients when or how personal continuity and access should be balanced.
'Access to appropriate care' from the patients' perspective is not fully addressed by GPs' focus on personal continuity, nor by performance targets focused only on speed of access. GPs need to make enabling access as much a part of their core values as personal continuity, and access targets need to be based on less simplistic measures that account for the appropriateness of care as well as speed of access.
BACKGROUND: The new United Kingdom general practice contract proposes that up to a third of general practitioners' income will come from achieving quality targets. AIM: To examine selected quality markers in terms of their robustness to case-mix variation and chance effects, and in the attribution of quality to practices. STUDY DESIGN AND METHODS: Data were extracted from a population-based diabetes clinical information system in Tayside, Scotland, for patients with type 2 diabetes registered in 67 practices with complete ascertainment. RESULTS: Most practices would have received relatively high levels of payment for the process measures examined. Outcome measures appeared more challenging. Case-mix adjustment for age, sex, and postcode-assigned deprivation altered measured performance by up to 7%, but payment by up to 14%. Despite no strong evidence of any real difference in quality, chance effects meant that there was greater apparent variability for smaller practices from year to year. Hospital attendance was common, but highly variable between practices. CONCLUSION: Case-mix adjustment to allow fairer comparison is routine in national performance indicators, and ignoring it risks making the new contract quality framework inequitable. Because of chance effects, smaller practices may have greater year-to-year variability in income. Reflecting National Health Service structure, the new contract provides no incentives for integrated care and offers a perverse incentive to refer more patients to hospital. There are trade-offs between the validity of measures, and the cost and bureaucracy of collecting data. The planned evaluation of the new contrast should examine the effectiveness and equity of the quality framework, and rapidly act on deficiencies found.