Background

Misplaced or poorly calibrated confidence in healthcare professionals’ judgments compromises the quality of health care. Using higher fidelity clinical simulations to elicit clinicians’ confidence 'calibration' (i.e. overconfidence or underconfidence) in more realistic settings is a promising but underutilized tactic. In this study we examine nurses’ calibration of confidence with judgment accuracy for critical event risk assessment judgments in a high fidelity simulated clinical environment. The study also explores the effects of clinical experience, task difficulty and time pressure on the relationship between confidence and accuracy.

Methods

63 student and 34 experienced nurses made dichotomous risk assessments on 25 scenarios simulated in a high fidelity clinical environment. Each nurse also assigned a score (0–100) reflecting the level of confidence in their judgments. Scenarios were derived from real patient cases and classified as easy or difficult judgment tasks. Nurses made half of their judgments under time pressure. Confidence calibration statistics were calculated and calibration curves generated.

Results

Nurse students were underconfident (mean over/underconfidence score −1.05) and experienced nurses overconfident (mean over/underconfidence score 6.56), P = 0.01. No significant differences in calibration and resolution were found between the two groups (P = 0.80 and P = 0.51, respectively). There was a significant interaction between time pressure and task difficulty on confidence (P = 0.008); time pressure increased confidence in easy cases but reduced confidence in difficult cases. Time pressure had no effect on confidence or accuracy. Judgment task difficulty impacted significantly on nurses’ judgmental accuracy and confidence. A 'hard-easy' effect was observed: nurses were overconfident in difficult judgments and underconfident in easy judgments.

Conclusion

Nurses were poorly calibrated when making risk assessment judgments in a high fidelity simulated setting. Nurses with more experience tended toward overconfidence. Whilst time pressure had little effect on calibration, nurses’ over/underconfidence varied significantly with the degree of task difficulty. More research is required to identify strategies to minimize such cognitive biases.

Su Golder and colleagues carry out an overview of meta-analyses to assess whether estimates of the risk of harm outcomes differ between randomized trials and observational studies. They find that, on average, there is no difference in the estimates of risk between overviews of observational studies and overviews of randomized trials.

Background

There is considerable debate as to the relative merits of using randomised controlled trial (RCT) data as opposed to observational data in systematic reviews of adverse effects. This meta-analysis of meta-analyses aimed to assess the level of agreement or disagreement in the estimates of harm derived from meta-analysis of RCTs as compared to meta-analysis of observational studies.

Methods and Findings

Searches were carried out in ten databases in addition to reference checking, contacting experts, citation searches, and hand-searching key journals, conference proceedings, and Web sites. Studies were included where a pooled relative measure of an adverse effect (odds ratio or risk ratio) from RCTs could be directly compared, using the ratio of odds ratios, with the pooled estimate for the same adverse effect arising from observational studies. Nineteen studies, yielding 58 meta-analyses, were identified for inclusion. The pooled ratio of odds ratios of RCTs compared to observational studies was estimated to be 1.03 (95% confidence interval 0.93–1.15). There was less discrepancy with larger studies. The symmetric funnel plot suggests that there is no consistent difference between risk estimates from meta-analysis of RCT data and those from meta-analysis of observational studies. In almost all instances, the estimates of harm from meta-analyses of the different study designs had 95% confidence intervals that overlapped (54/58, 93%). In terms of statistical significance, in nearly two-thirds (37/58, 64%), the results agreed (both studies showing a significant increase or significant decrease or both showing no significant difference). In only one meta-analysis about one adverse effect was there opposing statistical significance.

Conclusions

Empirical evidence from this overview indicates that there is no difference on average in the risk estimate of adverse effects of an intervention derived from meta-analyses of RCTs and meta-analyses of observational studies. This suggests that systematic reviews of adverse effects should not be restricted to specific study types.

Please see later in the article for the Editors' Summary

Editors' Summary

Background

Whenever patients consult a doctor, they expect the treatments they receive to be effective and to have minimal adverse effects (side effects). To ensure that this is the case, all treatments now undergo exhaustive clinical research—carefully designed investigations that test new treatments and therapies in people. Clinical investigations fall into two main groups—randomized controlled trials (RCTs) and observational, or non-randomized, studies. In RCTs, groups of patients with a specific disease or condition are randomly assigned to receive the new treatment or a control treatment, and the outcomes (for example, improvements in health and the occurrence of specific adverse effects) of the two groups of patients are compared. Because the patients are randomly chosen, differences in outcomes between the two groups are likely to be treatment-related. In observational studies, patients who are receiving a specific treatment are enrolled and outcomes in this group are compared to those in a similar group of untreated patients. Because the patient groups are not randomly chosen, differences in outcomes between cases and controls may be the result of a hidden shared characteristic among the cases rather than treatment-related (so-called confounding variables).

Why Was This Study Done?

Although data from individual trials and studies are valuable, much more information about a potential new treatment can be obtained by systematically reviewing all the evidence and then doing a meta-analysis (so-called evidence-based medicine). A systematic review uses predefined criteria to identify all the research on a treatment; meta-analysis is a statistical method for combining the results of several studies to yield “pooled estimates” of the treatment effect (the efficacy of a treatment) and the risk of harm. Treatment effect estimates can differ between RCTs and observational studies, but what about adverse effect estimates? Can different study designs provide a consistent picture of the risk of harm, or are the results from different study designs so disparate that it would be meaningless to combine them in a single review? In this methodological overview, which comprises a systematic review and meta-analyses, the researchers assess the level of agreement in the estimates of harm derived from meta-analysis of RCTs with estimates derived from meta-analysis of observational studies.

What Did the Researchers Do and Find?

The researchers searched literature databases and reference lists, consulted experts, and hand-searched various other sources for studies in which the pooled estimate of an adverse effect from RCTs could be directly compared to the pooled estimate for the same adverse effect from observational studies. They identified 19 studies that together covered 58 separate adverse effects. In almost all instances, the estimates of harm obtained from meta-analyses of RCTs and observational studies had overlapping 95% confidence intervals. That is, in statistical terms, the estimates of harm were similar. Moreover, in nearly two-thirds of cases, there was agreement between RCTs and observational studies about whether a treatment caused a significant increase in adverse effects, a significant decrease, or no significant change (a significant change is one unlikely to have occurred by chance). Finally, the researchers used meta-analysis to calculate that the pooled ratio of the odds ratios (a statistical measurement of risk) of RCTs compared to observational studies was 1.03. This figure suggests that there was no consistent difference between risk estimates obtained from meta-analysis of RCT data and those obtained from meta-analysis of observational study data.

What Do These Findings Mean?

The findings of this methodological overview suggest that there is no difference on average in the risk estimate of an intervention's adverse effects obtained from meta-analyses of RCTs and from meta-analyses of observational studies. Although limited by some aspects of its design, this overview has several important implications for the conduct of systematic reviews of adverse effects. In particular, it suggests that, rather than limiting systematic reviews to certain study designs, it might be better to evaluate a broad range of studies. In this way, it might be possible to build a more complete, more generalizable picture of potential harms associated with an intervention, without any loss of validity, than by evaluating a single type of study. Such a picture, in combination with estimates of treatment effects also obtained from systematic reviews and meta-analyses, would help clinicians decide the best treatment for their patients.

Additional Information

Please access these Web sites via the online version of this summary at http://dx.doi.org/10.1371/journal.pmed.1001026.

The US National Institutes of Health provide information on clinical research; the UK National Health Service Choices Web site also has a page on clinical trials and medical research

The Cochrane Collaboration produces and disseminates systematic reviews of health-care interventions

Medline Plus provides links to further information about clinical trials (in English and Spanish)

Background

There is insufficient evidence on the effectiveness of acupuncture for irritable bowel syndrome (IBS) for conclusions to be drawn. Given the current interest in acupuncture by patients, it is in the public interest to establish more rigorous evidence. Building on the positive findings from a pilot study, in this paper we present the protocol for a fully-powered trial designed to establish whether or not acupuncture is effective and cost-effective.

Methods/Design

In this pragmatic randomised controlled trial we will randomise patients recruited directly from GP databases to either 10 sessions of acupuncture plus usual GP care or to usual GP care alone. The primary clinical outcome will be the IBS Symptom Severity Score (SSS) (maximum score 500) at three months, and at 12 month assessing whether there is an overall benefit. We estimate the sample size required to detect a minimum clinical difference at 90% power and 5% significance to be 188 patients. To allow for loss to follow up we will recruit 220 patients drawn from an estimated primary care population of 140 000. Analysis will be by intention-to-treat, and multiple imputation is to be used for missing data.

In a nested qualitative study using in-depth interviews, we will explore how patients, acupuncturists, and GPs explain and subsequently understand acupuncture to work. We will use purposive sampling to identify patients and flexible topic guides for the interviews. The data analysis will lead to a thematic description of how patients and practitioners explain how acupuncture works, and whether or not the explanations influence treatment outcome and/or referrals.

We will undertake a cost-effectiveness analysis at 12 months by comparing resource use in the two groups with any treatment benefit. We will use the EQ-5D to measure health-related quality of life and convert into quality adjusted life years (QALYs). We will generate cost effectiveness acceptability curves (CEACs) exploring the probability that acupuncture will produce an acceptable cost per QALY at different cost-effectiveness thresholds.

Discussion

The trial has received NHS ethics approval and recruited 233 patients between November 2008 and June 2009. Results are expected in 2011.

Trial Registration

Current Controlled Trials ISRCTN08827905

Background

Population monitoring has been introduced in UK primary schools in an effort to track the growing obesity epidemic. It has been argued that parents should be informed of their child's results, but is there evidence that moving from monitoring to screening would be effective? We describe what is known about the effectiveness of monitoring and screening for overweight and obesity in primary school children and highlight areas where evidence is lacking and research should be prioritised.

Design

Systematic review with discussion of evidence gaps and future research.

Data sources

Published and unpublished studies (any language) from electronic databases (inception to July 2005), clinical experts, Primary Care Trusts and Strategic Health Authorities, and reference lists of retrieved studies.

Review methods

We included any study that evaluated measures of overweight and obesity as part of a population‐level assessment and excluded studies whose primary outcome measure was prevalence.

Results

There were no trials assessing the effectiveness of monitoring or screening for overweight and obesity. Studies focussed on the diagnostic accuracy of measurements. Information on the attitudes of children, parents and health professionals to monitoring was extremely sparse.

Conclusions

Our review found a lack of data on the potential impact of population monitoring or screening for obesity and more research is indicated. Identification of effective weight reduction strategies for children and clarification of the role of preventative measures are priorities. It is difficult to see how screening to identify individual children can be justified without effective interventions.

Background

A large number of randomised controlled trials in health settings have consistently reported positive effects of brief intervention in terms of reductions in alcohol use. However, although alcohol misuse is common amongst offenders, there is limited evidence of alcohol brief interventions in the criminal justice field. This factorial pragmatic cluster randomised controlled trial with Offender Managers (OMs) as the unit of randomisation will evaluate the effectiveness and cost-effectiveness of different models of screening to identify hazardous and harmful drinkers in probation and different intensities of brief intervention to reduce excessive drinking in probation clients.

Methods and design

Ninety-six OMs from 9 probation areas across 3 English regions (the North East Region (n = 4) and London and the South East Regions (n = 5)) will be recruited. OMs will be randomly allocated to one of three intervention conditions: a client information leaflet control condition (n = 32 OMs); 5-minute simple structured advice (n = 32 OMs) and 20-minute brief lifestyle counselling delivered by an Alcohol Health Worker (n = 32 OMs). Randomisation will be stratified by probation area. To test the relative effectiveness of different screening methods all OMs will be randomised to either the Modified Single Item Screening Questionnaire (M-SASQ) or the Fast Alcohol Screening Test (FAST). There will be a minimum of 480 clients recruited into the trial. There will be an intention to treat analysis of study outcomes at 6 and 12 months post intervention. Analysis will include client measures (screening result, weekly alcohol consumption, alcohol-related problems, re-offending, public service use and quality of life) and implementation measures from OMs (the extent of screening and brief intervention beyond the minimum recruitment threshold will provide data on acceptability and feasibility of different models of brief intervention). We will also examine the practitioner and organisational factors associated with successful implementation.

Discussion

The trial will evaluate the impact of screening and brief alcohol intervention in routine probation work and therefore its findings will be highly relevant to probation teams and thus the criminal justice system in the UK.

Ethical approval was given by Northern & Yorkshire REC

Trial Registration number

ISRCTN 19160244

Background

There have been many randomized controlled trials of screening and brief alcohol intervention in primary care. Most trials have reported positive effects of brief intervention, in terms of reduced alcohol consumption in excessive drinkers. Despite this considerable evidence-base, key questions remain unanswered including: the applicability of the evidence to routine practice; the most efficient strategy for screening patients; and the required intensity of brief intervention in primary care. This pragmatic factorial trial, with cluster randomization of practices, will evaluate the effectiveness and cost-effectiveness of different models of screening to identify hazardous and harmful drinkers in primary care and different intensities of brief intervention to reduce excessive drinking in primary care patients.

Methods and design

GPs and nurses from 24 practices across the North East (n = 12), London and South East (n = 12) of England will be recruited. Practices will be randomly allocated to one of three intervention conditions: a leaflet-only control group (n = 8); brief structured advice (n = 8); and brief lifestyle counselling (n = 8). To test the relative effectiveness of different screening methods all practices will also be randomised to either a universal or targeted screening approach and to use either a modified single item (M-SASQ) or FAST screening tool. Screening randomisation will incorporate stratification by geographical area and intervention condition. During the intervention stage of the trial, practices in each of the three arms will recruit at least 31 hazardous or harmful drinkers who will receive a short baseline assessment followed by brief intervention. Thus there will be a minimum of 744 patients recruited into the trial.

Discussion

The trial will evaluate the impact of screening and brief alcohol intervention in routine practice; thus its findings will be highly relevant to clinicians working in primary care in the UK. There will be an intention to treat analysis of study outcomes at 6 and 12 months after intervention. Analyses will include patient measures (screening result, weekly alcohol consumption, alcohol-related problems, public service use and quality of life) and implementation measures from practice staff (the acceptability and feasibility of different models of brief intervention.) We will also examine organisational factors associated with successful implementation.

Trial registration

Current Controlled Trials ISRCTN06145674.

Background

There is a wealth of evidence regarding the detrimental impact of excessive alcohol consumption on the physical, psychological and social health of the population. There also exists a substantial evidence base for the efficacy of brief interventions aimed at reducing alcohol consumption across a range of healthcare settings. Primary research conducted in emergency departments has reinforced the current evidence regarding the potential effectiveness and cost-effectiveness. Within this body of evidence there is marked variation in the intensity of brief intervention delivered, from very minimal interventions to more intensive behavioural or lifestyle counselling approaches. Further the majority of primary research has been conducted in single centre and there is little evidence of the wider issues of generalisability and implementation of brief interventions across emergency departments.

Methods/design

The study design is a prospective pragmatic factorial cluster randomised controlled trial. Individual Emergency Departments (ED) (n = 9) are randomised with equal probability to a combination of screening tool (M-SASQ vs FAST vs SIPS-PAT) and an intervention (Minimal intervention vs Brief advice vs Brief lifestyle counselling). The primary hypothesis is that brief lifestyle counselling delivered by an Alcohol Health Worker (AHW) is more effective than Brief Advice or a minimal intervention delivered by ED staff. Secondary hypotheses address whether short screening instruments are more acceptable and as efficient as longer screening instruments and the cost-effectiveness of screening and brief interventions in ED. Individual participants will be followed up at 6 and 12 months after consent. The primary outcome measure is performance using a gold-standard screening test (AUDIT). Secondary outcomes include; quantity and frequency of alcohol consumed, alcohol-related problems, motivation to change, health related quality of life and service utilisation.

Discussion

This paper presents a protocol for a large multi-centre pragmatic factorial cluster randomised trial to evaluate the effectiveness and cost-effectiveness of screening and brief interventions for hazardous alcohol users attending emergency departments.

Trial Registration

ISRCTN 93681536

Background

There is a wealth of evidence regarding the detrimental impact of excessive alcohol consumption. In older populations excessive alcohol consumption is associated with increased risk of coronary heart disease, hypertension, stroke and a range of cancers. Alcohol consumption is also associated with an increased risk of falls, early onset of dementia and other cognitive deficits. Physiological changes that occur as part of the ageing process mean that older people experience alcohol related problems at lower consumption levels. There is a strong evidence base for the effectiveness of brief psychosocial interventions in reducing alcohol consumption in populations identified opportunistically in primary care settings. Stepped care interventions involve the delivery of more intensive interventions only to those in the population who fail to respond to less intensive interventions and provide a potentially resource efficient means of meeting the needs of this population.

Methods/design

The study design is a pragmatic prospective multi-centre two arm randomised controlled trial. The primary hypothesis is that stepped care interventions for older hazardous alcohol users reduce alcohol consumption compared with a minimal intervention at 12 months post randomisation. Potential participants are identified using the AUDIT questionnaire. Eligible and consenting participants are randomised with equal probability to either a minimal intervention or a three step treatment approach. The step treatment approach incorporates as step 1 behavioural change counselling, step 2 three sessions of motivational enhancement therapy and step 3 referral to specialist services. The primary outcome is measured using average standard drinks per day and secondary outcome measures include the Drinking Problems Index, health related quality of life and health utility. The study incorporates a comprehensive economic analysis to assess the relative cost-effectiveness of the interventions.

Discussion

The paper presents a protocol for the first pragmatic randomised controlled trial evaluating the effectiveness and cost-effectiveness of stepped care interventions for older hazardous alcohol users in primary care.

Trial registration

ISRCTN52557360

Background

Acupuncture is increasingly being used for many conditions including chronic neck pain. However the evidence remains inconclusive, indicating the need for further well-designed research. The aim of this study was to conduct a pilot randomised controlled parallel arm trial, to establish key features required for the design and implementation of a large-scale trial on acupuncture for chronic neck pain.

Methods

Patients whose GPs had diagnosed neck pain were recruited from one general practice, and randomised to receive usual GP care only, or acupuncture (up to 10 treatments over 3 months) as an adjunctive treatment to usual GP care. The primary outcome measure was the Northwick Park Neck Pain Questionnaire (NPQ) at 3 months. The primary analysis was to determine the sample size for the full scale study.

Results

Of the 227 patients with neck pain identified from the GP database, 28 (12.3%) consenting patients were eligible to participate in the pilot and 24 (10.5%) were recruited to the trial. Ten patients were randomised to acupuncture, receiving an average of eight treatments from one of four acupuncturists, and 14 were randomised to usual GP care alone. The sample size for the full scale trial was calculated from a clinically meaningful difference of 5% on the NPQ and, from this pilot, an adjusted standard deviation of 15.3%. Assuming 90% power at the 5% significance level, a sample size of 229 would be required in each arm in a large-scale trial when allowing for a loss to follow-up rate of 14%. In order to achieve this sample, one would need to identify patients from databases of GP practices with a total population of 230,000 patients, or approximately 15 GP practices roughly equal in size to the one involved in this study (i.e. 15,694 patients).

Conclusion

This pilot study has allowed a number of recommendations to be made to facilitate the design of a large-scale trial, which in turn will help to clarify the existing evidence base on acupuncture for neck pain.

Objective To examine the efficacy of different methods of detecting a high death rate and determining whether an increase in deaths after heart transplantation could be explained by chance.

Design Retrospective analysis of deaths after heart transplantation. Seven methods were used: mortality above national average, mortality excessively above national average, test of moving average mortality, test of number of consecutive deaths, sequential probability ratio test (SPRT), cusum graph with v-mask, and CRAM chart. The national average mortality was not available and a rate of 15% was used instead as the benchmark.

Setting Regional cardiothoracic unit.

Participants All 371 patients who received a heart transplant in the programme, 1986-2000.

Main outcome measures 30 day survival after transplantation.

Results All methods provided evidence that the 30 day mortality had been high at some stage. The probability that the finding was a false positive depended on which test was used. At the end of the series the average mortality, sequential probability ratio, and cusum tests indicated a level of deaths higher than the benchmark while the remaining four tests yielded negative results.

Conclusions If the decision to test for outlying mortality is made retrospectively, in the light of the data, it is not possible to determine the false positive rate. Prospective on-site mortality monitoring with the CRAM chart is recommended as this method can quantify the death rate and identify periods when an audit of cases is indicated, even when data from other institutions are not available. A hospital mortality monitoring group can routinely monitor all deaths in the hospital, by specialty, using hospital episode statistics (HES) data and appropriate statistical methods.

Objective To evaluate the effectiveness of different brief intervention strategies at reducing hazardous or harmful drinking in primary care. The hypothesis was that more intensive intervention would result in a greater reduction in hazardous or harmful drinking.

Design Pragmatic cluster randomised controlled trial.

Setting Primary care practices in the north east and south east of England and in London.

Participants 3562 patients aged 18 or more routinely presenting in primary care, of whom 2991 (84.0%) were eligible to enter the trial: 900 (30.1%) screened positive for hazardous or harmful drinking and 756 (84.0%) received a brief intervention. The sample was predominantly male (62%) and white (92%), and 34% were current smokers.

Interventions Practices were randomised to three interventions, each of which built on the previous one: a patient information leaflet control group, five minutes of structured brief advice, and 20 minutes of brief lifestyle counselling. Delivery of the patient leaflet and brief advice occurred directly after screening and brief lifestyle counselling in a subsequent consultation.

Main outcome measures The primary outcome was patients’ self reported hazardous or harmful drinking status as measured by the alcohol use disorders identification test (AUDIT) at six months. A negative AUDIT result (score <8) indicated non-hazardous or non-harmful drinking. Secondary outcomes were a negative AUDIT result at 12 months, experience of alcohol related problems (alcohol problems questionnaire), health utility (EQ-5D), service utilisation, and patients’ motivation to change drinking behaviour (readiness to change) as measured by a modified readiness ruler.

Results Patient follow-up rates were 83% at six months (n=644) and 79% at 12 months (n=617). At both time points an intention to treat analysis found no significant differences in AUDIT negative status between the three interventions. Compared with the patient information leaflet group, the odds ratio of having a negative AUDIT result for brief advice was 0.85 (95% confidence interval 0.52 to 1.39) and for brief lifestyle counselling was 0.78 (0.48 to 1.25). A per protocol analysis confirmed these findings.

Conclusions All patients received simple feedback on their screening outcome. Beyond this input, however, evidence that brief advice or brief lifestyle counselling provided important additional benefit in reducing hazardous or harmful drinking compared with the patient information leaflet was lacking.

Trial registration Current Controlled Trials ISRCTN06145674.