This systematic review brings together the available evidence concerning the educational impact of workplace based assessment and its ability to change doctors’ performance. Considering the emphasis now placed on workplace based assessment as a method of formative performance assessment, there are surprisingly few published articles exploring these areas, and the strength of the findings is questionable.
The strongest evidence for workplace based assessment improving performance comes from studies examining multisource feedback. Work reported in the psychology literature has shown that multisource feedback can lead to small improvements in performance over time,33
and a 10 year old study of medical education also showed that doctors exposed to specific feedback from peers, coworkers, and patients can use the data to inform changes in their practice.34
The studies we reviewed show conflicting evidence, however. Although some junior doctors18
and most surgeons19
displayed little willingness to change in response to multisource feedback, family physicians seemed more prepared to initiate performance changes.21
This variability may be due to individual differences; it is already known that performance improvement is more likely to occur when feedback indicates a need for change, when recipients have a positive view of feedback, and when they believe that change is feasible.33
The single randomised controlled trial in our review17
attempted to show improved performance in the intervention group allocated to multisource feedback, but the positive results seen might have been due to the coaching session that was also part of the intervention. The positive influence of facilitation in the effectiveness of multisource feedback has recently been established,11 35
especially in the context of negative feedback.36
It seems, therefore, that multisource feedback can lead to improved performance, but individual factors, the context of the feedback, and the presence (or absence) of facilitation can have a profound effect on the magnitude of the response.
We were unable to unearth any clear evidence to show that the mini-clinical evaluation exercise, direct observation of procedural skills, or case based discussion can lead to improvements in performance. The studies examining the mini-clinical evaluation exercise and multiple assessment methods showed largely positive results in terms of learner satisfaction but could not show changes in attitudes, skills, knowledge, or behaviour. The study of the impact of direct observation of procedural skills30
revealed that some house officers felt it could improve their clinical skills, but this evidence has not been captured objectively, and participant numbers were small. A previous systematic review investigating tools for direct observation and assessment of clinical skills found similarly few studies describing educational outcome.37
Most of the articles included in this review were non-comparative descriptive or observational studies. Strength of findings may be limited by the uncontrolled nature of the studies, but given the methodological difficulties of evaluating topics such as educational impact and doctor performance,38
descriptive and observational studies can still provide useful information. Indeed, some of the strongest evidence for improved performance after workplace based assessment comes from detailed focus group data.22
The single randomised controlled trial we identified attempted to establish causality (“multisource feedback causes performance improvement”),17
but, as discussed above, the results are undoubtedly affected by confounding factors.
Methodological rigour is clearly apparent in some articles, especially those aiming to evaluate multiple facets of workplace based assessment,20 32
but, because the focus here tends to be on reliability and feasibility, they may be less suitable for gathering data about educational impact or performance change.
Quality is also affected by the voluntary nature of participation in most of the studies. Potentially biased30
or highly motivated21
study populations can lead to profoundly different results. The reliance on self reporting and the small study populations in most of the studies also limit the quality and strength of their findings.
Our review methodology also has its limitations. Although our database search was extensive, we did not review the grey literature and so may have missed some relevant studies. The Ovid database search was also limited to English language publications, so there may have been publication bias. The search may also have been limited by the terms used (for example, multisource feedback is also known as 360° feedback, mini-peer assessment tool, and team assessment of behaviours, but these terms were not included in the search).
This review has highlighted once again the need for further research in the area of formative performance assessment: the increasing use of workplace based assessment methods in postgraduate medical training and recertification should provide fertile ground for this work. Serious consideration needs to be given to the use of study designs that are able to show conclusive links between workplace based assessment and performance improvement. So often workplace based assessment has been implemented wholesale and evaluation has subsequently and understandably focused on feasibility and self reported outcomes. We need to move to an interventionist, experimental model to establish whether workplace based assessment makes a difference. Future studies will need to be ambitious, not only in size to show significant change but over extended periods to expose matched groups of doctors to different interventions. This will require collaboration within and across nations.
Further avenues for future work are also clearly signposted from here. A focus on assessment programmes to show how workplace based assessment instruments can be used together would be of great practical benefit. The role of facilitation in workplace based assessment, and the extent of its involvement in performance improvement, must also be fully investigated. Finally, we need to discover whether formative assessment strategies such as workplace based assessment can reach Kirkpatrick’s highest levels of change, leading to improvements in care delivery and patient outcome.
What is already known on this subject
- Workplace based assessment involves feedback on performance of doctors in their everyday activities
- It purports to provide useful information to both doctors and to their training and regulatory bodies, and is often reported to support educational impact and learning
What this study adds
- This review found little evidence in the literature to date for workplace based assessment as an educational initiative
- There is limited evidence that multisource feedback may lead to improved performance, but factors such as the context of the feedback and the presence of facilitation seem to have a profound effect on the response