|Home | About | Journals | Submit | Contact Us | Français|
This paper outlines the experience of the Child Outcomes Research Consortium—formerly known as the CAMHS Outcomes Research Consortium; the named changed in 2014 in recognition of the widening scope of the work of the collaboration; a learning collaboration of service providers, funders, service user groups and researchers across the UK and beyond, jointly committed to collecting and using routinely collected outcome data to improve and enhance service provision and improve understanding of how best to help young people with mental health issues and their families.
The Child Outcomes Research Consortium (CORC—formerly known as the CAMHS Outcomes Research Consortium; the named changed in 2014 in recognition of the widening scope of the work of the collaboration) was formed in 2002 by a group of child mental health clinicians, managers and funders all working in the National Health Service (NHS) in England. They worked across five different service providers across the country, but shared a mutual curiosity as to the effectiveness of their and their colleagues’ practice and how best to improve their own practice.
They determined that one way to find out about the impact of their work was to ask those they worked with (not routine practice then or now) and thus set about exploring appropriate tools to try to access these views in a systematic way (Wolpert et al. 2012). Interest grew amongst other services and interested academics joined the founding group. The collaboration opened to wider membership in 2004 and was formalised as a not-for-profit learning consortium in 2008 (see www.corc.uk.net).
Over the last decade the collaboration has grown to include over half of all services across the UK (70 membership groupings) with members also in Scandinavia and Australia, and seeks to act as a peer-learning group (Fullan 2009). It also increasingly includes a range of voluntary sector and counselling services.
The collaboration has pioneered the routine use of patient-reported outcome and experience measures (PROMs and PREMs) across child mental health services in England (supported by research reviewed elsewhere in this special issue) and has informed and contributed to policy development (Department of Health 2004, 2012). Its work and learning has underpinned the current national service transformation initiative: children and young people’s improving access to psychological therapies (CYP IAPT; http://www.cypiapt.org/) which seeks to implement patient-reported routine outcome measurement across children’s mental health services in England.
The Child Outcomes Research Consortium has recently introduced a self-review and accreditation system to allow members to internally assess quality and gain external assurance that they are implementing best practice in outcome evaluation.
From the outset, CORC has sought to bridge the worlds of clinical decision-making, evaluation and research. Table 1 offers a conceptualisation of the way that the collaboration conceived this continuum and outlines the role of CORC at each level.
This is a challenging agenda and there are clear tensions, as well as interdependencies, between the desire to use outcomes to directly inform clinical practice and using them to inform research and service evaluation (Wolpert 2014). Below we elaborate the key challenges faced in trying to use patient-reported routine outcome and experience measurement to contribute to research, evaluation and practice, and how CORC has tried to address them. In this paper we are reflecting on the practical issues and sustainability, rather than implementation (see CORE paper for a methodological approach) of CORC methodologies.
The Child Outcomes Research Consortium emphasises that any feedback measure should be used in the context of collaborative working and with an aspiration to shared decision-making to directly inform clinical work (Law 2012; Law and Wolpert 2014). Practitioners are encouraged to consider the outcomes of clients they see using normative data and to discuss this in supervision (Law and Wolpert 2014). This approach is supported by service users themselves (Roberson 2011).
It should be noted that the collaboration has not yet finalised ways to support members to track progress for individual clients against trajectories of change. This is something that the collaboration is seeking to pursue: learning from the approach pioneered by Lambert, Bickman, Duncan, Miller and others,work is underway to develop trajectories of change using a range of measure for a UK population.
As reported elsewhere in this special issue, there are well-recognised challenges to encouraging clinicians to use such measures as part of their routine practice including: a) concerns about inappropriate use and impact on therapeutic relationship; b) lack of confidence in choosing and using measures; c) concerns about insufficient support for increased administrative demands to inadequate data systems to support the collection of considerable amounts of additional data fields (Badham 2011; Curtis-Tyler 2011; de Jong et al. 2012; Johnston and Gowers 2005; Moran et al. 2012; O’Herlihy 2013; Wolpert 2013.)
The collaboration addresses these challenges as follows:
Collaborating services send their data to a central team of researchers and data analysts who produce reports that allow comparison with relevant comparators. A dashboard is being trialled to allow for a rapid review of key data. These reports are tailored to members’ needs in relation to four main domains of service metrics: 1) Who is my service seeing; 2) How well are we addressing their needs; 3) What do service users think of their support; 4) How good is our evidence on what we are doing and what could we be doing better?
Members are also offered bespoke reporting in more depth, which includes statistical comparisons of service outcomes with those of other services using funnel plots and other relevant visual representation.
Members are encouraged to use these reports to consider their outcomes in comparison with others, to inform discussions with commissioners and others in line with practice-based evidence (Wolpert et al. 2014). CORC recommends a systematic and collaborative approach to consideration of such data by service providers, funders and users adopting the ‘MINDFUL’ framework, whereby appropriate statistical comparisons are made in relation to the most meaningful clinical unit (in the UK this is the multidisciplinary team) employing multiple perspectives and harnessing the strength of a learning collaboration (Wolpert et al. 2014).
This MINDFUL framework (see Box Box1)1) involves: a consideration of multiple perspectives, interpreting differences in the light of the current base of evidence, a focus on negative differences when triangulated with other data, directed discussions based on ‘what if this were a true difference’ which employ the 75–25 % rule (discussed further below), the use of funnel plots as a starting point to consider outliers, the appreciation of uncertainty as a key contextual reality and the use of learning collaborations to support appropriate implementation and action strategies.
Key challenges to using data for service evaluation include a) data completeness b) data quality and c) inappropriate use of data.
The Child Outcomes Research Consortium has sought to respond these challenges as follows:
Over the last decade, CORC members have built up a rich (if flawed) dataset consisting of over a quarter of a million records (263,928 as of 24th February 2014) although only 24 % have meaningful outcome data. CORC has started to mine this data on behalf of members to answer key questions that may help inform our understanding of how best to help children and young people with mental health issues, always bearing in mind the need for caution given the missing data (Clark et al. 2008). In doing so, we are able to close the loop, turning practice-based evidence to evidenced-based practice.
The Child Outcomes Research Consortium now has a clear protocol whereby members (and non-members) can apply to use the data or request for analyses to be carried out by the central team. Key analyses already published include consideration of the sort of goals young people set for themselves when they come to therapy (Bradley et al. 2013) analysis of measure of service satisfaction (Brown et al. 2012) and analysis of service-level outcome (Wolpert et al. 2012). Further analyses currently in progress include an exploration of impact of evidence-based practice and a comparison of outcomes achieved between those seen in clinical services and those not seen in the community.
Bridging the worlds of research, service evaluation and clinical decision-making remains a complex and challenging agenda. CORC certainly does not have all the answers and daily obstacles remain. We hope that by sharing our experience we can help advance further work in this challenging but worthwhile area.
The authors would like to thank all members of the Child Outcomes Research Consortium; the CORC committee at the time of writing (includes M.W.): Alan Ovenden, Alison Towndrow, Ann York, Ashley Wyatt, Duncan Law, Evette Girgis, Julie Elliott, Mick Atkinson and Tamsin Ford; and the CORC Central Team at the time of writing (includes M.W., J.B. and I.F.): Robbie Newman, Rachel Argent, Slavi Savic and Thomas Booker. The authors would also like to thank Julian Edbrooke-Childs for his insightful comments.