Generalisability of research is not a new issue, and discussion of the subject can usefully draw on resolutions devised by many different disciplines.1 2
As regards medical treatments, many practitioners try to assess whether improved outcomes found in a randomised controlled trial (RCT) of a treatment will apply to their patients. The practitioners consider how their patients differ from those selected to participate in the RCT (eg, comorbidities) or the likelihood of their patients following the treatment.
The extent to which the effect seen in a RCT is associated with the treatment is related to the concept of ‘internal validity’ (defined in box 1
Whether the treatment is likely to have similar effects in other settings is related to the concept of ‘external validity’ or ‘generalisation’.1 2
The purpose of this paper is not to give an overview of validity or to repeat design issues that are well-discussed elsewhere.2
Conventionally, generalisation is discussed in relation to sampling, guided by a theory. In the case of QI, however, theory needs development which is usefully informed by a better understanding of practice. And any sampling frame for QI is inadequate without better information about the characteristics of patients, providers, organisations and notably—treatments—to be sampled.4
Box 1. Definitions used in this paper
- A quality improvement change (QIC) is an individual improvement change addressing a specific problem or opportunity, carried out in a specific context: a specific healthcare organisation and time period, with a specific patient population and improvement activities. It may or may not be described in a research study.
- A class of QICs is the set of underlying improvement principles or concepts from which individual QICs are derived and to which we would like to generalise from the individual QICs that are observed.
- Theory of a QIC is the use of a class of QICs, its outcomes, and related mediators and moderators that permit better understanding of why and how a QIC would produce an outcome in a given situation.
- Internal validity of a QIC study is the validity of a claim that a QIC caused a change in outcome.3
- External validity about the outcomes of a QIC study has a two-part definition.2
- Generalisation about a class of QICs: Validity of the claim that the causal relationship holds across variation in organisations, patient populations, time periods, individual QIC activities and measurement of outcomes
- Generalisation to a new QIC: Decision-makers' degree of certainty about whether a QIC, or a class of QICs, will produce similar outcomes for their organisations, patients and situation.
- Generalisability describes the range of situations or units of study to which findings or methods from elsewhere can be applied.
- Decision-maker: clinician, manager, policy maker or others who decide about whether or how to implement a quality improvement reported to be effective in one or more places.
Where ‘the subject’ of the study is not a patient but an individual provider or organisational unit, and the ‘treatment’ is an intervention to change behaviour or organisation, uncertainty about generalising from an external study is even greater. If the study design is a randomised or matched controlled trial of a quality intervention to providers, then generalisability can be increased by describing the intervention and the context in more detail. The science of quality improvement is in the early stages of recommending how these details should be reported.5
Compared with controlled trials, findings reported from the field have less strong internal and external validity. Because many such reports have no control group, the intervention effects at the study sites may be uncertain.2 6
In such reports, assessment of internal validity requires a description of the context factors ‘surrounding’ the intervention in order to understand the possible influences of these factors on the outcomes. Both the intervention and the context are often not well enough described for others to make a judgement about whether the QICs might be implementable with the same results in their situation. Similar points about descriptions of interventions for RCTs have also been made recently.7
Thus adequate description of context and intervention are necessary.5 7
Yet there is no consensus about which context factors should be reported or assessed, both to help practitioners and to build theory about which factors are important for which interventions. We also do not have a standard vocabulary for describing the features of QIC interventions. Many quality improvements are not detailed prescriptions that can be described in a fixed protocol. Rather they are concepts and ideas or objectives to be achieved, which provide latitude to practitioners to adapt them during their implementation.
For example, the central line blood stream infection (CLBSI) bundle contains a hand hygiene component. One change concept is that having hand cleaning material easily available will lead to its greater use, increased hand hygiene and lower rates of CLBSI.8
But few studies describe well the details of how this change concept and other aspects of the bundle are interpreted locally or the details of implementation.9
Moreover, many changes are made during the implementation using a Plan–Do–Study–Act cycle or equivalent.10