|Home | About | Journals | Submit | Contact Us | Français|
Few clinical rules have been derived let alone validated in primary care. A rule was derived to predict complications of acute cough in preschool children presenting to primary care. The clinical rule used the presence/absence of fever and/or chest signs to distinguish children at low, medium, and high risk of complications.
To validate a clinical rule for predicting complications of acute cough in preschool children in primary care.
Prospective cohort study.
Thirteen general practices in Bristol and Tayside, UK.
Preschool children with cough up to 28 days and without asthma were recruited. The same sociodemographic, clinical history, examination, and complications data as for the derivation study were collected. First, univariable logistic regression was used to explore the associations with complications, and then predictors with stronger relationships (P<0.2) were modelled using multivariable logistic regression. These predictors were compared with derivation predictors with respect to their strength of association with complications. The derivation predictors were used in the validation dataset to allow comparison of the post-test probabilities of complications between derivation and validation studies.
The presence of fever and chest signs in the validation study tended to be protective for complications, with univariable odds ratios (ORs) of 0.37 and 0.81 respectively, compared with ORs of 4.86 and 2.72 in the derivation study. However, 95% confidence limits were wide and evidence for two other possible reasons for these results were found: spectrum bias and confounding by indication.
No evidence was found to validate the clinical rule for predicting complications of acute cough, possibly as a result of spectrum bias, confounding by indication, and/or chance. As paediatric infectious illness is costly and associated with high rates of antibiotic use, further research is needed to derive and validate prediction rules.
Infection-related cough in children is the most frequently managed problem by health services around the world.1–4 Twelve per cent of children experience complications, and up to 24% reconsult.5,6 Although between 20% and 40% of children with cough receive antibiotics, current evidence suggests that, on average, antibiotics do not reduce complications.6–8 While outpatient antibiotic prescribing in other European countries has remained relatively static since 1996, reductions in UK primary care prescribing in the same period may be associated with rises in invasive bacterial infections, suppurative complications and, in adults, mortality due to pneumonia.9–13 Hence, there is an international need to individualise treatment and identify the children for whom antibiotics are most and least likely to be effective.14
There is growing interest as to how diagnostic research can help with this sort of clinical problem.15 Clinical prediction rules provide the individualised approach needed: they calculate a probability estimate from key variables that predict the risk of an event (diagnosis or prognosis).15,16 There are three stages in the development of a prediction rule: derivation, validation, and assessment of impact on clinical behaviour.17 Despite the frequency of the diagnostic and prognostic challenges faced in general practice, few clinical prediction rules have been derived, let alone validated in primary care.18 The current authors previously derived a clinical rule that predicts complications (as a future event as opposed to detection of serious illness when a child first presents) of acute cough in primary care.19 It was found that preschool children had a pre-test probability for complications of 10%, and post-test probabilities of 6% for children with neither fever nor chest signs, 18% for children with chest signs, 28% for children with fever, and 40% for children with fever and chest signs. However, this rule needs to be validated before it can be applied clinically. Validation is required for two reasons:17 first, associations between predictors and outcome may be due to chance, and second, the predictors may be idiosyncratic to a particular group of patients, clinicians, or study setting. In these cases, when tested in a different population, a new set of predictors would emerge or the rule would not adequately distinguish children with differing post-test probabilities. For these reasons, validation is generally taken to mean that the rule works satisfactorily in patients other than those in whom the rule was derived.20 Therefore, the aim of this study was to validate the rule in a separate population of children and clinicians. Specifically, the objectives of the study were to assess: 1) if the same variables are important; 2) if the estimated coefficient variables are compatible; 3) how well the model fits the new data; and 4) if the post-test probabilities of the prognostic groups are markedly different.
The derivation study was conducted in Leicester. Career changes of two authors provided the opportunity to validate the study in Bristol and Tayside. One-hundred and forty-eight practices in the Bristol area and 23 in Tayside were invited to participate. Children were recruited between October 2004 and May 2005 at morning and afternoon surgeries that included routine and emergency appointments. The design, recruitment, and data-collection methods were identical to those used in the derivation study.19 A researcher was located in the waiting areas to maximise the identification of eligible children and minimise selection bias.19 Eligible children were aged from 3–59 months seeing a GP or nurse practitioner with acute (up to 28 days) cough, and without known asthma or other chronic disease.
Cough is the most common presenting problem to health services internationally. Few clinical rules have been derived, let alone validated, in primary care. A rule has been derived that distinguishes children at low, medium, and high risk of complications of acute cough. This study did not find evidence to support the use of the clinical rule for predicting complications of acute cough. Evidence was found of three possible reasons why validation was not achieved in this study: spectrum bias, confounding by indication, and chance. Other validation studies should be designed to minimise confounding by indication and attempt to measure and adjust for spectrum bias.
After consent, the following predictor variables were collected: sociodemographic data from the parent in the waiting room and clinical data from the GP or nurse practitioner immediately after a routine consultation. As per the derivation study, the cough duration outcome was collected using a validated symptom diary,21 and the outcome of interest was complications. This was defined as any new symptom, sign, or diagnosis suggesting a deterioration in condition. Complications could be due to treatment or the underlying illness and resulted in a parent-initiated primary or secondary care NHS contact prior to cough resolution. Complications were identified (blind to the child's sociodemographic and clinical data) from the child's primary care medical record.
Data were entered onto an Access database and analysed using Stata (version 9). A random 10% of the data were double entered and checked for errors; none was found. For objectives ‘1’ and ‘2’, the analytic methods described in detail for the derivation of the rule were repeated.19 Briefly, logistic regression was used to ascertain univariable associations between all predictor variables and complications. Those with stronger relationships (using a liberal threshold of P<0.2) were then grouped into sociodemographic, clinical history, and clinical examination predictors. Multivariable logistic regression was used to determine the independent relationships with complications, first within and then between groups; variables with P<0.05 were retained. For objectives ‘3’ and ‘4’, the performance of fever and chest signs (the original model) in the new data was assessed, and comparison was made of the strength of expected (derivation) with observed (validation) associations and post-test probabilities. To assess for spectrum bias (i.e. where the two cohorts differ with respect to one or more factors also associated with the outcome of interest) baseline and outcome data were compared between derivation and validation studies as were the incidence of bronchitis, bronchiolitis, pneumonia, and influenza circulating nationally during the recruitment periods using data from the Royal College of General Practitioners' (RCGPs') Birmingham Research Unit (www.rcgp.org.uk/bru). Three analyses were performed to assess for confounding by indication; this occurs when a risk marker triggers the use of treatment, which in turn alters the risk of the outcome of interest and hence the apparent association between the risk marker and the outcome.22 First, confounding or interactions between antibiotics and the predictors independently associated with the outcome in the validation study, but not the derivation study, were examined. Second, the study investigated whether antibiotic prescribing confounded the relationships between fever and chest signs with complications in the validation study. Third, the factors associated with antibiotic use in each of the two studies were examined.
There is no single agreed method for determining the sample size needed for validation, as it depends on whether statistical or clinical validation is required.20 One sample size guideline for statistical validity is to provide at least 10 participants with the outcome of interest for every explanatory variable in the final model.23 With a complication rate of 12% and a final model with two variables, this would require (2 × 10 × [1/0.12]) = 167 children. An alternative sample size calculation focusing on clinical validity would be based on the precision with which the post-test probability for children with neither chest signs nor fever could be determined. This post-test probability is selected because this is the most common combination of clinical signs (71% of children), and at 6% this post-test probability is low enough to rule out complications and increase clinical confidence to withhold antibiotics. A sample of 200 children without these signs would yield a 95% confidence interval (CI) of 3% to 11 % around an estimate of 6%, using the exact binomial calculation. Allowing for the children with chest signs and/or fever and 85% follow-up, a total of (200 × [1/0.71] × [1/0.85]) = 331 children would be required.
Eleven Bristol and four Tayside practices agreed to participate. Recruitment was stopped at two of the Bristol practices because of low recruitment rates associated with low numbers of registered preschool children. The remaining 13 practices were located in urban and inner-city areas, with a total of 104 473 patients registered, of whom 5879 (5.6%) were under 5 years of age. Derivation study practices were also located in urban or inner-city areas and had a total of 105 689 patients registered, of whom 6722 (6.4%) were under 5 years. Figure 1 summarises the recruitment of children to the validation study, and shows that 856/913 (94%) of children were assessed for eligibility and 152/166 (92%) had data collected on the outcome of interest (complications), demonstrating similar levels of selection and attrition to the derivation study at 95% and 87% respectively. Table 1 shows the sociodemographic, clinical, and outcome data of the children in the derivation and validation cohorts. Comparing cohorts, most factors were similar: sex, deprivation, number of consultations in the 12 months prior to recruitment, clinical examination and assessment, cough duration prior to and post-recruitment, reconsultations, and complications.
However, possible differences existed for other factors: there was a reduction in the mean number of children attending the validation recruitment sessions (4.7 versus 7.1) and the number recruited with cough (0.8 versus 2.1); children in the validation study were older (median age 25 versus 21 months); fewer were white (75% versus 82%); more attended day care (58% versus 44%); fewer lived with smokers (36% versus 47%); more were recruited at their first healthcare contact for cough (82% versus 71%); and more were treated with antibiotics (24% versus 18%). Although illness severity was judged by the clinicians to be similar at the index consultation and the complication rates were similar between cohorts, Table 2 suggests that the complications in the validation study may have been qualitatively less severe, with fewer hospital admissions (none) compared with 2% in the derivation study.
Supplementary Table 1 shows the univariable associations between predictors and complications, comparing derivation and validation studies. For the rule to be valid, one would expect to see chest signs, or fever, or both associated with complications in the validation cohort. In the event, there was no evidence that the presence of chest signs or fever was associated with complications, but the CIs for these predictors overlapped with those from the derivation study, suggesting that validating these associations has been neither confirmed nor ruled out. Instead, predictor variables that were found to be independently associated with complications in the validation cohort were: younger age, female sex, non-white ethnicity, increased material deprivation, and higher prior consulting behaviour (Table 3). With the exception of age, which was found to be weakly associated with complications, these are different from the predictor variables found in the derivation cohort. Table 4 shows how the original model fits the new data. As expected from the univariable associations, results show that the presence of fever and fever with chest signs reduced, rather than increased, the post-test probability of complications.
Nonetheless, data were used from the RCGPs' Birmingham Research Unit to look for evidence of spectrum bias. These show broad similarity in the incidence of bronchitis, bronchiolitis, and pneumonia affecting preschool children between derivation and validation study recruitment periods; however, Supplementary Figure 1 suggests there may have been a higher peak mean weekly incidence of influenza-like illnesses in the derivation than in validation study periods.
No evidence was found in the derivation or validation datasets of confounding by indication or effect modification for the five new predictors identified in the validation cohort. However, regarding fever and chest signs in the validation dataset, it was found that antibiotic prescribing attenuated (reduced the strength of) the associations between fever (univariable odds ratio [OR] 0.81, adjusted OR 1.07, P = 0.05) and chest signs (univariable OR 0.37, adjusted OR 0.75, P = 0.03) with complications. Univariable associations with antibiotic prescribing for fever and chest signs were 9.9 (P<0.001) and 3.1 (P = 0.006) respectively in the validation study, compared with 3.9 (P = 0.003) and 7.2 (P<0.001) in the derivation study.
Although this is one of the first attempts to validate the prognostic value of common clinical signs in primary care, the authors' rule to predict complications in preschool children with acute cough has not been validated by this second study. If anything, the presence of fever and chest signs tended to be protective for complications, and although evidence was found of spectrum bias, the 95% CIs were wide, and the ORs were attenuated after adjustment for antibiotic prescribing. In the validation cohort, younger age, increased deprivation, non-white ethnicity, male sex and high previous consulting behaviour were observed to be risk factors for complications.
Despite the derivation and validation studies being of good methodological quality,24 there are three explanations that individually, or in combination, may explain why fever and chest signs were not found to predict complications in the validation study. While specific to the study of childhood cough, these explanations and the study limitations that follow illustrate some of the generic methodological challenges facing diagnostic and prognostic research. The first is spectrum bias. Reductions were found in consultation and recruitment rates consistent with the general UK trend for falling rates of respiratory tract infections seen in primary care;25 there were possible sociodemographic differences, reduced rates of hospital admission, and possible reduced levels of circulating influenza-like illness between the derivation and validation cohorts. Together, these factors indicate a rule that may be valid for children with cough associated with more severe illness including influenza, but not in less severe illness.
The second possible explanation is confounding by indication. This could explain the failure to validate the prediction rule if clinicians' antibiotics prescription tended to be targeted at children with chest signs and/or fever, and if antibiotics were effective in preventing complications. A slight increase was found in the overall use of antibiotics between the first and second studies (against the national trend),10 and there was an increase in use of antibiotics for children with fever from derivation to validation. Also, it was found that antibiotic prescribing attenuated the (protective) association between chest signs and fever with complications in the validation dataset.
Finally, while the factors from the derivation model appear consistent with established clinical wisdom and they do not appear to be a chance finding, particularly for fever (P = 0.005), those from the validation study do not have the same face validity and were more likely to be due to chance (P-values ranging from 0.02 to 0.11; Table 3), so it is possible that the differences observed between the derivation and validation regression models are chance findings. Recruitment was not extended beyond the end of the first winter as funding was limited, and at that time there were full data on 152 children, of whom 20 had experienced a complication. If fever and chest signs had been associated with complications then there would have been sufficient numbers to meet conventional statistical validity as the ‘10 outcomes per variable’ sample size criterion was fulfiled.23
It is possible that the clinician assessment of fever and chest signs were insufficiently reliable for use in a clinical rule.26 κ agreement values of at least 0.6 have been proposed.24 Although this may be unrealistically stringent, there is a trade-off between a rule consisting of common, easily obtained clinical findings and one with reliable predictors that are rare, difficult, or costly to obtain. As the derivation study19 used symptoms and signs as assessed in routine clinical care (so making it generalisable to everyday practice), the intention in this investigation was to not alter or standardise clinicians' methods of ascertaining the presence of fever or chest signs in the validation study. A thermometer-measured temperature in degrees Celsius was recorded in 24% of children in the derivation study, and this percentage was similar in the validation study at 20% when the investigators also asked how temperature was assessed: touch only was used in 75%, tympanic thermometer in 18%, electronic axillary thermometer in 4%, and other in 3%. Similarly, the study used the same outcome in the validation as derivation study. The strengths of this outcome (any new symptom, sign, or diagnosis identified by a health professional at a parent-initiated reconsultation) are that it is consistent with the literature,8 and it includes both the minor and more serious events so that any rule would be certain not to miss complications. However, its weaknesses are that it uses primary care clinician-derived diagnostic labels, which can be unreliable, particularly in the face of variable parental thresholds to reconsult.27 Given that one possible factor in the validation model was ‘higher prior consulting behaviour’, it is possible that some children of higher-consulting families did not have true complications but were simply given a new diagnostic label. It is because of these uncertainties that all the complications listed in Table 2 were included, for example ‘viral-induced wheeze’ and ‘chickenpox’, as only then was it possible to be certain that a validated rule would have helped clinicians to rule out the need for subsequent antibiotic treatment, which was its main potential application.
The problem of how to determine the ‘correct’ factors despite conflicting evidence is not new, as illustrated by the recent hormone-replacement therapy (HRT) and coronary heart disease (CHD) controversy in which observational studies found the use of HRT was protective, and randomised controlled trial data found HRT was a risk factor for CHD.28 Although no clear consensus has been reached on that issue, it is agreed that there must be a biologically plausible explanation for any observed association for it to be regarded as causal.29 There appears to be greater pathophysiological evidence for fever and chest signs being associated with complications than the validation study factors. Fever may be a proxy marker for more severe infection resulting from the release of pyrogenic cytokines such as interleukin-1, tumour necrosis factor, interferon, and prostaglandins.30 Chest signs, unless transmitted, suggest lower-respiratory tract involvement in what is usually a sterile area so may be an early marker of pneumonia or bronchiolitis.31 This, along with the relatively wide CIs found in the validation study,19 lead to a conclusion that the ‘correct’ factors were probably identified in the derivation study.
Given the paucity of primary care clinical rules, there are few published studies with which to compare and contrast this experience of validating this rule, and certainly none for acute cough.17 Rules for the diagnosis of group A β-haemolytic streptococcus have been derived and validated for use in primary care, both studies found that fever, lymphadenopathy, tonsillar exudates, and the absence of cough predicted bacterial infection.32,33 Fever has also been shown in a subgroup analysis to predict response to antibiotics in children with acute otitis media, adding further face validity to at least one of the factors in the present derivation model.34
The prediction rule has not been validated for clinical practice. However, it remains important that validation studies are conducted and reported irrespective of their findings. Future research aiming to validate respiratory infection prediction rules should measure and adjust for likely factors leading to spectrum bias. Where possible, this should include measuring differences in circulating respiratory pathogens and standardising the assessment of predictor variables and outcome. Future studies could also examine different outcomes, such as reconsultations or more serious complications requiring hospital admission. Although confounding by indication can be minimised by recruiting patients from clinicians who generally use fewer antibiotics, its effects can only be eliminated using a placebo-controlled randomised trial study design; opportunities to derive and validate clinical rules nested within randomised controlled trials should be taken.
While this is one of only a few primary care rules in which validation has been attempted, no evidence was found to validate the rule for predicting complications of acute cough in preschool children in primary care. As paediatric infectious illness is costly and associated with high rates of antibiotic use, further research to derive and validate prediction rules remains necessary and should also examine secondary care complications. The present results illustrate the reasons why unvalidated clinical rules should be treated with caution.
We are grateful to the practices, children and parents who participated in the study and to Ritu Khinda-Jones for her help with data collection and the follow-up of children in Tayside.
Additional information accompanies this article at http://www.rcgp.org.uk/bjgp-suppinfo
The Scientific Foundation Board of the Royal College of General Practitioners funded the study (reference no. SFB/2003/36). Alastair Hay holds a postdoctoral award from the National Coordinating Centre for Research Capacity Development (NCCRCD), Department of Health. The views and opinions expressed in this paper do not necessarily reflect those of the NCCRCD or the Department of Health
The Southmead (Bristol) and Tayside Research Ethics Committees approved the study (reference numbers 04/Q2002/17and 04/S1401/142)
The authors have stated that there are none