PMCCPMCCPMCC

Search tips
Search criteria 

Advanced


 
Formats
Article sections
Authors
Related links
Logo of plosonePLoS OneView this ArticleSubmit to PLoSGet E-mail AlertsContact UsPublic Library of Science (PLoS)
 
PLoS One. 2012; 7(1): e27793.
Published online 2012 January 4. doi:  10.1371/journal.pone.0027793
PMCID: PMC3251620
Copyright This is an open-access article, free of all copyright, and may be freely reproduced, distributed, transmitted, modified, built upon, or otherwise used by anyone for any lawful purpose. The work is made available under the Creative Commons CC0 public domain dedication.
Development of the Respiratory Index of Severity in Children (RISC) Score among Young Children with Respiratory Infections in South Africa
Carrie Reed,1,2* Shabir A. Madhi,3,4 Keith P. Klugman,3,4 Locadiah Kuwanda,3 Justin R. Ortiz,2,5 Lyn Finelli,1 and Alicia M. Fry1
1Influenza Division, National Center for Immunization and Respiratory Diseases, Centers for Disease Control and Prevention, Atlanta, Georgia, United States of America
2Epidemic Intelligence Service, Office of Workforce and Career Development, Centers for Disease Control and Prevention, Atlanta, Georgia, United States of America
3Respiratory and Meningeal Pathogens Research Unit, Department of Science and Technology/National Research Foundation, Vaccine Preventable Diseases and Medical Research Council, University of the Witwatersrand, Bertsham, South Africa
4Hubert Department of Public Health, Rollins School of Public Health, Emory University, Atlanta, Georgia, United States of America
5University of Washington, Seattle, Washington, United States of America
Ravi Jhaveri, Editor
Duke University School of Medicine, United States of America
* E-mail: creed1/at/cdc.gov
Conceived and designed the experiments: CR SM KPK LK LF AMF. Analyzed the data: CR JRO. Wrote the paper: CR AMF.
Received June 8, 2011; Accepted October 25, 2011.
Abstract
Objective
Pneumonia is a leading cause of death in children worldwide. A simple clinical score predicting the probability of death in a young child with lower respiratory tract infection (LRTI) could aid clinicians in case management and provide a standardized severity measure during epidemiologic studies.
Methods
We analyzed 4,148 LRTI hospitalizations in children <24 months enrolled in a pneumococcal conjugate vaccine trial in South Africa from 1998–2001, to develop the Respiratory Index of Severity in Children (RISC). Using clinical data at admission, a multivariable logistic regression model for mortality was developed and statistically evaluated using bootstrap resampling techniques. Points were assigned to risk factors based on their coefficients in the multivariable model. A child's RISC score is the sum of points for each risk factor present. Separate models were developed for HIV-infected and non-infected children.
Results
Significant risk factors for HIV-infected and non-infected children included low oxygen saturation, chest indrawing, wheezing, and refusal to feed. The models also included age and HIV clinical classification (for HIV-infected children) or weight-for-age (for non-infected children). RISC scores ranged up to 7 points for HIV-infected or 6 points for non-infected children and correlated with probability of death (0–47%, HIV-infected; 0–14%, non-infected). Final models showed good discrimination (area under the ROC curve) and calibration (goodness-of-fit).
Conclusion
The RISC score incorporates a simple set of risk factors that accurately discriminate between young children based on their risk of death from LRTI, and may provide an objective means to quantify severity based on the risk of mortality.
Pneumonia is a major cause of morbidity and mortality in children less than age five years. The World Health Organization (WHO) estimates that lower respiratory tract infection (LRTI) is responsible for approximately 20% of deaths in children less than five years worldwide, 90% of which is pneumonia [1]. The highest mortality from childhood respiratory infection is seen in resource-limited countries and in children younger than 24 months of age [2]. In children with HIV infection, the burden of pneumonia is even greater, with a broader range of pathogens and a case-fatality ratio three- to six-times higher than children without HIV infection [3],[4].
Effective management of pneumonia in children includes an assessment of the severity of illness, and in many settings around the world, this may rely primarily on signs and symptoms ascertained at presentation. Case management in many resource-limited settings is based on the Integrated Management of Childhood Illness (IMCI) system developed by the World Health Organization to aid treatment decisions for major causes of death in children under five [5]. The IMCI classifies pneumonia, severe pneumonia, and very severe pneumonia with simple case definitions that include elevated respiratory rate, chest wall indrawing, and other danger signs. These deliberately sensitive definitions maximize the number of children identified who could benefit from antibiotic treatment or referral to a hospital. However, many countries report poor compliance with hospital referral due to geographic and socioeconomic constraints, undermining efforts to reduce childhood mortality [6], [7]. More specific discrimination of children with LRTI based on their risk of mortality may help refine decisions about case management, such as the most appropriate site of treatment or the need for additional supportive care.
Clinical prediction scores have been developed and validated to aid clinicians in managing and treating adult patients with community-acquired pneumonia [8], [9]. These scores assign points to a variety of patient characteristics, clinical signs and laboratory measurements to calculate a total score which objectively discriminates patients based on their risk of mortality [10]. No similar method has been developed to quantify the severity of pediatric pneumonia.
A simple pediatric severity score could compliment the current IMCI framework by predicting the probability of death in a child who presents with respiratory illness or provide a standardized means of quantifying severity among children during vaccine trials and other epidemiologic studies. The objective of this study was to develop a severity score for LRTI in children with and without HIV infection – the Respiratory Index of Severity in Children (RISC) – and assess its ability to predict mortality by evaluating the performance of the scoring model.
Ethics Statement
The original study was approved by the Committee for the Study of Human Subjects at the University of the Witwatersrand, and permission for the trial was obtained from the Medicines Control Council of South Africa. All participants were enrolled after written informed consent had been obtained from a parent or legal guardian. Secondary analysis of the original data for this manuscript was approved by the Institutional Review Board of the Centers for Disease Control and Prevention.
Participants
We performed a secondary analysis of data from children <24 months of age who were enrolled in a clinical trial of the 9-valent pneumococcal conjugate vaccine in Soweto, South Africa and hospitalized with LRTI between 1998 and 2001. Details of the trial have been described elsewhere [11], [12]. Briefly, 39,836 children were randomized to receive 3 doses of the 9-valent pneumococcal conjugate vaccine or a placebo at 6, 10, and 14 weeks of age. All children also received Haemophilus influenzae type b (Hib) vaccine. Study staff monitored all admissions at the local hospital (Chris Hani-Baragwanath Hospital; Soweto, South Africa) through November 2001 and determined when study participants were admitted with LRTI (clinical diagnosis of pneumonia or bronchiolitis, irrespective of radiographic features). Children were tested for HIV infection when they were hospitalized for any reason. Details of HIV testing have been described previously [11].
All children hospitalized with LRTI during the follow-up period were potentially eligible for this analysis. Hospitalizations were excluded from the analysis if the child's HIV infection status was unknown, data on illness history or physical exam was not collected, or the child was ≥24 months at the time of admission. Children who received the study vaccine were less likely to be hospitalized with respiratory illness than those in the comparison arm [11], but among those hospitalized with LRTI, vaccinated children did not have reduced mortality compared with unvaccinated children. Consequently, children in both the vaccine and comparison arms were included in this analysis.
Measures
The outcome of interest was in-hospital mortality among children hospitalized with LRTI. As potential predictors of mortality, we considered the following classes of variables: Demographics, medical history, history of present illness, signs on physical exam, growth standards, chest radiography, and C-reactive protein levels. Information on these variables was collected by study physicians on a standardized case report form when a child was hospitalized. Subjective information on symptoms occurring prior to hospitalization was obtained from the child's caregiver at the time of hospitalization.
For this analysis, age was categorized based on IMCI categories: 6 weeks–2 months, 3–12 months, and 12–23 months. Children were considered to have low oxygen saturation if a pulse oximetry reading on room air was ≤90%. Three growth standards were also evaluated: weight for age, weight for length, and length for age, categorized based on the WHO z-scores [13]. Tables of growth standards were accessed at: http://www.who.int/childgrowth/standards/. Chest radiographs were evaluated independently by a pediatrician and a radiologist. C-reactive protein levels were categorized as >40 mg/L or ≤40 mg/L, which may indicate bacterial pneumonia [14]. For children with HIV infection, the clinical classification of HIV disease without CD4 count was recorded using the CDC categories – N (asymptomatic), A (mildly symptomatic), B (moderately symptomatic), C (severely symptomatic, AIDS-defining) [15].
Impact of HIV status
Because of the high prevalence of HIV infection in the study population and the substantial impact of HIV infection on mortality in young children, separate models were developed for HIV-infected and non-infected children. The cohort was stratified by HIV status and the following methods were used first to develop a model for children without HIV infection, and then repeated in the subset of children with HIV infection.
According to the local standard of care at the time, antiretroviral drugs for the treatment of HIV infection were not routinely used. Trimethoprim-sulfamethoxazole prophylaxis was recommended for HIV-exposed children aged 6 weeks or as soon as the child received a diagnosis of HIV infection if prophylaxis was not previously started; prophylaxis continued past the age of 1 year in children with AIDS of clinical category B or C [16].
Identifying risk factors for mortality
Using mortality as a dichotomous outcome variable, we assessed the relationship between each risk factor and mortality. Odds ratios for mortality and their 95% confidence intervals, as well as p-values, were calculated for each risk factor using logistic regression with generalized estimating equations (GEE) to account for the inclusion of multiple episodes among some children. Variables with p<0.20 in univariate analysis were considered as candidate predictors for multivariable regression. To develop the multivariable model, we evaluated included variables for interaction, and then all candidate variables were included in a GEE model and non-significant variables were eliminated one at a time, starting with the variable with the smallest magnitude of effect, until all remaining variables had p<0.05 or removing an additional variable significantly increased the −2 log likelihood of the model.
When developing the multivariable model, we initially chose to exclude some variables that, although associated with mortality, could have reduced the usability of the index in some settings, including chest radiography and laboratory measurement of CRP levels. After selecting a multivariable model, each of these variables was then added to assess whether they improved the predictive performance of the model.
Developing a scoring system
A point-based scoring system was developed from the final multivariable logistic regression model in which a number of points was assigned to each predictor in the model by rounding each β coefficient to the nearest integer. For consecutive cut-offs of the summed scores, the sensitivity, specificity and positive predictive values for mortality were calculated.
Statistical validation
To assess the predictive accuracy of our scoring system, two measures of model performance were evaluated – discrimination and calibration. Models with good discrimination distinguish well between patients with and without the outcome of interest. Discrimination was measured using the c-statistic, or area under the receiver operating characteristic (ROC) curve. The c-statistic ranges from 0.5 to 1, with higher values indicating better discrimination. Calibration measures how well the predicted probabilities of death match the actual mortality observed across the data. Calibration was assessed using the Hosmer-Lemeshow goodness of fit test. A low chi-square and high p-value indicates good calibration. Calibration was also assessed visually by graphing the observed frequencies of mortality against the predicted probabilities of mortality at each score level.
The performance of a prediction model is generally worse in new patients than initially observed when developing the model. This ‘optimism’ can be assessed with standard bootstrapping procedures [17]. To assess the internal validity of our model, bootstrap samples were drawn with replacement and of the same size as the original sample. The multivariable model was re-estimated in 300 bootstrap samples and each model was evaluated on the new sample and the original data. The average difference in the c-statistic indicates the optimism in the initially estimated discrimination [18].
All statistical analysis was performed using SAS software version 9.2 (SAS Institute, Cary, NC).
Study population
From 1998 to 2001, 4,584 hospitalizations with LRTI were observed in 3,140 children enrolled in the 9-valent pneumococcal conjugate vaccine trial in South Africa. Episodes of LRTI were excluded from the present study if they had unknown HIV status (n = 101, 2.2%), if either illness history (n = 21, 0.5%) or physical exam (n = 23, 0.5%) were not assessed, or the child was ≥24 months of age at the time of hospitalization (n = 319, 6.9%). Overall, 4,148 (90.5%) LRTI hospitalizations observed during the vaccine trial were included in this study.
Thirty-six percent (n = 1,502) of all LRTI hospitalizations occurred among children with HIV infection. Mortality varied greatly by HIV status, as 17.6% of all episodes in children with HIV infection (n = 265) resulted in death, compared to 1.3% of all episodes in children without HIV infection (n = 33).
Scoring model
Risk factors associated with mortality were identified for children with and without HIV infection (see Table 1). In multivariable regression (Table 2), independent risk factors for mortality in HIV non-infected children included: low oxygen saturation on room air, chest wall indrawing, low weight for age, and refusal to feed. Wheezing was associated with a decreased risk of mortality. Independent risk factors for mortality in HIV-infected children were slightly different and included: low oxygen saturation on room air, chest wall indrawing, and refusal to feed, but also age, and HIV clinical classification. Wheezing was also associated with a decreased risk of mortality in HIV-infected children. Due to interaction between oxygen saturation and chest indrawing in the multivariable model, we combined these factors in the final model; chest indrawing was only associated with increased mortality if oxygen saturation was normal.
Table 1
Table 1
Description and univariate analysis of risk factors for mortality among HIV non-infected and HIV-infected children <24 months of age hospitalized with LRTI.
Table 2
Table 2
Independent risk factors for mortality in children <24 months hospitalized with LRTI, by HIV status.
A number of points was assigned to each of the risk factors in the final models and is shown in Table 3. An overall score was calculated for each episode of LRTI by adding the points for each risk factor identified at presentation. For example, a child without HIV infection who presented with low oxygen saturation (3 points) and was wheezing (subtract 2 points), but was of normal weight for age (0 points) and was not refusing feedings (0 points) would have a total RISC score of 1 point.
Table 3
Table 3
RISC scoring system.
The RISC scores for children ranged up to 7 points for HIV-infected children and 6 points for non-infected children. For each score level, the observed risk of mortality and predicted probability of death from the model is shown in Figure 1. Among HIV non-infected children in this population, the median RISC score was 1 (Table 4), with a corresponding risk of mortality of 0% (95%CI: 0%–0.6%) (Figure 1). Among HIV-infected children, with a higher overall risk of mortality, the median RISC score was 4 (Table 4), with a mortality of 14.4% (95%CI: 10.6%–18.2%).
Figure 1
Figure 1
Observed mortality (with 95% confidence intervals) and mean predicted mortality by RISC score for children <24 months hospitalized with LRTI, (A) without HIV infection and (B) with HIV infection.
Table 4
Table 4
Distribution of RISC scores and screening performance in children <24 months of age with and without HIV infection, hospitalized with LRTI in Soweto, South Africa 1998–2001.
We considered three variables as additions to the multivariable model for HIV non-infected children to determine whether they improved the performance of the final model. In multivariable analysis, hepatomegaly (aOR = 3.1, 95%CI: 1.4–6.8) or the presence of alveolar consolidation on chest radiograph (aOR = 2.0, 95%CI: 0.9–4.7) were each independently associated with mortality in HIV non-infected children, though the discrimination of the multivariable model was already high and did not improve with either addition. In addition, 198 (7.5%) episodes would have been excluded due to missing data on chest radiograph. Elevated CRP level was not a statistically significant independent predictor of mortality in children without HIV infection (OR = 1.6, 95%CI: 0.7–3.7), and excluded 908 (34.3%) episodes. Thus, these variables were not retained in the final RISC score, but may warrant further consideration.
Validation
The discrimination and calibration for both the HIV-infected and non-infected models were measured to evaluate the performance of the RISC score. The c-statistic, or area under the ROC curve, is a measure of model discrimination and ranges from 0.5 to 1, with higher values indicating better discrimination. In the full dataset, the models showed good discrimination (c-statistic = 0.776 for HIV-infected and 0.923 for non-infected children) and calibration (p-value for goodness-of-fit = 0.95 for HIV-infected and 0.87 for non-infected children), although the model for HIV non-infected children provided better discrimination of mortality than the model for HIV-infected children (see Table 4 and Figure 2). We also evaluated the internal validity of the RISC model using standard bootstrapping techniques to estimate the optimism, or possible over-fitting, in our measurement of model discrimination. From 300 bootstrap samples, the c-statistic from models among HIV non-infected children ranged from 0.742–0.926, with an estimated average optimism of 0.013 in the original data. For HIV-infected children, the c-statistic from models of 300 bootstrap samples ranged from 0.665–0.732, with an estimated average optimism of 0.004.
Figure 2
Figure 2
Distribution of RISC score stratified by mortality for children <24 months hospitalized with LRTI, (A) without HIV infection and (B) with HIV infection.
We developed and validated a clinical prediction score, the RISC score, which uses a simple set of clinical factors to discriminate between young children with varying risks of death from LRTI. Variables retained in the score represent known risk factors for severe outcomes of respiratory illness in children, including: hypoxemia, chest indrawing, refusal to feed, malnutrition, age, and stage of HIV disease [19], [20], [21], [22], [23], [24], [25]. We demonstrate that a simple combination of these factors in a scoring model provides good discrimination and calibration for predicting mortality in young children with LRTI, without the need for radiographic or laboratory measurements.
This severity index has several potential uses in both research and clinical settings. In the research setting this tool may provide a useful means of quantifying the severity of LRTI in epidemiologic studies and clinical trials. In clinical settings, the ability to quantify the risk of mortality may aid clinicians in the management of young children that present with LRTI and compliment current IMCI guidelines. Recent studies have indicated that some children with severe pneumonia by IMCI criteria may be successfully managed at home [26], and that in regions with barriers to hospital referral, limiting referrals to those most in need actually improved the overall number of children who received appropriate treatment and may have reduced mortality [6]. The use of the RISC score may help refine decisions about case management in resource-limited countries or areas with geographic and socioeconomic barriers to hospital referral by facilitating decisions about the most appropriate site of treatment (i.e., home vs. hospital) or the need for additional supportive care (i.e., supplemental oxygen or intensive care).
Hypoxemia was an important risk factor among both HIV-infected and non-infected children. While pulse oximeters may not be currently available in all settings, our finding supports several recent studies that have shown the importance of pulse oximetry in guiding the use of oxygen therapy and other supportive care for reducing the mortality in children with respiratory infection [27], [28]. In contrast, wheezing was associated with reduced mortality in both models. There have been concerns about the potential for misclassifying pneumonia in children with wheeze, which can also result in rapid breathing and thus an IMCI classification of pneumonia. Studies of children with IMCI-defined pneumonia and wheeze have found deterioration was more likely in children with additional danger signs such as chest indrawing or malnutrition [29]. In the absence of additional danger signs, children with wheeze would have a low RISC score, which supports suggestions that in many instances these young children might be successfully managed without hospitalization [29].
Predictive models are expected to perform better in the population in which they were developed than in other populations [30]. We used bootstrapping techniques to estimate the possible optimism associated with the apparent performance of the RISC model in this population. However, while the RISC score includes recognized predictors of severe LRTI, differences in the distribution of risk factors and/or health seeking behaviors in different populations may impact the performance of this model at predicting the risk of death, and will need to be evaluated. Efforts are ongoing to validate these scoring models in other populations of young children with respiratory illness to further define the predicted mortality across the RISC score in a variety of settings. Further study should also focus on evaluating the usefulness and applicability of this tool in different settings [31].
For children with HIV infection, this study was conducted prior to antiretroviral medication use in South Africa. Antiretroviral medication programs can reduce the incidence and severity of HIV-associated pneumonia in children [32], and the impact on this score is unclear. This highlights the need for further evaluation of this model in other populations to determine how the RISC score may best be applied or refined in other contexts.
In order to develop a score that would have high utility in resource-limited settings, which have the greatest burden of childhood mortality from LRTI, we considered the effect of radiographic and laboratory measurements only after developing a strong multivariable model. In this population, these variables did not provide any additional discrimination after other risk factors in the multivariable model and thus was not included in the final RISC score, but may warrant future consideration.
Finally, these data come from a cohort of children enrolled in a pneumococcal vaccine trial. Although there were no differences in mortality between children hospitalized with LRTI in either the vaccine or comparison group, half of the children did receive pneumococcal conjugate vaccine, and all received Hib vaccine. Additionally, although study investigators passively monitored hospitalizations with LRTI and did not directly participate in their clinical care, mortality observed in a vaccine trial may underestimate the risk of mortality that would exist in populations with less access to care. This underscores the need for further study of RISC in additional populations, to understand how it might be adapted in different settings.
The RISC score incorporates a simple set of variables that discriminate the probability of death in children hospitalized with LRTI. The ability to estimate a child's risk of mortality with limited clinical information may provide the ability to improve outcomes for children in resource-limited settings where the burden of pediatric pneumonia is highest. With further validation in additional populations, the RISC score may be a tool to more effectively manage LRTI, a major cause of death in children worldwide, and to better understand the impact of vaccination or other public health interventions on the severity of childhood respiratory illness.
Footnotes
Competing Interests: The authors have declared that no competing interests exist.
Funding: The authors have no support or funding to report for this study. The funders of the original trial that supplied the data for this secondary analysis had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
References
1. Black RE, Cousens S, Johnson HL, Lawn JE, Rudan I, et al. Global, regional, and national causes of child mortality in 2008: a systematic analysis. Lancet. 2010;375:1969–1987. [PubMed]
2. Williams BG, Gouws E, Boschi-Pinto C, Bryce J, Dye C. Estimates of world-wide distribution of child deaths from acute respiratory infections. Lancet Infect Dis. 2002;2:25–32. [PubMed]
3. Enarson PM, Gie RP, Enarson DA, Mwansambo C, Graham SM. Impact of HIV on standard case management for severe pneumonia in children. Expert Rev Respir Med. 2010;4:211–220. [PubMed]
4. Graham SM, Gibb DM. HIV disease and respiratory infection in children. Br Med Bull. 2002;61:133–150. [PubMed]
5. World Health Organization. Cough and difficult breathing. Pocket book of hospital care for children: guidelines for the management of common illnesses with limited resources. Geneva: World Health Organization; 2005.
6. Chowdhury EK, El Arifeen S, Rahman M, Hoque DE, Hossain MA, et al. Care at first-level facilities for children with severe pneumonia in Bangladesh: a cohort study. Lancet. 2008;372:822–830. [PubMed]
7. Simoes EA, Peterson S, Gamatie Y, Kisanga FS, Mukasa G, et al. Management of severely ill children at first-level health facilities in sub-Saharan Africa when referral is difficult. Bull World Health Organ. 2003;81:522–531. [PubMed]
8. Fine MJ, Auble TE, Yealy DM, Hanusa BH, Weissfeld LA, et al. A prediction rule to identify low-risk patients with community-acquired pneumonia. N Engl J Med. 1997;336:243–250. [PubMed]
9. Lim WS, van der Eerden MM, Laing R, Boersma WG, Karalus N, et al. Defining community acquired pneumonia severity on presentation to hospital: an international derivation and validation study. Thorax. 2003;58:377–382. [PMC free article] [PubMed]
10. Aujesky D, Fine MJ. The pneumonia severity index: a decade after the initial derivation and validation. Clin Infect Dis. 2008;47(Suppl 3):S133–139. [PubMed]
11. Klugman KP, Madhi SA, Huebner RE, Kohberger R, Mbelle N, et al. A trial of a 9-valent pneumococcal conjugate vaccine in children with and those without HIV infection. N Engl J Med. 2003;349:1341–1348. [PubMed]
12. Madhi SA, Kuwanda L, Cutland C, Klugman KP. The impact of a 9-valent pneumococcal conjugate vaccine on the public health burden of pneumonia in HIV-infected and -uninfected children. Clin Infect Dis. 2005;40:1511–1518. [PubMed]
13. WHO Multicentre Growth Reference Study Group. WHO Child Growth Standards: Methods and development: Length/height-for-age, weight-for-age, weight-for-length, weight-for-height and body mass index-for-age. Geneva: World Health Organization; 2006.
14. Madhi SA, Kohler M, Kuwanda L, Cutland C, Klugman KP. Usefulness of C-reactive protein to define pneumococcal conjugate vaccine efficacy in the prevention of pneumonia. Pediatr Infect Dis J. 2006;25:30–36. [PubMed]
15. Caldwell MB, Oxtoby MJ, Simonds RJ, Lindegren ML, Rogers MR. 1994 Revised Classification System for Human Immunodeficiency Virus Infection in Children Less Than 13 Years of Age. MMWR. 1994;43:1–10.
16. Madhi SA, Cutland C, Ismail K, O'Reilly C, Mancha A, et al. Ineffectiveness of trimethoprim-sulfamethoxazole prophylaxis and the importance of bacterial and viral coinfections in African children with Pneumocystis carinii pneumonia. Clin Infect Dis. 2002;35:1120–1126. [PubMed]
17. Steyerberg EW, Harrell FE, Jr, Borsboom GJ, Eijkemans MJ, Vergouwe Y, et al. Internal validation of predictive models: efficiency of some procedures for logistic regression analysis. J Clin Epidemiol. 2001;54:774–781. [PubMed]
18. Harrell FE, Jr, Lee KL, Mark DB. Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors. Stat Med. 1996;15:361–387. [PubMed]
19. Young Infants Clinical Signs Study Group. Clinical signs that predict severe illness in children under age 2 months: a multicentre study. Lancet. 2008;371:135–142. [PubMed]
20. Chisti MJ, Tebruegge M, La Vincente S, Graham SM, Duke T. Pneumonia in severely malnourished children in developing countries - mortality risk, aetiology and validity of WHO clinical signs: a systematic review. Trop Med Int Health. 2009;14:1173–1189. [PubMed]
21. Duke T, Mgone J, Frank D. Hypoxaemia in children with severe pneumonia in Papua New Guinea. Int J Tuberc Lung Dis. 2001;5:511–519. [PubMed]
22. Graham SM. HIV-related pulmonary disorders: practice issues. Ann Trop Paediatr. 2007;27:243–252. [PubMed]
23. Lozano JM. Epidemiology of hypoxaemia in children with acute lower respiratory infection. Int J Tuberc Lung Dis. 2001;5:496–504. [PubMed]
24. McNally LM, Jeena PM, Gajee K, Thula SA, Sturm AW, et al. Effect of age, polymicrobial disease, and maternal HIV status on treatment response and cause of severe pneumonia in South African children: a prospective descriptive study. Lancet. 2007;369:1440–1451. [PubMed]
25. Opiyo N, English M. What clinical signs best identify severe illness in young infants aged 0–59 days in developing countries? A systematic review. Arch Dis Child 2011 [PMC free article] [PubMed]
26. Hazir T, Fox LM, Nisar YB, Fox MP, Ashraf YP, et al. Ambulatory short-course high-dose oral amoxicillin for treatment of severe pneumonia in children: a randomised equivalency trial. Lancet. 2008;371:49–56. [PubMed]
27. Duke T, Wandi F, Jonathan M, Matai S, Kaupa M, et al. Improved oxygen systems for childhood pneumonia: a multihospital effectiveness study in Papua New Guinea. Lancet. 2008;372:1328–1333. [PubMed]
28. Weber MW, Mulholland EK. Pulse oximetry in developing countries. Lancet. 1998;351:1589. [PubMed]
29. Hazir T, Qazi S, Nisar YB, Ansari S, Maqbool S, et al. Assessment and management of children aged 1–59 months presenting with wheeze, fast breathing, and/or lower chest indrawing; results of a multicentre descriptive study in Pakistan. Arch Dis Child. 2004;89:1049–1054. [PMC free article] [PubMed]
30. Bleeker SE, Moll HA, Steyerberg EW, Donders AR, Derksen-Lubsen G, et al. External validation is necessary in prediction research: a clinical example. J Clin Epidemiol. 2003;56:826–832. [PubMed]
31. Reilly BM, Evans AT. Translating clinical research into clinical practice: impact of using prediction rules to make decisions. Ann Intern Med. 2006;144:201–209. [PubMed]
32. Gona P, Van Dyke RB, Williams PL, Dankner WM, Chernoff MC, et al. Incidence of opportunistic and other infections in HIV-infected children in the HAART era. JAMA. 2006;296:292–300. [PubMed]
Articles from PLoS ONE are provided here courtesy of
Public Library of Science