PMCCPMCCPMCC

Search tips
Search criteria 

Advanced

 
Logo of nihpaAbout Author manuscriptsSubmit a manuscriptNIH Public Access; Author Manuscript; Accepted for publication in peer reviewed journal;
 
Arch Pediatr Adolesc Med. Author manuscript; available in PMC Oct 4, 2013.
Published in final edited form as:
PMCID: PMC3790639
NIHMSID: NIHMS469353
Validation and Refinement of a Prediction Rule to Identify Children at Low Risk for Acute Appendicitis
Anupam B Kharbanda, MD, MSc,1,11 Nanette C Dudley, MD,2 Lalit Bajaj, MD, MPH,3 Michelle D Stevenson, MD, MS,4 Charles G Macias, MD, MPH,5 Manoj K Mittal, MD,6 Richard G Bachur, MD,7 Jonathan E Bennett, MD,8 Kelly Sinclair, MD,9 Craig Huang, MD,10 Peter S Dayan, MD, MSc,1 and for the Pediatric Emergency Medicine Collaborative Research Committee of the American Academy of Pediatrics
1Department of Pediatrics, Columbia University College of Physicians and Surgeons, New York, NY
2Department of Pediatrics, University of Utah School of Medicine, Salt Lake City, UT
3Department of Pediatrics, University of Colorado School of Medicine, Denver, CO
4Department of Pediatrics, University of Louisville, Louisville, KY
5Department of Pediatrics, Baylor College of Medicine, Houston, TX
6Department of Pediatrics, The Children’s Hospital of Philadelphia and Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA
7Division of Emergency Medicine, Children’s Hospital Boston, Harvard Medical School, Boston, MA
8Department of Pediatrics, Alfred I. duPont Hospital for Children, Jefferson Medical College, Wilmington, DE
9Department of Pediatrics, University of Missouri, Kansas City, MO
10Department of Pediatrics, University of Texas Southwestern Medical Center, Dallas TX
11Department of Pediatrics, University of Minnesota, Minneapolis, MN
Corresponding Author and Reprints: Anupam Kharbanda, MD, MSc, Pediatric Emergency Medicine, East Building, Room M653, 2450 Riverside Avenue, Minneapolis, MN 55454, Office: 612-625-6678; Fax: 612-626-1144
Objective
To validate and refine a clinical prediction rule to identify which children with acute abdominal pain are at low risk for appendicitis (Low Risk Appendicitis Rule).
Design
Prospective, multi-center cross-sectional study.
Setting
Ten pediatric hospital emergency departments.
Participants
Children 3–18 years old who presented with suspected appendicitis from May 2009 – April 2010.
Main Outcome Measures
The test performance of the Low Risk Appendicitis Rule.
Results
Among 2625 patients enrolled, 1018 (38.8%; 95% confidence interval [CI] 36.9% – 40.7%) had appendicitis. Validation of the rule resulted in a sensitivity of 95.5% (95% CI 93.9 – 96.7%), specificity of 36.3% (33.9 – 38.9%) and negative predictive value (NPV) of 92.7% (90.1 – 94.6%). Theoretical application would have identified 573 (24%) as low risk, misclassifying 42 patients (4.5%; 95% CI 3.4% – 6.1%) with appendicitis. We refined the prediction rule, resulting in a model that identified patients at low risk if: a) absolute neutrophil count (ANC) ≤ 6.75 × 103/µL and no maximal tenderness in right lower quadrant (RLQ) or b) ANC ≤ 6.75 × 103/µL, maximal tenderness in the RLQ but no abdominal pain with walking/jumping or coughing. This refined rule had a sensitivity of 98.1% (97.0 – 98.9%), specificity of 23.7% (21.7 – 25.9%) and NPV of 95.3% (92.3 – 97.0%).
Conclusions
We have validated and refined a simple clinical prediction rule for pediatric appendicitis. For patients identified as low risk, clinicians should consider alternative strategies such as observation or ultrasound, rather than proceed to immediate imaging with CT.
Keywords: Appendicitis, Clinical prediction rules
Appendicitis is the most common surgical emergency in children and acute abdominal pain accounts for 5–10% of all pediatric emergency department (PED) visits.13 The diagnosis of appendicitis can be difficult, with many children misdiagnosed on initial presentation.4 Furthermore, negative appendectomy and perforation rates remain high, indicating a need to re-evaluate the diagnostic assessment for this disease.58
Computed tomography (CT) has high sensitivity and specificity for appendicitis and is heavily relied upon in the evaluation of patients with possible appendicitis.9 However, despite dramatic increases in CT utilization, substantial improvements in patient outcomes have not been realized.5, 1013 This discrepancy is potentially the result of over-utilization of CT which is problematic as it results in unnecessary exposure to ionizing radiation, prolonged ED visits, and increased costs.6, 1314
Prior studies have described substantial variability in the evaluation and management of children with suspected appendicitis.10, 15 Standardizing the approach to patients with suspected appendicitis through clinical prediction rules could reduce variability and reliance on CT, thus promoting the delivery of efficient, safe and cost-effective health care.16 Clinical prediction rules can be utilized to risk stratify patients, allowing for tailored management based upon patients’ risks for disease.17
In 2005, our research team published a low risk clinical prediction rule for pediatric appendicitis.18 Single-center internal validation revealed a sensitivity and negative predictive value (NPV) of 98% [95% CI 89–100%] and 98% [95% CI 85–100%], respectively.18 Hypothetical application of the rule could have led to a 20% reduction in CT utilization. Prior to implementation, independent validation of this rule is important. The objective of the current study was to validate and potentially refine our clinical prediction rule in a multi-center cohort of children with suspected appendicitis.
Study Design and Setting
We performed a prospective, cross-sectional study of children with suspected appendicitis at 10 PED’s that are members of the Pediatric Emergency Medicine Collaborative Research Committee (PEM-CRC) of the American Academy of Pediatrics. The PEM-CRC reviewed and approved the final study protocol. The study was approved by each participating site’s Institutional Review Board (IRB) and data user agreements were formalized between sites and the central data center. Seven IRBs granted a waiver of written informed consent/assent and instead allowed for verbal consent. At the three remaining sites, written consent from the guardians and assent from children seven years of age and older was obtained.
Study Patients
Children 3 to 18 years presenting to the PED with acute abdominal pain of less than 96 hours duration and who were being evaluated for suspected appendicitis were approached for enrollment. We defined “suspected appendicitis” patients as those for whom the treating physician obtained blood tests, radiological studies [CT and/or ultrasound (US)] or a surgical consultation for the purpose of diagnosing appendicitis. Radiological studies or surgical consultations were obtained at the discretion of the treating physician. We excluded patients with: pregnancy, prior abdominal surgery (i.e. gastrostomy tube, abdominal hernia repair), chronic abdominal illness or pain (e.g. inflammatory bowel disease, chronic pancreatitis, chronic/recurrent appendicitis), sickle cell anemia, cystic fibrosis, or a medical condition affecting the provider’s ability to obtain an accurate history. We also excluded patients who had radiologic studies (CT or US) of the abdomen performed prior to PED arrival or a history of abdominal trauma within 7 days of the PED evaluation.
Procedures
Prior to initiation, site principal investigators (PI’s) received standardized training that included a detailed manual of operations and instructions on the proper completion of case report forms (CRFs). Site PI’s subsequently conducted group and one-on-one instructional sessions with clinicians who worked in their respective PEDs.
A PEM attending or fellow physician completed a standardized history and physical examination on a structured CRF. A resident physician, nurse practitioner or physician assistant was allowed to complete the CRF with attending oversight. A subset of participants had a separate, independent assessment performed by a second clinician within 30–60 minutes of the first evaluation. Clinicians completed CRFs prior to knowledge of CT or US results.
CRFs were completed on paper and subsequently entered into Adobe Pro (Adobe Systems, San Jose, CA) for electronic transfer to the central data management warehouse through an electronic CRF (Teleforms™). Quality-assurance practices at the data warehouse included surveillance for missing and duplicate data. We determined capture rate by reviewing the PED visit, admission, pathology and radiology logs for 2 random days of each study month. Two sites were able to perform active surveillance (daily data capture monitoring). We compared demographic, clinical and outcome data between enrolled and missed patients to detect possible enrollment bias.
Outcome Measures
The primary outcome was the performance of the clinical prediction rule to identify children at low risk for appendicitis. Patient disposition was based upon physician discretion. Among patients undergoing surgery, we determined the presence of appendicitis from the attending pathologist’s written report. Appendiceal perforation was determined from the attending surgeon’s written operative report. A priori, we standardized the terminology to code pathology and operative reports.
For patients discharged from the PED, we conducted telephone follow-up within 2 weeks to determine resolution of signs and symptoms, visits to other sites of care, and need for surgery. If we were unable to contact the guardian, we reviewed the medical record for 90 days after the index PED visit to determine if the patient underwent a CT, US, or operation at that facility.
Data analysis
The previously published low risk prediction rule consisted of the following variables: ANC ≤ 6.75 × 103/µL, absence of nausea, and absence of maximal tenderness in the RLQ of the abdomen. On the CRFs, clinician’s had the option of coding the presence of nausea as “yes”, “no” or “don’t know” and maximal tenderness in the RLQ as “yes”, “no” or “unsure”. Responses of “don’t know” or “unsure” were analyzed as if the patient had the finding. We excluded patients if any of the prediction rule components were missing. A sensitivity analysis was performed to determine the effect on test performance of recoding “don’t know/unsure” findings as present, absent or missing. We calculated performance of the rule as sensitivity, specificity, positive predictive value (PPV) and negative predictive value (NPV). We assessed the accuracy of the low risk rule based on whether patients were identified as “low risk” in either of the terminal decision tree nodes (as analyzed in the original study).18
Rule Refinement
We anticipated that our validated prediction rule may have diminished performance, thus a priori we planned to refine the rule. We conducted binary recursive partitioning analyses (CART 6.0, Salford Systems, San Diego, CA) to refine our prediction rule, in order to create models that had higher sensitivity (> 95%) without affecting specificity (25–35%). We aimed to create rules for which the risk of appendicitis in the low risk group was less than or, at minimum, similar to the approximately 6–7.5% false negative rate of CT.9, 19 We entered variables into the model that were included in our original study as well as any patient history and physical examination variables that had at least moderate inter-rater reliability (kappa > 0.4).20 The following variables were entered: duration of abdominal pain, nausea, emesis, history of focal RLQ pain, presence of abdominal tenderness, maximal tenderness in the RLQ, abdominal pain with walking, abdominal pain on the right side with walking, the ANC and WBC (using both continuous and categorical cut-points) We identified the categorical cut-points through the use of univariate recursive partitioning. For this analysis, responses that were marked “unsure” or “don’t know” were coded as missing data. We used the Gini splitting method for classification trees and internally validated the results of our refined model using ten-fold cross validation. To create the models, we varied costs to always favor not missing a case of appendicitis rather than diagnosing appendicitis in a child that did not have the illness.
Study Population
Patients were enrolled in 10 PEDs with broad US geographic distribution from March 2009 through April 2010. We removed data from one site prior to analysis as their capture rate was below 40%. Therefore, the study cohort was comprised of 2625 patients across the remaining 9 sites, representing 71% of eligible patients. Enrollment by site ranged from 223 to 473 patients and the capture rate varied from 48% – 96%. A total of 1018 (38.8%; 95% confidence interval [CI] 36.9% – 40.7%) patients were diagnosed with appendicitis, of whom 275 (27%; 95% CI 24.4% – 30.0%) had a perforated appendix. Of those undergoing an operation, no evidence of appendicitis by pathology was found in 95 patients (negative appendectomy rate = 8.5%; 95% CI 7.0 – 10.3%). We completed telephone follow-up on 88% of patients discharged from the PED. None of the 186 patients lost to telephone follow-up had evidence of an appendectomy via review of the medical record (Figure 1).
Figure 1
Figure 1
Flow diagram of study population and final diagnosis
Characteristics of patients
The mean age of enrolled patients was 10.8 [SD ± 3.8] years and 51% were male. The most common diagnoses among patients who did not undergo an appendectomy included: non-specific abdominal pain (42%), gastroenteritis (14%), and constipation (12%). Clinicians obtained a CT in 55%, an ultrasound in 37% and both studies in 9% of patients. In total, 2116 (81%) patients underwent diagnostic imaging. Missed patients were similar to enrolled, with a mean age 11.6 years [SD ± 3.6], 53% of male sex, a 42% rate of appendicitis, and 29% having perforated appendicitis (Table 1). Among missed patients, clinicians used US more frequently (68%), CT less frequently (44%), and there was a higher rate of using CT or US (93%).
Table 1
Table 1
Comparison of Enrolled Patients to Those Eligible but not Enrolled (Missed)
Low Risk Rule Validation
Complete data for rule performance was available for 2390 patients (91%). The most common reason for exclusion from analysis was the absence of a white blood cell count (188 patients). The test characteristics of validation are provided in Table 2; we include the test characteristics of the derivation sample from our prior published study for comparison.
Table 2
Table 2
Comparison of Test Performance in Derivation and Validation Cohorts
Theoretical Application of the Low Risk Rule
Theoretical application of the low risk rule is presented in Figure 2. A sensitivity analysis revealed no significant change in test performance based on coding of unsure/don’t know responses (data available upon request). In total, 573 patients (24% of those with complete data) were identified as low risk; 64 (11%) underwent an operation for presumed appendicitis of whom 42 had pathology proven appendicitis and 22 had negative appendectomies. In addition, 296 (52%) underwent a CT, 241 (42%) an US and, in total, 465 (81%) had either a CT or US performed. Application of the low risk rule would have theoretically prevented 22 unnecessary operations, 465 (24%) diagnostic imaging studies, but missed 42 (4.5%; 95% CI 3.4% – 6.1%) patients who were ultimately diagnosed with appendicitis. In Table 3, we present the clinical characteristics of the 42 patients with appendicitis who were misclassified by the prediction rule.
Figure 2
Figure 2
Effect of hypothetical application of the Low Risk Appendicitis Rule
Table 3
Table 3
Characteristics of Patients Misclassified by Low Risk Rule
Low Risk Rule Refinement
The refined model identified patients as low risk for appendicitis through use of the following a) ANC ≤ 6.75 × 103/µL and no maximal tenderness in the right lower quadrant (RLQ) or b) ANC ≤ 6.75 × 103/µL, maximal tenderness in the RLQ but no abdominal pain with walking/jumping or coughing (Figure 3). Test characteristics of the refined model are presented in Table 4. Of the 400 patients identified as low risk, 27 (6.7%) underwent an operation, 19 of whom had appendicitis. Additionally, of these 400 patients, clinicians obtained CT or US in 301 (75%), including 180 patients (45%) who had a CT.
Figure 3
Figure 3
Refined Low Risk Appendicitis Rule and rule performance
Table 4
Table 4
Refinement of Low Risk Appendicitis Rule
In this large, prospective, multi-center study of children with suspected appendicitis, our previously derived low risk prediction rule maintained high sensitivity and modest specificity in a validation cohort. Furthermore, we refined our low risk rule in order to improve test sensitivity. These low risk rules identify children with suspected appendicitis at low but not zero risk for appendicitis.
Our study adds to a growing literature on the use of clinical prediction rules for managing patients in the emergency department.17, 21 2223 Similar to prior studies, our goal was to identify patients at low risk for illness in order to reduce reliance on diagnostic imaging and to reduce inefficient care delivery. As our study confirms, CT is heavily relied upon to diagnose and manage children with acute abdominal pain.10 The potential benefit of our clinical prediction rule lies in its ability to stratify patients, identifying those at low risk for appendicitis.
Several previous investigators have developed clinical prediction rules or scores for the diagnosis of appendicitis.2427 The Alvarado and Samuel scores are the most commonly cited and although the original studies noted excellent test performance, external validation by independent investigators revealed conflicting results.2830 It should be noted, that both scoring systems were intended to identify patients with appendicitis rather than identify a low risk group.24 Compared to these prior scores, advantages of our prediction rule include its simplicity, external validation in a large sample across multiple PEDs, and ability to more accurately identify a low risk cohort. Lastly, a decision tree format may be easier than a numerical based score for clinicians to remember and use.
Although the sensitivity of our validated low risk prediction rule was high, the NPV was lower than in the derivation study (98% derivation vs. 92.7% validation). As a result, 42 children (4.5% of patients with appendicitis) were misclassified as not having appendicitis. This rate of misclassification may concern clinicians given the potential medical and legal consequences associated with missed appendicitis. We anticipated this issue and thus refined our rule with the goal of improving the sensitivity and NPV. Our refined prediction rule provides sensitivity and NPV which are somewhat higher (98.1% and 95.3%, respectively), but the specificity and PPV of the rule do diminish. Furthermore, the refined rule would still miss some cases of appendicitis (19 patients). Consequently, either rule may be appropriate to identify a low risk population (risk of appendicitis: 7.3% validated rule, 4.8% refined rule) who clinicians may choose to observe for progression of abdominal symptoms. The use of ultrasound and/or surgical consultation may also be viable alternatives. Given the high rate of negative appendectomies in the low risk cohort (> 30%) as compared to the overall study cohort (8.5%), it would be prudent for surgeons to be cautious operating on low risk patients. Ultimately, our prediction rules may be best suited for integration into an appendicitis care algorithm to help stratify risk and guide clinical management (e.g. observation with serial examination for low risk patients).
It is important to consider the potential use of our low risk prediction rules in relation to the performance of CT. Although CT has demonstrated a sensitivity of 94% [95% CI 92–97%] and a specificity of 95% [95% CI 94–97%] for appendicitis, the PPV of CT will be lower when it is used in populations with a low prevalence of appendicitis.9 In addition, the NPV of CT is not 100%. 19 In our present study, if clinicians had acted upon CT results in isolation, 20 patients would have had missed appendicitis (inappropriately discharged home) and 27 patients would have had negative appendectomies (data available upon request). These results support concerns raised by several investigators that the excessive use of CT may lead to unnecessary operations, delays in care and increased costs.3133
Physicians may have concerns regarding the reliability of the clinical variables included in our prediction rules. Through the course of our study, we collected data on the inter-rater reliability of clinical history and physical examination findings, the results of which have been presented previously.20 The presence of nausea had a kappa of 0.44 [95% CI 0.37–0.52], maximal tenderness in the RLQ 0.45 [95% CI 0.36–0.54], and pain with walking 0.54 [95% CI 0.45–0.63], indicating moderate reliability for all three parameters.
Ultimately, the clinical utility of our prediction rules is in their ability to provide a quantitative assessment of risk for appendicitis. In this study we elected to stratify patients as either “low risk” or “not low risk” for appendicitis. In this scheme, patients identified as “low risk” had a risk of appendicitis of 7.3% (validated rule) or 4.8% (refined rule). However, by observing how patients flow within the decision trees, specific risks for appendicitis can be determined depending on a patient’s particular signs and symptoms (range of 4–12% for the various terminal nodes). As electronic health record-based clinical decision support becomes more common within EDs, the ability to calculate an appendicitis risk may allow physicians to tailor management based on their own risk tolerance and availability of diagnostic imaging and surgical resources.
Our study had the following limitations. Enrollment of patients varied considerably by site. To assess for enrollment bias, we conducted random medical record audits which revealed that missed patients were similar to those enrolled. Although we enrolled pediatric patients from numerous geographical regions, enrollment occurred exclusively in PEDs. Therefore, our results may not be able to be generalized to other settings. Our clinical prediction rule was developed and validated in cohorts where the rate of appendicitis was quite high (> 30%). Use of the rule in an urgent care or clinic setting, where the rate of appendicitis is lower, might result in a higher NPV but lower PPV. We collected clinical parameters only at the time of enrollment, thus the patients’ exam may have changed prior to final disposition. Although we made every attempt to follow-up patients discharged from the PED, we cannot exclude the possibility that some underwent appendectomies at alternative facilities. Lastly, we stress that our study was not an implementation study; clinicians should understand the potential risks and benefits of using the validated rule prior to formal implementation of the refined rule before external validation.
Conclusion
We validated and refined a clinical prediction rule for pediatric appendicitis, identifying a population of children with suspected appendicitis who are at low but not zero risk for appendicitis. If applied, clinicians will need to balance the risks of missing a case of appendicitis with the increased risk of negative appendectomies and the potential long-term risks associated with exposure to ionizing radiation. Clinicians should consider alternative strategies such as observation or ultrasound for patients identified as low risk rather than proceeding to immediate imaging with CT.
Acknowledgements
The PEMCRC data center is supported in part by the Center for Clinical Effectiveness at Baylor College of Medicine/Texas Children's Hospital. We thank all of the clinicians who enrolled patients into this study and the research coordinators who greatly facilitated study completion.
Funding
Supported by Grant Number UL1 RR024156 from the National Center for Research Resources (NICRR), a component of the National Institute of Health (NIH) and NIH Roadmap for Medical Research. In addition, Anupam Kharbanda received salary support through the Empire Clinical Research Program (New York State).
The funding agencies took no part in data analysis, interpretation or manuscript preparation. No person received any honorarium or other payment to produce this manuscript. This manuscript was written by ABK and all authors take full responsibility for the integrity of the data and the accuracy of data analysis.
Abbreviations
CTComputed Tomography
USUltrasound
PEMPediatric Emergency Medicine
PEDPediatric Emergency Department
CRFCase Report Form
RLQRight lower quadrant
NPVNegative predictive value

Footnotes
Presented in part at the annual meeting of Pediatric Academic Societies, Denver, Colorado, May 2011
Conflict of Interest
None of the authors have any financial disclosures or conflicts of interest to report.
1. Addiss DG, Shaffer N, Fowler BS, Tauxe RV. The epidemiology of appendicitis and appendectomy in the United States. Am J Epidemiol. 1990 Nov;132(5):910–925. [PubMed]
2. Bundy DG, Byerley JS, Liles EA, Perrin EM, Katznelson J, Rice HE. Does this child have appendicitis? JAMA. 2007 Jul 25;298(4):438–451. [PMC free article] [PubMed]
3. Reynolds SL, Jaffe DM. Children with abdominal pain: evaluation in the pediatric emergency department. Pediatr Emerg Care. 1990 Mar;6(1):8–12. [PubMed]
4. Rothrock SG, Skeoch G, Rush JJ, Johnson NE. Clinical features of misdiagnosed appendicitis in children. Ann Emerg Med. 1991 Jan;20(1):45–50. [PubMed]
5. Flum DR, Morris A, Koepsell T, Dellinger EP. Has misdiagnosis of appendicitis decreased over time? A population-based analysis. JAMA. 2001 Oct 10;286(14):1748–1753. [PubMed]
6. Partrick DA, Janik JE, Janik JS, Bensard DD, Karrer FM. Increased CT scan utilization does not improve the diagnostic accuracy of appendicitis in children. J Pediatr Surg. 2003 May;38(5):659–662. [PubMed]
7. Korner H, Sondenaa K, Soreide JA, et al. Incidence of acute nonperforated and perforated appendicitis: age-specific and sex-specific analysis. World J Surg. 1997 Mar-Apr;21(3):313–317. [PubMed]
8. Flum DR, Koepsell T. The clinical and economic correlates of misdiagnosed appendicitis: nationwide analysis. Arch Surg. 2002 Jul;137(7):799–804. discussion 804. [PubMed]
9. Doria AS, Moineddin R, Kellenberger CJ, et al. US or CT for Diagnosis of Appendicitis in Children and Adults? A Meta-Analysis. Radiology. 2006 Oct;241(1):83–94. [PubMed]
10. Newman K, Ponsky T, Kittle K, et al. Appendicitis 2000: variability in practice, outcomes, and resource utilization at thirty pediatric hospitals. J Pediatr Surg. 2003 Mar;38(3):372–379. discussion 372–379. [PubMed]
11. Kieran JA, Curet MJ, Schermer CR. Institutional variations in the management of patients with acute appendicitis. J Gastrointest Surg. 2003 May-Jun;7(4):523–528. [PubMed]
12. Martin AE, Vollman D, Adler B, Caniano DA. CT scans may not reduce the negative appendectomy rate in children. J Pediatr Surg. 2004 Jun;39(6):886–890. discussion 886–890. [PubMed]
13. Karakas SP, Guelfguat M, Leonidas JC, Springer S, Singh SP. Acute appendicitis in children: comparison of clinical diagnosis with ultrasound and CT imaging. Pediatr Radiol. 2000 Feb;30(2):94–98. [PubMed]
14. Brenner D, Elliston C, Hall E, Berdon W. Estimated risks of radiation-induced fatal cancer from pediatric CT. AJR Am J Roentgenol. 2001 Feb;176(2):289–296. [PubMed]
15. Muehlstedt SG, Pham TQ, Schmeling DJ. The management of pediatric appendicitis: a survey of North American Pediatric Surgeons. J Pediatr Surg. 2004 Jun;39(6):875–879. discussion 875–879. [PubMed]
16. Stiell IG, Wells GA. Methodologic standards for the development of clinical decision rules in emergency medicine. Ann Emerg Med. 1999 Apr;33(4):437–447. [PubMed]
17. Kuppermann N, Holmes JF, Dayan PS, et al. Identification of children at very low risk of clinically-important brain injuries after head trauma: a prospective cohort study. Lancet. 2009 Oct 3;374(9696):1160–1170. [PubMed]
18. Kharbanda AB, Taylor GA, Fishman SJ, Bachur RG. A clinical decision rule to identify children at low risk for appendicitis. Pediatrics. 2005 Sep;116(3):709–716. [PubMed]
19. Kharbanda AB, Taylor GA, Bachur RG. Suspected appendicitis in children: rectal and intravenous contrast-enhanced versus intravenous contrast-enhanced CT. Radiology. 2007 May;243(2):520–526. [PubMed]
20. Kharbanda AB, Stevenson MD, Macias CG, et al. Inter-rater Reliability of Clinical Findings in Children with Possible Appendicitis. Pediatrics. 2012 Apr;129(4) [PMC free article] [PubMed]
21. Holmes JF, Mao A, Awasthi S, McGahan JP, Wisner DH, Kuppermann N. Validation of a prediction rule for the identification of children with intra-abdominal injuries after blunt torso trauma. Ann Emerg Med. 2009 Oct;54(4):528–533. [PubMed]
22. Dayan PS, Vitale M, Langsam DJ, et al. Derivation of clinical prediction rules to identify children with fractures after twisting injuries of the ankle. Acad Emerg Med. 2004 Jul;11(7):736–743. [PubMed]
23. Stiell IG, Clement CM, McKnight RD, et al. The Canadian C-spine rule versus the NEXUS low-risk criteria in patients with trauma. N Engl J Med. 2003 Dec 25;349(26):2510–2518. [PubMed]
24. Samuel M. Pediatric appendicitis score. J Pediatr Surg. 2002 Jun;37(6):877–881. [PubMed]
25. Alvarado A. A practical score for the early diagnosis of acute appendicitis. Ann Emerg Med. 1986 May;15(5):557–564. [PubMed]
26. van den Broek WT, van der Ende ED, Bijnen AB, Breslau PJ, Gouma DJ. Which children could benefit from additional diagnostic tools in case of suspected appendicitis? J Pediatr Surg. 2004 Apr;39(4):570–574. [PubMed]
27. Garcia Pena BM, Cook EF, Mandl KD. Selective imaging strategies for the diagnosis of appendicitis in children. Pediatrics. 2004 Jan;113(1 Pt 1):24–28. [PubMed]
28. Schneider C, Kharbanda A, Bachur R. Evaluating appendicitis scoring systems using a prospective pediatric cohort. Ann Emerg Med. 2007 Jun;49(6):778–784. 784, e771. [PubMed]
29. Mandeville K, Pottker T, Bulloch B. Using appendicitis scores in the pediatric ED. Am J Emerg Med. 2010 Jul 30; [PubMed]
30. Bhatt M, Joseph L, Ducharme FM, Dougherty G, McGillivray D. Prospective validation of the pediatric appendicitis score in a Canadian pediatric emergency department. Acad Emerg Med. 2009 Jul;16(7):591–596. [PubMed]
31. Flum DR, McClure TD, Morris A, Koepsell T. Misdiagnosis of appendicitis and the use of diagnostic imaging. J Am Coll Surg. 2005 Dec;201(6):933–939. [PubMed]
32. Safran DB, Pilati D, Folz E, Oller D. Is appendiceal CT scan overused for evaluating patients with right lower quadrant pain? Am J Emerg Med. 2001 May;19(3):199–203. [PubMed]
33. Lee SL, Walsh AJ, Ho HS. Computed tomography and ultrasonography do not improve and may delay the diagnosis and treatment of acute appendicitis. Arch Surg. 2001 May;136(5):556–562. [PubMed]