|Home | About | Journals | Submit | Contact Us | Français|
The use of molecular-based methods for the diagnosis of bacterial infections in blood is appealing, but they have not yet passed the threshold for clinical practice. A systematic review of prospective and case-control studies assessing the diagnostic utility of PCR directly with blood samples for the diagnosis of invasive pneumococcal disease (IPD) was performed. A broad search was conducted to identify published and unpublished studies. Two reviewers independently extracted the data. Summary estimates for sensitivity and specificity with 95% confidence intervals (CIs) were calculated by using the hierarchical summary receiver operating characteristic method. The effects of sample processing, PCR type, the gene-specific primer, study design, the participants' age, and the source of infection on the diagnostic odds ratios were assessed through meta-regression. Twenty-nine studies published between 1993 and 2009 were included. By using pneumococcal bacteremia for case definition and healthy people or patients with bacteremia caused by other bacteria as controls (22 studies), the summary estimates for sensitivity and specificity were 57.1% (95% CI, 45.7 to 67.8%) and 98.6% (95% CI, 96.4 to 99.5%), respectively. When the controls were patients suspected of having IPD without pneumococcal bacteremia (26 studies), the respective values were 66.4% (95% CI, 55.9 to 75.6%) and 87.8% (95% CI, 79.5 to 93.1%). With lower degrees of proof for IPD (any culture or serology result and the clinical impression), the sensitivity of PCR decreased and the specificity increased. All analyses were highly heterogeneous. The use of nested PCR and being a child were associated with low specificity, while the use of a cohort study design was associated with a low sensitivity. The lack of an appropriate reference standard might have caused underestimation of the performance of the PCR. Currently available methods for PCR with blood samples for the diagnosis of IPD lack the sensitivity and specificity necessary for clinical practice.
Identification of the pathogens responsible for sepsis and septic shock is of primary importance for therapeutic decision making. Survival is dramatically influenced by the early initiation of treatment with the appropriate antibiotics (17, 35). Unnecessary or superfluous antibiotic treatment is associated with side effects and leads to the induction of resistance without the provision of a benefit (19).
Blood culture findings are classically used to define infections. However, they lack sensitivity for most types of infections, particularly following antibiotic treatment, and the results are delayed by the time required for pathogen growth. The case is especially noteworthy for community-acquired pneumonia (CAP), for which positivity rates are very low and blood cultures cannot be used to direct treatment (3). The results of sputum Gram stain and culture are positive for 31 to 68% and 28 to 86% of patients with pneumococcal pneumonia who can produce sputum, respectively (8, 21, 43, 46, 54), while the test for pneumococcal urine antigen is positive for about 70 to 87% of patients (8, 22, 61, 63), with all values being highly dependent on previous antibiotic treatment.
Molecular methods for pathogen identification are appealing since, in principle, the results should not depend on the amount of pathogen present in the sample, identification defined by DNA can be highly specific, and noncultivable pathogens can be detected (18, 44). The results can be made available within hours of a patient's presentation with an infection, during the time window that allows the direction of early antibiotic treatment. In practice, molecular methods have not replaced classical microbiology (7). While specificity is the main hindrance in general, mainly due to the contamination of PCR products with exogenous material or previously used extracts and primers, the poor sensitivities of PCRs with blood samples arose as a major issue. The presence of human DNA and inhibitors in blood may interfere with the PCRs or the hybridization reactions and cause false-negative results. The nonspecific binding of primers due to the presence of repeat sequences in the DNA template, nonspecific binding between the primer and the template, and incomplete primer binding reduce both the sensitivity and the specificity (51).
We systematically reviewed studies that assessed the sensitivity and specificity of PCR-based molecular methods applied to blood samples for the detection of invasive disease caused by Streptococcus pneumoniae. Our objectives were to estimate the pooled test accuracy as currently reported and to analyze the reasons for differences in the sensitivity or the specificity between different tests, infections, and populations studied. The review was conducted and reported according to the guidelines for systematic reviews of diagnostic test accuracy (34).
We included prospective cohort studies, prospective nested case-control studies, and retrospective case-control studies that assessed adults or children. The target condition was invasive pneumococcal disease (IPD), defined as primary bacteremia (signs and symptoms of sepsis without a documented source of infection), pneumonia, empyema, or meningitis caused by S. pneumoniae. Studies that assessed only sinusitis and otitis media were excluded. The index tests included any PCR-based molecular method performed with blood samples for the identification of S. pneumoniae, with or without susceptibility traits. We included PCR, nested PCR, reverse transcription-PCR, real-time PCR, and multiplex PCR. We excluded PCR testing of blood cultures after incubation or after the first identification of growth.
The reference standard was defined at several levels, in which positive results were used to confirm the presence of IPD and negative results were used to rule out IPD, as follows. Level I was the growth of S. pneumoniae in blood cultures. Level II was the growth of S. pneumoniae in cultures of blood or specimens from other sterile sites (cerebrospinal fluid, pleural fluid, lung biopsy specimens). Level III was the growth of S. pneumoniae as detailed for level 2 by the use of pneumococcal antigen-based tests with blood or urine or serological tests for S. pneumoniae. Level IV was any of the methods detailed for level 3 or a clinical rule that defined IPD.
Cultures of nonsterile upper airway samples (sputum, nasopharynx, throat, etc.) were not considered for the diagnosis of IPD in our analysis.
We searched the entries in the PubMed database provided up to October 2009; the Cochrane database, issue 4, 2008; the LILACS, NLM, and KOREAMED databases for entries provided up to December 2008; and the proceedings of the European Congress of Clinical Microbiology and Infectious Diseases and the Interscience Conference on Antimicrobial Agents and Chemotherapy between 2000 and 2008 using the following terms and their medical subject headings (adapted for each database): (sepsis OR pneumonia OR bacteremia OR meningitis OR cellulitis OR erysipelas OR urinary tract infection OR septicemia OR bloodstream-infection) AND (PCR OR real-time OR RT-PCR OR reverse-transcription OR nested-PCR) AND (Streptococcus pneumoniae OR pneumococ*). In addition, we scanned the references of all studies included. We searched the websites for studies involving commercially available products (e.g., LightCycler SeptiFast [Roche] and VYOO [SIRS-Lab, Jena, Germany]).
Two reviewers independently applied inclusion criteria and extracted the data from the studies included. Disagreements were resolved by discussion and in consultation with a third reviewer. When the same study population was analyzed in more than one publication, the study's results were accounted for only once. When more than one index test (e.g., tests with different PCR primers) or reference standard was assessed, we extracted the data separately for each test and reference standard. Authors were contacted to complement the data.
Methodological quality was appraised by using the Quality Assessment of Diagnostic Accuracy Studies (QUADAS) checklist (72). We adapted the components of the list to our review (see Table S1 in the supplemental material). For each component, a score was given: 2 for yes (a low risk for bias), 1 for unknown or unclear, and 0 for no (a high risk for bias).
We listed the number of true positives, true negatives, false positives, and false negatives per study and the test and reference standard, from which pairs of sensitivity and specificity values were calculated. If a study reported results for more than one gene-specific primer, the results obtained with the ply gene were the default for the primary analysis. The results obtained with other primers were used for primer-specific analyses. Sensitivity and specificity were plotted on a receiver operating characteristic (ROC) plot. We used the hierarchical summary ROC (HSROC) method to calculate summary estimates for sensitivity, specificity, and positive and negative likelihood ratios with 95% confidence intervals (CIs) and a 95% prediction region (23, 24).
To assess heterogeneity, we computed diagnostic odds ratios (DORs) for each study (odds for a positive PCR result for a patient with IPD divided by the odds for a positive PCR result for a patient without IPD), with 0.5 being imputed for null values. We assessed the effects of several covariates on DORs through univariable meta-regression (36), including blood sample type (whole blood, serum, plasma, or other), PCR method (nested, real time, or other), the pneumococcal gene(s) used as the primer for the PCR, the study design (cohort versus case-control), age (studies recruiting adults versus studies recruiting children), the source of infection (studies assessing pneumonia only versus studies assessing any IPD), the study year, the DNA extraction method (methods with commercial kits or enzymatic or mechanical methods), the use of serial dilution, and the use of a protocol to circumvent the effects of inhibitors in blood. Only significant (P < 0.05) associations are reported. We intended to assess the effects of prior antibiotic treatment and colonization status, but the data in the primary studies were insufficient. Adjustment for multiple testing was performed by using a uninvariable permutation test (the adjusted P value is reported) (27). When the unadjusted meta-regression results indicated a significant effect of the variable assessed on the DORs, summary HSROC sensitivity and specificity values are presented by category. Analyses were performed with STATA/IC (version 10.1) software.
The search strategy identified 364 citations, of which 40 potentially relevant citations were selected for further evaluation (Fig. (Fig.1).1). Twenty-one studies were excluded (2, 6, 11-13, 15, 20, 26, 29, 31, 33, 38, 40, 47-49, 56, 60, 67, 69, 74) and 10 studies were added through searches of references and conference proceedings. A total of 29 studies published between 1993 and 2009 met the inclusion criteria (1, 4, 5, 9, 10, 14, 16, 25, 28, 30, 32, 37, 39, 41, 42, 45, 50, 52, 55, 57-59, 62, 64-66, 68, 70, 73).
The study characteristics are presented in Table S2 in the supplemental material. Ten studies assessed children alone, seven assessed adults alone, and the others did not state a specific age group and most probably assessed a mix that was representative of individuals with IPD. Fourteen studies targeted any IPD (corresponding to the inclusion criteria definitions), 13 evaluated pneumococcal pneumonia alone, and 2 evaluated meningitis alone. Only one study evaluated children with otitis media, in addition to IPD (14). Standard, real-time, and nested PCRs were each used in approximately one-third of the studies. The studies used whole blood, serum, plasma, or the buffy coat; several studies evaluated more than one type of blood specimen. Primers for the pneumococcal ply gene were used for PCR in most studies (n = 22), primers for the lytA gene were used in 10 studies, primers for the gene for PBP 2b were used in 2 studies, and primers for psaA and SPN9802 were used in 1 study each (several studies assessed more than one gene). Three studies performed PCR with fresh samples (32, 68, 73), while all others reporting on the methods used stored, frozen samples.
The study design and the QUADAS items used in the methodological assessment are detailed in Table S3 in the supplemental material. Seventeen studies were prospective cohort studies, in which the participants with signs suggestive of pneumococcal infections (e.g., community-acquired pneumonia) were evaluated for IPD. Three were considered nested case-control studies that assessed a prospective cohort of participants evaluated for IPD, and patients with proven IPD from that cohort served as cases, while a separate cohort of healthy participants served as controls. Nine studies were classified as case-control studies. These studies used stored clinical samples from patients with IPD (usually collected prospectively), and the results of those studies were compared with those of studies performed with samples from a cohort of healthy people or patients with infections caused by bacteria other than S. pneumoniae. Methodological variability between the studies was noted with regard to the recruitment of consecutive participants, the timing of PCR in relation to the time of collection of blood for culture, testing of all participants by the use of reference tests, and the description of the clinical information available at the time that the PCR was conducted. Only one study addressed withdrawals.
When “patients with disease” were defined as patients with pneumococcal bacteremia and “patients with no disease” were defined as healthy people or patients with bacteremia caused by other bacteria (22 studies), the summary estimates for sensitivity and specificity were 57.1% (95% CI, 45.7 to 67.8%) and 98.6% (95% CI, 96.4 to 99.5%), respectively (Fig. (Fig.2A).2A). Case-control studies showed higher sensitivities than cohort studies (unadjusted P = 0.03, adjusted P = 0.19), and pneumonia as the source of infection was associated with a slightly lower sensitivity than other types of IPD (unadjusted P = 0.04, adjusted P = 0.24) (Table (Table1).1). When “patients with disease” were as defined above but “patients with no disease” were defined as those patients presenting with symptoms and signs suggestive of IPD without pneumococcal bacteremia (26 studies), the summary values were 66.4% (95% CI, 55.9 to 75.6%) for sensitivity and 87.8% (95% CI, 79.5 to 93.1%) for specificity (Fig. (Fig.2B).2B). Only the type of PCR affected the DORs (unadjusted P = 0.03, adjusted P = 0.18), with the nested PCR having a specificity and a sensitivity lower than the specificities and sensitivities of the other PCR types (mainly real-time PCR) (Table (Table11).
Patients with a clinical syndrome suggestive of IPD and proof of pneumococcal infection by culture of a specimen from a sterile site (including blood) were considered “patients with disease,” and healthy people or patients with infections bacteriologically proven to be caused by other bacteria were considered “patients with no disease” (22 studies). The summary sensitivity value was 56.1% (95% CI, 47.1 to 64.7%), and the summary specificity value was 98.6% (95% CI, 96.2 to 99.4%) (Fig. (Fig.3).3). None of the covariates assessed significantly affected the DORs.
IPD proven by means of culture of specimens from sterile sites, serology, or antigen testing was considered “disease”; and individuals who were healthy or patients who did not have IPD constituted the control population (21 studies). The summary sensitivity value was 46.6% (95% CI, 34.8 to 58.8%), and the specificity was 98.8% (95% CI, 96.6 to 99.6%) (Fig. (Fig.4A).4A). Studies that assessed pneumococcal pneumonia had higher sensitivities (unadjusted P = 0.04, adjusted P = 0.19) than studies that included any IPD, while studies that included only children reported lower sensitivities than studies that included adults or mixed populations (unadjusted P = 0.03, adjusted P = 0.12) (Table (Table1).1). When IPD was similarly proven by any means but the controls were those with suspected IPD for whom there was no proof for IPD (22 studies), the summary sensitivity and specificity values were 51.6% (95% CI, 39.6 to 63.4%) and 91.1% (95% CI, 82.7 to 95.6%), respectively (Fig. (Fig.4B).4B). Studies that assessed children reported higher sensitivities and lower specificities than studies that assessed adult or mixed patient populations (unadjusted P = 0.002, adjusted P = 0.029). Nested PCR had a lower specificity than real-time PCR or other, more recent PCR methods (unadjusted P = 0.05, adjusted P = 0.33) (Table (Table11).
Six studies used a clinical definition of IPD, in addition to conventional diagnostic techniques. This included a clinical syndrome compatible with IPD after other pathogens had been ruled out, with or without a response to therapy. The summary sensitivity point was very low, 31.7% (95% CI, 16 to 53%), with the specificity being 98.3% (95% CI, 80.9 to 100%).
The predictive regions in all analyses were large, indicating our lack of ability to predict the “true” sensitivity and specificity (95% prediction regions in Fig. Fig.22 to to4).4). The overall predictive ability was lower (and the 95% predictive region was larger) when the control population consisted of patients with suspected IPD (Fig. (Fig.2B2B and and4B)4B) than when the control population consisted of healthy individuals or patients with other confirmed infections (Fig. (Fig.2A2A and and4A).4A). Narrower prediction regions could be obtained by using the covariates identified above as the underlying heterogeneity. For example, the pooled sensitivity and specificity of seven studies that used real-time or other recently developed PCR methods to diagnose any IPD for the comparison of patients with pneumococcal bacteremia and healthy people or patients with nonpneumococcal bacteremia were 50.9% (95% CI, 42.7 to 59.0%) and 97.3% (95% CI, 93.2 to 99.0%), respectively, with the prediction regions being narrower (see Fig. S1 in the supplemental material).
Finally, in cohort studies including patients (n = 25) with a clinical syndrome compatible with IPD but no proof for IPD at the baseline, the adjusted pooled positivity rate of PCR with blood compared with a final diagnosis of IPD (random-effects model) was 15% (95% CI, 14 to 16%; range, 1 to 85%). These studies included one study that was not included in previous analyses, since it reported only the PCR positivity rate (25). In the same set of studies, the pooled rate of bacteremia was 9% (95% CI, 9 to 10%; range, 4 to 75%). The rate of PCR positivity was higher than the rate of blood culture positivity in 16/25 studies. We contacted all authors of the studies included in the analysis to ask whether the PCR under study entered clinical use. One author replied that the test was integrated into the routine work flow locally and at a national level in Italy (5). Others did not reply or stated that the PCR was used for nonblood samples only.
We compiled all studies that assessed the diagnostic characteristics of PCR applied to blood samples for the diagnosis of IPD. When pneumococcal bacteremia or positive cultures of specimens from other sterile sites were used to define IPD in patients with sepsis, the sensitivity of PCR with blood averaged 66 to 57%. When lower degrees of proof for IPD were used as the reference standard, sensitivity values were lower, as expected; with samples from patients with disease proven by microbiological or serological methods, the sensitivity range was 47 to 52%, and with samples from patients with disease diagnosed by use of a clinical rule, the sensitivity decreased to 32%. Although the specificity was nearly 100% when healthy participants or patients infected by bacteria other than S. pneumoniae were considered, the more relevant population was that of patients suspected of having IPD (e.g., community-acquired pneumonia). Among those patients, the specificity rates were lower (88 to 92%). This might reflect the inadequacy of currently available modalities for the diagnosis of IPD and thus a potential added value of PCR or false-positive PCR results. In concordance with this, in cohort studies of patients suspected of having IPD, PCR was more commonly positive than blood cultures. All analyses were highly heterogeneous, as reflected in the wide confidence intervals and predictive regions (Fig. (Fig.22 to to4).4). The most consistent estimate for the sensitivity of PCR (50.9%; 95% CI, 42.7 to 59.0%) was obtained in studies that used real-time or other recent PCR methods to diagnose pneumococcal bacteremia.
We attempted to identify variables affecting the diagnostic accuracy of PCR with blood, other than the IPD case definition and reference standard used. As expected, nested PCR had a lower sensitivity and a lower specificity than real-time PCR or other advanced PCR methods. When pneumonia was the IPD under study, the sensitivity was higher than that for any IPD (including primary pneumococcal bacteremia and meningitis). The specificity observed among children with suspected IPD was lower than that observed among adults. Children not only are more likely to be colonized with pneumococci than adults (a true lower specificity) but also are more likely to have a pneumococcal disease with a compatible clinical syndrome (results reflecting a better performance for PCR than the reference standard). Case-control studies exaggerated the sensitivity values compared to those achieved in cohort studies, a finding that has previously been observed with other diagnostic tests (71). We did not find a consistent effect on diagnostic performance of the type of blood sample used, the sample processing methods (including DNA extraction methods, the use of serial dilutions, and the use of a protocol to circumvent inhibitors), the year of study performance, and the gene-specific primer used. Notably, these analyses depended on the level of reporting in the primary studies, which was highly variable.
Two important possible modifiers could not be assessed because data in the primary studies were not provided by patient subgroup: prior antibiotic treatment and colonization status. Prior antibiotic treatment decreased the sensitivity of PCR in four studies (1, 14, 32, 39), while four other studies claimed that it did not affect their sensitivity (28, 42, 70, 73). Dagan et al. showed that nasopharyngeal colonization by S. pneumoniae strongly affected the specificity, causing false-positive results among both healthy children and those with viral respiratory tract infections (14). Michelow et al. found no such association (42). Several other variables potentially affecting the diagnostic accuracy could not be assessed, given the various levels of reporting of the methods in the primary studies. These included the volume used for testing (a low concentration of bacteria may necessitate the use of a certain minimum volume), the techniques and the timing of the collection of blood for PCR in relation to the timing of the collection of blood for culture, and the use of a single or a multiplex PCR. Finally, nearly all studies used stored frozen blood samples; thus, we could not assess whether the use of fresh samples improved the sensitivity.
Invasive pneumococcal infections may be lethal if they are not treated. Their main differential diagnosis is viral infection. Thus, a rapid diagnostic test that may be used to rule out IPD would be helpful. Direct PCR with blood samples cannot yet qualify for this role. The negative likelihood ratios in the current analysis ranged from 0.38 to 0.54, values that are unhelpful in clinical practice. The main caveat refers to our specificity estimates, since the currently available diagnostic methods used as the reference standards in the studies analyzed do not diagnose all cases of IPD.
In summary, PCR performed directly with blood samples has a poor sensitivity compared to the sensitivities of current culture-based methods and a poor specificity for patients with sepsis. The results of individual studies are highly heterogeneous, and we could not define the clinical scenario or methods that will result in uniformly better test characteristics. Given the current data, PCR directly with blood samples cannot be considered for use in clinical practice. Similar analyses should be conducted to assess the performance of PCR applied to blood samples for the detection of other bacterial infections, and these results should be considered prior to the introduction of novel costly technologies (53).
We highlight the methodological difficulty with assessing the accuracy of molecular test methods. Defining the reference standard is complex because the currently available culture-based diagnostics have limited sensitivities and negative predictive values. An optimal diagnostic study would include a cohort of patients suspected of having the targeted infection, should monitor the patients prospectively to define outcomes, and should use a clinical “expert” definition for the reference standard, in addition to definitive proof. While the highest degree of proof is a positive culture result with specimens from sterile sites, the lowest degree of proof for “infection” (and the highest degree of proof for “no infection”) would be an accepted clinical definition of infection that considers the clinical presentation, follow-up, and the response to treatment. The development of guidelines for a uniform design and a uniform means of reporting for studies assessing the diagnostic accuracies of molecular methods for the diagnosis of infections is warranted.
T.A. was supported by a research grant (for optimal antibiotic treatment of moderate to severe bacterial infections: integration of PCR technology and medical informatics/artificial intelligence) from the Rothschild-Caesarea Foundation.
None of us has any conflicts of interest to declare.
Published ahead of print on 9 December 2009.
†Supplemental material for this article may be found at http://jcm.asm.org/.