PMCCPMCCPMCC

Search tips
Search criteria 

Advanced

 
Logo of etepidemBioMed CentralBiomed Central Web Sitesearchsubmit a manuscriptregisterthis articleEmerging Themes in EpidemiologyJournal Front Page
 
Emerg Themes Epidemiol. 2010; 7: 2.
Published online 2010 May 14. doi:  10.1186/1742-7622-7-2
PMCID: PMC2885382

Testing bias in clinical databases: methodological considerations

Abstract

Background

Laboratory testing in clinical practice is never a random process. In this study we evaluated testing bias for neutrophil counts in clinical practice by using results from requested and non-requested hematological blood tests.

Methods

This study was conducted using data from the Utrecht Patient Oriented Database. This clinical database is unique, as it contains physician requested data, but also data that are not requested by the physician, but measured as result of requesting other hematological parameters. We identified adult patients, hospitalized in 2005 with at least two blood tests during admission, where requests for general blood profiles and specifically for neutrophil counts were contrasted in scenario analyses. Possible effect modifiers were diagnosis and glucocorticoid use.

Results

A total of 567 patients with requested neutrophil counts and 1,439 patients with non-requested neutrophil counts were analyzed. The absolute neutrophil count at admission differed with a mean of 7.4 × 109/l for requested counts and 8.3 × 109/l for non-requested counts (p-value < 0.001). This difference could be explained for 83.2% by the occurrence of cardiovascular disease as underlying disease and for 4.5% by glucocorticoid use.

Conclusion

Requests for neutrophil counts in clinical databases are associated with underlying disease and with cardiovascular disease in particular. The results from our study show the importance of evaluating testing bias in epidemiological studies obtaining data from clinical databases.

Background

In recent years, large health care databases are increasingly used and provide important tools in epidemiological research [1,2]. Advantages are that large amounts of clinical data are available at relatively low cost, and that these databases usually reflect daily practice [3,4]. However, in contrast to randomized clinical trials, where data collection is well-controlled, bias should always be considered when using routinely collected data in automated databases and methodological issues should be taken into account [3,5-7].

Laboratory testing in clinical practice is never a random process, as the physician has reasons to perform a test. Physicians selectively request tests for patients with a high probability of abnormalities and less frequently for patients with a low probability, because of patient burden and costs [8]. Such selective processes might induce testing bias in clinical database studies. There are several strategies to minimize testing bias, including selection of proper patient populations, measuring outcomes for all study participants, blind testing, or using imputation techniques to deal with missing data [8-10], but these techniques do not provide insight into size and direction of testing bias.

One example where testing bias might occur is in physicians' requests of blood tests. Neutrophil counts in peripheral blood are considered a useful biomarker for disease severity in many conditions [11-14]. However, testing bias might occur because of underlying disease or medication use, as neutrophil counts differ in several diseases and clinical observations have shown that patients using glucocorticoids often have higher neutrophil counts. Requesting neutrophil counts specifically for certain diseases or for glucocorticoid users might cause testing bias in clinical databases. The aim of this study was to evaluate testing bias for neutrophil counts in clinical practice by using results from requested and non-requested hematological blood tests.

Methods

Setting

This study was conducted using data from the Utrecht Patient Oriented Database (UPOD). UPOD is an infrastructure of relational databases comprising administrative data on patient characteristics, laboratory test results, medication orders, discharge diagnoses and medical procedures for all patients treated at the University Medical Center (UMC) Utrecht, a 1,042-bed tertiary teaching hospital in the center of the Netherlands. Each year, approximately 165,000 patients are treated during more than 28,000 hospitalizations, 15,000 day-care treatments, and 334,000 outpatient visits. UPOD data acquisition and data management is in accordance with current Dutch privacy and ethical regulations. A more complete description of UPOD has been published elsewhere [15].

UPOD is a unique clinical database as it contains results of hematological blood tests measured with Cell-Dyn Sapphire automated blood cell analyzers (Abbott Diagnostics, St. Clara, California, USA) [15]. A feature of this analyzer is that it measures all hematological parameters irrespective of whether these are requested or not [15]. The non-requested parameters are measured because one hematological test is technically linked to the other hematological tests and conducted automatically when one of these tests is requested. In other words, UPOD contains requested (Figure (Figure1)1) and non-requested test results (Figure (Figure2).2). Although non-requested neutrophil counts are not reported to the clinician, these neutrophil counts are collected in UPOD.

Figure 1
Illustration of scenarios. Scenario 1 represents the situation in a typical clinical database where all neutrophil counts were requested by physicians.
Figure 2
Is unique for UPOD as this includes non-requested neutrophil counts. These non-requested neutrophil counts are measured because this test is conducted automatically when one hematological test, for example hemoglobin, is requested.

Scenarios

By comparing the measured hematological parameters with the routine hospital laboratory reporting system, which reports laboratory results to physicians, neutrophil counts can be categorized as requested or non-requested. Neutrophil counts appearing in the laboratory reporting system were categorized as requested; other neutrophil counts were categorized as non-requested. Using these data, we conducted two scenario analyses. Scenario 1 reflects the situation as in a typical clinical database, where all blood tests were requested. With scenario 2 we were able to study testing bias by including non-requested blood tests in our analysis.

Study population

The source population comprised 3,467 adult (18 years or older) users and non-users of glucocorticoids who were hospitalized in the UMC Utrecht in 2005 and had at least two hematological blood tests, where these tests should cover at least a one-day period. For each glucocorticoid user, the first blood test during admission and the last blood measurement during in-hospital glucocorticoid use were selected for analysis. Up to four unexposed patients were sampled to each glucocorticoid user according to calendar time (with a maximum of 15 days before or after the test date of the user), neutrophil count at time of admission (max 2 × 109 neutrophils/l around the neutrophil count of the user) and days between the two blood samples (max two days around the number of days for the glucocorticoid user). According to our laboratory normal reference range for neutrophils (1.6-8.3 × 109/l), there is large inter-individual variation in the absolute neutrophil count. Using two blood tests, we were able to study testing bias in both blood tests separately, but also in the change in neutrophil count during hospitalization for each patient. Within the source population, we contrasted patients with both blood tests requested and with both blood tests non-requested, one at time of admission and one at the end of hospitalization. For all participants the discharge diagnosis was defined according to the ICD-9-CM code [16].

Data analysis

Student t-tests, Mann-Whitney tests, and chi-square tests were used to test for differences between groups, as appropriate. Confounding was studied using logistic regression. The absolute neutrophil count was categorized into tertiles to obtain three equally-sized groups. These three groups were defined as an increase, decrease or no change in neutrophil count where no change was the comparator group. Potential confounding factors were included in a multivariable logistic model in sequence with decreasing confounding strength. Potential confounders that were studied were age, gender, number of days between blood samples, length of hospitalization, death during hospitalization, and diagnosis. All variables that changed the regression coefficient for glucocorticoid use by less than ten percent were excluded from the model. Of these potential confounders, only diagnosis had a substantial effect on the comparison between scenarios. Glucocorticoid use was studied because of the ongoing discussion about the effect of glucocorticoids on the neutrophil count [12,17,18]. Subsequently, linear regression analysis was used to estimate the proportion of bias associated with diagnostic subgroups and glucocorticoid use. The beta-coefficient for the contrasted scenarios was calculated for all patients in the study population as well as for only patients exposed to one factor (for example a diagnostic subgroup or glucocorticoid use). The proportion of bias explained by one factor was calculated as the weighted fraction of beta-coefficients. All analyses were conducted using SPSS for Windows, version 14.0 (SPSS Inc., Chicago, Illinois, USA).

Results

A total of 567 patients with requests for the absolute neutrophil count (scenario 1) and 1,439 patients with non-requested neutrophil counts (scenario 2) were identified. It appeared that the absolute neutrophil count was most frequently requested in the context of a leukocyte differential request, which includes the absolute counts of neutrophils, eosinophils, lymphocytes, monocytes, and basophils (99.8% of all neutrophil count requests). Of patients with requested neutrophil counts, there was also a hemoglobin request for 97.2% of patients. For patients with non-requested neutrophil counts, 96.1% of the requests were for hemoglobin. Hemoglobin values were lower when requested compared with non-requested hemoglobin values (Table (Table11).

Table 1
Baseline characteristics of the study population*

For the first blood test, lower neutrophil counts were found for patients with requested neutrophil counts compared with non-requested neutrophil counts (Table (Table1).1). Comparable neutrophil counts were found in the second blood test. For both blood tests, there were more patients with neutropenia and fewer patients with neutrophilia for requested neutrophil counts. Studying the change in the absolute neutrophil count during hospitalization for each patient, patients with non-requested neutrophil counts had a mean decrease of 0.50 × 109 neutrophils/l compared with a slight increase of 0.14 × 109 neutrophils/l for patients with requested neutrophil counts (p-value 0.008, Table Table1,1, Figure Figure33).

Figure 3
Absolute neutrophil counts and change in neutrophil count. Distribution of the absolute neutrophil count of the first (A) and second blood test (B) and the change in neutrophil count for each patient (C) for requested neutrophil counts (scenario 1) and ...

Overall, the main diagnostic subgroups were cardiovascular disease (28.9%), neoplasms (14.5%), and respiratory disease (4.5%). Requests for neutrophil counts were more often conducted for patients suffering from cardiovascular or respiratory diseases, whereas diagnoses for non-requested neutrophil counts were much more diffuse with multiple diagnoses (Table (Table1).1). There were no differences in absolute neutrophil count or change in neutrophil counts among patients with neoplasms and respiratory disease (Table (Table2).2). However, among patients with cardiovascular disease there was a lower absolute neutrophil count in the first blood test for requested counts compared with non-requested neutrophil counts. Excluding cardiovascular patients from analysis, the absolute neutrophil counts in the first blood test were equal with 8.1 × 109/l in both scenarios (Figure (Figure4).4). The difference in absolute neutrophil count between scenarios in the first blood test could be explained for 83.2% by cardiovascular disease (p-value for effect modification 0.002). Incorporating glucocorticoid use in the linear regression model showed that diagnosis was far more important than glucocorticoid use (p-value for effect modification 0.240). Taking diagnosis into account, 4.5% of the difference in absolute neutrophil count between scenarios in the first blood test could be explained by glucocorticoid use.

Table 2
Neutrophil counts for the main diagnostic subgroups
Figure 4
The effect of cardiovascular disease on the absolute neutrophil count of the first blood test. The higher neutrophil count in scenario 2 is explained by a high neutrophil count among cardiovascular patients in scenario 2. Excluding cardiovascular disease ...

With respect to the absolute neutrophil count in the second blood test, there were no differences between the scenarios, either in the overall analysis or in the main diagnostic subgroups. An increase in neutrophil count of 2.1 × 109/l was shown for requested neutrophil counts in cardiovascular patients, whereas non-requested counts revealed a decrease of 0.4 × 109 neutrophils/l (Table (Table2).2). Excluding cardiovascular disease from analysis, the change in neutrophil count was comparable in both scenarios with a mean decrease of 0.9 × 109/l for each patient with requested neutrophil counts and a mean decrease of 0.5 × 109/l for each patient including only non-requested neutrophil counts (p-value = 0.211).

Discussion

In this study, we used UPOD to study bias in neutrophil testing, as this database contains both requested and non-requested neutrophil counts. Among tests for which neutrophil counts were requested, hemoglobin was also requested in 97%. For non-requested neutrophil test results, 96% were generated by hemoglobin requests. Therefore, hemoglobin requests approximate random testing and can be used as comparator group. Absolute neutrophil counts differed for requested tests (scenario 1) compared with non-requested tests (scenario 2), which leads to the conclusion that testing bias was found in this study.

The bias in absolute neutrophil count in the first blood test could be explained for 83.2% by cardiovascular disease. This finding could reflect the role of neutrophils in cardiovascular disease [13,19,20]. After excluding cardiovascular disease from analysis, there were no differences in absolute neutrophil count or change in neutrophil count for each patient. This could be explained by the fact that the absolute neutrophil count was mainly requested in the context of a leukocyte differential count. Similar findings were observed for the change in neutrophil count. For the second neutrophil test, no differences were found between the scenarios. The second neutrophil test was at the end of hospitalization. The first neutrophil test, at time of admission, is more informative because the patients are likely to be severely ill at that point. At time of the second blood test the difference in absolute neutrophil count has evened out, as patients are healthier at the end of hospitalization.

The results of this study are in accordance with other studies finding that data are not missing at random [21,22]. Further research is needed to study the clinical relevance of the bias found in this study. Distributions of diagnostic subgroups and testing guidelines might vary between health care institutions. As a consequence, generalizability of clinical implications, like the association with cardiovascular disease as an example in this study, might be limited. However, testing bias is an issue in all centers and should be evaluated to be able to adjust for this bias. Using laboratory tests, a random tested parameter, like hemoglobin testing in this study, could serve as comparator group to study testing bias. With knowledge about the size and direction of testing bias, strategies such as imputation techniques [8,21] could adjust for this bias in order to obtain an unbiased risk estimate in epidemiological studies.

With development of automated machines for routine analysis, more parameters are measured than requested. When these non-requested parameters are collected, testing randomness is introduced. UPOD contains requested neutrophil counts and non-requested neutrophil counts, as well as other non-requested hematological blood tests. Therefore, the database is especially suitable to study and adjust for testing bias in clinical research questions. Conducting studies with laboratory markers in UPOD, correction factors for requested testing can be added to the statistical model to minimize testing bias in order to obtain an unbiased risk estimate.

A classic example of testing bias is the association between thrombosis and use of oral contraceptives. Many studies state traditionally that the size of this association is overestimated because of diagnostic suspicion bias and referral bias, both types of testing bias [23,24]. However, a case-control study with the same referral and diagnostic strategies for cases and controls, showed that neither type of bias played a major role in previous studies, and that the risk of thrombosis while using oral contraceptives is not solely due to bias [9]. This example and the results from our study show the importance of evaluating testing bias in epidemiological studies obtaining data from clinical databases.

List of abbreviations

UMC: University Medical Center; UPOD: Utrecht Patient Oriented Database.

Competing interests

None of the authors report a personal conflict of interest. The division of Pharmacoepidemiology & Pharmacotherapy employing authors KJV, HGML, PCS, and WWS has received unrestricted funding for pharmacoepidemiological research from GlaxoSmithKline, Novo Nordisk, the private-public funded Top Institute Pharma (http://www.tipharma.nl, includes co-funding from universities, government, and industry), the Dutch Medicines Evaluation Board, and the Dutch Ministry of Health.

Authors' contributions

KJV participated in designing the study, performed the analyses and drafted the final manuscript. HGML, PCS, RCS and WWS participated in the design of the study and helped in the interpretation of the data. All authors critically reviewed draft versions of the manuscript. All authors read and approved the final manuscript.

Acknowledgements

The authors are grateful to Maarten ten Berg for creating the Figure Figure11 and and22 drafts and to Svetlana Belitser for statistical advice.

References

  • Lawrenson R, Williams T, Farmer R. Clinical information for research; the use of general practice databases. J Public Health Med. 1999;21:299–304. doi: 10.1093/pubmed/21.3.299. [PubMed] [Cross Ref]
  • Flouris AD, Duffy J. Applications of artificial intelligence systems in the analysis of epidemiological data. Eur J Epidemiol. 2006;21:167–70. doi: 10.1007/s10654-006-0005-y. [PubMed] [Cross Ref]
  • Schneeweiss S, Avorn J. A review of uses of health care utilization databases for epidemiologic research on therapeutics. J Clin Epidemiol. 2005;58:323–37. doi: 10.1016/j.jclinepi.2004.10.012. [PubMed] [Cross Ref]
  • Hartz A, Bentler S, Charlton M, Lanska D, Butani Y, Soomro GM, Benson K. Assessing observational studies of medical treatments. Emerg Themes Epidemiol. 2005;2:8. doi: 10.1186/1742-7622-2-8. [PMC free article] [PubMed] [Cross Ref]
  • Strom BL. Methodologic challenges to studying patient safety and comparative effectiveness. Med Care. 2007;45:S13–5. doi: 10.1097/MLR.0b013e318041f752. [PubMed] [Cross Ref]
  • De Luise C, Lanes SF, Jacobsen J, Pedersen L, Sorensen HT. Cardiovascular and respiratory hospitalizations and mortality among users of tiotropium in Denmark. Eur J Epidemiol. 2007;22:267–72. doi: 10.1007/s10654-007-9106-5. [PubMed] [Cross Ref]
  • Rosso S, Ricceri F, Terracini L, Zanetti R. Methodological issues in estimating survival in patients with multiple primary cancers: an application to women with breast cancer as a first tumour. Emerg Themes Epidemiol. 2009;6:2. doi: 10.1186/1742-7622-6-2. [PMC free article] [PubMed] [Cross Ref]
  • Oostenbrink R, Moons KG, Bleeker SE, Moll HA, Grobbee DE. Diagnostic research on routine care data: prospects and problems. J Clin Epidemiol. 2003;56:501–6. doi: 10.1016/S0895-4356(03)00080-5. [PubMed] [Cross Ref]
  • Bloemenkamp KW, Rosendaal FR, Buller HR, Helmerhorst FM, Colly LP, Vandenbroucke JP. Risk of venous thrombosis with use of current low-dose oral contraceptives is not explained by diagnostic suspicion and referral bias. Arch Intern Med. 1999;159:65–70. doi: 10.1001/archinte.159.1.65. [PubMed] [Cross Ref]
  • Simons P, Algra A, Laak M Van de, Grobbee D, Graaf Y Van der. Second manifestations of ARTerial disease (SMART) study: rationale and design. Eur J Epidemiol. 1999;15:773–81. doi: 10.1023/A:1007621514757. [PubMed] [Cross Ref]
  • Wilson CB, Jones PW, O'Leary CJ, Hansell DM, Dowling RB, Cole PJ, Wilson R. Systemic markers of inflammation in stable bronchiectasis. Eur Respir J. 1998;12:820–4. doi: 10.1183/09031936.98.12040820. [PubMed] [Cross Ref]
  • Jatakanon A, Uasuf C, Maziak W, Lim S, Chung K, Barnes P. Neutrophilic inflammation in severe persistent asthma. Am J Respir Crit Care Med. 1999;160:1532–9. [PubMed]
  • Haumer M, Amighi J, Exner M, Mlekusch W, Sabeti S, Schlager O, Schwarzinger I, Wagner O, Minar E, Schillinger M. Association of neutrophils and future cardiovascular events in patients with peripheral artery disease. J Vasc Surg. 2005;41:610–7. doi: 10.1016/j.jvs.2005.01.013. [PubMed] [Cross Ref]
  • Velthove KJ, Van Solinge WW, Lammers JW, Leufkens HG, Schweizer RC, Bracke M. Hematocytometry analysis as discriminative marker for asthma phenotypes. Clin Chem Lab Med. 2009;47:573–8. doi: 10.1515/CCLM.2009.139. [PubMed] [Cross Ref]
  • Ten Berg MJ, Huisman A, Bemt PM Van den, Schobben AF, Egberts AC, Van Solinge WW. Linking laboratory and medication data: new opportunities for pharmacoepidemiological research. Clin Chem Lab Med. 2007;45:13–9. doi: 10.1515/CCLM.2007.009. [PubMed] [Cross Ref]
  • Anonymous, International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) National Center for Health Statistics; 2006.
  • Wenzel SE. Asthma: defining of the persistent adult phenotypes. Lancet. 2006;368:804–13. doi: 10.1016/S0140-6736(06)69290-8. [PubMed] [Cross Ref]
  • Chung K, Godard P, Adelroth E, Ayres J, Barnes N, Barnes P, Bel E, Burney P, Chanez P, Connett G, Corrigan C, De Blic J, Fabbri L, Holgate S, Ind P, Joos G, Kerstjens H, Leuenberger P, Lofdahl C-G, McKenzie S, Magnussen H, Postma D, Saetta M, Salmeron S, Silverman M, Sterk P. Difficult/therapy-resistant asthma. The need for an integrated approach to define clinical phenotypes, evaluate risk factors, understand pathophysiology and find novel therapies. ERS Task Force on Difficult/Therapy-resistant asthma. Eur Respir J. 1999;13:1198–208. [PubMed]
  • Pinto EM, Huppert FA, Morgan K, Mrc C, Brayne C. Neutrophil counts, monocyte counts and cardiovascular disease in the elderly. Exp Gerontol. 2004;39:615–9. doi: 10.1016/j.exger.2003.12.011. [PubMed] [Cross Ref]
  • Mehta J, Dinerman J, Mehta P, Saldeen TG, Lawson D, Donnelly WH, Wallin R. Neutrophil function in ischemic heart disease. Circulation. 1989;79:549–56. [PubMed]
  • Delaney JA, Moodie EE, Suissa S. Validating the effects of drug treatment on blood pressure in the General Practice Research Database. Pharmacoepidemiol Drug Saf. 2008;17:535–45. doi: 10.1002/pds.1553. [PubMed] [Cross Ref]
  • Lewis JD, Brensinger C. Agreement between GPRD smoking data: a survey of general practitioners and a population-based survey. Pharmacoepidemiol Drug Saf. 2004;13:437–41. doi: 10.1002/pds.902. [PubMed] [Cross Ref]
  • Vessey MP, Doll R. Investigation of relation between use of oral contraceptives and thromboembolic disease. BMJ. 1968;2:199–205. doi: 10.1136/bmj.2.5599.199. [PMC free article] [PubMed] [Cross Ref]
  • Sackett DL. Bias in analytic research. J Chronic Dis. 1979;32:51–63. doi: 10.1016/0021-9681(79)90012-2. [PubMed] [Cross Ref]

Articles from Emerging Themes in Epidemiology are provided here courtesy of BioMed Central