Within the last decade, three models to estimate an individual's absolute LC risk were developed: the Bach (Bach et al, 2003), Spitz (Spitz et al, 2007), and the Liverpool Lung Project (LLP) models (Cassidy et al, 2008). All the three models share risk factors (such as smoking duration and occupational exposure to asbestos); however, differences arise, with the inclusion of lung-related comorbidities or family history information. These models have not previously been compared in an independent data set in terms of discriminatory power, accuracy, and clinical utility. Such a comparison, to evaluate whether these published risk models have similar discriminatory power for a given population of individuals, is important, given the potential application of risk-prediction models to strategies for primary and secondary preventions.

In this study, we used each of these models to estimate 5-year absolute LC risks for an independent population of LC patients and healthy controls. We compared the discriminatory power of these three models by calculating the area under the curve (AUC) of the receiver operator characteristic (ROC) curve for each 5-year absolute risk estimate. We evaluated the accuracy and compared the positive predictive value (PPV; the probability of accurately categorising an affected participant) and the negative predictive value (NPV; the probability of accurately categorising an unaffected participant) among the three risk models; we also evaluated the clinical utility of each.

Inclusion of risk factors in the three LC risk models is summarised in Table 1. Smokers were defined as those who had smoked >400 cigarettes in their lifetime; former smokers were those who had quit smoking at least 1 year before the cancer diagnosis (patients) or the interview (controls). Smoking duration was determined by subtracting the age at which the participant had started smoking from either the age at which the participant had quit smoking (former smokers) or the participant's current age (current smokers). Pack-years were calculated by multiplying the smoking duration (in years) by the number of cigarettes smoked per day and then dividing by 20. Time of smoking cessation for former smokers was determined by subtracting the age at which the participant had quit smoking from the participant's current age.

Participants were classified as positive for asbestos exposure if they had been directly exposed for at least 8h per week for a year or if they were employed in an asbestos-related industry (according to the Standard Industrial Classification Manual (1972) and/or the Dictionary of Occupational Titles (1991)). Exposure to wood dusts (including sawdust or sanding dust) for at least 8h per week for a year was self-reported, or for a family history of any cancer if at least two first-degree relatives had cancer. Participants were classified as positive for a family history of any smoking-related cancer if at least one first-degree relative had had cancer at some point in his or her life. Participants were also classified by self-reported physician-diagnosed emphysema or hay fever at any time before study entry (Spitz) or by physician-diagnosed pneumonia at least 2 years before entry (LLP).

Any study participant with missing data for any of the risk factors for any model was excluded from analysis. As all three models were developed using data obtained from White participants, we only included individuals who self-reported being non-Hispanic White. For the comparison of discriminatory power between the LLP and Bach models and the Spitz and Bach models only ever smokers (total of 1066 LC and 677 controls) were used as the Bach model was developed only for ever smokers (Bach et al, 2003). In the comparison between the LLP and Spitz models, never, former, and current smokers were included (total of 1121 LC and 1024 controls).

The institutional review boards at the M. D. Anderson Cancer Center, MGH, and the Harvard School of Public Health approved this study.

We calculated the 5-year absolute risk of LC for the three models, using MatLab software (The MathWorks Inc., Natick, MA, USA). For the Bach model, they were obtained by running 1-year incidence and mortality models recursively five times, with each individual contributing to the predicted risk for 5 years. For the Spitz and LLP models, they were estimated by combining the risk of cancer from the relative risk model with age- and gender-specific LC incidence rates.

Details regarding the exact calculation of risk for each model were given in the original paper (Bach et al, 2003; Spitz et al, 2007; Cassidy et al, 2008). With the LLP model, the α-value used to calculate the 5-year absolute risk was adjusted for the US LC incidence rate (Appendix Table A1). For each participant, we had three estimates of absolute risk for LC (one from each model). For each model, we used NCSS statistical software (NCSS, Kaysville, UT, USA) to calculate the specificity and sensitivity required to construct ROC curves and estimate AUC (binomial method) and the 95% confidence interval (95% CI) for each of the three models. We also calculated the AUC after stratification of participants by sex and age (<50 vs 50 years). We then conducted pairwise comparisons of the AUCs of the three models using the method described in the NCSS package (Hanley and McNeil, 1983), the test statistic for comparing two ROC curves being given by where AUC_i is the area under the ROC curve from the ith model (i=1,2), s.e._i the s.e. of AUC_i, and r the correlation between AUC₁ and AUC₂ (Tyrer et al, 2004). The test statistic z follows a standard normal distribution; and values of z>z_1−α are interpreted as evidence that AUC₁ is significantly greater than AUC₂ for the given α-level.

From these risk factors, a relative risk was calculated and combined with age- and gender-specific incidence rates from x (SEER) (SEER, 2005), and all-cause mortality (excluding LC) rates from CDC (Centers for Disease Control) to estimate the absolute risk of LC (National Center for Health Statistics, 2003) (Appendix Table A2). As most of the absolute risk calculations involve pairwise comparisons from three different LC models, the Bonferroni correction was taken into account to adjust for any multiple comparisons issues.

The discriminatory power for the three models, overall and stratified by smoking, age, and sex, are summarised in Table 3, the AUCs being 0.69 for the Spitz (95% CI=0.66–0.71) and LLP (95% CI=0.67–0.71) models and 0.66 (95% CI=0.64–0.69) for the Bach model. The differences in discriminatory power between the LLP and Bach models were significant (P=0.023), and the differences between the Spitz and Bach models reached borderline significance (P=0.072). Among former smokers, the discriminatory power was 0.70 (95% CI=0.67–0.73) for the Spitz and LLP models and 0.65 (95% CI=0.62–0.68) for the Bach model. Among current smokers, the discriminatory power was 0.68 (95% CI=0.64–0.72) for the Spitz model, 0.65 (95% CI=0.60–0.69) for the Bach model, and 0.66 (95% CI=0.62–0.70) for the LLP model. Among former smokers, the Bach model was outperformed by both the LLP (P=0.002) and the Spitz (P=0.008) models, whereas among current smokers, only the Spitz model significantly outperformed the Bach model (P=0.024). When incorporating never smokers for testing discriminatory power, the LLP model (AUC=0.72, 95% CI=0.70–0.74) outperformed the Spitz model (AUC=0.68, 95% CI=0.66–0.71) significantly (P=0.001).

We also tested the discriminatory power of all models when participants were stratified by age and sex (Table 3) and for women over the age of 50 years, observed significant differences in discriminatory power between the Spitz and Bach models, and the LLP and Bach models.

Table 4 summarises the NPV and PPV results for each. Overall, the three models had reasonable PPV levels (all >70%); the Spitz model had a significantly higher PPV (88.2%) than those of the LLP (75.9% P<0.001) and the Bach (80.9% P=0.009) models. Among former smokers, the Spitz model had significantly higher PPV (85.5%) than did the LLP model (72.6% P<0.001) but not significantly higher PPV than the Bach model (83.6% P=0.851). However, among current smokers, the Spitz model had higher PPV (91.9%) than did the Bach (80.4% P=0.002) and the LLP (80.9% P<0.001) models. The overall NPV for each of the three models were lower than the PPV (range=45.0–56.0%), with the LLP model having a substantially better probability of accurately categorising an unaffected participant. The LLP model was also significantly better for the NPV among former smokers, but both the Spitz and Bach models were competitive with the LLP model in calculating the NPV among current smokers.

Each model included some form of tobacco exposure. In the Bach model, the variables – duration of smoking (in years) and number of cigarettes smoked per day – are included for both former and current smokers. In the Spitz model (controls matched to cases on smoking status), the duration of smoking and numbers of cigarettes smoked per day are combined into pack-years for current smokers only and into age at smoking cessation for former smokers. The Bach and LLP models do not include a smoking cessation variable. The Bach model included smokers aged 50–75 years who are/were heavy smokers (10–60 cigarettes per day for 25–60 years) and who had quit no more than 20 years previously (Bach et al, 2003).

In terms of clinical utility, the Spitz and Bach models performed reasonably well in identifying LC patients at defined levels of risk while limiting the number of false-positive results. However, the LLP model was much better at identifying individuals with LC but also had a much higher false-positive rate than the Spitz and Bach models had. This could be attributed to the importance of smoking in the LLP model. The Spitz model's relatively low recognition of cancer patients with a >2.5% absolute LC risk could be caused by smoking being a matching variable in the model rather than a risk factor. The overall high (>75%) PPVs for the three models indicate that they can identify high-risk individuals; however, the overall relatively low NPVs (between 45 and 56%) indicate that many low-risk individuals would be identified as well. The scaled rectangle diagrams illustrate more clearly the modest discriminatory performances of the Spitz, Bach, and LLP models and provide a sobering message about LC risk prediction. To substantially improve LC risk discriminatory power for individual patients, we need to identify a risk factor (other than smoking habits) that has a different distribution in LC patients from those who will not develop it; to date, there is no evidence for such a factor. High expectations have been pinned on genome-wide association studies, which have successfully identified hundreds of common genetic variants that are strongly associated with the risk of more than 40 diseases, including LC (Kraft and Hunter, 2009). However, a strong association does not necessarily guarantee good classification or discriminatory ability (Jakobsdottir et al, 2009). It was recently shown that on average, 80 common variants with odds ratios of 1.25 each were required to develop a model useful for the identification of high-risk individuals (AUC>0.80) for genetic profiling studies (Janssens et al, 2006).

Our study had some limitations. The most important limitation is that the study design is a case–control study, which could lead to some recall bias with the self-reported variables, such as smoking and environmental tobacco smoke exposure (Asomaning et al, 2008). However, in this study, controls were recruited from family and friends of those being treated for LC at the MGH, so that exposures for the self-reported variables would be similar or non-differential, among cases and controls, which would limit recall bias (Miller et al, 2003). With non-differential biases, AUC results will regress to the null, so it is possible that the AUC results are conservative instead of overstated (Greenland and Lash, 2008).

Other minor limitations include that the risk-prediction models compared in our study were developed in Caucasian populations, so the validation was also restricted to Caucasians, and thus, the models may not be applicable to other racial or ethnic groups. In addition, for most of the analysis, we only included ever-smoker cases and controls in our analysis, and thus, the enormous contribution of smoking to LC risk was effectively underestimated. The ultimate test of a model's application is its accurate prediction of risk in an independent data set. However, direct comparison of risk models is complicated by the fact that few studies have population samples that are large enough and diverse enough in age and risk factor backgrounds (Cassidy et al, 2007). Thus, to avoid possible information bias, it was imperative for our analysis to select only patients and controls from those who had complete information relating to risk-model covariates.