Search tips
Search criteria 


Logo of nictobLink to Publisher's site
Nicotine Tob Res. 2012 December; 14(12): 1483–1487.
Published online 2012 February 7. doi:  10.1093/ntr/nts002
PMCID: PMC3509007

Systematic Biases in Cross-sectional Community Studies may Underestimate the Effectiveness of Stop-Smoking Medications

Ron Borland, B.Sc.,corresponding author1 Timea R. Partos,1 and K. Michael Cummings, Ph.D., MPH2



Randomized, controlled trials typically indicate stop-smoking medications (SSMs: e.g., Varenicline, Bupropion, and over-the-counter nicotine replacement therapies) to be effective, whereas cross-sectional community-based studies have found them to be less effective, ineffective, or even associated with higher risk of relapse. Consequently, some critics have suggested SSMs have no useful applications in “real-world” settings. This discrepancy may, however, be due to systematic biases affecting cross-sectional survey outcomes. Namely, failed quit attempts where SSMs were used may be better recalled than failed unassisted attempts. Moreover, smokers who choose to quit using SSMs may be more addicted and thus less likely to succeed. Either of these factors would lead to an over-representation of failed quit attempts among SSM users in cross-sectional surveys even if there were real benefits.


We report on data from the International Tobacco Control 4-country cohort study to examine the relationship between SSM use, level of nicotine addiction, and the reported date since the start of participants’ (N = 1,101) most recent quit attempt.


The last quit attempt was reported to have begun longer ago among participants who used SSMs than those who did not. Scores on the Heaviness of Smoking Index, measuring addiction severity, were also higher among SSM users, with no interactions.


Better recall of quit attempts and stronger addiction to nicotine are two characteristics found more often among smokers using SSMs compared with self-quitters, which could potentially bias the assessed effects of SSMs on cessation outcomes in cross-sectional surveys.


Stop-smoking medications (SSMs) are becoming increasingly available to smokers, particularly those residing in affluent Western nations. SSMs include prescription-only (Rx) medications such as Varenicline and Bupropion and over-the-counter nicotine replacement therapies (NRT) in the form of patches, lozenges, tablets, gum, and inhalers. Randomized, controlled clinical trials (Walsh, 2008) and prospective studies (West & Zhou, 2007) have generally found such aids to be more effective than unassisted quit attempts, but community-based cross-sectional studies have failed to find similar effects (Pierce & Gilpin, 2002). This has led some critics to comment that SSMs have no useful applications in real-world settings (Chapman, 2011; Walsh, 2011). The criticisms are that the medicines either do not work because they are used inappropriately when provided over the counter or that they need the structure of a program to be effective.

It has, however, been suggested that the effectiveness of SSMs may be underestimated by cross-sectional studies because failed quit attempts where SSMs were used are better recalled than failed unassisted quit attempts (Berg et al., 2010; Walsh, 2008; West, 2006). Shorter quit attempts are also forgotten more quickly (Berg et al., 2010; Borland, Partos, Yong, Cummings, & Hyland, 2012). Such differential recall effects would result in the over-representation of failed quit attempts among the SSM users compared with unassisted attempters when relying on recall of attempts. An alternative explanation of the reduced effectiveness of SSMs observed in cross-sectional studies is that unassisted quitters may be less addicted than those who choose to use help (Shiffman, Di Marino, & Sweeney, 2005; Walsh, 2008).

This study explores differences in recall of recent quit attempts and level of addiction to cigarettes between SSM users and self-quitters, using data from the International Tobacco Control (ITC) four-country cohort study, where adult smokers from Australia, Canada, the United Kingdom, and the United States are surveyed regularly (every 6–24 months). Research suggests that quit attempts that began longer ago are more likely to be forgotten (Gilpin & Pierce, 1994; West, 2006). We therefore predict that the time since the start of the most recent recalled quit attempt will be longer, on average, for participants who used SSMs than for those who did not. Furthermore, we will explore whether any differential recall effects are maintained in the presence of any differences in level of addiction between SSM users and self-quitters.



At each wave of the ITC, participants are surveyed via standardized computer-assisted telephone interviews. Wave 8 was the first wave where participants were asked about their use of SSMs on their most recent quit attempt, rather than attempts over the past year. Participants were selected for the current analysis if they were daily smokers at Wave 7 of the ITC 4-country survey and were also surveyed at Wave 8 (N = 3,888). The data collection period for Wave 7 occurred between October 2008 and June 2009 and for Wave 8 between July 2010 and February 2011. The mean Wave 7–8 interwave interval was 611 days (SD = 45 days). Further detail on ITC methodology is available in Thompson et al. (2006).

We excluded from the sample described above anyone who did not report making any attempts to quit smoking during the interval between the two surveys (n = 2,288) and anyone who had made an attempt but was currently still quit at the Wave 8 survey (n = 429). We also excluded participants who could not provide valid data on their use of SSMs on their last quit attempt (n = 34) or on the length of time ago that their last quit attempt had started (n = 36). One thousand one hundred and one adults (60.0% female; mean age = 48.1 years, SD = 12.8 years) met our final selection criteria.


Use of SSMs on the last quit attempt was coded into three categories: (a) no medications used; (b) only NRT used; and (c) any prescription SSMs used, by cross-tabulating responses to the following three questions:

On your last quit attempt, did you use any type of nicotine replacement therapy? (n = 405 responded “yes”)

On your last quit attempt, did you use Bupropion, also called Zyban? (n = 57 responded “yes”)

On your last quit attempt, did you use Varenicline, also called Champix or Chantix? (n = 194 responded “yes”)

Participants reporting use of both NRT and Bupropion or Varenicline were coded into Category (c). The time (in days) since participants’ last quit attempt was assessed by asking, “When did your most recent quit attempt start?” SSM use and start of the last quit attempt were all assessed at the follow-up wave (Wave 8). To estimate baseline levels of addiction, we also calculated the Heaviness of Smoking Index (HSI: Heatherton, Kozlowski, Frecker, Rickert, & Robinson, 1989) from the reported number of cigarettes smoked per day and minutes to first cigarette of the day at the baseline wave (Wave 7: 3.2% missing). HSI scores range from 0 (least addicted) to 6 (most addicted).


We examined the prevalence of over-the-counter NRT and Rx SSM use on the most recent failed quit attempt and report 95% C I. We also computed mean levels of HSI and recalled time since the start of the last quit attempt across the three SSM use groups. We conducted a 3 (SSM group) × 7 (HSI levels) analysis of variance (ANOVA) to determine whether levels of addiction influenced the recalled time since the start of the last quit attempt. We explored these relationships both including and excluding participants who smoked less than 10 cigarettes/day at baseline (n = 344), as use of SSMs is not generally recommended as a quitting strategy for this group. The recalled time since the start of the last quit attempt was highly positively skewed so the square root transform of this variable was used in all parametric analyses.


Summary statistics for the variables of interest are provided in Table 1. On their most recent failed quit attempt, 49.7% did not report using any SSMs. Use of over-the-counter NRT was reported by 29.2% and use of some type of prescription medication by 21.1%. For the sample as a whole, the mean HSI was higher among SSM users (both NRT and Rx) than among those who did not use any SSMs, and the mean recalled time since the start of the last quit attempt was also longer among SSM users than nonusers. The prevalence of SSM use was higher among those individuals who smoked 10 or more cigarettes at baseline (55.4%, 95% CI: 51.8–58.9) than those who did not (39.2%, 95% CI: 34.0–44.0). A significant main effect of SSM use was still evident among these heavier smokers, F(2, 754) = 15.9, p < .001, with the recalled time since the start of the last quit attempt being longer among NRT and Rx SSM users than among nonusers (both p < .005). HSI also remained influential, F(2, 740) = 4.1, p < .05. As may be seen from the overlapping 95% CI in Table 1, however, the influence of HSI was much reduced among the group who smoked 10 or more cigarettes. Post-hoc comparisons revealed that the effect was attributable to the difference between the group who used prescription medications and those who did not use any SSMs (p < .05). We checked for different patterns of baseline HSI across the three SSM use categories among relapsed and successful quitters (by comparing the present relapsed sample to the sample of 405 smokers who were still successfully quit at follow-up) but did not find a significant SSM × quit success interaction, F(2, 1,454) = 1.2, p = .309. Both main effects were in the expected direction with higher baseline HSI among SSM users, F(1, 1,454) = 28.4, p < .001, and among the relapsed group, F(1, 1454) = 22.1, p < .001.

Table 1.
Prevalence of Stop-Smoking Medication (SSM) Use, Baseline Heaviness of Smoking Index (HSI), and Recalled Time Since the Start of the Last Quit Attempt for All Participants (N = 1,101) and Only Those Smoking 10 or More Cigarettes at Baseline (n = 757)

Results of the 3 (SSM group) × 7 (HSI levels) ANOVA on the whole sample revealed a significant main effect of SSM use, F(2, 1045) = 10.8, p < .001, and of HSI, F(6, 1045) = 2.9, p < .01, on the mean recalled time since the last quit attempt started. The SSM use × HSI interaction was not significant, F(12, 1,045) = 1.2, p = .308. Post-hoc comparisons (Bonferroni corrected) revealed significant differences between the no SSM group and both the NRT-only and Rx groups (both p < .005) but not between the NRT-only and Rx group (p = .130). These outcomes remained the same when only participants smoking 10 or more cigarettes at baseline were included. There were few participants in the extreme HSI categories (0 or 6), so we recoded the lowest HSI category to include scores of 0 and 1 and the highest to include 5 and 6 and recomputed the ANOVA for the whole sample (see Figure 1). Although the differences between the SSM use groups in recalled time since the last quit attempt appear greater at the higher HSIs, only the main effect of SSM use remained statistically significant, F(2, 1,051) = 17.8, p < .001. The main effect of HSI was no longer significant, F(4, 1,051) = 1.8, p = .131, and neither was the SSM use × HSI interaction, F(8, 1,051) = 1.0, p = .405.

Figure 1.
Mean recalled time (days) since the start of the last quit attempt by use of stop-smoking medications (SSM) and Heaviness of Smoking Index (HSI) scores for all participants (N = 1,101). Error bars represent ±95% CI. NRT = nicotine replacement ...

As a further check on differential recall, we compared rates of attempts in the previous month reported by all participants (where forgetting of attempts is minimal; Borland et al., 2012) with attempts reported earlier in the previous year. Unassisted quit attempts were reported more in the last month (58%, 95% CI: 51.8–64.1) compared with the rest of the year (47.3%, 95% CI: 44.0–50.7), with use of both NRT and prescription medications being reported relatively less, χ2(2) = 9.4, p < .01.


Smokers who reported using some sort of SSM on their most recent unsuccessful quit attempt recalled that quit attempt as having started longer ago than those who did not use any SSMs. This remained the case even when we controlled for baseline levels of addiction using the HSI and also when we only examined a subgroup of heavier smokers who smoked at least 10 cigarettes at baseline, although it should be noted that differences in baseline HSI across the SSM groups were still apparent among these heavier smokers. Furthermore, taking reports of attempts in the last month as a gold standard (when few if any are forgotten), the higher proportion of unassisted attempts in this period confirms a differential memory effect. The longer period since the last quit attempt made by smokers who used SSMs cannot simply be attributed to this group being more addicted and therefore less likely to have recently tried to quit than smokers who did not use SSMs. The results demonstrate the existence of a recall bias where quit attempts made using pharmaceutical assistance are remembered for longer than unassisted attempts. This provides one mechanism by which retrospective accounts of quit attempts overestimate the success rate of unassisted attempts relative to assisted attempts. Successful attempts, because the person has quit when interviewed, are not subject to any memory loss.

In addition to the recall bias, the results also show that smokers who elected to use SSMs on their last quit attempt were more addicted, based on higher mean HSI scores, a known predictor of relapse (Borland, Yong, O’Connor, Hyland, & Thompson, 2010). This represents a potential real difference in likely relapse rates but one that makes the comparison between assisted and unassisted attempts invalid unless it is adequately controlled for. Together, these two factors are likely to contribute at least in part to the failure to observe a clear treatment benefit of SSMs over unassisted cessation in cross-sectional community surveys. It is likely that these biases are not restricted to SSM use. The use of other forms of help during a quit attempt, such as joining a support group or cognitive behavioral therapy, are also likely to be better recalled than unassisted attempts and may be more likely to be sought by more highly addicted smokers.

The implications of the present findings are that retrospective estimates of differential quit success for various groups, particularly those using assistance to quit, should not be relied upon unless they have properly controlled for differential memory effects and differences in smoker characteristics, particularly those related to cessation success such as levels of dependence. This is especially important as both biases operate to underestimate success from assisted attempts compared with unassisted ones. Our research group is going on to try to reestimate quit rates for attempts with and without assistance, controlling for these biases.


The ITC 4-country survey is supported by multiple grants including Roswell Park Transdisciplinary Tobacco Use Research Center (R01 CA 100362 and P50 CA111236) and also in part from grant Roswell Park Cancer Institute, Buffalo, New York (P01 CA138389), all funded by the National Cancer Institute of the United States, Robert Wood Johnson Foundation (045734), Canadian Institutes of Health Research (57897, 79551), National Health and Medical Research Council of Australia (265903, 450110, APP1005922), Cancer Research UK (C312/A3726), Canadian Tobacco Control Research Initiative (014578), Centre for Behavioural Research and Program Evaluation, National Cancer Institute of Canada/Canadian Cancer Society.

Declaration of Interests

The authors declare that they have no competing interests.


We would like to thank members of the Data Management Core at the University of Waterloo for assistance in preparing the data for this analysis.

Ethics clearance: All waves of the study have received ethical approval from the relevant institutional review board or research ethics committee at The Cancer Council Victoria (Australia), Roswell Park Cancer Institute (USA), University of Waterloo (Canada), and University of Strathclyde (UK).


  • Berg CJ, An LC, Kirch M, Guo H, Thomas JL, Patten CA, et al. Failure to report attempts to quit smoking. Addictive Behaviors. 2010;35:900–904. doi:10.1016/j.addbeh.2010.06.009. [PubMed]
  • Borland R, Partos TR, Yong HH, Cummings KM, Hyland A. How much unsuccessful quitting activity is going on among adult smokers? Data from the International Tobacco Control Four Country cohort survey. Addiction. 2012;107:673–682. 682. doi:10.1111/j.1360-0443.2011.03685.x. [PubMed]
  • Borland R, Yong H.-H., O’Connor RJ, Hyland A, Thompson ME. The reliability and predictive validity of the Heaviness of Smoking Index and its two components: Findings from the International Tobacco Control Four Country study. Nicotine & Tobacco Research. 2010;12:s45–s50. doi:10.1093/ntr/ntq038. [PMC free article] [PubMed]
  • Chapman S. Smoking cessation: Big pharma butts in. British Medical Journal. 2011;343:344–345. doi:10.1136/bmj.d5008.
  • Gilpin E, Pierce JP. Measuring smoking cessation: Problems with recall in the 1990 California Tobacco Survey. Cancer Epidemiology, Biomarkers & Prevention. 1994;3:613–617. [PubMed]
  • Heatherton TF, Kozlowski LT, Frecker RC, Rickert W, Robinson J. Measuring the heaviness of smoking: Using self-reported time to the first cigarette of the day and number of cigarettes smoked per day. British Journal of Addiction. 1989;84:791–800. doi:10.1111/j.1360-0443.1989.tb03059.x. [PubMed]
  • Pierce JP, Gilpin EA. Impact of over-the-counter sales on effectiveness of pharmaceutical aids for smoking cessation. Journal of the American Medical Association. 2002;288:1260–1264. doi:10.1001/jama.288.10.1260. [PubMed]
  • Shiffman S, Di Marino ME, Sweeney CT. Characteristics of selectors of nicotine replacement therapy. Tobacco Control. 2005;14:346–355. doi:10.1136/tc.2004.009183. [PMC free article] [PubMed]
  • Thompson M, Fong GT, Hammond D, Boudreau C, Driezen P, Hyland A, et al. Methods of the International Tobacco Control (ITC) Four Country Survey. Tobacco Control. 2006;15:iii12–iii18. doi:10.1136/tc.2005.013870. [PMC free article] [PubMed]
  • Walsh RA. Over-the-counter nicotine replacement therapy: A methodological review of the evidence supporting its effectiveness. Drug and Alcohol Review. 2008;27:529–547. doi:10.1080/09595230802245527. [PubMed]
  • Walsh RA. Australia's experience with Varenicline: Usage, costs, and adverse reactions. Addiction. 2011;106:449–452. doi:10.1111/j.1360-0443.2010.03282.x. [PubMed]
  • West R. Feasibility of a national longitudinal study (‘The Smoking Toolkit Study’) to monitor smoking cessation and attempts at harm reduction in the UK. 2006. Pilot report. Retrieved from (
  • West R, Zhou X. Is nicotine replacement therapy for smoking cessation effective in the “real world”? Findings from a prospective multinational cohort study. Thorax. 2007;62:998–1002. doi:10.1136/thx.2007.078758. [PMC free article] [PubMed]

Articles from Nicotine & Tobacco Research are provided here courtesy of Oxford University Press