Search tips
Search criteria 


Logo of procamiaLink to Publisher's site
AMIA Annu Symp Proc. 2012; 2012: 789–798.
Published online 2012 November 3.
PMCID: PMC3540477

Using a Resource Effect Study Pre-Pilot to Inform a Large Randomized Trial: The Decide2Quit.Org Web-Assisted Tobacco Intervention


Resource effect studies can be useful in highlighting areas of improvement in informatics tools. Before a large randomized trial, we tested the functions of the Web-assisted tobacco intervention using smokers (N=204) recruited via Google advertisements. These smokers were given access to for six months and we tracked their usage and assessed their six months cessation using a rigorous follow-up. Multiple, interesting findings were identified: we found the use of tailored emails to dramatically increase participation for a short period. We also found varied effects of the different functions. Functions supporting “seeking social support” (Your Online Community and Family Tools), Healthcare Provider Tools, and the Library had positive effects on quit outcomes. One surprising finding, which needs further investigation, was that writing to our Tobacco Treatment Specialists was negatively associated with quit outcomes.


Several promising informatics tools have been developed to support decision making and provide behavior support for patients, ranging from Web-assisted tools to smart phone applications [1, 2]. Evaluations of these tools have mostly focused on the tool as a black box [1, 2]; however, many of these tools are multi-modal including multiple functions that the tool developers considered to help the patient. Eliciting which functions or which combinations of functions have the greatest impact on patient outcomes would help tool developers focus and make targeted inclusions to their system, rather than working on functions which may have limited effect on the patient [2].

In the planning stage of two nationwide trials evaluating a Web-assisted tobacco intervention — [3] — we conducted a resource-effect study with smokers from across the United States to evaluate the functions of the system. Within, we have implemented multiple innovative functions designed to continuously engage users and provide support for smoking cessation. Our pre-randomized trial pilot resource-effect study was not hypothesis driven, rather it was a hypothesis-generating knowledge discovery evaluation, designed to inform refinement of the system for the randomized trial. Thus, we explored which functions we should target for the intervention. In this paper, we report our analysis of the impact of the intervention functions on repeat engagement with the system, and the association of their use with participant behavior change, six month smoking cessation outcomes.


Study Design

Resource effect studies evaluate the impact of the system, rather than the usability of the system; they are used to evaluate the influence of the resource on users [4, 5]. We used a prospective cohort of smokers recruited using Google advertisements to These smokers were given access to for a period of six months, and we tracked their usage and assessed six month cessation outcomes at the end of the study. This study was approved by the University of Alabama at Birmingham and University of Massachusetts Medical School Institutional Review Boards. Web-Assisted Tobacco Intervention

Over the last 10 years, we have continued to develop and evaluate, an evidence-based Web-assisted tobacco intervention. The initial development of focused on the development of interactive educational content [3]. These included self-help strategies (ex: helpful tips, interactive calculators, library of useful articles), but also content that encouraged seeking social support, and help from health care providers. [3] The educational content was developed based on formative research and using constructs from multiple behavioral theories including health belief model (HBM), [6] social cognitive theory (SCT), [7] and the transtheoretical model (TM). [8] These content were divided among the following functions in My Health Risks, Thinking About Quitting, Family Tools, Healthcare Provider Tools, and the Library (Table 1). For our current R01, was expanded with three more innovative functions to continuously engage smokers, including tailored email messaging, secure asynchronous messaging with a Tobacco Treatment Specialist, and an online smoker community (see Table 1 and Figure 1) [9].

Figure 1. Web-Assisted Tobacco Intervention
Table 1:
Major Functions of Decide2Quit

Setting and Sample

The current study included 204 smokers who were recruited using Google AdWords [10] from across the US and between the timeframe of August 5 2010 through December 4 2011. To recruit smokers through Google advertisements, we posted three advertisements that were linked to searches for keywords related to smoking (e.g., smoking, quit smoking, stop smoking, quit, quit smoking tips, quit smoking programs). Each advertisement included the link that would directly take the participant to the home page where they could choose to register as a new participant.

Data Collection

All smokers completed an online survey during registration on the website. We collected demographics (age, gender, race, education, and Internet usage) and smoker behavior (smoking status, quit history, readiness to quit, and information on other smokers). Logins and online activity was tracked through web page scripts. To assess smoking cessation, we conducted a rigorous six-month follow-up using multiple methods. Initially, smokers were emailed up to 5 times and given the option of completing the survey online. For those who did not complete online, we followed up with a telephone survey. Twenty attempts were made to reach the patient. After verbal or written consent was obtained, the survey was conducted and the patient’s address was gained so they could be mailed a $30 gift card for completing the survey. The six-month follow-up survey confirmed general demographic characteristics, and six-month smoking behavior.

Statistical Analyses

For this hypothesis-generation pre-pilot, we conducted two main analyses:

  1. Effects of tailored messaging on return visits to
    For this analysis, we used all 204 smokers. We compared return visits of the smokers to on message and non-message days using the chi-square statistic. Our dependent variable was return visits. Our independent variable was message days. We compared days immediately after the messages to days when no messages were sent. We then developed a multivariable model, adjusted for demographics, smoker characteristics, and readiness to quit. We used a generalized estimating equation for panel data, with a logit link and exchangeable correlation matrix. We also evaluated whether the effectiveness of the messages to induce return visits decayed over time.
  2. Effects of the functions of on six months cessation outcomes
    For this analysis, we excluded smokers who registered on after September 2011 and were not eligible for six months cessation follow-up. Our final sample size was 195 smokers. In this analysis, our dependent variable was six month point prevalent smoking abstinence as reported in the follow-up phone survey, which was assessed using the question — Did you smoke any cigarettes during the past 30 days. This is a standard, validated question that has been used in multiple studies including the California Adult Tobacco Survey [11], and Massachusetts Tobacco Survey Adults [12].

We first measured total level of engagement measured by number of visits to We also evaluated how the use of different functions of the system was interrelated. We then compared six month cessation outcomes with the use of the different functions of the system (Table 1). We then created a summary score of the functions that had the most differential in quit outcomes between those who use the functions compared with those who did not use the function. We then developed a multivariable logistic regression model, adjusting for the summary score, demographics, smoker characteristics, and readiness to quit.


Smokers were mostly female (64.7%), white (88.7%), and with some college education or a college graduate (67.2%). Most of the smokers were thinking of quitting (60.8%); 23% of smokers already quit. Most of the smokers had tried quitting before (53.4%). A lower proportion of smokers had visited a Web-assisted tobacco intervention before (32.8%). The mean cigarettes smoked per day was 17.4 (SD=10.4).

1. Effects of tailored messaging on return visits to

Overall, for these 204 participants, we collected six months of follow-up data. Thus, there were 36,222 person-days on the system. On the days a message was sent (5,962 person-days, 16% of total days), 52% of the return visits occurred. On the first day following a message, 27% of the return visits occurred; on the second day following a message, 13% of the return visits occurred. (Figure 2) On the remaining days, which represented the majority of total days, only 7% of the total visits occurred.

Figure 2:
Effects of tailored messages on return visits

Compared with all other days, return visit was most likely to occur on the three days surrounding the message (OR=5. 4, 95% CI=4.02, 7.31). We found stepwise impact on return visits on the days immediately following a message: message day (OR=25.96, 95% CI=13.08, 51.52), message day + 1 (OR=14.17, 95% CI=6.77, 29.63), message day + 2 (OR=9.40, 95% CI=4.79, 18.40). This effect was sustained after adjusting for patient age, gender, education, readiness to quit, and baseline number of cigarettes per day. However, the effectiveness of the messages to induce returns visits declined over time. The proportion of return visit was greater in the first four weeks and then declined in the subsequent months (Figure 3)

Figure 2:
Effects of tailored messages on return visits

2. Effects of the functions of on six months cessation outcomes

There was no linear association between number of visits and cessation. In bivariate association, we did not find any dose-response with Quit outcomes (OR=0.96, 95% CI=0.83, 1.10). However, we did find use of individual functions was associated with cessation. In bivariate comparison, use of Family Tools, Healthcare Provider Tools, The Library, and Your Online community had the most positive association with the outcomes (Table 3).

Table 3.
Effects of the functions of on six months cessation outcomes

We found that the use of Family Tools, Healthcare Provider Tools, and The Library were highly correlated (range 0.74 to 0.65). The use of My Health Risks and Thinking About Quitting were modestly correlated with other components and themselves (range 0.56 to 0.41). The two communication function of the system — Your Online Community and talking with Tobacco Treatment Specialists — demonstrated weak correlations with each other and other functions (range 0.33 to 0.13). We created a summary score, from 0 being use of no functions, 1 being use of one or two functions, to 2 being use of three or four functions. After adjusting for the summary score, number of visits, age, sex, education, smoking status, and number of cigarettes per day, we found a linear association - for every increase by one in this scale, the odds of smoking cessation increased by (OR= 2.10, 95% CI = 1.03, 4.30). In addition to this significant trend, we also looked at odds for each level of use. We found increased number of functions used was associated with increased cessation (Table 4).

Table 4:
Impact of Summary Score of Use of 4 Functions Adjusted


Our resource effect study with smokers (N=204) to test the functions of yielded interesting results. We found that tailored emails were critical to longitudinal participation in the intervention. On days when no emails had been sent, very little participation occurred. We also found increased participation on the days immediately following the messages, even after adjustment for patient characteristics. We found that general increased participation, measured by number of visits (OR=0.96, 95% CI=0.83, 1.10) was not associated with an increase in smoking cessation. However, various functions had wide ranging effects on the six-month cessation outcomes. We found that a summary score of four components was strongly predictive of cessation. This was significant even after adjusting for number of visits. Previous studies have shown a relationship between longitudinal engagement with the Intervention and improved outcomes [13, 14, 15, 16, 17]; however, based on our findings, we speculate that it is not how many times you come, but what you do when you come that makes a difference in cessation.

Attrition rates are usually high in Web-based interventions [18, 19, 20, 21], and identifying effective strategies to increase engagement is crucial. Our study strongly shows that continuous emails are effective in bringing patients back to the system. This is consistent with other studies that also used emails to engage smokers [18, 22, 23, 24, 25]. All our emails contained links to the website and provided a direct cue for the patients to engage with the system. Of note, each of our email reminders included an “opt-out” option to no longer receive messages, and 14% of our users chose to opt out from the emails. However, the effectiveness of these messages to bring back smokers decayed over time, potentially indicating a need to continuously evolve these messages and their tailoring. Machine learning recommender approaches that companies like Google, Amazon, Netflix and Pandora use to provide content with enhanced personal relevance may be useful [26, 27], and we are exploring them to improve our message tailoring.

Functions enhancing a patient’s need for “seeking social support” were associated with six months cessation. The Your Online Community function had the highest differential (+5.5) in quit outcomes between those who used the functions compared with others. This is consistent with other studies that have used an online community, and found they have some association with improving outcomes [10, 28]. Our study also adds to the increasing body of evidence that highlight the influence of peer voice in changing behavior [29, 30, 31, 32]. There are several implications for us. One potential enhancement is to improve the ease of access and communication with other smokers. In most online communities, there are natural leaders — users who consistently post on the forums. Most users only lurk, i.e., they read the leaders’ posts and do not actively contribute. Potential opportunities exist to proactively enhance this natural tendency. We could recruit these leaders and provide them proactive tools to engage with other smokers. This may increase participation in the community and improve outcomes. Such “community leaders” approaches have been outside informatics applications with success [33, 34, 35].

Another “seeking social support” function that was associated with improved outcomes was Family tools. Family tools provide helpful tips to the patient on how to get help from their family, and deal with issues like. This is not surprising because family members can have a huge influence on changing the behavior of the patient. We are considering ways of enhancing this function. One approach would be to develop content targeting the family member and providing tips on how to best support the patient. It would be interesting to test if “tailored” messages to family members would increase the patient’s participation on Patients in other settings have expressed interest in sharing health information with family members [36]. Having delegate function using which patients can provide restricted access to their family members may be important to improve outcomes [36, 37, 38]. Examples of information that could be shared are the patient’s visit rates, and use of the different functions on the system.

Another function that was associated with improved outcomes was the health provider tools. Our toolset included tips on how to include the patient’s healthcare provider in their quit smoking plan. In our randomized trial, we are also implementing an e-referral system, using which providers can refer smokers. Included in the e-referral system are proximal reports to the providers, including the patients’ engagement with A secure messaging function using which the provider and their patients can interact with the provider is also being developed. Such function may further enhance the communication between the provider and their patients, and may lead to improved six months cessation outcomes.

We also found that the Library which provided information on multiple smoking related topics was associated with improved outcomes. One way to enhance the library content may be to include user generated content, such as stories on how a patient used different tools at their disposal to quit smoking. While use of a single component was not significant associated with the six months outcome, we found that the summary score was significantly associated (OR=2.06, 95% CI = 1.02, 4.14), implying that smokers who use two or more components were more likely to quit smoking.

Our study is the first to test the use of asynchronous messaging with a Tobacco Treatment Specialists in web environments. The results were surprising that the use of Tobacco Treatment Specialists was negatively associated with six months cessation. This was surprising because we studies of counseling have consistently showed improvements in smoking cessation outcomes [39, 40, 41]. A meta-analysis of nine studies showed that counseling was effective in improving outcomes compared with usual care (1.37 (1.26–1.50)) [42]. One reason could be confounding by indication; our patients having the most difficulty with quitting smoking self-selected to use the Tobacco Treatment Specialists function. However, even after adjusting for demographics and smoking characteristics, the effect remained. More research is needed to understand this negative association. We are currently looking at the messages between the Tobacco Treatment Specialists and the smokers and communicating with our Tobacco Treatment Specialists about improving the function.

This resource effect study was observational; therefore, we cannot directly infer causality. However, despite having a limited sample size of 204 patients, our study yielded important findings. Our messaging system is also limited by the potential that smokers may not have received the emails we sent them. We were unable to confirm email delivery. We also collected a limited number of characteristics for these smokers; the samples may also vary on important, unmeasured characteristics. We only evaluated the impact on one intervention ( preventing strict generalizability to other technology interventions.

We also assessed smoking cessation using only self-report, not biochemical verification. In a smoking cessation study, there is always misclassification in self-report; however, the need for biochemical verification of nicotine depends on the intervention [40]. Studies that are in-person and intense generally have more misclassification because of the personal connection between the smoker and the counselor and therefore require biochemical verification. Low intensity and light-touch studies have less misclassification. In these studies, not only is biochemical verification less needed, it also is more difficult to do because of the light touch.


Resource effect studies are useful in evaluating technology interventions before they are conducted to identify areas of improvement. Our study suggests that different functions may have a differential impact on the patient. The results of this study helped us identify several areas of improvement.

Table 2:
Demographic Characteristics (N=204)**


Funding for the study was received from the US National Cancer Institute, grant 1 R01 CA129091.


1. Ekeland AG, Bowes A, Flottorp S. Methodologies for assessing telemedicine: a systematic review of reviews. Int J Med Inform. 2012 Jan;81(1):1–11. [PubMed]
2. Strecher VJ, McClure JB, Alexander GL, Chakraborty B, Nair VN, Konkel JM, et al. Web-based smoking-cessation programs: results of a randomized trial. Am J Prev Med. 2008 May;34(5):373–81. [PMC free article] [PubMed]
3. Houston TK, Ford DE. A tailored Internet-delivered intervention for smoking cessation designed to encourage social support and treatment seeking: usability testing and user tracing. Inform Health Soc Care. 2008 Mar;33(1):5–19. [PubMed]
4. Friedman CP, Wyatt J. Evaluation and technology assessment. In: Shortliffe EH, Cimino JJ, editors. Biomedical informatics : computer applications in health care and biomedicine. 3rd ed. New York, NY: Springer; 2006. pp. 379–403.
5. Friedman CP, Wyatt J. Evaluation Methods in Biomedical Informatics (Health Informatics) Softcover reprint of hardcover. 2nd ed. 2006 ed. Springer; 2010. Evaluation as a field; pp. 21–48.
6. Rosenstock IM, Strecher VJ, Becker MH. Social learning theory and the Health Belief Model. Health Educ Q. 1988 Summer;15(2):175–83. [PubMed]
7. Bandura A. Social Foundations of Thought and Action: a social cognitive theory. Englewood Cliffs, NJ: Prentice Hall; 1986.
8. DiClemente CC, Prochaska JO, Fairhurst SK, Velicer WF, Velasquez MM, Rossi JS. The process of smoking cessation: an analysis of precontemplation, contemplation, and preparation stages of change. J Consult Clin Psychol. 1991 Apr;59(2):295–304. [PubMed]
9. Houston TK, Sadasivam RS, Ford DE, Richman J, Ray MN, Allison JJ. The QUIT-PRIMO provider-patient Internet-delivered smoking cessation referral intervention: a cluster-randomized comparative effectiveness trial: study protocol. Implement Sci. 2010;5:87. [PMC free article] [PubMed]
10. Cobb NK, Graham AL, Byron MJ, Niaura RS, Abrams DB. Online social networks and smoking cessation: a scientific research agenda. J Med Internet Res. 2011;13(4):e119. [PubMed]
11. California Department of Public Health’s Cancer Surveillance and Research Branch (CSRB) California Adult Tobacco Survey. California Department of Public Health’s Cancer Surveillance and Research Branch (CSRB); [cited 2012 July 6]; Available from:
12. Massachusetts Tobacco Survey Adults. Available from:
13. Glasgow RE, Christiansen SM, Kurz D, King DK, Woolley T, Faber AJ, et al. Engagement in a diabetes self-management website: usage patterns and generalizability of program use. J Med Internet Res. 2011;13(1):e9. [PMC free article] [PubMed]
14. Saul JE, Schillo BA, Evered S, Luxenberg MG, Kavanaugh A, Cobb N, et al. Impact of a statewide Internet-based tobacco cessation intervention. J Med Internet Res. 2007;9(3):e28. [PMC free article] [PubMed]
15. Davies C, Corry K, Van Itallie A, Vandelanotte C, Caperchione C, Mummery WK. Prospective associations between intervention components and website engagement in a publicly available physical activity website: the case of 10,000 Steps Australia. J Med Internet Res. 2012;14(1):e4. [PubMed]
16. Strecher VJ, McClure J, Alexander G, Chakraborty B, Nair V, Konkel J, et al. The role of engagement in a tailored web-based smoking cessation program: randomized controlled trial. J Med Internet Res. 2008;10(5):e36. [PMC free article] [PubMed]
17. Cobb NK, Graham AL, Bock BC, Papandonatos G, Abrams DB. Initial evaluation of a real-world Internet smoking cessation system. Nicotine Tob Res. 2005 Apr;7(2):207–16. [PMC free article] [PubMed]
18. Robroek SJ, Lindeboom DE, Burdorf A. Initial and Sustained Participation in an Internet-delivered Long-term Worksite Health Promotion Program on Physical Activity and Nutrition. J Med Internet Res. 2012;14(2):e43. [PMC free article] [PubMed]
19. Postel MG, de Haan HA, ter Huurne ED, van der Palen J, Becker ES, de Jong CA. Attrition in web-based treatment for problem drinkers. J Med Internet Res. 2011;13(4):e117. [PMC free article] [PubMed]
20. Manzoni GM, Pagnini F, Corti S, Molinari E, Castelnuovo G. Internet-based behavioral interventions for obesity: an updated systematic review. Clin Pract Epidemiol Ment Health. 2011;7:19–28. [PMC free article] [PubMed]
21. Neve MJ, Collins CE, Morgan PJ. Dropout, nonusage attrition, and pretreatment predictors of nonusage attrition in a commercial Web-based weight loss program. J Med Internet Res. 2010;12(4):e69. [PMC free article] [PubMed]
22. Houston TK, Coley HL, Sadasivam RS, Ray MN, Williams JH, Allison JJ, et al. Impact of content-specific email reminders on provider participation in an online intervention: a dental PBRN study. Stud Health Technol Inform. 2010;160(Pt 2):801–5. [PMC free article] [PubMed]
23. Woodall WG, Buller DB, Saba L, Zimmerman D, Waters E, Hines JM, et al. Effect of emailed messages on return use of a nutrition education website and subsequent changes in dietary behavior. J Med Internet Res. 2007;9(3):e27. [PMC free article] [PubMed]
24. Abdolrasulnia M, Collins BC, Casebeer L, Wall T, Spettell C, Ray MN, et al. Using email reminders to engage physicians in an Internet-based CME intervention. BMC Med Educ. 2004 Sep 29;4:17. [PMC free article] [PubMed]
25. Bot M, Milder IE, Bemelmans WJ. Nationwide implementation of Hello World: a Dutch email-based health promotion program for pregnant women. J Med Internet Res. 2009;11(3):e24. [PMC free article] [PubMed]
26. Segaran T. Programming collective intelligence : building smart web 2.0 applications. 1st ed. Beijing ; Sebastapol CA: O’Reilly; 2007.
27. Marmanis H, Babenko D. Algorithms of the intelligent web. Greenwich, CT: Manning; 2009.
28. Fallin A, Zuercher R, Rayens MK, Adkins S, York N, Hahn EJ. A Short Online Community Readiness Survey for Smoke-Free Policy. Nicotine Tob Res. 2012 Mar 5; [PMC free article] [PubMed]
29. McQueen A, Kreuter MW, Kalesan B, Alcaraz KI. Understanding narrative effects: the impact of breast cancer survivor stories on message processing, attitudes, and beliefs among African American women. Health Psychol. 2011 Nov;30(6):674–82. [PMC free article] [PubMed]
30. Houston TK, Cherrington A, Coley HL, Robinson KM, Trobaugh JA, Williams JH, et al. The art and science of patient storytelling-harnessing narrative communication for behavioral interventions: the ACCE project. J Health Commun. 2011 Aug;16(7):686–97. [PubMed]
31. Brouwer W, Kroeze W, Crutzen R, de Nooijer J, de Vries NK, Brug J, et al. Which intervention characteristics are related to more exposure to internet-delivered healthy lifestyle promotion interventions? A systematic review. J Med Internet Res. 2011;13(1):e2. [PMC free article] [PubMed]
32. Kreuter MW, Holmes K, Alcaraz K, Kalesan B, Rath S, Richert M, et al. Comparing narrative and informational videos to increase mammography in low-income African American women. Patient Educ Couns. 2010 Dec;81(Suppl):S6–14. [PMC free article] [PubMed]
33. Fouad MN, Partridge E, Dignan M, Holt C, Johnson R, Nagy C, et al. A community-driven action plan to eliminate breast and cervical cancer disparity: successes and limitations. J Cancer Educ. 2006 Spring;21(1 Suppl):S91–100. [PubMed]
34. Savage CL, Xu Y, Lee R, Rose BL, Kappesser M, Anthony JS. A case study in the use of community-based participatory research in public health nursing. Public Health Nurs. 2006 Sep-Oct;23(5):472–8. [PubMed]
35. Eng E. The Save our Sisters Project. A social network strategy for reaching rural black women. Cancer. 1993 Aug 1;72(3 Suppl):1071–7. [PubMed]
36. Zulman DM, Nazi KM, Turvey CL, Wagner TH, Woods SS, An LC. Patient interest in sharing personal health record information: a web-based survey. Ann Intern Med. 2011 Dec 20;155(12):805–10. [PubMed]
37. Turvey CL, Zulman DM, Nazi KM, Wakefield BJ, Woods SS, Hogan TP, et al. Transfer of Information from Personal Health Records: A Survey of Veterans Using My HealtheVet. Telemed J E Health. 2012 Mar;18(2):109–14. [PubMed]
38. Ralston JD, Carrell D, Reid R, Anderson M, Moran M, Hereford J. Patient web services integrated with a shared medical record: patient use and satisfaction. J Am Med Inform Assoc. 2007 Nov-Dec;14(6):798–806. [PMC free article] [PubMed]
39. Meites E, Thom DH. Telephone counseling improves smoking cessation rates. Am Fam Physician. 2007 Mar 1;75(5):650. [PubMed]
40. Rollins G. Smoking quit lines work in clinical trials and actual practice. Rep Med Guidel Outcomes Res. 2002 Nov 1;13(21):7–9. [PubMed]
41. Zhu SH, Tedeschi G, Anderson CM, Rosbrook B, Byrd M, Johnson CE, et al. Telephone counseling as adjuvant treatment for nicotine replacement therapy in a “real-world” setting. Prev Med. 2000 Oct;31(4):357–63. [PubMed]
42. Mahvan T, Namdar R, Voorhees K, Smith PC, Ackerman W. Clinical Inquiry: which smoking cessation interventions work best? J Fam Pract. 2011 Jul;60(7):430–1. [PubMed]

Articles from AMIA Annual Symposium Proceedings are provided here courtesy of American Medical Informatics Association