|Home | About | Journals | Submit | Contact Us | Français|
Resource effect studies can be useful in highlighting areas of improvement in informatics tools. Before a large randomized trial, we tested the functions of the Decide2Quit.org Web-assisted tobacco intervention using smokers (N=204) recruited via Google advertisements. These smokers were given access to Decide2Quit.org for six months and we tracked their usage and assessed their six months cessation using a rigorous follow-up. Multiple, interesting findings were identified: we found the use of tailored emails to dramatically increase participation for a short period. We also found varied effects of the different functions. Functions supporting “seeking social support” (Your Online Community and Family Tools), Healthcare Provider Tools, and the Library had positive effects on quit outcomes. One surprising finding, which needs further investigation, was that writing to our Tobacco Treatment Specialists was negatively associated with quit outcomes.
Several promising informatics tools have been developed to support decision making and provide behavior support for patients, ranging from Web-assisted tools to smart phone applications [1, 2]. Evaluations of these tools have mostly focused on the tool as a black box [1, 2]; however, many of these tools are multi-modal including multiple functions that the tool developers considered to help the patient. Eliciting which functions or which combinations of functions have the greatest impact on patient outcomes would help tool developers focus and make targeted inclusions to their system, rather than working on functions which may have limited effect on the patient .
In the planning stage of two nationwide trials evaluating a Web-assisted tobacco intervention — Decide2Quit.org  — we conducted a resource-effect study with smokers from across the United States to evaluate the functions of the system. Within Decide2Quit.org, we have implemented multiple innovative functions designed to continuously engage users and provide support for smoking cessation. Our pre-randomized trial pilot resource-effect study was not hypothesis driven, rather it was a hypothesis-generating knowledge discovery evaluation, designed to inform refinement of the system for the randomized trial. Thus, we explored which functions we should target for the intervention. In this paper, we report our analysis of the impact of the intervention functions on repeat engagement with the Decide2Quit.org system, and the association of their use with participant behavior change, six month smoking cessation outcomes.
Resource effect studies evaluate the impact of the system, rather than the usability of the system; they are used to evaluate the influence of the resource on users [4, 5]. We used a prospective cohort of smokers recruited using Google advertisements to Decide2Quit.org. These smokers were given access to Decide2Quit.org for a period of six months, and we tracked their usage and assessed six month cessation outcomes at the end of the study. This study was approved by the University of Alabama at Birmingham and University of Massachusetts Medical School Institutional Review Boards.
Over the last 10 years, we have continued to develop and evaluate Decide2Quit.org, an evidence-based Web-assisted tobacco intervention. The initial development of Decide2Quit.org focused on the development of interactive educational content . These included self-help strategies (ex: helpful tips, interactive calculators, library of useful articles), but also content that encouraged seeking social support, and help from health care providers.  The educational content was developed based on formative research and using constructs from multiple behavioral theories including health belief model (HBM),  social cognitive theory (SCT),  and the transtheoretical model (TM).  These content were divided among the following functions in Decide2Quit.org: My Health Risks, Thinking About Quitting, Family Tools, Healthcare Provider Tools, and the Library (Table 1). For our current R01, Decide2Quit.org was expanded with three more innovative functions to continuously engage smokers, including tailored email messaging, secure asynchronous messaging with a Tobacco Treatment Specialist, and an online smoker community (see Table 1 and Figure 1) .
The current study included 204 smokers who were recruited using Google AdWords  from across the US and between the timeframe of August 5 2010 through December 4 2011. To recruit smokers through Google advertisements, we posted three advertisements that were linked to searches for keywords related to smoking (e.g., smoking, quit smoking, stop smoking, quit, quit smoking tips, quit smoking programs). Each advertisement included the link that would directly take the participant to the home page where they could choose to register as a new participant.
All smokers completed an online survey during registration on the Decide2Quit.org website. We collected demographics (age, gender, race, education, and Internet usage) and smoker behavior (smoking status, quit history, readiness to quit, and information on other smokers). Logins and online activity was tracked through web page scripts. To assess smoking cessation, we conducted a rigorous six-month follow-up using multiple methods. Initially, smokers were emailed up to 5 times and given the option of completing the survey online. For those who did not complete online, we followed up with a telephone survey. Twenty attempts were made to reach the patient. After verbal or written consent was obtained, the survey was conducted and the patient’s address was gained so they could be mailed a $30 gift card for completing the survey. The six-month follow-up survey confirmed general demographic characteristics, and six-month smoking behavior.
For this hypothesis-generation pre-pilot, we conducted two main analyses:
We first measured total level of engagement measured by number of visits to Decide2Quit.org. We also evaluated how the use of different functions of the system was interrelated. We then compared six month cessation outcomes with the use of the different functions of the system (Table 1). We then created a summary score of the functions that had the most differential in quit outcomes between those who use the functions compared with those who did not use the function. We then developed a multivariable logistic regression model, adjusting for the summary score, demographics, smoker characteristics, and readiness to quit.
Smokers were mostly female (64.7%), white (88.7%), and with some college education or a college graduate (67.2%). Most of the smokers were thinking of quitting (60.8%); 23% of smokers already quit. Most of the smokers had tried quitting before (53.4%). A lower proportion of smokers had visited a Web-assisted tobacco intervention before (32.8%). The mean cigarettes smoked per day was 17.4 (SD=10.4).
Overall, for these 204 participants, we collected six months of follow-up data. Thus, there were 36,222 person-days on the system. On the days a message was sent (5,962 person-days, 16% of total days), 52% of the return visits occurred. On the first day following a message, 27% of the return visits occurred; on the second day following a message, 13% of the return visits occurred. (Figure 2) On the remaining days, which represented the majority of total days, only 7% of the total visits occurred.
Compared with all other days, return visit was most likely to occur on the three days surrounding the message (OR=5. 4, 95% CI=4.02, 7.31). We found stepwise impact on return visits on the days immediately following a message: message day (OR=25.96, 95% CI=13.08, 51.52), message day + 1 (OR=14.17, 95% CI=6.77, 29.63), message day + 2 (OR=9.40, 95% CI=4.79, 18.40). This effect was sustained after adjusting for patient age, gender, education, readiness to quit, and baseline number of cigarettes per day. However, the effectiveness of the messages to induce returns visits declined over time. The proportion of return visit was greater in the first four weeks and then declined in the subsequent months (Figure 3)
There was no linear association between number of visits and cessation. In bivariate association, we did not find any dose-response with Quit outcomes (OR=0.96, 95% CI=0.83, 1.10). However, we did find use of individual functions was associated with cessation. In bivariate comparison, use of Family Tools, Healthcare Provider Tools, The Library, and Your Online community had the most positive association with the outcomes (Table 3).
We found that the use of Family Tools, Healthcare Provider Tools, and The Library were highly correlated (range 0.74 to 0.65). The use of My Health Risks and Thinking About Quitting were modestly correlated with other components and themselves (range 0.56 to 0.41). The two communication function of the system — Your Online Community and talking with Tobacco Treatment Specialists — demonstrated weak correlations with each other and other functions (range 0.33 to 0.13). We created a summary score, from 0 being use of no functions, 1 being use of one or two functions, to 2 being use of three or four functions. After adjusting for the summary score, number of visits, age, sex, education, smoking status, and number of cigarettes per day, we found a linear association - for every increase by one in this scale, the odds of smoking cessation increased by (OR= 2.10, 95% CI = 1.03, 4.30). In addition to this significant trend, we also looked at odds for each level of use. We found increased number of functions used was associated with increased cessation (Table 4).
Our resource effect study with smokers (N=204) to test the functions of Decide2Quit.org yielded interesting results. We found that tailored emails were critical to longitudinal participation in the intervention. On days when no emails had been sent, very little participation occurred. We also found increased participation on the days immediately following the messages, even after adjustment for patient characteristics. We found that general increased participation, measured by number of visits (OR=0.96, 95% CI=0.83, 1.10) was not associated with an increase in smoking cessation. However, various functions had wide ranging effects on the six-month cessation outcomes. We found that a summary score of four components was strongly predictive of cessation. This was significant even after adjusting for number of visits. Previous studies have shown a relationship between longitudinal engagement with the Intervention and improved outcomes [13, 14, 15, 16, 17]; however, based on our findings, we speculate that it is not how many times you come, but what you do when you come that makes a difference in cessation.
Attrition rates are usually high in Web-based interventions [18, 19, 20, 21], and identifying effective strategies to increase engagement is crucial. Our study strongly shows that continuous emails are effective in bringing patients back to the system. This is consistent with other studies that also used emails to engage smokers [18, 22, 23, 24, 25]. All our emails contained links to the Decide2Quit.org website and provided a direct cue for the patients to engage with the system. Of note, each of our email reminders included an “opt-out” option to no longer receive messages, and 14% of our users chose to opt out from the emails. However, the effectiveness of these messages to bring back smokers decayed over time, potentially indicating a need to continuously evolve these messages and their tailoring. Machine learning recommender approaches that companies like Google, Amazon, Netflix and Pandora use to provide content with enhanced personal relevance may be useful [26, 27], and we are exploring them to improve our message tailoring.
Functions enhancing a patient’s need for “seeking social support” were associated with six months cessation. The Your Online Community function had the highest differential (+5.5) in quit outcomes between those who used the functions compared with others. This is consistent with other studies that have used an online community, and found they have some association with improving outcomes [10, 28]. Our study also adds to the increasing body of evidence that highlight the influence of peer voice in changing behavior [29, 30, 31, 32]. There are several implications for us. One potential enhancement is to improve the ease of access and communication with other smokers. In most online communities, there are natural leaders — users who consistently post on the forums. Most users only lurk, i.e., they read the leaders’ posts and do not actively contribute. Potential opportunities exist to proactively enhance this natural tendency. We could recruit these leaders and provide them proactive tools to engage with other smokers. This may increase participation in the community and improve outcomes. Such “community leaders” approaches have been outside informatics applications with success [33, 34, 35].
Another “seeking social support” function that was associated with improved outcomes was Family tools. Family tools provide helpful tips to the patient on how to get help from their family, and deal with issues like. This is not surprising because family members can have a huge influence on changing the behavior of the patient. We are considering ways of enhancing this function. One approach would be to develop content targeting the family member and providing tips on how to best support the patient. It would be interesting to test if “tailored” messages to family members would increase the patient’s participation on Decide2Quit.org. Patients in other settings have expressed interest in sharing health information with family members . Having delegate function using which patients can provide restricted access to their family members may be important to improve outcomes [36, 37, 38]. Examples of information that could be shared are the patient’s visit rates, and use of the different functions on the system.
Another function that was associated with improved outcomes was the health provider tools. Our toolset included tips on how to include the patient’s healthcare provider in their quit smoking plan. In our randomized trial, we are also implementing an e-referral system, using which providers can refer smokers. Included in the e-referral system are proximal reports to the providers, including the patients’ engagement with Decide2Quit.org. A secure messaging function using which the provider and their patients can interact with the provider is also being developed. Such function may further enhance the communication between the provider and their patients, and may lead to improved six months cessation outcomes.
We also found that the Library which provided information on multiple smoking related topics was associated with improved outcomes. One way to enhance the library content may be to include user generated content, such as stories on how a patient used different tools at their disposal to quit smoking. While use of a single component was not significant associated with the six months outcome, we found that the summary score was significantly associated (OR=2.06, 95% CI = 1.02, 4.14), implying that smokers who use two or more components were more likely to quit smoking.
Our study is the first to test the use of asynchronous messaging with a Tobacco Treatment Specialists in web environments. The results were surprising that the use of Tobacco Treatment Specialists was negatively associated with six months cessation. This was surprising because we studies of counseling have consistently showed improvements in smoking cessation outcomes [39, 40, 41]. A meta-analysis of nine studies showed that counseling was effective in improving outcomes compared with usual care (1.37 (1.26–1.50)) . One reason could be confounding by indication; our patients having the most difficulty with quitting smoking self-selected to use the Tobacco Treatment Specialists function. However, even after adjusting for demographics and smoking characteristics, the effect remained. More research is needed to understand this negative association. We are currently looking at the messages between the Tobacco Treatment Specialists and the smokers and communicating with our Tobacco Treatment Specialists about improving the function.
This resource effect study was observational; therefore, we cannot directly infer causality. However, despite having a limited sample size of 204 patients, our study yielded important findings. Our messaging system is also limited by the potential that smokers may not have received the emails we sent them. We were unable to confirm email delivery. We also collected a limited number of characteristics for these smokers; the samples may also vary on important, unmeasured characteristics. We only evaluated the impact on one intervention (Decide2Quit.org) preventing strict generalizability to other technology interventions.
We also assessed smoking cessation using only self-report, not biochemical verification. In a smoking cessation study, there is always misclassification in self-report; however, the need for biochemical verification of nicotine depends on the intervention . Studies that are in-person and intense generally have more misclassification because of the personal connection between the smoker and the counselor and therefore require biochemical verification. Low intensity and light-touch studies have less misclassification. In these studies, not only is biochemical verification less needed, it also is more difficult to do because of the light touch.
Resource effect studies are useful in evaluating technology interventions before they are conducted to identify areas of improvement. Our study suggests that different functions may have a differential impact on the patient. The results of this study helped us identify several areas of improvement.
Funding for the study was received from the US National Cancer Institute, grant 1 R01 CA129091.