PMCCPMCCPMCC

Search tips
Search criteria 

Advanced

 
Logo of jmirEditorial BoardMembershipSubmitCurrent IssueArchiveReviewSearchAboutJMIR PublicationsJMIR
 
J Med Internet Res. 2017 September; 19(9): e312.
Published online 2017 September 6. doi:  10.2196/jmir.7449
PMCID: PMC5607435

Effectiveness of a Web-Based Tailored Intervention With Virtual Assistants Promoting the Acceptability of HPV Vaccination Among Mothers of Invited Girls: Randomized Controlled Trial

Monitoring Editor: John Torous and Tijn Kool
Reviewed by Maria Grandahl, Jason Glanz, and Nicola Bragazzi

Abstract

Background

In 2010, the human papillomavirus (HPV) vaccination was introduced in the Dutch National Immunization Program for 12-year-old girls, aiming to reduce the incidence of cervical cancer in women. HPV vaccination uptake turned out to be lower than expected: 61% versus 70%, respectively. Mothers were shown to play the most important role in the immunization decision about this vaccination. They had also expressed their need for interactive personal information about the HPV vaccination over and above the existing universal general information. To improve the effectiveness of the existing education about the HPV vaccination, we systematically developed a Web-based tailored intervention with virtual assistants providing mothers of girls to be invited with tailored feedback on their decision making about the HPV vaccination.

Objective

The aim of this study was to evaluate the effectiveness of the Web-based tailored intervention for promoting HPV vaccination acceptance by means of a randomized controlled trial (RCT).

Methods

Mothers were recruited via the Dutch vaccination register (Praeventis) (n=36,000) and three Web-based panels (n=2483). Those who gave informed consent (N=8062) were randomly assigned to the control (n=4067) or intervention condition (n=3995). HPV vaccination uptake, as registered by Praeventis once the HPV vaccination round was completed, was used as the primary outcome. Secondary outcomes were differential scores across conditions between baseline (before the provided access to the new tailored intervention) and follow-up (just before the first vaccination) regarding the mothers’ degree of informed decision making (IDM), decisional conflict, and critical determinants of HPV vaccination uptake among which are intention, attitude, risk perception, and outcome beliefs.

Results

Intention-to-treat analysis (N=8062) showed a significant positive effect of the intervention on IDM, decisional conflict, and nearly all determinants of HPV vaccination uptake (P<.001). No effect was found on uptake (P=.60). This may be attributed to the overall high uptake rates in both conditions. Mothers evaluated the intervention as highly positive, including the website as well as the virtual assistants that were used to deliver the tailored feedback.

Conclusions

This computer-tailored intervention has the potential to improve HPV vaccination acceptability and IDM and to decrease decisional conflict among mothers of invited girls. Implications for future research are discussed.

Trial Registration

Trialregister.nl NTR4935; http://www.trialregister.nl/trialreg/admin/rctview.asp?TC=4935 (Archived by WebCite at http://www.webcitation.org/6srT7l9EM)

Keywords: vaccination, health promotion, web-based intervention computer-tailoring, randomized controlled trial

Introduction

Worldwide, cervical cancer is the third most common cancer in women [1]. Persistent infection by the human papillomavirus (HPV) is the causative agent of cervical cancer [2]. In the Netherlands, yearly 750 new cases of cervical cancer are detected, of which 242 are with fatal consequences [3], despite the presence of a national cervical cancer screening program for women aged 30 to 60 years [4]. In 2008, the Health Council advised the Ministry to include the HPV vaccination for girls aged 12 years in the National Immunization Program (NIP) [5]. Initial implementation started with a catch-up campaign in 2009 for girls aged 13 to 16 years. From 2010, new cohorts of 12-year-old girls have been invited by the NIP to receive the HPV vaccination on a yearly basis. The municipal health services organize local sessions for group-based HPV vaccination, usually at large venues. This restricts the opportunity for interaction between the parent and girl with the professional. The vaccination itself is given by young health professionals (ie, medical doctors, nurses, or doctor’s assistants). The vaccination is voluntary and is offered free of charge. Complete vaccination includes 2 injections with a 6-month interval.

So far, HPV vaccination uptake in the Netherlands has remained lower than expected: 61% uptake in 2016 while 70% was targeted [6]. Research has indicated that mothers play the most important role in decision making about the girls’ immunization [7]. Currently, the regular invitation for the HPV vaccination comprises an introduction folder and a link to a website providing universal information about HPV and HPV vaccination. However, research has already indicated that mothers feel more in need for interactive personal information about the HPV vaccination, over and above this universal information [8]. To improve the existing educational strategy targeting HPV vaccination uptake, we developed a computer-tailored intervention with virtual assistants using the 6-step Intervention Mapping protocol for developing theory- and evidence-based health promotion interventions [9]. The intervention was aimed at Dutch mothers of girls to be invited for the HPV vaccination in 2015 (ie, girls born in 2002).

To date, only few tailored interventions to encourage HPV vaccination have been tested [10-13]. Three of these showed positive results. Hopfer [11] found that HPV vaccination uptake doubled among participants who were exposed to a culturally tailored video, compared with controls. Gerend and colleagues [12] found that, compared with general information, information tailored to the individual participant’s perceived barriers increased HPV vaccination intention. Grandahl and colleagues [13] found that an intervention delivered individually, that is face-to-face, by school nurses positively affected beliefs toward HPV prevention as well as vaccination uptake. However, to our knowledge, effective interventions promoting HPV vaccination that can reach large groups at relatively low costs (eg, Web-based tailored interventions) [14] do not exist yet.

Social cognitive determinants of the mothers’ decision making about their daughters’ HPV vaccination that appeared both relevant and changeable were selected as targets for developing the intervention [9]. These included HPV vaccination-related intention, attitude, outcome beliefs, risk perception, anticipated regret, subjective norms, habit, relative effectiveness of the HPV vaccination, and self-efficacy [7,8,15]. These determinants appeared to account for large proportions of variance in the mothers’ decision-making outcome (80-82%) [7]. Also, large proportions of the mothers do not actively acquire and process information about the pros and cons of the HPV vaccination and feel ambivalent about their decision [7,8]. This indicates that these decisions are based on rather unstable grounds, which makes them vulnerable for arguments challenging their initial attitudes and/or intention. Because informed decision making (IDM) is expected to make mothers less vulnerable for counterarguments, this was also chosen as a relevant intervention target. According to Marteau and colleagues [16], an informed decision is based in sufficient and relevant knowledge and in the congruence between the person’s values (ie, their attitude toward the HPV vaccination) and the behavioral outcome (ie, whether mothers had their daughters vaccinated against HPV). Consequently, knowledge was also targeted by the intervention. In addition, decisional conflict was selected as a target as this appears strongly related to IDM, with the possibility that decisional conflict may arise when feeling uninformed [17].

The aim of this study was to assess the effectiveness of the Web-based tailored intervention with virtual assistants on HPV vaccination uptake among the participants’ daughters (primary outcome). Secondary outcomes were the mothers’ degree of IDM, decisional conflict, and the social cognitive determinants of decision making about the daughters’ HPV vaccination uptake (eg, attitude, intention, and beliefs). When compared with the control condition, significantly positive effects were expected in the intervention condition with respect to HPV vaccination uptake, social cognitive determinants of the mothers’ decision making about the vaccination, levels of informed decision making, and levels of decisional conflict.

Methods

Participants

Mothers were randomly recruited from Praeventis, the Dutch National Immunization Register, and three Internet panels. The latter was to assure a high response rate [7]. This gave us the opportunity to assess differential effects of the intervention under (1) more controlled conditions (ie, panel sample) and (2) more naturalistic conditions as will be the case when the intervention has become part of the national implementation strategy (ie, Praeventis sample). This provided a basis for inferences concerning the intervention’s efficacy and effectiveness [18]. This is also why we did not reward the mothers from the Praeventis sample. The panel members received a small financial reimbursement for each survey that they completed (2-3 euros per survey). The amount of money received was panel specific. Extra financial reimbursement was provided to those in the intervention condition who completed all surveys (1-3 euros extra). In total, the participants in the control group could receive between 4 and 6 euros (completing baseline and follow-up), whereas participants in the experimental group could receive between 5 and 9 euros (completing baseline, intervention, and follow-up). Panel members were prestratified by region to ensure geographic diversity.

Power Calculation

Power analyses indicated that 1200 mothers per sample (ie, Praeventis and panels) were needed at baseline (600 per arm) to detect a 10% difference in HPV vaccination uptake between the intervention and control group, and small effects on the continuous secondary outcomes (Cohen d=0.10-0.30), with a power of 0.80, a two-sided alpha of .05, and an expected dropout of 30% at the last survey. Given previous experiences [7,8], a response rate of 3% was expected in the Praeventis sample. A total of 36,000 mothers were initially invited to participate via Praeventis and 1200 mothers via the Internet panels.

Design

This study, approved by the Ethical Committee of the VU Medical Center in Amsterdam, was conducted between January 2015 and March 2015. Effectiveness was evaluated by a 2-arm randomized controlled trial (RCT). Intervention effects on HPV vaccination uptake were assessed objectively using Praeventis. Because invited girls were given the opportunity to catch up on their missed HPV vaccinations, complete data on uptake were only available 18 months after baseline (ie, July 2016). The effects on secondary outcomes were examined by two Web-based surveys; at baseline, just before they had access to the experimental education, and at follow-up, just before they received the first HPV vaccination (time intervals around 2 months). Participants in the intervention condition were invited to visit the Web-based intervention between baseline and follow-up. Participants in both arms had access to the universal information about the HPV vaccination as part of the regular invitation for the HPV vaccination.

Procedure

An invitation letter to participate in the study was sent in January 2015 by postal mail to the Praeventis sample and by email to the panel sample. This letter included information about the study, a link to a secured website, and a unique code for entrance to the baseline survey. The same code was used for entering the follow-up survey and for gaining access to the tailored intervention (only participants in the intervention condition). Securing the website and providing unique codes was done to reduce the risk of spillover effects. The mothers in the intervention condition were explicitly requested not to share the link with others. On the website, the participants were assured of their privacy, confidentiality, and security in handling their responses and were informed that they could withdraw from participation at any time. Participants were then asked to provide informed consent and to give us permission to derive their daughters’ HPV vaccination status from Praeventis. After having provided informed consent, participants were randomly assigned to either the control or intervention condition. A reminder was sent 1 week after the first invitation. One week after the reminder, participants in the intervention condition received an email inviting them to visit the Web-based tailored feedback. Two weeks after this invitation, a reminder was sent. The website could be visited until the invitation to complete the follow-up questionnaire, 8 weeks after the initial invitation for the baseline questionnaire. All participants were given 2 weeks to complete the follow-up survey; a reminder was sent after the first week. These timelines (see Figure 1) were chosen as these fit with the standard procedures for the HPV vaccination in the Netherlands.

Figure 1
Timeline of data collection for the control and intervention group.

Intervention

The intervention consisted of a website providing mothers with tailored feedback from 2 virtual assistants, one being visualized in Figure 2. In Multimedia Appendix 1, a selection of screenshots of the website is presented. Computer-tailoring was the basic method for change and fitted the outcome of a previously conducted needs assessment indicating that the mothers preferred personalized feedback [8]. Tailoring is a health communication strategy in which messages are individualized to the person’s preferences and needs [19]. Meta-analyses have shown computer-tailored interventions to be more effective than universal interventions in achieving behavioral change outcomes (eg, [20,21]). Examples of change techniques that were used in addition to tailoring were consciousness raising (targeting risk perception) [22], belief selection (targeting beliefs) [23], active learning (targeting knowledge retention) [24], and motivational interviewing (targeting decisional conflict and attitude change) [25]. Before completing the concept intervention, we experimentally pretested three different intervention components [26,27] and conducted three focus groups for pretesting the prototype’s feasibility.

Figure 2
Screenshot of the decisional balance and the mother-like virtual assistant on the website.

Also, innovative was the use of 2 virtual assistants for delivering the tailored feedback; a mother- and doctor-like assistant. They provided opportunities for two-way interactions and for creating a highly personal experience. The added value of using a virtual assistant over a text and/or picture-based website is that it improves information recall [28], transfer of learning [29], amount of learning [30], self-efficacy expectations, literacy, and behavioral change [31-33].

The website consisted of four menu options: (1) two-sided information about the HPV vaccination, (2) a decisional balance, (3) practical background information, and (4) frequently asked questions (see Multimedia Appendix 1). In the first menu, mothers were able to collect tailored information about the HPV vaccination such as information about the risk of contracting an HPV infection, which may cause cervical cancer, as well as the risks and effectiveness of the HPV vaccine. In the second menu, a decisional balance gave mothers the opportunity to weigh their perceived pros and cons to balance the mothers’ position toward vaccinating versus not vaccinating the daughter (see Figure 2). In the third menu, mothers received practical information such as how and where to get the HPV vaccination and how to talk to their daughter and/or partner about the HPV vaccination.

Outcome Measures

Primary Outcome: HPV Vaccination Uptake

HPV vaccination uptake was derived from Praeventis, which was registered as having received no, 1, or 2 injections. We dichotomized HPV vaccination uptake into having received no HPV injection (0=not vaccinated) versus having received 1 or 2 HPV injections (1=vaccinated), as data showed that the determinants of HPV vaccination contrasted in these groups contrasted the most.

Secondary Outcomes: IDM and Decisional Conflict

According to Marteau and colleagues [16], a choice is considered to be informed when people have sufficient and relevant knowledge (knowledge) and when the person’s values (attitude) and behavior match. As such, IDM is usually expressed dichotomously (eg, see [34]). However, as we think the selection of the cut-off points is somewhat arbitrary, yet critical for the outcome [35], we also constructed a continuous measure for IDM. Post hoc analyses showed the correlation to be high between the two (Spearman ρ [rho]=.78). Both scores were derived from the Multi-dimensional Measure of Informed Choice [16,34,36].

By constructing the dichotomous IDM, the mother was classified as an informed decision maker, if (1) she had sufficient knowledge and (2) her attitude was consistent with the behavior (HPV vaccination uptake). Knowledge was considered to be sufficient when it was higher than the baseline mean score. A decision was considered to be consistent when the mother’s attitude was positive (higher than 4 on a 7-point scale) and the daughter was vaccinated, or when her attitude was negative (lower than 4 on a 7-point scale) and her daughter was not vaccinated. Any other combination was categorized as an uninformed decision.

By constructing the continuous measure of IDM, we first recoded the attitude scale into a scale ranging from −3 (extremely negative) to 3 (extremely positive). Consistency (ranging from −3 to 3) resulted from multiplying this attitude score with uptake (−1=no injection vs 1=1 or 2 injections). Then, the resulting consistency score was recoded from 0 (least consistent) to 6 (most consistent). Next, knowledge (−8 to 8, see Table 1) was recoded into a scale ranging from 0 (least knowledgeable) to 8 (most knowledgeable); the original scores below 0 were all recoded to 0. Finally, the continuous scale for IDM was computed by multiplying knowledge (range 0-8) with consistency (range 0-6), resulting in a scale ranging from 0 (not/least informed) to 48 (most informed).

Table 1
Overview of primary and secondary outcome measures.

Decisional conflict was measured using the “Uncertainty” subscale of the Decisional Conflict Scales [37], which included three 7-point scaled items (eg, “as regards to the HPV-vaccination, I felt sure about my choice) (1=completely disagree to 7=completely agree). Internal consistency was high (Cronbach alpha=.94).

Determinants of HPV Vaccination Uptake

Composite scores were computed for determinants of HPV vaccination uptake in case the scaled items showed internal consistency (Cronbach alpha >.60 or Pearson r>.64). See Table 1 for an overview of the primary and secondary outcome measures. All scores on scaled items were averaged into a scale because they showed sufficient internal consistency (Cronbach alpha ≥.78/Pearson r ≥.64); Cronbach alpha was used for scales consisting of more than 2 items, whereas Pearson r was used for scales consisting of 2 items; items with an (R) were reverse coded.

Sociodemographics

Sociodemographics were modeled as background variables (ie, age, educational level, country of birth, and religion). Level of education referred to the highest level the mother had completed. Educational level was classified into low (less than secondary or vocational education), intermediate (secondary through preuniversity education) or high (professional or university education) [7,8]. Country of birth was dichotomized into “Netherlands” versus “other,” as in our sample only 562 (6.97%) of 8062 (100%) mothers appeared to be born in a country other than the Netherlands. Religion was measured by asking the mothers about their religious convictions (Protestant, Roman Catholic, Muslim, Jewish, Buddhist, Hindu, other, or no religion). This was later classified as “Protestant” versus “not Protestant” as Protestants most refrain from vaccination compared with other religious or nonreligious groups in the Netherlands [7,8].

Subjective Program Evaluation and Objective Program Use

Subjective program evaluation was assessed at follow-up for mothers in the intervention condition by asking them to evaluate the website and the virtual assistants on a 10-point scale; the higher the score, the more positive the evaluation. Objective program use was evaluated by computer logs assessing the number of visits and amount of time logged in per session. If participants logged in more than once, these were summed.

Statistical Analyses

Descriptive statistics were used to describe the baseline sample. For analyzing the effects of the intervention, we used intention-to-treat (ITT) instead of complete case analysis. By using ITT, power increases while the risk of bias possibly caused by selective dropout decreases [42]. To deal with missing data, we applied multiple imputation by chained equations [42,43]. There were 15 imputed datasets generated using the predictive mean matching algorithm in Statistical Package for the Social Sciences (SPSS, IBM Corp) . The results from the imputed datasets were pooled together using Rubin’s rules [37]. Convergence of the imputations was checked by inspecting the iteration plots.

Intervention effects were examined by logistic and linear regression analyses (for dichotomous and continuous variables, respectively) by using the outcome at the follow-up as the criterion and the outcome-score at baseline and condition as the independent variables [44]. In view of multiple testing, an effect was considered significant when P<.003 (Bonferroni corrected alpha=.05/15 factors). The odds ratio was used as an indicator for effect size (Bonferroni corrected alpha=.05/14 factors). Effect sizes of the linear regressions were calculated in R (R Development Core Team) [45] using Cohen ]ƒ2 (R2 including the outcome at baseline and condition/1− R2 only including the outcome at baseline). These were interpreted as 0.02=small, 0.15=medium, 0.35=large [46]. Complete case analyses were performed as a sensitivity check for substantial differences with the results based on ITT.

Furthermore, we performed exploratory moderation analysis (Bonferroni corrected alpha=.05/15) to examine differences in effects regarding sociodemographics (ie, age, country of birth, education level, and religion) and sample (ie, Praeventis vs panels) by including a two-way interaction term (eg, condition × sample) in each of the aforementioned regression analyses [47]. In addition, we explored whether intention (at baseline) moderated the effects found in the primary and secondary outcomes. For this, intention was divided into three subgroups: (1) mothers with a negative intention (scores below half a standard deviation (SD) below the centered mean score of intention at baseline), (2) mothers who were hesitating (scores between half an SD below and above the centered mean of intention at baseline), and (3) mothers with a positive intention (scores more than half an SD above the centered mean score). Finally, subjective evaluations and objective use of the program were assessed by using descriptive analysis. IBM statistical package SPSS version 23 was used for analyzing the data [44].

Results

Response Rates and Attrition

We invited 36,000 participants via Praeventis and 2483 via the panels. A flow diagram of the recruitment and response is shown in Figure 3. From the 9124 participants who were initially randomized at T0, 8593 (94.18%; 4277 in the intervention group and 4316 participants in the control group) completed the baseline questionnaire, whereas 4678 (51.27%; 2197 in the intervention group and 2481 participants in the control group) completed the follow-up questionnaire 8 weeks later (T1). Dropout analysis showed significantly more dropout in the Praeventis sample. There was also selective nonresponse with regard to condition (ie, more dropout in the intervention condition), sociodemographics (ie, more dropout in those not born in the Netherlands, and in those low in education), HPV vaccination uptake (more dropout in mothers having a daughter not being vaccinated), and secondary outcomes (ie, more dropout in mothers with low levels of IDM, risk perception, and self-efficacy and in mothers with high attitude scores). In total, 1067 participants were excluded (564 in the intervention group and 503 participants in the control group), as they did not meet the inclusion criteria (ie, being a mother of a daughter born in 2002 and aged 24-62 years) or were found to be duplicates across the two samples. The final sample for ITT analysis consisted of 8062 mothers: 3995 mothers in the intervention condition versus 4067 in the control condition.

Figure 3
Flow diagram of the recruitment and response of study participants. Notes: (1) Participants could be excluded based on multiple criteria (eg, a male with an invalid age). Therefore, the total number of Praeventis participants excluded differed from the ...

Sample Description

See Table 2 for the sample description. As there were no data available on sociodemographics of the population from which the sample was derived (ie, Dutch mothers of girls aged 12 years in the Praeventis database), we were unable to assess the representativeness of the study sample. The mean age of mothers was 43.64 years (SD 4.25). On average, mothers had a positive intention toward the HPV vaccination of their daughter at baseline (mean 5.35 [SD 1.69]). Compared with the national HPV vaccination uptake, uptake was higher in the study sample (n=59,866; 60.98% vs n=5880; 72.93%, respectively).

Table 2
Sample description (N=8062). In case of missing values, the number of missing values (Nmissing) was presented. By reporting 2 decimal points for the percentages, summing the percentages for each category up may differ from 100%.

Intervention Effects on Primary and Secondary Outcomes

In Table 3, an overview of the intervention effects on the primary and secondary outcomes is given. A higher mean score means a higher X (eg, more positive attitude) except for decisional conflict; here, a higher mean score means less decisional conflict If an odds ratio (OR) is higher than one, this means that the higher the score on a factor, the higher the outcome of IDM or the higher the chance of the daughter being vaccinated. If an OR is less than one, this means that the higher the score on a factor, the lower outcome of IDM or the lower the chance of the daughter being vaccinated. ITT analyses showed that there was no effect of the intervention on HPV vaccination uptake (odds ratio, OR=1.03, P=.60). The intervention had a significant positive effect on all secondary outcomes (P<.001), except for risk perception when not vaccinated, anticipated regret, and self-efficacy (P=.01; P=.01; P=.03, respectively). Compared with the control group, at follow-up, mothers in the intervention group were more informed (dichotomous measure: OR=1.28, P<.001; continuous measure: beta=1.72, P<.003), experienced less decisional conflict (beta=.21, P<.003), were more intended to vaccinate their daughter (beta=.18, P<.004), had a more positive attitude toward vaccinating their daughter (beta=0.15, P<.004), had more positive beliefs (eg, beliefs about the safety and effectiveness of the HPV vaccination; beta=.12, P<.003), had a lower risk perception when they imagined that their daughter was vaccinated (beta=−.11, P<.003), perceived more positive subjective norms (beta=.82, P<.003), reported a higher relative effectiveness (beta=.46, P<.003), and had more knowledge (beta=.35, P<.003). Effect sizes were small (see Table 3). Results from complete case analyses were similar, except for an additional effect of the intervention on anticipated regret and self-efficacy.

Table 3
Effects of the intervention on the outcome measures according to intention-to-treat analyses (N=8062).

Moderation of Intervention Effects

Regarding sociodemographics, no significant interaction effects on any of the outcome measures were found for country of birth (P ≥.08) or religion (P ≥.08). For educational level, we found an interaction effect with condition on relative effectiveness (beta=.59, P<.001): the intervention had more positive effects on relative effectiveness for those with high education compared with those low in education. There was no significant interaction between condition and sample on any of the outcome measures (P ≥.04). For the interaction effects between intention at baseline and the outcome measures, See Table 4. If an OR is higher than one, this means that the higher the score on a factor, the higher the outcome of IDM or the higher the chance of the daughter being vaccinated. If an OR is less than one, this means that the higher the score on a factor, the lower outcome of IDM or the lower the chance of the daughter being vaccinated. In the first 2 columns, the reference category is those with a negative intention. For a comparison between those in doubt (reference category) and a positive attitude, see the third column. Significant interaction effects between intention at baseline and condition were found on intention, attitude, decisional conflict, subjective norm, and relative effectiveness . For mothers who had a negative intention, the intervention had more positive effects on intention and relative effectiveness compared with mothers who were doubting (beta=.26, P=.002; beta=.39, P=.001, respectively) or had a positive intention (beta=.40, P<.001; beta=.53, P<.001, respectively). In addition, for mothers with a negative intention, the intervention had a more positive effect on attitude (beta=.21, P=.001) and on subjective norms (beta=1.64, P<.001) compared with mothers with a positive intention. For mothers who were doubting, the intervention had more positive effects on decisional conflict compared with mothers who had a negative intention (beta=.26, P=.001). No differences on intervention outcomes were found between mothers who were doubting and mothers who had a positive intention (P ≥.004).

Table 4
Moderation effects of intention subgroups on the outcome measures according to the intention-to-treat analyses (N=8062).

Subjective Program Evaluation and Objective Program Use

Mothers in the intervention condition evaluated the website with a 7.6 (SD=1.36) and the virtual assistants with a 7.4 (SD=1.53). According to the computer logs, 2509 (62.80%) of the 3995 (100.00%) invited mothers logged on to the website. Of these, 1835 (73.14%) visited the website once, 498 (19.84%) visited twice, and 176 (7.02%) more than twice. On average, mothers spent 22 min on the website (SD=13 min).

Discussion

Principal Findings

This study investigated the effectiveness of a Web-based tailored intervention with virtual assistants promoting HPV vaccination acceptability among mothers of invited girls. As hypothesized, positive intervention effects were found with respect to the social cognitive determinants of the mothers’ decision making about the vaccination (eg, HPV vaccination-related intention, attitude, and outcome beliefs), levels of IDM, and levels of decisional conflict. The positive effect of tailored education on HPV vaccination intention was also found by Gerend et al [12] among young women. However, they did not assess other determinants of HPV vaccination acceptability (next to intention) nor did they measure levels of IDM, levels of decisional conflict, or actual HPV vaccination uptake.

The findings described above suggest that this intervention has potential in promoting HPV vaccination acceptability and IDM. This is important, given the currently moderate HPV vaccination uptake and the fact that large proportions of the mothers do not actively acquire and process information about the pros and cons of this HPV vaccination and that many feel ambivalent about the decision [7,8]. Less informed decisions are decisions constituted in rather instable beliefs that are susceptible to counterarguments. Nowadays, counterarguments are all around on the Internet and Web-based social media [48]. Because the intervention initiated active processing of verifiable information about the risks and effectiveness of the HPV vaccination, it inoculates mothers with arguments that become accessible at the moment they are confronted with (new) information that might challenge their initial positive attitudes and intentions [49,50].

No effects were found on mothers’ perceived risk of their daughter getting cervical cancer without the HPV vaccination, anticipated regret in case their daughter would get cervical cancer later in life, and self-efficacy. As for risk perception, baseline scores indicated that the mothers overestimated the probability of contracting cervical cancer to a great extent when taken into account the actual population incidence [51]. Because the intervention presented mothers this actual low probability of attracting cervical cancer, it seems unlikely that their perceived risk was brought to higher levels. The lack of effect on anticipated regret might be explained by the fact that we removed the intervention component specifically targeting anticipated regret. This was removed because our pilot studies and focus groups revealed that resistance was evoked by asking mothers how much regret they would have if they did not vaccinate their daughter against HPV and their daughter developed cervical cancer later in life. Also, emphasizing the impact of cervical cancer might be fear-arousing, which, in turn, may have been detrimental for exploring and processing other information provided by the program [52]. Finally, it appeared that we encountered a ceiling effect for self-efficacy as the scores at baseline among both groups were above 6 on a 7-point scale.

No effects of the tailored intervention were found on HPV vaccination uptake. This is contrary to both our expectations and to what has been found by others; Hopfer found that among female college students, HPV vaccination uptake doubled after they were exposed to a tailored video compared with controls (22% vs 12%, respectively) [11]. However, in their study, vaccination rates were quite low in the control condition (12%), leaving much room for improvement. In our study, however, the uptake rates were high in both conditions (intervention: n=3995; 73.17% and control: n=4067; 72.71%) especially when compared with the national Dutch uptake (n=59,866; 60.98) [6]. This may explain why we, as opposed to Hopfer, did not find an effect on HPV vaccination uptake. After all, we did find a larger increase in intention among mothers in the intervention compared with the control condition, and according to theory [23] and empirical findings [53], intention is an important predictor of (HPV vaccination) behavior.

Mothers evaluated the intervention as positive, specified by the high subjective evaluation of both the website (7.6 on a 10-point scale) and the virtual assistants that were used to deliver the tailored feedback (7.4 on a 10-point scale). Objective program use was also high, with 62.80% (n=2509) of the invited mothers having visited the website. In addition, on average, they spent quite some time on the website (22 min). Taken together, the intervention has potential for broad national dissemination and implementation.

Furthermore, subgroup analysis with 3 intention groups (ie, negative, in doubt, and positive) showed that the intervention had the most positive effects on decisional conflict for mothers who were doubting. For mothers with a negative intention, the most positive effects were found on intention, attitude, and subjective norms. This is promising, as for a population-wide program it is relevant to guide those in doubt toward making an informed choice without decisional conflict and to persuade those having negative intentions toward vaccination. Fortunately, we did not find any adverse effects of the intervention in mothers with a negative intention, such as a further decrease in their intention or attitude.

Methodological Considerations

There are two methodological considerations. First, with large sample sizes, as in this study, even small effects can become statistically significant [54]. However, the positive intervention effects were consistently found on almost all outcomes. In addition, the large sample provided us with sufficient data for conducting analyses on subsamples (ie, based on sociodemographics, sample, and three intention groups) while maintaining sufficient levels of power [54]. The effect sizes that we found are in line with other Web-based interventions targeting health behavior outcomes [55]. We believe these, even small effects, are of relevance in public health as they become substantial at the population level. In addition, the intervention was of help for those in doubt and did not have any detrimental effect. We therefore find this Web-based tailored program is a substantial step forward in improving both research and practice in the context of the promotion of HPV vaccination acceptability.

Second, we used a scale comprising two items of the Self-Report Habit Index for measuring habit [41]. This accounted for the extent to which getting the HPV vaccination was something mothers did (1) naturally and (2) without thinking. The positive intervention effect on the composite measure might indicate that the intervention induced perceptions about the HPV vaccination as something you take for granted, without thinking. The latter is unwanted, considering the aim of initiating active processing of information about the vaccination. Fortunately, secondary analysis separating the two items showed that there was a positive effect of the intervention on the first item (ie, “naturally”) but not on the second (ie, “without thinking”). In retrospect, the label “habit” attributed to the two-item operationalization appears misleading, though the intervention strengthened the mothers’ belief about getting the HPV vaccination as something natural.

Strengths and Limitations

Important strengths of this study are the randomized controlled design, adequate sample sizes, and the reliable objective assessment of HPV vaccination uptake. Furthermore, the systematic, stepwise development of the intervention was a notable strength, as well as the mothers’ positive subjective evaluation of the intervention and the objectively assessed high level of program use.

Some limitations of this study should be considered as well. First, the study was subject to a considerable amount of attrition. Unfortunately, attrition is quite common in studies on Web-based interventions [56]. Dropout analysis showed that dropout was selective. For instance, there were higher dropout rates in the intervention condition, which has also been reported for other (tailored) interventions [57-59]. In this study, we handled the missing data and selective dropout by using multiple imputation [60]. Results from the complete case analyses only slightly differed from those from the ITT analyses. Thus, it seems unlikely that the observed effects are spurious or due to selective dropout. Second, caution is needed when generalizing the results of this study to the general population (ie, Dutch mothers of girls aged 12 years) because we were unable to check the sample’s representativeness. However, we did not find any differences in effectiveness of the intervention in specific subgroups of participants, as indicated by the conducted moderation analyses with sociodemographics.

Conclusions and Recommendations

The study findings suggest that this Web-based tailored intervention has the potential to improve both HPV vaccination acceptability and IDM, and decrease decisional conflict among mothers of invited girls. Therefore, we recommend nationwide dissemination and implementation. Furthermore, we recommend developing (tailored) interventions targeting the daughter and the mother’s partner, as these have appeared to be important social referents [7,8]. Research has indicated that tailoring the intervention could have beneficial effects for girls, as they also expressed their need for interactive and personal information about the HPV vaccination [8]. This still needs to be investigated for the partners. In addition, boys may also become a relevant target group. In other countries, such as Australia, boys are already included in the national immunization program. This may contribute to the achievement of herd immunity and to a reduction of the global burden of a variety of HPV-related cancers in women and men [61-63].

Acknowledgments

We would like to thank Petra Oomen (RIVM) and the panels (Veldkamp BV, Intromart GFK, and NGO Flycatcher) for their help with data collection. ZonMw (The Netherlands Organization for Health Research and Development) and NWO (Netherlands Organization for Scientific Research) support this study (grant-number: 50-51515-98-259).

Abbreviations

HPV
human papillomavirus
IDM
informed decision making
ITT
intention-to-treat analysis
NIP
National Immunization Program
RCT
randomized controlled trial
SD
standard deviation

Multimedia Appendix 1

Intervention screenshots.

Multimedia Appendix 2

CONSORT-EHealth (V 1.6.1).

Footnotes

Conflicts of Interest: None declared.

References

1. Ferlay J, Shin H, Bray F, Forman D, Mathers C, Parkin DM. Estimates of worldwide burden of cancer in 2008: GLOBOCAN 2008. Int J Cancer. 2010 Dec 15;127(12):2893–917. doi: 10.1002/ijc.25516. http://dx.doi.org/10.1002/ijc.25516. [PubMed] [Cross Ref]
2. Schiffman M, Castle PE. Human papillomavirus: epidemiology and public health. Arch Pathol Lab Med. 2003 Aug;127(8):930–4. doi: 10.1043/1543-2165(2003)127<930:HPEAPH>2.0.CO;2. [PubMed] [Cross Ref]
3. Bruni L, Barrionuevo-Rosas L, Albero G, Serrano B, Mena M, Gómez D, Muñoz J, Bosch F, de SS. Hpvcentre. [2017-04-19]. ICO Information centre on HPV and cancer (HPV Information centre) http://www.hpvcentre.net/statistics/reports/XWX.pdf webcite.
4. Braspenning J, Tacken M, Penders A, Van DHH, De Bakker D. Nivel. 2001. Opkomst bevolkingsonderzoek baarmoederhalskanker, 1997-1999 https://www.nivel.nl/node/2430?database=ChoicePublicat&priref=1459 webcite.
5. Gezondheidsraad Vara. 2008. Vaccinatie tegen baarmoederhalskanker https://zembla.vara.nl/pdf/GR%20baarmoederhalskanker.pdf webcite.
6. van Lier E, Oomen P, Giesbers H, van Vliet JA, Drijfhout I, Zonnenberg-Hoff I, de Melker H. Rivm. 2016. [2017-01-31]. Vaccinatiegraad Rijksvaccinatieprogramma Nederland: verslagjaar http://www.rivm.nl/Documenten_en_publicaties/Wetenschappelijk/Rapporten/2016/juni/Vaccinatiegraad_Rijksvaccinatieprogramma_Nederland_Verslagjaar_2016 webcite.
7. van KH, Otten W, Ruiter RA, Fekkes M, van SJ, Dusseldorp E, Paulussen TW. Determinants of HPV vaccination intentions among Dutch girls and their mothers: a cross-sectional study. BMC Public Health. 2013 Feb 06;13:111. doi: 10.1186/1471-2458-13-111. https://bmcpublichealth.biomedcentral.com/articles/10.1186/1471-2458-13-111. [PMC free article] [PubMed] [Cross Ref]
9. Bartholomew EL, Markham C, Ruiter R, Fernàndez ME, Kok G, Parcel GS. Planning health promotion programs: An Intervention Mapping approach, 4th edition. Hoboken, NJ: Wiley; 2016.
10. Bennett AT, Patel DA, Carlos RC, Zochowski MK, Pennewell SM, Chi AM, Dalton VK. Human papillomavirus vaccine uptake after a tailored, online educational intervention for female university students: a randomized controlled trial. J Womens Health (Larchmt) 2015 Nov;24(11):950–7. doi: 10.1089/jwh.2015.5251. http://europepmc.org/abstract/MED/26488269. [PMC free article] [PubMed] [Cross Ref]
11. Hopfer S. Effects of a narrative HPV vaccination intervention aimed at reaching college women: a randomized controlled trial. Prev Sci. 2012 Apr;13(2):173–82. doi: 10.1007/s11121-011-0254-1. [PubMed] [Cross Ref]
12. Gerend MA, Shepherd MA, Lustria ML. Increasing human papillomavirus vaccine acceptability by tailoring messages to young adult women's perceived barriers. Sex Transm Dis. 2013 May;40(5):401–5. doi: 10.1097/OLQ.0b013e318283c8a8. [PubMed] [Cross Ref]
13. Grandahl M, Rosenblad A, Stenhammar C, Tydén T, Westerling R, Larsson M, Oscarsson M, Andrae B, Dalianis T, Nevéus T. School-based intervention for the prevention of HPV among adolescents: a cluster randomised controlled study. BMJ Open. 2016 Jan 27;6(1):e009875. doi: 10.1136/bmjopen-2015-009875. http://bmjopen.bmj.com/cgi/pmidlookup?view=long&pmid=26817639. [PMC free article] [PubMed] [Cross Ref]
14. Neville LM, O'Hara B, Milat AJ. Computer-tailored dietary behaviour change interventions: a systematic review. Health Educ Res. 2009 Aug;24(4):699–720. doi: 10.1093/her/cyp006. http://her.oxfordjournals.org/cgi/pmidlookup?view=long&pmid=19286893. [PMC free article] [PubMed] [Cross Ref]
15. Pot M, Van Keulen H, Ruiter RA, Eekhout I, Mollema L, Paulussen TW. Motivational and contextual determinants of HPV-vaccination uptake: a longitudinal study among mothers of girls invited for the HPV-vaccination. Prev Med. 2017 Apr 04;100:41–9. doi: 10.1016/j.ypmed.2017.04.005. https://linkinghub.elsevier.com/retrieve/pii/S0091-7435(17)30125-1. [PubMed] [Cross Ref]
16. Marteau TM, Dormandy E, Michie S. A measure of informed choice. Health Expect. 2001 Jun;4(2):99–108. [PMC free article] [PubMed]
17. O'Connor AM, Jacobsen MJ, Stacey D. An evidence-based approach to managing women's decisional conflict. J Obstet Gynecol Neonatal Nurs. 2002;31(5):570–81. [PubMed]
18. Ernst E, Pittler MH. Efficacy or effectiveness? J Intern Med. 2006 Nov;260(5):488–90. doi: 10.1111/j.1365-2796.2006.01707.x. http://dx.doi.org/10.1111/j.1365-2796.2006.01707.x. [PubMed] [Cross Ref]
19. Kreuter MW, Wray RJ. Tailored and targeted health communication: strategies for enhancing information relevance. Am J Health Behav. 2003 Dec;27 Suppl 3:S227–32. [PubMed]
20. Lustria ML, Noar SM, Cortese J, Van Stee SK, Glueckauf RL, Lee J. A meta-analysis of web-delivered tailored health behavior change interventions. J Health Commun. 2013;18(9):1039–69. doi: 10.1080/10810730.2013.768727. [PubMed] [Cross Ref]
21. Krebs P, Prochaska JO, Rossi JS. A meta-analysis of computer-tailored interventions for health behavior change. Prev Med. 2010;51(3-4):214–21. doi: 10.1016/j.ypmed.2010.06.004. http://europepmc.org/abstract/MED/20558196. [PMC free article] [PubMed] [Cross Ref]
22. Prochaska J, Wright J, Velicer W. Evaluating theories of health behavior change: a hierarchy of criteria applied to the transtheoretical model. Appl Psychol. 2008;57(4):561–88. doi: 10.1111/j.1464-0597.2008.00345.x. [Cross Ref]
23. Fishbein M, Ajzen I. Predicting and changing behavior: The Reasoned Action Approach. New York: Taylor & Francis; 2010.
24. Petty R, Barden J, Wheeler S. The Elaboration Likelihood Model of Pursuasion: Developing health promotions for sustained behavioral change. In Diclimente RJ, Crosby RA. & Kegler M (Eds.), Emerging theories in health promotion practice and research, 2nd edition. San Francisco, CA: Jossey-Bass; 2009. pp. 185–214.
25. Miller W, Rollnick S. Motivational interviewing: Preparing people for change, 2nd edition. New York: The Guilford Press; 2002.
26. Whittingham J, Ruiter RA, Zimbile F, Kok G. Experimental pretesting of public health campaigns: a case study. J Health Commun. 2008;13(3):216–29. doi: 10.1080/10810730701854045. [PubMed] [Cross Ref]
27. Whittingham JR, Ruiter RA, Castermans D, Huiberts A, Kok G. Designing effective health education materials: experimental pre-testing of a theory-based brochure to increase knowledge. Health Educ Res. 2008 Jun;23(3):414–26. doi: 10.1093/her/cym018. [PubMed] [Cross Ref]
28. Beun R, de Vos E, Witteman E. Proceedings of the International Conference on Intelligent Virtual Agents. New York: Springer-Verlag; 2003. Embodied conversational agentsffects on memory performance and anthropomorphisation; pp. 315–9.
29. Gerend MA, Shepherd MA, Lustria MLA. Increasing human papillomavirus vaccine acceptability by tailoring messages to young adult women's perceived barriers. Sex Transm Dis. 2013 May;40(5):401–5. doi: 10.1097/OLQ.0b013e318283c8a8. [PubMed] [Cross Ref]
30. Baylor A, Kim Y. LNCS, intelligent tutoring systems, volume 3320. Berlin/Heidelberg: Germany Springer; 2004. Pedagogical agent design: the impact of agent realism, gender, ethnicity, and instructional role; pp. 592–603.
31. Jin SA. The effects of incorporating a virtual agent in a computer-aided test designed for stress management education: the mediating role of enjoyment. Comput Human Behav. 2010 May;26(3):443–51. doi: 10.1016/j.chb.2009.12.003. [Cross Ref]
32. Blanson HO, Rogers W, Fisk AD, Neerincx M, Lindenberg J, van der Mast CA. Usability of an adaptive computer assistant that improves self-care and health literacy of older adults. Methods Inf Med. 2008;47(1):82–8. http://europepmc.org/abstract/MED/18213433. [PMC free article] [PubMed]
33. Blanson HOA, van der Boog PJ, Lindenberg J, van der Mast CA, Neerincx MA, Zwetsloot-Schonk BJ. An online lifestyle diary with a persuasive computer assistant providing feedback on self-management. Technol Health Care. 2009;17(3):253–67. doi: 10.3233/THC-2009-0545. [PubMed] [Cross Ref]
34. Van der Pal S, Otten W, Detmar S. Evaluatie van de voorlichting aan ouders over de hielprik. TSG. 2010;88:449–53. doi: 10.1007/s12508-010-0679-7. [Cross Ref]
35. Altman DG, Royston P. The cost of dichotomising continuous variables. BMJ. 2006 May 6;332(7549):1080. doi: 10.1136/bmj.332.7549.1080. http://europepmc.org/abstract/MED/16675816. [PMC free article] [PubMed] [Cross Ref]
36. Michie S, Dormandy E, Marteau TM. The multi-dimensional measure of informed choice: a validation study. Patient Educ Couns. 2002 Sep;48(1):87–91. [PubMed]
37. Rubin D. Multiple Imputation for Nonresponse in Surveys. New York: Wiley; 1987.
38. Paulussen TG, Lanting CI, Buijs GB, Hirasing RA. Google. 2000. Ouders over het Rijksvaccinatieprogramma: Tevredenheid en vaccinatiebereidheid van ouders van jonge kinderen in Nederland http://bit.ly/2wEgIuL webcite.
39. Reiter PL, Brewer NT, Gottlieb SL, McRee A, Smith JS. Parents' health beliefs and HPV vaccination of their adolescent daughters. Soc Sci Med. 2009 Aug;69(3):475–80. doi: 10.1016/j.socscimed.2009.05.024. [PubMed] [Cross Ref]
40. Gerend MA, Weibley E, Bland H. Parental response to human papillomavirus vaccine availability: uptake and intentions. J Adolesc Health. 2009 Nov;45(5):528–31. doi: 10.1016/j.jadohealth.2009.02.006. [PubMed] [Cross Ref]
41. Verplanken B, Orbell S. Reflections on past behavior: a self-report index of habit strength. J Appl Social Pyschol. 2003 Jun;33(6):1313–30. doi: 10.1111/j.1559-1816.2003.tb01951.x. [Cross Ref]
42. Van Buuren S. Flexible Imputation of Missing Data. Boca Raton: Chapman Hall/CRC; 2012.
43. White IR, Royston P, Wood AM. Multiple imputation using chained equations: issues and guidance for practice. Stat Med. 2011 Feb 20;30(4):377–99. doi: 10.1002/sim.4067. [PubMed] [Cross Ref]
44. Van Breukelen GJ. ANCOVA versus change from baseline: more power in randomized studies, more bias in nonrandomized studies [corrected] J Clin Epidemiol. 2006 Sep;59(9):920–5. doi: 10.1016/j.jclinepi.2006.02.007. [PubMed] [Cross Ref]
45. R Development Core Team . R: A language and environment for statistical computing. Vienna: R Foundation for Statistical Computing; 2009.
46. Cohen J. Statistical power analysis for the behavioral science. Hillsdale, NJ: Lawrence Erlbaum Associates Publishers; 1988.
47. Fayers PM, King MT. How to guarantee finding a statistically significant difference: the use and abuse of subgroup analyses. Qual Life Res. 2009 Jun;18(5):527–30. doi: 10.1007/s11136-009-9473-3. [PubMed] [Cross Ref]
48. Kata A. Anti-vaccine activists, Web 2.0, and the postmodern paradigm--an overview of tactics and tropes used online by the anti-vaccination movement. Vaccine. 2012 May 28;30(25):3778–89. doi: 10.1016/j.vaccine.2011.11.112. [PubMed] [Cross Ref]
49. McGuire W. In: Advances in experimental social psychology, volume I. Berkowitz L, editor. New York: Academic Press; 1964.
50. Paulussen TG, Hoekstra F, Lanting CI, Buijs GB, Hirasing RA. Determinants of Dutch parents' decisions to vaccinate their child. Vaccine. 2006 Jan 30;24(5):644–51. doi: 10.1016/j.vaccine.2005.08.053. [PubMed] [Cross Ref]
51. Integraal Kankercentrum Nederland Cijfersoverkanker. 2016. [2017-01-31]. Cijfers over kanker http://www.cijfersoverkanker.nl/selecties/Dataset_1/img58906e45e92b1 webcite.
52. Ruiter RA, Abraham C, Kok G. Scary warnings and rational precautions: a review of the psychology of fear appeals. Psychol Health. 2001 Nov;16(6):613–30. doi: 10.1080/08870440108405863. [Cross Ref]
53. Hofman R, van Empelen P, Richardus JH, de Kok IM, de Koning HJ, van Ballegooijen M, Korfage IJ. Predictors of HPV vaccination uptake: a longitudinal study among parents. Health Educ Res. 2014 Feb;29(1):83–96. doi: 10.1093/her/cyt092. [PubMed] [Cross Ref]
54. Lin M, Lucas HC, Shmueli G. Too big to fail: large samples and the p-value problem. Inf Syst Res. 2013 Dec;24(4):906–17. doi: 10.1287/isre.2013.0480. [Cross Ref]
55. Webb TL, Joseph J, Yardley L, Michie S. Using the internet to promote health behavior change: a systematic review and meta-analysis of the impact of theoretical basis, use of behavior change techniques, and mode of delivery on efficacy. J Med Internet Res. 2010;12(1):e4. doi: 10.2196/jmir.1376. http://www.jmir.org/2010/1/e4/ [PMC free article] [PubMed] [Cross Ref]
56. Kohl LFM, Crutzen R, de Vries NK. Online prevention aimed at lifestyle behaviors: a systematic review of reviews. J Med Internet Res. 2013;15(7):e146. doi: 10.2196/jmir.2665. http://www.jmir.org/2013/7/e146/ [PMC free article] [PubMed] [Cross Ref]
57. Oenema A, Brug J, Dijkstra A, de WI, de VH. Efficacy and use of an internet-delivered computer-tailored lifestyle intervention, targeting saturated fat intake, physical activity and smoking cessation: a randomized controlled trial. Ann Behav Med. 2008 Apr;35(2):125–35. doi: 10.1007/s12160-008-9023-1. [PubMed] [Cross Ref]
58. Oenema A, Tan F, Brug J. Short-term efficacy of a web-based computer-tailored nutrition intervention: main effects and mediators. Ann Behav Med. 2005 Feb;29(1):54–63. doi: 10.1207/s15324796abm2901_8. [PubMed] [Cross Ref]
59. Spittaels H, De Bourdeaudhuij BI, Vandelanotte C. Evaluation of a website-delivered computer-tailored intervention for increasing physical activity in the general population. Prev Med. 2007 Mar;44(3):209–17. doi: 10.1016/j.ypmed.2006.11.010. [PubMed] [Cross Ref]
60. Twisk J. Applied Multilevel Analysis. Cambridge: Cambridge University Press; 2006.
61. Georgousakis M, Jayasinghe S, Brotherton J, Gilroy N, Chiu C, Macartney K. Population-wide vaccination against human papillomavirus in adolescent boys: Australia as a case study. Lancet Infect Dis. 2012 Aug;12(8):627–34. doi: 10.1016/S1473-3099(12)70031-2. [PubMed] [Cross Ref]
62. Wakeham K, Kavanagh K. The burden of HPV-associated anogenital cancers. Curr Oncol Rep. 2014 Sep;16(9):402. doi: 10.1007/s11912-014-0402-4. [PubMed] [Cross Ref]
63. Braakhuis BJM, Leemans CR, Visser O. Incidence and survival trends of head and neck squamous cell carcinoma in the Netherlands between 1989 and 2011. Oral Oncol. 2014 Jul;50(7):670–5. doi: 10.1016/j.oraloncology.2014.03.008. https://linkinghub.elsevier.com/retrieve/pii/S1368-8375(14)00098-0. [PubMed] [Cross Ref]

Articles from Journal of Medical Internet Research are provided here courtesy of Gunther Eysenbach