|Home | About | Journals | Submit | Contact Us | Français|
CME providers may be interested in identifying effective marketing strategies to direct users to specific content. The use of online advertisements to recruit participants for clinical trials, public health programs, and Continuing Medical Education (CME) has been shown to be effective in some but not all studies. The purpose of this study was to compare the impact of two marketing strategies in the context of an online CME cultural competence curriculum (www.c-comp.org).
In an interrupted time-series quasi-experimental design, two marketing strategies were tested: a) wide dissemination to relevant organizations over a period of approximately four months, and b) Internet paid search using Google Ads (five consecutive eight-week periods--control 1, cultural/ CME advertisement, control 2, hypertension/ content advertisement, control 3). Outcome measures were CME credit requests, Web traffic (visits per day, page views, pages viewed per visit), and cost.
Overall, the site was visited 19,156 times and 78,160 pages were viewed. During the wide dissemination phase, the proportion of visits requesting CME credit decreased between the first (5.3%) and second halves (3.3%) of this phase (p= .04). During the Internet paid search phase, the proportion of visits requesting CME credit was highest during the cultural/ CME advertisement period (control 1, 1.4%; cultural/CME ad, 4.3%; control 2, 1.5%; hypertension/content ad, 0.6%; control 3, 0.8%; p<.001). All measures of Web traffic changed during the Internet paid search phase (p<.01); however, changes were independent of the advertisement periods. The incremental cost for the cultural advertisement per CME credit requested was $0.64US.
Internet advertisement focusing on cultural competence and CME was associated with about a three-fold increase in requests for CME credit at an incremental cost of under $1; however, Web traffic changes were independent of the advertisement strategy.
Physicians are increasingly using Web-based continuing medical education (CME) to meet CME requirements. In 2008, accredited CME providers offered a total of 47,304 hours of instruction1 and data suggest that online CME comprises approximately seven to nine percent of CME utilized.2 Given that some studies suggest that Web-based CME improves knowledge, behaviors, and skills, 3, 4 CME providers may be interested in identifying effective marketing strategies to direct users to specific content.5, 6 However, to serve this function, physicians need to find relevant online CME activities. Effective marketing can be an important strategy for helping physicians locate and access online educational opportunities.5
The wide reach of the Internet makes online advertisement an attractive tool for recruiting physicians to CME websites. However, evidence for the effectiveness of online advertisement in the health care field is inconclusive. When used for recruiting participants for clinical trials7 and public health programs,8 it has been effective in some but not all studies.9 In addition, it can be costly. In a study to improve the recognition and management of intimate partner violence, promoting a free online CME activity with search engine advertisement and e-mail solicitation cost $30 to $80 per physician enrolled;10 while direct mail yielded 44% of the participants, it cost $143 per physician participant. We build on available evidence by focusing on the least expensive strategies and comparing two marketing strategies to recruit to an online CME program.
The present study is grounded on a conceptual framework—the Engel-Kollat-Blackwell (EKB) model of consumer behavior for online activities as modified by Darley et al.11 In this model, the decision making process has five main stages: problem recognition (“What do I need?”), search of alternative options (“Where can I find this information?”), alternative evaluation (“Which one should I select”), purchase (option selected), and outcomes (satisfaction). We focused on the search of alternative options aspect, where internal and external factors may influence the search behavior. In particular, personal characteristics, social influences, specific situations, and the online environment shape the nature of the decisions and the effort in searching for additional information. Importantly, several characteristics of the online environment such as Web quality and satisfaction with the site also influence search behavior. The EKB model helped frame the problem for this study. It addresses the question “What marketing strategies are effective during the search process?” More specifically, the purpose of this study was to compare the impact of two marketing strategies on recruitment to CME, Web traffic, and cost in the context of an online CME cultural competence curriculum (www.c-comp.org). The marketing strategies were 1) wide dissemination and 2) Internet paid searches.
The Cultural Competence Online for Medical Practice (CCOMP) (www.c-comp.org) Website was designed and implemented after extensive formative evaluation.12 The main objective of the educational site was to teach effective cross-cultural approaches to care for African-American patients with hypertension. The site includes videos with real patient scenarios and case-based modules. Designed for practicing physicians, residents and medical students, the Website is available in the public domain. We used an interrupted time-series quasi-experimental design to explore the impact of two marketing strategies: 1) wide dissemination to relevant organizations; and, 2) Internet paid searches, consisting of consecutive periods of control (no paid searches) and paid searches (FIGURE 1).
The wide dissemination phase occurred between June 3, 2008 and October 17, 2008. The purpose of this phase was to determine the impact of a wide dissemination strategy on the outcome variables.13After identifying Websites focusing on cultural competence training, minority research, cardiovascular disease and others, we asked 37 to post the CCOMP Website address with a brief description of its content. We also submitted a request to the American Association of Medical Colleges (AAMC) clearinghouse portal for peer review and posting14 and posted an entry in Wikipedia, for a total of 39 sites. To identify which Websites posted the requested link to the cultural competence Website, we reviewed each of the 39 sites at the end of the wide dissemination phase to verify that the link had been posted.
After the wide dissemination phase, we included a pilot phase to inform the length of time for the Internet paid search phase. The pilot phase consisted of a cultural ad pilot (October 18–November 18, 2008) and a no ad washout period (November 19, 2008– January 11, 2009); the length of time for the washout period was empirically determined and extended past the December holidays. The purpose of the pilot phase was to provide estimates for the number of weeks for each control and Internet paid search advertisement periods; the purpose of the washout period was to avoid any delayed contamination with the subsequent phase.
The Internet paid search phase consisted of five consecutive eight-week periods between January and October, 2009; the periods were control #1 (January 12 to March 8), cultural ad period (March 9 to May 3), control #2 (May 4 to June 28), hypertension ad period (June 29 to Aug 23), and control #3 (Aug 24-Oct 18, 2009).
During the control periods, none of the ads were displayed and no other marketing strategies were in place. We did not market the site on any print venue nor make any attempts to remove the link from any of the Websites with previous link postings.
We designed and purchased Internet advertising using Google Ads to display two types of ads. The first one focused on cultural competence, the CME offering, and an authoritative sponsor (National Institutes of Health)(advertisement #1; reported here as cultural/CME ad). The second one focused on content, ie, tips to control hypertension and video (advertisement #2; reported here as hypertension/ content). Initially, a set of 26 tentative phrases were generated by a group with expertise on online CME development, with final phrases selected having face validity for cultural or hypertension issues. The first ad read: “Cultural Competency, Videos with Real Patient Stories, Online CME, NIH-sponsored www.C-Comp.org.” The second ad read: “Tips to Control Hypertension,… in your African American Patient, Videos with Real Patient Stories www.C-Comp.org.” The availability of CME credit was not prominently displayed on the main page; the viewer had to click on a link labeled “Objectives and Credit Information” to get access to that information (CME was only mentioned in the first ad).
Ads appeared on the right side of the screen after any US user typed any of the selected keywords. The keywords for the cultural ad were: cultural competence in health care. The keywords for the hypertension ad were: African Americans, African Americans hypertension, hypertension, African American hypertension treatment, African American with hypertension, hypertension in African American, hypertension in African American men, hypertension in African American women, and hypertension in the African American.
Participants were Internet users and we had no exclusion criteria. Participants interested in obtaining CME credit registered by providing name, e-mail address, and medical training level. CME credit was provided at no charge and no additional incentives were provided for participation.
The main dependent variables were recruitment to CME, Web traffic, and costs (TABLE 1). We defined recruitment to CME or conversion rate as the total number of CME credit requests / total number of visits (not unique); this metric is commonly used for online commercial sites.15 We defined traffic as a) frequency (total number of visits per day), b) volume (the total number of page views), and c) engagement with the site (total number of pages viewed per visit).16, 17 We calculated the incremental cost per additional CME credit obtained using a cost-effectiveness framework18 during the Internet paid search phase, incremental cost-effectiveness ratio = [Cost during advertisement periods– cost during control periods]/[# CME credits during advertisement periods - # CME credits during control periods].
We used Urchin Software, Version 5.0 to analyze Web server log files,19 which compile data that are not linked to a specific user. We used data provided by Google Ads to determine the number of times an ad was displayed (impressions) or selected (clicks) and the average position of the ad. The average position is attributed to each keyword, “1” is the highest position on the first page of search results (keywords with an average position of 1–8 generally trigger ads on the first page of search results).
To assess the changes in the dependent outcome variables during the wide dissemination phase, we divided this phase in two halves. We then used the Kruskall-Wallis test to compare Web traffic (number of visits, pages viewed, pages per visit per day) and the Chi square test to compare CME credit between these two halves.
We also compared the outcomes between periods in the Internet paid search phase using the Kruskall-Wallis test, the Chi square test, and run charts including control limits20 as appropriate. Using data from the pilot phase, we assumed that the mean number of pages viewed would increase from 120 (SD 120) to 190 (SD 120); at an alpha of .05, power 0.80, the number of weeks required for each period was seven weeks (47 days); we elected eight weeks. We used STATA SE, Version 10.0 (College Station, Texas, USA) for statistical analysis and used a p value of .05 for statistical significance.
The Institutional Review Board at the University of Alabama at Birmingham approved the study.
During the study period, the site was visited 19,156 times, 78,160 pages were viewed, and the median number of pages per visit per day was 3.7 (interquartile range, Q1–Q3, 2.2–6.8); CME credit was requested in 1.8% of the visits (339/19,156). The run charts shown in FIGURE 2 display the measures over time for frequency (# visits/day) and engagement (# of pages per visit). During the entire study period, the number of visits per day increased (Panel A) and the number and variability of engagement decreased (Panel B).
The Website URL was posted on 41% of the sites we requested (16/39), including MedEdPortal21 and Wikipedia.22 The median number of visits increased between the first and second halves of the wide dissemination phase (median 5.5, Q1– Q3: 3–11; 23, Q1– Q3: 10–30; respectively, Kruskall-Wallis p<.001), suggesting that the marketing strategy had a positive impact on Web traffic (frequency). The median number of pages viewed also increased between the first and second halves of the wide dissemination phase (median 28, Q1– Q3: 9–96; 115, Q1–Q3: 58–181; respectively; Kruskall-Wallis p<.001), also suggesting that the marketing strategy had a positive impact on Web traffic (volume). However, the median number of pages per visit per day remained unchanged between the first and second halves of the wide dissemination phase (median 5.9, Q1– Q3: 1.2–11.6; 5.5, Q1– Q3: 3.4–9.8; respectively, Kruskall-Wallis p=.8), suggesting that the marketing strategy did not have an impact on engagement with the Website.
The proportion of visits resulting in requests for CME credit decreased between the first and second halves of the wide dissemination phase, from 5.3% (27/508) to 3.3% (47/1,417); respectively (Chi square p= .04), suggesting that the wide dissemination marketing strategy did not increase recruitment to CME.
During the Internet paid search phase of 40 weeks, the site was visited 13,976 times (daily median 44; Q1– Q3, 28 – 70), 50,209 pages were viewed (daily median 148; Q1–Q3, 100–232), and the median number of pages per visit per day was 3.2 (Q1–Q3, 2.0–4.9).
As shown in TABLE 2, the proportion of visits resulting in requests for CME credit was highest during the cultural/CME ad period (control #1 1.4%, cultural/CME ad 4.3%, control #2 1.5%, hypertension/content ad 0.6%, and control #3 0.8%, Chi square p<.001), suggesting that the ad focusing on cultural competence and CME had the highest recruitment to CME.
The Web traffic measure, number of visits per day, was lowest during the first control period (median 27) and the cultural/ CME ad period (median 29), and highest during the second and third control periods (median 65 and 69 respectively) (Kruskall-Wallis p<.001, TABLE 2). The number of pages viewed per day were lowest during the first control period (median 118) and the hypertension ad period (median 119) and highest during the cultural/CME ad period (median 175) and the second control period (median 188)(Kruskall-Wallis p<.001, TABLE 2). The engagement measure, number of pages per visit per day, was lowest during the hypertension ad and third control periods (both median 2.3) and highest during the cultural/CME ad period (median 5.1)(Kruskall-Wallis p<.001, TABLE 2). In summary, we did not observe a specific and consistent increase on any measures of Web traffic during the cultural/CME or hypertension advertisement periods. The total cost of the display ads was $125. The cultural/CME ad was shown 1,460 times at an average position of 1.3; the ad was clicked 33 times (2.3%) at a total cost of $36 ($1.08 per click). The hypertension/content ad was shown 48,009 times at an average position of 5.9; the ad was clicked 89 times (0.19%) at a total cost of $89 ($1.01 per click). As compared with the first two control periods, the incremental cost for both ads per CME credit obtained was $5.43 ([$125-$0]/[99 CME credit requests-76 requests]); however; as compared with the first control period, the incremental cost for the cultural/CME ad per CME credit obtained was $0.64 ([$36-$0]/[81 CME credit requests-25 CME credit requests]).
In promoting an open access CME online cultural competence curriculum, during the wide dissemination marketing strategy phase, Web use (number of visits) increased while engagement (number of pages viewed per visit) and recruitment to CME decreased However, during the Internet paid search marketing strategy phase, Internet advertisement focusing on cultural competence, CME offering, and an authoritative sponsor (National Institutes of Health) resulted in the highest proportion of users requesting CME credit. Our paid search advertisement did not have an advertisement banner, which is one of the most expensive online advertisements. The incremental cost for the cultural/ CME ad per additional CME credit requested of under $1 seemed reasonable.
The increase in the number of visits could be explained by higher rankings of the Website over time, independent of the advertisement used. As the site was visited more often over time, and as other Websites posted a link, the proprietary algorithm used by Google probably increased the priority rankings during searches. We suspect that the decrease in engagement could be explained by market saturation. Additionally, the greater number of keywords in the hypertension/content ad may explain the difference on the number of times the ad was shown during this period (48,009 times) as compared to the cultural/ CME ad (1,460 times).
The Web produces vast amounts of information and methods for tracking utilization, also called Web analytics, are rapidly evolving.15, 23, 24 Web traffic statistics on frequency and volume are routinely reported;16, 17 however, how these statistics are calculated varies widely, making comparisons between studies challenging.8 Also, standard statistical techniques to track information over time may be ill-suited for analysis and more complex techniques may be more difficult to implement. Hence, the run charts we used offer another approach for CME providers to monitor process variation (such as Web traffic) over time and monitor the effects of interventions.20
Causation cannot be concluded from this study as we used a quasi-experimental design. Because the Web site did not require participants to register, we do not know if factors such as specialty, age, gender, or frequency of computer use influenced our findings. We were unable to quantify any potential impact of confounders. Finally, the greater number of keywords in the hypertension/content ad may explain the 33-fold increase in the number of times the ad was shown during this period as compared to the cultural/ CME ad (48,009 vs.1,460 times); however, one has to keep in mind that more people may be interested in blood pressure content as over 70 million people in the US suffer from hypertension.
For an online cultural competence curriculum, an Internet advertisement focusing on cultural competence and CME was associated with an approximately three-fold increase in recruitment to obtain CME credit at an incremental cost of under $1. The number of visits and level of engagement seemed to be independent of the advertisement strategy.
Future studies could further explore the strategies we used to increase Web traffic in online CME activities; for example, examining whether cycles of changes in the advertisement wording or change in the number of keywords- rather than content as was done in this study - has any effect on Web traffic.
We thank Michael Schoen PhD and Trudi Horton PhD, for critical revisions of the manuscript.
Funding and Support: This study was funded by a grant from the National Heart, Lung, and Blood Institute (K07 HL081373-01) to Dr. Estrada.
Ethical approval: This study was approved by the institutional review boards of the University of Alabama at Birmingham.
DISCLOSURES The authors report no conflict of interest.
Other Disclosures: None
Disclaimer: The opinions expressed in this article are those of the authors alone and do not reflect the views of the Department of Veterans Affairs.