The five major steps that we formulated and carried out in developing the web site are presented in logical order; however, the process was iterative and insight gained in one area often resulted in the revision of another area. Furthermore, extensive detail is provided in Step 4, Conceptualize the Purpose and Objectives of the Web Site, as we incorporated information about contextual and design decisions in this step.
Step 1: Review the Literature
The literature was reviewed to gain insight and understanding of web site development and Internet recruitment. Although there is an abundance of information on web site development in general, few reports addressed systematic information about how to develop a web site for recruiting eligible participants.16
To obtain beneficial information, we sought out reports from investigators who had used the Internet successfully for recruiting hidden, concealed, or disparate populations and/or when the study focused on sensitive topics. 17–19
We were also interested in gleaning knowledge from nurse investigators who used the Internet successfully for accessing hidden populations using qualitative methods on sensitive topics.9, 10
We also sought out and paid thorough attention to reports where Internet recruitment was slower and more challenging than expected20
or resulted in disappointingly low response rates.21
These investigators often provided insightful information and strategies for facilitating future Internet recruitment success. In particular, we noted that the majority of participants who enrolled in several research projects did so after visiting a project web site.20
Furthermore, if a project web site was developed, care should be taken to make the project web site attractive with pictures and special fonts,20
and text should be carefully selected to convey a trustworthy and legitimate message.21
Other investigators made suggestions about web site design that were helpful. Zhang and von Dran22
encouraged web site designers to carefully consider the overall purpose of the web site (e.g., business, education, recruitment) and to pay particular attention to details important to the future users.22, 23
Gehrke and Turban24
determined that web site users appear to be most concerned about page-loading speed, quality and usefulness of the content, and navigation efficiency, whereas others suggested a contextual reading level of less than tenth grade or the most appropriate level to the target population.25, 26
Finally, we sought out literature that provided valuable information on broader issues surrounding Internet recruitment. Predominant issues reported were establishing participant authenticity;27
identifying and working with moderators or gatekeepers of Internet communities and web sites to obtain access; 27
and addressing security, confidentiality, and ethical issues27–30
Another important issue was negotiating use of and navigation with stakeholders for the Internet server.28
Step 2: Incorporate and Expand Previously Successful Recruitment Strategies
Prior experiences of the research team in recruiting similar concealed, disparate, or vulnerable populations for sensitive research guided our willingness to develop a project web site even though it extended our prior recruitment strategies.31–34
For example, knowledge gained from the first author’s qualitative study examining the lived experience among donor egg recipient women in their second and third trimesters of pregnancy was evaluated.31
In addition, the second author’s expertise in theoretical recruitment approaches was incorporated into all aspects of the web site.35
Step 3: Seek Expert Advice
Early in the process, experts were sought out to assist with the development of the web site and also to explore in-depth the issues surrounding web site development and Internet recruitment. Consultants who were sought out were an individual with expertise in Internet recruitment (Dr. Rebekah Hamilton), a psychologist with clinical experience and expertise with the population (Dr. Susan Klock), a well-established nurse researcher with expertise in women’s reproductive health (Dr. Ellen Olshansky), and an expert in decision science (Dr. Penny Pierce). A graduate student with expertise in web design and healthcare graphics (Ms. Lisa Merry) was also added to the team to assist in the technological development of the web site.
In addition to seeking out experts in substantive areas related to web site design and recruitment, representatives of the University of Illinois at Chicago Institutional Review Board (IRB) were contacted early in the process to obtain information about procedures to ensure human subject protections. Suggestions made by the IRB representatives were then incorporated into the development of the web site. These included having text within the web site that contained elements similar to that of traditional research recruitment flyers or brochures and clearly stating the IRB approval notice and protocol number on the web site. Noteworthy was the suggestion to include off-line hardcopies of the web pages and a transcript of any audio materials used in the web site included in the IRB application. Once IRB approval was obtained, the web site was uploaded to a web server, making it available online and accessible via the Internet.
During formative work, experts in budgetary considerations were consulted for cost estimates of implementing a research web site. The range of cost was variable from no fees for an investigator-designed and -developed web site to moderate estimates of 2,000 U.S. dollars for contractual work. After much consideration, a hybrid model was adopted where the first author, the principal investigator (PI), contracted and worked closely with a graduate student (i.e., fifth author) who contributed to the technological design and development of the web site.
Step 4: Conceptualize the Purpose and Objectives of the Web Site
After reviewing the literature, evaluating prior recruitment successes and challenges, and consulting with experts, the purpose and objectives of the web site were finalized. This was an important step and sufficient time and effort for reflective work was necessary. For our study, the main purpose of the web site was to inform potential participant couples about the study to ultimately foster participation. The objectives we set for the web site were to (a) communicate information, (b) establish credibility, (c) facilitate contact, and (d) demonstrate caring.
A. Communicate Information
Communicating and transmitting information effectively about the purpose of the study was paramount. Providing the eligibility criteria with the purpose allowed for potential participants to self-evaluate their own eligibility and optimize participant involvement from the onset of the study. It also served to minimize investigator workload regarding eligibility screening. Additionally, we also informed potential participants about the protocol and expectations of participating in the research.
In addition to deciding what information to communicate, we deliberated about how to communicate the information. Initially, we determined the number and style of visuals including an attractive color scheme and web site navigation aids. The color scheme was considered with respect to the target population of men and women of reproductive age, thus, we opted for colors that would appeal to both genders. Because the main aim of the study was to explore decision-making across a wide range of cultures, we developed and incorporated an ethnically neutral silhouette of a couple in a contemplative pose for the study logo, which served both as a visual reminder of the overall aim of the study and the target population. The logo, designed to color coordinate with the web site color scheme, can be seen in the top left corner of . We also wrote the text for the web site at a tenth grade reading level (Flesch-Kincaid Grade Level score = 10.2). Although this is relatively high, our eligibility criteria required that couples have an awareness of PGD, which we found difficult to discuss at lower reading levels.
Next, we contemplated how best to display the information on the web site. The two options we considered were to display the information on one large web page where potential participants could scroll down the page for additional information, or to display the information on several web pages contained on the web site and controlled by navigation tabs. We opted for the latter and developed a Home page that displayed vital study information to maximize interested participant preferences with brief information, and developed four additional web pages for detailed information: Study Details, Research Team, Contact, and PGD Resources. is a screenshot of the web site Home Page with the navigation tabs visible.
Another strategy to communicate information was to develop and include a brief (~4 minute) video on the Home page. In the video, the PI offers a brief welcome to the web site and provides key information about the study. The video was inserted for potential participants who might prefer receiving information through an audiovisual media versus a text media and was programmed to begin automatically when viewing the Home page.
B. Establish Credibility
The second objective was to inform potential participant couples about the high quality of the study and the expertise of the research team. This was accomplished through several strategies. First, the official University of Illinois at Chicago College of Nursing logo was displayed on the top right corner of each of the web pages. Additionally, information about funding sources, including grant numbers, and information about institutional IRB approvals, including protocol numbers, was delineated at the top of the Study Details page.
Another strategy to demonstrate the credibility of the study was to host the web site on the university’s web server that was linked to the college affiliated with the PI for the study, rather than have the web site as a stand-alone site. This was also financially feasible as the university-linked site was available at no cost. The study web site URL, linking to both the university and the college of nursing, became http://www.uic.edu/nursing/fertilitydecisions/PGD
To demonstrate the high quality of the research team members, the Research Team web page was developed. Each of the key research team members (i.e., PI and Co-Investigators) had his or her name, credentials, title, a photograph, and a short biography that delineated his or her role and expertise for the study displayed on the Research Team page. The biographies were structured so that the expertise and experience of each key researcher was highlighted to reflect his or her particular contribution to one or more substantive areas of the study. An electronic hyperlink was inserted into each research team member’s name and linked to his or her individual faculty page at his or her respective university. The three consultants for the study were also listed on the Research Team page with their names, credentials, titles, and university affiliations.
C. Facilitate Contact
Facilitating contact between potential participants and the PI was an important objective because the PI determined eligibility and obtained follow-up information, if appropriate, for entry of the couple into the study. Because enrolling couples was paramount to the success of the study, information detailing how potential participant couples could establish contact with the PI was displayed on two web site pages. The Home and Contact pages contained the PI’s name, credentials, office phone number, and e-mail address. The e-mail address had an embedded hyperlink to minimize participant burden should a potential participant choose to establish contact with the PI. Information about how to contact the PI was also discussed verbally in the video and displayed on text superimposed during the video.
D. Demonstrate Caring
The objective to demonstrate caring was implemented to provide a virtual milieu through which the web site would be philosophically consistent with the nursing discipline and the underlying philosophical stance of the PI and the research team members. Specifically, the objective was to demonstrate care and compassion for potential and actual participants in the study and high-genetic-risk couples in general.
This objective was implemented through careful and considerate use of tone embodied in the text and audiovisuals. Another mechanism to demonstrate caring was the creation of the PGD Resource web page. This page was designed to include information and resources, available on the Internet and hyperlinked to corresponding web pages that might be helpful to high-genetic-risk couples regardless of whether they choose to participate in the study. For example, one of the resources hyperlinked is the National Institutes of Health’s Genetic Testing web page.
Step 5: Determine Conduits, Links, and Analytics
Brainstorming sessions were held among research team members to identify and determine appropriate Internet organizations and mechanisms that could serve as conduits for the web site and announce the study to potential participants. Prior to obtaining funding for the study, two Internet community groups, one professional group, and one professional group’s electronic mailing list were identified as key conduits for announcing the study and linking to the web site. Because we were also using traditional recruitment approaches involving brochures that announced the study at clinics and through advertisements in printed newsletters, we also included the web site address in these traditional approaches.
After the web site was developed and online, a gatekeeper at one of the previously supportive Internet community groups failed to respond to the PI’s numerous phone calls and e-mail messages to announce the study and post the web site link. This experience led to the PI developing a recruitment journal where names, dates, and records detailing discussion and talking points were recorded to facilitate communication and follow-up. Additionally, other Internet organizations where potential participant couples could be made aware of the study and a link to the web site posted were identified for potential use, if our initial strategies faltered.
We also formulated strategies to optimize web site searchability and visibility and decided on analytic procedures that could provide information about web site use. Regarding web site searchability, we opted to include meta tags in the html code of the web site to maximize search engine optimization. Meta tags enable search engines, such as Google and Yahoo!, to better understand how to categorize the web site and improve the web site’s search rankings. Thus, meta tags can increase and ease searchability and visibility by facilitating awareness of the web site among Internet users. The meta tags we used were PGD, Preimplantation Genetic Diagnosis, study, research, information, resources, high risk, genetic disorder, genetic testing, reproductive testing, and prevention. To further enhance searchability and visibility, we accelerated the process of having Internet search engines locate, retrieve, and display the study web site following queries by submitting the web site URL to the indexes of Google and Yahoo!.
Another strategy we used was to insert a two-sentence description of the study web site in the html code, enabling search engines to list a description of the site. In this instance, when the web site was retrieved after an Internet search, a description of the web site appeared underneath the URL link. provides the text used for this strategy. Lastly, we set up a free analytic system using Google Analytics that would record the number of visitors to our web site and collect useful data that can enhance our understanding of web site usage.
Description of Web Site for Search Engines.
The web site went live in April 2009 and was subsequently linked to community and professional web sites and the electronic mailing list, which took several months to accomplish. During this time, gatekeepers and web site managers who had agreed to announce the study were contacted about posting an announcement or e-message that would provide the PI’s contact information and a link to the study web site. We also implemented our analytic strategies after the web site was live on the Internet. The results from Google Analytics for April 2009 through May 2010 demonstrated that there were 853 web site visits, of which 634 were first-time visitors. The average number of web pages viewed during visits was 2.59 pages. As a result of all of our efforts, the first couple entered the study in early June 2009 and recruitment was closed after the 22nd couple entered into the study in May 2010. Results for recruitment indicate that of the 22 participating couples, ~82% (n = 18) were recruited via the Internet. delineates how the couples found out about the study, as reported by the couples themselves.
We questioned each partner within the dyad as to whether they had viewed the study web site. Consistent with our success of Internet recruitment strategies, a large majority of the couples (73%) acknowledged that they viewed the web site prior to entering the study. Among the couples that viewed the web site were two couples that were made aware of the study by traditional recruitment strategies. Additionally, four couples that were recruited via the Internet did not view the web site, but opted to contact the PI directly after learning about the study from Internet announcements or e-messages from electronic mailing lists. Furthermore, we found that 59% of the female partners and only 14% of the male partners viewed the web site prior to participating in the study.
To obtain more in-depth understanding about participants’ perceptions and evaluations regarding the web site, using a semi-structured interview guide we asked each individual partner who viewed the web site to provide feedback on 1.) what was beneficial about the web site for recruiting participants and 2.) how the web site could be improved. Overall, the participants reported that the web site was helpful and informative. For example, participants said, “I … found it very helpful. It was very clear and easy to read and understand. I like that you kept it fairly simple and brief, yet explained all of the study details,” and “[I found] out, you know, what was going to be involved.” Another participant stated, “ [I used the web site to] just to get a sense of, you know, who you were, and, you know, all you researchers, and, you know, where you guys were coming from, and, you know, just to kind of inform myself more about it [the study].” Although few participants offered suggestions for improving the web site, we did ascertain that for several participants, the amount of time spent viewing the web site was limited. Comments such as, “Well, honestly, I didn’t spend a lot of time on it,” and “I actually did not look at the web site. I just jumped straight in [to enrolling in the study],” helped us gain insight about participant use.
Several participants also offered their perceptions about the video, which appeared to be an effective strategy based on this typical comment: “I thought that it was brief and to the point, but it gave enough information to pique an interest in participating in the study. I really liked that it seemed very personal and that you actually really care about helping others who are in the process of making a decision to use or not use PGD.” Another participant summed up the thought of others by saying succinctly, “I saw what you looked like. I heard you talk.” Yet, there was discrepancy about the content and length of the video. As one participant remarked, “I thought it was very thorough. You spoke very carefully and used simple language, which made the video understandable. For me, it was a little too slow, but I think others may need the more thorough video.”
We learned foremost what a tremendous amount of effort, planning, and coordination is needed to develop a web site. We underestimated the amount of time needed for thoughtful consideration of how to convey content, communicate information, navigate human subject protections, and work with Internet gatekeepers and managers. We did find our extensive review of the literature and consultation with multiple media and content experts valuable. However, there is a dearth of literature on web site design for recruiting concealed and disparate populations for sensitive research, and we often found ourselves traversing undocumented territory even with the advice and direction of experts. It should also be noted that if our budget was larger, we may have been able to minimize our effort and time. Because this was not the case, the PI undertook a systematic process where she was involved with all aspects and coordination of the web site, along with the assistance of a technologically experienced and knowledgeable graduate student (fifth author).
Nevertheless, about 82% of our sample was recruited via the Internet, and most couples had at least one partner view the web site prior to entering the study. We conclude that our multifaceted recruitment plan, aided by an informative web site, was highly successful for this study. We are keenly aware, however, that there may have been potential participant couples that found the web site not helpful or difficult to navigate or understand. Given the high number of individuals who viewed our web site (634 were first-time visitors) relative to our sample size, it is likely that there were individuals who opted not to participate in the study because of web site issues and thus their perceptions about the web site were not captured. What was demonstrated is that for the majority of participating couples, one partner viewed the web site and found the information helpful, and the couple enrolled in the study. It is noteworthy, too, that for couples entering the study by traditional recruitment approaches, 50% viewed the web site. Our findings also indicate that women were more likely to view the web site than men, and that our efforts to interject a caring philosophical stance throughout all aspects of the web site were positively perceived.
It is unknown whether or not the sensitive nature of the research, the focus on reproductive decision-making, or other unknown factors compelled the female partners to view the web site more than their male counterparts. Future research that explores these gender issues within dyadic couples’ research would be beneficial. In addition, we did learn that despite efforts to communicate effectively, some potential participants prefer brief and focused information. Furthermore, our interview data demonstrated the effectiveness of and preference for a short video in reaching some participants, while our analytic data suggested that web site visitors viewed about two to three web pages. In-depth research into the contextual factors that influence and enhance recruitment would aid understanding about web site development and design.
Regarding the broader issues surrounding Internet recruitment and use of a study web site to augment recruitment, we did not find authenticity of the couples to be problematic. Whether the study procedures (e.g., mailing informed consent documents to eligible couples’ private residences) and/or the qualitative and sensitive nature of the study deferred spurious participants remains unknown. Whatever the underlying instances were, there were no participants whose authenticity we questioned. Yet, because of the effort required by the couple and the PI to send and receive official study documents by mail and given our lack of bogus participants, we are considering using an online informed consent strategy in future studies as suggested by other investigators.28
Another issue that warrants discussion is the effort expended to establish and achieve support from gatekeepers and web site managers. For this study, the PI developed and kept a recruitment journal to facilitate and ease follow-up communication. When recruitment closed, a total of 36 individuals had been contacted, representing 22 Internet organizations, electronic mailing lists, or private community web sites, and the journal contained 104 pages and consisted of 27,803 words. Although our efforts in recruiting the target sample were successful, obtaining access from Internet gatekeepers and managers took a significant amount of time and was challenging, as other investigators have also discovered.20, 27, 36
In contrast, in several instances, we experienced an effortless and supportive milieu among gatekeepers and managers. We are in the process of evaluating our strategies and data in this area to understand the phenomena and how best to navigate the relationship with the wider Internet community and professional groups.
In summary, developing an informational web site to aid in recruiting high-genetic-risk couples on the Internet was challenging, yet beneficial. The five-step process we undertook to develop and implement the web site resulted in the successful recruitment of a well-concealed and disparate sample to participate in sensitive research. We also found that half of the traditionally recruited couples viewed the web site, while not all participants recruited through the Internet viewed the web site, rather several “jumped straight in.” Research in this area is needed to understand how investigators can maximize use of a study web site as the Internet evolves and to enhance the research experience for the individuals, couples, and communities that we serve.