|Home | About | Journals | Submit | Contact Us | Français|
To describe the advantages and limitations of an ACASI system in a multisite trial with African American couples and to present the steps in designing, testing, and implementing a system.
The ACASI system evolved from a paper and pencil interview (PAPI) that was pilot tested. Based on this initial work, the PAPI was translated into story boards that were the basis of the development of ACASI system. Story Boards consisted of one page per question and provided the programmers with the test of the question, valid responses, and any instructions that were to be read to the participants. Story boards were further translated into flow diagrams representing each module of the survey and illustrating the skip patterns used to navigate a participant through the survey. Provisions were also made to insert a face-to-face interview, into the ACASI assessment process, to elicit sexual abuse history data which typically requires specially trained data collectors with active listening skills in order to help participants reframe and coordinate times, places and emotionally difficult memories.
The ACASI was successfully developed and implemented in the main trial. During an exit interview, respondents indicated that they liked using the ACASI and indicating that they favored it as the method to answer questions.
It is feasible to implement an ACASI system in a multisite study in a timely and efficient way.
There is an evolving trend in health research, investigating sensitive and/or personal issues, to use more technologically advanced data collection methods as a way to ensure the privacy of participants.1 A recent article by Tourangeau and Yan2 reviews the survey methodology research on reporting errors in surveys on sensitive topics. Their findings suggest that misreporting about sensitive topics is common and largely situational. They go on to conclude that misreporting on sensitive topics is a process in which respondents edit the information they report to avoid personal embarrassment or repercussions from third parties. One such method is the Audio Computer Assisted Self-Interview (ACASI) system. ACASI is a user friendly computer interface that guides users through a survey, using digitally recorded instructions, questions, and answers. This is in contrast to more traditional face-to-face interviews or self-administered questionnaires (SAQ), which are common in survey research.3 Generally, ACASI participants listen to a series of digitally recorded survey questions through headphones, while the corresponding text is displayed on the computer monitor. ACASI users advance through the survey by responding to the question displayed, using the keyboard and/or mouse.
Research has shown that participants who volunteer for studies that investigate sensitive and personal issues often prefer advanced methods of data collection such as ACASI for perceived privacy advantages over traditional data collection methods. We recently searched PUBMED for keyword “ACASI” in either the title or abstract of published articles and retrieved 39 such publications. As this paper is not intended to provide a comprehensive review of the ACASI literature, we will highlight only a few of those articles here and note the increasing frequency of “ACASI” and growing range of targeted study populations using or studying ACASI in publications in the literature over time.
The earliest publications were four articles from 20004–7 where each assessed the feasibility of using ACASI in a variety of settings and found ACASI to be an effective mode for collecting sensitive data in their respective settings. Additionally, two of these early publications were in HIV affected populations.4;6 These were followed by a single publication in 2001 by van Griensven and colleagues8 examining the assessment of sexual behavior, illicit drug use, HIV and sexually transmitted infections (STIs) in a sample of youth in Thailand. This study found that 10% of participants in the study had a positive urine test for methamphetamines, but in the ACASI interview, 16% of those who tested positive denied ever using methamphetamines, thus concluding that ACASI use may lead to increased, but not necessarily complete, self-reported responses to the sensitive outcomes measured in their study. In 2002, a single publication found that ACASI is an important data collection tool in reducing socially desirable responses to sensitive questions related to sexual behavior, especially among HIV seropositive participants.9 Four additional publications appeared in 200310–13 followed by eight more in 2004.14–21 This series of manuscripts included a example where ACASI was used as the data collection method in a sample of youth in Northern Thailand to assess the prevalence of not wearing a helmet while riding motorcycles,18 which is represents a different context for measuring risky behavior compared any of the previously mentioned study populations. There were eight publications which appeared in 200522–28 including one by Johnson and colleagues29 which explored different data collection modalities (ACASI versus telephone interviews) to address social desirability issues commonly associated with self-reported cancer screening behaviors and one by Johnson and Fendrich28 which modeled sources of self-report bias in an illicit drug use survey and found that both memory difficulties and social desirability concerns are independent sources of measurement error in studies of drug use epidemiology. Seven publications appeared in 200630–35 including a paper by van Griensven and colleagues36 examining the palmtop technology as a cheaper more mobile alternative to ACASI for collecting sensitive behavioral risk data. At the time of this writing, there have already been seven such publications which have appeared in the literature in 200737–42 including the published findings of a feasibility study of ACASI in a multisite, international trial.43
We further highlight two of the above examples from studies of populations affected by HIV. In a study to understand breastfeeding practices and attitudes of women affected by HIV, researchers found that a majority of women preferred ACASI to face-to-face interviews (65%) because of the added privacy and usability.27 Additionally, most respondents in this study indicated a preference for ACASI over face-to-face interviews for future studies (79%). Furthermore, Metzger and colleagues6 summarized the results of a test of ACASI system in a longitudinal study of HIV risk behavior and infection, in a sample of gay men and injection drug users (IDUs). Respondents were randomly assigned to one of two assessment modalities: ACASI or face-to-face interview at their second follow-up visit, 12 months after baseline. They found significantly more sexually active gay men who reported having sexual partners who were HIV antibody positive (OR = 1.36, 95% CI: 1.08, 1.72), and a higher proportion reported unprotected receptive anal intercourse assessed via ACASI than in face-to-face interviews. Additionally, this study found that IDUs randomized to ACASI reported higher usage of needles without cleaning them (OR = 2.40, 95% CI: 1.34, 4.30).
ACASI has several advantages over face-to-face interviewing and SAQ. First, ACASI provides anonymity, and a seemingly safe space for participants to respond honestly and candidly to questions that are highly personal and sensitive in nature or that may be regarded as socially undesirable. For example, Waruru and colleagues27 found that respondents endorsed more socially undesirable survey items using ACASI than in face-to-face interviews. Another advantage is the ability to collect critical data using complex survey instruments from populations with lower literacy levels while allowing participants to advance through the survey at their own pace. Additionally, participants at all literacy levels are able to proceed through the survey quickly because they do not have to simultaneously respond to each item of the survey and navigate complex skip patterns since those have been directly pre-programmed into an ACASI system. Data quality is also improved since data are collected directly from the participant thus minimizing data transcription and data entry errors. ACASI also provides the opportunity to minimize human errors by programming various consistency checks into the ACASI system and enforcing response constraints.
This article describes a customized ACASI system implemented in a randomized controlled clinical trial of an intervention for African American Couples conducted simultaneously at four urban clinical centers: Columbia University, Emory University, the University of California at Los Angeles, and the University of Pennsylvania (Project Eban). The primary goal of the study is to test the efficacy of a couples-based Eban HIV/STD Risk Reduction Intervention (treatment) compared to an individual focused Eban Health Promotion Intervention (control). The primary outcomes for this ongoing study are (1) the rate of condom protected sex for couples and (2) incidence of biologically confirmed sexually transmitted diseases (STDs). The primary endpoints will be assessed at baseline, immediately following the completion of the 8-week behavioral intervention (IPT), and 6- and 12-months post intervention. Due to the sensitive nature of the data collected in relation to the aims of the study, ACASI technology was implemented as the primary means to collect behavioral outcome data (condom protected sex) at each of the four assessment times. Detailed descriptions of the trial’s study design and both behavioral modification interventions are provided in other articles in this special issue.
The ACASI system evolved from a paper and pencil interview (PAPI), self-administered survey. Created by study staff, including the Principal Investigators and Project Directors from each of the four clinical sites, the PAPI prototype was reviewed by each site’s Community Advisory Board (CAB), as well as several focus groups before being pilot tested in a small sample of African American, serodiscordant, heterosexual couples at two of the four clinical sites. The primary purpose of the PAPI was to identify and reduce participants’ problems in interpreting survey questions, understanding corresponding response categories, and ensure that the assessment could be completed in a reasonable amount of time (e.g., reduce participant response burden). Feedback from pilot testing was incorporated into a finalized version of the PAPI, which was then translated into story boards that were ultimately used to develop the Eban ACASI system. Story boards consisted of one page per question that provided the ACASI programmers with the text of the question, valid responses, and any instructions that were to be read to the participant. Two gender-specific versions of Module F were created in order to elicit sexual behavior. An example of two ACASI story boards from female version Module F of ACASI (Question #: F2f “In the past 90 days, how many times did your study partner put his penis into your vagina?” and Question#: F3f “In the past 90 days, when your study partner put his penis into your vagina about how many of these times was a male condom used?”) are presented in Figure 1. Not included in this figure is the corresponding index question for this section of the assessment: Question #: F1f “In the past 90 days, has your partner put his penis into your vagina?”. Each story board also contains internal coding notes for the ACASI system programmers which are not visible to participants completing the assessment. For example, responses to F2f must be numeric with no more than 3 digits (Annotation: N3) and must be greater than zero (Constraint: F2f must be greater than zero). Additionally, since F3f would be used as a numerator value in constructing a condom-protected, vaginal sex measure, participants are reminded that their response to this question should not be larger than the denominator which they have entered in F2f in a note that also appears (and is read) on the screen while their value from F2f is displayed as the single entry in the box labeled “Value = F2f” near the bottom of the screen. This restriction also is hard-coded by ACASI programmers (Violation: F3f > F2f) directly into the ACASI system, so that if a participant receives an error message if the value they enter larger than their previously entered denominator and are instructed to change one or both responses. Because this series of questions are critical in constructing the primary sexual behavioral endpoint, if participants choose to “go back” in this section of the survey, they must go back and re-enter their response to the index question (PREVIOUS go to F1f). The Eban ACASI system tracks the number of “Preferred not to Answer” responses and a report is provided to data collectors when each participant completes the ACASI assessment. Finally, because this is the female version of Module F, males skip out of this entire section of the assessment (Skip if participant is male).
Story boards were further translated into flow diagrams representing each module of the survey and illustrating the skip patterns used to navigate a participant through the survey. ACASI programmers used the story boards and flow diagrams to develop an initial version of the ACASI system which was also pilot tested in a different sample than its PAPI predecessor. Feedback from piloting the initial system was incorporated into a finalized ACASI version that was ultimately distributed to each clinical study site and administered to the research participants. In total, the Eban ACASI system contains 19 Modules (A-R, with gender-specific Module F) consisting of 215 items, excluding Module F. A detailed description of the measures used in Eban are presented in another manuscript in this issue (HIV/STD Risk-Reduction: Strategies for Enhancing the Utility of Behavioral and Biological Outcome Measures for African American Couples). The gender-specific Module F contains appropriate language in order to assess sexual behavior outcomes with respect to the participant’s study partner as well as activity with other partners. Male participants were asked additional questions about sexual behavior with other men and women and female participants were asked additional questions regarding sexual activity with other men (e.g., multiple partners). The net result was a 45 item male version of Module F and a 38 item female version of Module F. In total the female ACASI contained 253 items and the male ACASI contained 260 items, designed to be completed in approximately 90 minutes. Given the length and complexity of the survey, the story boards and flow diagrams were essential to the programming of the ACASI system. As an example, the female ACASI flow diagram for Module F is shown in Figure 2. “Yes/No” index sexual activity questions (e.g., F1f and Question #: F10f “In the past 90 days, has your partner put his penis into your anus (butt)?”) are included as shaded boxes and if a participant responds “yes” to any of these questions, they proceed to the various assessment items related to that behavior (e.g., wearing both male and female condoms) and if they respond “no” they skip those questions. The set of boxed questions in the top left are the series of questions related using male condoms, in the top right are the series of questions related to using female condoms and the boxed questions near the bottom are the series of questions regarding sexual activities with other male partners.
A portable ACASI system (dedicated laptop computers complete with the customized Eban assessment instrument and all the required hardware and software for complete ACASI functionality) was required that provided users one question per screen, read each question, the corresponding responses, and instructions to the user. Additionally, the Eban ACASI system would programmatically control navigation through the survey, and enforce data range checks and consistency across all responses. For example, in collecting the primary behavioral outcome (proportion condom protected sex), the ACASI system was programmed to prohibit participants from responding having had more condom protected sexual experiences than their reported sexual experiences. Due to the complex nature of the Eban ACASI system requirements, a custom application utilizing an Oracle Database and Oracle Interface design tools was developed instead of using an off the shelf ACASI tool. As indicated earlier, the ACASI system was divided into 19 structured question groupings. Once a module is completed, responses are saved to a database that resides locally on the computer being used to conduct the survey. The user has the option to discontinue the survey after completing a module and resuming at a later time with the first question of the next uncompleted module. This feature facilitated study staff’s ability to administer the face-to-face interview and then have the research participant return to the ACASI.
Each module has an “introductory” page indicating the start of the module and a “final” page indicating the completion of the module. Each data entry screen within a module displays the question number, the module section, the actual question, and the range of valid responses available to the user. Each question has a pre-recorded digital sound file that contains the text of the question and valid responses. These pre-defined questions and responses are read to the user every time they move to a new question. The user also has the option to replay the question and answers at any time by pressing a button on the screen. Within a module, users have the option to move back and forth one question at a time to change answers. However, once a user has completed a module and moved on to the next module he or she is not able to return to a previously completed module to modify responses. In addition to the acceptable responses to a question, each screen within each module (except Module F described previously) has these four choices at the bottom of the screen:
HELP - Provides additional descriptions and explanation of the question to the user.
PREVIOUS - Navigates the user to the previous question.
NEXT - Navigates the user to the next question within the survey. (The user is not allowed to navigate to the next question until the current question is answered.)
Preferred Not to Answer - Allows the participant to choose not to answer a question and proceed to the next question.
To minimize mistakes made by users when answering questions, several features were programmed into the ACASI system. Users are prompted by the system to correct values outside the range of plausible answers, based on a set of constraints that were defined by project personnel based on pilot testing. Examples include the customized programming for Module F as previously described. Data quality checks are essential to the quality and consistency of the data being collected and were directly programmed into the ACASI system.44
To address potential literacy issues, the denominator is described in the question on the screen, and in the audio when read to users. Examples include those previously described relating to Module F. Additional prompts were built into the ACASI system to help users accurately recall past events. Because the primary behavioral endpoint references behavior in the past 90 days, every question in Module F, includes three dates that are programmed at the top of each screen. The first date is 90 days ago; the second date is 30 days ago; and the third date is the current date. So for example, if the question was about "the past 90 days", the current date, the date 90 days ago, as well as 30 days ago, are displayed on the screen. In addition to the dates provided on the screen, prior to beginning the ACASI, participants are oriented to a paper calendar which the study staff reviews with them. They are asked to include on the calendar holidays, birthdays, anniversaries, hospitalizations, things they’ve done with family, friends, or alone, places they’ve been, etc. in the past 90 days. Prior to beginning Module F, ACASI stops and prompts the participant to get the attention of one of the study staff. Prior to proceeding with Module F, a staff member sits down with the participant, reviews the events listed during the past 90 days, and instructs them to use the calendar to calculate the last time they had sex. Gender-specific questions were recorded and administered in such a way that male partners were asked about sexual behavior with their female study partners, and female partners were asked about sexual behavior with their male study partners (for the primary behavioral outcomes).
At each assessment visit, study participants are escorted to a gender specific data collection room and given instructions on completing the ACASI by the trained study data collectors. Instructions include general computer use and navigation, as well as practice with assessment style questions. Specifically, the Eban ACASI system has a built in “practice module” that users are required to complete before going on to the actual study modules. These questions are not checked for accuracy, but serve as a means to orient users to the ACASI system and the different types of questions they will encounter during the assessment. Data collectors are also available to answer participants' questions or provide other assistance they might need in using the ACASI. To maintain the integrity of the intervention, data collectors are blinded to assigned intervention condition, and only interact with their assigned couples regarding issues related to the assessment and data collection at the assessment visits.
The Eban ACASI system was developed and installed on individual laptops at the AAC data coordinating center (DCC) and delivered to the sites, and each laptop was loaded with a self-contained, stand alone system that allows for portability. So that the data collected on each laptop could be backed up and incorporated into a centralized database at the DCC, each laptop has the ability to export data onto Compact Discs (CD). CDs from each laptop are sent to the DCC after each cohort of participants completes a particular assessment visit, at which time the data are loaded into a single centralized database that contains data from all sites and all laptops. For security reasons, access to the Eban ACASI system on the laptop is controlled by a username/password for logging onto the computer, as well as a username/password that is required to log onto the Eban ACASI system. To further protect the primary outcome data, an additional feature was built into the Eban ACASI that allows the clinical center to print an entire session, or report of the participants’ responses to questions in Module F.
Great efforts were made to ensure the best sound quality possible. To preserve consistency throughout the questionnaire, storyboards corresponding to each question of the four assessments were recorded by a single African American female with American standard accent.
The primary challenges associated with the implementation of the Eban ACASI system stemmed from the required portability requested by study investigators in order to logistically collect data in a variety of settings. To meet this requirement, each laptop had to be a stand alone system lacking connections to the internet or a centralized database for purposes of data collection. This limitation was a result of an inability to ensure that internet access would be available at all locations where the ACASI was being administered. For example, we could not ensure that reasonable internet access would be available at sites where data were being collected in the field (e.g., community centers, etc) where it was logistically more feasible for participants to access. As a result, it was possible for study staff to accidentally administer the wrong ACASI to a participant. For example, a participant in for his/her baseline visit could accidentally be given a 12 month ACASI survey. If the laptops were connected to a central database, this error could have been captured and would have prevented staff from administering the wrong survey to a study participant. Luckily, most of the data collected in these situations can be corrected manually, in house at the DCC, but it is manually detailed and tedious. Such instances occurred several times throughout the course of the study and resulted resources expended at the DCC to clean and correct the data. Future ACASI systems that could be deployed on laptops and/or desktops equipped with internet access to connect to a centralized database from any location would circumvent this technical limitation.
An exit interview was administered upon completion of the Eban ACASI pilot assessment. In general, the response was positive. For example, all participants reported feeling “empowered” by the use of Eban ACASI. One participant responded: “I never used a computer, and I was proud of myself; very empowering.” Another participant reported favoring it over face-to-face in survey research: “Using the computer to answer questions was a great experience; better than a person reading the questions.” Finally, respondents commented on the usefulness of the data collectors as a resource during the process: “The staff was available to help, and they were wonderful.” These experiences are consistent with the literature from longitudinal studies of high risk behavior. For example, in a study of gay men and injection drug users,6 the majority of participants reported few problems, and (60%) felt that the Eban ACASI elicited more honest responses than did interviewer-administered questionnaires.
Disadvantages of Eban ACASI include the inability to probe respondents when their responses may be unclear. In Eban, for example, we chose to collect sexual abuse data with data collectors who received special training in active listening in order to help participants define past experiences where they were abused and to help them reorganize their memories of those events so that times, places and their memories are aligned. In some instances, we discovered participants who experienced abuse, but were not aware of it and these sensitive and potentially highly emotional experiences, we thought, were best conducted outside of the context of a computerized ACASI system. In Eban, we were able to facilitate the flow of the assessment process so that participants could transition from one modality to the other. Therefore, ACASI may not be ideal in settings where active listening is essential to the data collection process. This potential limitation also exists in Short Answer Questionnaires (SAQs). Additionally, because of their portability and value, laptops may be stolen or break down over the course of the study. Although critical data may be lost, because none of the respondents’ identifying information is attached to their data, participant privacy is always preserved.
While these are real limitations, on balance, the advantages of the ACASI system in collecting data over mutliple time periods in a multisite study far outweigh them. The technical advantages of Eban’s ACASI in this study that provided a cultural familiarity aided in collecting sensitive data in vulnerable populations. One African American voice with an American standard accent asking each question for each of the four assessments, of each participant in exactly the same way, ensured the assessment’s uniformity across all participants in the study. Because data were entered directly by the research participant and there were guidelines embedded in the system to collect accurate data, there was less time spent cleaning the data. The most important advantage, however, is the fact that the research participants enjoyed using the ACASI system.
A special thanks to Shawn Ballard, Randy Hilderbrand and Edward Barrell from the Clinical Research Computing Unit (CRCU) within the DCC for developing, editing and testing the Eban ACASI system. Thanks also to David Metzger for consulting with the group in customizing ACASI for the AAC Study.