The NSHAP team faced considerable challenges in developing the instruments and protocols required for the collection of questionnaire and biomeasure data. This section of the article describes how the team successfully integrated each of these components so that the data were collected efficiently, cost-effectively, and with appropriate quality.
Instrument and Biomeasure Protocol Development
NSHAP Wave I administration consisted of an in-home questionnaire, biomeasure collection, and a self-administered postinterview or “leave-behind” questionnaire. All questionnaires and survey materials were developed in English and translated into Spanish.
Development process for the questionnaire.—
The research staff met regularly over the course of several months to determine the measurement domains necessary to address the specific aims of the project. Once measurement domains were chosen, research staff conducted thorough reviews of relevant literature and contacted substantive experts to determine if existing instruments could be used, how much time the instrument took to administer, and validity and reliability of the instrument. If no existing instrument could be found, questions were developed specifically for the project.
A small pretest of the questionnaire and biomeasures procedures was conducted from July to September 2004 with 53 respondents. Unfortunately, the results of the pretest indicated that the in-home administration was taking significantly longer time to complete than the planned 2 hr. There were anecdotal reports of some interviews taking in excess of 4 hr to complete. Consequently, the questionnaire was subjected to a series of cuts, and the remaining questionnaire domains were prioritized according to the specific aims of the project. Any domain that was considered not essential to these aims was either cut completely or modularized (details of this modularization are provided subsequently). The questionnaire sections and domains for the in-home interview are shown in .
NSHAP Wave I Questionnaire Sections and Domains
Although most of the in-person questionnaire sections were administered by the field interviewer, a small subset of questions was determined to be sensitive enough to warrant self-administration (fifth section in ). These questions included the number of lifetime sexual partners (both male and female), lifetime use of prostitutes, frequency of masturbation, frequency of urinary problems, and stool incontinence.
Development process for the biomeasures.—
The integration of biomeasures into the NSHAP study design enhanced the study’s scientific objectives in three major ways. First, biomeasures permit estimation of the incidence and prevalence among older adults of certain diseases (e.g., bacterial vaginosis [BV], human papilloma virus [HPV]). Second, they permit the detection of undiagnosed or subclinical conditions such as hypertension and diabetes that may have a potent effect on sexual function and/or overall well-being. Third, they provide objective measures of health status (e.g., height, weight, vision) for use in conjunction with self-reported measures.
Biomeasure collection and shipping procedures were collaboratively developed over several years as part of a unique interdisciplinary NSHAP biomeasure working group. Because many existing protocols for collecting biomeasures were based on clinical or laboratory administration, the working group focused on ensuring that the measures were universally collectable in the home setting. Primary considerations included (a) adaptability to in-home collection and shipping, (b) adaptability to lay administration, (c) adaptability to the target population, (d) choosing portable, reliable, valid, and economical equipment, and (e) evaluating the impact of interviewer and respondent burden (for each measure and the sum of all measures in the interview).
The range of biomeasures collected in NSHAP represents a balance between the scientific value of the data, minimally invasive techniques accepted by respondents, practical considerations of implementation, and budget constraints. The pretest provided invaluable data and experience that showed that the collection of high-quality biomeasure data was possible using nonmedically trained interviewers; but it also showed that some of the protocols required streamlining or exclusion from the main data collection to improve administration efficiency and reduce respondent burden. Weighing the main aims of the study with the target time of around 2 hr spent in the home conducting the interview, the NSHAP research team agreed that the assessments of hearing, lung function (peak flow), mobility (chair stand), glucose test and toxicology (urine collection) should be dropped from Wave I administration.
A total of 13 biomeasures were collected for the first wave of the study:
- Waist circumference
- Blood pressure/pulse
- Saliva collection (for cotinine, dehydroepiandrosterone [DHEA], testosterone, progesterone, and estradiol)
- Get Up and Go test of mobility
- Distance vision
- OMT test for HIV (using Orasure®)
- Blood spots (for C-reactive protein [CRP], Epstein–Barr virus [EBV], hemoglobin, and hemoglobin A1c)
- Vaginal swab (self-administered) for female respondents (for BV, yeast vaginitis, and HPV)
Modularization of the Questionnaire and Biomeasures
As discussed, the pretest had shown that although the collection of comprehensive questionnaire items and an extensive range of biomeasures was feasible, the respondent burden and potential financial costs of the fieldwork were considered unacceptable. To decrease respondent burden and minimize in-home interview time while maximizing the analytic content of the interview, the NSHAP interview was modularized. All respondents received a set of core interview and biomeasure items. Five modules were developed: Two modules contained survey questions and three modules included the collection of biomeasures. Respondents were randomly assigned to one of six paths containing a set of these modules. Respondents assigned to paths where modularized interview questions were not asked were instead given a mail-in self-administered questionnaire that included these items. depicts the content of each randomly assigned path (note that this figure does not represent the order in which the questions were asked).
Overview of random path assignment and content in each path. FI = field interviewer.
The modularized questions were asked based on which one of the six paths the interview had been randomly assigned. Modules A and B were questionnaire modules. Module A contained items on support from family and friends, elder mistreatment, and nonsexual physical contact. Module B contained items on children and grandchildren, sexual interest, health care utilization, sexually transmitted infections, and medical decision makers. If a respondent did not receive Module A or B during the in-person interview, he or she received these items in the self-administered, postinterview questionnaire. Modules C, D, and E were biomeasure modules: Module C contained blood spot collection; Module D contained the Orasure® test (for HIV); and Module E contained the vision, touch assessments, and the Get Up and Go test of mobility and balance. Questionnaire items on perceived taste, touch, and smell were asked of respondents in Module E during the in-person interview but were omitted erroneously from the postinterview questionnaire provided to others.
Questionnaire Administration and Management
The main portion of the in-home NSHAP interview was conducted in English or Spanish using a Computer-Assisted Personal Interview (CAPI) questionnaire developed using SPSS’ mrInterview (Version 2.2). The software accommodated the complexity of the questionnaire but remained user-friendly. The CAPI program controlled which questionnaire and biomeasure modules were to be administered and which of the postinterview questionnaires was to be given to the respondent, so that the whole interview appeared to the interviewer and the respondent as one continuous smooth process. As described earlier, there were several questions that were considered particularly sensitive, so respondents were offered the choice of completing this series in one of two ways: reading the question text on the laptop computer and entering responses directly into the CAPI program or reading and answering the sensitive questions on a separate paper-and-pencil questionnaire. To further maintain privacy, these paper questionnaires were immediately sealed by the interviewer in a separate envelope and returned (still sealed) to NORC’s production center for processing.
To decrease the occurrence of interviewer data entry errors, both soft and hard range checks were programmed into the CAPI instrument. CAPI-programmed range checks alert the interviewer that an unlikely (soft check) or impossible (hard check) response was entered. An impossible response, such as entering a weight that is higher than the scale’s maximum measurable weight, cannot be entered into the CAPI, whereas an unlikely response can be entered into CAPI once the interviewer confirms it is correct. The interviewer instructions and the data fields for collecting the in-home biomeasure responses were also incorporated into the CAPI interview.
Procedures were carefully designed for all aspects of the NSHAP data collection including the sample frame and selection, field staff preparations, informed consent, processes to conduct the in-home interviews including the collection of biomeasures, and the postinterview questionnaire. The complexity of the interview required comprehensive and efficient protocols for all data collection tasks.
Field interviewer recruitment.—
Running parallel to the sample frame preparation and instrument development, the field staff was planning for data collection. As is the case in most field or telephone surveys, interviewers are often the “face” of the project to respondents and are therefore critical to its ultimate success. Consequently, NORC field interviewers completed a rigorous recruitment process before being staffed on NSHAP. Recruitment began several months prior to the start of data collection. All NORC field interviewers were sent a letter explaining the details of NSHAP in order to let them know when the project would be in the field. This letter helped to build enthusiasm for the project as well as creating a list of interviewers who were interested in working on NSHAP. Experienced interviewers were staffed wherever possible and were evaluated on their previous work and feedback from the field management staff. Experienced interviewers also participated in a project-based telephone interview. It was necessary to hire a handful of new-to-NORC interviewers in areas where the sampled areas did not overlap with areas in which NORC had existing field staff. New hires completed in-person mock interviews with field managers to evaluate their interviewing skills and proficiency with computers.
All interviewers were evaluated to ensure that they were comfortable asking questions about the sensitive issues addressed in the NSHAP interview. Because of the biomeasures included in the NSHAP study, the recruitment process also focused on the ability and willingness of interviewers to learn new skills. Potential interviewers were given fact sheets about the biomeasures that they would be collecting as well as information about storing and shipping biological samples. In addition, they were also shown pictures of the biomeasure collection techniques and when possible shown the actual equipment used to collect the biomeasures. The NSHAP project team wanted to make sure that interviewers were comfortable with all the tasks they would be asked to complete.
NSHAP hired both male and female interviewers, although the majority of the field interviewing staff were female. Due to some sensitive questionnaire items and biomeasures (e.g., self-administered vaginal swabs), only female interviewers were allowed to interview female respondents. Field management staff selected the most capable interviewers in their areas to comprise a national field interviewing staff.
Field interviewer training.—
Interviewer training included the completion of a home study packet, a four-day in-person training in Chicago, and booster trainings throughout data collection. Over the summer of 2005, NORC conducted three in-person trainings to prepare field interviewers to administer the NSHAP questionnaire and biomeasures. Having three training sessions allowed for smaller groups and more interaction between interviewers and the trainers in each of the training rooms. In total, 130 field interviewers and 12 field managers completed the NSHAP training.
Prior to the in-person training, interviewers were required to complete a home study packet that included reviewing the training manual, watching a video demonstrating biomeasure collection, and answering questions about the project. Interviewers then attended an in-person training that consisted of two project training days, a self-study day, and a certification day. The two project trainings days were taught through a series of brief lectures, hands-on practice drills and mocks, and individual instruction.
Project training focused on (a) field operations and administering the questionnaire and (b) collecting the biomeasures (Jaszczak, Lundeen, & Smith, 2009
). Field operations and questionnaire administration training included several mock interviews to give the interviewers in-depth experience with the questionnaire. Interviewers also completed modules addressing confidentiality, case management, gaining cooperation, and refusal aversion. For biomeasure training, a standardized technique was used for each of the 13 measures: The trainer first would explain the collection procedure; the trainer would then demonstrate the collection procedure with a “respondent”; and finally, the interviewer would practice the collection procedure. The interviewer practice step was an essential aspect of the training, and it was repeated at multiple times during the in-person training to enable the interviewer to become familiar and comfortable with the collection procedures. Individual and small group feedback was integrated into this training process to ensure correct collection techniques.
During the self-study day, interviewers rotated through a series of modules that provided individual instruction from trainers on such topics as blood spot collection and the shipping of biological samples. The final day of training required interviewers to pass a certification interview. NSHAP interviewers also had to complete at least one practice interview after returning home before entering the field. Additionally, booster trainings were held by phone during data collection to provide help with field efficiency and gaining cooperation for the biomeasures. Selected field interviewers participated in conference calls to address these issues.
The data collection began in a staggered format due to the timings of the three training sessions. Once all three training sessions had been completed, the entire interviewing staff was out in the field by mid-August 2005.
In an effort to show that the project was a serious and scholarly undertaking, a conscious decision was made to devote resources to developing a comprehensive set of high-quality project materials that had a project logo and a consistent layout and color scheme. All materials were tailored to meet the unique aspects of NSHAP. A series of respondent materials were developed to introduce the project to the respondent and aid the interviewers with gaining cooperation. Following standard practice in field studies, an advance letter was sent to the respondent’s home before the interviewer made the first contact. The NSHAP pretest protocol included a very detailed and informative advance letter. However, too much information seemed to overwhelm the respondents, making them less likely to participate. So a brief letter with limited detail to introduce NSHAP was developed that allowed the interviewer to explain the project in detail and provide additional materials at the door.
At the first contact, interviewers were provided with an array of tailored project materials to use as necessary to address respondents’ questions and concerns. Depending on the respondent’s questions or concerns, the interviewer could provide any of the following materials: an NSHAP brochure, examples of questions asked in NSHAP, a letter to community authorities, a dossier that contained letters of support and articles about the investigators’ research, a legitimacy handout, or the health information booklet that showed the results they would receive from the biomeasures.
If respondents were hesitant or initially refused, tailored advance letters were mailed to them to address their concerns. NSHAP created letters to address concerns for those who were too busy, not interested, did not like surveys, were worried about their privacy, did not like the government, did not want to participate in the biomeasures, or their child/spouse did not want them to participate. The $100 incentive to participate in the study was also very beneficial in gaining the cooperation of many respondents; although in the following, we discuss that the incentive was increased in the closing stages of the fieldwork to help boost the response rate. Interviewers also had great success in using small “in-kind” gifts such as small potted plants, a pumpkin (at Halloween), or gift cards.
Field managers worked very closely with the interviewers to help develop strategies for each individual case. The number and type of contact an interviewer made with a respondent varied greatly depending on the case. Interviewers and field management staff read through call notes from previous attempts to determine the best time of day to contact the respondent, what concerns they had, and any other relevant information that would help them make a connection with the respondent. Sometimes it helped to send another interviewer who had more similarities (e.g., race or age) with the respondent. Having interviewers travel from other regions to attempt the interview also proved to be very successful. Respondents were sometimes more willing to participate if the interviewer had a different accent or was unique in some way.
Respondents had the right to refuse any of the biomeasures at any time during the interview process. Consequently, gaining cooperation for each biomeasure was an important part of the interviewer’s job. The interviewers used several gaining cooperation techniques to increase biomeasure participation. Interviewers were thoroughly trained to explain each biomeasure, address concerns, and obtain consent throughout the interview. In addition, biomeasures were completed during the latter half of the interview when most respondents were engaged in the interview, had already built a strong rapport with their field interviewer, and recognized the importance of the data being collected. In the final months of data collection, fieldwork largely focused on refusal conversion. To increase the probability of converting these refusal cases and reach the goal of completing 3,000 interviews, the following steps were taken: (a) extend the end of the field period from December 2005 to March 2006, (b) mail a refusal conversion packet, (c) form a refusal conversion team, (d) increase interviewer traveling, and (e) increase the monetary incentive.
In December 2005, a refusal conversion packet was mailed to potential respondents who had not completed the interview. Anecdotally, the biggest concern respondents were reporting was a need to verify the legitimacy of the study. A one-page handout was developed addressing issues of legitimacy and sent via FedEx with a letter signed by the principal investigators and an NSHAP pen. It was hoped that respondents would be more likely to open the envelope by sending it FedEx and with a small trinket that they could feel through the packaging. At the same time, a group of talented NSHAP interviewers was also assembled to become part of a national refusal conversion team. Team members would contact respondents by telephone and use their finely tuned gaining cooperation skills to attempt to schedule interviews for local field interviewers to complete.
The last refusal conversion effort was focused around increasing the incentive amount offered to the respondent. To ensure that the optimal incentive was offered that would facilitate participation in NSHAP, a split-ballot incentive experiment was conducted (Wargo, Jaszczak, Smith, & O’Muircheartaigh, 2007
). A letter was sent via FedEx to a random sample of finalized refusal cases offering either a $200 or a $400 incentive. The letter also explained that the respondents would not be contacted again; instead respondents were provided with a toll-free number to schedule an interview. The $200 incentive resulted in an approximately 2% conversion rate (i.e., approximately 2% of those offered the $200 incentive then completed the interview). The $400 incentive resulted in an approximately 9% conversion rate. Based on the results of the experiment, the $400 incentive was offered to all remaining pending (i.e., in-scope but not interviewed) cases. Ultimately, 221 respondents (7% of those interviewed) received an increased incentive.
Results Notification for the Biomeasures
In addition to the monetary incentive, it was hoped that the in-home collection of the biomeasures and being provided with the results would be viewed by the respondent as an added benefit to participating in the study. Respondents received results in two phases: (a) immediately following the interview and (b) three weeks postinterview on a toll-free, anonymous call-in system. The biomeasures that were either experimental or not clinically validated (e.g., cotinine, DHEA, testosterone, progesterone, and estradiol from saliva, and CRP, EBV, hemoglobin, and hemoglobin A1c from blood spots) were not reported to respondents.
At the end of the interview, interviewers provided respondents with their height; weight; waist circumference; blood pressure; pulse; and ratings for smell, taste, mobility, vision, and touch. To provide these results, the interviewer was instructed to copy the results from the CAPI screen to a health information booklet that was left with the respondent. Interviewers were told not to comment on these results because they were not medically qualified to do so, but the booklet did provide guidance on several measures, such as whether their weight (using their body mass index) was below, within, or above a healthy weight, and whether their blood pressure was normal, prehypertension, or high.
The interviewers also provided respondents with information on how to obtain results from the anonymous counseling service for biomeasures that required additional processing at a laboratory. Specifically, respondents were given an identification number to access their results. To ensure anonymity and security, respondents were asked to create a unique password and answer a verification question to be used if they forgot their password. This call-in information was provided only to individuals who were eligible to receive laboratory results. To remind respondents to call for their results, NORC sent respondents letters to inform them that their results were ready.
NSHAP contracted with the American Social Health Association (ASHA) to provide anonymous counseling services. ASHA provided respondents with test results for HIV, BV, HPV, and yeast vaginitis; pretest and posttest result counseling; materials; and referrals via a toll-free number. The NSHAP call protocols used by ASHA counselors were collaboratively created by ASHA and NSHAP researchers. These protocols served as important tools and guidelines for hotline staff to use during NSHAP calls. To protect the confidentiality of the respondents, laboratory results were communicated to ASHA using de-identified respondent identification numbers. A reconciliation system was developed to ensure that respondents’ identification numbers and laboratory results were received and matched accurately.
Shipping of the Biomeasure Specimens
Blood spots, OMT, vaginal swab, and saliva samples collected for NSHAP were analyzed by several outside laboratories: Magee Women’s Health Corporation, Northwestern University’s Department of Anthropology, Salimetrics, and the University of Chicago’s Institute of Mind and Biology. Interviewers were responsible for storing, packing, and shipping the samples to the respective laboratories. Each interviewer was provided with detailed instructions on storage and shipping requirements, which existed both for the interviewers’ safety and to maintain the integrity of the sample. In particular, many samples had specific temperature requirements that had to be followed to ensure sample quality; for example, to limit bacterial growth, the saliva samples needed to be frozen as soon as possible after the interview and needed to be shipped on dry ice to avoid them thawing. All shipping protocols adhered to federal shipping guidelines for the proper packing of diagnostic samples.
Quality Control of the Biomeasure Specimens
Careful monitoring of the biomeasures was performed to ensure the quality of the data collected. Specifically, the quality control measures included:
- Reconciliation. Throughout the data collection period, NORC performed weekly reconciliation tasks to compare laboratory data with questionnaire data indicating whether or not a sample was collected. Tracking was performed with an interactive reconciliation system that allowed for the remote tracking of biological specimens from the point of in-home collection through the delivery in the final data set. Anomalies were investigated and resolved with the assistance of the field and laboratory staff, field notes, laboratory sample requisition forms, and FedEx tracking numbers.
- Laboratory sample quality. The quality of the samples collected and delivered to outside laboratories for analysis was monitored throughout the data collection period. NSHAP’s laboratory subcontracting partners provided ongoing feedback on the quality of the specimens received, including how they were collected, stored, and shipped, as well as if the results of the analyses fell within expected ranges. All four of the laboratories that processed NSHAP specimens reported receiving high-quality samples that were similar in quality and outcomes to other studies.
At the end of the in-person interview, respondents were given a paper-and-pencil postinterview questionnaire. Respondents were asked to complete this questionnaire and return it via U.S. mail in a provided postage-paid envelope. The interviewers were instructed to not stay in the home while the respondents completed this portion of the interview. The self-administered questionnaires were designed with simple instructions and few skip patterns and took approximately 30 min or less to complete. To encourage completion, respondents were told that a portion of their incentive payment (25%) was for completing and returning the postinterview questionnaire. However, the full incentive was paid at the time of the in-person interview, with written documentation itemized to note this “prepayment.”
After completing the in-person interview, most respondents were fully engaged in NSHAP and were willing to complete and mail in the postinterview questionnaire. Respondents who had not returned their questionnaire within three weeks of the in-person interview received a telephone call from a member of the NSHAP field staff. These calls served multiple functions including (a) thanking respondents for their participation in NSHAP, (b) inquiring if the postinterview questionnaire had been returned, and (c) offering to send a new version of the questionnaire if the original had been misplaced.
The postinterview questionnaires were mailed to NORC’s production center where they were receipted and data captured. The receipt control process was specifically designed to enable NORC staff to track the receipt of paper documents on a flow basis. Detailed specifications for computer-assisted data entry were developed to ensure proper range edits, skip patterns, and missing data flags, which all served to prevent errors and promote data integrity. Ten percent of the cases were reentered for verification and quality control purposes.
Institutional Review Board Approval and Informed Consent Procedures
Given the age of the study population, the collection of biomeasure data, and the sensitive or personal nature of some elements of the questionnaire, it was critical that protocols be developed to adequately protect the rights and welfare of human subjects in the study.
Institutional review board approval.—
Full approval was granted for NSHAP from two institutional review boards (IRBs): the Social and Behavioral Sciences Institutional Review Board at the University of Chicago (the Federal Wide Assurance [FWA] No. is FWA00005565) and the NORC Institutional Review Board (FWA00000142).
National Institutes of Health Certificate of confidentiality.—
The NSHAP in-home interview included sensitive information about physical and mental health, history of sexual and intimate partnerships, patient–physician communication regarding intimacy and sexuality, sexual function, behavior, desire, opportunity, and attitudes about sexuality. Additionally, disease status was verified during the biomeasure collection. For these reasons, the researchers obtained a Certificate of Confidentiality from the National Institute on Aging to further protect the privacy of the respondents (National Institutes of Health, 2003
). With this certificate, researchers cannot be forced to disclose information that may identify respondents, even by a court subpoena, in any federal; state; or local civil, criminal, administrative, legislative, or other proceedings.
Unique human subjects considerations.—
The NSHAP interview included components that made IRB approval different compared with most face-to-face interviews. Most notably, the integration of biomeasures into the in-home interview presented the researchers with unique human subjects considerations.
Before the NSHAP interview could be conducted, it was necessary for the interviewer to obtain informed consent from the respondent. To reduce any real or perceived respondent burden as well as to make the information contained in the consent documents more salient to the different questionnaire paths, the consent documentation process was structured to reflect a continuous approach to informed consent. The specific steps in the consent documentation process were as follows:
Verification to read consent. At the beginning of the interview, respondents were asked whether they would like to read the Questionnaire and Physical Measures Consent Form or have the interviewer read the form to them. To ensure that respondents did not receive information that did not apply to them, separate male and female versions of the consent form were used. The respondent was asked to sign the consent form (a copy of the form was given to respondents for their records).
Blood spot storage.
During the results notification, respondents who had provided a blood spot were given this particular form indicating whether they consented to the storage of their blood spots and, if so, whether or not they would like to be contacted before any potential future analysis is conducted. The decision to offer respondents a choice as to whether or not they would like to be notified before beginning analysis on stored blood specimens was based on our interpretation of the guidelines published by the National Bioethics Advisory Commission’s (1999)
Research Involving Human Biological Materials: Ethical Issues and Policy Guidance
, Vol. 1.
Call-in information. The final step of consent documentation also occurred during results notification and applied to respondents who provided an OMT and/or vaginal swab sample during the interview. The purpose of the form was to provide respondents with information about calling the toll-free number to access their results. In addition, respondents were asked to sign the form if they wanted to receive a results reminder letter.
The process of verbal consent continued throughout the interview, and the interviewer always confirmed willingness to participate in biomeasure collection prior to each assessment. At any time, the respondent was able to seek and obtain additional information regarding the questions or measures or withdraw their consent.
To maintain respondent and data confidentiality after the interview, the researchers instituted the use of an “honest broker” when dealing with the transmission of biological results. Specifically, the honest broker was one individual and a small number of directly supervised, trained assistants within NORC who functioned to protect respondent confidentiality. These specially designated individuals de-identified all data prior to transferring them to the investigator team.
To further protect respondent confidentiality, steps were taken to maintain the anonymity of the results from biological assays. During the NSHAP interview, blood spots, saliva, OMT, and vaginal swab samples collected from consenting respondents were labeled with unique identification numbers. These numbers provided no individually identifiable information to the laboratories. Samples were mailed by interviewers directly to the appropriate laboratories and labeled only with the unique identification number. Once analysis of a specimen was completed, laboratories sent results with the unique identification number to NORC.
At the completion of the interview, NORC transmitted the respondent’s unique identification number, password, and verification question to the results notification service (ASHA). One laboratory sent HIV, HPV, yeast, and BV test results with the unique identification number to the results service. No personally identifying information was contained in any transmittals. These results were then available to the respondents on the anonymous call-in system two to three weeks after the completion of the interview.