There is ample evidence that attaining high levels of survey participation is increasingly difficult (Berk et al. 2007
; Curtin et al. 2005
; Hartge 1999
; Hox, and de Leeuw 1994
; Morton et al. 2006
; Steeh et al. 2001
; Tickle et al. 2003
) and that deployment of a mixed-mode data collection protocol can be an effective way of increasing survey response rates.(Beebe et al. 2005
; Gallagher et al. 2000
; Griffin, and Obenski 2002
) However, emerging evidence suggests that a low response rate does not necessarily portend major study bias (Groves 2006
; Groves, and Peytcheva 2008
) and little evidence that mixing modes minimizes the latter. In our general population survey with an overall response rate of 51.2%, contrary to expectations, we found that switching modes from a mail survey to a telephone interview did not
uniformly increase the representativeness of the responding sample. Indeed, we found evidence that switching modes may make the sample less
representative of the population in terms of at least one clinical variable. Incidentally, we also found that a second contact in the same mode did not increase sample representativeness either.
Our finding that switching mode did not increase the representation of the final sample runs counter to the few studies investigating this issue. In the two studies most similar to our study with respect to order of contact, this approach yielded a more representative sample, although only one study had health and health care utilization for nonrespondents (Fowler et al. 2002
; Gallagher et al. 1999
). However, the populations from neither study were representative of the general population and, as such, may be more attuned to the nuances of data collection strategies and more susceptible to the deployment of specific modes. Tacit support for this notion is supplied by the juxtaposition of two studies deploying a mixed-mode design representing the converse of ours; initial telephone
contact followed by another mode (e.g., mail, web). Whereas switching to a mailed survey after a telephone interview reached a segment of the population quite different from the segment that would have been reached through telephone alone among adult patients enrolled in a trial to promote treatment for relapsed smokers at five Veteran’s Administration (VA) Centers (Baines et al. 2007
), a similar effect was not seen in a similarly-designed general population survey of close to 9000 households, albeit in an area unrelated to health (Dillman et al. 2009a
For general populations, switching modes may be more akin to a multiple attempt strategy, perceived only as an increased effort on our part to enlist cooperation, rather than the introduction of a new method, per se
. As such, our results are more aligned with the literature investigating the effects of multiple attempts on response rates (Davern et al., 2010
; Keeter et al. 2000
). The impact of additional measures to enlist participation, such as multiple contacts and/or switching modes, may actually bring in respondents for whom the topic is less salient leading to an under-representation of those who are less healthy and higher utilizers. This interpretation is consistent with Leverage-Salience Theory proposed by Groves and colleagues (Groves, Presser, and Dipko 2004
; Groves, Singer, and Corning 2000
) which posits that survey features, such as mode, could have variable leverage for different types of sample members and that switching modes may make a given survey more or less salient for certain types of people, thus increasing or decreasing participation. Regardless of the cause, it appears that use of a mixed mode approach does not represent a wholesale good when considering use among general population samples, particularly if the topic of survey pertains to health.
In considering our findings, we note potentially important limitations. Our data may not be generalizable to the U.S. population because the racial composition of the population is predominantly white; the prevalence of clinical disease status may vary by ethnicity, but at a minimum our data are probably generalizable to the U.S. white population. Additionally, our study relied on the medical chart to determine disease status and utilization, which may be subject to underreporting of mild symptoms or disease status. However, we assume that more severe symptoms or disease conditions would have been charted and that utilization history was accurately characterized as payment is based on such documentation. Finally, this relatively health literate population has been heavily surveyed and lives in close proximity to a well known medical center with close community ties which may have reduced nonresponse bias; the results may not apply in all other US population-based studies.
Survey researchers usually work with fixed resources and are faced with difficult choices of how to allocate efforts to maximize study goals. The choice to use multiple-modes of data collection is increasingly popular because it is assumed to serve multiple goals. First, starting with a relatively inexpensive mode such as mail, allows one to reach a substantial proportion of the sample at relatively low costs. Second, multiple modes typically are effective at reaching the goal of achieving higher response rates. The research presented here, however, suggests that it is overly simplistic to assume that reaching higher response rates in itself is consistent with a goal of reduced bias. Finally, sample size is also an important goal of survey research, especially when it comes to providing precise estimates for small subpopulations. Balancing the competing goals of survey research will always prove difficult, but further study of which types of designs actually reduce nonresponse bias is essential for informed decisions about how to allocate efforts.