PMCCPMCCPMCC

Search tips
Search criteria 

Advanced

 
Logo of nihpaAbout Author manuscriptsSubmit a manuscriptNIH Public Access; Author Manuscript; Accepted for publication in peer reviewed journal;
 
Public Opin Q. Author manuscript; available in PMC Sep 20, 2010.
Published in final edited form as:
PMCID: PMC2942770
NIHMSID: NIHMS205475
Eliciting Subjective Probabilities in Internet Surveys
Adeline Delavande and Susann Rohwedder
Adeline Delavande, RAND and Universidade Nova de Lisboa, 1776 Main Street, Santa Monica, CA 90407-2138, Tel 310-393-0411, ext 6593, Fax 310-260-8176, adeline/at/rand.org;
Individuals’ subjective expectations are important in explaining heterogeneity in individual choices, but their elicitation poses some challenges, in particular when one is interested in the subjective probability distribution of an individual. We have developed an innovative visual representation for Internet surveys that has some advantages over previously used formats. In this paper we present our findings from testing this visual representation in the context of individuals’ Social Security expectations. Respondents are asked to allocate a total of 20 balls across seven bins to express what they believe the chances to be that their future Social Security benefits would fall into any one of those bins. Our data come from the Internet Survey of respondents to the Health and Retirement Study, a representative survey of the U.S. population age 51 and older. To contrast the results from the visual format with a previously used format we divided the sample into two random groups and administered both, the visual format and the more standard percent chance format. Our findings suggest that the main advantage of the visual format is that it generates usable answers for virtually all respondents in the sample while in the percent chance format a significant fraction (about 20 percent) of responses is lost due to inconsistencies. Across various other dimensions the visual format performs similarly to the percent chance format, leading us to conclude that the bins-and-balls format is a viable alternative that leads to more complete data.
Studies of individual decision making in a dynamic setting need to take into account individuals’ expectations about future events. Traditionally, researchers have met this need by making assumptions about expectations or the expectation formation process. This shortcut is largely due to the lack of adequate data. Over the last 15 years there have been some important advances in the collection of information on subjective expectations, most importantly the move from a deterministic survey question format to one that captures some of the uncertainty necessarily associated with an event that has not yet materialized.1 Several general-purpose surveys now collect expectations data across several domains in the form of subjective probabilities rather than point expectations.2 Questions usually take the “percent chance” format
“On a scale from 0 to 100 where 0 means you think there is no chance and 100 means that you are absolutely certain, what do you think are the chances that […]”
Subjective probabilities collected in this manner have been shown to vary systematically with covariates in the same way as the actual outcomes, to be predictive of actual outcomes and of actual and expected economic behavior (e.g., Dominitz 1998, Hurd and McGarry 1995, 2002, Gan et al. 2004, Delavande and Willis 2008, Delavande 2008). However, this question format tends to produce focal answers, in particular bunching of responses at 50 percent, but also at 0 and 100. Bruine de Bruin et al. (2002) find that the fraction of focal answers is reduced when supplying a visual representation such as a response scale that explicitly shows the numeric response options from 0 to 100. Internet-based interviews open the possibility to much richer visual representations, including tailoring to individual respondents’ previous answers. We have designed a survey instrument that takes advantage of these features of the Internet and that goes beyond asking just about one point on individuals’ subjective probability distribution. Using an innovative visual representation we elicit information about individuals’ entire subjective distribution of beliefs in the context of individuals’ expectations about their future Social Security benefits. We present respondents with seven bins each of which represents a range into which an individual’s future Social Security benefits might fall. Respondents are asked to allocate a total of 20 balls across those seven bins to express what they believe the chances to be that their future Social Security benefits would fall into any one of those bins. We chose a design with 20 balls and seven bins as a compromise between respondent burden when allocating the balls and the precision of the elicited information.
We are not the first to elicit information from the entire distribution of individuals’ expected Social Security benefits. Dominitz and Manski (2006) have done so in the Survey of Economic Expectations using the percent chance format. They ask about six different points on the cumulative distribution function. Even though the format is somewhat repetitive for respondents they achieved a response rate of 97 percent among those who provided a minimum and a maximum Social Security benefit amount.3 Intuitively, the bins-and-balls format has several advantages: first, it avoids the repetitive series of questions inherent in the percent-chance format (due to asking about several different thresholds); second, it yields internally consistent data (no violations of monotonicity of the cumulative distribution function); third, the information comes in a form that visualizes an approximation of the probability density which may help respondents and, finally, by increasing the number of bins, it can provide more detail about the probability distribution in just one question.
To compare our bins-and-balls format with the percent chance format, we randomized our sample into two groups and administered the bins-and-balls format to one group and the percent chance format to the other. Both groups answered the questions over the Internet so that our comparisons of the two formats are not affected by interview mode effects. Our data come from the Internet Survey of respondents to the Health and Retirement Study (HRS) which is a longitudinal survey that is representative of the U.S. population age 51 and older. We link the observations on Social Security expectations from the Internet Survey to the rich background information collected on the same respondents in the main HRS survey. We can therefore investigate the informational content of the Social Security expectations; find how the uncertainty about expected Social Security benefits varies by individual characteristics and whether the bins-and-balls format differs in this respect from the percent-chance format. Our objective is to find whether and to what extent the visual format – rendered possible by the Internet mode – is a viable alternative for eliciting the distribution of beliefs and to what extent it might even be advantageous compared to the percent chance format.
1.1. The HRS Internet Survey
The data on subjective probabilities about future Social Security benefits come from a module of the HRS Internet Survey which is a supplementary survey of the Health and Retirement Study (HRS). The HRS is a panel survey that is representative of the U.S. population age 51 and over. In the core survey the HRS collects data on close to 20,000 individuals and their spouses in about 13,000 households.4 Eligibility for the second wave of the HRS Internet Survey, which we use in this study, is determined by whether a respondent reports regularly using the Internet in the core survey in HRS 2004 or HRS 2006. A random sub-sample of 7,261 respondents qualified, but only 77.7 percent were invited to participate in the Internet Survey; the remainder of the sample was retained as a control group.5 The data for the second wave of the HRS Internet Survey was collected in two phases: the first part of the sample (34.0 percent or 1,919 individuals) was invited to participate in the spring of 2006 (Phase I) and the second part of the sample (66.0 percent or 3,721 individuals) was invited to participate in the summer of 2007.6 In both phases the unit response rate, conditional on being invited to participate, was 70%. There were no breakoffs in phase I. In phase II, 2,618 respondents completed the entire survey out of 3,721 invited. Most of our analytical sample comes from phase II, but we bring in some data from phase I (see Section 1.3 for further detail).
1.2 Experimental design
The survey questions that we focus on in this study are embedded in a sequence eliciting several aspects related to future Social Security receipt: whether the person expects to receive any Social Security benefits in the future, at what age and how much. To elicit the subjective distributions of beliefs about future Social Security benefits, respondents are randomized into one of two different formats: (i) a new visual format that we designed and (ii) the standard percent chance format.7
1.2.1 Question wording of lead-up questions
Respondents who reported not currently receiving Social Security benefits are asked the percent chance that they will receive Social Security benefits some time in the future. Those who provide a positive probability are asked about their subjective expectations regarding their claiming age and future Social Security benefit amounts (conditional on receiving any). Before providing information on the subjective distributions of beliefs about the benefits amounts, respondents are first asked to give a point estimate:
How much do you expect your monthly Social Security benefits to be in today's dollars?
We use this point estimate to tailor the thresholds for the questions about individuals’ subjective distributions of their future benefits which follow next.
1.2.2 Subjective distributions of Social Security benefit amounts: Visual format
The visual format elicits information on individuals’ subjective probability distributions about their future Social Security benefits in a way that mimics the density function of their subjective beliefs. The introduction includes a couple of examples to familiarize respondents with the exercise:
Often people are uncertain about their future Social Security benefits.
In the next question, we ask you to think about what your monthly benefits might be. We will show you 20 balls that you can put in seven different bins, reflecting what you think are the chances out of 20 that your future Social Security benefits fall in each bin. The more likely you think it is that your benefits fall in a given bin, the more balls you should assign to this bin. For example, if you put all the balls in the bin $500 – $800, it means you are certain that the amount you will receive is between $500 and $800. Another example is illustrated on the next screen.
The next screen shows a graph of balls allocated across bins replicated in Figure 1 with additional explanations underneath. We included the examples of interpretation to provide guidance to those who may not be comfortable with thinking in terms of percentages.
Figure 1
Figure 1
Introductory example to bins-and-balls question
Figure 2 shows the screen that respondents see when allocating the 20 balls into 7 bins. In the design we chose 20 balls as the best compromise between respondent burden involved in the process of allocating balls and available precision. From response patterns to other percent chance questions in the HRS core we found that only rarely would anybody provide answers that require a smaller unit than 5 percent (=1 ball out of 20). Similarly, we chose 7 bins as a compromise between potentially obtaining additional detail from a large number of bins and constraints to fit the bins on the screen.
Figure 2
Figure 2
Screen where the respondent is asked to allocate balls
1.2.3 Subjective distributions of Social Security benefit amounts: Percent chance format
The percent chance format follows Dominitz and Manski (2006) in requiring respondents to provide multiple points on their cumulative distribution function. In particular, respondents are asked to provide four points by stating the chance that their benefits might lower or higher than a threshold using the following wording: 8
Often people are uncertain about their future Social Security benefits. In the next few questions we will ask you about the chances that your future Social Security benefits turn out to be higher or lower than certain values.
Now what about the chances that your Social Security benefits might be higher/lower: On a scale from 0 to 100, where 0 means no chance and 100 means you are absolutely certain, what do you think is the percent chance that your Social Security benefits will be more than $[X] per month?
The exact sequence of questions is provided in Appendix 1. We chose to use only four thresholds for the percent chance format, defining five different bins, to limit the amount of repetition resulting from separate questions about every one of the thresholds.
1.2.4 Design of thresholds
For both the visual and the percent chance format, the thresholds depend on respondents’ reported expected Social Security amount (see Section 1.2.1).9 The motivation for using tailored thresholds is presented in Dominitz and Manski (1997). First, it ensures that the thresholds cover a range that is relevant for the respondent, rather than asking about a range that might be too wide or too narrow and thus less informative. Second, it decreases potential anchoring effects where respondents’ answers might be influenced by the amounts associated with the thresholds.
We designed the thresholds to be centered around the respondents’ point estimate of their expected future Social Security benefits.10 They are computed in the same way for the visual and the percent chance format. However, the bins-and-balls format has two additional thresholds in the extreme right and left (see Figure 3 where A denotes the respondent’s point estimate). For example the central bin covers the range of (just under) +/− 15 percent of the respondent’s point estimate. The thresholds are rounded to the next 5 dollars. If a respondent did not provide a point estimate, we administered standardized thresholds.11
Figure 3
Figure 3
Threshold Design
1.3 Analytical sample
2,618 respondents participated in phase II of the HRS Internet wave 2. Of these 1,835 were administered the Social Security module.12 Data on expected Social Security benefits are only relevant to respondents who are not currently receiving these benefits and who give a positive subjective probability of receiving them some time in the future. This leaves us with 1,021 observations from phase II who were asked about their subjective distribution of future Social Security benefits, randomized across the percent format and the bins-and-balls format. The design and sampling in phase I followed the same principles, but the piloted version of the percent format was not exactly comparable to the bins-and-balls format. We therefore use only the bins-and-balls observations from phase I. This adds 397 observations from phase I to our analytical sample bringing the total number of observations to 1,418. Combining the two phases, 55 percent of respondents were randomized into the bins-and-balls format and 45 into the percent chance format.
While the original sampling frame is the population representative pool of HRS respondents, eligibility for the Internet survey is based on respondents using the Internet on a regular basis. In comparison with respondents from the entire HRS sample who do not yet receive Social Security benefits, we find that our analytical sample is more highly educated, slightly younger on average, more likely to live in the Midwest and less likely to live in the South. Most of these differences are rather small with the exception of education (41.5 percent with college or above compared to 32.6 percent). Respondents from the HRS Internet sample randomized into the two formats share similar characteristics (see Table 1). In Table 1, the only variable for which we can reject equality of the averages at 5% across the two formats is age, with respondents in the bins-and-balls format being slightly older on average (55.7 vs. 55.1 years old).13 We do not use weights in our analysis as we are presenting results for randomized groups to make advances in survey methodology rather than population estimates.
Table 1
Table 1
Descriptive statistics of the analytical sample
2.1. Response Rates
Item non-response rates, conditional on participating in the survey, are very low throughout: about three percent for the point expectations and less than two percent for the subjective distribution of beliefs for both formats. In the percent chance format 98.4 percent answered all four questions about the different thresholds.14 These minimal differences do not lead to any one of the question designs to stand out over the other.
2.2. Survey Time
Using timing data, we find only a small difference in how long it took respondents to answer the questions in the different formats (84.7 seconds to go through the two introductory screens of the visual format and to allocate the balls and 90.2 seconds to complete the sequence of the four percent chance questions15). Note that allocating the balls accounted for only 21 seconds on average. This suggests that if a researcher is interested in eliciting more than one distribution within the same survey, there might be gains in terms of survey time from using the bins-and-balls format, as the introduction screens can easily be skipped or shortened when eliciting additional distributions.
2.3. Usable Answers
A common problem with administering a series of percent chance questions to elicit multiple points on respondents’ subjective probability distribution about an event is that respondents may not respect the monotonicity across thresholds (i.e. the subjective probability of, say, Social Security benefits being more than $500 is larger than the probability of Social Security benefits exceeding any other higher threshold). Respondents whose answers violate this monotonicity property may not master the concept of probabilities and their answers are difficult to interpret and use in empirical analysis. This is what we call “unusable answers” in the context of the percent chance format. This problem does not arise in the bins-and-balls format. However, in the bins-and-balls setting respondents may fail to allocate all 20 balls. This is what we call “unusable answers” in this format.
More than 97 percent of the answers are usable for the bins-and-balls format. This is in stark contrast to the percent chance format where observations are lost first because some respondents do not answer all the four questions about the four different thresholds (1.6 percent), and second because a considerable fraction of respondents’ answers violate the monotonicity property (about 20 percent of those who answered all four questions). This not only reduces the usable data in the percent chance format, but it also introduces selection effects. Table 2 highlights that respondents who respected the monotonicity tend to be more educated and healthier; they also score higher on cognitive tests and are more likely to live in the Northeast and the Midwest. Those who violated the monotonicity property have a lower score in the serial 7 test (a test assessing working memory by asking respondent to subtract 7 from 100, and from each subsequent number for a total of five trials, Ofstedal et al., 2005) and seem to have more difficulty overall with other probabilistic questions. Using all the expectations questions from HRS 2006, we constructed an index which captures one’s ability to “think probabilistically” in the spirit of Lillard and Willis (2002). The index computes for a particular respondent the proportion of “Don’t know”, “Refuse” and “50 percent” out of all the expectations questions that are phrased using the percent chance format in the HRS core survey instrument. The average index is 20.0 percent among respondents who violated the monotonicity, compared to 17.9 among those who did not.
Table 2
Table 2
Characteristics of respondents in the percent chance format, by whether their answer respects basic principles of probabilities
2.4. Precision of Information
One of the advantages of the bins-and-balls design is that we can present a relatively large set of bins (or intervals) without increasing much the burden to respondents. In contrast, having answers for seven intervals requires asking respondents six percent chance questions. At the same time, however, the bins-and-balls format limits respondents’ precision in the allocation of the probability mass in a given bin. In the current design, respondents have to allocate 20 balls, which implies that the smallest positive probability that they can allocate to a bin is 5 percent. The percent chance format allows respondents to report positive probabilities with precision down to the single integer.
We evaluate to what extent respondents use the precision provided by each format. Table 3 presents the total number of bins used by respondents. While respondents randomized into the bins-and-balls format could use up to seven bins, only one percent chose to use more than five bins. We also check the proportion of respondents who provide a percent chance which is not a multiple of five. This is the case for less than 3.5 percent of the answers per bin.16 Overall, very few respondents take advantage of the full precision provided by the format they are given and the limited precision in the bins-and-balls format imposed by the number of balls is not expected to lead to any material difference between the bins-and-balls format and the percent format.
Table 3
Table 3
Distribution of total number of bins used by respondents (in percent)
2.5. Shape of the Elicited Distributions
In this section, we compare the shape of the elicited distributions. To make the data from the two formats directly comparable, we consider the probability mass (on a scale from 0 to 100) distributed across five bins, that is, we aggregate the number of balls in the first and second bin, and in the sixth and seventh bin for the bins-and-balls format. We restrict the analysis to respondents who provided usable answers, according to the definition above.
2.5.1 Most Common Answers
For both formats, the most common answer is to allocate all the probability mass in the middle which contains the respondent’s previously reported point estimate. However, this answer is more common in the visual format than in the percent chance format (21 percent vs. 10 percent).17 The second most common answer (6 percent of respondents) in the visual format is to allocate 50 percent of the probability mass in the middle bin, and 50 percent in the second bin. The third most common answer (5 percent of respondents) is to allocate 75 percent of the probability mass to the middle bin, and 25 percent to the fourth bin. For the percent chance format the second and third most common answers are different: two percent of the respondents allocated 50 percent of the probability mass in the middle bin and 50 percent in the fourth bin; while two percent allocated 50 percent of the probability mass in the middle bin and 50 percent in the fifth bin.
One potential drawback of the percent chance format that may be mitigated in a visual format is the tendency of respondents to use focal answers. People who answer “50 percent” may either truly believe that the chance is about half or they may be uncertain (e.g., Bruine de Bruin et al., 2000; Hill et al., 2006). We investigate the pattern of focal answers for the four percent chance questions in Table 4.18 As expected, zero-percent answers are substantially more frequent for the extreme thresholds (49 and 40 percent) as many respondents are fairly certain that their expected Social Security benefits will not be that far off their previously reported point estimate. Subjective probabilities about the two middle thresholds attract a large proportion of zeros and 50s (about 45 percent of the answers), and there appears to be excessive bunching at 50 percent. The tendency to answer 50 percent for the two middle thresholds may generate less probability mass in the middle bin for the percent chance format. We discuss this further in section 2.6 on anchoring bias.
Table 4
Table 4
Distribution of answers for the percent chance format (in percent)
2.5.2 Central Tendencies and Spread
The visual and percent chance format generate similar central tendency. Respondents tend to allocate most probability mass in the middle bin. Yet, the percent chance format generates more spread out distributions: respondents who were administered the percent chance format allocate on average more mass in the two extreme bins and less in the middle bin. On average the middle bin attracts 59.9 percent of the probability mass for the visual format compared to 43.0 in the percent chance format and the difference is statistically significant at the conventional level. The first bin attracts on average 2.7 and 9.6 percent of the probability mass respectively. The difference across formats is also reflected in the total number of bins used (see Table 3): respondents in the bins-and-balls format tend to use a smaller number of bins than those in the percent chance format. For example, 73 percent of the bins-and-balls format respondents used two bins or less, compared to 32 percent of the percent chance format respondents.
The difference in probability mass in the middle bin across formats is driven in part by two facts: respondents to the visual format are more likely to allocate all the probability mass in the middle bin (20% of the respondents vs. 10%) and less likely to allocate no probability mass in the middle bin (9% of the respondents vs. 17%). Given that the middle bin contains respondents’ own previously reported point estimate, this latter group seems surprisingly high in the percent chance format.
We investigate in more detail whether the two formats generate similar means of the subjective distributions. To construct the individual-specific mean of the elicited subjective distribution, we assume that the probability mass reported by a respondent is uniformly distributed within a bin (see the Appendix 2 for details). Table 5 presents the distribution of these computed means for both formats and for the point expectations and shows that the percentiles of these three distributions are very similar.19 We cannot reject equality of the computed means of the percent chance format and the bins-and-balls format using a t-test; nor can we reject equality of the means of the point estimates compared with the computed means derived from either format. Despite generating different dispersion and shapes of distributions, the two formats yield distributions with very similar means that are in line with the point expectations.
Table 5
Table 5
Distribution of the point expectations and computed means of the elicited distribution
2.5.3 Symmetry of the Distributions
Restricting our sample to respondents who did not allocate all the probability mass in one bin Table 6 shows that respondents who were administered the percent chance format are more likely to put the same probability on either side of the central bin, that is, they provide a symmetric distribution (20.5 percent vs. 5.5 percent in the bins-and-balls format).20
Table 6
Table 6
Distribution of answers excluding respondents allocating all the probability mass in the central bin
The higher proportion of asymmetric distributions in the bins-and-balls format may indicate that visualizing the density helps respondents to report the shape they have in mind.
2.6. Anchoring Bias
We have found that the visual format attracts more probability mass in the middle than the percent chance format. We investigate whether this may be due to anchoring toward the middle, possibly because respondents are attracted visually towards the center of the scale and/or possibly because they see all the thresholds at once and again tend towards amounts that are closer to the midpoint of the amounts shown on the screen. To test this hypothesis we compared the subjective distributions obtained from two different bins-and-balls formats that we specifically designed for this purpose.21 We randomly administered to some respondents the same visual format as in the HRS Internet survey, and to the other respondents a similar design where the third bin, rather than the middle bin, is centered around the point estimate. We find that for both groups most of the probability mass is attracted to the bin containing the point estimate, even though for one group this bin is not located in the middle.22 Respondents tend to provide strikingly similar subjective distributions relative to the point estimate when the point estimate is located in the middle bin or in the third bin. This finding lends support that there is no anchoring bias toward the middle in the bins-and-balls format.
In view of the differences in the distributions generated by the bins-and-balls format compared to the percent format, does this imply that the percent chance format generates distributions with excessive dispersion? We believe that this is the case. Hurd (2008) documents that the tendency to report probabilities that are drawn towards 50 percent, and excessive bunching at 50 in particular, is found across percent-chance format questions from several domains. For example, Hurd shows that it introduces considerable bias to the population distribution of subjective probabilities on survival and on subjective probabilities on working past age 62. If these patterns also hold in the responses to the percent chance format about Social Security benefit amounts they would bias the distribution towards finding larger dispersion. This is because in each percent chance question we ask about the chances that benefits might be further away from the initial point estimate. In addition, the surprisingly large proportion of respondents (see Section 2.5.2) who allocate no probability mass in the middle bin that contains respondents’ own previously reported point estimate may further suggest a bias in the dispersion of the elicited distribution in the percent chance format.
2.7. Unfolding bins
The objective behind the bins-and-balls format was to generate data that even in its raw form approximates respondents’ subjective probability distributions. However, a large proportion of respondents in the bins-and-balls format used only one or two bins revealing only limited detail about their distributions. Taking advantage of the possibilities offered by the Internet survey mode we administered a follow-up bins-and-balls screen to respondents who provided a concentrated distribution of beliefs, i.e. those who allocated all the balls into one or two adjacent bins. These “unfolding bins” split the initial bin(s) containing all the balls into narrower bins and the respondent is re-asked to place 20 balls into those narrower bins (See Appendix 3 for exact wording). The number of new bins introduced varies between 2 and 5 bins depending on the width of the initial bin(s) where all the probability mass was placed (See Appendix 4 for details).
Out of the 386 respondents from phase II who were randomized into the visual format, 186 were eligible for the unfolding bins. Ninety-four percent of those eligible re-allocated the 20 balls into the narrower bins. Table 7 gives the distribution of unfolding bins that respondents used. About 80 percent of those who were presented with two unfolding bins allocated all the balls into one bin. About two thirds of those who were presented with three, four or five unfolding bins used two bins. As before respondents tend to provide distributions concentrated around their point estimates, with on average more probability mass to the right of the point estimates.
Table 7
Table 7
Number of unfolding bins that respondents used
This suggests that respondents are willing to provide more precise answers about their distributions of beliefs if asked. The bins-and-balls format is a very efficient way of doing so because only one additional screen is needed irrespective of the level of additional detail to be elicited. As such this application is particularly well suited to Internet surveys. While a similar follow-up could be implemented in the percent chance format, this would increase survey time (see discussion in section 2.2) and the number of repetitive questions and with that respondent burden more heavily than in the bins-and-balls format.
The plausibility of response patterns and how they vary with other covariates may indicate which format yields higher quality data. We investigate in this section how uncertainty in Social Security benefits correlates with uncertainty about related outcomes. For this analysis we restrict our sample to usable answers and exclude 10 respondents who provided an expected monthly Social Security benefit above $10,000 and who are thus asked about very wide bins.
We show in Table 8 regression analyses using three measures of uncertainty in Social Security benefits as dependent variables: the number of bins used by the respondents, the probability mass allocated in the middle bin and a constructed standard deviation of the subjective distribution (see Appendix 2 for details). We employ an ordered probit model when the number of bins used by the respondents is the dependent variable and look at the best linear predictors when the probability mass in the middle and the standard deviation are the dependent variables.
Table 8
Table 8
Multivariate analysis
Uncertainty in the timing of future benefit receipt (expected claiming age) is likely to translate into uncertainty about the amount of future Social Security benefits. Also the longer the time to claiming the higher one would think the uncertainty that the individual faces regarding the Social Security benefit amount. For both formats respondents who used more bins to express the distribution of future claiming ages or those who are further away from to their expected claiming age expressed more uncertainty regarding their future Social Security benefits, i.e., they used more bins for the distributions of future Social Security benefits, allocated less probability mass in the central bin and have a distribution with a larger standard deviation.
Eligibility and the possibility of Social Security reform are likely to affect respondents’ expectations. Respondents were asked the probability that they would receive Social Security benefits in the future. Almost half of the respondents reported a probability of 95 percent or more. Those who report a lower probability of eligibility report a more spread out distribution of future Social Security benefits. The coefficient is statistically significant at 5% for the specifications using the total number of bins and the number of balls in the middle bin as dependent variables in the bins-and-balls format; it is significant at 10% for the number of balls in the middle bin for the percent chance format. Also, respondents who provide a higher probability that a reform would reduce their own benefits provide a more spread-out distribution regarding their future benefits. The coefficient is statistically significant for all formats and all dependent variables, except for the computed standard deviation in the bins-and-balls format.
Health may impact one’s ability to work, and thus one’s future benefits. Consistent with this idea, we find that, in the bins-and-balls format, respondents who report to be in excellent or very good health have a more concentrated distribution than those who report to be in poor health, and the coefficient is statistically significant in the specifications using the total number of bins or the number of balls in the middle bins. In the percent chance format, respondents who report to be in good health have a more concentrated distribution than those who report to be in poor health in the regression using the number of balls in the middle bin.
Wealth has a statistically significant coefficient for the percent chance format only: respondents in the higher wealth tercile have a more concentrated distribution than respondents in the lower wealth tercile. An explanation might be that respondents who have accumulated more wealth have better financial knowledge and know more about the Social Security rules and in many cases have contributed to Social Security at the maximum level for an extended period of time and therefore know that they qualify for the maximum benefit or something close to it.
Gender and income have a statistically significant coefficient for both formats only in the regressions using the computed standard deviation as dependent variables. Women and respondents with lower income are found to have a distribution with a smaller standard deviation.
Overall, the plausibility of response patterns and how they vary with other covariates suggest that both formats yield comparable quality data. The regressions presented in Table 8 are however not without caveat for comparison between the two formats. First, the number of observations is larger in the bins-and-balls format. Caution needs to be taken when comparing the significance levels of coefficients across formats. Second, the sample of usable answers in the percent chance format suffers from selection (see section 2.3), and some of the relationships presented in the regression might be biased as a result.
This paper presents the results of an exploratory data collection that we undertook to elicit subjective probabilities using a new visual format made possible in Internet surveys. Our point of departure was that a visual format might elicit more detailed information about individuals’ distribution of beliefs without increasing respondent burden, and allow respondents to visualize the density s/he provides. In Internet surveys this visual format could provide an alternative to the more standard percent chance format. The percent chance format has been shown in previous work to yield coherent subjective distributions, but it may require respondents to be more proficient with probabilities than would be required with the visual format.
When comparing the two formats, we show that they yield similar response rates, survey time and precision of information. In addition, the dispersion of the elicited distribution correlates with other sources of uncertainty in the expected direction for both designs suggesting that both yield high quality data from a substantive point of view.
However, the two formats differ importantly in two dimensions. First, the visual format attracts more probability in the central bin than the percent chance format. Additional experiments lead us to conclude that this is not due to anchoring towards the middle in the bins-and-balls format. We suspect instead that the difference is due to biased distributions in the percent chance format. Hurd (2008) shows for several percent chance questions in the core HRS survey an excessive tendency to report values drawn towards 50. An increased tendency to answer 50 percent combined with the fact that in our percent chance design respondents were not asked directly about the middle bin would generate distributions that are biased towards finding larger spread.
Second and most importantly, 20 percent of the observations are lost in the percent chance format because respondents fail to respect a basic property of probability. Moreover, those 20 percent tend to be less educated, less healthy, less wealthy and less comfortable with probabilities questions of the percent format than the rest of the sample introducing selection in the remaining usable answers. This is a serious drawback because it prohibits the study of expectation formation and decision making under uncertainty among the most vulnerable group in the population which could potentially benefit most from public policies. While the fraction of unusable answers in the percent format may be reduced if respondents were informed about this violation, little is known about whether adding warnings or teaching respondents about probabilities improves the resulting data quality. Overall, our results suggest that the bins-and-balls format for eliciting individuals’ subjective distribution of beliefs in Internet surveys is a feasible option that leads to more complete data.
Acknowledgments
This research was supported by a grant from the National Institute on Aging (P01AG008291). Rohwedder is grateful for additional funding from the National Institute on Aging provided under grant R03AG024269. We would like to thank Norbert Schwarz and other participants of the 2007 conference on “Subjective Probabilities and Expectations: Methodological Issues and Empirical Application to Economic Decision-Making” in Jackson, Wyoming for helpful comments on an early draft of this paper. We are also grateful to two anonymous referees and the editor for their constructive advice.
Appendix 1: Wording of the percent chance question
Often people are uncertain about their future Social Security benefits. In the next few questions we will ask you about the chances that your future Social Security benefits turn out to be higher or lower than certain values.
<next screen>
Now what about the chances that your Social Security benefits might be higher: On a scale from 0 to 100, where 0 means no chance and 100 means you are absolutely certain, what do you think is the percent chance that your Social Security benefits will be more than $[X3] per month?
<next screen>
If answer to previous question>0, then:
Still about the chances that your Social Security benefits might be higher: On the same scale, what do you think is the percent chance that your Social Security benefits will be more than $[X4] per month?
<next screen>
Now what about the chances that your Social Security benefits might be lower?: On a scale from 0 to 100, where 0 means no chance and 100 means you are absolutely certain, what do you think is the percent chance that your Social Security benefits will be less than $[X2] per month?
<next screen>
If answer to previous question>0, then:
Still about the chances that your Social Security benefits might be lower: On the same scale, what do you think is the percent chance that your Social Security benefits will be less than $[X1] per month?
Where X1 < X2 < X3 < X4. Half the sample is first asked about the chances that benefits could be higher and then about the chances that benefits might be lower. For the other half the ordering is reversed.
Appendix 2: Construction of the mean and variance for the distribution of future Social Security benefits
We use respondents’ answers about the subjective distribution of future Social Security benefits to compute individual-specific mean and variance for this distribution based on two assumptions. First, we assume that the probability mass reported by a respondent is uniformly distributed within an interval (or bin). Second, we make assumptions on the support of the distribution. For the bins-and-balls format, we assume that the extreme right bin is bounded and has the size of the other bins. For the percent chance format, we also assume that the upper interval is bounded to the right such that the size of the extreme right interval is comparable to that of the bins-and-balls format.
Based on these assumptions, we compute the expectation of the distribution as follows:
equation M1
Where Ti is i’s bin’s left threshold, and M=5 for the percent chance format, and M=7 for the bins and balls format.
Under the uniform assumption, equation M2 for Ti+1xTi, where bi is the probability mass allocated between [Ti+1,Ti].
Therefore:
equation M3
c
Based on these assumptions, we compute the variance of the distribution as follows:
equation M4
Appendix 3: Wording of the introduction for the unfolding bins
Thank you for your answer. In order to get more precise information, we have now narrowed the size of the bins. By clicking on the + and − buttons under each bin, please put the 20 balls into the bins such that it reflects what you think are the chances out of 20 that your monthly Social Security benefits fall in each bin.
Appendix 4: Thresholds definition for the unfolding bins
Let L1 be the left bound of the unique bin or one of the adjacent bins, and L2 be the right bound of the unique bin or one of the adjacent bins. The following algorithm summarizes the design of the unfolding bins:
  • If L2-L1>=300 and L2-L1<450, use 2 bins of equal width starting at L1.
  • If L2-L1>=450 and L2-L1<600, use 3 bins of equal width starting at L1.
  • If L2-L1>=600 and L2-L1<750, use 4 bins of equal width starting at L1.
  • If L2-L1>=750, use 5 bins of equal width starting at L1.
So for example, if a respondent first allocated 20 balls into a bin ranging from $1,710 to $2,280, he is then asked to allocate 20 balls into 3 bins: $1,710 to $1,900, $1,900 to $2,090 and $2,090 to $2,280.
If the respondent allocated 20 balls into one bin of width smaller than $300, or into two bins of total width smaller than $450, the unfolding bins are not asked.
Footnotes
1For an overview of and the state of knowledge of expectations data, see Manski (2004).
2Examples of large household surveys that have collected data on individuals’ subjective expectations are the Health and Retirement Study, the Panel Study of Income Dynamics, the National Longitudinal Survey of Youth, the Michigan Survey of Consumers, the Survey of Economic Expectations, and the Survey of Health, Ageing and Retirement in Europe. See for example Hurd and McGarry (1995), Dominitz and Manski (1997, 2004, 2007) and Fischhoff, et al. (2000) for analyses of these expectations data.
3In Dominitz and Manski (2006), only respondents who provided a smallest and a highest possible value of expected Social Security benefits were asked the six follow-up questions to elicit several points on individuals’ subjective probability distribution. However, a large fraction (31%) of the sample did not provide the minimum or maximum value resulting in a significant reduction in usable answers.
4More precisely, in 2004 the HRS interviewed 20,129 individuals in 13,645 households achieving a response rate of 87.8%; in 2006 HRS interviewed 18,469 individuals in 12,605 households and the response rate was 88.9%.
5The purpose of the control group is to allow HRS to study potential effects of participation in the Internet Survey on participation in subsequent interviews of the core survey.
6The reason for postponing the interview to a later second phase for one group of the sample was that this group had been previously assigned (at random) to participate in another supplemental study and would have had three HRS-related interviews within months of each other had their internet interview not been postponed.
7Respondents are also randomized into whether they are asked about their future Social Security benefits conditional or unconditional on their self-reported expected claiming age. In earlier work (Delavande and Rohwedder 2007), we show that conditioning on expected claiming age leads to distributions that are not statistically different from each other. This is because the respondents’ underlying distribution of expected claiming ages is almost universally symmetric around the point estimate of the expected claiming age resulting in conditional and unconditional distributions that are very similar. In this paper we therefore pool respondents who were asked their expectations conditional or unconditional on claiming age.
8Within the percent chance format there is another randomization: half the sample is first asked about the chances that benefits could be higher and then about the chances that benefits might be lower. For the other half the ordering is reversed. The rationale for this randomization is that the “higher-” or “lower-“ wording might lead to anchoring bias in the answers which would average out in summary statistics at the population level. While this does not generate a different pattern of answers for the probability mass lower than a threshold, we find that the average probabilities about the chance that benefits might be higher than certain values differ statistically using a t-test depending on the order.
9The median expected amount is $1,100 per month.
10Dominitz and Manski (2006) also use preliminary questions to determine individual-specific thresholds. However, rather than using the point estimate they ask respondents about the lowest and highest future Social Security benefits amount. We decided to use the point estimate because non-response to these lowest and highest amounts was high in their survey (about 30 percent).
11The standard thresholds are $0, $300, $600, $900, $1,200, $1,500 and $1,800 for the bins-and-balls format, and $600, $900, $1,200, $1,500 for the percent chance format.
12Beginning in the second week of the field period, respondents age 65 or older were no longer asked the Social Security module in phase II of the HRS Internet wave 2. This step was taken to reduce survey burden for respondents after it turned out that the average time to complete the entire survey, which included several other modules, was longer than anticipated.
13Due to competing survey time, respondents age 65 or older who were asked the Social Security module in phase I were not longer asked in phase II, resulting in a slightly older sample for the bins-and-balls format since some of the sample comes from wave I.
14As a comparison, about 96.8 percent of respondents to the Survey of Economic Expectations answered the six thresholds about future Social Security benefits (Dominitz and Manski, 2006).
15The medians are 61 and 65 seconds respectively.
16Similar patterns are reported in Manski and Molinari (2008) for the whole HRS.
17Respondents with all the probability mass in the middle bin are those who answered 0 percent to all percent chance questions.
18Note that the percent chance questions do not ask directly about the central bin which contains the point estimate.
19Table 5 excludes 12 respondents who reported a monthly point estimate above $10,000.
20Note that out of the 20 percent of respondents to the percent chance format who provided equal probability on either side, about half gave the same answers to thresholds 1 and 3 and the same answers to thresholds 2 and 4. (e.g. the sequence of answers would be for example 20, 10, 20, 10).
21These data were collected as part of the American Life Panel (ALP) Internet survey. See http://rand.org/labor/roybalfd/american_life.html for details about the ALP.
22If we combine the two designs into comparable bins, we cannot reject equality of the average number of balls for each of the bins at the 5 percent level.
Contributor Information
Adeline Delavande, RAND and Universidade Nova de Lisboa, 1776 Main Street, Santa Monica, CA 90407-2138, Tel 310-393-0411, ext 6593, Fax 310-260-8176, adeline/at/rand.org.
Susann Rohwedder, RAND, 1776 Main Street, Santa Monica, CA 90407-2138, Tel 310-393-0411, ext 7885, Fax 310-541-6923, susannr/at/rand.org.
  • Bruine de Bruin Wändi, Fischbeck Paul S, Stiber Neil A, Fischhoff Baruch. What number is 'fifty-fifty'? Redistributing excess 50% responses in risk perception studies. Risk Analysis. 2002;22:725–735. [PubMed]
  • Bruine de Bruin Wändi, Fischhoff Baruch, Millstein Susan G, Halpern-Felsher Bonnie L. Verbal and Numerical Expressions of Probability: ‘It’s a Fifty-Fifty Chance’ Organizational Behavior and Human Decision Processes. 2000;81:115–131. [PubMed]
  • Dominitz Jeff. Earnings Expectations, Revisions, and Realizations. Review of Economics and Statistics. 1998;80:374–388.
  • Dominitz Jeff, Manski Charles F. Expected Equity Returns and Portfolio Choice: Evidence from the Health and Retirement Study. Journal of the European Economic Association. 2007;5:369–379.
  • Dominitz Jeff, Manski Charles F. Measuring Pension-Benefit Expectations Probabilistically. Labour. 2006;20(2):201–236.
  • Dominitz Jeff, Manski Charles F. How Should We Measure Consumer Confidence? Journal of Economic Perspectives. 2004;18:51–66.
  • Dominitz Jeff, Manski Charles F. Using Expectations Data to Study Subjective Income Expectations. Journal of the American Statistical Association. 1997;92:855–867.
  • Delavande Adeline. Pill, Patch or Shot? Subjective Expectations and Birth Control Choice. International Economic Review. 2008;49:3.
  • Delavande Adeline, Rohwedder Susann. Rand Corporation, Working Paper. 2007. Eliciting Subjective Probabilities about Social Security Expectations.
  • Delavande Adeline, Willis Robert J. Managing the Risk of Life. Michigan Retirement Research Center Working Paper 2007-167. 2008.
  • Fischhoff Baruch, Parker Andrew M, Bruine de Bruin Wändi, Downs Julie, Palmgren Claire, Dawes Robyn, Manski Charles F. Teen Expectations for Significant Life Events. Public Opinion Quarterly. 2000;64:89–205. [PubMed]
  • Gan Li, Gong Guan, Hurd Michael D, McFadden Daniel L. Subjective Mortality Risk and Bequests. NBER Working Paper No. W10789. 2004.
  • Hill Daniel, Perry Michael, Willis Robert J. University of Michigan, Working Paper; 2006. Estimating Knightian Uncertainty from Survival Probability Questions on the HRS.
  • Hurd Michael D. Subjective Probabilities in Economics. RAND. 2008 manuscript.
  • Hurd Michael D, McGarry Kathleen. The Predictive Validity of the Subjective Probabilities of Survival. Economic Journal. 2002;112:966–985.
  • Hurd Michael D, McGarry Kathleen. Evaluation of the Subjective Probabilities of Survival in the Health and Retirement Study. Journal of Human Resources. 1995;30:268–292.
  • Lillard Lee A, Willis Robert J. Cognition and Wealth: The Importance of Probabilistic Thinking. Michigan Retirement Research Center Research Paper No. WP 2001-007. 2002.
  • Manski Charles F. Measuring Expectations. Econometrica. 2004;72:1329–1376.
  • Manski Charles F, Molinari Francesca. Northwestern University, Working Paper; 2008. Rounding Probabilistic Expectations in Surveys. [PMC free article] [PubMed]
  • Ofstedal Mary Beth, Fisher Gwenith G, Herzog A Regula. Documentation of Cognitive Functioning Measures in the Health and Retirement Study. HRS Documentation Report DR-006. 2005.