The plausibility of response patterns and how they vary with other covariates may indicate which format yields higher quality data. We investigate in this section how uncertainty in Social Security benefits correlates with uncertainty about related outcomes. For this analysis we restrict our sample to usable answers and exclude 10 respondents who provided an expected monthly Social Security benefit above $10,000 and who are thus asked about very wide bins.

We show in regression analyses using three measures of uncertainty in Social Security benefits as dependent variables: the number of bins used by the respondents, the probability mass allocated in the middle bin and a constructed standard deviation of the subjective distribution (see

Appendix 2 for details). We employ an ordered probit model when the number of bins used by the respondents is the dependent variable and look at the best linear predictors when the probability mass in the middle and the standard deviation are the dependent variables.

Uncertainty in the timing of future benefit receipt (expected claiming age) is likely to translate into uncertainty about the amount of future Social Security benefits. Also the longer the time to claiming the higher one would think the uncertainty that the individual faces regarding the Social Security benefit amount. For both formats respondents who used more bins to express the distribution of future claiming ages or those who are further away from to their expected claiming age expressed more uncertainty regarding their future Social Security benefits, i.e., they used more bins for the distributions of future Social Security benefits, allocated less probability mass in the central bin and have a distribution with a larger standard deviation.

Eligibility and the possibility of Social Security reform are likely to affect respondents’ expectations. Respondents were asked the probability that they would receive Social Security benefits in the future. Almost half of the respondents reported a probability of 95 percent or more. Those who report a lower probability of eligibility report a more spread out distribution of future Social Security benefits. The coefficient is statistically significant at 5% for the specifications using the total number of bins and the number of balls in the middle bin as dependent variables in the bins-and-balls format; it is significant at 10% for the number of balls in the middle bin for the percent chance format. Also, respondents who provide a higher probability that a reform would reduce their own benefits provide a more spread-out distribution regarding their future benefits. The coefficient is statistically significant for all formats and all dependent variables, except for the computed standard deviation in the bins-and-balls format.

Health may impact one’s ability to work, and thus one’s future benefits. Consistent with this idea, we find that, in the bins-and-balls format, respondents who report to be in excellent or very good health have a more concentrated distribution than those who report to be in poor health, and the coefficient is statistically significant in the specifications using the total number of bins or the number of balls in the middle bins. In the percent chance format, respondents who report to be in good health have a more concentrated distribution than those who report to be in poor health in the regression using the number of balls in the middle bin.

Wealth has a statistically significant coefficient for the percent chance format only: respondents in the higher wealth tercile have a more concentrated distribution than respondents in the lower wealth tercile. An explanation might be that respondents who have accumulated more wealth have better financial knowledge and know more about the Social Security rules and in many cases have contributed to Social Security at the maximum level for an extended period of time and therefore know that they qualify for the maximum benefit or something close to it.

Gender and income have a statistically significant coefficient for both formats only in the regressions using the computed standard deviation as dependent variables. Women and respondents with lower income are found to have a distribution with a smaller standard deviation.

Overall, the plausibility of response patterns and how they vary with other covariates suggest that both formats yield comparable quality data. The regressions presented in are however not without caveat for comparison between the two formats. First, the number of observations is larger in the bins-and-balls format. Caution needs to be taken when comparing the significance levels of coefficients across formats. Second, the sample of usable answers in the percent chance format suffers from selection (see section 2.3), and some of the relationships presented in the regression might be biased as a result.