Table presents the number of correct responses to individual ICBM items, and indicates any group differences in correct responding. An indication of linear trend (students, trainees, consultants) in correct responses is also provided. Item names are based on the key content of each scenario. Table summarises all relevant study variables. Probability values for both absolute group difference and linear trend are again included. While the total sample comprised approximately 61% males, they were differentially distributed across the student, trainee and consultant subgroups. Predictably, age increased from students to trainees to consultants. Need for cognition provided subtle and varying results across the subsamples, while there was a clear trend toward a decrease in faith in intuition with experience. Correct responses to both the ICBM22 and a 10 item ICBM (ICBM10), based on findings from an unpublished data (items 2, 4, 5, 6, 7, 10, 13, 14, 15, 17), increased with experience.
Correct responses to ICBM items.
Summary statistics for key study variables.
The internal reliability (α) of the ICBM22 was 0.57 (0.55 students, 0.52 trainees, 0.56 consultants), indicating a relatively poor level of internal consistency. The ICBM10 demonstrated similarly poor reliability of 0.57 (0.55 students, 0.54 trainees, 0.52 consultants). In light of these internal consistency figures, two strategies were used in the attempt to improve this property of the ICBM.
Item-total Correlation Analysis
First, iterative removal of those items with the lowest item-total correlation was undertaken, identical to the procedure previously used to derive the ICBM10. With the current data, α was maximised with a 17 item scale (ICBM17). Items 8, 11, 18, 21 and 22 were omitted. These items assessed misconceptions of chance (8), framing or anchoring bias (11 and 22), insensitivity to superior reliability of objective over subjective data (18), and insensitivity to the principle of regression (21). The resultant 17 item scale achieved an α of 0.61 (0.56 students, 0.57 trainees, 0.61 consultants), with item-total correlations ranging from 0.11 to 0.37.
Removal of a further item resulted in the total sample α being maintained at 0.61 but a reduction in consistency for two of the three subsamples (0.58 students, 0.55 trainees, 0.59 consultants). This result suggested that further gains in internal consistency could only be attained through differential item removal for each subsample. That is, different scales for different target groups. Summary statistics for the ICBM17 are included in Table . As with the ICBM22, correct responses increased with experience.
Second, factor analysis was used as a pragmatic guide to potential subscale membership among the ICBM items [22
]. Decisions regarding the nature of the factor analysis were based on this strategy. Maximum likelihood extraction was used to allow generalisation from a sample to a population [23
] and for correlations with more unique variance and less error variance to be given more weight [22
]. Most importantly, adjustment is made for the constraints imposed on the data based on the increased potential for non-random measurement error associated with dichotomous variables.
This procedure resulted in an initial solution comprising 10 factors, each with an eigenvalue greater than one, although only 60% of the variance among items was accounted for by these factors. This result reflects the tendency for dichotomous variables to cluster together due to similar response distributions rather than actual item content [24
], producing additional factors that are essentially statistical artefacts. Therefore, a conservative decision rule for retaining factors for rotation was employed. Parallel analysis criteria [25
] takes account of both sample size and the number of items being analysed and is more reliable than the misunderstood 'eigenvalues greater than one' rule [26
], particularly when there are many coefficients of modest size within the available correlation matrix (communality range 0.025 to 0.197). Further, an oblique rotation of the retained factors was undertaken using the oblimin criterion (delta = 0), to acknowledge the expected intercorrelations among any dimensions of the ICBM.
Using this strategy only two factors were retained for rotation, accounting for a mere 19% of variance. Scales were nevertheless computed from these factors. Scale 1 (ICBMs1) comprised items predominantly concerned with insensitivity to sample size and representativeness (items 1, 4, 5, 6, 9, 10, 12, 14, 15, 16, 17, 20). Scale 2 (ICBMs2) tended to tap availability and representativeness (items 2, 3, 4, 6, 7, 10, 12, 13, 14, 15, 19). The substantial overlap in the content of these two scales should be noted (r = .68, p < .001). Interestingly, the five items that did not load on either or both of these scales were identical to those excluded by the item-total correlation analysis. Summary statistics for these scales are included in Table . For both scales, correct responses increased with experience. Internal consistency was 0.58 (0.52 students, 0.56 trainees, 0.61 consultants) for ICBMs1 and 0.58 (0.60 students, 0.47 trainees, 0.51 consultants) for ICBMs2.
Correlations between the various versions of the ICBM that we have presented, and scores from the REI, are shown in Table separately for students, trainees and consultants. Among students there are a number of significant, albeit modest, coefficients. In accord with theory, all ICBM versions are negatively correlated with faith in intuition, although this is accounted for by favourability rather than ability. There is also a less consistent positive association between ICBM and need for cognition among students. This is reflected most by the ICBM22, ICBM17 and ICBMs2 (availability/representativeness). Among trainees significant correlations are more patchy, although again the direction of associations is in accord with theory. These data offer little evidence to support one version of the ICBM over another. Similarly there is no evidence available from the consultant data to either guide the choice of ICBM version or support the assumption that the ICBM taps a related construct to that operationalised by the REI.
Correlations between ICBM scores and the Rational Experiential Inventory.