|Home | About | Journals | Submit | Contact Us | Français|
In two experiments, we studied how people’s strategy choices emerge through an initial and then a more considered evaluation of available strategies. The experiments employed a computer-based paradigm where participants solved multiplication problems using mental and calculator solutions. In addition to recording responses and solution times, we gathered data on mouse cursor movements. Participants’ motor behavior was revealing; although people rapidly initiated movement to the calculator box or the answer input box, they frequently changed their minds and went to the other box. Movement initiation direction depended on problem difficulty and calculator responsiveness. Ultimate strategy selection also depended on these factors, but was further influenced by movement initiation direction. We conclude that strategy selection is iterative, as revealed by these differences between early cursor movement and eventual strategy implementation. After rapidly initiating movement favoring one strategy, people carefully evaluate the applicability of that strategy in the current context.
The question of how people select among problem solving strategies is central to psychological research. Although this question has received substantial attention, researchers have primarily studied situations in which strategies are selected by a single irrevocable action. In many situations, however, the physical act of executing one strategy provides an opportunity to consider the wisdom of that choice. While searching for a calculator, one can instead decide to compute the tip mentally; while commuting along a congested roadway, one can consider the speed of alternate routes; while composing a spiny response to an email, one can reflect on whether it is prudent to respond at all. Might the need to second-guess oneself, and the second guess itself, be adaptive? In this paper, we explore this question by considering subtleties of the mouse-based movements that people make while selecting and implementing strategies at a computer interface. We will show that as people implement a strategy, they continue to consider whether that strategy is truly preferable. We will show that both the rapid initial selection and the further consideration of a solution method are sensitive to the relative utility of available strategies.
Many formal models of human problem solving posit the existence of a strategy selection phase (Lovett & Anderson, 1996; Lovett & Schunn, 1999; Payne, Bettman, & Johnson, 1988; Payne, Johnson, Bettman, & Coupey, 1990; Schunn, Reder, Nhouyvanisvong, Richards, & Stroffolino, 1997; Siegler & Shipley, 1995). Upon reaching an impasse, the problem solver evaluates the applicability of each available strategy to the current problem. This evaluation is informed by history of strategy use; what has worked in the past is likely to work again (Lovett & Anderson, 1996; Lovett & Schunn, 1999; Siegler & Shipley, 1995). Based on the current context and past experiences, the solver attempts to identify the strategy that minimizes solution time and effort while maximizing accuracy. Such evaluations do not require conscious awareness (Cary & Reder, 2002). Following selection, a single strategy is executed and its outcome is integrated into the problem solver’s history.
Selections are generally sound. People adaptively apply strategies in a variety of tasks to improve performance. For example, while using mental arithmetic, people retrieve answers to problems that are easy and familiar while they compute answers to problems that are difficult and unfamiliar (Reder & Ritter, 1992). Similarly, when quizzed after reading a story, people shift from using retrieval to using inferences as the elapsed time since reading increases (Reder, 1987, 1988). People display adaptivity in many other domains including numerical estimation, currency conversion, reading, and noun pair learning (Aaronson & Ferres, 1986; Lemaire, Arnaud, & Lecacheur, 2004; Lemaire & Lecacheur, 2001; Touron & Hertzog, 2004). In each instance, selected strategies facilitate performance by increasing efficiency and accuracy.
In the preceding cases, solutions are reached via mental strategies. The case we consider involves selection between a mental strategy and a technologically facilitated strategy. Technologically facilitated strategies are more likely to incur perceptual-motor costs while mental strategies primarily tax memory. People seem to be sensitive to each of these demands. Evidence supporting this point comes from work by Gray and Fu (2004). In their experiments, participants programmed a virtual VCR at a computer interface. To program the VCR, participants first needed to retrieve trial-relevant data from an occluded or unoccluded window. Gray and Fu found that when the data window was occluded, participants accessed it less frequently and committed more items to memory per access. In contrast, when the window was unoccluded, participants accessed it more frequently and encoded fewer items per access. Such behavior is consistent with the emerging view of a distributed cognitive representational system (Clarke & Chalmers, 1998; Hutchins, 1995; Kirsh & Maglio, 1994; Zhang & Norman, 1994). People seem to be tuned to the costs and benefits of epistemic actions, actions intended to augment cognitive processes. For instance, in the video game Tetris, experts rotate shape “zoids” manually rather than mentally to discover where the zoid best fits (Kirsh & Maglio, 1994). The explanation given for this finding was that manual rotation was faster than mental rotation, allowing players greater opportunity to identify the ideal zoid placement. As this example illustrates, when the benefits outweigh the costs, people use epistemic actions to manage information in the environment rather than in the mind.
Further examples come from studies of problem solving at computer interfaces. When interface controls are costly to manipulate or when system response time is slow, people plan extended movement sequences before acting (Gray, Sims, Fu, & Schoelles, 2006; O’Hara & Payne, 1998; Svendsen, 1991). Conversely, when an interface is responsive and easy to operate, people act more and plan less. Presumably, these different behaviors reflect an adaptive approach to problem solving (Anderson, 1991; Simon, 1978); when the cost of generating further partial plans exceeds the cost of acting, action ensues. As such, people adopt more cognitively demanding strategies when the cost of acting is high.
While these studies confirm that people alter their behaviors based on characteristics of available technology, they do not fully address whether strategy selection at the interface is adaptive. To support such a claim, one could show that people perform as well or better when they are allowed to select between a mental and a technological strategy versus when they are required to use the mental or the technological strategy. Our studies differ from existing work in that we use the choice/no-choice method (Siegler & Lemaire, 1997). Using this method, we can make these very comparisons.
In addition to testing determinants of strategy selection, we sought to explore the process by which the performer selects and implements a strategy. Researchers interested in strategy choice frequently rely on performance outcomes such as the distribution of strategy use, error rates, and solution times (Lovett & Anderson, 1996; Reder, 1987; Reder & Ritter, 1992; Siegler & Shipley, 1995). Although these measures are informative, they force us to infer the underlying process based solely on its final outcomes. More recently, researchers have analyzed eye movements accompanying strategy selection in complex addition (Green, Lemaire, & Dufau, 2007). While patterns of steady fixation provide valuable clues, the time required to plan and execute saccades limits the number of fixations occurring in short-duration tasks (Matin, Shao, & Boff, 1993).
As such, in addition to recording responses and solution times, we gathered a more dynamic source of data; mouse cursor position. Because motion is modified online (Goodale, Pélisson, & Prablanc, 1986), cursor movement offers sensitive information about the processes accompanying selection and execution. The practice of inferring internal processes from overt motor behavior is not new. This technique has been applied to topics ranging from spoken-language processing (Spivey, Grosjean, & Knoblich, 2005) to algebraic representation (Alibali, Bassok, Solomon, Syc, & Goldin-Meadow, 1999). These applications have demonstrated the value of monitoring motor behaviors and confirmed that guided limb movements, much like saccades, provide veridical measures of attention and intention. Currently, we attempted to gain a deeper understanding of strategy use by studying the mouse cursor trajectories that accompany strategy selection and execution.
While humans are often characterized as apt strategists, certain questions remain minimally understood. First, do people accurately evaluate the relative utility of technological and mental strategies and do they select accordingly? Second, how do people interleave action with the selection of a strategy?
To address these questions, we devised a task in which participants rapidly solved multiplication problems presented at a computer. Participants solved problems mentally or with the use of a calculator built into the interface. Calculator responsiveness varied so that solutions were presented immediately in one condition and following a delay in the other condition. Problems of varying difficulty (according to problem types NN × 10, N × NN, & NN × NN) were presented during both conditions. Pay was granted for correct responses and decreased as time passed.
We used the choice/no-choice method in our experiments (Siegler & Lemaire, 1997). During choice trials, participants were allowed to use mental or calculator solutions. During no-choice trials, participants were told which strategy to use. Because performance during no-choice trials is not biased by strategy selections, performance estimates derived from no-choice trials can be used to assess the adaptivity of selections during choice trials (Siegler & Lemaire, 1997).
If people try to maximize the payoff function defined by solution time and accuracy, calculator responsiveness and problem difficulty should influence strategy selection. Because mental solutions for N × NN (e.g. 7 × 16) problems are moderately slow (Siegler & Lemaire, 1997), selections on these problems should be particularly sensitive to calculator delay – participants should show greater reliance on mental strategies in the delay condition. In addition, solution times and error rates should increase as participants use mental solutions on a harder superset of problems in the delay condition. Conversely, calculator usage should be rare on NN × 10 (e.g. 17 × 10) problems and frequent on NN × NN (e.g. 17 × 13) problems because mental solutions are especially accurate and quick for NN × 10 problems, while they are inaccurate and slow for NN × NN problems (Siegler & Lemaire, 1997). Lastly, if participants selectively apply mental solutions to the most suitable problems, mental solution times and error rates should be lower during choice than during no-choice trials. In sum, adaptive selection depends on at least three factors: problem difficulty, technological responsiveness, and sensitivity to personal ability.
We were also interested in participants’ immediate behavior following problem presentation. Immediate behavior can be considered in light of two additional questions. Both questions relate to early cursor motion, beginning at movement onset and commencing with a mouse click in the calculator box or in the answer box.
Strategy selection takes time. Even when pressed to decide as quickly as possible, people take about 700 ms to select a mental multiplication strategy and to indicate their selection by striking a key (Reder & Ritter, 1992). While no such estimates exist for our task, the duration of such a relatively slow decision process presumably exceeds the time required to initiate a guided limb movement, which occurs rapidly (Rosenbaum, 1980). As such, participants could conceivably initiate motion before selecting a final strategy.
Early cursor movements need not predict ultimate strategy use. Participants might initially move towards a neutral point that allows equally rapid access to both the calculator and the answer box. After starting towards this point, participants would select a strategy and smoothly converge to the associated box. Alternatively, participants could always initiate movement corresponding to the favored strategy. They would frequently complete these movements, but they would sometimes revise the motion and converge to the opposite box.
Twenty-one Carnegie Mellon students participated on a paid volunteer basis (13 males and 8 females, ages ranging from 18–35 with a mean age of 24 years). All participants were blind to the study’s aim and hypotheses.
Stimuli included 112 multiplication problems of three structural types (NN × 10, N × NN, & NN × NN). Because we expected the largest effect of calculator delay on problems of intermediate difficulty, we included 56 N × NN problems, 28 NN × 10 problems, and 28 NN × NN problems. The appendix contains the complete problem set.
Problem order was randomized during two calculator conditions (0 and 4 second delay), resulting in 224 experimental trials. Multiplicand order was randomized during the first condition and each problem’s commutative pair was presented during the second condition. Eleven participants completed the delay condition first and ten completed the no delay condition first.
Choice/no-choice trials were intermixed. During each calculator condition, 14 problems of each type were assigned to no-choice trials. Half required mental solutions and half required calculator solutions. Thus, within each calculator condition, 21 problems required mental solutions, 21 problems required calculator solutions, and 70 problems allowed choice1. Each participant received a randomly selected set of choice/no-choice problems.
Each block began with 10 unique practice problems. These problems allowed participants to adjust to the interface and to gauge calculator responsiveness, a necessary precondition to accurately evaluating its utility.
Throughout the experiment, participants interacted with a MATLAB graphical user interface (GUI) centered on a 17' Dell Ultrasharp monitor. To initiate each trial, participants used the computer mouse to click the button labeled Next that was centered on the screen. After clicking Next, participants saw a prompt that specified the upcoming trial. The prompt read “Mental”, “Calculator”, or “Choice” and was aligned above the Next button. If the prompt read “Mental” or “Calculator”, the participant was required to use that strategy. If the prompt read “Choice”, the participant could use either. When participants clicked Next a second time, the prompt disappeared and the problem appeared.
The calculator and answer box were vertically aligned, with the calculator box above and to the left of the Next button (Figure 1). The calculator and answer box had identical dimensions and were equidistant from the Next button. Problems were presented in 20 point, Times New Roman font and were aligned above the answer box with a standard times symbol to the left.
To use the calculator, participants moved the computer mouse to and clicked in the calculator box, typed the multiplicands separated by the multiplication symbol “*”, and hit enter. The product then appeared in the calculator box. During the delay condition, a dotted line appeared in the calculator box while the product was computed. Participants then clicked the answer box, input the solution, and pressed enter. To solve a problem mentally, participants simply input their response in the answer box and pressed enter. The interface automatically deleted entries that violated the trial condition, forcing participants to adhere to prompts. Participants were instructed to use their right hand while moving the mouse and using the number pad.
After responding, participants saw whether they were correct and if so, how much they earned for the problem. This feedback was horizontally centered above the Next button. Cumulative earnings were displayed on the bottom right corner of the screen.
Pay was determined according to the equation:
where Δtime corresponded to the elapsed time since problem presentation. As solution time approached 0 or 14 seconds, pay approached 100% or 0%. Responses after 14 seconds received 0% pay. Maximum pay for each problem was 10 cents2.
We will first describe participants’ selections and performance, and then report findings related to mouse cursor movements.
As shown in Figure 2 (top panel), participants used mental solutions less frequently as problem difficulty increased, F(2,40) = 157.15, p < .0001, and more frequently when the calculator was delayed F(1,20) = 32.89, p < .0001. The interaction between problem difficulty and calculator condition was also significant, F(2,40) = 19.99, p < .0001. As expected, the difference between the delay and no delay condition was greatest for the intermediate N × NN problems, t(20) = 5.41, p < .0001.
Figure 2 also shows the effects of problem type and calculator condition on solution times during choice and no-choice trials (bottom panels). Some participants failed to contribute to all cells during choice trials. The means and standard errors displayed pertain to all participants who contributed to that cell. We evaluated mental solution times during choice trials for the 20 participants who contributed fully to a 2 (problem type: NN × 10, N × NN) X 2 (calculator condition) layout, and then for the subset of 9 participants who contributed fully to a 3 (problem type: NN × 10, N × NN, NN × NN) X 2 (calculator condition) layout.
The first ANOVA revealed significant effects of problem type, F(1,19) = 119.25, p < .0001, and calculator condition, F(1,19) = 5.86, p = .026, as well as a significant interaction, F(1,19) = 9.17, p = .007. Mental solutions on N × NN problems were slower when the calculator was delayed, t(19) = 2.82, p = .011, presumably because participants decided to solve more difficult problems mentally during that condition. The second ANOVA confirmed effects of problem type, F(2,16) = 80.47, p < .0001, and calculator condition, F(1,8) = 8.12, p = .022, as well as a significant interaction, F(2,16) = 5.85, p = .012. Mental solutions on NN × NN problems were slower when the calculator was delayed, t(8) = 2.95, p = .018. Corresponding error rates in each group depended on problem type (F(1,19) = 78.24, p < .0001; F(2,16) = 12.06, p = .0006), increasing with problem difficulty (NN × 10 =.004 ± 0.002; N × NN = .155 ± .015; NN × NN = .273 ± .072). Calculator error rates, in turn, were consistently low (aggregated error rate = .02 ± .007). Neither the effect of calculator condition nor the interaction between calculator condition and problem type affected error rates3.
Mental solution times appeared to be faster during choice than during no-choice trials for N × NN and NN × NN problems, presumably because participants opted to solve easier problems mentally during choice trials. We confirmed this with additional 2 (calculator condition) X 2 (choice/no-choice) ANOVAs for N × NN and NN × NN problems. Both showed a significant effect of choice on solution time (F(1,19) = 29.27, p < .0001; F(1,8) = 6.84, p = .031). Error rates were not affected by the choice status of trials.
One additional point warrants explanation. When forced to use the delayed calculator, participants’ solution times were especially quick on NN × 10 problems. Astute participants realized that they could solve NN × 10 problems mentally, move to the answer box, and type the solution while waiting for the delay to end. When the calculator responded, they could quickly strike enter thereby reducing solution times.
Mouse cursor position was sampled at ~90 Hz yielding an average of 66 points between movement onset (defined by the earliest cursor motion following problem presentation) and operator selection (defined by the first mouse click in the calculator or answer box). We used a shape preserving cubic spline interpolation to calculate x and y cursor coordinates at 100 time-normalized points between movement onset and operator selection for each trial.
Participants’ motor behavior was revealing. On average, participants initiated motion within 220 ms. Following movement initiation, participants frequently redirected motions mid-execution. Figure 3 shows two representative trajectories as they approach the calculator box. One trajectory clearly begins towards the answer box while the other approaches the calculator directly. This distinction between direct and indirect motions is also apparent when we consider trajectories collectively. Figure 4 displays how frequently participants’ mouse trajectories passed through different regions of the screen. The region of raised relief connecting the calculator and answer box results from movements initiated towards the answer box but redirected to the calculator, and from movements initiated towards the calculator but redirected to the answer box.
To quantify the difference between direct and indirect motions, we calculated an index of curvature for the portion of movement preceding operator selection. The index was defined as the maximum deviation from the straight line connecting the movement start point and the movement end point divided by the length of that line (with positive values assigned to portions of curvature towards the non-selected box and negative values assigned to portions away from the non-selected box)4. Figure 5 shows the frequencies of resulting curvature values for calculator solutions (left panel) and mental solutions (right panel). The distributions in both panels are bi-peaked, with values near zero reflecting direct movements and values near one reflecting indirect movements curved towards the non-selected box. These histograms show that participants frequently began towards the answer box before using calculator solutions, while they rarely began towards the calculator box before using mental solutions. This is especially apparent in the delayed calculator condition.
Another way to understand these data is to distinguish between the initial movement and the strategy ultimately selected. Using a K-means cluster analysis (Seber, 1984), we partitioned the aggregated values of movement curvature into two groups. Observations in the cluster with smaller curvature, centered at .045, were considered direct movements. Observations in the cluster with larger curvature, centered at .801, were considered indirect, late aborts. To ensure that the assumption of bimodality was valid, we compared how well a unimodal normal distribution could fit the curvature distribution to how well a bimodal distribution formed by a pair of overlapping normal distributions could fit the curvature distribution. The ratio of the Bayesian Information Criterion (BIC) for the unimodal solution to the BIC of the bimodal solution (4.2) was substantially greater than 1, indicating that the bimodal solution was preferable despite its added complexity. Additionally, the BIC ratio for every participant substantially exceeded 1, confirming that all participants exhibited a blend of direct movements and indirect late aborts.
The initial movement direction and the ultimate selection coincide for direct movements but differ for late aborts. Figure 6A shows the direction of the initial movements, independent of eventual solution method. As seen, participants initiated fewer movements towards the answer box on harder problems, F(2,40) = 14.18, p < .0001, and they initiated more movements towards the answer box when the calculator was delayed, F(1,20) = 27.40, p < .0001. The interaction between problem difficulty and calculator condition was not significant (p > .5). It is noteworthy that participants were able to consider these factors in the first 220 ms before movement initiation on at least some trials.
This rapid initial action affected eventual strategy selection (Figure 6B). On NN × 10 problems, there were no effects of delay or initial decision – participants almost always used mental solutions. However, there was a substantial interaction between the other two problem types (N × NN and NN × NN) and the type of movement initiated, as confirmed by analysis of the 20 participants who contributed to all cells, F(1,19) = 21.18, p = .0002. Participants tended to use mental solutions after starting towards the answer box for N × NN problems, t(19) = 4.88, p = .0001, while they showed a non-significant effect in the opposite direction for NN × NN problems, t(19) = −.39, p > .7.
There were costs associated with revising selections mid-execution. These costs are clear at the level of operator selection times. Sensibly, participants took longer to click in the calculator or answer box if they had begun in the opposite direction (Table 1). The resulting difference in the time to commit to a strategy averaged approximately a second.
Finally, behavior in the previous trial strongly influenced behavior in the current trial. Table 2 shows current state probabilities (columns) conditioned on prior state (rows). The first letter in each classification describes the direction of movement initiation (movement to answer box denoted by M; movement to calculator denoted by C), and the second letter describes the solution used (mental solutions denoted by M, calculator solutions denoted by C). Both aspects of prior behavior affected current behavior. An ANOVA on the percent of current movements initiated to the answer box revealed large effects of prior initial direction (77% for mental vs. 58% for calculator, F(1,16) = 20.18, p = .0004) and prior solution (80% for mental vs. 54% for calculator, F(1,16) = 56.59, p < .0001). Similarly, an ANOVA on the percent of current problems solved mentally revealed large effects of prior initial direction (63% for mental vs. 48% for calculator, F(1,16) = 10.16, p = .006) and prior solution (63% for mental vs. 48% for calculator, F(1,16) = 17.11, p = .0008).
Prior behaviors seemed to indirectly influence current solutions through their effect on current initial movements. The percent of current problems solved mentally, conditioned on initiating the current movement towards the answer box, depended minimally on prior initial direction (74% for mental vs. 65% for calculator, t(19) = 1.49, p = .15) and prior solution (76% for mental vs. 72% for calculator, t(20) = 1.43, p = .17). Similarly, the percent of current problems solved mentally, conditioned on initiating the current movement towards the calculator box, depended minimally on prior initial direction (40% for mental vs. 27% for calculator, t(15) = 2.08, p = .055) and prior solution (43% for mental vs. 36% for calculator, t(18) = 1.64, p = .12).
In contrast, much larger effects remained for the inverse conditional probabilities – probability of a current initial direction conditioned on the current solution. The percent of movements initiated to the answer box, conditioned on solving the current problem mentally, depended on prior initial direction (90% for mental vs. 73% for calculator, t(19) = 2.95, p = .008) and prior solution (93% for mental vs. 77% for calculator, t(20) = 5.09, p < .0001). Similarly, the percent of current movements initiated to the answer box, conditioned on solving the current problem with the calculator, depended on prior initial direction (70% for mental vs. 44% for calculator, t(18) = 4.77, p = .0001) and prior solution (73% for mental vs. 42% for calculator, t(17) = 7.15, p < .0001). Thus, behavior during the previous trial primarily affected the direction of movement initiation during the current trial, which in turn influenced the solution applied.
To confirm that these contingencies were not driven by the behavior of individual participants (e.g. participants who frequently used calculator solutions also frequently initiated movement to the calculator) we replicated the prior set of analyses using nonparametric binomial tests. The motivation behind these analyses was to determine whether the directions of contingencies were consistent across participants. In all but one case, the resulting p-values were less than .02, indicating that contingency effects were consistent across participants. In the one discrepant case, the effect of prior initial direction on the percent of movements initiated to the answer box, conditioned on solving the current problem mentally, was marginally significant (p = .11).
Strategy selections depended on the interaction between problem difficulty and calculator responsiveness. Mental solutions were generally superior for NN × 10 problems and calculator solutions were generally superior for NN × NN problems, but the optimal strategy for N × NN problems depended on calculator responsiveness. Accordingly, participants predominantly used mental solutions on NN × 10 problems, they rarely used mental solutions on NN × NN problems, and they used far more mental solutions on N × NN problems when the calculator was delayed.
The pattern of solution times further suggests that participants adaptively selected between strategies. Mental solutions during choice trials tended to be faster than mental solutions during no-choice trials, as would be expected if participants only attempted mental solutions on the most suitable problems. Mental solution times also increased when the calculator was delayed, indicating that people applied mental solutions to a more difficult superset of problems during that condition.
Results from Experiment 1 support an iterative decision process. Participants quickly adopted an initial direction based on problem characteristics, calculator responsiveness, and behavior during the previous trial. They continued to consider the choice before ultimately committing to a strategy on the basis of problem characteristics, calculator responsiveness, and the direction of movement initiation.
Although the results of Experiment 1 seem clear, we have two concerns. First, participants began towards the answer box on the majority of trials (74%). We suspect that this occurred because mental solutions were ultimately chosen for the majority of problems (64%). As such, participants initiated the movement that was most likely to correspond with the strategy ultimately used. However, aspects of the interface design or task format may have primarily motivated participants’ initial motions. Problems always appeared above the answer box and every trial ended with the submission of a response in the answer box. Participants may have initiated a disproportionate number of movements towards the answer box for either of these reasons.
To test these accounts, we altered the distribution of problem difficulty by removing NN × 10 problems and doubling the number of NN × NN problems. This manipulation was intended to cause participants to predominantly use the calculator during Experiment 2. If movement initiation depends on a rapid problem judgment, participants will now initiate more movements towards the calculator, which is the favored strategy. Movements to the calculator will be direct while movements to the answer box will sometimes be direct but will sometimes begin towards the calculator.
Our second concern relates to the ambiguity of selections following movement initiation on NN × NN problems. While calculator responsiveness, problem difficulty, and movement initiation direction clearly influenced strategy use on N × NN problems, the effect of direction of movement initiation on NN × NN problems was unclear. Because the prior study included few NN × NN problems, this likely relates to the small number of observations. By increasing the number of NN × NN problems in the current study, we expected to attain a definitive result.
Twenty Carnegie Mellon students participated on a paid volunteer basis (9 males and 11 females, ages ranging from 18–27 with a mean age of 23 years). All participants were blind to the study’s aim and hypotheses, and none had participated in Experiment 1.
We replaced NN × 10 problems with 28 new NN × NN problems (Appendix). During each calculator condition, 14 problems of each type were randomly assigned to no-choice trials. Half required mental solutions and half required calculator solutions. Thus, within each calculator condition, 14 problems required mental solutions, 14 problems required calculator solutions, and 84 problems allowed choice. Each participant received a randomly selected set of choice/no-choice problems. The procedure was identical to that used in Experiment 1.
As shown in Figure 7 (top panel), participants used mental solutions less frequently as problem difficulty increased, F(1,19) = 45.09, p < .0001, and more frequently when the calculator was delayed, F(1,19) = 65.44, p < .0001. The interaction between problem difficulty and calculator delay was not significant (p > .2). In the delay versus the no delay condition, participants used mental solutions on a higher proportion of N × NN problems, t(19) = 5.13, p < .0001, and NN × NN problems, t(19) = 2.66, p = .015.
Figure 7 also shows the effects of problem type and calculator delay on solution times during choice and no-choice trials (bottom panels). Some participants failed to contribute to all cells during choice trials. The means and standard errors displayed pertain to participants who contributed to that cell. On choice trials, participants’ mental solutions were slower when the calculator was delayed, although this effect was not significant (NN × NN: t(5) = .17, p > .8; N × NN: t(14) = 1.64, p > .1). After aggregating across calculator conditions, we compared mental solution times during choice and no-choice trials. Mental solutions were faster during choice trials for NN × NN problems, t(14) = 2.54, p = .024, but not for N × NN problems, t(19) = 1.25, p > .2. Error rates increased with problem difficulty (N × NN = .166 ± .031; NN × NN = .381 ± .095), but did not depend on calculator condition or choice status of trials. During choice and no-choice trials, calculator error rates remained uniformly low (aggregated error rate = .021 ± .005).
On average, participants initiated motion within 208 ms. As seen in Figure 8, initial motions were clearly directed towards the calculator or answer box. Many of these initial motions were then redirected, as revealed by the region of raised relief connecting the calculator and answer box.
Movement curvature was calculated as in Experiment 1. Figure 9 shows frequencies of movement curvature values for calculator solutions (left panel) and mental solutions (right panel). Participants were equally likely to start towards the calculator or the answer box before using the delayed calculator. Conversely, participants were much less likely to start towards the answer box before using the responsive calculator. In both conditions, participants frequently started towards the calculator before using mental solutions.
Using a K-means cluster analysis, we partitioned the aggregated values of movement curvature into two groups. Mean curvature of direct movements was .064, and mean curvature of indirect, late aborts was .837. To ensure the validity of the bimodality assumption, we compared the BIC for a single normal distribution to the BIC for a pair of overlapping normal distributions. The BIC ratio (3.7) was substantially greater than 1 across participants, and the ratio substantially exceeded 1 for each participant, confirming the observation that participants exhibited a blend of direct movements and indirect late aborts. Based on the results of the K-means analysis, we identified initial movement selections during each trial (Figure 10A). As predicted, participants began towards the answer box less frequently than in Experiment 1. However, as in Experiment 1, participants initiated fewer movements towards the answer box as problem difficulty increased, F(1,19) = 22.25, p < .0001, and they initiated more movements towards the answer box when the calculator was delayed, F(1,19) = 34.13, p < .0001. The interaction between problem difficulty and calculator condition was not significant (p > .6).
These rapid initial actions influenced participants’ eventual selections (Figure 10B). Participants were more likely to use a mental solution if they started towards the answer box, both for N × NN problems, t(19) = 4.23, p = .0004, and for NN × NN problems, t(19) = 2.33, p = .03. A 2 (problem type) X 2 (initial direction) ANOVA revealed a non-significant interaction (p > .1).
Movement redirection came at a cost. This cost is apparent in operator selection times. Within-subject comparisons confirmed that participants took longer to click in the calculator or answer box if they had begun in the opposite direction (Table 3). As in Experiment 1, the average cost of a late abort was approximately a second.
Prior behaviors influenced current behaviors. Table 4 shows current state probabilities (columns) conditioned on prior state (rows), with states again defined by the direction of movement initiation and the solution ultimately applied. An ANOVA on the percent of current movements initiated towards the answer box showed large effects of prior initial direction (69% for mental vs. 44% for calculator, F(1,16) = 56.49, p < .0001) and prior final solution (68% for mental vs. 44% for calculator, F(1,16) = 24.49, p = .0001). An ANOVA on the percent of current problems solved mentally showed weaker effects of prior initial direction (49% for mental vs. 39% for calculator, F(1,16) = 3.15, p = .1) and prior final solution (48% for mental vs. 40% for calculator, F(1,16) = 3.42, p = .08).
We again tested the causal relationship between prior behavior and current solutions. The percent of current problems solved mentally, conditioned on initiating the current movement towards the answer box, was independent of prior initial direction (60% for mental vs. 58% for calculator, t <1) and prior solution (59% for mental vs. 57% for calculator, t <1). The percent of current problems solved mentally, conditioned on initiating the current movement towards the calculator box, was again independent of prior initial direction (24% for mental vs. 24% for calculator, t <1) and prior solution (27% for mental vs. 24% for calculator, t <1).
Conversely, large effects remained for the inverse conditional probabilities – probability of a current direction conditioned on the current solution. The percent of current movements initiated to the answer box, conditioned on solving the current problem mentally, depended on prior initial direction (83% for mental vs. 63% for calculator, t(17) = 4.21, p = .0005) and prior solution (80% for mental vs. 62% for calculator, t(18) = 4.01, p = .0008). The percent of current movements initiated to the answer box, conditioned on solving the current problem with the calculator, depended on prior initial direction (49% for mental vs. 21% for calculator, t(18) = 5.71, p < .0001) and prior solution (55% for mental vs. 26% for calculator, t(18) = 4.95, p = .0001). We repeated each of these comparisons using binomial tests to ensure that contingencies were not driven by individual participants. As expected, each contingency effect was consistent across participants (all ps < .005). Thus, as in Experiment 1, previous behaviors affected current direction of movement initiation, which influenced the ultimate selection of a strategy.
These results are consistent with findings from the first experiment. Additionally, these results address the earlier two concerns. First, as participants used fewer mental solutions (41%), they initiated fewer movements to the answer box (45%). Although aspects of the interface and task format may induce a slight bias, problem type, calculator responsiveness, and prior behavior largely accounted for initial movement direction. Second, a consistent set of factors determined ultimate selections across problem types. On both N × NN and NN × NN problems, strategy selection depended on problem type, calculator responsiveness, and direction of movement initiation.
Participants’ behavior suggests an iterative selection process. Participants quickly decide whether a mental or calculator solution is preferable and they initiate the corresponding mouse motion. While moving, participants then conduct a more thorough analysis and decide whether to continue with the initially favored strategy or to revise the selection. Here, we investigate the rationality of two outcomes of this selection process, namely the adopted movement initiation direction and the strategy ultimately applied.
These analyses are driven by two assumptions. First, rather than trying to optimize solution time or accuracy alone, participants attempt to optimize pay, which depends on both. This is plausible because the feedback given to participants pertained to pay. To understand selections, we should then consider strategy profitability. Second, different selection patterns between experiments should depend on the distribution of problem difficulty and the applicability of each strategy given that level of difficulty.
To derive unbiased estimates of strategy profitability, we looked to no-choice trials. For each combination of problem type and calculator condition, participants were required to solve 7 problems mentally and to solve 7 problems with the calculator. We computed the average proportion of times that each mental solution paid as much or more than the 7 corresponding calculator solutions (Table 5, top three rows). These proportions are local because they relate to specific problem types within each calculator condition. The fourth row of Table 5 shows proportions averaged across problem type and within calculator condition. These proportions are global because they relate to all problem types within each calculator condition.
If participants responded optimally, they would always start towards a mental solution when it was more profitable than a calculator solution (values greater than .5 in Table 5). Likewise, participants would always start towards a calculator solution when it was more profitable than a mental solution (values less than .5 in Table 5). However, participants were probably not perfectly tuned to these local probabilities. Additionally, participants may have sometimes initiated movement before carefully considering problem type. In these instances, participants would give greater weight to the probability global to the calculator condition.
To evaluate the importance of local and global probabilities, we performed a multiple regression analysis. We sought the best fitting combination of the global and local values to predict the observed probability of initiating movement towards the answer box during both experiments (observed probabilities contained in Figures 6A and 10A). The resulting regression equation was
This equation accounted for 97.9% of the variance. Both variables contributed significantly (local weight: t(7) = 7.28; and global weight: t(7) = 13.19). Global weight was significantly greater than local weight t(7) = 9.86, indicating that participants were more sensitive to the global context. In addition, the intercept was significantly greater than .5, t(7) = 13.02, indicating that participants tended to start in the direction of the answer box even when the global and local statistics did not support a mental solution.
In summary, participants were sensitive to problem type and calculator condition at the time of movement initiation. However, because they acted so quickly, participants gave relatively little weight to local features of the problem. Instead, they mainly responded to the global statistics of the current condition. Additionally, participants exhibited an optimistic bias: they seemed to overestimate the relative utility of mental solutions.
Strategy selection depended on similar considerations of strategy profitability by problem type and calculator condition. Selections additionally depended on the direction of movement initiation. If one had already moved to the answer box, the critical comparison was between a direct mental solution and a late abort calculator solution. If one had already moved to the calculator box, the critical comparison was between a direct calculator solution and a late abort mental solution.
To conduct these comparisons, we computed the increase in solution time brought about by late aborts during choice trials (based on average differences within Tables 1 and and3).3). This value (.913 seconds) represented the temporal penalty associated with deviating from the initial movement direction. By adding this penalty to solution times from no-choice trials, we could calculate expected pay following a late abort. To calculate the probability that a mental solution was preferable after first moving to the answer box, we computed the probability of a mental solution paying as much or more than a time-penalized calculator solution. Similarly, to calculate the probability that a mental solution was preferable after first moving to the calculator box, we computed the probability of a time-penalized mental solution paying as much or more than a calculator solution.
This gave us three variables for predicting the probability of using a mental solution – the global and local statistics used in predicting initial direction and the fine-grain statistic that accounted for the direction of movement initiation. In a multiple regression analysis, neither the global nor the local information was predictive of selections. The only variable that proved relevant was the fine-grain statistic. The best fitting equation was
This equation accounted for 97.2% of the variation in the 20 data points from the two experiments (observed probabilities in Figures 6B and 10B). The contribution of the fine-grain statistic was highly significant t(18) = 25.17. The resulting equation describes a probability-matching situation (Herrnstein, 1961), where participants chose a solution strategy as frequently as it proved to be the better choice. Such a situation would be characterized by an equation with an intercept of .5 and a slope of 1. The actual intercept was slightly but significantly greater than .5, t(18) = 2.80, but the slope was not significantly different from 1, t(18) = 0.78.
Figure 11 shows a scatter plot of the observed percent of mental solutions applied against the probability of a mental solution paying as much or more than a calculator solution. A set of points along the main diagonal would indicate perfect probability matching. As seen, the observed points deviate minimally from the main diagonal. Corresponding to the significant intercept, the percent of mental solutions applied averaged slightly more (.036) than the probability of a mental solution paying as much or more than a calculator solution. This difference, although small, was significant across the 20 points, t(19) = 2.83.
The ability of these simple linear equations to account for participants’ selections supports an iterative selection process and indicates that as participants approached a decision, their behaviors depend on the relative profitability of mental and calculator solutions. Adoption of a movement initiation direction, which occurred rapidly, reflected imperfect sensitivity to the current problem. Commitment to a specific strategy, which occurred later, reflected nearly perfect sensitivity to the relative profitability of mental and calculator solutions.
Given that strategy selections were more finely tuned to local statistics than were directional selections, one might wonder why participants did not wait longer to initiate motion. The answer is simple. When participants correctly anticipated how they would solve a problem, they saved about a second by moving and evaluating the problem concurrently. When participants incorrectly anticipated how they would solve a problem, they were no worse off than had they waited, in which case they would have still needed to program a movement to the desired region.
The applicability of the same linear equations to both experiments is not trivial. Participants’ behavior, particularly while selecting a movement initiation direction, differed substantially between experiments. The equations’ generalizability shows that the primary differences between experiments were quantitative rather than qualitative. Specifically, the global probability of a mental solution paying as much or more than a calculator solution decreased from the first to the second experiment. As a result, participants initiated fewer motions towards the answer box during the second experiment.
Thus far, we have focused on the outcomes of the underlying decision process. How might we characterize the process itself? One simple account is that people quickly select a strategy and only deviate when that strategy fails to produce a solution. In this case, a single selection occurs. Aborted movements do not require further deliberation, but instead reflect default use of the remaining solution method after the initially favored method failed. We have two arguments against this account. First, because the correct solution could always be found with the calculator, participants should have never redirected movements from the calculator to the answer box. Second, movement aborts occurred around 1–3 seconds. The time required to solve N × NN and NN × NN problems mentally far exceeds this time (Siegler & Lemaire, 1997). Therefore, late aborts could not have required completion of mental solutions, as suggested by this account.
While features of the data seem to indicate that participants were engaging in two discrete selections, an initial selection based on a quick evaluation and a later selection based on a more deliberate evaluation, this is not the only possible account of the data. A feasible alternative is that within-trial preferences emerged from an ongoing deliberation process that drew on gradually accumulating information, as envisioned for instance in Busemeyer & Townsend’s (1993) dynamical decision field theory. The initial direction of movement might be based on an early read of the accumulated evidence and a decision to redirect to the other solution based on a later read of the accumulated evidence. Diederich (2003) provides evidence for such a basis for preference reversals. Whether participants engaged in a discrete or a continuous decision process, however, these analyses quite clearly show that participants continued to deliberate as they acted, and that they exhibited progressively greater sensitivity as they approached a final decision.
Adaptive strategy selection is a pervasive characteristic of human performance. People adaptively select between solution methods in diverse tasks and across the life span (Siegler, Adolph, & Lemaire, 1996). The reported work extends these findings to situations that require selection between mental and technologically facilitated strategies. In our experiments, people accurately represented the utility of mental and technologically facilitated strategies, and they selected accordingly. This conclusion follows from two observations. First, factors that influenced relative strategy utility, as assessed during no-choice trials, also influenced selections during choice trials. Second, mental solutions were faster during choice than during no-choice trials, as would be expected if participants adaptively applied mental solutions to the most suitable problems.
These findings are consistent with, but distinct from, existing work on distributed cognition (Clarke & Chalmers, 1998; Hutchins, 1995; Kirsh & Maglio, 1994; Zhang & Norman, 1994). Although previous studies have shown that people rely on internal and external representations of information, the current studies more directly show that people accurately represent the utility of internal and external solution methods. When choice is afforded, people seamlessly switch between methods to generate the required information.
While we have stressed the benefits of choice, selections do incur costs. For instance, calculator solution times for NN × NN problems were about 450 ms slower during choice than during no-choice trials (Experiment 1: F(1,20) = 8.57, p = .008; Experiment 2: F(1,19) = 35.59, p <.0001)5. The interaction between choice and calculator condition was also significant (Experiment 1: F(1,20) = 7.67, p = .01; Experiment 2: F(1,19) = 8.61, p = .008), such that the temporal cost of selecting a calculator solution was greater in the delay condition. This interaction likely reflects the fact that the optimal choice in the delay condition was less clear, prompting longer deliberation during choice trials. Additionally, mental solutions were favored during the delay condition, resulting in fewer direct movements to the calculator during choice trials. When strategies are assigned, the time required to select a strategy and the time spent modifying selections is eliminated. Despite these saving, however, the benefit of being allowed to choose the more suitable strategy far outweighed the costs in these experiments.
Participants’ motor behavior was revealing. Although people rapidly initiated movement towards the calculator or answer box, they frequently redirected their initial motions. Thus, some trajectories were direct while others first approached the non-selected box. We interpret this as evidence for an iterative decision process. Participants quickly initiated movement corresponding to an initially favored strategy, and they then decided whether to complete the problem using that strategy. By conducting a more thorough evaluation while moving, participants could reduce trial completion times.
As these studies show, the implementation of a single solution method can follow multiple subtler behaviors. Here, these behaviors include the initial adoption of a movement direction and the ultimate commitment to a solution strategy. We can then consider the sensitivity exhibited in each behavior. As seen, the proportion of movements initiated towards the answer box exceeded the probability of a mental solution paying as much or more than a calculator solution. It seems that participants initially overestimated their mathematical prowess or underestimated the value of using a calculator, as also found by Siegler and Lemaire (1997).
Although participants initiated a disproportionate number of movements towards the answer box, the likelihood of ultimately using a mental solution was remarkably similar to the probability of a mental solution paying as much or more than a calculator solution under those circumstances. In this sense, participants’ ultimate selections were decidedly adaptive. Highly refined ultimate selections are consistent with the idea that participants carefully considered the problem after initiating movement. More generally, differences between initial directional selections and ultimate strategy selections are consistent with the idea that participants evaluated an increasingly broad set of option attributes as they moved towards a solution (Diederich, 2003).
One consequence of this evaluation process is that the initial adoption of a movement direction influences the final selection of a solution method. Participants were significantly more likely to use a strategy if they had begun towards the associated box, which makes sense because the cost of accessing a box decreases as one approaches it. This sort of self-fulfilling prophecy is not unique to our task. For instance, when navigating an online database, one’s early hyperlink selections influence the cost of accessing the desired information via different paths. As one pursues a specific path, it becomes increasingly sensible to follow that path to completion rather than to backtrack to the initial decision point. This illustrates that a poor initial decision can veil an otherwise rational selection process. However, as we have shown, that initial decision is more often right than wrong. Moreover, the practice of making a rapid initial decision is adaptive, even if the decision is sometimes wrong, because of the afforded motor savings.
The fact that participants initiated predictive motions within about 200 ms is surprising. Had they actually begun to consider the problem by that time? We suspect so. Trajectories were aligned with either the calculator or answer box as early as 100 ms following movement initiation. Because motion during this brief initial-impulse phase is ballistic, and hence not subject to correction (Meyer et al., 1988), it seems that participants were sensitive to problem type and calculator condition at movement onset. This timescale is substantially shorter than other reports of rapid strategy selection. Reder and colleagues showed that people take 600–700 ms to decide whether to retrieve or compute answers during mental arithmetic (Reder & Ritter, 1992; Schunn et al., 1997). Our task differs from the paradigm used by Reder and colleagues in an important way, however. In their task, participants were required to use the selected strategy. In our task, participants could alter initial selections as they moved. It makes sense that strategy selection times would increase with required commitment, particularly because people perceive irreversible decisions as being especially demanding (Beach & Mitchell, 1978; McAllister, Mitchell, & Beach, 1979). How would people act if the cost of redirecting initial movements varied incrementally? As redirection costs increased, participants would likely exhibit graded increases in movement initiation time and decreases in late abort frequency. This prediction can, of course, be tested in future studies.
Thus far, we have discussed the influence of initial movements on strategy use within trials. Motor behaviors also influenced strategy use between trials. As previously found, participants were more likely to use a strategy on the current trial if they had used the strategy during the preceding trial (Lovett & Anderson 1996; Lovett & Schunn, 1999). Surprisingly, participants were also more likely to use a strategy on the current trial if they had initiated movement corresponding to that strategy during the previous trial. Why would past movements have such a pervasive effect? We can think of two reasons. First, by initiating consistent motions between trials, participants could rely on memory of past target positions and spatio-temporal motor plans to reduce current movement planning costs (Jax & Rosenbaum, 2007; Marteniuk & Roy, 1972; Van der wel, Fleckenstein, Jax, & Rosenbaum, 2007). Second, the selection of an initiation direction was based on relatively stable estimates of strategy utility. As a result, the correlation between initial directional selections across neighboring trials could relate to the correlation between perceived strategy utilities across neighboring trials.
These experiments show that people can adaptively select between strategies, both while rapidly initiating action and then while carefully considering the decision as they act. The current experiments may under-represent people’s actual ability to adaptively change their minds. In our experiments, participants were required to use a strategy once they clicked in the associated box. In actuality, such points of finality are rare and they are reached more gradually, allowing people more opportunity to alter initial choices as more information becomes available. While relatively brief decision episodes characterized our task, real-life decisions may emerge across multiple stages with more information available at each.
Two aspects of these experiments’ results caution for a tempered view of the human as an adaptive decision maker, however. First, although participants’ ultimate selections were highly tuned, participants were probability matching rather than optimizing. One could argue that they should have always chosen the more successful strategy. However, this line of reasoning ignores the fact that participants do not possess perfect knowledge of the relative success of each strategy, and that participants acquire more precise estimates by sampling both strategies. Also, the relative values of the two strategies may evolve over the course of the experiment, making it worthwhile to resample. Probability matching effectively balances the needs to exploit the best option and to explore the space of available options (Hardy-Vallée, 2007).
Second, the proportion of movements initiated towards the answer box exceeded the probability of a mental solution paying as much or more than a calculator solution. A similar, though weaker, bias for mental solutions was apparent in final selections. Perhaps our measure of strategy profitability was too restrictive. By frequently using mental solutions, participants may have increased the efficiency of the mental strategy through practice. Repeated mental solutions would also allow participants to determine whether that strategy was indeed becoming more effective with practice.
Collectively, these studies highlight the adaptivity of the human performer. The sensibility exhibited in ultimate strategy use follows a series of subtler choices. These subtler choices include the rapid initial selection of a movement direction, and the further consideration of whether it is best to proceed in that manner or to change one’s mind.
|NN × 10||N × NN||NN × NN|
|10 × 13||3 × 17||6 × 39||12 × 16||12 × 23|
|10 × 14||3 × 26||6 × 41||12 × 17||12 × 26|
|10 × 15||3 × 29||6 × 47||12 × 21||12 × 29|
|10 × 16||3 × 34||6 × 48||12 × 28||12 × 31|
|10 × 17||3 × 37||7 × 13||13 × 14||13 × 22|
|10 × 18||3 × 42||7 × 16||13 × 19||13 × 24|
|10 × 21||3 × 46||7 × 19||13 × 28||13 × 27|
|10 × 23||3 × 49||7 × 24||13 × 29||13 × 32|
|10 × 25||4 × 18||7 × 27||14 × 16||14 × 26|
|10 × 27||4 × 23||7 × 31||14 × 18||14 × 29|
|10 × 29||4 × 27||7 × 35||14 × 19||14 × 31|
|10 × 31||4 × 31||7 × 48||14 × 23||14 × 32|
|10 × 32||4 × 36||8 × 14||16 × 18||16 × 13|
|10 × 34||4 × 38||8 × 17||16 × 22||16 × 21|
|10 × 35||4 × 43||8 × 21||16 × 27||16 × 23|
|10 × 38||4 × 46||8 × 28||16 × 29||16 × 28|
|10 × 39||5 × 19||8 × 35||17 × 14||17 × 13|
|10 × 42||5 × 23||8 × 38||17 × 21||17 × 16|
|10 × 45||5 × 26||8 × 41||17 × 24||17 × 18|
|10 × 46||5 × 29||8 × 43||17 × 26||17 × 28|
|10 × 49||5 × 34||9 × 13||18 × 13||18 × 12|
|10 × 51||5 × 37||9 × 16||18 × 22||18 × 19|
|10 × 52||5 × 39||9 × 18||18 × 24||18 × 21|
|10 × 53||5 × 49||9 × 24||18 × 27||18 × 23|
|10 × 54||6 × 14||9 × 32||19 × 12||19 × 16|
|10 × 56||6 × 21||9 × 36||19 × 23||19 × 17|
|10 × 58||6 × 28||9 × 42||19 × 26||19 × 22|
|10 × 60||6 × 32||9 × 47||19 × 27||19 × 24|
Experiment 1 used all NN × 10 and N × NN problems and the left column of NN × NN problems. Experiment 2 used all N × NN and NN × NN problems and no NN × 10 problems.
1Because we were mainly interested in behavior during choice trials, we included more choice than no-choice trials. The number of no-choice trials was selected to provide a sufficient number of observations to estimate the utility of each strategy for each problem type.
2Because of the low sinusoidal frequency, the payoff function behaves similarly to a linear function, which we also considered using. During piloting work however, the adopted function generated a greater range of payoffs by allowing participants to earn the near maximum amount on some problems.
3Trials with errors were included in all analyses based on the fact that the distributions of solution times were statistically indistinguishable between error trials and error-free trials. Inclusion of these trials ensured an adequate number of observations for comparisons. No trimming of solution times was used.
4We replicated all results using an alternate measure of movement curvature, area between the movement trajectory and the straight line connecting the movement start point and end point. Both methods are common in studies of human motor control (Spivey, Grosjean, & Knoblich, 2005; Jax & Rosenbaum, 2007).
5Corresponding comparisons for mental solutions are compromised because participants could choose the easiest problems in a category for mental solutions, whereas all problems in a category are equally difficult with a calculator.