|Home | About | Journals | Submit | Contact Us | Français|
How our central nervous system (CNS) learns and exploits relationships between force and motion is a fundamental issue in computational neuroscience. While several lines of evidence have suggested that the CNS predicts motion states and signals from motor commands for control and perception (forward dynamics), it remains controversial whether it also performs the ‘inverse’ computation, i.e. the estimation of force from motion (inverse dynamics). Here, we show that the resistive sensation we experience while moving a delayed cursor, perceived purely from the change in visual motion, provides evidence of the inverse computation. To clearly specify the computational process underlying the sensation, we systematically varied the visual feedback and examined its effect on the strength of the sensation. In contrast to the prevailing theory that sensory prediction errors modulate our perception, the sensation did not correlate with errors in cursor motion due to the delay. Instead, it correlated with the amount of exposure to the forward acceleration of the cursor. This indicates that the delayed cursor is interpreted as a mechanical load, and the sensation represents its visually implied reaction force. Namely, the CNS automatically computes inverse dynamics, using visually detected motions, to monitor the dynamic forces involved in our actions.
The dynamics of physical systems can often be represented as a deterministic relationship between force and motion. The extent to which our central nervous system (CNS) internally represents and further exploits this relationship has long been a central issue in computational neuroscience (see [1–4] for reviews). A series of results has supported the idea that the CNS computes forward dynamics to predict motion from applied force for motor control [5–8] and sensory perception [9–11]. However, it is still debated whether it also solves the computational problem of inverse dynamics [2,12,13], i.e. the estimation of force from motion.
Physiological evidence for the inverse computation has been provided by in vivo recording of Purkinje cells in monkeys during their reflexive eye movements . The firing patterns of the cells were largely explained by an inverse dynamics model of the eye, but the interpretation of this observation remains controversial [2,13]. Behavioural studies have demonstrated our ability to compensate for external forces during arm movements, presumably by acquiring internal models [12,15,16], but this referred to predicting the forces instead of estimating them from motion. Studies on the visual perception of lifted weights [17,18] and the relative mass of objects in collision [19,20] also referred to the concept of inverse dynamics, but these studies examined the ability to estimate the objects' properties rather than the forces involved. Although the phenomena introduced so far could each involve the inverse computation, the indirect evidence enabled explanations that omit the computation [13,21].
The peculiar sensation experienced when a delayed cursor is manipulated [22–24] is of particular interest in seeking more direct evidence of the inverse computation. As subjects tend to report a sense of resistance from the cursor motion, the kinetic percept may arise from a motion-correlate of force by means of inverse dynamics estimation, i.e. imaginary inertia of the lagging cursor is perceived from its acceleration. Interestingly, a recent study showed that when a delayed cursor is manipulated, our grip force pattern shifts forward in time relative to the applied load force, and the size of this shift correlates with the amount of delay . The study explained this behavioural change as compensation for the imaginary inertial force of the delayed cursor internally represented as a mechanical load. It was further speculated that a similar process could also underlie the sensation.
Another recent study, however, suggested an alternative hypothesis to account for the impact of the visual delay . It showed that a carried object is perceived as having a larger inertia than it actually does when its visual motion is delayed. On the basis of an observation that this bias decreases after subjects have adapted to the delay, it was concluded that the effect essentially reflects our perceiving a prediction error of object motion and our attributing the error to the inertia. The hypothesis is based on a prevailing theory that our motor awareness, as well as our motor control, is mediated by internal forward models (see [27,28] for reviews), associating impacts of altered visual feedback in general with internal computations of forward dynamics [10,11,29]. Importantly, considering that grip force during manipulation is also modulated by uncertainties about applied forces , the mere temporal shift of the grip force pattern, mentioned earlier, is also interpretable as a safety margin introduced in response to the prediction error.
Here, to clearly specify the computational basis of the sensation, we systematically varied the visual feedback and examined its effect on the strength of the sensation. If errors in cursor motion due to the delay explain the sensation, this would suggest the involvement of the forward estimation. On the other hand, if a cue related to a motion-correlate of force appears as the critical factor, this would suggest the involvement of the inverse estimation. Three experiments, each designed to segregate the contributions of the different potential factors, revealed that the resistive sensation correlates with a motion-correlate of force, suggesting that it reflects the inverse dynamics computation.
Thirteen participants took part in the first experiment (six males; mean age = 29.3 years old), 11 in the second experiment (five males; mean age = 28.4 years old) and 13 in the last experiment (three males; mean age = 29.3 years old). All participants were healthy, right-handed and naive to the purpose of the study.
Subjects moved a stylus pen on a digitized tablet (Wacom Intuos4 XL), and a cursor on a screen placed horizontally above the tablet followed the position of the pen hidden by the screen (figure 1a). A fixation point, two target markers and a cursor were displayed on the screen with a DLP projector at 60 Hz (figure 1b). A chin rest fixed the position of the head, and periodic beeps from earphones specified the timing and velocity of the arm movements.
All subjects were familiarized with the set-up before the experiment. During the familiarization, we occasionally delayed the cursor and instructed the subjects to focus on the sensation they felt in their arm. They reported a peculiar sensation, often described as heavy or resistive. We instructed the subjects to compare the strength of this sensation, defined by the prior experience, between two successive trials, each with a different delayed feedback (procedure illustrated in figure 1c). As a value representative of the sensation strength experienced under each condition, we calculated the probability of each feedback condition's being selected as causing a stronger sensation than that in a reference condition.
Trajectory and pressure of the pen were recorded at 125 Hz and low-pass filtered at 20 Hz. After the preprocessing, the following quantities were calculated as plausible factors of the resistive sensation. The first three quantities represented prediction errors of cursor motion, which are indicative of the involvement of forward computation. The latter two represented a motion-based force cue and a motion-based work cue, both of which are indicative of the involvement of the inverse computation. As the examined sensation was generally referred to as resistive rather than leftwards or rightwards, the quantities were calculated relative to the moving/accelerating direction of the non-delayed cursor (representative of the expected cursor and hand positions) so that their positive values could each be associated with a resistance to the hand movement. Importantly, however, while the former three quantities essentially represent changes in hand motion and provide only rough estimates of kinetic perturbations applied to the hand, the latter two quantities represent exact correlates of the reaction force and mechanical work when the cursor is interpreted as a pulled mechanical load. In what follows, each quantity is described in detail.
The distance of the delayed cursor behind the non-delayed one (figure 2a) was calculated as
as a value representative of the prediction error of cursor position. Here, x and y represent the non-delayed and delayed position of the cursor (i.e. y(t) = x(t − ΔT), where ΔT corresponds to the amount of cursor delay), represents the velocity of the non-delayed cursor, and represents its sign. Because of the elasticity of our musculoskeletal system, displacement of the hand from its default position (equilibrium point) is associable with a force applied to the hand towards its direction of motion. Backward displacement (positive error) of the cursor could therefore be attributed to an increase in resistive force (e.g. friction) and thus explain the resistive sensation.
Because there is sufficient evidence [6,11] that cursor velocity is predicted during our action, we considered the prediction error of cursor velocity as another potential factor. This error was calculated as
where represents the velocity of the delayed cursor. As a resistive impulse applied to the hand would decrease its velocity, a lack of forward velocity (positive error) could be attributed to an increase in the impulse (e.g. dynamic friction) and thus explain the sensation.
We also considered the prediction error of cursor acceleration. As for this error, we evaluated the error component relative to the direction of effort,
rather than that relative to the movement direction (i.e. ). This is because an assumption that the sensation correlates with the latter component leads to the implausible prediction that an assistive sensation would be experienced during cyclic movement with the delay (electronic supplementary material, text S1, explains how we excluded variants of the considered quantities). Note that the term represents the acceleration direction of the non-delayed cursor which approximates the direction of effort (i.e. force we apply to the pen). As an external force resistive to our effort would decrease the acceleration of the hand towards the intended direction, a lack of acceleration (positive error) could be attributed to an increase in such resistance (e.g. inertia of pen) and thus explain the sensation.
The delayed cursor is also interpretable as a mechanical load pulled by the preceding hidden hand. A possible case where the delayed cursor is interpreted as the load of a spring–mass–damper system, suggested earlier in the context of motor control , is shown in figure 2b. In such a case, the acceleration of the cursor implies an inertial force of the load applied to the hand towards the opposite direction (recall Newton's second and third laws of motion). As this force would be interpreted as resistive when it opposes the hand motion and assistive when it acts towards the hand-motion direction, the motion-correlate of resistive force experienced under the delayed feedback was calculated considering the motion direction of the hand () as
On the basis of a hypothesis that mechanical work cues underlie our haptic perception , we also evaluated the instantaneous mechanical work required in pulling the cursor interpreted as the load. The work cue calculation is also based on implied force cue :
In the first four hypotheses, we assumed that the kinetic sensation caused by the delay basically reflects cursor-motion errors () or cursor acceleration () per se, which are associated with leftwards or rightwards forces. The term was applied only to evaluate the components interpretable as resistive to match the subjective report. Absolute values of the errors and cues were also considered as potential factors (see electronic supplementary material, text S1). Recorded trajectories were also examined to account for possibilities that the sensation arose as artefacts of behaviour changes due to the delay  (see electronic supplementary material, text S2).
In our first experiment, with the cursor moved in a cyclic manner, we examined what amount of cursor delay caused the strongest resistive sensation. Subjects moved the delayed cursor between two targets with a movement cycle of either 1000 or 1333 ms and compared the strength of the sensation experienced for delays of 83, 167, 250, 333, 416 and 500 ms with that for the reference delay (250 ms). The delay with the strongest sensation, i.e. the delay most frequently selected as causing a stronger sensation, was obtained for each movement cycle from each subject. Temporal averages of the position error, force cue and work cue were predicted to be a function of sin(ωΔT) and to peak at a cursor phase delay of 90° (ωΔT = π/2), whereas averages of the velocity and acceleration errors were predicted to be a function of 1 − cos(ωΔT) and to peak at the largest delay (see electronic supplementary material, text S3, for details). The interest was on the delay at which the sensation was maximized.
Subjects moved the cursor back and forth between the targets while fixating on the centre marker. To finish each trial, they needed to maintain the cycle and width of the movement within specific margins (5% and 20%, respectively) for a predefined number of cycles in a row (four cycles for the 1000-ms-cycle trials and three cycles for the 1333-ms-cycle trials). The colour of the cursor indicated whether the conditions were satisfied, and subjects were instructed to refer to the sensation only when the requirements were met. The cursor disappeared for 150 ms every time the delay was changed to avoid transient effects. A trial with no delay was experienced before each comparison to avoid adaptation to the delay. All conditions were compared with the reference condition 24 times. The order of test and reference trials and the order of the compared conditions were randomized.
Figure 3 shows the normalized frequency distributions of the delays with the strongest sensation. The distribution centred at the delay of 250 ms for the movement cycle of 1000 ms and at the delay of 333 ms for the 1333-ms cycle. Importantly, both of these delays corresponded to the phase delay of 90°. The significant difference between the means of the two distributions (t12 = 2.55, p = 0.025) indicates that the peak depended on the phase delay rather than on the absolute delay. Temporal averages of the position error, force cue and work cue peaked at the 90° delays, whereas averages of the velocity error, acceleration error and absolute values of the cursor-motion errors peaked around the largest delays as predicted (electronic supplementary material, table S1). The coincidence of the peak delays suggests that the strength of the resistive sensation correlates with the position error, force cue or work cue. Meanwhile, neither the velocity error, nor the acceleration error, nor the absolute values of the cursor-motion errors correlated with the sensation. These quantities continued to show inconsistency with the sensation when we further considered the possibilities that the sensation judgements reflected only the maximum values of the factors within the trials or only the components interpretable as resistive (see electronic supplementary material, text S4, tables S2 and S3 for details). Differences in the movement trajectories among the delay conditions were negligible and are therefore also unlikely to explain the changes in the sensation (see electronic supplementary material, text S2).
To determine which quantity among the three remaining alternatives explains the sensation, we conducted a second experiment to directly test their contributions. This experiment was based on a theoretical prediction that each of the three quantities peaks at a different phase of the cyclic movement. Coloured lines in figure 4a,b indicate how the quantities were expected to change over each cycle for the 90° and 180° delays. The predictions are based on the definitions given in §2d, assuming sinusoidal trajectories of the pen (see electronic supplementary material, text S3). Here, we displayed the delayed cursor only for a short time around the phases in which the quantities were expected to peak (durations indicated by coloured regions in figure 4a,b). Then, by examining which phase produced the stronger sensation, we determined the underlying factor. The experiment consisted of two comparisons, one at a delay of 90° and another at a delay of 180°. For the 90° delay, we compared the sensation of the phase around the force and work peaks (0/180°, coloured in magenta) and the sensation of the phase around the position error peak (45/225°, coloured in blue). For the 180° delay, we compared the sensation of the phase around the force peak (0/180°, coloured in magenta) and the sensation of the phase around the work peak (−45/135°, coloured in yellow). Red portions in the upper illustrations of figure 4c,d indicate the position where the cursor would appear in the compared conditions of 90° and 180° delays. Red arrows in the bottom illustration indicate the corresponding trajectories in the phase plane.
Subjects moved the cursor back and forth between the targets as they did in the first experiment. The movement cycle was set to 1200 ms, and the delayed cursor appeared only for 133 ms at the specified phases. The appearance and disappearance of the cursor were controlled by a timer, which was reset to 0 every time the pen turned at the edges. The paired conditions were compared 32 times in random order. Prior to the experiment, subjects experienced trials with a full view of the delayed cursor, and they were informed that the cursor would appear only at specific phases in the experiment. We verbally confirmed that the subjects perceived the resistive sensation from the limited exposure to the delayed cursor.
Figure 4e,f shows the results of the comparisons for the 90° and 180° delays, respectively. For both delays, the condition with the visible phase near the edges was selected significantly more often than chance (50%) as the cause of the stronger sensation (90°: t10 = 5.41, p = 2.96 × 10−4; 180°: t10 = 3.86, p = 3.1 × 10−3). The time average of the force cue calculated from the visible phases of the trajectories was larger in the selected conditions. Meanwhile, the conditions had significantly smaller values of the time average of position error (at 90° delay) or of the work cue (at 180° delay). Here again, differences in the actual movements were minimum (see electronic supplementary material, text S2). The findings suggest that the sensation correlates with the force cue rather than with the other two quantities.
To further confirm our theory, we tested in our final experiment whether the theory also applies to discrete (non-periodic) movements. Here, by temporarily occluding the cursor at different intervals and measuring the resulting strengths of the sensation, we examined to what extent the motion of the delayed cursor at different time intervals of a discrete reaching movement each contributed to the resistive sensation. While the subjects moved the cursor from left to right with a cursor delay of 150 ms, the cursor disappeared for 100 ms at 17–117 ms (T1), 100–200 ms (T2), 200–300 ms (T3), 300–400 ms (T4) or 400–500 ms (T5) after movement onset, as illustrated in figure 5a. Each test trial was compared with a reference trial in which the occlusion always occurred at T3 (200–300 ms).
Subjects moved the cursor from the left target to the right while fixating on the centre marker. The timing and duration of the movement were specified by four beeps with a constant interval of 700 ms. Subjects started the reach after the third beep and arrived at the right target in sync with the fourth beep. Each comparison of the conditions consisted of four reaching trials, the first two without the delay to avoid adaptation and the latter two as the test and reference trials. All conditions were compared with the reference condition 16 times. The order of test and reference trials and the order of the compared conditions were randomized. The occlusion was controlled by a timer that started just after the movement detection (this is why time-interval T1 started at 17 ms instead of 0 ms).
The time interval of the occlusion had a significant effect on the strength of the resistive sensation (figure 5b; F4,48 = 4.75, p = 3.0 × 10−3). To evaluate the average strength of the implied force observed under the compared conditions, we calculated the forward acceleration of the cursor (force cue) averaged across only its visible time intervals. Then, to examine its involvement in the sensation, we plotted this value and the resistive sensation index. We found a statistically significant positive relation (figure 5c; correlation for all time intervals of all participants: R = 0.45, t63 = 3.95, p = 1.99 × 10−4). Note that we calculated the values for the force cue as differences from the mean obtained for each subject in order to exclude any pseudo-effects of inter-subject variances on the correlation. The correlation strongly supports the theory that the sensation results from the inverse dynamics computation. The modulation of the sensation was again not explained by the position error of the cursor; i.e. the error did not correlate with the sensation strength (figure 5d; correlation for all time-intervals of all participants: R = 0.08, t63 = −0.65, p = 0.52). The judgements made by the subjects were also unlikely to reflect the behavioural changes due to the delay (see electronic supplementary material, text S2).
Manipulation of a delayed cursor was accompanied by a peculiar resistive sensation. Our goal was to reveal the computational basis of the illusory sensation by specifying its principle factor.
Following the prevailing theory that sensory prediction errors underlie our motor awareness [27,28], an earlier study suggested that prediction error of cursor motion underlies the visual contribution to our haptic perception . The results of our experiments, however, showed that neither the position error, nor the velocity error, nor the acceleration error, nor the absolute values of these errors correlated with the sensation. The strongest resistive sensation at the 90° delay strongly limited the options to explain the sensation, leaving only the position error as a candidate within the errors (electronic supplementary material, text S4). The second and third experiments showed that the sensation does not correlate with the position error. One could argue that the effects of the prediction errors do not necessarily correlate with their sizes as there exist critical limits of the delay for processing visual feedback [26,34]. However, the finding that the peak delay of the sensation is characterized by a phase delay rather than an absolute delay rules out explanations based on such temporal limits. The limits were also unlikely to explain our second and third results as the delays were identical between the compared conditions.
Earlier studies assumed that behavioural changes due to altered visual feedback underlie the effects on our haptic perception . In our experiment, however, we excluded this secondary effect by rigorously controlling the participants' motor behaviour, i.e. differences in movement trajectories between the compared conditions were either non-significant or negligibly small. As difficulty in controlling the cursor was evident at large delays in our first experiment, we also considered its effect and confirmed that it does not explain the subjective reports (electronic supplementary material, text S2, provides details on these issues).
Instead of the above-mentioned factors, the results of our experiments revealed a striking coincidence between the acceleration of the delayed cursor towards hand-motion direction (force cue) and the resistive sensation. When the cursor was moved in a cyclic manner, the magnitude of the sensation peaked at a cursor delay of 90°, where the temporal average of the force cue maximized (first result). The turning phase of cursor motion, in which the force cue peaked, produced a stronger resistive sensation than the other phases (second result). Furthermore, when the cursor was occluded at different time intervals of a discrete reaching movement, the sensation scaled in correlation with the force cue averaged across the visible intervals (third result). In essence, the strength of the sensation scaled in correlation with the temporal average of the force cue (the contribution of negative components of the force cue were irrelevant to the correlation, as described in electronic supplementary material, text S5). As the force cue correlates with the imaginary inertial force of the dragged cursor which virtually resists hand motion, these results suggested that the resistive sensation corresponds to the visually implied force. While it was also possible that the implied force was perceived as mechanical work , the comparison at 180° delay in our second experiment suggested that the force cue itself was the factor.
To summarize, the resistive sensation of delayed visual feedback appeared as evidence of the inverse dynamics estimation. Namely, it suggested we control our ability to perceive the reaction forces of the objects from their visual motion. Importantly, this differs from the widely accepted theory that vision specializes in monitoring the spatial aspects of our physical interaction and only biases our force perception on the basis of the cognitive contrasting effect (e.g. size–weight illusion ). The visual percept of reaction force explains why we are so easily deceived by altered visual feedback in haptically perceiving an object's properties such as its inertia , stiffness [36,37] and surface shape  (perception of surface shape also depends on applied reaction force ).
Although earlier studies have suggested the inverse dynamics computation in the context of force control [12,15,16] or visual perception of observed weights in motion [17–20], the indirect evidence enabled alternative explanations that omit the inverse computation. For instance, grip force modulation in synchrony with applied dynamic load forces  could be partly attributed to low-level coupling of grip and arm forces . Relative weights of objects lifted by others [17,18] can be speculated from their relative velocities without the inverse computation. In contrast to these findings, our findings provide direct evidence that the fundamental relationship between force and motion is indeed encoded in our CNS and that we solve the computational problem of inverse dynamics.
While our study focused on whether the resistive sensation reflects the inverse computation, it also provides insight into how our CNS reacts to sudden changes in sensory feedback. In fact, as the resistive sensation is evident only under visual delay and fades after prolonged exposure to it , sudden changes in the visual feedback appear to be essential for the sensation of additional inertia. This suggests the involvement of the prediction error, but our results revealed that the errors per se do not explain the sensation. Our study rather supports the theory that sudden changes in visual feedback, and possibly the resulting prediction error, trigger a re-selection of our internal model, i.e. the mismatch between the hand and cursor is resolved by interpreting the cursor as a pulled mechanical load (e.g. spring–mass–damper system) . Note that peak sensation at the 90° delay can be explained only by assuming a non-delayed trajectory of the hand separate from the delayed cursor. The idea that temporal perturbation of sensory feedback can be interpreted as physical dynamics is consistent with previous findings [41,42]. The assumption of the additional load explains the sensation of the additional inertia. Prolonged exposure to the delay could remove the imaginary inertia, not only by updating our internal forward model of hand motion but also by learning that the inertia of the imaginary load is small or that the visual feedback is unreliable.
An interesting question remaining is why the inverse dynamics computation yields a somatic sensation. One possibility is the existence of functional associations between visual motion signals and somatic signals, which could be complementary in encoding our somatic sensation. A similar example is found in multimodal filling-in . Modulation in the somatosensory areas by altered visual feedback  is interpretable as an effect of the visually coded somatic information.
Finally, we may ask what role the visual motion cue could play in force estimation. The rationality of the cue may be that it directly represents the forces of the objects we control. Imagine putting a golf ball into a hole. Somatosensory signals associated with the hit will vary depending on the state of our body, the state of contact and the tool used. Furthermore, these signals are, in essence, represented in intrinsic coordinates and contaminated with signals of intrinsic forces. Meanwhile, the visual motion cue consistently provides direct information on the interaction force represented in extrinsic coordinates. Accordingly, integration of the visual and somatosensory signals could contribute to obtaining a robust, extrinsic representation of force, which would be essential for flexible and accurate motor control.
We thank P. Haggard, M. Kashino, S. Nishida and N. Saijo for their comments. The experiments used Cogent Graphics developed by John Romaya at the LON at the Wellcome Department of Imaging Neuroscience.
Ethical approval was obtained from the NTT Communication Science Laboratories Ethical Committee. All subjects gave written informed consent.
The datasets supporting this article have been uploaded as part of the electronic supplementary material.
S.T. and H.G. designed the experiments and wrote the paper; S.T. collected and analysed the data.
We declare we have no competing interests.
JST CREST Project for ‘Developing a communication environment by decoding and controlling implicit interpersonal information’ funded the study.