Behavioral changes driven by reinforcement and punishment are referred to as simple or model-free reinforcement learning. Animals can also change their behaviors by observing events that are neither appetitive nor aversive, when these events provide new information about payoffs available from alternative actions. This is an example of model-based reinforcement learning, and can be accomplished by incorporating hypothetical reward signals into the value functions for specific actions. Recent neuroimaging and single-neuron recording studies showed that the prefrontal cortex and the striatum are involved not only in reinforcement and punishment, but also in model-based reinforcement learning. We found evidence for both types of learning, and hence hybrid learning, in monkeys during simulated competitive games. In addition, in both the dorsolateral prefrontal cortex and orbitofrontal cortex, individual neurons heterogeneously encoded signals related to actual and hypothetical outcomes from specific actions, suggesting that both areas might contribute to hybrid learning.
belief learning; decision making; game theory; reinforcement learning; reward
Game theory analyses optimal strategies for multiple decision makers interacting in a social group. However, the behaviours of individual humans and animals often deviate systematically from the optimal strategies described by game theory. The behaviours of rhesus monkeys (Macaca mulatta) in simple zero-sum games showed similar patterns, but their departures from the optimal strategies were well accounted for by a simple reinforcement-learning algorithm. During a computer-simulated zero-sum game, neurons in the dorsolateral prefrontal cortex often encoded the previous choices of the animal and its opponent as well as the animal's reward history. By contrast, the neurons in the anterior cingulate cortex predominantly encoded the animal's reward history. Using simple competitive games, therefore, we have demonstrated functional specialization between different areas of the primate frontal cortex involved in outcome monitoring and action selection. Temporally extended signals related to the animal's previous choices might facilitate the association between choices and their delayed outcomes, whereas information about the choices of the opponent might be used to estimate the reward expected from a particular action. Finally, signals related to the reward history might be used to monitor the overall success of the animal's current decision-making strategy.
prefrontal cortex; decision making; reward
The value of an object acquired by a particular action often determines the motivation to produce that action. Previous studies found neural signals related to the values of different objects or goods in the orbitofrontal cortex, while the values of outcomes expected from different actions are broadly represented in multiple brain areas implicated in movement planning. However, how the brain combines the values associated with various objects and the information about their locations is not known. In this study, we tested whether the neurons in the dorsolateral prefrontal cortex (DLPFC) and striatum in rhesus monkeys might contribute to translating the value signals between multiple frames of reference. Monkeys were trained to perform an oculomotor intertemporal choice in which the color of a saccade target and the number of its surrounding dots signaled the magnitude of reward and its delay, respectively. In both DLPFC and striatum, temporally discounted values (DVs) associated with specific target colors and locations were encoded by partially overlapping populations of neurons. In the DLPFC, the information about reward delays and DVs of rewards available from specific target locations emerged earlier than the corresponding signals for target colors. Similar results were reproduced by a simple network model built to compute DVs of rewards in different locations. Therefore, DLPFC might play an important role in estimating the values of different actions by combining the previously learned values of objects and their present locations.
intertemporal choice; prefrontal cortex; reward; temporal discounting; utility
Human behaviors can be more powerfully influenced by conditioned reinforcers, such as money, than by primary reinforcers. Moreover, people often change their behaviors to avoid monetary losses. However, the effect of removing conditioned reinforcers on choices has not been explored in animals, and the neural mechanisms mediating the behavioral effects of gains and losses are not well understood. To investigate the behavioral and neural effects of gaining and losing a conditioned reinforcer, we trained rhesus monkeys for a matching pennies task in which the positive and negative values of its payoff matrix were realized by the delivery and removal of a conditioned reinforcer. Consistent with the findings previously obtained with non-negative payoffs and primary rewards, the animal’s choice behavior during this task was nearly optimal. Nevertheless, the gain and loss of a conditioned reinforcer significantly increased and decreased, respectively, the tendency for the animal to choose the same target in subsequent trials. We also found that the neurons in the dorsomedial frontal cortex, dorsal anterior cingulate cortex, and dorsolateral prefrontal cortex often changed their activity according to whether the animal earned or lost a conditioned reinforcer in the current or previous trial. Moreover, many neurons in the dorsomedial frontal cortex also signaled the gain or loss occurring as a result of choosing a particular action as well as changes in the animal’s behaviors resulting from such gains or losses. Thus, primate medial frontal cortex might mediate the behavioral effects of conditioned reinforcers and their losses.
cingulate cortex; decision making; prefrontal cortex; reinforcement learning; reward; punishment; neuroeconomics
The primate prefrontal cortex contributes to stimulus-guided behavior, but the functional specializations among its areas remain uncertain. To better understand such specializations, we contrasted neuronal activity in the dorsolateral prefrontal cortex (PFdl) and the orbital prefrontal cortex (PFo). The task required rhesus monkeys to use a visual cue to choose a saccade target. Some cues instructed the monkeys to repeat their most recent response; others instructed them to change it. Responses were followed by feedback: fluid reward if correct, visual feedback if incorrect. Previous studies, using different tasks, have reported that PFo neurons did not encode responses. We found PFo did encode responses in this task, but only near feedback time, after the response had been completed. PFdl differed from PFo in several respects. As reported previously, some PFdl neurons encoded responses from the previous trial and others encoded planned responses. PFo neurons did not have these properties. After feedback, PFdl encoded rewarded responses better than unrewarded ones and thus combined response and outcome information. PFo, in contrast, encoded the responses chosen, rewarded or not. These findings suggest that PFdl and PFo contribute differently to response knowledge, with PFo using an outcome-independent signal to monitor current responses at feedback time.
Decision; feedback; monitoring; evaluation; frontal lobe; prefrontal cortex
Mentalizing is defined as the inference of mental states of fellow humans, and is a particularly important skill for social interactions. Here we assessed whether activity in brain areas involved in mentalizing is specific to the processing of mental states or can be generalized to the inference of non-mental states by comparing brain responses during the interaction with an intentional and an artificial agent. Participants were scanned using fMRI during interactive rock-paper-scissors games while believing their opponent was a fellow human (Intentional agent, Int), a humanoid robot endowed with an artificial intelligence (Artificial agent, Art), or a computer playing randomly (Random agent, Rnd). Participants' subjective reports indicated that they adopted different stances against the three agents. The contrast of brain activity during interaction with the artificial and the random agents didn't yield any cluster at the threshold used, suggesting the absence of a reproducible stance when interacting with an artificial intelligence. We probed response to the artificial agent in regions of interest corresponding to clusters found in the contrast between the intentional and the random agents. In the precuneus involved in working memory, the posterior intraparietal suclus, in the control of attention and the dorsolateral prefrontal cortex, in executive functions, brain activity for Art was larger than for Rnd but lower than for Int, supporting the intrinsically engaging nature of social interactions. A similar pattern in the left premotor cortex and anterior intraparietal sulcus involved in motor resonance suggested that participants simulated human, and to a lesser extend humanoid robot actions, when playing the game. Finally, mentalizing regions, the medial prefrontal cortex and right temporoparietal junction, responded to the human only, supporting the specificity of mentalizing areas for interactions with intentional agents.
social cognition; neuroscience; artificial intelligence; fMRI
Humans and animals often must choose between rewards that differ in their qualities, magnitudes, immediacy, and likelihood, and must estimate these multiple reward parameters from their experience. However, the neural basis for such complex decision making is not well understood. To understand the role of the primate prefrontal cortex in determining the subjective value of delayed or uncertain reward, we examined the activity of individual prefrontal neurons during an inter-temporal choice task and a computer-simulated competitive game. Consistent with the findings from previous studies in humans and other animals, the monkey’s behaviors during inter-temporal choice were well accounted for by a hyperbolic discount function. In addition, the activity of many neurons in the lateral prefrontal cortex reflected the signals related to the magnitude and delay of the reward expected from a particular action, and often encoded the difference in temporally discounted values that predicted the animal’s choice. During a computerized matching pennies game, the animals approximated the optimal strategy, known as Nash equilibrium, using a reinforcement learning algorithm. We also found that many neurons in the lateral prefrontal cortex conveyed the signals related to the animal’s previous choices and their outcomes, suggesting that this cortical area might play an important role in forming associations between actions and their outcomes. These results show that the primate lateral prefrontal cortex plays a central role in estimating the values of alternative actions based on multiple sources of information.
game theory; inter-temporal choice; reinforcement learning; utility theory; temporal discounting
Prefrontal cortex mediates cognitive control by means of circuitry organized along dorso-ventral and rostro-caudal axes. Along the dorso-ventral axis, ventrolateral PFC controls semantic information, whereas dorsolateral PFC encodes task rules. Along the rostro-caudal axis, anterior prefrontal cortex encodes complex rules and relationships between stimuli, whereas posterior prefrontal cortex encodes simple relationships between stimuli and behavior. Evidence of these gradients of prefrontal cortex organization has been well documented in fMRI studies, but their functional correlates have not been examined with regard to integrity of underlying white matter tracts. We hypothesized that (a) the integrity of specific white matter tracts is related to cognitive functioning in a manner consistent with the dorso-ventral and rostro-caudal organization of the prefrontal cortex, and (b) this would be particularly evident in healthy older adults. We assessed three cognitive processes that recruit the prefrontal cortex and can distinguish white matter tracts along the dorso-ventral and rostro-caudal dimensions –episodic memory, working memory, and reasoning. Correlations between cognition and fractional anisotropy as well as fiber tractography revealed: (a) Episodic memory was related to ventral prefrontal cortex-thalamo-hippocampal fiber integrity; (b) Working memory was related to integrity of corpus callosum body fibers subserving dorsolateral prefrontal cortex; and (c) Reasoning was related to integrity of corpus callosum body fibers subserving rostral and caudal dorsolateral prefrontal cortex. These findings confirm the ventrolateral prefrontal cortex's role in semantic control and the dorsolateral prefrontal cortex's role in rule-based processing, in accordance with the dorso-ventral prefrontal cortex gradient. Reasoning-related rostral and caudal superior frontal white matter may facilitate different levels of task rule complexity. This study is the first to demonstrate dorso-ventral and rostro-caudal prefrontal cortex processing gradients in white matter integrity.
Our choices often require appropriate actions in order to obtain a preferred outcome, but the neural underpinnings that link decision making and action selection remain largely undetermined. Recent theories propose that action selection occurs simultaneously, i.e. parallel in time, with the decision process. Specifically, it is thought that action selection in motor regions originates from a competitive process which is gradually biased by evidence signals originating in other regions, such as those specialized in value computations. Biases reflecting the evaluation of choice options should thus emerge in the motor system before the decision process is complete. Using transcranial magnetic stimulation, we sought direct physiological evidence for this prediction by measuring changes in cortico-spinal excitability in human motor cortex during value-based decisions. We found that excitability for chosen versus unchosen actions distinguishes the forthcoming choice before completion of the decision process. Both excitability and reaction times varied as a function of the subjective value-difference between chosen and unchosen actions, consistent with this effect being value-driven. This relationship was not observed in the absence of a decision. Our data provide novel evidence in humans that internally generated value-based decisions influence the competition between action representations in motor cortex before the decision process is complete. This is incompatible with models of serial processing of stimulus, decision, and action.
Functional impairment of the orbital and medial prefrontal cortex underlies deficits in executive control that characterize addictive disorders, including alcohol addiction. Previous studies indicate that alcohol alters glutamate neurotransmission and one substrate of these effects may be through the reconfiguration of the subunits constituting ionotropic glutamate receptor (iGluR) complexes. Glutamatergic transmission is integral to cortico-cortical and cortico-subcortical communication, and alcohol-induced changes in the abundance of the receptor subunits and/or their splice variants may result in critical functional impairments of prefrontal cortex in the alcohol-addicted state.
Methods and results
The effects of chronic ethanol self-administration on glutamate receptor ionotropic NMDA (GRIN), as well as GRIN1 splice variant mRNA expression was studied in the orbitofrontal cortex (OFC; Area 13), dorsolateral prefrontal cortex (DLPFC; Area 46) and anterior cingulate cortex (ACC; Area 24) of male cynomolgus monkeys. Chronic ethanol self-administration resulted in significant changes in the expression of NMDA subunit mRNA expression in the DLPFC and OFC, but not the ACC. In DLPFC, the overall expression of NMDA subunits was significantly decreased in ethanol treated monkeys. Slight but significant changes were observed for synaptic associated protein 102 kD (SAP102) and neuronal nitric oxide synthase (nNOS) mRNAs. In OFC, the NMDAR1 variant GRIN1-1 was reduced while GRIN1-2 was increased. Furthermore, no significant changes in GFAP protein levels were observed in either the DLPFC or OFC.
Results from these studies provide the first demonstration of post-transcriptional regulation of iGluR subunits in the primate brain following long-term ethanol self-administration. Furthermore, changes in these transcripts do not appear to reflect changes in glial activation or loss. Further studies examining the expression and cellular localization of subunit proteins and receptor pharmacology would shed more light on the findings reported here.
Ethanol; Glutamate; messenger RNA; Prefrontal Cortex; qPCR; Primate
We assessed the involvement of the orbital prefrontal cortex (PFo), the prelimbic region of the medial prefrontal cortex (PL), and the amygdala in goal-directed behavior. Rhesus monkeys were trained on a task in which two different instrumental responses were linked to two different outcomes. One response, called ‘Tap’, required the monkeys to repeatedly touch a colored square on a video monitor, to produce one kind of food reward. The other response, called ‘Hold’, required persistent contact of an identical stimulus, and it produced a different kind of food reward. Following training, we assessed the effects of satiety-specific reinforcer devaluation as a way to probe each monkey’s use of goal-directed behavior. In this procedure, monkeys were allowed to consume one of the two foods to satiety, and were then tested for Tap/Hold preference under extinction. Unoperated control monkeys showed a reduction in the response associated with obtaining the devalued food, called the devaluation effect, a hallmark of goal-directed behavior. Monkeys with bilateral lesions of PFo or the amygdala exhibited significantly reduced devaluation effects. Results from monkeys with PL lesions were equivocal. We conclude that both PFo and the amygdala play a significant role in goal-directed behavior in monkeys. Notably, the findings for PFo challenge the idea that orbital and medial prefrontal regions are exclusively dedicated to object- and action-based processes, respectively.
Studies in monkeys show clear anatomical and functional distinctions among networks connecting with subregions within the prefrontal cortex. Three such networks are centered on lateral orbitofrontal cortex, medial frontal and cingulate cortex, and lateral prefrontal cortex and all have been identified with distinct cognitive roles. Although these areas differ in a number of their cortical connections, some of the first anatomical evidence for these networks came from tracer studies demonstrating their distinct patterns of connectivity with the mediodorsal (MD) nucleus of the thalamus. Here, we present evidence for a similar topography of MD thalamus prefrontal connections, using non-invasive imaging and diffusion tractography (DWI–DT) in human and macaque. DWI–DT suggested that there was a high probability of interconnection between medial MD and lateral orbitofrontal cortex, between caudodorsal MD and medial frontal/cingulate cortex, and between lateral MD and lateral prefrontal cortex, in both species. Within the lateral prefrontal cortex a dorsolateral region (the principal sulcus in the macaque and middle frontal gyrus in the human) was found to have a high probability of interconnection with the MD region between the regions with a high probability of interconnection with other parts of the lateral prefrontal cortex and with the lateral orbitofrontal cortex. In addition to suggesting that the thalamic connectivity in the macaque is a good guide to human prefrontal cortex, and therefore that there are likely to be similarities in the cognitive roles played by the prefrontal areas in both species, the present results are also the first to provide insight into the topography of projections of an individual thalamic nucleus in the human brain.
Anatomy; DTI; Human; Macaque; Thalamus
Studies investigating response reversal consistently implicate regions of medial and lateral prefrontal cortex when reinforcement contingencies change. However, it is unclear from these studies how these regions give rise to the individual components of response reversal, such as reinforcement value encoding, response inhibition, and response change. Here we report a novel instrumental learning task designed to determine whether regions implicated in processing reversal errors are uniquely involved in this process, or whether they play a more general role in representing response competition, reinforcement value, or punishment value in the absence of demands for response change. In line with previous findings, reversal errors activated orbitofrontal cortex, dorsomedial prefrontal cortex, ventrolateral prefrontal cortex, caudate, and dorsolateral prefrontal cortex. These regions also showed increased activity to errors in the absence of contingency changes. In addition, ventrolateral PFC, caudate, and dorsolateral PFC each exhibited increased activity following correct reversals. Activity in these regions was not significantly modulated by changes in reinforcement value that were not sufficient to make an alternative response advantageous. These data do not support punishment-processing or prepotent reponse inhibition accounts of ventrolateral prefrontal cortex function. Instead, they support recent conceptualizations of ventrolateral prefrontal cortex function that implicate this region in resolving response competition by manipulating the representation of either motor response options, or object features. These data also suggest that dorsolateral prefrontal cortex plays a role in reversal learning, probably through top down attentional control of object or reinforcement features when task demands increase.
Response reversal; affective shift; response competition; ventrolateral prefrontal cortex; decision-making
Motivational interventions to improve health behaviors based on conventional cognitive and behavioral theories have been extensively studied; however, advances in neuroimaging technology make it possible to assess the neurophysiological basis of health behaviors, such as physical activity. The goals of this approach are to support new interventions to achieve optimal outcomes.
This study used functional magnetic resonance imaging (fMRI) to assess differences in brain responses in healthy weight to obese midlife women during a goal-directed decision task.
Thirty nondiabetic, midlife (age 47–55 years) women with body mass index (BMI) ranging from 18.5 to 40 kg/m2 were recruited. A descriptive, correlational design was used to assess the relationship between brain activations and weight status. Participants underwent a goal-directed behavior task in the fMRI scanner consisting of a learning and implementation phase. The task was designed to assess both goal-directed and habitual behaviors. One participant was omitted from the analysis because of excessive motion (>4 mm), and six were omitted because of fewer than 50% correct responses on the exit survey. Four participants developed claustrophobia in the scanner and were disqualified from further participation. The remaining 19 participants were included in the final analysis.
Brain responses while participants learned goal-directed behavior showed a positive correlation with BMI in the dorsomedial prefrontal cortex (dmPFC) and a negative correlation with BMI in the insula. During the implementation of goal-directed behavior, brain responses in the dorsolateral prefrontal cortex (dlPFC) negatively correlated with BMI.
These results indicate that overweight women activate regions associated with cognitive control to a greater degree than healthy weight women during goal-directed learning. The brain regions activated (dmPFC, dlPFC, insula) are associated with cognitive control and self-regulation. On the other hand, healthy weight women activate regions associated with emotion processing, planning, and self-regulation (lateral orbitofrontal cortex, anterior insula) to a greater degree than overweight women during goal-directed learning and implementation of goal-directed behavior. Overweight women activate cognitive control regions while learning associations between actions and outcomes; however, this is not the case during the implementation phase—which may make it more difficult to transform goals into action (e.g., maintain physical activity over time). Overall, these results indicate that overweight midlife women respond differently during learning and implementation of actions that lead to positive outcomes during a general test of goal-directed behavior. Future study is needed to assess the transfer of goal-directed and habitual behavior to specific aspects of energy balance to improve health outcomes.
fMRI; health behavior; neuroimaging; neurophysiology; obesity; women’s health
Considering the neuroscientific findings on reward, learning, value, decision-making, and cognitive control, motivation can be parsed into three sub processes, a process of generating motivation, a process of maintaining motivation, and a process of regulating motivation. I propose a tentative neuroscientific model of motivational processes which consists of three distinct but continuous sub processes, namely reward-driven approach, value-based decision-making, and goal-directed control. Reward-driven approach is the process in which motivation is generated by reward anticipation and selective approach behaviors toward reward. This process recruits the ventral striatum (reward area) in which basic stimulus-action association is formed, and is classified as an automatic motivation to which relatively less attention is assigned. By contrast, value-based decision-making is the process of evaluating various outcomes of actions, learning through positive prediction error, and calculating the value continuously. The striatum and the orbitofrontal cortex (valuation area) play crucial roles in sustaining motivation. Lastly, the goal-directed control is the process of regulating motivation through cognitive control to achieve goals. This consciously controlled motivation is associated with higher-level cognitive functions such as planning, retaining the goal, monitoring the performance, and regulating action. The anterior cingulate cortex (attention area) and the dorsolateral prefrontal cortex (cognitive control area) are the main neural circuits related to regulation of motivation. These three sub processes interact with each other by sending reward prediction error signals through dopaminergic pathway from the striatum and to the prefrontal cortex. The neuroscientific model of motivational process suggests several educational implications with regard to the generation, maintenance, and regulation of motivation to learn in the learning environment.
motivation; neuroeducation; educational neuroscience; reward; value; goal; decision-making; self-regulation
Flexible behaviors are organized by complex neural networks in the prefrontal cortex. Recent studies have suggested that such networks exhibit multiple dynamical states, and can switch rapidly from one state to another. In many complex systems such as the brain, the early-warning signals that may predict whether a critical threshold for state transitions is approaching are extremely difficult to detect. We hypothesized that increases in firing irregularity are a crucial measure for predicting state transitions in the underlying neuronal circuits of the prefrontal cortex. We used both experimental and theoretical approaches to test this hypothesis. Experimentally, we analyzed activities of neurons in the prefrontal cortex while monkeys performed a maze task that required them to perform actions to reach a goal. We observed increased firing irregularity before the activity changed to encode goal-to-action information. Theoretically, we constructed theoretical generic neural networks and demonstrated that changes in neuronal gain on functional connectivity resulted in a loss of stability and an altered state of the networks, accompanied by increased firing irregularity. These results suggest that assessing the temporal pattern of neuronal fluctuations provides important clues regarding the state stability of the prefrontal network. We also introduce a novel scheme that the prefrontal cortex functions in a metastable state near the critical point of bifurcation. According to this scheme, firing irregularity in the prefrontal cortex indicates that the system is about to change its state and the flow of information in a flexible manner, which is essential for executive functions. This metastable and/or critical dynamical state of the prefrontal cortex may account for distractibility and loss of flexibility in the prefrontal cortex in major mental illnesses such as schizophrenia.
Vagus nerve stimulation (VNS) can improve depression. Cognitive models of depression highlight an over-representation of negative thoughts and memories, with depressed individuals showing memory facilitation for negative material. We hypothesized that the antidepressant action of VNS may emerge through corrective influences on ‘negativity bias’ in memory. We therefore examined the impact of VNS on emotional memory and its underlying brain activity.
We tested a single patient undergoing VNS for treatment-resistant depression (TRD). Stimulation was set at a 30/66 second on/off cycle during three encoding blocks when the patient viewed randomly presented positive, negative and neutral words. Following each block, VNS was switched off and the patient identified previously seen words from distractors in a subsequent recognition memory task. The patient was scanned using functional magnetic resonance imaging (fMRI) during the first encoding block.
There was robust recall of negative material viewed during ‘off cycles’ of VNS but subsequent memory of negative words was attenuated during active VNS (on cycles). These effects were not apparent for neutral and positive words. In neuroimaging, direct modulatory effects of VNS were observed in dorsomedial, dorsolateral and orbital regions of prefrontal cortex. Moreover, during encoding of negative words, compared to positive and neutral words, VNS also modulated activity within orbitofrontal, ventromedial and polar prefrontal cortices, mid cingulate cortex and brainstem.
Our observations show that VNS can interfere with memory for negative information, an effect that may contribute to its antidepressant role. Neuroimaging implicates regions including ventral and medial prefrontal cortex as an underlying neural substrate.
Research in animal learning and behavioral neuroscience has distinguished between two forms of action control: a habit-based form, which relies on stored actio n values, and a goal-dir ected form, which forecasts and compares action outcomes based on a model of the environment. While habit-based control has been the subject of extensive computational research, the computational principles underlying goal-directed control in animals have so far received less attention. In the present paper, we advance a computational framework for goal-directed control in animals and humans. We take three empirically motivated points as founding premises: (1) Neurons in dorsolateral prefrontal cortex represent action policies, (2) Neurons in orbitofrontal cortex represent rewards, and (3) Neural computation, across domains, can be appropriately understood as performing structured probabilistic inference. On a purely computational level, the resulting account relates closely to previous work using Bayesian inference to solve Markov decision problems, but extends this work by introducing a new algorithm, which provably converges on optimal plans. On a cognitive and neuroscientific level, the theory provides a unifying framework for several different forms of goal-directed action selection, placing emphasis on a novel form, within which orbitofrontal reward representations directly drive policy selection.
Functional clusters of neurons in the monkey prefrontal and anterior cingulate cortex are involved in guiding attention to the most valuable objects in a scene.
Attentional control ensures that neuronal processes prioritize the most relevant stimulus in a given environment. Controlling which stimulus is attended thus originates from neurons encoding the relevance of stimuli, i.e. their expected value, in hand with neurons encoding contextual information about stimulus locations, features, and rules that guide the conditional allocation of attention. Here, we examined how these distinct processes are encoded and integrated in macaque prefrontal cortex (PFC) by mapping their functional topographies at the time of attentional stimulus selection. We find confined clusters of neurons in ventromedial PFC (vmPFC) that predominantly convey stimulus valuation information during attention shifts. These valuation signals were topographically largely separated from neurons predicting the stimulus location to which attention covertly shifted, and which were evident across the complete medial-to-lateral extent of the PFC, encompassing anterior cingulate cortex (ACC), and lateral PFC (LPFC). LPFC responses showed particularly early-onset selectivity and primarily facilitated attention shifts to contralateral targets. Spatial selectivity within ACC was delayed and heterogeneous, with similar proportions of facilitated and suppressed responses during contralateral attention shifts. The integration of spatial and valuation signals about attentional target stimuli was observed in a confined cluster of neurons at the intersection of vmPFC, ACC, and LPFC. These results suggest that valuation processes reflecting stimulus-specific outcome predictions are recruited during covert attentional control. Value predictions and the spatial identification of attentional targets were conveyed by largely separate neuronal populations, but were integrated locally at the intersection of three major prefrontal areas, which may constitute a functional hub within the larger attentional control network.
To navigate within an environment filled with sensory stimuli, the brain must selectively process only the most relevant sensory information. Identifying and shifting attention to the most relevant sensory stimulus requires integrating information about its sensory features as well as its relative value, that is, whether it's worth noticing. In this study, we describe groups of neurons in the monkey prefrontal cortex that convey signals relating to the value of a stimulus and its defining feature and location at the very moment when attention is shifted to the stimulus. We found that signals conveying information about value were clustered in a ventromedial prefrontal region, and were separated from sensory signals within the anterior cingulate cortex and the lateral prefrontal cortex. The integration of valuation and other “top-down” processes, however, was achieved by neurons clustered at the intersection of ventromedial, anterior cingulate, and lateral prefrontal cortex. We conclude that valuation processes are recruited when attention is shifted, independent of any overt behavior. Moreover, our analysis suggests that valuation processes can bias the initiation of attention shifts, as well as ensure sustained attentional focusing.
Chronic cocaine administration regulates the expression of several proteins related to dopaminergic signaling and synaptic function in the mesocorticolimbic pathway, including the prefrontal cortex. Functional abnormalities in the prefrontal cortex are hypothesized to be due in part to the expression of proteins involved in dopamine signaling and plasticity. Adult male rhesus monkeys self-administered cocaine (i.v.) under limited (n = 4) and extended access conditions (n = 6). The abundance of surrogate markers of dopamine signaling and plasticity in the dorsolateral prefrontal cortex (DLPFC), orbitofrontal cortex (OFC), and anterior cingulate cortex (ACC) were examined: glycosylated and non-glycosylated forms of the dopamine transporter (efficiency of dopamine transport), tyrosine hydroxylase (TH; marker of dopamine synthesis) and phosphorylated TH at Serine 30 and 40 (markers of enzyme activity), extracellular signal-regulated kinase 1 and 2 (ERK1 and ERK 2), and phosphorylated ERK1 and ERK2 (phosphorylates TH Serine 31; markers of synaptic plasticity), and markers of synaptic integrity, spinophilin and post-synaptic density protein 95 (roles in dopamine signaling and response to cocaine). Extended cocaine access increased non-glycosylated and glycosylated DAT in DLPFC and OFC. While no differences in TH expression were observed between groups for any of the regions, extended access induced significant elevations in pTHSer31 in all regions. In addition, a slight but significant reduction in phosphorylated pTHSer40 was found in the DLPFC. Phosphorylated ERK2 was increased in all regions; however, pERK1 was decreased in ACC and OFC but increased in DLPFC. PSD-95 was increased in the OFC but not in DLPFC or ACC. Furthermore, extended cocaine self-administration elicited significant increases in spinophilin protein expression in all regions. Results from the study provide insight into the biochemical alterations occurring in primate prefrontal cortex.
cocaine; dopamine; orbitofrontal cortex; anterior cingulate cortex; dorsolateral prefrontal cortex; rhesus monkey
Substance use disorders (SUDs) can be conceptualized as a form of risk-taking behavior with the potential for highly aversive outcomes such as health or legal problems. Risky decision-making likely draws upon several related brain processes involved in estimations of value and risk, executive control, and emotional processing. SUDs may result from a dysfunction in one or more of these cognitive processes.
We performed a systematic literature review of functional neuroimaging studies examining risk-related decision making in individuals with SUDs. A quantitative meta-analysis tool (GingerALE) and qualitative approach was used to summarize the imaging results.
Meta-analysis findings indicate that individuals with SUDs exhibit differences in neural activity relative to healthy controls during risk-taking in the anterior cingulate cortex, orbitofrontal cortex, dorsolateral prefrontal cortex, striatum, insula, and somatosensory cortex. In addition, a qualitative review of the literature suggests that individuals with SUDs may have altered function in the amygdala and ventromedial prefrontal cortex.
The neuroimaging literature reveals that several neural substrates involved in the computation of risk may function suboptimally in SUDs. Future research is warranted to elucidate which computational processes are affected, whether dysfunctional risk-related processing recovers with sobriety, and whether different drugs of abuse have specific effects on risk-taking.
Risk-taking; Drug Abuse; Addiction; Neuroimaging; Decision-making
Functional impairment of the orbital and medial prefrontal cortex underlies deficits in executive control that characterize addictive disorders, including alcohol addiction. Previous studies indicate that alcohol alters glutamate neurotransmission and one substrate of these effects may be through the reconfiguration of the subunits constituting ionotropic glutamate receptor (iGluR) complexes. Glutamatergic transmission is integral to cortico-cortical and cortico-subcortical communication and alcohol-induced changes in the abundance of the receptor subunits and/or their splice variants may result in critical functional impairments of prefrontal cortex in alcohol dependence. To this end, the effects of chronic ethanol self-administration on glutamate receptor ionotropic AMPA (GRIA) subunit variant and kainate (GRIK) subunit mRNA expression were studied in the orbitofrontal cortex (OFC), dorsolateral prefrontal cortex (DLPFC), and anterior cingulate cortex (ACC) of male cynomolgus monkeys. In DLPFC, total AMPA splice variant expression and total kainate receptor subunit expression were significantly decreased in alcohol drinking monkeys. Expression levels of GRIA3 flip and flop and GRIA4 flop mRNAs in this region were positively correlated with daily ethanol intake and blood ethanol concentrations (BEC) averaged over the 6 months prior to necropsy. In OFC, AMPA subunit splice variant expression was reduced in the alcohol treated group. GRIA2 flop mRNA levels in this region were positively correlated with daily ethanol intake and BEC averaged over the 6 months prior to necropsy. Results from these studies provide further evidence of transcriptional regulation of iGluR subunits in the primate brain following chronic alcohol self-administration. Additional studies examining the cellular localization of such effects in the framework of primate prefrontal cortical circuitry are warranted.
ethanol; AMPA; kainate; messenger RNA; prefrontal cortex; qPCR; primate
The process of decision making in humans and other animals is adaptive and can be tuned through experience so as to optimize the outcomes of their choices in a dynamic environment. Previous studies have demonstrated that the anterior cingulate cortex plays an important role in updating the animal’s behavioral strategies when the action-outcome contingencies change. Moreover, neurons in the anterior cingulate cortex often encode the signals related to expected or actual reward. We investigated whether reward-related activity in the anterior cingulate cortex is affected by the animal’s previous reward history. This was tested in rhesus monkeys trained to make binary choices in a computer-simulated competitive zero-sum game. The animal’s choice behavior was relatively close to the optimal strategy, but also revealed small but systematic biases that are consistent with the use of a reinforcement learning algorithm. In addition, the activity of neurons in the dorsal anterior cingulate cortex that was related to the reward received by the animal in a given trial was often modulated by the rewards in the previous trials. Some of these neurons encoded the rate of rewards in previous trials, whereas others displayed activity modulations more closely related to the reward prediction errors. By contrast, signals related to the animal’s choices were only weakly represented in this cortical area. These results suggest that neurons in the dorsal anterior cingulate cortex might be involved in the subjective evaluation of choice outcomes based on the animal’s reward history.
reinforcement learning; game theory; neuroeconomics; decision making; dopamine
It is generally assumed that choice between different actions reflects the difference between their action values yet little direct evidence confirming this assumption has been reported. Here we assess whether the brain calculates the absolute difference between action values or their relative advantage, that is, the probability that one action is better than the other alternatives. We use a two-armed bandit task during functional magnetic resonance imaging and modelled responses to determine both the size of the difference between action values (D) and the probability that one action value is better (P). The results show haemodynamic signals corresponding to P in right dorsolateral prefrontal cortex (dlPFC) together with evidence that these signals modulate motor cortex activity in an action-specific manner. We find no significant activity related to D. These findings demonstrate that a distinct neuronal population mediates action-value comparisons, and reveals how these comparisons are implemented to mediate value-based decision-making.
In humans, choice between actions depends on the ability to compare action–outcome values. Here, the authors show that action–outcome values are compared on the basis of the relative advantage of a particular action over alternative actions, which takes place in the right dorsolateral prefrontal cortex of the brain.
Monkeys adjust their behavior in response to outcomes that they have observed but not directly experienced, and single neurons within the anterior cingulate cortex respond to these fictive rewards they same way they respond to experienced rewards.
The neural mechanisms supporting the ability to recognize and respond to fictive outcomes, outcomes of actions that one has not taken, remain obscure. We hypothesized that neurons in anterior cingulate cortex (ACC), which monitors the consequences of actions and mediates subsequent changes in behavior, would respond to fictive reward information. We recorded responses of single neurons during performance of a choice task that provided information about the reward values of unchosen options. We found that ACC neurons signal fictive reward information, and use a coding scheme similar to that used to signal experienced outcomes. Thus, individual ACC neurons process both experienced and fictive rewards.