Reversal learning as a measure of impulsive and compulsive behavior in addictions

Orbitofrontal Cortex

If we restrict our description of the circuitry to that which supports two-stimuli odor or visual discrimination reversal learning, the literature is quite consistent across species. Mice (Bissonette et al., 2008), rats (Chudasama and Robbins, 2003, McAlonan and Brown, 2003) and monkeys (Jones and Mishkin, 1972, Dias et al., 1996, Izquierdo et al., 2004) with lesions that include the orbitofrontal cortex (OFC) exhibit normal acquisition of the initial discrimination but are impaired at reversal, often exhibiting perseverative responding to the previously rewarded stimulus (Rudebeck and Murray, 2008). Conversely, medial and dorsal subregions of the frontal cortex are not critical to discrimination reversal learning in mice, rats, or monkeys (Birrell and Brown, 2000, Boulougouris et al., 2007, Bissonette et al., 2008, Rudebeck et al., 2008). Instead, the medial wall of the frontal cortex in the rodent brain (to include prelimbic and infralimbic cortices) appears to be more important in task and strategy switching (Ragozzino et al., 1999a, Ragozzino et al., 1999b, Floresco et al., 2008, Ghods-Sharifi et al., 2008, Rich and Shapiro, 2009), which have been described as serving more of a working memory, action-monitoring function (Ragozzino et al., 1998, Delatour and Gisquet-Verrier, 1999, 2000, Gisquet-Verrier and Delatour, 2006). Similar results have been obtained in monkey studies (Dias et al., 1996, Rudebeck et al., 2008), with a recent report showing that a subregion within the OFC, Walker’s area 14, may be most critical in learning to inhibit responses to a previously rewarded stimulus (Rudebeck and Murray, 2011). Importantly, behavioral and neuroimaging studies in human subjects with OFC damage (Fellows and Farah, 2003, Hornak et al., 2004), patients diagnosed with OCD (Remijnse et al., 2006) and healthy controls (O’Doherty et al., 2001, Ghahremani et al., 2010), all show that this region is important for accurate performance across a variety of reversal learning paradigms. In sum, the functional localization of discrimination reversal learning within the frontal cortex is well-preserved across species (Chudasama, 2011).

Striatum

The idea that areas of the striatum may also mediate “frontocortical”-like processes was proposed and supported by empirical evidence as early as the late 1960s (Divac et al., 1967, Winocur and Mills, 1969, Winocur and Eskes, 1998). The anatomical interconnectivity (Fig. 2) may account for some functional overlap in reversal learning: lesions of the medial striatum, like damage limited to the OFC, produce perseverative responding on reversal learning in rats (Castane et al., 2010) and in marmoset monkeys (Clarke et al., 2008, Man et al., 2008). Additionally, mediodorsal thalamus, heavily interconnected with OFC, is a contributor to accurate performance during reversal learning (Chudasama et al., 2001), yet its circumscribed contribution has not been fully identified.

Medial temporal lobe structures

Severe discrimination and reversal learning impairments are found following rhinal cortex, specifically perirhinal cortex, probably due to this region’s essential role in object recognition and object identification (Murray and Richmond, 2001). Damage restricted to the hippocampus also results in object reversal learning impairments in monkeys (Murray et al., 1998); yet in rodents, hippocampal recruitment appears to depend on the spatial nature of the reversal task (Morellini et al., 2010). Certain regions of the medial temporal lobe, though substantially involved in discrimination learning and in forming stimulus-reward associations generally, are not critically important to reversal learning per se. For example, selective lesions of the amygdala do not disrupt reversal learning (Izquierdo and Murray, 2007, Stalnaker et al., 2007) and even facilitate reversal learning (Stalnaker et al., 2007) and instrumental extinction (Izquierdo and Murray, 2005). Thus, medial temporal lobe structures are critical to associative learning processes involving reward, yet frontocorticostriatal circuitry is most reliably implicated in support of reversal-specific learning.

Serotonin

Some of the first evidence linking serotonin mechanisms to discrimination reversal came from studies of the effects of ondansetron – a centrally active 5-HT3 receptor antagonist – on object discrimination acquisition and reversal in monkeys. In both marmosets and rhesus monkeys, ondansetron improved performance, but this effect was noted for both initial acquisition and reversal learning, meaning that the pharmacological effect could not be localized to the inhibitory control processes measured in reversal (Barnes et al., 1990, Domeney et al., 1991, Arnsten et al., 1997).

Reductions in serotonin concentrations produced by dietary deficiencies in tryptophan have largely found no effect of low indoleamine transmission on reversal learning (Murphy et al., 2002, Evers et al., 2005a, Evers et al., 2005b, van der Plasse and Feenstra, 2008). On the other hand, toxin-mediated depletions of serotonin (which potentially produce larger magnitude reductions in transmitter) have often been reported to affect reversal learning in a behaviorally selective manner. Focal destruction of serotonin terminals in the OFC impairs the ability to update behavior during reversal in marmosets (Clarke et al., 2004, Clarke et al., 2005, Clarke et al., 2007). Similar findings have been reported after cortical serotonin depletions in rats (Masaki et al., 2006, Lapiz-Bluhm et al., 2009) yet this manipulation appears to have no effect on reversal learning in mice, even though chronic fluoxetine treatment produces a reversal enhancement in this species (Brigman et al., 2010). Very little study has been focused on the receptor mechanisms within prefrontal cortex that mediate effects on reversal learning, though 5-HT2 subtype receptors appear to be plausible candidates (Boulougouris et al., 2008b, Boulougouris and Robbins, 2010).

Additional evidence relating serotonin to reversal learning derives from genetic findings in laboratory animals. In rhesus monkeys, variation in the non-coding regions of the gene encoding the serotonin transporter (that inactivates synaptic serotonin) is linked to reversal learning (Izquierdo et al., 2007, Vallender et al., 2009, Jedema et al., 2010); since the studies conducted so far linked independent, non-segregating variants with reversal learning, the relationship between this gene and reversal appears to be a robust finding. Experimental studies in mice and rats (involving genetic or pharmacological inhibition of the serotonin transporter) support these findings, as well (Brigman et al., 2010, Lapiz-Bluhm et al., 2009; Nonkes et al., 2011). The influence of genetic variation in the serotonin transporter on reversal learning probably depends on an interaction of factors, including stress (Graybeal et al. 2011).

Unfortunately, much remains unknown about the details of the control of reversal learning by central serotonin systems. The types of manipulations conducted so far (dietary and pharmacological depletions, reductions in reuptake) are not specific enough to uncover the cellular and molecular mechanisms that mediate these influences and that are relevant to treatment of disorders linked with reversal learning problems. These findings are summarized in Table 1.

Dopamine

Though depletion of dopamine in OFC does not impair reversal learning the way that serotonin depletion does (Clarke et al., 2007), there are clear data linking dopaminergic systems to reversal learning. A recent study found that depletion of dopamine, but not serotonin, in the medial caudate nucleus results in a non-perseverative impairment on reversal learning in marmosets (Clarke et al., 2011).

Pharmacological studies, on the other hand, have provided convincing evidence for dopaminergic modulation of reversal learning. Using an object discrimination task, Ridley and colleagues reported that both indirect dopamine agonists and dopamine receptor antagonists, d-amphetamine and haloperidol, respectively, produced behaviorally-specific problems with reversal (Ridley et al., 1981a, Ridley et al., 1981b). While both selectively impaired performance in reversal, amphetamine increased perseverative errors, and haloperidol produced “non-perseverative” responding. Haloperidol was found to block the effect of amphetamine (Ridley et al., 1981b), showing that amphetamine-induced perseveration was due to increased dopamine output produced by the drug (as opposed to an action on another monoamine system).

Studies of the effects of amphetamine, or its analogues, on reversal learning in rodent models have generated variable results, including impairments (Idris et al., 2005, McLean et al., 2010), no effect (Wilpizeski and Hamilton, 1964) or improvements (Kulig and Calhoun, 1972, Weiner and Feldon, 1986); the variability in types of reversal used (spatial vs. visual; appetitively-reinforced vs. escape-reinforced; Pavlovian vs. instrumental) and the inconsistency in doses and routes of administration make a direct resolution of these disparate findings challenging.

The fact that haloperidol blocked the effects of amphetamine, while producing an impairment itself (Ridley et al., 1981b, Idris et al., 2005), raises the hypothesis that over or under stimulation of dopamine D2-like receptor function causes impairments in reversal learning, and over the years, this hypothesis has received remarkable support. In rats, activation or antagonism of D2-like receptors (D2 and D3 receptors, in particular) affects reversal learning (Idris et al., 2005, Boulougouris et al., 2008a), and the particularly important role of the D2 receptor gene has been confirmed through the study of dopamine D2 receptor gene knockout mice (Kruzich and Grandy, 2004, Kruzich et al., 2006, De Steno and Schmauss, 2009). In non-human primates, activation or antagonism of D2 and D3 receptors disrupts reversal learning (Smith et al., 1999, Lee et al., 2007), with similar effects in human subjects (Mehta et al., 2001). It is important to note that not all these studies agree on the symmetrical effects of agonists and antagonists, with some showing effects of one, alone with no effects of the other; however, these discrepancies are almost certainly related to variation in doses administered and the difficulty of the reversal problem being solved. These studies do not imply that dopamine D1 receptors are unrelated to reversal learning; indeed, systemic treatment with the D1 agonist SKF81297 results in a selective early reversal learning impairments in mice, leaving visual discrimination learning unaffected (Izquierdo et al., 2006).

Beyond the behavioral pharmacological data, recent studies in a variety of species indicate that individual differences in reversal learning are related to D2-mediated dopamine transmission, with low receptor availability predicting poor reversal in otherwise normal mice (Laughlin et al., 2011), non-human primates (Groman et al., 2011) and humans (Jocham et al., 2009). Additionally, the reversal learning deficits in patients with psychostimulant addiction, a condition associated with low D2 receptor availability (Volkow et al., 2001, Lee et al., 2009), are reversed by administration of a D2-like receptor agonist (Ersche et al., 2011), suggesting that the treatment implications of this dopamine-mediated effect on reversal learning are strong. These findings are summarized in Table 2.

Together, these data collectively support the notion that dopamine D2 receptors are major players in coordinating effective behavioral flexibility during reversal learning and that low receptor complement/function is associated with poor inhibitory control in this task.

Noradrenaline

Increases in synaptic noradrenaline levels elicited by selective reuptake inhibitors or alpha-2 adrenergic autoreceptors are sufficient to trigger improvements in reversal learning (Lapiz and Morilak, 2006, Lapiz et al., 2007, Seu and Jentsch, 2009, Seu et al., 2009). Though it is tempting to speculate that this effect is mediated by increased noradrenaline output into OFC, this remains untested. Consequently, it remains possible that noradrenergic mechanisms in another cortical or subcortical region are mediating this effect or that these mechanisms are secondary to increases in dopamine output in the frontal cortex (Millan et al., 2000, Bymaster et al., 2002). This is an area worthy of much more systematic study.

Poor inhibitory control: Cause and consequence?

Addiction and cognitive rigidity are related in a complex circuitous relationship that is not yet fully understood (Schoenbaum and Shaham, 2008). The observation that substance abusers exhibit inhibitory control deficits such as increased perseveration in reversal learning (Ersche et al., 2008) and risky decision-making may indicate pre-morbid vulnerability factors for addiction, direct consequences of long-term drug intake, or a combination of these (Verdejo-Garcia et al., 2008). Animal studies have substantiated both views: that exposure to addictive drugs causes cognitive deficits (Jentsch et al., 1997, Jentsch et al., 2002, Shoblock et al., 2003, Schoenbaum et al., 2004, Kantak et al., 2005) and that individual variation in inhibitory control influences addiction vulnerability (Dalley et al., 2007, Belin et al., 2008, Diergaarde et al., 2008, Perry and Carroll, 2008).

Daily exposure to cocaine over a 2-week period produces persistent deficits in reversal learning (Jentsch et al., 2002). This is strong evidence for deficits of inhibitory control following prolonged stimulant exposure; a similar long-lasting impairment has been found in rodent models of cocaine or methamphetamine administration, using a range of reversal-like tasks (Shoblock et al., 2003, Schoenbaum et al., 2004, Kantak et al., 2005, Schoenbaum and Setlow, 2005). Additionally, even brief exposure to methamphetamine results in selective impairments on a wide range of reversal tasks in rats: spatial reversal (White et al., 2009), response reversal (Cheng et al., 2007), and discrimination reversal learning (Izquierdo et al., 2010). The learning impairment observed in the latter study was reversal-specific, leaving initial discrimination learning, as well as attentional set shifting, unaffected. Follow-up studies in our lab confirm that impaired reversal learning is just one example of cognitive rigidity after relatively brief exposures to methamphetamine that do not produce any significant, measured dopaminergic neurotoxicity (Kosheleff et al., 2011).

Rats that, in turn, exhibit poor inhibitory control (assessed prior to any experience with drug) have been shown to acquire cocaine or nicotine self-administration faster and to exhibit greater overall intake and diminished extinction of the drug-taking response (Dalley et al., 2007, Belin et al., 2008, Diergaarde et al., 2008). While the precise relationship between poor inhibitory control and addiction liability is still unknown, this difference in propensity may be due to the fact that impulsive individuals are more sensitive to the acute stimulus effects of the drugs (Perkins et al., 2008). Additionally, inhibiting prepotent responses may be a phenotype that contributes to a more rapid change in drug-taking behavior from a “goal-directed” pattern of use to a more compulsive pattern (Groman et al., 2008). This relationship has not yet been shown for reversal learning measures, specifically.

A putative mechanism for the loss of control of drug use involves frontocorticostriatal adaptations after protracted and perhaps even brief exposure to drug. For example, imaging studies in humans reveal an increasing involvement of ventral-to-dorsal striatum with increasing severity of stimulant craving and habit (Volkow et al., 2006). This may represent the neurophysiological correlate to increased automaticity and habitized behavior (Takahashi et al., 2007) measured in reversal learning. Plasticity within frontocorticostriatal circuitry resulting from drug exposure could contribute to the transition from recreational use to addiction (Everitt et al., 2007). With more striatal control in addiction, there is a concomitant decrease in frontocortical (e.g. OFC) involvement in inhibitory control, and consequently decreased control over use of the drug.

The neurochemical mechanisms by which pre-morbid differences in inhibitory control function affect addiction liability are unknown, but recent studies suggest that low D2-like receptor function is a potential mechanistic determinant (Dalley et al., 2007, Zald et al., 2008, Ersche et al., 2011, Groman and Jentsch, in press). These results suggest that low D2-like receptor function is a molecular convergence point for pre-morbid genetic factors influencing impulsivity and for chronic stimulant-drug induced deficits of inhibitory control. This underscores the possibility that pre-morbid differences in inhibitory control affect vulnerability, and that addiction further impairs inhibitory control, as well as that a common molecular alteration may mediate the relationship between both associations.