Surprise-induced enhancements in the associability of Pavlovian cues facilitate learning across behavior systems

Associability enhancements across behavior systems 1 Brief communication 1 Surprise-induced enhancements in the associability of Pavlovian 2 cues facilitate learning across behavior systems 3 4 Inmaculada Márquez1, Gabriel Loewinger2, Juan Pedro Vargas1, Juan Carlos 5 López1, Estrella Díaz1*, Guillem R. Esber3* 6 7 1Animal Behavior and Neuroscience Lab, Department of Experimental Psychology, Faculty of 8 Psychology, University of Seville, 41018, Spain 9 2 Department of Biostatistics, Harvard School of Public Health, Boston, USA. 10 3Brooklyn College, City University of New York, USA 11 IM’s current address: Centro de Investigación en Conducta y Cognición Comparada, 12 Universidad de Guadalajara, Centro Universitario de la Ciénega, Av. Universidad 1115, Col. 13 Lindavista, Ocotlán, Jalisco, 47820, México 14 *These authors should be considered joint senior authors.15 Corresponding author: Guillem R. Esber ([email protected]) 16 Keywords: associability, salience, attention, behavior systems, surprise, uncertainty, Pearce-17 Hall model, counterconditioning 18 19 Acknowledgements 20 This research was supported by Spain’s Ministry of Science, Innovation and Universities grant PID2019-21 110739GB-I00 (ED and JPV) and by the National Institute on Drug Abuse grants 5R00DA036561 and 22 1R15DA051795 (GE). We declare no conflict of interest, financial or otherwise, related to the current 23 research. We thank Felipe Cabrera for helpful comments on a previous version of the MS. 24 25 Author contributions 26 Inmaculada Marquez: Methodology, Investigation, Validation, Data curation, Visualization, Writing- 27 Original draft preparation; Gabriel Loewinger: Formal analysis; Juan Pedro Vargas: Project 28 Administration, Funding acquisition; Juan Carlos López: Conceptualization, Project Administration, 29 Funding acquisition, Software, Visualization, Methodology, Supervision, Writing- Reviewing and Editing; 30 Estrella Díaz: Conceptualization, Project Administration, Funding acquisition, Methodology, Supervision, 31 Writing- Reviewing and Editing; Guillem R. Esber: Conceptualization, Writing- Original draft preparation 32 33 Author ORCIDs 34 Inmaculada Márquez: https://orcid.org/0000-0001-7914-6907 35 Gabriel Loewinger: https://orcid.org/0000-0002-0755-8520 36 Juan Pedro Vargas: https://orcid.org/0000-0002-4358-5737 37 Juan Carlos López: https://orcid.org/0000-0001-8106-5006 38 Estrella Díaz: https://orcid.org/0000-0001-9465-9460 39 Guillem R. Esber: https://orcid.org/0000-0001-8874-4255 564 40 ©2022 American Psychological Association. This paper is not the copy of record and may not exactly replicate the authoritative document published in the APA journal. The final article is available, upon publication, at: https://doi.org/10.1037/bne0000505 Associability enhancements across behavior systems 2 Abstract 41 Surprising violations of outcome expectancies have long been known to enhance the 42 associability of Pavlovian cues; that is, the rate at which the cue enters into further associations. 43 The adaptive value of such enhancements resides in promoting new learning in the face of 44 uncertainty. However, it is unclear whether associability enhancements reflect increased 45 associative plasticity within a particular behavior system, or whether they can facilitate learning 46 between a cue and any arbitrary outcome, as suggested by attentional models of conditioning. 47 Here, we show evidence consistent with the latter hypothesis. Violating the outcome 48 expectancies generated by a cue in an appetitive setting (feeding behavior system) facilitated 49 subsequent learning about the cue in an aversive setting (defense behavior system). In addition 50 to shedding light on the nature of associability enhancements, our findings offer the 51 neuroscientist a behavioral tool to dissociate their neural substrates from those of other, 52 behavior system- or valence-specific changes. Moreover, our results present an opportunity to 53 utilize associability enhancements to the advantage of counterconditioning procedures in 54 therapeutic contexts. 55 56 57 58 59 60 61 62 Associability enhancements across behavior systems 3 63 In an ever-changing world, brain mechanisms have evolved to modulate the associability of 64 Pavlovian cues in order to meet the learning demands of the environment (e.g., Mitchell & Le 65 Pelley, 2010). One form of modulation is captured by the so-called uncertainty principle, 66 according to which a cue’s associability increases whenever its consequences are surprising 67 (Pearce & Hall, 1980; Pearce et al., 1982). In support of this notion, cues that predict an 68 outcome inconsistently (i.e., partial reinforcement) are subsequently learned about more rapidly 69 than cues that predict the outcome consistently (i.e., continuous reinforcement; Haselgrove et 70 al., 2010, Collins & Pearce, 1985). Similarly, repeated confirmation of outcome expectancies 71 decreases a cue’s associability (Pearce & Hall, 1979; Griffiths et al., 2011; Mackintosh & Turner, 72 1971), whereas a sudden violation of those expectancies restores it (Hall & Pearce, 1982; 73 Dickinson et al., 1976; Holland, 1984). Surprise-induced associability enhancements have been 74 documented both in appetitive (e.g., Holland, 1984) and aversive (e.g., Dickinson et al., 1976) 75 procedures as well as across phylogenetically distant species (e.g., rats: Kaye & Pearce, 1984; 76 pigeons: Collins & Pearce, 1985; humans: Hogarth et al., 2008; Russo et al., 2019), suggesting 77 they might constitute a widespread, if not universal property of learning systems. 78 While these findings have fostered important neurobiological discoveries (reviewed in: 79 Holland & Schiffino, 2016; Roesch et al., 2012; Holland & Maddux, 2010) and theoretical 80 developments (Dayan et al., 2000; Le Pelley, 2004; Courville et al., 2006; Pearce & Mackintosh, 81 2010; Esber & Haselgrove, 2011), the nature and scope of associability enhancements remains 82 poorly understood. On the one hand, such enhancements might reflect a state of heightened 83 associative plasticity involving a specific association or behavior system (e.g., feeding, mating, 84 defense, etc.; Timberlake, 1993; 1994; Cabrera et al., 2019). Such a labile state would facilitate 85 the updating of associative representations involving the cue and outcomes within that behavior 86 system. It follows from this view that a surprise-induced associability enhancement by a food-87 Associability enhancements across behavior systems 4 predictive cue, for instance, should translate as more rapid learning between that cue and food-88 related outcomes (including food omission), but not necessarily outcomes related to other 89 behavior systems, such as the presence of a sexual partner (Domjan & Gutiérrez, 2019). On the 90 other hand, associability enhancements might arise from increased attentional processing of the 91 cue, as assumed by attentional models of associative learning (Mackintosh, 1975; Pearce & 92 Hall, 1980). If so, those enhancements should manifest as faster learning regardless of the 93 nature of the outcome and the behavior system engaged. Since studies on associability 94 modulation have traditionally employed a single reinforcer or reinforcer class (thus engaging a 95 single behavior system), this fundamental issue remains unresolved. 96 To decide between these alternatives, we violated the outcome expectancies generated 97 by a cue in an appetitive setting (feeding system) and tested the associability of the cue in an 98 aversive setting (defense system). To achieve this, we modified a serial prediction task (Wilson 99 et al., 1992) that has been extensively used to investigate the neural substrates of surprise-100 induced associability changes in rats (e.g., Holland & Gallagher, 1993, 2006; Chiba et al., 1995; 101 Bucci & MacLeod, 2007; Esber et al., 2015). In the original task, a light stimulus is initially 102 followed by a tone that is partially reinforced with food (LTfood, LTnothing). After 103 developing an expectancy of the tone during light presentations, animals in the Surprise 104 condition experience the unexpected omission of the tone on nonreinforced trials (LTfood, 105 Lnothing), whereas control subjects continue to receive the initial training (LTfood, 106 LTnothing). The omission of the tone is intended to increase the associability of the light 107 without fundamentally changing its predictive or incentive properties (which, if anything, should 108 decrease during tone omission). This increase in associability is typically revealed in a 109 subsequent test in which the light is paired with food (Lfood) and more rapid learning is 110 observed in Surprise than control animals. The fact that greater associability can be detected 111 Associability enhancements across behavior systems 5 days after the end of the Surprise phase rules out transient increases in arousal and suggests 112 more enduring changes in the mnemonic representation of the cue. 113 Here, we tested the associability of the light by pairing it with foot shock in order to 114 determine whether associability increases can be expressed across behavior systems. Our 115 results disconfirmed the hypothesis that associability enhancements reflect heightened plasticity 116 within a particular behavior system. Rather, they are consistent with the view that such 117 enhancements result from increased attentional processing of the cue (Mackintosh, 1975; 118 Pearce & Hall, 1980; Pearce et al., 1982). Our procedure will provide neuroscientists with a tool 119 to dissociate the neural bases of associability changes from those of other, behavior system- or 120 valence-specific changes that a cue representation may undergo during learning. In the clinical 121 setting, our findings suggest the possibility of administering associability-boosting treatments to 122 bolster counterconditioning-based interventions. 123 Methods 124 Subjects. Thirty-four experimentally naïve, male Wistar rats were used in the study, run in three 125 cohorts that included animals from all three groups. They were obtained from the Animal 126 Production and Experimentation Center at the University of Seville. Upon arrival, rats were 127 acclimated to the colony room for two weeks with free access to food and water. The colony 128 room was maintained on a 14:10 light/dark cycle schedule at a constant temperature of 21ºC. 129 Rats were housed individually in standard clear-plastic tubs (35×20×20 cm) with woodchip 130 bedding. At the start of the experiment, they were 7–9 weeks old and weighed 230–280 g. One 131 week prior to the beginning of the study, they were food deprived by progressively restricting 132 their diet until they reached 90% of their original body weight and maintained at that weight 133 thereafter. Once training began, they were fed a restricted amount immediately after the 134 experimental sessions. They had free access to water in their home chambers at all times. All 135 procedures and methods were carried out in accordance with the European Directive 136 Associability enhancements across behavior systems 6 2010/69/EU for the maintenance and use of laboratory animals and following Spanish 137 regulations (R.D. 53/02013). The protocol was approved by the Ethics Committee for Animal 138 Research of the University of Seville (Protocol Number: CEEA-US2015-27/4). 139 140 Apparatus. Rats were trained in four identical, modular conditioning chambers (31.8 × 25.4 × 141 34.3 cm, Med Associates, Inc.) enclosed in a ventilated light- and sound-attenuating cubicle 142 (63.5×41.9×49.4 cm, Med Associates, Inc.). An extractor fan was fitted on the right wall of the 143 cubicle and produced a ~60-dB background noise in the conditioning chamber. The side walls 144 of the conditioning chambers were made of aluminum, while the front and back walls and the 145 roof were made of transparent acrylic plastic. The floor consisted of 0.4 mm-diameter steel bars 146 oriented perpendicular to the front wall and spaced 1.4 cm apart as measured from their 147 centers. This floor grid was connected to a shock dispenser capable of delivering a foot shock 148 unconditioned stimulus (US). Each conditioning chamber housed a 6-W white jewel lamp 149 mounted 20 cm above the floor on the center panel of the left wall. Illumination of this lamp 150 provided the visual stimulus used during behavioral training. A speaker was mounted 20 cm 151 above the floor on the left panel of the left wall. This speaker was connected to a tone generator 152 set to deliver a 1500-Hz, 80-dB tone which served as the auditory stimulus used during training. 153 Each chamber also housed a recessed food cup located 2 cm above the floor on the center 154 panel of the right wall. This food cup was equipped with an infrared sensor for detecting head 155 entries and connected to a pellet dispenser capable of delivering 45-g sucrose pellets (DietTM; 156 Mlab Rodent Tablet-45mg; St Andrews University). The chambers remained dark throughout 157 the experimental session except during presentations of the light stimulus. In the same 158 experimental room was a computer running Med PC IV software (Med Associates Inc., St. 159 Albans, VT, USA) on Windows OS which controlled and automatically recorded all experimental 160 events via a Fader Control Interface. 161 162 Associability enhancements across behavior systems 7 Behavioral Procedure. Rats initially received a single session of magazine training in which a 163 pellet was delivered in the food cup once every minute for a total of 30 minutes. They were then 164 randomly assigned to three groups (Figure 1, table). In the first, Appetitive serial conditioning 165 phase, the No-surprise and Surprise groups received Pavlovian magazine-approach training 166 with a serial compound consisting of a 10-s light immediately followed by a 10-s tone. On half 167 the trials, two pellets were delivered immediately after the termination of the tone 168 (lighttonefood, lighttonenothing). This training was intended to establish the light as a 169 predictor of the tone. A Naïve group also received partial reinforcement training with the tone, 170 but the latter was not preceded by the light (tonefood, tonenothing). In each session, 10 171 trials were presented in pseudorandom order (reinforced and nonreinforced), with the constraint 172 that no more than 2 reinforced trials could occur in succession. The mean intertrial interval (ITI) 173 was 300 s. In this and the remainder of the phases, the total number of magazine head-entries 174 during the cues was taken as a measure of appetitive conditioning. That is, we summed the 175 number of head entries for each animal and for each cue across all trials in a session, and 176 calculated the group means based on those sums. Training continued for 10 sessions 177 conducted over a period of 5 days, with two daily sessions run at 8 am and 3 pm. 178 In the next, Surprise phase, the No-surprise and Naïve groups continued to receive the 179 same training for an additional 4 sessions (conducted over 2 days as in the first phase). In the 180 Surprise group, however, the tone was omitted on nonreinforced trials (lighttonefood, 181 lightnothing) in order to boost the associability of the light (Wilson et al., 1992). All other 182 procedural details remained the same in this phase. 183 On the next day, at 9 am, all groups received a threat conditioning session in which a 184 single presentation of the light was followed by a 0.25-mA foot shock (Lshock). We arrived at 185 the use of a single 0.25 mA shock after piloting our experimental design with two 0.5-mA and 186 one 0.5-mA shocks and finding a floor effect; that is, almost complete suppression during the 187 final test in all three groups. This single threat conditioning trial was preceded and followed by a 188 Associability enhancements across behavior systems 8 300-s period. No responses were recorded on this session. Later in the day, at 4 pm, rats were 189 placed back in the conditioning chambers to receive an appetitive session consisting of 5 190 reinforced trials with the tone. The mean ITI was 300 s. The purpose of the latter session was to 191 extinguish contextual threat conditioning and provide a baseline of magazine approach to the 192 tone across the groups ahead of the final test. 193 On the following day, at 9 am, all groups received a suppression test consisting of 4 194 trials with the light and the tone presented simultaneously and reinforced with the delivery of two 195 pellets (LTfood). The purpose of this test was to measure threat conditioning to the light by 196 assessing the extent to which it was capable of suppressing magazine approach during the tone 197 relative to the tone baseline taken on the previous day. If the unexpected omission of the tone in 198 the Surprise group enhances the light’s associability, and if associability changes can cross 199 behavior systems, then greater suppression of responding to the tone should be observed in 200 that group relative to the No-surprise group. The Naïve group provided a positive control for 201 associability since those animals experienced the light as a novel stimulus on the threat 202 conditioning session. Thus, we expected threat conditioning to be the strongest in Naïve rats. 203 204 Statistical analysis. Analyses were conducted in R version 3.6.1. Generalized Linear Models 205 (GLMs) were conducted using the stats package, Generalized Linear Mixed Models (GLMMs) 206 were conducted using lme4 package. To assess magazine approach performance in the 207 Appetitive Serial Conditioning and Surprise phases, we collapsed (summed responses) across 208 the last two sessions of each phase in order to probe asymptotic behavior. Before running any 209 statistical inference, we selected to proceed with this contrast (the sum of the last two sessions) 210 since the progression of responding across sessions was not of interest. We conducted all 211 analyses with a single Generalized Linear Mixed Effects Model, adopting a Poisson as the 212 conditional distribution of our outcome given the random effects and the covariates. We 213 included a random intercept for each rat. We proceeded with analyses that adopted a count 214 Associability enhancements across behavior systems 9 distribution for the outcome because all responses were head entries and thus treating the 215 outcome as continuous (e.g., using ANOVA, repeated measures ANOVAs or t-tests) would not 216 yield valid inference given the sample sizes in this study. For the Surprise phase, animals in the 217 Surprise group received 5 trials in a session, whereas animals in the No-Surprise and Naïve 218 groups received 10. To compare rates of responding between conditions with different numbers 219 of trials, we included in the model an offset of the log number of trials that each animal received. 220 For all statistical analyses in which post-hocs were necessary, we used Bonferroni corrections 221 to account for multiple comparisons. Post-hocs analyses were conducted with the glht function 222 in the multicomp package in R. All statistical tables are shown in the Supplementary Materials. 223 Data as well as code to reproduce statistical analyses and tables are available at the github 224 repository: https://github.com/gloewing/marquez-et-al-2021. 225 To assess magazine approach performance on the Tone baseline and Suppression test 226 phases, we proceeded by collapsing across trials. We opted not to conduct a repeated 227 measures analysis and no analyses of that kind were ever inspected. This was motivated by the 228 fact that the progression of responding across trials within the test day was not of interest and 229 thus the associated loss in statistical power from the increase in parameters we would need to 230 estimate was not justifiable. The temporal nature of the data (i.e., the trial-specific structure) was 231 a nuisance needed to probe the impact of the behavioral task, but did not provide any 232 meaningful or interpretable information. Before conducting any statistical inference, all analyses 233 were planned to avoid having to conduct any adjustment for multiple comparisons and to ensure 234 analysis results were not selected to maximize statistical significance. Any comparisons of 235 models were conducted without viewing p-values, confidence intervals or otherwise. Moreover, 236 models were parameterized to provide the comparisons/contrasts of interest and thus no post-237 hocs were necessary. As such, Naive - No-surprise comparisons were not conducted. To 238 assess whether there were differences in observed rates of head entries across the entire test 239 day between the three groups, we conducted a GLM adjusting for baseline responding. 240 Associability enhancements across behavior systems 16 For many animals, foraging for food implies increasing their exposure to predators, making it 342 essential to simultaneously attend to signals for food and threat. Crucially, preys and predators 343 may share common predictive cues (e.g., a glimpse of a moving object, a rustle in the 344 undergrowth), and thus it makes adaptive sense for associability increases to food cues to also 345 benefit learning in connection with threat. It remains to be established, however, whether 346 associability increases occurring within the defensive system can in turn be expressed in the 347 feeding system, or indeed whether associability changes can universally transfer across any 348 arbitrary pair of behavior systems. Such is of course the prediction of attentional theories of 349 associative learning (e.g., Pearce & Hall, 1980; Mackintosh, 1975; Esber & Haselgrove, 2011). 350 From this perspective, associability increments reflect greater attentional processing of the cue 351 rather than enhanced associative plasticity within a particular behavior system. This assumption 352 is consistent with evidence that the same uncertainty conditions that facilitate learning also 353 promote stronger overt attentional responses to the cue (Kaye & Pearce, 1984; Swan & Pearce, 354 1988; Beesley et al., 2015; Luque et al., 2017; Easdale et al., 2019). 355 Critically, our assertion that associability enhancements can be expressed across 356 behavior systems hinges on the assumption that the light—the target cue in the current study—357 was able to gain access to both the feeding and defense systems. One question raised by the 358 low level of magazine approach evoked by the light in the Surprise and No-surprise groups 359 during the first two phases of the study is whether this stimulus was capable of engaging the 360 feeding system at all. That it did so is suggested by the greater level of responding to the tone in 361 these groups relative to the Naïve group, which, as mentioned above, is readily explained if the 362 light alerted the rats of the imminence of the tone and prepared them to respond at the 363 magazine. A related question is whether the greater suppression of magazine activity observed 364 at test in Surprise than No-surprise animals truly reflects a stronger activation of the defense 365 system by the light (e.g., more freezing) or simply its greater proclivity to elicit orienting 366 Associability enhancements across behavior systems 17 responses that compete with magazine approach. While our study does not directly address this 367 possibility, it should be noted that, by the same token, one would expect orienting responses to 368 the light to increase similarly in the Surprise group of a standard serial prediction task, where 369 the associability of the light is tested by directly pairing it with food. If so, those orienting 370 responses should be expected to compete with magazine approach there as well, thus 371 hindering rather than facilitating conditioning, which is of course the opposite result to that 372 typically observed. 373 A competing explanation for the greater suppression of magazine activity we observed 374 at test in Surprise relative to No-surprise animals is that the omission of the tone in the Surprise 375 phase may have extinguished some of the positive valence of the light rather than enhancing its 376 associability. Compounded with threat conditioning, this reduced positive valence would, 377 according to this account, produce less magazine approach at test relative to No-surprise 378 animals. The issue with this interpretation is that it predicts a drop in responding to the light (i.e., 379 extinction) in Surprise animals during the Surprise phase, which was not observed. Given the 380 low level of responding to the light (a serially conditioned cue), it is possible that such extinction 381 was obscured by a floor effect. We regard it as more plausible, however, that the unaltered 382 relationship between light and food as well as the remaining tone presentations in the Surprise 383 group prevented a loss of positive valence substantial enough to produce a sizeable decrement 384 at test. 385 Due to various advantages, the standard serial prediction task has been extensively 386 used to characterize the neural substrates of surprise-induced associability enhancements. One 387 such advantage is that it permits decoupling the encoding of associability increases at the time 388 of surprise induction from the expression of those increases at the time of learning. This 389 advantage has permitted the discovery, for instance, that the central nucleus of the amygdala 390 (CeA; Holland & Gallagher, 2006) and the substantia nigra pars compacta (SNc; Lee et al., 391 Associability enhancements across behavior systems 18 2008) are critical for the encoding, but not the expression of associability increases, although 392 this has only been demonstrated in appetitive conditioning with food. On the other hand, the 393 substantia innominata/nucleus basalis magnocellularis (SI/nBM; Holland & Gallagher, 2006), the 394 secondary visual cortex (V2; Schiffino & Holland, 2016), and the dorsolateral striatum (DLS; 395 Asem et al., 2015) are necessary for the expression, but not the encoding of associability 396 increases. Such associability expression, however, has only been tested within the same 397 behavior system (feeding), and thus it is unclear whether these regions would also be 398 necessary in the current version of the task. Interestingly, the posterior parietal cortex (PPC), 399 which has long been implicated in attention in humans and non-human primates (e.g., Mesulam, 400 1981; Posner & Petersen, 1990; Desimone & Duncan, 1995; Corbetta & Shulman, 2002) is so 401 far the only region identified as being critical to the encoding, consolidation and expression of 402 associability enhancements (Schiffino et al., 2014), suggesting it might constitute a locus for 403 storing the cue-specific associability memory. Whether this mnemonic representation in PPC is 404 fully detached from the motivational, emotional and behavior system-specific properties of the 405 cue (i.e., whether it provides a neural substrate for the results observed here) remains to be 406 established. Once again, the current procedure should help make this determination. 407 The present findings may also carry clinical significance. For instance, they suggest the 408 possibility of expediting counterconditioning procedures by coupling them with associability-409 boosting manipulations (Keller et al., 2020). In behavioral therapy, counterconditioning refers to 410 a collection of procedures that seek to modify maladaptive behaviors by associating their 411 triggering events with an outcome of the opposite affective valence (Keller et al., 2020; Konorski 412 & Szwejkowska, 1956). Recent studies show that counterconditioning therapies have greater 413 efficacy (Engelhard et al., 2014; Kerkhof et al., 2011; Raes & De Raedt, 2012) and resistance to 414 relapse (Kang et al., 2018) than exposure therapies based on extinction procedures. This 415 relative advantage is thought to derive from the greater evaluative learning that takes place 416 when the triggering stimulus is experienced not merely in the absence of its associated outcome 417 Associability enhancements across behavior systems 19 (as in extinction), but in the presence of another of opposite affective sign. On the downside, 418 however, counterconditioning typically requires more training than de novo acquisition or 419 extinction (Scavio & Gormezano, 1980; Peck & Bouton, 1990; Bouton & Peck, 1992), and, to 420 that extent, it could benefit from prior manipulations that enhance the associability of the stimuli 421 and responses involved. The induction of surprise in a manner similar to that used here might 422 provide one such manipulation. While our findings remain to be extended and replicated in 423 humans, they point in a promising direction for future research. 424 References 425 Asem, J. S. A., Schiffino, F. L., Holland, P. C. (2015). Dorsolateral striatum is critical for the 426 expression of surprise-induced enhancements in cue associability. European Journal of 427 Neuroscience, 42, 2203–2213. https://doi.org/10.1111/ejn.13001 428 Beesley, T., Nguyen, K. P., Pearson, D., & Le Pelley, M. E. (2015). Uncertainty and 429 predictiveness determine attention to cues during human associative 430 learning. Quarterly Journal of Experimental Psychology, 68(11), 2175-2199. 431 https://doi.org/10.1080/17470218.2015.1009919 432 Bouton, M.E., Peck, C.A. (1992) Spontaneous recovery in cross-motivational transfer 433 (counterconditioning). Animal Learning & Behavior 20, 313–321. 434 https://doi.org/10.3758/BF03197954 435 Bucci, D. J., & MacLeod, J. E. (2007). Changes in neural activity associated with a surprising 436 change in the predictive validity of a conditioned stimulus. European Journal of 437 Neuroscience, 26(9), 2669–2676. https://doi.org/10.1111/j.1460-9568.2007.05902.x 438 Cabrera, F., Jiménez, Á. A., & Covarrubias, P. (2019). Timberlake’s behavior systems: A 439 paradigm shift toward an ecological approach. Behavioural Processes, 167, Article 440 103892. https://doi.org/10.1016/j.beproc.2019.103892 441 Associability enhancements across behavior systems 20 Chiba, A. A., Bucci, D. J., Holland, P. C., Gallagher, M. (1995) Basal forebrain cholinergic 442 lesions disrupt increments but not decrements in conditioned stimulus processing. J 443 Neurosci 15:7315–7322. https://doi.org/10.1523/JNEUROSCI.15-11-07315.1995 444 Collins, L., & Pearce, J. M. (1985). Predictive accuracy and the effects of partial reinforcement 445 on serial autoshaping. Journal of Experimental Psychology: Animal Behavior 446 Processes, 11(4), 548–564. https://doi.org/10.1037/0097-7403.11.4.548 447 Corbetta, M., & Shulman, G. L. (2002). Control of goal-directed and stimulus-driven attention in 448 the brain. Nature reviews. Neuroscience, 3(3), 201–215. https://doi.org/10.1038/nrn755 449 Courville, A. C., Daw, N. D., & Touretzky, D. S. (2006). Bayesian theories of conditioning in a 450 changing world. Trends in Cognitive Sciences, 10(7), 294-300. 451 https://doi.org/10.1016/j.tics.2006.05.004 452 Dayan, P., Kakade, S., & Montague, P. R. (2000). Learning and selective attention. Nature 453 neuroscience, 3 Suppl, 1218–1223. https://doi.org/10.1038/81504 454 Desimone, R., & Duncan, J. (1995). Neural mechanisms of selective visual attention. Annual 455 review of neuroscience, 18, 193–222. 456 https://doi.org/10.1146/annurev.ne.18.030195.001205 457 Dickinson, A., Hall, G., & Mackintosh, N. J. (1976). Surprise and the attenuation of 458 blocking. Journal of Experimental Psychology: Animal Behavior Processes, 2(4), 459 313322. https://doi.org/10.1037/0097-7403.2.4.313 460 Domjan, M., & Gutiérrez, G. (2019). The behavior system for sexual learning. Behavioural 461 processes, 162, 184-196. https://doi.org/10.1016/j.beproc.2019.01.013 462 Easdale, L. C., Le Pelley, M. E., & Beesley, T. (2019). The onset of uncertainty facilitates the 463 learning of new associations by increasing attention to cues. Quarterly journal of 464 experimental psychology (2006), 72(2), 193–208. 465 https://doi.org/10.1080/17470218.2017.1363257 466 Associability enhancements across behavior systems 21 Engelhard, I. M., Leer, A., Lange, E., Olatunji, B. O. (2014) Shaking That Icky Feeling: Effects of 467 Extinction and Counterconditioning on Disgust-Related Evaluative Learning. Behavior 468 Therapy, Vol 45, (5), 708-719. https://doi.org/10.1016/j.beth.2014.04.003. 469 Esber, G. R., Haselgrove, M. (2011) Reconciling the influence of predictiveness and uncertainty 470 on stimulus salience: a model of attention in associative learning. Proceedings of the 471 Royal Society B: Biological Sciences 278(1718):2553-61. 472 https://doi.org/10.1098/rspb.2011.0836 473 Esber, G. R., Torres-Tristani, K., Holland, P. C. (2015). Amygdalo-striatal interaction in the 474 enhancement of stimulus salience in associative learning. Behav Neurosci.;129(2):87-475 95. http://dx.doi.org/10.1037/bne0000041 476 Griffiths, O., Johnson, A. M., & Mitchell, C. J. (2011). Negative Transfer in Human Associative 477 Learning. Psychological Science, 22(9), 1198–1204. 478 https://doi.org/10.1177/0956797611419305 479 Hall, G., & Pearce, J. M. (1982). Restoring the Associability of a Pre-Exposed CS by a 480 Surprising Event. The Quarterly Journal of Experimental Psychology Section B, 34(3b), 481 127–140. https://doi.org/10.1080/14640748208400881 482 Haselgrove, M., Esber, G. R., Pearce, J. M., & Jones, P. M. (2010). Two kinds of attention in 483 Pavlovian conditioning: Evidence for a hybrid model of learning. Journal of 484 Experimental Psychology: Animal Behavior Processes, 36(4), 456–470. 485 https://doi.org/10.1037/a0018528 486 Hogarth, L., Dickinson, A., Janowski, M., Nikitina, A., & Duka, T. (2008). The role of attentional 487 bias in mediating human drug-seeking behaviour. Psychopharmacology, 201(1), 29–488 41. https://doi.org/10.1007/s00213-008-1244-2 489 Holland, P. C. (1984). Unblocking in Pavlovian appetitive conditioning. Journal of Experimental 490 Psychology: Animal Behavior Processes, 10(4), 476–497. https://doi.org/10.1037/0097-491 7403.10.4.476 492 Associability enhancements across behavior systems 22 Holland, P. C., & Gallagher, M. (1993). Amygdala central nucleus lesions disrupt increments, 493 but not decrements, in conditioned stimulus processing. Behavioral 494 neuroscience, 107(2), 246–253. https://doi.org/10.1037//0735-7044.107.2.246 495 Holland, P. C., & Gallagher, M. (2006). Different roles for amygdala central nucleus and 496 substantia innominata in the surprise-induced enhancement of learning. The Journal of 497 neuroscience : the official journal of the Society for Neuroscience, 26(14), 3791–3797. 498 https://doi.org/10.1523/JNEUROSCI.0390-06.2006. 499 Holland PC, Maddux J-M. (2010). Brain systems of attention in associative learning. In: Mitchell 500 CJ, LePelley ME, editors. Attention and learning. Oxford University Press; Oxford; pp. 501 305–349 502 Holland, P. C., Schiffino, F. L. (2016) Mini-review: Prediction errors, attention and associative 503 learning. Neurobiol Learn Mem.;131:207-15. https://doi.org/10.1016/j.nlm.2016.02.014 504 Kang, S., Vervliet, B., Engelhard, I. M., van Dis, E., & Hagenaars, M. A. (2018). Reduced return 505 of threat expectancy after counterconditioning verus extinction. Behaviour research and 506 therapy, 108, 78–84. https://doi.org/10.1016/j.brat.2018.06.009 507 Kaye, H., & Pearce, J. M. (1984). The strength of the orienting response during Pavlovian 508 conditioning. Journal of Experimental Psychology: Animal Behavior Processes, 10(1), 509 90–109. https://doi.org/10.1037/0097-7403.10.1.90 510 Keller, N. E.; Hennings, A. C.; Dunsmoor, J. E. (2020) Behavioral and neural processes in 511 counterconditioning: Past and future directions. Behaviour Research and Therapy, Vol 512 125, 103532. https://doi.org/10.1016/j.brat.2019.103532. 513 Kerkhof, I., Vansteenwegen, D., Baeyens, F., Hermans, D. (2011) Counterconditioning: An 514 effective technique for changing conditioned preferences. Experimental 515 Psychology, 58, pp. 31-38, https://doi.org/10.1027/1618-3169/a000063 516 Konorski, J., Szwejkowska, G. (1956) Reciprocal transformations of heterogeneous conditioned 517 reflexes. Acta Biologica Experimentalis, 11, 141-165. 518 Associability enhancements across behavior systems 23 Le Pelley, M. E. (2004). The role of associative history in models of associative learning: A 519 selective review and a hybrid model. The Quarterly Journal of Experimental 520 Psychology B: Comparative and Physiological Psychology, 57B(3), 193–243. 521 https://doi.org/10.1080/02724990344000141 522 Lee, H. J., Youn, J. M. O. M. J., Gallagher, M., Holland, P. C. (2008) Temporally-limited role of 523 substantia nigra-central amygdala connections in surprise-induced enhancement of 524 learning. Eur J Neurosci.; 27(11): 3043–3049. https://doi.org/10.1111/j.1460-525 9568.2008.06272.x 526 Luque, D., Vadillo, M. A., Le Pelley, M. E., & Beesley, T. (2017). Prediction and uncertainty in 527 associative learning: Examining controlled and automatic components of learned 528 attentional biases. Quarterly Journal of Experimental Psychology, 70(8), 1485-1503. 529 https://doi.org/10.1080/17470218.2016.1188407 530 Mackintosh, N. J. (1975). A Theory of Attention: Variations in the Associability of stimuli with 531 Reinforcement. Psychological Review, Vol. 82, No. 4, 276-298. 532 https://doi.org/10.1037/h0076778 533 Mackintosh, N. J. & Turner, C. (1971) Blocking as a function of novelty of CS and predictability 534 of UCS. Quarterly Journal of Experimental Psychology, 23:4, 359-366. 535 https://doi.org/10.1080/14640747108400245 536 Mesulam M. M. (1981). A cortical network for directed attention and unilateral neglect. Annals of 537 neurology, 10(4), 309–325. https://doi.org/10.1002/ana.410100402 538 Mitchell, C. J., Le Pelley, M. E. (2010) Attention and Learning. Cambridge University Press, 539 Cambridge. 540 Pearce, J. M., & Hall, G. (1979). Loss of associability by a compound stimulus comprising 541 excitatory and inhibitory elements. Journal of Experimental Psychology: Animal 542 Behavior Processes, 5(1), 19–30. https://doi.org/10.1037/0097-7403.5.1.19 543 Associability enhancements across behavior systems 24 Pearce, J. M., & Hall, G. (1980). A model for Pavlovian learning: Variations in the effectiveness 544 of conditioned but not of unconditioned stimuli. Psychological Review, 87(6), 532–545 552. https://doi.org/10.1037/0033-295X.87.6.532 546 Pearce J. M., Mackintosh N. J. (2010) Two theories of attention: a review and a possible 547 integration. In Attention and associative learning: from brain to behaviour (eds Mitchell 548 C. J., Le Pelley M. E.), pp. 11–39 Oxford, UK: Oxford University Press. 549 Pearce, J. M., Kaye, H., & Hall, G. (1982). Predictive accuracy and stimulus associability: 550 Development of a model for Pavlovian conditioning. In M. L. Commons, R. J. 551 Herrnstein, & A. R. Wagner (Eds.), Quantitative analyses of behavior: Acquisition, vol 552 3 (pp. 241–255). Cambridge, MA: Ballinger. 553 Peck, C. A.; Bouton, M. E. (1990) Context and performance in aversive-to-appetitive and 554 appetitive-to-aversive transfer. Learning and Motivation, Vol. 21, (1), 1–31. doi: 555 https://doi.org/10.1016/0023-9690(90)90002-6 556 Posner, M.I. & Petersen, S.E. (1990). The attention system of the human brain. Annu. Rev. 557 Neurosci., 13, 25–42. https://doi.org/10.1146/annurev.ne.13.030190.000325 558 Raes, A. K., Raedt, R. D. (2012) The Effect of Counterconditioning on Evaluative Responses 559 and Harm Expectancy in a Fear Conditioning Paradigm. Behavior Therapy, Vol 43, (4), 560 757-767, https://doi.org/10.1016/j.beth.2012.03.012. 561 Roesch, M. R., Esber, G. R., Li, J., Daw, N. D., & Schoenbaum, G. (2012). Surprise! Neural 562 correlates of Pearce-Hall and Rescorla-Wagner coexist within the brain. The European 563 journal of neuroscience, 35(7), 1190–1200. https://doi.org/10.1111/j.1460-564 9568.2011.07986.x 565 Russo, S., Burns, N., & Baetu, I. (2019). Mackintosh, pearce-hall and time: An EEG study on 566 Inhibition ofreturn. Biological psychology,146, 107731. 567 https://doi.org/10.1016/j.biopsycho.2019.107731 568 Associability enhancements across behavior systems 25 Scavio, M. J., & Gormezano, I. (1980). Classical–classical transfer: Effects of prior appetitive 569 conditioning upon aversive conditioning in rabbits. Animal Learning & Behavior, 8(2), 570 218–224. https://doi.org/10.3758/BF03199598 571 Schiffino, F. L., & Holland, P. C. (2016). Secondary visual cortex is critical to the expression of 572 surprise‐induced enhancements in cue associability in rats. European Journal of 573 Neuroscience, 44(2), 1870-1877. https://doi.org/10.1111/ejn.13286 574 Schiffino, F. L., Zhou, V., Holland, P. C. (2014). Posterior parietal cortex is critical for the 575 encoding, consolidation, and retrieval of a memory that guides attention for learning. 576 European Journal of Neuroscience, 39, 640–649. https://doi.org/10.1111/ejn.12417 577 Swan, J. A. & Pearce, J. M. (1988). The orienting response as an index of stimulus associability 578 in rats. J. Exp. Psychol: Anim. Behav. Proc. 4, 292–301. 579 https://doi.org/10.1037/0097-7403.14.3.292 580 Timberlake W. (1993). Behavior systems and reinforcement: an integrative approach. Journal of 581 the experimental analysis of behavior, 60(1), 105–128. 582 https://doi.org/10.1901/jeab.1993.60-105 583 Timberlake, W. (1994) Behavior systems, associationism, and Pavlovian conditioning. Psychon 584 Bull Rev 1, 405–420. https://doi.org/10.3758/BF03210945 585 Wilson, P. N., Boumphrey, P., & Pearce, J. M. (1992). Restoration of the Orienting Response to 586 a Light by a Change in its Predictive Accuracy. The Quarterly Journal of Experimental 587 Psychology Section B, 44(1b), 17–36. https://doi.org/10.1080/02724999208250600 588