Pombal’s maze: a novel decision making paradigm

Ana Rita Secca da Fonseca Dissertation presented to obtain the Ph.D degree in Biology | Neuroscience Instituto de Tecnologia Química e Biológica António Xavier | Universidade Nova de Lisboa Oeiras, May, 2016 Insert here an image with rounded corners Pombal’s maze: a novel decisionmaking paradigm Ana Rita Secca da Fonseca Dissertation presented to obtain the Ph.D degree in Biology | Neuroscience Instituto de Tecnologia Química e Biológica António Xavier | Universidade Nova de Lisboa Oeiras, May, 2016 Pombal’s maze: a novel decisionmaking paradigm Research work coordinated by: Cá dentro inquietação, inquietação É só inquietação, inquietação Porquê, não sei Mas sei É que não sei ainda José Mário Branco, in “Inquietação” À memória de meu pai de quem herdei a inquietação que me trouxe até aqui This work was developed in the context of the International Neuroscience Doctoral Programme (INDP) of the Champalimaud Research Programme, Champalimaud Center for the Unknown, Lisbon, Portugal. The project entitled “Pombal’s maze: a novel decision-making paradigm” was carried out at Instituto Gulbenkian de Ciência, Oeiras, Portugal and at the Champalimaud Research Programme, Champalimaud Center for the Unknown, Lisbon, Portugal, under the scientific supervision of Zachary Mainen, Ph.D, and under the guidance of the Thesis Committee composed by Carlos Ribeiro, Ph.D, and Rui Costa, Ph.D. This work was supported by the fellowship SFRH/BD/33944/2009 from Fundação para a Ciência e Tecnologia, Portugal. i ACKNOWLEDGEMENTS I feel extremely privileged to have been offered the possibility to undertake such an incredible intellectual endeavor. Above all, I thank the people who throughout these years have generously given me the opportunity to become a better person, towards the world and towards myself Zach as an example of extra-sensorial intuition and intellectual freedom Masa as an example of rigor and endless generosity Patrícia for all the cognitive rough and tumble Gil for silence Maria for throwing ladders at the moon Bass as an example of making-it-happen Cindy as an example of thorough kindness towards Nature Eric as an example of scientific culture Nico as an example of detail Sara as an example of endurance Sam as an example of insurrection Alex as an example of rebellion Susana and Marta as examples of seriously playful science Rui and Carlos for putting things into perspective INDP 2008 for all the fun and support CNP community as an example of diversity and active listening world wide web as an example of human cooperation Alexandra Elbakyan as example of open science Rats as reminders of a universal bond Ana C. H. for helping me discover new ways to look within ii À minha família xoné como exemplo de amizade e estupidez natural À minha família de sangue por me ter ensinado o berro e o riso À avó Augusta como exemplo de leveza À avó Luz como um exemplo do brincar Ao irmão Filipe como exemplo de destemor À mãe Ana como exemplo de resistência A Matilde como exemplo de impermanência A Frederico como exemplo de amor incondicional iii RESUMO Os animais são constatemente confrontados com decisões: o que comer? Onde encontrar parceiro sexual? Como alcançar aquela gota de água? Actualmente, decorre um debate dentro do campo cujo o objecto de estudo são as tomadas de decisão, sobre quais são os mecanismos subjacentes à selecção de acções: será que os animais escolhem de entre o que pretendem obter - o retorno, ou escolhem de entre o que precisam de fazer para alcançar esse retorno? Existem dados comportamentais e de actividade neuronal que suportam a hipótese de que há competição entre as acções potenciais a acontecer em áreas relacionadas com a componente motora, enquanto os animais têm que decidir de entre múltiplas opções qual a que leva ao objectivo. Mais especificamente, tem sido avançada a hipótese de que a selecção de acções depende dum mecanismo de competição entre acções potenciais, onde um sinal polarizador favorece uma das acções, levando à sua selecção. Por forma a melhor compreender em que medida a selecção de acções depende efectivamente dum mecanismo de competição, é necessário explorar melhor quais os tipos de interacção entre acções potenciais, simultaneamente ao nível da actividade neuronal e do comportamento. Uma das formas em que este objectivo poderá ser atingido é através da investigação da actividade neuronal e do comportamento enquanto animais decidem entre múltiplas acções em que todas levam ao objectivo. Dentro das neurociências, existem duas linhas de investigação cujo um dos objectivos é determinar o que acontece no cérebro quando múltiplas acções permitem atingir o objectivo: o estudo de navegação espacial, que por um lado, pretende compreender a selecção de acções pela perspectiva do processamento de informação espacial, focando-se nos pontos do iv espaço que obrigam a uma decisão - pontos de escolhae correspondentes escolhas discretas, e o planeamento motor, que pretende compreender a seleção de acções pela perspectiva do controlo motor, focando-se nas dínâmicas contínuas do movimento. Assim, desenhámos e desenvolvemos um novo paradigma comportamentala que chamámos de labirinto de Pombalonde são oferecidas múltiplas opções de acção para chegar a um objectivo. Este paradigma apresenta a roedores o problema de escolher entre caminhos interconnectados quais os que lhe permitem chegar a uma recompensa sinalizada por um sinal luminososo, o que implica que esta tarefa tem simultaneamente características de uma tarefa de navegação e de uma tarefa de planeamento motor. O paradigma proposto baseia-se num labirinto que apresenta uma estrutura de uma grelha com múltiplos pontos de recompensa e múltiplas intersecções. Estas intersecções tinham o potencial para se tornarem pontos de escolha, dando a oportunidade aos animais de fazer várias decisões ao longo do caminho. Foi observado que os roedores adoptam a estratégia óptima de escolher os caminhos que são mais curtos para chegar à recompensa, na maioria das vezes. Dentro do conjunto dos caminhos mais curtos, os animais mostram tendências para certas opções baseando-se em características geométricas (e.g. o número de mudanças de direcção). Estas tendências reflectem custos internos implícitos associados a diferentes característiscas dos caminhos que combinadas podem gerar competição entre diferentes acções, pois podem ser mutuamente exclusivas. Adicionalmente, foi observado que os animais diminuiam a velocidade quando se aproximavam da intersecção, o que é indicativo de que estes locais eram vistos como sítios onde havia a possibilidade de decidir entre acções. Comparando trajectórias de caminhos em xi Figure 15 Trials categorized according to different criteria.. .................. 50 Figure 16 Example of start-goal pair for multiple option analysis.. ........ 51 Figure 17 Example of start-goal pair for center avoidance analysis. ....... 52 Figure 18 Numbering of each port.. ......................................................... 53 Figure 19 Example start-goal pair for maze experience.. ........................ 52 Figure 20 Goal probabilities per start port.. ............................................. 53 Figure 21 Metrics associated with direction of movement. ..................... 54 Figure 22 Animals avoid center paths.. ................................................... 56 Figure 23 Animals preferred to the go back through the direction of arrival at the port. ..................................................................................... 57 Figure 24 Preference for type A is maintained across the majority of start-goal pairs.. ........................................................................................ 59 Figure 25 Rats prefer paths that minimize distance to goal faster.. ......... 58 Figure 26 Rats prefer paths that go through the center, in a three options condition.. ................................................................................................ 59 Figure 27 Animals prefer to continue in the direction that they arrived at the port.. ................................................................................................... 60 Figure 28 Animals prefer paths with one turn.. ....................................... 61 Figure 29 Animals are faster to reach the goal when taking a path with one turn. ................................................................................................... 62 Figure 30 Different factors define different expected proportions for each option. ...................................................................................................... 68 Figure 31 Trial type A minimizes number of turns and trial type C minimizes Euclidean distance to the goal.. .............................................. 63 Figure 32 Different factors define different expected proportions for each option.. ..................................................................................................... 64 xii Figure 33 Rats don't prefer paths that minimize Euclidean distance to goal.. ......................................................................................................... 65 Figure 34 Animals prefer paths with one turn. ........................................ 66 Figure 35 Movement time is independent of number of turns................. 67 Figure 36 The probability of choice for option A or option B should be the same.. ................................................................................................. 72 Figure 37 Window of interest for behavioral analysis.). ......................... 77 Figure 38 Linear transformation of trajectories for spatial alignment.. ... 78 Figure 39 Raw velocities and corresponding trajectories.. ...................... 78 Figure 40 Examples of binned trajectories and corresponding velocities.. .................................................................................................................. 79 Figure 41 Example trial dynamics profile where one rat left the choice point ROI and then re-entered it. ............................................................. 79 Figure 42 Pairing of start-goal pairs according to spatial overlap.. ......... 80 Figure 43 Animals are faster to leave start port when they are about to do a shortest path.. ........................................................................................ 81 Figure 44 Animals take more time to leave the start port ROI when more options are available.. .............................................................................. 83 Figure 45 Animals were faster to leave start port when the goal was closer.. ...................................................................................................... 84 Figure 46 Bell-shaped velocity profile.. .................................................. 85 Figure 47 Animals slow down while reaching the choice point.. ............ 86 Figure 48 Extra non-optimal option does not affect decrease in velocity, when one optimal option is available....................................................... 87 Figure 49 Extra non-optimal option does not affect decrease in velocity, when more than one optimal option is available.. ................................... 89 xiii Figure 50 Animals slow down more when choice point offers two optimal path options.. ............................................................................... 90 Figure 51 Distance to the light does not affect velocity decrease at the choice point .............................................................................................. 92 Figure 52 Trajectories from competing options in a three option condition diverge 20 cm after animal has left the start port ROI.. ........................... 94 Figure 53 Trajectories from competing options in a six option condition diverge 20 cm after animal has left the start port ROI. ............................ 95 xiv AUTHORS CONTRIBUTION Ana Rita Fonseca (ARF) and Zachary Frank Mainen (ZFM) designed the experiments (Chapter 2). ARF conducted the experiments and analyzed the data (Chapter 3 and 4). xv TABLE OF CONTENTS ACKNOWLEDGEMENTS ______________________________________ i RESUMO _________________________________________________ iii ABSTRACT ________________________________________________ vi ABREVIATION LIST _________________________________________ ix FIGURE INDEX ______________________________________________ x AUTHORS CONTRIBUTION __________________________________ xiv TABLE OF CONTENTS _______________________________________ xv 1 GENERAL INTRODUCTION ________________________________ 1 1.1 The entrance… _________________________________________ 2 1.1.1 What _____________________________________________________ 3 1.1.2 Why ______________________________________________________ 3 1.1.3 How ______________________________________________________ 3 1.2 Framing decision-making processes _________________________ 4 1.3 Action selection mechanisms ______________________________ 6 1.3.1 Competition between actions _________________________________ 8 1.4 Multiple actions, one goal _______________________________ 11 1.4.1 Navigation and action selection _______________________________ 12 1.4.2 Motor planning and action selection ___________________________ 16 1.5 Guide for the reader ____________________________________ 20 2 POMBAL’S MAZE ______________________________________ 22 2.1 Introduction __________________________________________ 23 2.2 Methods _____________________________________________ 24 xvi 2.2.1 Animal subjects __________________________________________ 24 2.2.2 Experimental apparatus ___________________________________ 25 2.2.3 Pombal’s maze paradigm __________________________________ 26 2.2.4 Spatial distribution of reward _______________________________ 27 2.2.5 Training _________________________________________________ 28 2.2.6 Video processing _________________________________________ 29 2.2.7 Trajectories into sequences ________________________________ 30 2.2.8 Video and BControl synchronization ________________________ 32 2.2.9 Data base _______________________________________________ 32 2.2.10 Validation criteria _______________________________________ 33 2.3 Results _______________________________________________ 35 2.3.1 Preference for shortest paths ______________________________ 35 2.3.2 Strength of light-water pairing ______________________________ 38 2.3.3 Biased choices ___________________________________________ 39 2.4 Discussion ____________________________________________ 44 3 DISCRETE CHOICES _____________________________________ 47 3.1 Introduction __________________________________________ 48 3.2 Methods _____________________________________________ 48 3.2.1 Trial classification _________________________________________ 49 3.2.2 Movement time ___________________________________________ 54 3.2.3 Direction of movement ____________________________________ 54 3.2.4 Measuring preference _____________________________________ 55 3.3 Results _______________________________________________ 55 3.3.1 Preference for allocentric features: center versus edge ________ 55 3.3.2 Preference for egocentric features: backwards versus forward _ 56 3.3.3 Preference for geometrical features: minimizing the Euclidean distance to goal __________________________________________________ 57 3.3.4 Preference for geometrical features: number of turns _________ 61 xvii 3.3.5 Qualitative framework _____________________________________ 67 3.3.6 Competition between options ______________________________ 62 3.4 Discussion ____________________________________________ 67 4 DYNAMICS OF CHOICE __________________________________ 75 4.1 Introduction __________________________________________ 76 4.2 Methods _____________________________________________ 76 4.2.1 Spatial window of interest __________________________________ 77 4.2.2 Spatial data alignment and binning _________________________ 77 4.2.3 Velocity exclusion criterion _________________________________ 79 4.2.4 ROC analysis ____________________________________________ 80 4.3 Results _______________________________________________ 81 4.3.1 Latency to leave the start port ______________________________ 81 4.3.2 Bell-shaped velocity profile_________________________________ 84 4.3.3 Velocity at the choice point_________________________________ 86 4.3.4 Trajectories for competing paths ____________________________ 92 4.4 Discussion ____________________________________________ 95 5 GENERAL DISCUSSION _________________________________ 100 5.1 Conceptual framing ____________________________________ 101 5.2 Overview of main empirical findings ______________________ 101 5.3 Critical points ________________________________________ 103 5.4 Pombal’s maze and action selection ______________________ 105 5.5 Final remarks and future directions _______________________ 108 6 REFERENCES _________________________________________ 110 1 GENERAL INTRODUCTION I get in in what, from the outside, it’s called the inside searching for the path that will get to the center leaving longing and fear on the outside Sérgio Godinho, in “The maze” (Freely translated by ARF) 2 1.1 The entrance… Nature has provided humans with a lot to wonder about. Far away flickering points up above our heads, sea rages, clamorous lightning cruising dark skies, apples that persist to fall and days that persist to die. But maybe, and just maybe, the phenomenon that relentlessly takes more of our surprise is ourselves, what is within us. And, whatever that might be, it lives in an enclosed finite body which interacts with others – conspecifics or notand with the environment, through behavior. Even though there is not consensus about a definition of behavior, neuroscientists agree that it is generated by different types of biological matter (nerve cells, muscles) which work together to generate actions. There are many parts but in the end the body is just one. If a rat goes on a quest to find water then its head, legs, tail and 200 000 000 neurons have to get there, in full commitment with that decision, together. But animals have to make decisions constantly: what to eat, where to mate, how to reach that water drop... There are always many options so how do animals select a single one? There is an ongoing debate within the decision-making field about the underlying mechanisms of action selection: is action selection achieved by first making a choice between abstract representation of outcomes and then performing a sensorimotor transformation of that abstraction to the action space or is it achieved by evaluating several potential actions while at the same time weighing the costs and benefits of each, before arriving at a decision? 3 1.1.1 What One of the fundamental problems in decision-making is to understand how actions are selected (1.2 Framing decision-making processes). For a long time it was thought that action selection relied mostly on a central executive system that would then send the selected abstract action to be implemented by the motor system. There is currently compelling evidence that higher motor areas are involved in the action selection process. It has been hypothesized that action selection is generated by a competition process between potential actions which underlying architecture relies on mutual inhibition that is fed by a value-based bias signal (1.3 Action selection mechanisms). 1.1.2 Why On one side, there is a fair amount of research that has brought understanding about the generation of the bias signal. On the other side, there is both behavior and neural data that supports the hypothesis that competition between potential actions is taking place in motor-related areas while animals are deciding between multiple options to reach a goal (1.3.1 Competition between actions). Nonetheless, in order to further understand to what extent action selection relies on competition, it is necessary to further explore the types of interactions between potential actions both at the level of neural activity and behavior. 1.1.3 How One way in which this subject could be addressed is by investigating decision-making problems where multiple options lead to a goal and the animal can freely choose which option to take (1.4 Multiple actions, one 10 presented to a monkey that has to saccade to one of two parallel targets to implicitly report which motion direction it perceived, allowed us to understand that in lateral intraparietal area (LIP) there are neurons whose activity correlates with the integration of information. Another example is the odor-mixture categorization task, where a mixture of two odors is presented to a rat that has to nose poke into one of two parallel ports to implicitly report which odor identity it perceived, allowed to understand that not only SC was active before movement initiation but that this activity varied with sensory uncertainty and task difficulty (Felsen & Mainen 2012). Value-based decision-making paradigms allowed answering questions like where and how subjective experience information is integrated. For example, in a task where monkeys chose between two types of juice offered in different amounts, OFC neurons encode value independently of visuospatial factors and motor responses (Padoa-Schioppa & Assad 2006). Another example is a dynamic foraging task, where rats chose freely between two spatial goals that delivered a fixed amount of water reward with different probabilities, allowed to understand that neurons in secondary motor cortex (M2) encode value of upcoming choice (Sul et al. 2012). In summary, using these categories of paradigms behavioral and neural data was investigated while sensorial and/or subjective information was manipulated, creating decision conflicts that the subject had to solve. This lead to better understanding on how the brain represents and integrates information that is valuable for decisions whether this information is sensorial or derived from subjective experience (Glimcher et al. 2009; Sugrue et al. 2005). The previously described paradigms contributed for the understanding about how the bias is generated but it still remains to be understood if there is a competition circuit that computes the action selection 11 and that would be fed with this bias signal. There are other related lines of study that have addressed these questions more specifically. 1.3.1.2 Representation of potential actions The hypothesis that action selection is implemented through competition relies heavily on the implicit premise that potential actions under selection are simultaneously represented while there is an ongoing decision-making (Cisek 2006; Tipper et al. 1998). The theory that perceiving and acting on the world depends on the simultaneous specification of multiple potential actions than can be afforded by the environment has been around for 40 years (Gibson 1979). During that time convincing evidence has been accumulated showing that when an animal is offered the possibility to choose between multiple actions to reach a goal these potential actions are simultaneously represented in motor-related areas (Cisek & Kalaska 2002; Cisek & Kalaska 2005; Basso & Wurtz 1998; Houk 1995). In summary, at least some of the necessary conditions for the implementation of action selection through competition are met, both for the existence of a bias signal and for the simultaneous representation of potential actions. Nonetheless, it is not fully understood how and what type of interactions dynamically occur between potential actions, both at the level of neural activity and behavior. It is necessary to further explore the types of interactions between potential actions both at the level of neural activity and behavior. One way in which this subject could be addressed is by investigating neural activity and behavior while animals are deciding between multiple options that all lead to goal 1.4 Multiple actions, one goal 12 Within the decision-making field, there are two types of problems that require the subject to make a decision about which action to take, among alternatives, to reach the same goal: motor planning tasks and spatial navigation tasks. Both types of tasks involve making choices about which movement or sequence of movements to optimize in order to reach a goal (Penner & Mizumori 2012; Whitlock et al. 2008; Wolpert & Landy 2012), being the main difference that, on the first case, the movements under choice are performed by one body part (hand, arm or eye) and, in the second case, the whole body. Also, there is an important line of complementarity between these two fields associated to the type of action selection correlates they focus on. If, on one side, in navigation paradigms one of the key windows for understanding action selection is the choice point, where discrete choices are measured and interpreted according to different strategies (Tolman 1938); in the case of motor planning, one of the key windows is the movement between start and goal positions, where the continuous dynamics of movement is assessed (Flash & Hogan 1985). It seems that considering both of these two perspectives can bring a more complete understanding about action selection correlates and may lead to a more detailed characterization of both the type of choices at choice points and the dynamics of those choices. Consequently, this can contribute for a more complete comprehension of how animals select actions namely by exposing correlates of the interactions between potential actions. 1.4.1 Navigation and action selection It has been proposed extensively that animals select actions in navigation contexts using two distinct schemes derived from instrumental conditioning: goal-directed action selection, driven by response-outcome associations, and 13 habitual action-selection, driven by stimulus-response associations (Glimcher et al. 2009; Penner & Mizumori 2012; Niv et al. 2006; Balleine & Ostlund 2007; Daw & Shohamy 2008). Goal-directed action selection is sensitive to the contingencies between actions and their outcomes and to the utilities of these outcomes. It depends on having a schema of how the world works that can be used to determine the consequences of actions (Redish 2016). This argues for animals being capable of learning the casual relationship between their actions and the resulting outcomes, allowing them control over their own action based on their desire for a particular outcome goal (Tolman 1948). This type of schema confers animals with the ability to plan actions to maximize expected rewards within their environment and flexibly integrate new information like a novel stimuli and or a change in reward contingency. In that sense, this system does not impose that a specific action has to be selected but animals can resort to it to guide behavior (Redish 2016; Johnson et al. 2007). It is thought that this flexibility comes with a time cost because it involves deliberation. This system is critically dependent on intact hippocampal function (Poldrack & Packard 2003; Packard & McGauch 1996). On the other side, habitual action selection is sensitive to the pairing between antecedent stimuli within the environment and actions without regard to their consequential outcomes (Niv et al. 2006). Animals automate their behavior when actions reliably led to goals, by developing stored action chains that can be released at appropriate times but that, once initiated, tend to run to conclusion (Hull 1943; Dezfouli et al. 2014). The habit system allows for fast action selection but also permits arbitrary stimulus-action pairings. Hence, it provides the basis for simpler, non-integrated navigation functions such as stimulus recognition and approach and is critically dependent on intact striatal function (Penner & Mizumori 2012). 14 From the perspective of action selection, solving a navigation task is also solving an optimization problem that depends on the processing of spatial contextual information (Niv et al. 2006; Penner & Mizumori 2012). Reinforcement learning is a theoretical framework that aims at describing the process through which an organism learns to optimize behavior within a decision environment. The decision-making environments in which reinforcement learning occurs is defined by a set of states (Sutton & Barto 1998), which in the case of navigation, can be represented by locations on a maze, such as a corner or an intersection; a set of possible actions that the decision-maker can choose from, such as turn left or travel west; and, finally, a set of rules that the decision-maker must learn about the environment through persistent interaction with it, such as where is reward located and when. In reinforcement learning, outcomes such as food or water have numerical utilities, and the imperative is to choose actions to maximize a long-term measure of total utility (Niv et al. 2006; Glimcher et al. 2009). The actions implemented by the decision-maker implicate a change from one state to another, which leads to different outcomes. Then the problem is to decide which strategy maximizes utility given many different possibilities of states. In the reinforcement learning framework, on one side, using goaldirected system implicates that, when at a choice point, animals work out the ultimate outcomes consequent on a sequence of their actions by searching through a tree of states, joined up according to the actions that lead between them, and choosing actions based on the outcomes’ current utilities; on the other side, habitual action selection system implicates that, when at the choice point, animals chose an action by comparing their relative stored values, rather than the actual identity of the outcomes consequent on sequences of actions . 15 In navigational contexts, the outcomes are often dependent on the implementation of whole sequences of action choices dependent on the spatial context. Hence, understanding how animals select which path to take to reach a goal is also a matter of spatial cognition. This area of knowledge focus, for example, in investigating how spatial information from the environment is processed and represented in the brain (Moser et al. 2008; O’Keefe 1979), how internal cues are used to navigate the environment (Taube 2007), how animals integrate information from a travelled path to then return to an initial position (McNaughton et al. 2009) or how animals learn to solve a maze (Tolman & Postman 1953). Among other behavioral paradigms, mazes have been the most popular setting to understand how animals navigate the environment (Knierim & Hamilton 2011; Sharma et al. 2010). Usually, inside mazes, animals are presented with the problem of navigating through a network of interconnecting paths and dead ends where only one or few options lead to reward. Generally, the underlying strategy of these paradigms is to let the animal to familiarize itself with the maze by learning spatial features and reward contingencies, and then manipulate either or both. These manipulations are designed so that the observed behavioral effects can be explained by different learning factors. For instance, in the case of Tolman–Hull plus maze where rats are trained for collecting reward according with a certain rule - turn left from the south arm to the west armto test what animals have learned, trials starts in a different arm and, then depending on which strategy the animal is following, the behavioral effect should be different (Gardner et al. 2013; Ruprecht et al. 2014). Another example is the Multiple-T task where rats are trained to run through a central track and to turn left or right for food (Brown 1946; Schmitzer-Torbert & Redish 2004). On each day the central track has a 16 different shape and the side on which the final reward is placed can change, but both variables generally remain constant within a day. Here the aim is to determine what animals learn by observing, for instance, when animals take wrong paths. Given that one of the aims of this type of studies is to understand how animals process and integrate spatial information, the goal is usually not cued to allow for the animals to reveal their expectation of where the goal is, which may bring up how the animals is using spatial information. An exception to this is the Cue-T task where rats are trained to turn left or right at a T intersection where the correct choice depends on a sensory cue provided on the stem of the T (Johnson & Redish 2007; Thorn et al. 2010). In this case, the aim can be to better understand how animals learn to associate stimulus with reward (Barnes et al. 2005). Overall, these paradigms allow for asking different questions from how is spatial information processed and learned and how possible choices map on to different strategies. While solving a maze, the animal is faced with choice points where a decision takes place. At the choice point, animals should consider which action among alternatives leads to reward and, depending on the underlying strategy and on the decision variables that the animal is taking into consideration, the selected action should be different. As in any other decision-making study, the experimental strategy is to pose a problem to the subject where, given an optimization problem, the different choices reveal the underlying processes of action selection. 1.4.2 Motor planning and action selection In planning a movement, the brain has to select one of many possible movement plans. Determining whether the motor system naturally prepares 17 multiple potential movements when merely presented with alternative targets for action, one of which will subsequently be selected before a movement is even required, is critical to the understanding of the underlying mechanisms by which the brain initially represents and makes decisions between competing options in the environment (Gallivan et al. 2015; Murakami & Mainen 2015). As described before, it has been proposed that selection of actions may be performed through competition among neurons preparing for available actions (Cisek 2007). This further supports the view that motor systems do not passively reflect the result of completed cognitive processes; rather, it is crucially linked to the dynamic decision-making process itself (McKinstry et al. 2008; O’Hora et al. 2016; Haith et al. 2015). Considering the linkage between cognitive and motor processes, and the concurrent processing of multiple motor plans, competing cognitive states are likely to prepare corresponding plans in parallel (Song & Nakayama 2009; Cisek 2012). Hence, the quality of the interactions between potential actions could, in potential, spill to the movement execution dynamically modulating reaction time or trajectory. In the particular setting of motor planning paradigms, for instance, it has been shown that when reaching for a target in the presence of a distractor, where the distractor afforded a movement with the other hand, resulted in higher reaction time in comparison when the distractors afforded responses for the same hand (Ray et al. 2014). Moreover, there is evidence that the number of actions under consideration for selection modulates reaction time that increases with the increase in the number of possible reach targets (Churchland et al. 2008). Evidence for dynamical interactions between action plans has come from kinematic studies of eye and arm movements made to targets in the presence 18 of distractors in ‘go-before-you-know’ tasks. It has been observed that the trajectory of a reach to a target is influenced by the presence and placement of distracters (Welsh et al. 1999; Tipper et al. 1998), suggesting that multiple target and distractor related directional signals coexist and interfere, in neural populations specifying reach direction. When a target and distractor appear in close proximity, the endpoints of target-directed eye and arm movements often land between them, an observation which has been termed ‘spatial averaging’ (Van Der Stigchel & Theeuwes 2006; Eggert et al. 2002; Theeuwes et al. 1998). In a case where the presented targets corresponded to potential reach locations and not as distractor, it has been shown that, trajectories are spatially sensitive to the probabilistic distribution of targets within the display. Specifically, when presented with two or three target displays, subjects initiate their reaches toward an intermediary or averaged location before correcting the trajectory of the movement towards the cued target location. This indicates that the observed spatial averaging of the reaching trajectory depends not only on the spatial position of the stimulus in the display, like in the previously described case of the distractors, but also on the probability that these stimulus are targets that afford reach movements (Chapman et al. 2010). There has been some debate about whether the potential reach targets were being encoded in visual versus motor coordinates so that the spatial averaging was a consequence of visual averaging but, recently, it was shown that by introducing an obstacle in some trials while subjects where solving a ‘go-before-you-know’ task generated a correct spatial averaged reach trajectory predicted by averaging of motor target locations rather than averaging of visual target locations (Stewart et al. 2014). One of the hypothesis that has been put forward to explain all these behavioral observations is that closely spaced visual stimuli (targets or 19 distractors) create overlapping hills of activity in the corresponding motor maps of structures involved in movements of the eye in structures like SC (McPeek 2003), and arm in structures like motor cortex (Georgopoulos et al. 1988) with the final movement vector being determined by the averaging of these signals (Tipper et al. 2001). Under this view, both targets and distractors are therefore thought to be initially represented as potential targets by the visuo-motor system (McPeek & Keller 2004). In agreement with this interpretation, it was found that activity in the SC appears greatest at an intermediary location when spatial averaging of eye movement occurs (Glimcher & Sparks 1993) arguing for the simultaneous initiation of multiple movements and consequent averaging of the resultant activity. On the other side, it has also been shown that in a cued-response delayed task, when two targets of different value are presented before the go signal and, after the goal signal, the less valued target is cued for reach, the average trajectory tends to be initially deviated towards the higher value target (Pastor-Bernier et al. 2012). Congruently with this view, for instance, it has been observed that neural activity correlated with movement direction is preshaped by information about the available motor choices (Kurata 1993). For example, neural activity related to an action tends to be stronger if the action is more likely or yields higher rewards (Kable & Glimcher 2009; Sugrue et al. 2005). This observation would again enforce the interpretation that there is an underlying ongoing competition between the two representations of the two reach movements but the one that is more valued is winning before the cue is presented, and then when the cued signal is presented the subject has to update the value of each target and take the reach that leads to reward. More specifically, the hypothesis that competition is implemented through mutual inhibition finds its stronger evidence in studies that show that peaks 26 two was 75 cm. The tracks that composed the grid were 15 cm wide. The hard structure of the maze was built with metal rails and tracks where plastic sheets parts that slid into the rails slit. Moreover, ~2 cm away from each port there was a white visible led encapsulated in a 10 cm height plastic tube which would signal for reward availability. A Point Grey FL3-U3-13S2M camera with lens Fujinon YV28x28SA-2 was placed at 1.5 meters above the center of the maze and video recordings where performed at 73 fps with resolution of 1280x960. Experiments were performed under dim light (~10 lux) to increase the contrast between background and light cue. The video was recorded on the infrared spectrum and the maze was illuminated by standard CCTV infrared lamps. 2.2.3 Pombal’s maze paradigm Figure 1 Top view of the maze. The maze was an elevated grid structure with nine water ports embedded on the floor, with no walls. Each port was distanced from the closest neighbor by 75 cm and each track was 15cm wide. Co-localized with each water port there was a visible led encapsulated in a plastic tube. 27 Each trial initiated when one of the LED lights, co-localized with each port, was switched on. When the animal poked in that goal port, light went off and water reward was released (Figure 2). If the animal was out of the port for more than 0.5 s, then a 100 ms inter-trial interval (ITI) started, after which a new trial started. After light was on, the animal could freely choose which path to take to reach the goal (Figure 3). There was no penalty for not choosing the shortest path. If the animal took more than 90 s to poke in, light went off and a new trial started. 2.2.4 Spatial distribution of reward Spatial distribution of reward was defined according to a set of probabilistic rules so that it would be unlikely that rats could predict where the next Figure 2 Trial structure. Schematic drawing of the succession of events in each state: light was switched on in port location A (orange); after rat poked in that port (blue trace), light went off and rat poked in and out, signaling reward consumption. When the rat was out of the port for more than 500ms, an inter-trial interval started, lasting for 100ms (grey). Figure 3 Example trial. Schematic drawing of an example trial where the start port corresponded to the location where reward was lastly available (blue circle) and the goal port corresponded to the location where light was on (yellow shape). Both reward consumption period and ITI are not represented. 28 reward was going to be available before light turned on. The aim was to make it costly for the rat to decide which path to take in the next trial before the light was on, and, hence, increasing the probability that the decision was happening in a defined time window between light is on and rat leaving start port. Nevertheless, at the same point exposing the rats, exclusively, to a uniform distribution of locations implied that all start-goal pairs would be equally represented in our sample which would lead to the overrepresentation of trials that were not proportional in interest for the analysis. In order to generate a bias, we defined categories of interest for the behavioral analysis and biased the distribution of reward location accordingly. These categories and generated biases will be explained in depth in Chapter 3, Section 3.2.1. 2.2.5 Training The training sequence consisted of: (I) handling (5 sessions); (II) 20 minutes free exploration of the maze (1 session); (III) nose poking water association where light was on for six random port locations simultaneously so that if animal poked in either of them reward would be released (2 sessions); (IV) light water association where number of lights per trial would decrease with increase in number of trials; (V) when only one light was on per trial the full session. This progression happened across several days and it was individualized to each rat. The rationale was that as long as the animals would continue running after rewards the exposure to the light-water pairing would happen. Hence, the criterion for training progress was the number of trials per day (Figure 4). Animals significantly do more trials in the last session than in the first session (one-sided, paired t-test, p=0.0020, n=8). After this stage, animals were considered to be sufficiently exposed to the maze and to the light-water pairing. 29 Rats took on average 9±0.5 sessions (mean±S.E.M.) from first session of stage III to the last session of stage IV to reach this stage. Overall, each rat performed ~5000 valid trials, that is, trials where the animal started the trial on the correct port and finished in the correct goal port and where no incorrect pokes were performed. Overall, each rat performed ~5000 valid trials, that is, trials where the animal started the trial on the correct port and finished in the correct goal port and where no incorrect pokes were performed. 2.2.6 Video processing Video acquisition and online analysis were performed with Bonsai (Lopes et al. 2015), where x and y position, body centroid and orientation of the animal were extracted from each frame using background subtraction and binary region analysis by ellipsoidal approximation. Area occupied by the rat corresponded to ~3500 pixels. It was estimated that, from the body center to Figure 4 Number of trials increases with experience. Comparison between the mean number of trials in the first training session and mean number of trials in last training session. Error bars are mean±S.E.M. 30 the tip of the snout there was a ~5 cm distance. Given that there was fish eye like image distortion, we applied a transformation matrix to the image using a standard image correction procedure. This transformation matrix was obtained after image calibration using a checkerboard. After the transformation, the error was equal to 0.9 cm in the x direction, and equal to 0.8 cm in y direction. Smoothing was applied to x, y values of center of mass in order to reduce noise from different sources such as video jitter or body wobble (Hen et al. 2004). We used a generalized version of the local weighted regression (LOWESS) technique (Cleveland 1979) called LOESS. In the latter, each smoothed value is given by a weighted quadratic least squares regression instead of a weighted linear least squares regression as in the former. We used a lag of 35 points, corresponding to ~ 0.5 s window (Figure 5). This smoothing technique was applied using MATLAB Curve Fitting Toolbox. Velocity over x and velocity over y were derived from center of mass position using a standard numerical method and then the module 𝑣 = 𝑠𝑞𝑟𝑡(𝑣𝑥^2 + 𝑣𝑦^2) was calculated. Video frame rate jitter was on average 3% (data not shown). 2.2.7 Trajectories into sequences Figure 5 Example of velocity trace over y dimension. In grey is represented the trace before smoothing y as a function of time and in black is represented the trace after smoothing, for a window of ~25s. 31 After smoothing, we transformed x,y coordinates into a sequence of numbers corresponding to port location. This procedure was performed according to the following steps: 1) x,y coordinates in pixels where transformed into x,y coordinates in the real world to cm, according to a linear relation extracted via a standard calibration procedure. 2) The center of mass position was converted into a sequence of locations in an offline procedure. The location of each port was manually defined by extracting x,y coordinates out of a distortion-corrected frame. Regions of Interest (ROI) where defined by a square centered in the x, y coordinates of each port center and side size equal to 13 cm (Figure 6). Each ROI was assigned the corresponding poke number between 1 and 9 and whenever the animal occupied one of the 9 areas, x,y coordinates where attributed the corresponding number. When the x,y coordinates did not match any of the ROIS, 0 was the assigned value. Finally, all consecutive equal numbers where reduce to one element such that sequence loses Figure 6 Regions of interest centered on water ports. Example session where x,y position (black) is detected in the different ROIs associated with each port (color). 32 information on how long the animal stayed in the same ROI and all zeros corresponding to movement in between ROIs were removed. Hence, information about reentering ROIs was preserved. 2.2.8 Video and BControl synchronization Video and nose poke data where aligned using the led visible light associated to each port so that in each trial the time of onset of light was given both by the clock from the BControl and the detection of light change in the video frame. The time lag between the BControl data and the video data was on ~ 17ms. 2.2.9 Data base All the analysis was performed in Matlab® 2014b (8.4.0.150421, The MathWorks, Inc). Per each rat and each session, we automatically generated a matrix where rows corresponded to trials and columns corresponded to different fields of two types:  Raw information such as frame number when trial started, frame number when it ended, time in session when trial started, time in session when trial ended, start port, goal port.  Processed variables: shortest path length, rat path length, shortest path {0 – non shortest path, 1 – shortest path}, turn {0 – straight, 1 - turn}, direction of movement at arrival to port {South, West, North, East}, direction of movement at departure, among others. Each trial was tagged with the corresponding state of the variable in the corresponding column. For example, for selecting trials where start port was 1, goal port was 3 and rat did shortest path - 1 we would apply simply a conjunction logic rule extracting the rows for which the columns would meet these criteria. All the analysis was performed using as basic units start-goal 33 pairs, which implicitly means that we aligned data on the start port nose poke event until goal port nose poke event. 2.2.10 Validation criteria For the validating the paradigm, we evaluated behavior according to three criteria: I. Maze familiarization and visual discrimination of cue, where we focused on choices between shortest paths and detours. More specifically we measured: a. Median latency to leave start port in trials where rats did shortest paths and where rats did a detour. Latency to leave start port was defined has the interval between trial started, that is, light turned on and the time that rat took to leave ROI of the start port. For each start-goal pair we calculated the median latency to leave start port, averaged across start-goal pairs for each rat and determined the average across the population. b. Population median proportion of shortest path trials in comparison with a null model that selected randomly at each choice point from the set of closest geometrical neighbors, that is, a stochastic choice model. This sampling was run iteratively until the goal was finally met. Firstly, this agent was run eight times, one for each rat, where it was presented with the same start-goal pairs as each of the rats. After determining the median proportion of shortest-path trials for each start-goal pair for each run, we calculated the median for each run and, finally, the median of medians across the eight runs. Finally, we run this procedure 1000 times and calculated the 95% confidence interval. 34 c. Population median proportion of shortest path trials in comparison with a stochastic choice model, as a function of Manhattan distance to goal. We split the data in b. into trials according to Manhattan distance to the goal and determine median proportion of shortest path trials across distance to goal. d. Population median proportion of shortest path trials across sessions, after training had finished. After determining the median proportion of shortest path trials for each start-goal pair for each rat, we determined the median across start-goal pairs for each rat and, finally, the median of medians across the population. This measure was calculated for each of the first five sessions after training had ended. II. Strength of light water pairing, where we focused on choices between poking or not in non-cued ports. More specifically, we measured the proportion of trials where animals poked at least once in a port where light was not on. We separated trials were animals did shortest paths and then quantified the median proportion of these where animals did incorrect poking. We used the median given that the there was a high probability that distribution across rats would be close to zero as time goes by. Hence, we selected the first 5 sessions, after animals were considered to be trained and from these we selected trials where animals did shortest paths. We excluded detours from this analysis because the main concern was that rats would use light as directional cue for shortest path choice but then would be stopping at each water port to check for reward. III. Biased behavior, where we focused on choices between different options within each set of possible shortest paths. More specifically, we separated trials into three conditions: trials where animals could 35 select between two shortest path options, three shortest path options or six shortest path options. For each start-goal pair, we arbitrarily categorized each of the options (option A, option B, etc) and determined the average proportion of trials consistently across rats corresponding to each option. We averaged over all start-goal pairs for each condition per rat and then averaged across rats. In the condition where two shortest paths were available, the comparison to determine a bias was made with a shuffling of the data, so that in each trial for each start-goal pair for each rat the choice was sampled from a binomial distribution. Moreover, we repeated the analysis but separating data into three batches: the first 1/3 of the sessions, the second 1/3 of the sessions and last 1/3 of the sessions. The data was split into batches instead of doing session by session analysis because there was not enough statistical power for that. Lastly, the overall biases were compared between the three batches corresponding to three experience epochs. 2.3 Results 2.3.1 Preference for shortest paths Firstly, in order to understand if animals were following an optimal strategy, we compared the choices of the animals with a null model that did not take into consideration any information about the goal, that is, at each choice point it selected randomly which path to take. Hence, we compared the median proportion of shortest path trials, across the population, with a stochastic choice model simulation which sampled randomly at each choice point (Figure 7). Animals chose shortest paths more often than a stochastic choice model simulation (permutation test, p<0.002, n=8). More specifically, 42 There was no significant different over the three epochs, for all the three Figure 13 There is a bias towards one of the options, in a three option condition. (i) Example start-goal pair where six shortest path options were available. Only three of the six shortest path options are depicted (A, B and C) (ii) Mean proportion of trials corresponding to each of the six options in the set of shortest paths for one rat across all start-goal pairs matching the condition. D, E and F are the corresponding diagonally symmetric paths of A, B and C (iii) Mean proportion of trials corresponding to each of the 6 options in the set of shortest paths (one-way ANOVA, p <0.001, n=8). Error bars are mean±SEM. 43 conditions with associated number of options. There was no significant difference across the three time epochs for all the three conditions (one way Figure 14 Biases towards certain options do not change across time. (i) Mean proportion of trials where option A was chosen out of two possibilities across rats (horizontal line) over session batches (lighter grey – first batch, middle grey –second batch, darker grey third batch) (one way ANOVA p=0.6759, n=8). (ii) Mean proportion of trials where options A, B and C where chosen out of three possibilities across rats (horizontal line) over session batches (lighter grey – first batch, middle grey –second batch, darker grey third batch) (two way ANOVA, interaction option x time, p=0.1789, n=8). (iii) Mean proportion of trials where options A, B, C, D, E and F where chosen out of six possibilities across rats (horizontal line) over session batches (lighter grey – first batch, middle grey – second batch and darker grey third batch) (two way ANOVA, interaction option x epoch p=0.1228, n=8). Error bars are mean±SEM. 44 ANOVA, p=0.6759, n=8), three options (two way ANOVA interaction option x epoch, p=0.1579, n=8) and, finally, six options (two way ANOVA interaction option x epoch p=0.8322, n=8). This observation indicated that preferences for certain options were robust across time which possibly meant that these biases conferred an advantage to the behavioral strategy that animals were using to solve this task. 2.4 Discussion Given our interest in gaining a better understanding about action selection phenomenon, we aimed at developing a novel decision-making paradigm which would have simultaneously navigation and motor planning features. Hence, we designed and developed what we termed Pombal’s maze paradigm where rats chose from multiple options in a grid-like maze which path to take to reach a water reward signaled by a visual cue, on a trial-bytrial basis. In order to validate the paradigm premises, behavior was characterized and validated using different criteria. Animals were able to learn the task both because they selected shortest paths most of the trials and because they mostly checked for reward in the ports where light was on. Additionally, the proportion of shortest paths increased with experience to the maze. This could be explained by the need to learn to visually discriminate the light position being that, given the maze symmetries, the goal could have been in one of five possible locations, and/or the need to learn the maze geometry, being that it imposed constraints on which action sequences could be performed (diagonals across ports were not an option) and, consequently, which paths could be taken to reach the goal and from those which were at the shortest distance. 45 In comparison with a stochastic choice model simulation, animals chose shortest paths more often which enforced our hypothesis that animals were following an optimal strategy of using visual information about the goal to select the shortest path. Predictably, in order for the stochastic choice model to generate choices that would match the optimal strategy it would just require the introduction of the constraint where options that would not minimize distance to the light would have zero probability. This would correspond to an ideal subject with a shortest path proportion equal to one. Animals can perform close to what would be expected by an ideal subject so it was concluded that animals were using visual information to choose the optimal paths to reach the cued goal. Even though the learning processes that rats go through while being trained were not subject to investigation in this project, the potential interest of doing so will be alluded to in Chapter 5. Pombal’s is a two dimensional maze that was designed so that there were multiple paths to reach the goal. The solution found was inspired by motor planning tasks where typically the start position and the goal position are the same for all possible actions. Accordingly, all paths were at the same Manhattan distance to the goal. Given the two dimensional nature of the maze, this necessarily implicated that their geometry had to be different and, hence, the necessary action sequences to travel those paths would also be different. Hence, we considered that even though there were different action sequences to travel the same distance and to reach the same goal, it could be argued that the difference in cost across action sequences was probably residual. Nevertheless, rigorously, there were differences between these action sequences. In that sense it was important to quantify if animals showed biases towards some options over others, for instance, it would have been possible to choose always the same shortest path within a given startgoal pair. We observed that animals consistently show biases towards certain 46 options within the set of shortest paths. Hence, the observed patterns of choice seemed an indication that animals were taking into consideration other decision variables besides distance to goal. Possibly, these were also decision variables relevant for the optimization problem that animals were solving and hence potentially revealing of the underlying decision-making processes. Nevertheless, the problem that animals were trying to solve was also a navigation problem where the goal did not have to be inferred from landmarks or did not have to be remembered, but still information about the goal position had to be integrated with information about maze geometry, somehow. Pombal’s maze is also a cued navigation paradigm. In conclusion, the strategy that animals were using to solve this optimization problem remains to be understood. Specifically, it is necessary to understand in greater depth how animals were using:  goal visual information. Did animals discriminate the exact location of the goal among five possibilities and only then they selected the shortest path option among all possible paths? Did animals discriminate the direction of the light in relation to their head position and then followed through the path that seemed to be closer to the light?  maze geometry information. Which variables are animals weighing when they choose between different options? Answering these questions will add to a better understanding of the strategy that animals used to solve the task and it will also allow us to determine which pool of actions was being considered at each trial, hence, better defining the action selection layer of the decision-making problem. These matters will be subject of further investigation in the following chapters. 47 3 DISCRETE CHOICES and near the center of myself there is an half open gate and I have ahead the uncovered monster the beast that I guard and that guards what lives within Sérgio Godinho, “The maze” (Freely translated by ARF) 48 3.1 Introduction In the previous chapter, we introduced Pombal’s maze paradigm and discussed the optimal strategy to solve the underlying decision-making problem. In this chapter, we will introduce the reasoning we used to frame behavioral analysis for the decision-making paradigm described in Chapter 2, and, then, we will summarize the behavior of rats trained on that paradigm. On a first level, in Pombal’s maze, animals could freely choose which path to take to reach the goal. As long as the animal poked in the port where light was on, it would get reward, so, in that sense, this task did not implicate a forced choice. Nevertheless, given the water deprivation state of the animal, rationally, the optimal strategy would be to select paths that minimize distance to the goal, so that, animals would get to goal as fast as possible, maximizing the reward rate. On a second level, for trials where multiple shortest paths options were available, there were geometrical features which distinguished these paths (e.g. number of turns). But there may have been other features that were dependent on internal biases of the animals, for instance, preference for moving in a given direction. Such features may have imposed a cost or a benefit for the strategy of the animal. More specifically, we investigated which decision variables the rats were taking into consideration, by describing revealed preferences for given types of paths. Our approach was based on selecting and comparing trial conditions that could isolate only one potential decision variable. Overall, we were aiming in defining the underlying navigation problem. 3.2 Methods 49 3.2.1 Trial classification Trials were collapsed depending on different classification criteria: A. Manhattan distance to the goal: where trials were grouped according to distance in maze segments between port location and goal port (Figure 15i); B. Euclidean distance to the goal: where trials were grouped according to Euclidean distance in segments between port location and goal port (Figure 15ii); C. Number of path options: where trials were grouped according to number of shortest path options to the goal (Figure 15iii); D. Number of turns: where trials were grouped according to the number of turns across shortest path options (Figure 15iv); E. Allocentric features: where trials were grouped according to where the paths goes through in the maze (Figure 15v); F. Egocentric features: where trials were grouped according to features that depend on body position (Figure 15vi); 50 Figure 15 Trials categorized according to different criteria. (i) Schematic drawings of examples of four start-goal pairs where the rat had to travel between start and goal one maze segment, two maze segments, three maze segments or six maze segments. (ii) Schematic drawings of five examples of start-goal pairs where rat would have to travel between start and goal the Euclidean distance corresponding to one maze segment, 𝟐 maze segments, two maze segments, 𝟓 maze segments or 2 𝟐 maze segments. (iii) Schematic drawing of four examples of start-goal pairs where there is a set of shortest paths with one option, two options, three options or six options. (iv) Schematic drawings of three examples of start-goal pairs where path required one turn, two turns or three turns. (v) Schematic drawing of an example start-goal pair where there were only two shortest paths and where one would require going through the edge and the alternative would require going through the center. Schematic drawing of an example of a start-goal pair where rat could select one of two shortest path options where it would require to go backwards in the opposite direction in relation the direction of arrival. 51 In Chapter 2, we mentioned that spatial location of reward was probabilistic but subjected to a bias factor (Section 2.2.4). Here we explain in more detail what this entailed. There were 72 possible start-goal pairs but, for example, 32 pairs corresponded to trials where only one shortest path option was available. So, in order to get a more evenly distribution of trial types we presented rats ~7 session days where all start-goal pairs were equally probable (uniform distribution) and ~15 session days where there was a bias. These biases where defined to maximize the number of trials associated to different categories of trial types. We considered three main categories of trials: center avoidance trials, which aim was to determine if animals showed avoidance to the center; multiple option trials, which aim was to analyze behavior when more than two options were available and, finally, experience trials, which aim was allowing the animals to experience enough rewards in the center. Splitting trials into these different conditions allowed for a more detailed investigation of the preferences for certain paths. For this analysis only shortest path trials were considered. According to this frame, we defined three main categories of trials: 1. Multiple option trials, which aim was to analyze behavior when more than 2 options were available (Figure 16). As a target, we allocated 45% of trials. Figure 16 Example of start-goal pair for multiple option analysis. Schematic drawing of one example start-goal pair where more than 2 shortest path options were available. 58 trials across rats where paths that would minimize Euclidean distance to the light would be selected (Figure 24). Rats showed a significant preference for paths that minimize Euclidean distance to goal (permutation test, p=0<0.002, n=8). In order to test for robustness of this observation, we split the data into startgoal pairs and measured the mean proportion of trials of type A across rats in comparison with sampling from a binomial distribution (Figure 25). It was observed that 10/16 start-goal pairs show means above the 95% confidence interval (permutation test, p<0.002, n=8). This indicates that preference for type A paths is robust across different locations in the maze and across rats, enforcing the hypothesis that taking this paths is part of strategy. Secondly, we analyzed mean proportion of trials where animals selected type A but separated by start-goal pairs that go through the center vs through the edge and compared these means in order to understand if choosing A was dependent on location in the maze (Figure 26). Figure 24 Rats prefer paths that minimize distance to goal faster. (a) Schematic drawing of an example start-goal pair where, after first segment of path, a type A path would take the rat closer to the light position in terms of Euclidean distance and a type B path would take rat further away from the light position in terms of Euclidean distance. (b) Mean proportion of trials where rats choose type A paths and the expected level of chance sampled from a binomial distribution (permutation-test p<0.002, n=8). Error bars for rats are mean±SEM. Error bars for binomial sampling are 95% confidence. 59 The mean proportion of type A trials that go through the center is Figure 25 Rats prefer paths that go through the center, in a three options condition. (i) Schematic drawing of example start-goal pairs where the paths go through the edge (solid line) or through the center (dashed line). (ii) Mean proportion of trials where rats choose option center and the expected level of chance sampled from a binomial distribution (permutation-test p<0.002, n=8). Error bars for rats are mean±SEM. Error bars for binomial sampling are 95% confidence. Figure 26 Preference for type A is maintained across the majority of start-goal pairs. (i) Schematic drawing of the numbering for each port location. (ii) Mean proportion of trials where type A paths were selected across rats for each start-goal pair (white bars) in comparison with and the expected level of chance sampled from a binomial distribution sampling from a binomial distribution. Error bars are 95% confidence. 60 significantly higher than the mean proportion of type A trials that go through the edge (one sided, paired t-test, p < 0.002). Moreover, given that we, previously, observed a preference for selecting paths that go in the opposite direction of port arrival, in two option condition, we repeated that analysis for the three option condition. We compared mean proportion of trials that rats chose paths that go on the opposite direction of arrival at the port (Figure 27) with sampling from a binomial distribution. Animals showed preference for going in the same direction, in the three option case (permutation test, p<0.002, n=8). For the condition where six options were available, analyzing the preference for minimizing distance to light was problematic because there was the inseparable confound of the number of turns. Figure 27 Animals prefer to continue in the direction that they arrived at the port. Comparison between mean proportion of trials that rats left start port in the opposite direction of port arrival with sampling from a binomial distribution (permutation test, p=0<0.002, n=8). Error bars for rats are mean±SEM. Error bars for binomial sampling are 95% confidence. 61 3.3.4 Preference for geometrical features: number of turns Another factor that might have played a role in preferences for certain options over others was the number of turns. If the aim of the animal was to reach goal as fast as possible, making a turn physically implied a reduction in velocity which could introduce a cost to certain options. In order to investigate if animals had preference for doing less turns, we analyzed trials where three shortest path options were available and where animals selected between paths that take one or two turns. Within trials that require one turn, we selected only the trials where there was a divergence point between the path with one turn and the path with two turns. We observed that rats preferred paths that implicate less turns, both in the trials where three options were available (Figure 28) (permutation test, p<0.002, n=8). Figure 28 Animals prefer paths with one turn. (i) In three option trials, there were two types of conditions: two paths that require one turn and one path that required two turns .(ii) Comparison between mean proportion of trials across the population where option with one turn was selected and the expected level of chance sampled from a binomial distribution (permutation test, p<0.002, n=8). Error bars for rats are mean±SEM. Error bars for binomial sampling are 95% confidence. 62 Additionally, we quantified if there was a difference between the intervals of time that rats took to travel through paths with different number of turns. We calculated the mean movement time for each option type (Figure 29). The mean movement time for one turn was significantly lower than for paths where rats took two turns (one-sided, paired t-test, p<0.001, n=8). As previously, for the condition where six options were available, analyzing the preference for number of turns was problematic because there was the inseparable confound of minimizing distance to light. 3.3.5 Competition between options In the condition where three options were available, between the two paths that minimized the Euclidean distance to the light there was one which had less turns, hence, we would not expect that those would be competing choices. Nonetheless, in the condition where six options were available animals were presented with trials where, there may have been competition between options. In the six options condition, the preference for minimizing Euclidean distance to the light and the preference for minimizing number of Figure 29 Animals are faster to reach the goal when taking a path with one turn. Comparing mean movement time across population between trials where animals took a one turn path and trials where animals took a two turn path (paired t-test, p<0.001, n=8). Errorbar is SEM. 63 turns were competing because the paths that best minimize the Euclidean distance to light are the ones with more turns (Figure 30). At this point we could put forward a set of predictions for preferences in the six option condition given these minimization factors and then check if these would match rat’s behavior (Figure 31), seemingly to what was previously observed for the three option condition (3.3.6). We observed that the population data did not match the predictions associated to each of the two minimization factors. For instance, according to the minimization Euclidean distance, paths A and B should never be chosen, which animals did, similarly, according to the minimization of number of turns, paths B and F should never be chosen. Figure 30 Trial type A minimizes number of turns and trial type C minimizes Euclidean distance to the goal. Schematic drawing of two example paths, in the six option condition, where A would minimize the number of turns but not Euclidean distance to goal and where C would minimize Euclidean distance to goal but not the number of turns. 64 In the end, the path that was the best compromise between the two factors Figure 31 Different factors define different expected proportions for each option. (i) Example start-goal where six shortest paths with differentiated geometry (A, B and C and their symmetrical paths D, E, F no represented) were available (left panel) and population mean proportion of trials corresponding to each option. (ii) Choice probabilities associated to each intersection given the criterion of minimization of Euclidean distance (left panel) and expected proportions for each option, given the derived conditional probabilities (right panel). (iii) Choice probabilities associated to each intersection given the criterion of minimization of number of turns (left panel) and expected proportions for each option, given the derived conditional probabilities (right panel). 65 was path B which did not match the most preferred choice by the animals. In order to quantify with more detail the patterns of choices in the six option condition, firstly, we quantified the mean proportion of trials where rats choose type A trials, after the having crossed the first segment of path (Figure 32). Symmetrical trials were collapsed so that the top three options were mapped to the three bottom options. This case is the only symmetry axis across options, that is, there is no path with same geometry that could go through the center or through the edge. The aim of this mapping was to make it equivalent in conditions to the analysis for three options. Figure 32 Rats don't prefer paths that minimize Euclidean distance to goal. (i) Schematic drawing of an example start-goal pair where, after the first initial segment, type A path put rat further away from light and type B path put rat further away from light, in terms of Euclidean distance. (ii) Comparison between mean proportion of trials where rats choose type A trials and sampling from a binomial distribution (permutation test, p <0.002, n=8). Errorbar is SEM. 66 We observed that rats preferred to take type A paths bellow chance (permutation test, p <0.002, n=8). For the six option condition, animals did not show preference for staying close to the light position. Second, we quantified the mean proportion of trials where rats could choose between taking one, two or three turns paths. Animals prefer to take paths with one turn (Figure 33) (one-way ANOVA, post-hoc Tukey-Kramer test, p < 0.001, 1 turn x 2 turns, p <0.001, 1 turn x 3 turns, p<0.001). We also quantified movement time as a function of the number of turns of the path (Figure 34) and there was not a significant difference between the movement times for paths with different number of turns (one-way ANOVA, p= 0.4760, n=8). Figure 33 Animals prefer paths with one turn. (i) In six option trials, there are three types of paths: paths that require one turn, paths that require two turns and paths that require three turns (ii) Comparison between median proportion of trials across the population for each option with associated different number of turns (one-way ANOVA, post-hoc Tukey-Kramer, p= 2.2743e-06, 1 turn x 2 turns, p< 0.001, 1 turn x 3 turns, p<0.001, n=8). Errorbar is SEM. 67 Going back to the question of what animals would do if there was a conflict between minimizing Euclidean distance and the number of turns, it seemed that animals preferred to take paths with less turns over minimizing the Euclidean distance to the light, even though movement time was not different for paths with different number of turns. 3.3.6 Qualitative framework We identified what were two potentially key features of behavior. It seems that animals preferred to choose paths that minimized Euclidean distance to the goal and paths that minimize the number of turns. On that account, we could ask: how well can these premises explain all the observed choices? In order to answer this question we derived analytically the expected probabilities associated to different biases and compared those with the behavioral results. For each intersection, depending on each bias factor, we defined the probability of choosing one of the divergent segments according to each minimization factor and then we calculated the respective expected Figure 34 Movement time is independent of number of turns. Comparing mean movement time across population between trials where animals took one turn paths, two turn paths and trials where animals took three turn paths (one-way ANOVA, p= 0.4760, n=8). Errorbar is SEM. 74 At this point, it remains to be better understood which of the above described algorithms are being implemented. Do animals maintain a look up table which they consult at the beginning of each trial to consider the set of potential action sequences to reach the goal? Alternatively, do animals make sequential decisions along the path taking advantage of each choice point? Or is a combination of these two algorithmic solutions, where animals leave the start port with a path in mind but then they take the choice point as a decision affordance to re-evaluate previous choice? Certainly, investigating behavioral correlates of choice dynamics could help shed light into these questions, as it has been shown by the study of reaching and eye trajectories in equivalent contexts such as motor planning (Ray et al. 2014; Kaufman et al. 2015; Cos et al. 2014; Song & Nakayama 2008). 75 4 DYNAMICS OF CHOICE I feel that the mind is almost like a maze where each street has un undistinguishable name where each choice is made at the last moment Sergio Godinho, in “The maze” (Freely translated by ARF) 76 4.1 Introduction In the previous chapter, we investigated discrete choices in order to understand which strategies animals were using to solve Pombal’s maze paradigm and discussed which decision variables were animals taking into account. In this chapter, we investigated dynamics of action such as latency to start movement, velocity and trajectory across types of paths with the aim to relate these dynamics with decision-making processes. More specifically, we wanted to determine if trajectory and/or velocity profiles could reveal correlations with conflict between potential actions or commitment with a choice. The aim was to gain a better understanding of the underlying strategy of decision. Our hypothesis was that if animals were weighing options sequentially across choice points we would not be able to separate trajectories of different types of paths until animals reached choice points, similarly with the phenomenon of spatial averaging described in motor planning Van Der Stigchel & Theeuwes 2006; Eggert et al. 2002; Theeuwes et al. 1998; additionally, we hypothesized that if animals were deciding sequentially, we would expect a velocity decrease while reaching a choice point, similarly with what was observed in a task where reaction time increased when last minute choice was made available (Kaufman et al. 2015). First, we will summarize the dynamics of the animal’ behavior in different trial types. Secondly, we will investigate the hypotheses about dynamics of choice above described. 4.2 Methods 77 4.2.1 Spatial window of interest For this chapter, we mainly focused on a specific window of interest to analyze behavior (Figure 37). Mainly, the aim was to study specifically what happened just before the first choice point. In terms of conditions, the focus was the case of three options where, at the first choice point, there are only two possible symmetrical paths diverging. 4.2.2 Spatial data alignment and binning In order to allow for comparison between different start-goal pairs, trajectories were linearly transformed to be aligned to the x,y coordinates of the center of port 1 (left most bottom port), to always go upwards and to the left (Figure 38). Afterwards, raw trajectories and velocities (Figure 39) where binned in the y direction (Figure 40). Figure 37 Window of interest for behavioral analysis. Schematic drawing of the spatial window used for analysis (red box) which corresponds to the space between leaving the start port and the first intersection (grey dashed box). 78 It was inherent to the binning process the reduction of information over the chosen dimension (y in this case) so it meant that for analyzing parts of the trajectory where the dominant dimension of movement is along x dimension, this binning was not appropriate. Figure 38 Linear transformation of trajectories for spatial alignment. i) Two example trials where trajectories was linearly transformed to get the trajectories all aligned ii) Transformed trajectories, aligned on the same position. Figure 39 Raw velocities and corresponding trajectories. (i) Trajectory profile for three different trials, where each color corresponds to a trial. (ii) Velocity profiles correspondent to trajectories in (i) with same color match. 79 4.2.3 Velocity exclusion criterion In some trials, trajectories show that rats left the choice point ROI and then re-entered it (Figure 41). This criterion was defined as a trial exclusion factor Figure 41 Example trial dynamics profile where one rat left the choice point ROI and then reentered it. (i) Trajectory corresponding to a trial where rat left the intersection ROI and re-entered it (ii) derived velocity. Figure 40 Examples of binned trajectories and corresponding velocities. (i) Trajectories binned over y for all trials of one start-goal pair for the same rat (grey traces) with associated mean (light blue) (ii) Corresponding velocities with the mean trace (light blue). 80 because these trials could be, potentially, behavioral correlates of changes of the decision (Burk et al. 2014; Song et al. 2008). These trials represented less than 1% of overall trials per start-goal pair (data not shown). 4.2.4 ROC analysis In order to determine discriminability between two different types of paths we used receiver operant characteristic curve and the respective measure of discriminability, area under the curve (AUC) (Marill & Electronics 1956). This analysis was used both for discriminating trajectories or velocities. More specifically, first, we separated trials for each start-goal pair by path type; second, we performed the ROC analysis over the two types of path per 5cm bin of y; third, we determined the mean AUC as a function y across star-goal pairs for each rat and, finally, we averaged across rats. Each startgoal pair had to have at least 7 trials of each of the two conditions and both distributions had to have the same size. For that, we determined the smallest sample size between the two conditions and used the n samples from the distribution with more samples, where n corresponded to the size of the smallest distribution. Given that this type of analysis is performed as a function of spatial location, for the comparison between overlapping path types we paired each start-goal pair for one of the path types with the Figure 42 Pairing of start-goal pairs according to spatial overlap. (i) Example comparison between two overlapping path types where paths start in the start port and completely overlap in space (ii) Two examples of pairing between two trial types where paths from the two different types where overlapping in space. 81 corresponding overlapping start-goal pair for the other path type (Figure 42). This was to account for potential effects with image distortion errors (0). 4.3 Results For decision-making processes it has been shown that time related measures such as reaction time and movement time have proven very informative behavioral correlates (Barack & Gold 2016). Seemingly, we considered latency to leave start port ROI, after light was on, as a potential decisionmaking correlate. 4.3.1 Latency to leave the start port We started by comparing the median latencies to leave start port in trials where animals selected shortest path in comparison with trials were animals did detours (Figure 43). The hypothesis was that if, in general, animals were aiming at reaching the goal as fast as possible, if trials where animals did detours were different already at beginning of the trial, indicating an explicit Figure 43 Animals are faster to leave start port when they are about to do a shortest path. Mean latency to leave start port for trials where rats choose shortest paths or detours (paired t test, one-sided, p < 0.001, n=8). Errorbars are SEM. 82 disengagement in those trials. Rats were, on average, significantly faster to leave the start port in shortest path trials, in comparison with trials where rats chose a detour path (paired t test, one-sided, p<0.001, n=8). This could be interpreted as if animals were less motivated to pursue reward, from the beginning of the trial, when they were about to choose a detour. Alternatively, it could be thought that this were cases of trials were visual stimulus was not readily detected, for instance, due to animal being bent over the edge of maze; we considered that, as soon as the animal left the start port, independently of the position, light would be visible so taking a shortest path would still be possible. Within trials where animals selected shortest paths, we considered that latency to leave start port was potentially associated with the difficulty of making a decision when more options were involved (Churchland et al. 2008). Hence, we hypothesized that, if there was an associated cognitive weight of considering multiple options than latency to leave start port would increase with the number of optimal options. Accordingly, we compared mean latency to initiate movement as a function of the number of options (Figure 44). Multiple comparisons between pairs of number of options show significant differences except between two options and three options condition (two-way ANOVA, factor number of options, p<0.001, post-hoc Tukey Kramer, *p<0.05, **p<0.01 n=8). Nevertheless, the number of options is correlated with distance to goal so we separated trials where only one option was available but distance to goal was different (Figure 45i). The aim was to determine if, independently from the number of options, there was an effect associated to distance to the goal. More specifically, we hypothesized that latency to leave start port would increase with the Euclidean distance to the light. 83 We compared mean latencies between trials where, in one case, Euclidean distance to the goal was one maze segment and, in the other case, Euclidean distance to the goal was two maze segments but both conditions only comprised one shortest path to the goal (Figure 45ii). We observed that animals were significantly faster to leave start port when the goal was closer (paired t-test, p= 0.0059, n=8). This would argue for the hypothesis that the latency to initiate movement was correlated with difficulty in discriminating of light position. One alternative interpretation for this observation was that animals were more motivated to pursue an easier goal (Reppert et al. 2015). Figure 44 Animals take more time to leave the start port ROI when more options are available. (i) Schematic drawing of examples of start-goal pairs where one, two, three or six options are available to reach the goal (ii) Comparison between mean latencies to leave start port as a function of number of options (two-way ANOVA, factor number of options , p<0.001, post-hoc Tukey Kramer, *p<0.05, **p<0.01 n=8). Errorbar is SEM. 90 (Figure 50iv). The discriminability progressively increased until a maximum at 45 cm, that is, 9.5 cm away from the choice point (permutation test, *p<0.01). On one way, it seems that arriving at the choice point when it offered more than one optimal option seemed to maximally decrease velocity which argued for considering this observation as a behavioral correlate of competion between different options. Moreover, velocity associated to trials Figure 50 Animals slow down more when choice point offers two optimal path options. (i) Schematic drawing of an example type A path and type B, where these two path types are mutually exclusive in the same trial. (ii) Mean velocity for type A path (solid line) and for type B path (dashed line) as a function of position, for one rat across all start-goal pairs (n=16). The shaded area is SD (iii) Mean velocity for type A path (solid line) and for type B path (dashed line) as a function of position across the population (n=8). The shaded area is SEM. (iv) ROC area under the curve as a function of y position between distribution of velocities of type A path and distribution of velocities of type B path across population (n=8) compared with shuffled data (light grey shade) (permutation test, *p< 0.01). Shaded dark area is SEM. 91 where choice point only offered one optimal option was significantly higher as soon as the rat left the start port ROI (permutation test, *p<0.01, n=8). This could be interpreted as an increased motivation to pursue a closer goal or an effecet of the light position being closer and hence, easier to discriminate. To try to disentangle if decrease in velocity was a light discrimination correlate, we compared path types where the first choice point offered more than one non-optimal option but the distance to light was different (Figure 51). Hence, we would expect that animals would slow down more while reaching the choice point if the light was further away as in being harder to discriminate. There is no significant difference between the velocity profiles of the two types of paths which makes it less probable that the decrease in velocity is due to visual discrimination. Before, it was also considered the hypothesis that this decrease in velocity could be relate to the physical presence of the port on the floor. Against this hypothesis is the observation that the decrease in velocity at the arrival of the choice point was bigger when there were more optimal path options. Overall, it could be argued that the observed decrease in velocity is related to a deliberation process that takes place due to the presence of choice point which affords a decision. 92 4.3.4 Trajectories for competing paths If that is the case then we would expect that the trajectories of competing path options would only become separable after the deliberation process at the arrival of the choice point as a correlate for resolution of the decisionmaking process. This means that it was expected to find significant Figure 51 Distance to the light does not affect velocity decrease at the choice point (i) Schematic drawing of an example type A path and type B. (ii) Mean velocity for type A path (solid line) and for type B path (dashed line) as a function of position, for one rat across all start-goal pairs (n=16). The shaded area is SD (iii) Mean velocity for type A path (solid line) and for type B path (dashed line) as a function of position across the population (n=8). The shaded area is SEM. (iv) ROC area under the curve as a function of y position between distribution of velocities of type A path and distribution of velocities of type B path across population (n=8) compared with shuffled data (light grey shade) (permutation test, *p< 0.05, n=8). Shaded dark area is SEM. 93 discriminability between two types of trajectories only at the arrival of the choice point. Hence, we quantified at which point in space was possible to discriminate between trajectories of two concurrent path types. The rationale was such that if it could be determined a point where different path types could be separated using trajectory, this was an indication that a choice had been made. Moreover, we would expect that considering the idea of spatial averaging (Van Der Stigchel & Theeuwes 2006; Eggert et al. 2002; Theeuwes et al. 1998) that the trajectories would overlap until a divergence for one of the targets. More specifically, we compared two potentially competing paths types in a condition where three path options where available, each path was mutually exclusive option in a condition such that each type would overlap and geometrically diverge at a choice point (Figure 52). Mean discriminability between two trajectories was first significant 40 cm after animals left the start port ROI (permutation test, *p<0.01, n=8) which corresponds to 15.5 cm before arrival at the choice point. Given that the distance between the limit of the start port ROI and the limit of the choice point is 54.5 cm, this means that animals traveled almost two thirds of the of the track, more precisely 73% until it was possible to anticipate which path they were going to take. Furthermore, we did the same analysis to determine the divergence point of trajectories but for two competing paths from the condition where six options are available (Figure 53i). Discriminability between trajectories was first significant 40 cm after leaving the start port ROI. In both conditions of three and six options, the point in space where two competing trajectories diverge is the same. 94 This result is unexpected because it seems that trajectory divergence, which we hypothesized as a behavioral correlate of commitment to a given choice, was taking place 5 cm before the maximum decrease of velocity at the choice point, which we hypothesized as being a behavioral correlate of competition between potential actions at the choice point. Figure 52 Trajectories from competing options in a three option condition diverge 20 cm after animal has left the start port ROI. (i) Schematic drawing of an example type A path and type B, where these two path types are mutually exclusive in the same trial. (ii) Mean trajectory for type A path (solid line) and for type B path (dashed line) as a function of position, for one rat across all startgoal pairs (n=16). The shaded area is SD (iii) Mean trajectory for type A path (solid line) and for type B path (dashed line) as a function of position across the population (n=8). The shaded area is SEM. (iv) ROC area under the curve as a function of y position between distribution of trajectories of type A path and distribution of trajectories of type B path across population (n=8) compared with shuffled data (light grey shade) (permutation test, *p< 0.01). Shaded dark area is SEM. 95 4.4 Discussion In this chapter, we investigated dynamics of action such as latency to start movement, velocity and trajectory across types of paths with the aim to relate these dynamics with decision-making processes. More specifically, we wanted to determine if trajectory and/or velocity profiles could reveal correlations with conflict between potential actions or commitment with a choice. The main findings are summarized below: Figure 53 Trajectories from competing options in a six option condition diverge 20 cm after animal has left the start port ROI. (i) Schematic drawing of an example type A path and type B, where these two path types are mutually exclusive in the same trial. (ii) Mean trajectory for type A path (solid line) and for type B path (dashed line) as a function of position, for one rat across all start-goal pairs (n=16). The shaded area is SD (iii) Mean trajectory for type A path (solid line) and for type B path (dashed line) as a function of position across the population (n=8). The shaded area is SEM. (iv) ROC area under the curve as a function of y position between distribution of trajectories of type A path and distribution of trajectories of type B path across population (n=8) compared with shuffled data (light grey shade) (permutation test, *p< 0.01). Shaded dark area is SEM. 96  Latency to leave start port – we found that that animals left the start port faster if they were about to do a shortest path than if they were about to do a detour. We interpreted this observation as a motivation correlate such that, at the beginning of the trial, animals had already decided to take a shortest path or not. On the other hand, the difficulty in discriminating the light at the beginning of the trial due to body position and orientation could only explain a difference in latency to leave start port between shortest paths and detours on the other direction. If rats would take less time to leave start port when about to do a detour, it could be argued that they left the start port without enough information about the goal and, hence, choose a nonoptimal path at the beginning that compromised the overall shortest path. Within shortest path trials, we found that animals take more time to leave start port in trials where six options are available which could indicate that the number of options available could be increasing the time to deliberate at the start port. Nevertheless, there is no difference in latency between trials where two options or three options are available. Given that the number of options also increased with the distance to the goal which introduced a potential confound. Accordingly, in trials where the distance to the goal is different and there is only one option animals take more time to leave the start port which could be either associated to motivation or difficulty in discriminating goal location when the goal is further away.  Bell-shaped velocity – we found that animals exhibit a bell-shaped velocity profile when travelling between two ports separated by one maze segment, as expected from motor planning literature (Flash & Hogan 1985). 97  Center avoidance – we found that animals reach a higher velocity peak in trials that take them through the center which is congruent with data observed in the open field (Lipkind et al. 2004) when the distance to goal is of two segments and there is only one optimal option. This result did not hold through for the case where three options were available. One possible explanation for this type of incongruence was that the center avoidance factor was being suppressed by other factors such minimizing Euclidean light.  Velocity at the choice point - animals slowed down while arriving at the choice point even though there was only one available optimal option at that point. This observation could be interpreted as a consequence of the need to check for the light position, given the opportunity to reevaluate a previous decision. Nevertheless, we observed no difference between velocity profiles in terms of decrease at the choice point for two path types which goals were at different distances so we would argue that this observation is not explained by visual discrimination effect. Moreover, the decrease in velocity did not depend on the number of non-optimal choices (center vs edge) which argues for a general effect of being presented with a choice point, as if the presence of a choice point afforded a decision which interfered with the current motor plan (Kaufman et al. 2015). Adding to this, animals slowed down more if the choice point afforded more than one optimal option. A bigger decrease in velocity could indicate that animals take more time to consider between two options when both of them are optimal than when one is optimal and the other is not. This could be interpreted as if the offer of an irrelevant option for the optimal strategy was interfering with the decision to take the optimal path, equivalently as in motor planning tasks where 98 distractors interfere with the reaches (Welsh et al. 1999; Tipper et al. 1998). Hence, this observation could be a correlate of strong competition between two optimal actions or a weak competition between an optimal action and a non-optimal action (‘distractor’). It cannot be excluded that, the decrease in velocity while arriving at the choice point does not have a component associated to a biomechanical adaption (Va et al. 1996) associated to the physical presence of the port on the floor.  Trajectory divergence – we found that trajectories of competing paths diverge at 15.5 cm before arrival at the choice point. This could be interpreted as a correlate that the decision about the choice at the choice point was taken at that location in space, and consequent execution of the chosen option started. In a way, this would be equivalent to the spatial averaging phenomenon observed in ‘gobefore-you-know’ tasks (Welsh et al. 1999; Tipper et al. 1998) or in tasks where reach trajectory was used to report binary choices (McKinstry et al. 2008), where in the latter case the cue for which target to move to was not defined externally by the experimenter but by internal processes, similarly to Pombal´s maze. Alternatively, it could be argued that animals had already decided which path to take at the beginning of the trial and were withholding the action execution until it was biomechanically necessary. This seems a less probable possibility because it would implicate that animals would have to maintain the memory of the outcome of the decision until it was biomechanically necessary to execute it. Yet another possibility, is that the divergence happened before and that the potentially insufficient spatial resolution of the video frame together with the small width of the track did not allow for measurable separation of 99 the trajectories. Finally, it cannot be excluded that these correlates correspond to a bias of the animals, not reflecting the commitment towards a decision. In conclusion, it seems that both motivation and Euclidean distance to the goal may influence latency to leave start port. Center avoidance seems to have different weight on the behavior depending on the number of options and correlated Euclidean distance to the goal. Additionally, there is evidence that animals take choice points as a place that affords a decision, whether there are only optimal path options or not. In that sense, it could be that even though there is an initial decision about what path to take at the choice point, the offer of the choice point may lead animals to reconsider. The hypothesis of reconsideration is enforced by the possibility to determine before the choice point which path are animals going to take from a set of two competing options. Overall, it seems that animals may be making decisions before the choice point but given the offered opportunity to change minds, animals reevaluate their previous decision. 106 aims at understanding action selection from the perspective of spatial processing focusing on choice points and associated discrete choices; and motor planning, which aims at understanding action selection from the perspective of motor control focusing on the continuous dynamics of movement. Pombal’s maze is a cued navigation task where animals have to gather spatial information about the goal, through visual stimulus, and choose one path to get there. From the perspective of an action selection problem, animals are required to choose between different actions – going forward, turning right or left. If we would consider a stochastic choice model that selects randomly between potential actions at each choice point, as it was described in Chapter 2, it would not mimic the rats’ behavior. In order for this model to mimic the rat behavior, at each choice point the potential actions should be considered but then different type of information would bias the choice probability of each action: preference for less turns, preference for avoiding the center, preference for staying closer to the light as possible. A bias associated to turn avoidance can be generated on the spot at the choice point by learning from previous experience that turning reduces velocity or introduces energetic costs – in a stimulus-response manner (Penner & Mizumori 2012; Niv et al. 2006). Nevertheless, a bias associated to minimizing Euclidean distance would require an extra step of computation for evaluating how much each potential action would lead to staying close to the light, in the future. For determining these relationships between future actions and spatial relation to the light, we would argue would be acquired through a goal-directed strategy. It could be expected that after being over trained in the task the animals would become habitual and use a stimulusoutcome association as a rule (Penner & Mizumori 2012; Niv et al. 2006). Nevertheless, this would result into the animal having to remember a 107 possibly extensive list of stimulus-outcome associations. Another aspect that should bias the probability of choice of a potential action is the factor if the animal is already moving or not, that is, if the choice point coincides with the start location or not, in which, for instance, making a turn would have a different associated cost from when the animal is already moving. Overall, in Pombal’s maze animals are confronted with the selection of actions where different biases change the probability of choice of each of the potential actions. Overall, in order for a stochastic choice model to mimic rat’s behavior it would have to weigh over the cost and benefit of the different features of the potential actions at the choice point, engaging in a sequential choice strategy. Nevertheless, this mathematical description of the action selection problem associated to solving Pombal’s maze would require the integration of dynamical properties to provide with a more accurate description of behavior. Hence, in between closest neighbors velocity would be modelled by a bell-shaped function with a modification because animals do not fully stop at choice points, they reduce velocity and, in fact the reduction seems to be modulated by the number of optimal choices. Lastly, considering a component related to trajectories, there is an observation that seems incongruent with the hypothesis that animals make sequential choices along the path, which is the result that position on the track anticipates which choice animals are going to take at the choice point. It is not completely clear if this divergence of paths is as a correlate of commitment towards a certain choice or a bias. If it is the case that trajectory divergence correlates with commitment this could implicate that decision-making processes are not exclusive from choice points but are ongoing while animals are moving, which has been argued is the case in motor planning typical tasks (Song & Nakayama 2009) so that the potential actions are represented and interacting dynamically, allowing for flexible adaption to new potential actions or 108 changes in plan (Cisek 2012; Song & Nakayama 2009). This hypothesis would predict that even after decision has been taken and action is under execution, there is still the possibility of reconsidering (Kaufman et al. 2015) which could be seen as interference between potential actions. One observation that seems to go on this direction is the existence of change in mind type of trials, where animals left the choice point following one direction and then returned to the choice point to go in the other direction. It seems, overall, that the choice point and inherent possibility it offers the animal to make choices generates an interference with decisions previously made. This type of dynamical interaction between potential actions seems to be at the core of understanding action selection and the underlying competition mechanisms. 5.5 Final remarks and future directions The focus of this thesis was to design and develop a novel decision-making paradigm for studying behavior while animals are selecting between multiple actions. The paradigm, due to its characteristics of being based on a maze and on a trial structure where multiple options lead to goal, stands both in the fields of navigation and motor planning. Hence, we aimed at framing the problem that animals were solving considering these two frameworks. On one side, it was necessary to characterize which problem were the animals solving in terms of optimization and decision variables, in order to determine which type of actions were potentially competing to be selected. On the other side, it was necessary to better characterize the dynamics of movement with the aim to find behavioral correlates of action selection that could be integrated with the patterns of discrete choices. In the end, integrating choice patterns with dynamical correlates led to the understanding that choice points 109 can afford changes in mind, where the previous decision competes with new possibilities. This seems like a particularly relevant window for studying how potential actions dynamically interact and, hence, gaining a better understanding about the underlying mechanisms of action selection. In future directions, we suggest to do electrophysiological recordings of neural activity in motor-related areas while animals are solving Pombal’s maze. More specifically, we consider that the window between the location where divergence of trajectories occurs, until the location where the animal leaves the choice point, is particularly relevant for searching for neural correlates of action selection. We consider that one of the brain areas which could be a good candidate for recoding neural activity is M2 which is an area associated with action planning and spontaneous action initiation (Sul et al. 2012; Murakami et al. 2014; Erlich et al. 2011), considered to be homologous to primate supplementary motor areas (Reep et al. 1990; Reep et al. 1986). We would expect to find activity correlated with simultaneous representation of potential actions, similarly with what has been found in PMd (Cisek & Kalaska 2005). More specifically, we would expect to characterize the interactions between potential representations, from the moment where animals show trajectory divergence onwards. These observations could help to disambiguate if trajectory divergence is associated to commitment towards a decision or not. If that is the case, it would be relevant to see if there are normalization effects between representations of different actions, which would argue for a competition mechanism taking place between potential actions. Hence, we believe that these observations would contribute to gaining a deeper understanding about action selection mechanisms and respective underlying circuitry. 110 6 REFERENCES 111 Alexander, R.M., 1997. A minimum energy cost hypothesis for human arm trajectories. Biological cybernetics, 76(2), pp.97–105. Andersen, R.A. & Cui, H., 2009. Intention, Action Planning, and Decision Making in Parietal-Frontal Circuits. Neuron, 63(5), pp.568–583. Bailey, K.R. & Crawley, J.N., 2009. Anxiety-related behaviors in mice. In J. J. Buccafusco, ed. Methods of Behavior Analysis. Boca Raton (FL): CRC press/Taylor & Francis. Baldauf, D., Cui, H. & Andersen, R.A., 2008. The Posterior Parietal Cortex Encodes in Parallel Both Goals for Double-Reach Sequences. Journal of Neuroscience, 28(40), pp.10081–10089. Balleine, B.W. & Ostlund, S.B., 2007. Still at the choice-point: action selection and initiation in instrumental conditioning. Annals of the New York Academy of Sciences, 1104, pp.147–71. Barack, D.L. & Gold, J.I., 2016. Temporal trade-offs in psychophysics. Current Opinion in Neurobiology, 37, pp.121–125. Barnes, T.D. et al., 2005. Activity of striatal neurons reflects dynamic encoding and recoding of procedural memories. Nature, 437(7062), pp.1158–61. Basso, M. a & Wurtz, R.H., 1998. Modulation of neuronal activity in superior colliculus by changes in target probability. The Journal of neuroscience : the official journal of the Society for Neuroscience, 18(18), pp.7519–7534. Basso, M.A. & Wurtz, R.H., 1997. Modulation of neuronal activity by target uncertainty. Nature, 389(September), pp.66–69. 112 Bouwknecht, J.A. et al., 2007. Differential effects of exposure to low-light or high-light open-field on anxiety-related behaviors: Relationship to cFos expression in serotonergic and non-serotonergic neurons in the dorsal raphe nucleus. Brain Research Bulletin, 72(1), pp.32–43. Brown, W.L., 1946. The effects of a common start-finish locus on orientation and behavior in a multiple-t maze. , 1, pp.331–338. Burk, D. et al., 2014. Motor effort alters changes of mind in sensorimotor decision making. PLoS ONE, 9(3). Chapman, C.S. et al., 2010. Reaching for the unknown: Multiple target encoding and real-time decision-making in a rapid reach task. Cognition, 116(2), pp.168–176. Churchland, A.K., Kiani, R. & Shadlen, M.N., 2008. Decision-making with multiple alternatives. Nature neuroscience, 11(6), pp.693–702. Cisek, P., 2007. Cortical mechanisms of action selection: the affordance competition hypothesis. Philosophical transactions of the Royal Society of London. Series B, Biological sciences, 362(1485), pp.1585–1599. Cisek, P., 2006. Integrated Neural Processes for Defining Potential Actions and Deciding between Them: A Computational Model. Journal of Neuroscience, 26(38), pp.9761–9770. Cisek, P., 2012. Making decisions through a distributed consensus. Current Opinion in Neurobiology, 22(6), pp.927–936. Cisek, P. & Kalaska, J.F., 2005. Neural correlates of reaching decisions in dorsal premotor cortex: Specification of multiple direction choices and final selection of action. Neuron, 45(5), pp.801–814. 113 Cisek, P. & Kalaska, J.F., 2002. Simultaneous encoding of multiple potential reach directions in dorsal premotor cortex. Journal of neurophysiology, 87(2), pp.1149–54. Cleveland, W.S., 1979. Robust locally weighted regression and smoothing scatterplots. Corrado, G. & Doya, K., 2007. Understanding neural coding through the model-based analysis of decision making. The Journal of neuroscience : the official journal of the Society for Neuroscience, 27(31), pp.8178–80. Cos, I., Duque, J. & Cisek, P., 2014. Rapid prediction of biomechanical costs during action decisions. Journal of neurophysiology, 112(6), pp.1256– 66. Coulthard, E.J., Nachev, P. & Husain, M., 2008. Control over Conflict during Movement Preparation: Role of Posterior Parietal Cortex. Neuron, 58(1), pp.144–157. Coutlee, C.G. & Huettel, S.A., 2012. The functional neuroanatomy of decision making: Prefrontal control of thought and action. Brain Research, 1428, pp.3–12. Daw, N.D. & Shohamy, D., 2008. The Cognitive Neuroscience of Motivation and Learning. Social Cognition, 26(5), pp.593–620. Deaner, R.O., Khera, A. V. & Platt, M.L., 2005. Monkeys pay per view: Adaptive valuation of social images by rhesus macaques. Current Biology, 15(6), pp.543–548. Desimone, R. & Duncan, J.S., 1995. Neural mechanisms of selective visual attention. Annual Review of Neuroscience, 18(1), pp.193–222. 114 Dezfouli, A., Lingawi, N.W. & Balleine, B.W., 2014. Habits as action sequences: hierarchical action control and changes in outcome value. Philosophical transactions of the Royal Society of London. Series B, Biological sciences, 369(1655), p.20130482–. Eggert, T. et al., 2002. Differential effect of a distractor on primary saccades and perceptual localization. Vision Research, 42(28), pp.2969–2984. Erlich, J.C., Bialek, M. & Brody, C.D., 2011. A cortical substrate for memory-guided orienting in the rat. Neuron, 72(2), pp.330–343. Felsen, G. & Mainen, Z.F., 2012. Midbrain contributions to sensorimotor decision making. Journal of Neurophysiology, 108(1), pp.135–147. Flash, T. & Hogan, N., 1985. The coordination of arm movements: an experimentally confirmed mathematical model. The Journal of neuroscience : the official journal of the Society for Neuroscience, 5(7), pp.1688–1703. Gallivan, J.P. et al., 2015. Action plan co-optimization reveals the parallel encoding of competing reach movements. Nature communications, 6(May), p.7428. Gardner, R.S. et al., 2013. A secondary working memory challenge preserves primary place strategies despite overtraining. Learning & memory (Cold Spring Harbor, N.Y.), 20(11), pp.648–56. Georgopoulos, a P., Kettner, R.E. & Schwartz, a B., 1988. Primate motor cortex and free arm movements to visual targets in three-dimensional space. II. Coding of the direction of movement by a neuronal population. J.Neurosci., 8(8), pp.2928–2937. 115 Gibson, J.J., 1979. The ecological approach to visual perception, Houghton Miffli. Glimcher, P.W. et al., 2009. Neuroeconomics, Glimcher, P.W. & Sparks, D.L., 1993. Representation of averaging saccades in the superior colliculus of the monkey. Experimental Brain Research, 95(3), pp.429–435. Gold, J.I. & Shadlen, M.N., 2007. The neural basis of decision making. Annual review of Neuroscienceeuroscience, 30, pp.535–574. Haith, A.M., Huberdeau, D.M. & Krakauer, J.W., 2015. Hedging Your Bets: Intermediate Movements as Optimal Behavior in the Context of an Incomplete Decision. PLoS Computational Biology, 11(3), pp.1–21. Hen, I. et al., 2004. The dynamics of spatial behavior: How can robust smoothing techniques help? Journal of Neuroscience Methods, 133(12), pp.161–172. Hikosaka, O., Nakamura, K. & Nakahara, H., 2006. Basal ganglia orient eyes to reward. Journal of neurophysiology, 95(2), pp.567–584. Hoshi, E. & Tanji, J., 2007. Distinctions between dorsal and ventral premotor areas: anatomical connectivity and functional properties. Current Opinion in Neurobiology, 17(2), pp.234–242. Houk, C.J., 1995. Distributed modular architectures linking basal ganglia, cerebellum, and cerebral cortex their role in planning and controlling action.pdf. Cerebral Cortex, pp.95–110. Hull, C.L., 1943. Principles of Behavior: an introduction to behavior theory, Appleton-Century. 122 Taube, J.S., 2007. The head direction signal: origins and sensory-motor integration. Annual review of neuroscience, 30, pp.181–207. Theeuwes, J. et al., 1998. Our eyes do not always go where we want them to go: Capture of the Eyes by New Objects. Psychological Science, 9(5), pp.379–385. Thorn, C.A. et al., 2010. Differential dynamics of activity changes in dorsolateral and dorsomedial striatal loops during learning. Neuron, 66(5), pp.781–795. Tipper, S.P., Howard, L.A. & Houghton, G., 1998. Action-based mechanisms of attention. Philosophical transactions of the Royal Society of London. Series B, Biological sciences, 353(1373), pp.1385– 1393. Tolman, E.C., 1948. Cognitive maps in rats and men. The Psychological Review, 55(4), pp.189–208. Tolman, E.C., 1938. The determiners of behavior at a choice point. In Hothersall, ed. History of Psychology. p. 494. Tolman, E.C. & Postman, L., 1953. Learning. Annual Review of Psychology, 5, pp.27–56. Tversky, A. & Kahneman, D., 1981. The Framing of Decisions and the Psychology of Choice. Science, 211(4481), pp.453–458. Uchida, N. & Mainen, Z.F., 2003. Speed and accuracy of olfactory discrimination in the rat. Nature neuroscience, 6(11), pp.1224–1229. Va, R. a, Ebensperger, L. a & Bozinovic, F., 1996. The influence of habitat on travel speed , intermittent locomotion , and vigilance in a diurnal 123 rodent. Behavioral Ecology, 13(2), pp.182–187. Welsh, T.N., Elliott, D. & Weeks, D.J., 1999. Hand deviations toward distractors. Evidence for response competition. Experimental Brain Research, 127(2), pp.207–212. Whitlock, J.R. et al., 2008. Navigating from hippocampus to parietal cortex. Proceedings of the National Academy of Sciences, 105(39), pp.14755– 14762. Wolpert, D.M. & Landy, M.S., 2012. Motor control is decision-making. Current Opinion in Neurobiology, 22(6), pp.996–1003. Wunderlich, K., Rangel, A. & O’Doherty, J.P., 2009. Neural computations underlying action-based decision making in the human brain. Proceedings of the National Academy of Sciences of the United States of America, 106(40), pp.17199–17204.