Verhalten: Analyse und Modellierung Björn Brembs Neuroinformatik Verhalten: Analyse und Modellierung Björn Brembs

Computational Neuroscience

Ivan Petrowitsch Pavlov

Pavlov’s Hunde Valiet Tungus Barbos

Klassisches Konditionieren UR Pavlov‘s Hund

Zeitliche Paarung von CS und US

Komplexere Assoziationen (1960er) Overshadowing Training Test CS1+CS2+US CS1 alleine CS2 alleine Blocking Prä-Training Test Training CS2+US CS2=100% CS1+CS2+US CS1 alleine CS1+US CS1=100% CS2 alleine

Theoretische Formulierungen (1970er) Experiment/Analyse: „Lernen ist von der zeitlichen Paarung der Reize abhängig“ Hypothese: „Der prädiktive Wert des CS könnte für das Lernen entscheidend sein“ also: Lernen findet immer dann statt, wenn Erwartungen verletzt werden Formalisierte Hypothese: , mit DV - Lernrate, l - tatsächl. US - erwarteter US Rescorla-Wagner-Regel (1972): Mit ai - Salienz von CSi (0<a<1), b - Salienz des US (-1<b<1)

Rescorla Wagner Regel Anfangsbedingungen: Vmax = 100 (willkürlich) Vereinfacht: DVCS=c(Vmax-Vall) Anfangsbedingungen: Vmax = 100 (willkürlich) Vall = 0 (kein Lernen) Vcs = 0 (kein Lernen) c = 0.5 (willkürlich)

First Conditioning Trial Trial c (Vmax - Vall) = ∆Vcs 1 0.5 * 100 - 0 = 50

Second Conditioning Trial Trial c (Vmax - Vall) = ∆Vcs 2 .5 * 100 - 50 = 25

Third Conditioning Trial Trial c (Vmax - Vall) = ∆Vcs 3 .5 * 100 - 75 = 12.5

4th Conditioning Trial Trial c (Vmax - Vall) = ∆Vcs 4 .5 * 100 - 87.5 = 6.25

5th Conditioning Trial Trial c (Vmax - Vall) = ∆Vcs 5 .5 * 100 - 93.75 = 3.125

6th Conditioning Trial Trial c (Vmax - Vall) = ∆Vcs 6 .5 * 100 - 96.88 = 1.56

7th Conditioning Trial Trial c (Vmax - Vall) = ∆Vcs 7 .5 * 100 - 98.44 = .78

8th Conditioning Trial Trial c (Vmax - Vall) = ∆Vcs 8 .5 * 100 - 99.22 = .39

Extinktion

1st Extinction Trial Trial c (Vmax - Vall) = ∆Vcs 1 .5 * 0 - 99.61 = -49.8

2nd Extinction Trial Trial c (Vmax - Vall) = ∆Vcs 2 .5 * 0 - 49.8 = -24.9

Hypothetical Acquisition & Extinction Curves with c=.5 and Vmax = 100

Acquisitions & Extinktions Kurven mit c=0,5 vs. c=0,2 (Vmax = 100)

Rescorla-Wagner Spreadsheet

R-W sagt Übererwartung voraus DVCS=c(Vmax-Vall) Blocking Wenn Vall=Vmax=100 wird CS2 kaum noch assoziative Stärke erreichen (Blocking) Übererwartung Bei Vall>Vmax wird DV<0! Prä-Training Test Training CS2+US CS2=100% CS1+CS2+US CS1 alleine CS1+US CS1=100% CS2 alleine Prä-Training1 Test1 Prä-Training2 Test2 Training Test3 CS1+US CS1=100% CS2+US CS2=100% CS1+CS2+US CS1+CS2 CS1+CS2+CS3+US CS3

Computational Neuroscience

PER Konditionierung

R-W und Vummx1

Blocking und Dopamin

Seit Rescorla und Wagner Sutton und Barto 1990: Reinforcement learning, temporal difference models „actor-critic model“ assoziativen Lernens http://en.wikipedia.org/wiki/Temporal_difference_learning http://www.scholarpedia.org/article/Temporal_difference_learning Reinforcement Learning: An Introduction Richard S. Sutton and Andrew G. Barto: http://www.cs.ualberta.ca/~sutton/book/the-book.html http://machinelearning.org

Analyse von Spontanverhalten http://brembs.net/spontaneous/ http://www.plosone.org/doi/pone.0000443 Thank you very much for inviting me here and giving me an opportunity to present some of our research. We work in two model systems, the marine snail Aplysia and the fruit fly Drosophila. Today, I will only talk about the most recent developments in Drosophila. As you can see from the cartoon, it is a common concept that animals are in essence organisms which primarily react to external stimuli.

Die Organisation von Verhalten wird meist als Input/Output Modell beschrieben Brain function is ultimately best understood in terms of input/output transformations and how they are produced. Michael Mauk (2000): Nature Neuroscience 3, 649-651 This view is pertinent also among scientists and is reflected in a quote from Mike Mauk at my former department at the University of Texas in Houston: "Brain function is ultimately best understood in terms of input/output transformations and how they are produced". A general model of behavior is often depicted as a simple input/output model, in which the sensory apparatus perceives environmental stimuli and transmits the sensory data to the sensorimotor link which maps the incoming sensory data to motor commands. The motor commands control the biomechanical apparatus which produces the behavior. A prediction derived from such a model is that constant input should produce constant output. However, because of inevitable noise at all stages of the model, the output in reality is rarely constant but rather noisy. Such a more realistic model implies that not every environmental situation necessarily leads to the same behavior, but any given behavior is a response to a specific situation. For example, a person committing a murder is doing so in response to the specific sequence of situations the murderer has experienced leading up to the crime. Of course, in such a view, the question arises of how responsible the murderer is for his crime. Now you might say that this model is a scientific simplification and nobody would possibly use it in such a case. However, the Dana foundation, the ACLU and the AAAS all have sponsored meetings on "Neuroscience and the Law", where the participants discussed the impact of Neuroscience on culpability, including exactly such cases as the one we are talking about. However, one can show that already in small brains, such a model is indeed oversimplified. Konstanter Input Verrauschter Output Konstanter Output …aber jedes Verhalten ist eine Antwort auf einen Reiz Nicht jeder Reiz bedingt die gleiche Antwort…

Drosophila am Drehmoment-Kompensator Selbst bei konstantem sensorischem Eingang verhält sich die Fruchtfliege variabel spike Tethered Drosophila melanogaster performs a variety of behaviors with the abdomen, the legs, the wings and the antennae, while head and thorax remain fixed in space. Suspended at the torque compensator, we can measure one of these behaviors: yaw torque, the force the fly produces when attempting to turn (left or right) around its vertical body axis. Keeping the environment constant by surrounding the fly with a white, featureless panorama, we can confirm that the fly's behavior is indeed not constant at all. Instead, yaw torque fluctuates seemingly randomly from left to right. Zooming in onto a short 5min stretch, we can see that there are two components to this variability: a baseline fluctuation and superimposed, fast torque spikes. These spikes correspond to so-called body-saccades in free flight. As you might have noticed, free flying flies do not fly curves like airplanes, but rather zig-zag around. Each zig or zag corresponds to one such torque spike at the torque meter. baseline

Zustandsraum-Rekonstruktion Koordinaten-Einbettung Zeitreihe: 80, 60, 40,…… Einbettungs-Dimension: 3 Drei Datenpunkte bestimmen einen 3D Vektor: All computations: Alexander Maye, UKE Hamburg

Inter-Spike-Interval (ISI-)Analyse Fliegen gegen Poisson Fliegenverhalten sieht nicht wie einfaches Rauschen aus Geometric Random Inner Products: GRIP Inter-Spike-Interval (ISI-)Analyse Using the time intervals between the torque spikes we can evaluate the temporal structure of fly turning behavior. Every blue stripe denotes a spike extracted from data such as that depicted on the previous slide. The input/output model predicts a Poisson process for spike production. Much like the hiss of static from a radio tuned between stations, a Poisson process is a random process, so flies should be random number generators. A mathematical method has recently been developed which can quantify the randomness of a time series. The Geometric Random Inner Products or GRIP procedure computes the deviation from ideal randomness. The graph shows this deviation on the Y-axis. The computer-generated Poisson series is not perfect but very close to random. However, flies are clearly even worse random number generators and definitely do not show the predicted random noise in their behavior. Information theory tells us that any deviation from random entails information in a number sequence and that in principle at least parts of the sequence can be predicted using this information. Fliegen als Zufallsgeneratoren? All computations: Alexander Maye, UKE Hamburg (Maye et al. 2007)

Statistik erster Ordnung ISI-Verteilungen haben ein „schweres Ende“ Potenzgesetz: Skaleninvarianz All computations: Alexander Maye, UKE Hamburg (Maye et al. 2007)

Unabhängigkeit der Daten Mischung der Einzelwerte ähnelt den Originaldaten nicht All computations: Alexander Maye, UKE Hamburg (Maye et al. 2007)

Nichtlinearität? Noch nicht einmal sehr komplexe stochastische Modelle reichen aus Branched Poisson Process: BPP Root mean square fluctuation of displacement Offensichtlich scheint die Nichtlinearität ein wichtigerer Faktor zu sein als die Zufälligkeit! All computations: Alexander Maye, UKE Hamburg (Maye et al. 2007)

Nichtlineare Vorhersagen: S-Maps Mathematisch instabile, nichtlineare Prozesse steuern das Flugverhalten von Drosophila Korrelation Logistische Gleichung: nichtlinear linear/stochastisch Therefore we used a method called nonlinear forecasting to further analyze turning behavior. The S-maps procedure works similar to a weather forecast: mathematical models are derived from the first half of the time series to predict or forecast the second half of the series. You then calculate a correlation coefficient (rho) between the actual and the predicted values for forecasting just one time step ahead. Similar to how ou would judge the accuracy of a weather forecast, rho gives you an estimate of the accuracy of your model. In addition, the S-maps procedure uses a weighting parameter (theta) to denote how nonlinear the model was, with 0 being linear and 4 being most nonlinear. So rho tells you how good the model was and theta which model was used. Time series generated by nonlinear processes show an increase in rho with theta, random or linear time series show either a decrease or at least stable rho with theta. Analyzing ISIs with this method yields a nonlinear signature, albeit at a very low overall rho. In order to exclude information-loss due to spike extraction, we repeated the procedure with the raw yaw torque data. Now we find enhanced rho together with a very clear nonlinear signature. However, the brain is full of nonlinear feedback and feedforward circuits. Maybe what we find is just a reflection of this well-known fact? To test this hypothesis, we adapted a biologically plausible software agent from the literature with known nonlinear properties. It consists of a nonlinear activator (maybe located in the brain of the fly) which activates two mutually inhibiting nonlinear turn-generators (which may represent central pattern generators in the thoracic ganglia). Subtracting left from right turn-generator yields a torque signal. Not surprisingly, the resulting signal shows a strong nonlinear signature and the agent can be tuned such that the signal looks very similar to fly behavior. However, under these conditions, the agent does not show the nonlinear signature and if the agent is tuned so that it shows the nonlinearity, the output doesn't resemble fly behavior at all. Apparently, the agent is not as good a model for fly behavior as we thought. But maybe we can nevertheless learn something from this failure? The agent's components are modeled by the so-called logistic map, a well-known recursive function. The output s of the function (depicted on the Y-axis) is largely dependent on the parameter lambda, depicted on the X-axis. For low lambda, the agent's output looks similar to fly behavior, but is linear, because the function shows linear behavior. For larger lambda, the function becomes unstable and the agent shows the nonlinear signature. This reminds us that nonlinear systems can display linear behavior. We thus can conclude that spontaneous fly turning behavior is controlled by brain-circuits which are tuned to operate as nonlinear, unstable processes. Otherwise, we would not detect a nonlinear signature in the S-maps. Nichtlinearität All computations: Alexander Maye, UKE Hamburg (Maye et al. 2007)

Ein neues Verhaltensmodell Haben Fruchtfliegen einen freien Willen? Es muss eine input-unabhängige Output-Komponente geben: Our results show that a simple input/output model is insufficient even to explain fly behavior. Clearly, there must be an output component which is independent of input. Our data are consistent with the hypothesis that there is an initiator in the brain which is superimposed on top of the sensorimotor link. If this initiator were truly input-independent, one may wonder if even fruit flies may have free will. In this light, recent fMRI data showing spontaneous correlated activity in the resting human brain make a lot of sense. Similar to fly turning behavior, this activity is independent of environmental stimulation. Apparently, most of the brain's energy is spent on such intrinsic activity, with input from the environment using as little as 1% of the total energy budget. Where does this intrinsic initiating activity come from? Unfortunately, now I'll have to disappoint you: the rest of this presentation will not be about the neurobiology of the initiator. We don't yet know how the initiator works or where in the brain it is located. Finding out which circuits give rise to spontaneous activity and how is definitely next on our to-do-list. However, in the lack of more knowledge about the biology of the initiator, maybe we can ask why the brain appears to be devoting 99% of its energy to it? What is the advantage of having an initiator?