8 Theoretische Formulierungen (1970er) Experiment/Analyse: „Lernen ist von der zeitlichen Paarung der Reize abhängig“Hypothese: „Der prädiktive Wert des CS könnte für das Lernen entscheidend sein“also: Lernen findet immer dann statt, wenn Erwartungen verletzt werdenFormalisierte Hypothese:, mit DV - Lernrate, l - tatsächl. US- erwarteter USRescorla-Wagner-Regel (1972):Mit ai - Salienz von CSi (0<a<1), b - Salienz des US (-1<b<1)
24 R-W sagt Übererwartung voraus DVCS=c(Vmax-Vall)BlockingWenn Vall=Vmax=100 wird CS2 kaum noch assoziative Stärke erreichen (Blocking)ÜbererwartungBei Vall>Vmax wird DV<0!Prä-TrainingTestTrainingCS2+USCS2=100%CS1+CS2+USCS1 alleineCS1+USCS1=100%CS2 alleinePrä-Training1Test1Prä-Training2Test2TrainingTest3CS1+USCS1=100%CS2+USCS2=100%CS1+CS2+USCS1+CS2CS1+CS2+CS3+USCS3
29 Seit Rescorla und Wagner Sutton und Barto 1990: Reinforcement learning, temporal difference models„actor-critic model“ assoziativen LernensReinforcement Learning: An IntroductionRichard S. Sutton and Andrew G. Barto:
30 Analyse von Spontanverhalten Thank you very much for inviting me here and giving me an opportunity to present some of our research. We work in two model systems, the marine snail Aplysia and the fruit fly Drosophila. Today, I will only talk about the most recent developments in Drosophila. As you can see from the cartoon, it is a common concept that animals are in essence organisms which primarily react to external stimuli.
31 Die Organisation von Verhalten wird meist als Input/Output Modell beschriebenBrain function is ultimately best understood in terms of input/output transformations and how they are produced. Michael Mauk (2000): Nature Neuroscience 3,This view is pertinent also among scientists and is reflected in a quote from Mike Mauk at my former department at the University of Texas in Houston: "Brain function is ultimately best understood in terms of input/output transformations and how they are produced". A general model of behavior is often depicted as a simple input/output model, in which the sensory apparatus perceives environmental stimuli and transmits the sensory data to the sensorimotor link which maps the incoming sensory data to motor commands. The motor commands control the biomechanical apparatus which produces the behavior. A prediction derived from such a model is that constant input should produce constant output. However, because of inevitable noise at all stages of the model, the output in reality is rarely constant but rather noisy. Such a more realistic model implies that not every environmental situation necessarily leads to the same behavior, but any given behavior is a response to a specific situation. For example, a person committing a murder is doing so in response to the specific sequence of situations the murderer has experienced leading up to the crime. Of course, in such a view, the question arises of how responsible the murderer is for his crime. Now you might say that this model is a scientific simplification and nobody would possibly use it in such a case. However, the Dana foundation, the ACLU and the AAAS all have sponsored meetings on "Neuroscience and the Law", where the participants discussed the impact of Neuroscience on culpability, including exactly such cases as the one we are talking about. However, one can show that already in small brains, such a model is indeed oversimplified.Konstanter InputVerrauschter OutputKonstanter Output…aber jedes Verhalten ist eine Antwort auf einen ReizNicht jeder Reiz bedingt die gleiche Antwort…
32 Drosophila am Drehmoment-Kompensator Selbst bei konstantem sensorischem Eingang verhält sich die Fruchtfliege variabelspikeTethered Drosophila melanogaster performs a variety of behaviors with the abdomen, the legs, the wings and the antennae, while head and thorax remain fixed in space. Suspended at the torque compensator, we can measure one of these behaviors: yaw torque, the force the fly produces when attempting to turn (left or right) around its vertical body axis. Keeping the environment constant by surrounding the fly with a white, featureless panorama, we can confirm that the fly's behavior is indeed not constant at all. Instead, yaw torque fluctuates seemingly randomly from left to right. Zooming in onto a short 5min stretch, we can see that there are two components to this variability: a baseline fluctuation and superimposed, fast torque spikes. These spikes correspond to so-called body-saccades in free flight. As you might have noticed, free flying flies do not fly curves like airplanes, but rather zig-zag around. Each zig or zag corresponds to one such torque spike at the torque meter.baseline
33 Zustandsraum-Rekonstruktion Koordinaten-EinbettungZeitreihe: 80, 60, 40,……Einbettungs-Dimension: 3Drei Datenpunkte bestimmen einen 3D Vektor:All computations: Alexander Maye, UKE Hamburg
34 Inter-Spike-Interval (ISI-)Analyse Fliegen gegen PoissonFliegenverhalten sieht nicht wie einfaches Rauschen ausGeometric Random Inner Products: GRIPInter-Spike-Interval (ISI-)AnalyseUsing the time intervals between the torque spikes we can evaluate the temporal structure of fly turning behavior. Every blue stripe denotes a spike extracted from data such as that depicted on the previous slide. The input/output model predicts a Poisson process for spike production. Much like the hiss of static from a radio tuned between stations, a Poisson process is a random process, so flies should be random number generators. A mathematical method has recently been developed which can quantify the randomness of a time series. The Geometric Random Inner Products or GRIP procedure computes the deviation from ideal randomness. The graph shows this deviation on the Y-axis. The computer-generated Poisson series is not perfect but very close to random. However, flies are clearly even worse random number generators and definitely do not show the predicted random noise in their behavior. Information theory tells us that any deviation from random entails information in a number sequence and that in principle at least parts of the sequence can be predicted using this information.Fliegen als Zufallsgeneratoren?All computations: Alexander Maye, UKE Hamburg (Maye et al. 2007)
35 Statistik erster Ordnung ISI-Verteilungen haben ein „schweres Ende“Potenzgesetz: SkaleninvarianzAll computations: Alexander Maye, UKE Hamburg (Maye et al. 2007)
36 Unabhängigkeit der Daten Mischung der Einzelwerte ähnelt den Originaldaten nichtAll computations: Alexander Maye, UKE Hamburg (Maye et al. 2007)
37 Nichtlinearität?Noch nicht einmal sehr komplexe stochastische Modelle reichen ausBranched Poisson Process: BPPRoot mean square fluctuation of displacementOffensichtlich scheint die Nichtlinearität ein wichtigerer Faktor zu sein als die Zufälligkeit!All computations: Alexander Maye, UKE Hamburg (Maye et al. 2007)
38 Nichtlineare Vorhersagen: S-Maps Mathematisch instabile, nichtlineare Prozesse steuern das Flugverhalten von DrosophilaKorrelationLogistische Gleichung:nichtlinearlinear/stochastischTherefore we used a method called nonlinear forecasting to further analyze turning behavior. The S-maps procedure works similar to a weather forecast: mathematical models are derived from the first half of the time series to predict or forecast the second half of the series. You then calculate a correlation coefficient (rho) between the actual and the predicted values for forecasting just one time step ahead. Similar to how ou would judge the accuracy of a weather forecast, rho gives you an estimate of the accuracy of your model. In addition, the S-maps procedure uses a weighting parameter (theta) to denote how nonlinear the model was, with 0 being linear and 4 being most nonlinear. So rho tells you how good the model was and theta which model was used. Time series generated by nonlinear processes show an increase in rho with theta, random or linear time series show either a decrease or at least stable rho with theta. Analyzing ISIs with this method yields a nonlinear signature, albeit at a very low overall rho. In order to exclude information-loss due to spike extraction, we repeated the procedure with the raw yaw torque data. Now we find enhanced rho together with a very clear nonlinear signature. However, the brain is full of nonlinear feedback and feedforward circuits. Maybe what we find is just a reflection of this well-known fact? To test this hypothesis, we adapted a biologically plausible software agent from the literature with known nonlinear properties. It consists of a nonlinear activator (maybe located in the brain of the fly) which activates two mutually inhibiting nonlinear turn-generators (which may represent central pattern generators in the thoracic ganglia). Subtracting left from right turn-generator yields a torque signal. Not surprisingly, the resulting signal shows a strong nonlinear signature and the agent can be tuned such that the signal looks very similar to fly behavior. However, under these conditions, the agent does not show the nonlinear signature and if the agent is tuned so that it shows the nonlinearity, the output doesn't resemble fly behavior at all. Apparently, the agent is not as good a model for fly behavior as we thought. But maybe we can nevertheless learn something from this failure? The agent's components are modeled by the so-called logistic map, a well-known recursive function. The output s of the function (depicted on the Y-axis) is largely dependent on the parameter lambda, depicted on the X-axis. For low lambda, the agent's output looks similar to fly behavior, but is linear, because the function shows linear behavior. For larger lambda, the function becomes unstable and the agent shows the nonlinear signature. This reminds us that nonlinear systems can display linear behavior. We thus can conclude that spontaneous fly turning behavior is controlled by brain-circuits which are tuned to operate as nonlinear, unstable processes. Otherwise, we would not detect a nonlinear signature in the S-maps.NichtlinearitätAll computations: Alexander Maye, UKE Hamburg (Maye et al. 2007)
39 Ein neues Verhaltensmodell Haben Fruchtfliegen einen freien Willen?Es muss eine input-unabhängige Output-Komponente geben:Our results show that a simple input/output model is insufficient even to explain fly behavior. Clearly, there must be an output component which is independent of input. Our data are consistent with the hypothesis that there is an initiator in the brain which is superimposed on top of the sensorimotor link. If this initiator were truly input-independent, one may wonder if even fruit flies may have free will. In this light, recent fMRI data showing spontaneous correlated activity in the resting human brain make a lot of sense. Similar to fly turning behavior, this activity is independent of environmental stimulation. Apparently, most of the brain's energy is spent on such intrinsic activity, with input from the environment using as little as 1% of the total energy budget. Where does this intrinsic initiating activity come from? Unfortunately, now I'll have to disappoint you: the rest of this presentation will not be about the neurobiology of the initiator. We don't yet know how the initiator works or where in the brain it is located. Finding out which circuits give rise to spontaneous activity and how is definitely next on our to-do-list. However, in the lack of more knowledge about the biology of the initiator, maybe we can ask why the brain appears to be devoting 99% of its energy to it? What is the advantage of having an initiator?