Jens Zimmermann, Forschungszentrum Jülich, Astroteilchenschule 10/041 Jens Zimmermann Max-Planck-Institut für Physik, München MPI für extraterrestrische Physik, München Forschungszentrum Jülich GmbH Statistical Learning? Three Classes of Learning Methods Applications in Physics Analysis Training the Learning Methods Examples Conclusion Statistical Learning in Astrophysics

Jens Zimmermann, Forschungszentrum Jülich, Astroteilchenschule 10/042 Some Events x10 # formulas # slides x10 # formulas# slides Experimentalists Theorists

Jens Zimmermann, Forschungszentrum Jülich, Astroteilchenschule 10/043 First Analysis x10 # formulas x10 # slides Experimentalists Theorists x10 # formulas # slides x10

Jens Zimmermann, Forschungszentrum Jülich, Astroteilchenschule 10/044 Decision Trees x10 # formulas #formulas < 20 exp #formulas > 60th x10 # slides 20 < #formulas < 60? #slides > 40exp #slides < 40th #slides < 40#slides > 40 expth #formulas < 20 #formulas > 60 rest exp th all events subset 20 < #formulas < 60

Jens Zimmermann, Forschungszentrum Jülich, Astroteilchenschule 10/045 Local Density Estimators Search for similar events that are already classified and count the members of the two classes. (e.g. k-Nearest-Neighbour) x10 # formulas # slides x10 # formulas # slides x10

Jens Zimmermann, Forschungszentrum Jülich, Astroteilchenschule 10/046 Methods Based on Linear Separation Divide the input space into regions separated by one or more hyperplanes. Extrapolation is done! x10 # formulas # slides x10 # formulas # slides x10

Jens Zimmermann, Forschungszentrum Jülich, Astroteilchenschule 10/047 Neural Networks x # formulas# slides 0 1 Construct NN with two separating hyperplanes:Train NN with two hidden neurons (gradient descent):

Jens Zimmermann, Forschungszentrum Jülich, Astroteilchenschule 10/048 NN Training 8 hidden neurons = 8 separating lines Test-Error Train-Error signal background Training Epochs

Jens Zimmermann, Forschungszentrum Jülich, Astroteilchenschule 10/049 Training of Statistical Learning Methods Statistical Learning Method: From N examplesinfer a rule Important: Generalisation vs. Overtraining Without noise and separable by complicated boundary? x10 # formulas # slides x10 Easily separable but with noise? Too high degree of polynomial results in interpolation but too low degree means bad approximation

Jens Zimmermann, Forschungszentrum Jülich, Astroteilchenschule 10/0410 Applications in Physics Analysis Classification Online „Trigger“ H1L2NNCharged Current Classification Offline „Purification“ Gamma vs. Hadron MAGIC Regression ~300µm ~10µm XEUSX-ray CCD

Jens Zimmermann, Forschungszentrum Jülich, Astroteilchenschule 10/0411 Features Choose raw quantities PCA FFT SymmetrieFit

Jens Zimmermann, Forschungszentrum Jülich, Astroteilchenschule 10/0412 Regression of the Incident Position of X-ray Photons  of reconstruction in µm: Neural Networks 3.6classical methods k-Nearest-Neighbour 3.7 ETA 3.9 CCOM 4.0 XEUS ~300µm ~10µm electron potential transfer direction

Jens Zimmermann, Forschungszentrum Jülich, Astroteilchenschule 10/0413 Pileup Recognition – Setup Pileup vs. Single photon classical algorithm „XMM“ ? photon efficiency [%] pileup rejection [%] 99/52 99/67

Jens Zimmermann, Forschungszentrum Jülich, Astroteilchenschule 10/0414 MAGIC - Gamma/Hadron Separation Observation Mkn SuperCuts 39.0  SuperCuts + Neural Network 46.8  

Jens Zimmermann, Forschungszentrum Jülich, Astroteilchenschule 10/0415 Conclusion Three classes of statistical learning methods Decision Trees (Bagging) Local Density Estimators Linear Separation Many applications in current astrophysics experiments and analysis Compared to classical methods usually at least small improvements C4.5 CART Random Forest Maximum Likelihood k-Nearest-Neighbour Linear Discriminant Analysis Neural Networks Support Vector Machines

Jens Zimmermann, Forschungszentrum Jülich, Astroteilchenschule 10/0416 Theory of Communication: Minimum Description Length Principle Bayes Hypothesis H and Data D Our hypothesis should have the maximum probability given the data: Shannon MDLP Rissanen 18 th century

Jens Zimmermann, Forschungszentrum Jülich, Astroteilchenschule 10/0417 Statistical Learning Theory: Structural Risk Minimization We have N training events with input x i and correct output y i empirical riskactual risk Relationship is uniform convergence: h 1 <h 2 <h3h3 Create nested subsets of function spaces with rising complexity Upper bound for the actual risk (Vapnik): h : VC Dimension of learning method (complexity) 1996

Jens Zimmermann, Forschungszentrum Jülich, Astroteilchenschule 10/0418 Support Vector Machines Separating hyperplane with maximum distance to each datapoint: Maximum margin classifier Found by setting up condition for correct classfication and minimizing which leads to the Lagrangian Necessary condition for a minimum is So the output becomes Only linear separation? The mapping to feature space is hidden in a kernel No! Replace dot products: KKT: only SV have Non-separable case:

Jens Zimmermann, Forschungszentrum Jülich, Astroteilchenschule 10/0419 Finally x10 # formulas # slides x10 Include Statistical Learning Theory: ~25 formulas on 19 slides x10 # formulas # slides x10 Skip Theory: ~7 formulas on 16 slides