EduDDI – an application of DDI 3.x for large scale assessments in education IASSIST/IFDO 2009 Tampere, Finland May 26th-May 29th, 2009 Martin Mechtel, Dr. Michel Knigge

Outline IQB – Institute for Educational Progress Expected benefits of meta data standardisation Efforts so far (drawbacks) Introducing DDI 3.x (bridges) Road map of EduDDI (sorry no software demo)

IQB – Institute for Progress in Education Founded 2004 by the Bundesländer (Federal States); hosted by Humboldt-University at Berlin, Principal Prof. Dr. Olaf Köller Trigger: TIMSS 1995/PISA 2000; frequent large scale assessments of students competencies take place on a regular basis since then Mission: Test educational standards for Germany Staff: Meanwhile 40 plus 40 part time (students) Budget: About 4.5 Mio per year, plus extra projects (e.g. FDZ – research data center) My role: Responsible for IT services (infrastructure, development); IT-staff: 3 plus 4 part time collegues (most infrastructure is supported by Humboldt-University) Dr. Michel Knigge: Head of FDZ

Expected benefits of meta data standardisation Obvious: Interoperability: not only "meta data" but datasets too Exchange of items/tasks easily, support of re-analyses, nice presentation of data to the public etc. Thinking: when I have a standard then I have tools which do the work for me Side effects: Installation of structures, work flows and vocabularies Dissemination of best practises (statistical methods, software for analyses, reporting culture, access policies)

Efforts so far (drawbacks) – specific in education Item appearance is crucial for interpretation of results (line breaks, font, size etc.) Item structure is complex (task: stimulus, sub tasks, items for scoring, variables for analyses) 1

item appearance is crucial for interpretation of results (line breaks, font, size etc.) item structure is complex (task: stimulus, sub tasks, items for scoring, variables for analyses) Efforts so far (drawbacks) – specific in education Grupp, ich beschwöre dich, mein punzig Turteldrom Und drängel reifig mich mit krinklen Bindelwördeln Denn sonst werd ich dich rändern in deinen Gobberwarzen Mit meinem Börgelkranze, wart´s nur ab! Vogonische Lyrik Welches Sprichwort passt am ehesten zu der Aussage des Gedichts? Kreuze die richtige Antwort an. Reisende soll man nicht aufhalten Liebe macht blind Ohne Fleiß keinen Preis Nachts sind alle Katzen grau. Aufgabe 1 An welcher Zeile ist das erkennbar (Zeilennummer): Nenne zwei Klangfiguren in diesem Gedicht: Aufgabe 2 Task M042 Subtask A Stimulus Subtask B Variable a Variable b Item 01 Item 02 Item 03

Efforts so far (drawbacks) – specific in education Complex study design to estimate the level of a certain competence although the test person responds simply to one or two items of the scale Item position is important for interpretation Item appearance is crucial for interpretation of results (line breaks, font, size etc.) Item structure is complex (task: stimulus, sub tasks, items for scoring, variables for analyses) 1 2

Efforts so far (drawbacks) – specific in education test design (example) booklets timeline total 167 Min

Efforts so far (drawbacks) – specific in education Complex study design to estimate the level of a certain competence although the test person responds simply to one or two items of the scale Item position is important for interpretation Item appearance is crucial for interpretation of results (line breaks, font, size etc.) Item structure is complex (task: stimulus, sub tasks, items for scoring, variables for analyses) Mostly small institutions involved in assessment (EMSE – Bundesländer; isolated chairs in research); like simple procedures Mostly short-term projects; no pay-back of investment in comprehensive approaches development of a standard failed 3

Introducing DDI 3.x (bridges) But now: we HAVE a standard! Grupp, ich beschwöre dich, mein punzig Turteldrom Und drängel reifig mich mit krinklen Bindelwördeln Denn sonst werd ich dich rändern in deinen Gobberwarzen Mit meinem Börgelkranze, wart´s nur ab! Vogonische Lyrik Welches Sprichwort passt am ehesten zu der Aussage des Gedichts? Kreuze die richtige Antwort an. Reisende soll man nicht aufhalten Liebe macht blind Ohne Fleiß keinen Preis Nachts sind alle Katzen grau. Aufgabe 1 An welcher Zeile ist das erkennbar (Zeilennummer): Nenne zwei Klangfiguren in diesem Gedicht: Aufgabe 2 Task M042 Subtask A Stimulus Subtask B Variable a Variable b Item 01 Item 02 Item 03 "EduDDI"

Introducing DDI 3.x (bridges) DDI can do: specific structure, other material, scales DDI should do (3.2?): usage of Genericode to attach any property to any XML item (inline storage of data to get a question bank) Working with DDI expands the community (by YOU) DDI is the cheese for the mouse: possible partners are soon convinced and support the project Other fortunate conditions: New high stake/large scale project looks for technology: NEPS National Educational Panel Study In Germany, a sufficient number of large studies wait for further analyses (FDZ)

Road map of EduDDI Project/initiative members: DIPF Frankfurt/M., TestDaF Hagen, IEA-DPC Hamburg, IQB+FDZ, ifib Bremen, University of Bamberg (NEPS) Near future Discussion about meta data catalogues (two groups: item oriented and dataset oriented) Discussion about technology/architecture (inside institutions: ASP/VB.Net, MS SQL Server; offline tools: java) Acquirement of more support (project proposals, more institutions) Next year Definition of the DDI implementation Development of first tools and interfaces Launch of research studies to underpin the assumptions and to bring the project to a broad community

telefon +49[30] telefax +49[30] postal address Humboldt-Universität zu Berlin Unter den Linden Berlin Location Luisenstr Berlin