The Automatic Generation of Formal Annotations in a MultiMedia Indexing and Searching Environment Thierry Declerck, Peter Wittenburg and Hamish Cunningham DFKI GmbH, Max-Planck-Institut für Psycholinguistik and University of Sheffield ACL/EACL2001 Workshop on Human Language Technology and Knowledge Management

The MUMIS Consortium CTITUniversity of Twente, Enschede, NLNLP/IE TSI University of Nijmegen, Nijmegen, NLASR DFKISaarbrücken, DNLP/IE MPI Nijmegen, NLOnline SW DCSUniversity of Sheffield, UKNLP/IE ESTEAMGothenburg, SE (location Athens, GR)Translation Software VDAHilversum, NL Dissemination

Objectives Technology development to automatically index (with formal annotations) lengthy multimedia recordings (off-line process) Find and annotate relevant events, together with the involved entities and relations Technology development to exploit indexed multimedia archives (on-line process) Search for interesting scenes and play them via Internet Test Domain: Soccer Games / UEFA Tournament 2000

Off-line Task Automatic Speech Recognition (Radio/TV Broadcasts) Automatically transforms the speech signals into texts (for 3 languages Dutch, English and German) Natural Language Processing (Information Extraction) Analyse all available textual documents (newspapers, speech transcripts, tickers, formal texts...), identify and extract interesting entities, relations and events Merging all the annotations produced so far Create a database with formal annotations

The Generation of Formal Annotations Metadata (type of game, teams, date, final score, players etc.), as they can be used a.o. for classifying and filtering videos in the MM digital archive Events (particular actions with time codes, involved entities and related events), as they can be extracted from the video sequences All Formal Annotations available in XML Standard

The Event Table Related to domain ontology and multilingual terminology. Guiding the generation of formal annotations Final whistle#90>t>120Subj=referee, score etc…Final score Goal kick#0>t>120Subj=pl, loc=loc, cons=cons,.. Dribbling#0>t>120Subj=pl, loc=loc, … Substitution#0>t>120Subj=pl, I.obj=pl, cause=c, …Team (adding pl) Red Card#0>t>120Subj=ref, I.obj=pl, cause=c, …Team (red at t) Goal#0>t>=penSubj=pl, I.obj=team, score=s,Order of goal … EventIDTimeSubcat/ModificationMetadata

Off-line Task Events indexed in video recording 1:0 60 m25 m SchollBasler CampbellMatthäusBaslerNeville DribblingFreekickFoul 28min24 min18 min17 min DefensePassGoalFreekick Radio Commenting 3 Languages Radio Commenting 3 Languages Radio Commenting 3 Languages Audio Commenting (TV, Radio) 3 Languages Newspaper Text Newspaper Text Newspaper Text Newspaper Texts 3 Languages Newspaper Text Newspaper Text Newspaper Text Close caption 3 Languages multilingual IE => event tables Merging of Annotations Event = goal Player = Basler Dist. = 25 m Time = 18 Score = 1:0 Event = goal Type = Freekick Player = Basler Dist. = 25 m Time = 17 Score: leading Event = goal Player= Basler Team = Germany Time = 18 Score = 1:0 Finalscore = 1:0 Event = goal Type = Freekick Player = Basler Team = Germany Time = 18 Score = 1:0 Final score = 1:0 Distance = 25 m

Information Extraction IE is generally subdivided in following tasks: - Named Entity task (NE) - Template Element task (TE) - Template Relation task (TR) - Scenario Template task (ST) - Co-reference task (CO)

The Role of IE in MUMIS Information Extraction (IE) is the task of identifying, collecting and normalizing relevant information for a specific application or user. The relevant information is typically represented in form of predefined templates, which are filled by means of Natural Language (NL) analysis (Template = Event Table in MUMIS) IE combines pattern matching mechanisms, (shallow) NLP and domain knowledge (terminology and ontology).

Extension of our IE system in MUMIS Multilingual and multisource IE. Incremental information building Cross-document co-reference resolution Combine Metadata and event extraction => better organisation and dynamic updating of information (KM) Multiple presentation of results: Template, Event table and Hyperlinks (Named Entities, rel. to KM)

Subtasks of IE Named Entity task (NE): Mark into the text each string that represents, a person, organization, or location name, or a date or time, or a currency or percentage figure. Template Element task (TE): Extract basic information related to organization, person, and artifact entities, drawing evidence from everywhere in the text.

Subtasks of IE (2) Template Relation task (TR): Extract relational information on employee_of, manufacture_of, location_of relations etc. (TR expresses domain- independent relationships). Scenario Template task (ST): Extract pre-specified event information and relate the event information to particular organization, person, or artifact entities (ST identifies domain and task specific entities and relations). Co-reference task (CO): Capture information on co- referring expressions, i.e. all mentions of a given entity, including those marked in NE and TE.

IE applied to soccer Terms as descriptors for the NE task Team: Titelverteidiger Brasilien, den respektlosen Außenseiter Schottland Player:Superstar Ronaldo, von Bewacher Calderwood noch von Abwehrchef Hendry, von Jackson als drittem Stürmer, Torschütze Cesar, von Roberto Carlos (16.), Referee: vom spanischen Schiedsrichter Garcia Aranda Trainer: Schottlands Trainer Brown, Kapitän Hendry seinen Keeper Leighton Location: im Stade de France von St. Denis (more fine-grained location detection would be: Stadion: im Stade de France and City: von St. Denis ) Attendance: Vor Zuschauern

IE applied to soccer (2) Terms for NE Task Time: in der 73. Minute, nach gerade einmal 3:50 Minuten, von Roberto Carlos (16.), nach einer knappen halben Stunde, scheiterte Rivaldo (49./52.) jeweils nur knapp, das vor der Pause Versäumte versuchten die Brasilianer nach Wiederbeginn,... Date: am Mittwoch, der Turnierstart (?), im WM-Eröffnungsspiel (?) Score/Result: Brasilien besiegt Schottland 2:1, einen 2:1 (1:1)- Sieg, der zwischenzeitliche Ausgleich, in der 4. Minute in Führung gebracht, köpfte zum 1:0 ein

IE applied to soccer (3) Relations for TR Task Opponents: Brasilien besiegt Schottland, feierte der Top-Favorit... einen glücklichen 2:1 (1:1)-Sieg über den respektlosen Außenseiter Schottland, Player_of: hatte Cesar Sampaio den vierfachen Weltmeister... in Führung gebracht, Collins gelang... der zwischenzeitliche Ausgleich für die Schotten, der Keeper des FC Aberdeen, Brasiliens Keeper Taffarel Trainer_of: Schottlands Trainer Brown...

IE applied to soccer (4) Events for ST task : Goal: in der 4. Minute in Führung gebracht, das schnellste Tor... markiert, Cesar Sampaio köpfte zum 1:0 ein, Collins (38.) verwandelte den Strafstoß, hätte Kapitän Hendry seinen Keeper Leighton um ein Haar zum zweiten Mal bezwungen, von dem der Ball ins Tor prallte Foul: als er den durchlaufenden Gallacher im Strafraum allzu energisch am Trikot zog Substitution: und mußte in der 59. Minute für Crespo Platz machen...

IE applied to soccer (5) Description of the Templates: Team team-template TACTIC [ ] SCORE [ ] NAME [ ] PLAYER [ ] TRAINER [ ] goal-template TIME [ ] SCORE [S] PLAYER [P] TEAM [team-templ ] TYPE [ ] SUCCESS [ ] team-template TACTIC [ ] SCORE [S] NAME [ ] PLAYER [P] TRAINER [ ]

Example of Processing Formal Texts Formal Text The Formal Text annotated with domain- specific information

Example of Processing Semi-Formal Texts Semi-Formal Text The Semi-Formal Text annotated with domain-specific information

Merging Component Acting on the generated formal annotations (Metadata and Events), but also interleaving with the generation process of those Checking consistency, eliminating redundancy (Template Merging), in accordance with domain ontology Completing the information with domain knowledge, inference Machine

On-line Tasks Searching and Displaying Search for interesting events with formal queries Give me all goals from Overmars shot with his head in 1. Half. Event=Goal; Player=Overmars; Time<=45; Previous-Event=Headball Indicate hits by thumbnails & let user select scene Play scene via the Internet & allow scrolling etc Of course: slow motion, fast play, start/stop, etc User Guidance (Lexica and Ontology)

On-line Tasks Knowledge Guided User Interface & Search Engine München - Ajax 1998 München - Porto 1996 Deutschland - Brasilien 1998 Prototype Demo Play Movie Fragment of that Game 1:0 60 m25 m SchollBasler CampbellMatthäusBaslerNeville DribblingFreekickFoul 28min24 min18 min17 min DefensePassGoalFreekick

More about MPEG (Moving Picture Coding Experts Group) MPEG-1: low-level media encoding and compression format (VHS quality, ~ 2-3 Mbps - good for streaming) MPEG-2: improved media encoding and compression format (S-VHS quality, ~ 5-10 Mbps, digital TV and DVD standard MPEG-4: Codes content as objects and enables those objects to be manipulated at the client, optimized compression

On-line SW Architecture Client Applet JMF WWW Server Java Server Media Server MPEG1 Media Server MPEG1 Media Server MPEG1 DB Server rDBMS Media Server MPEG1 File Server HTTP RMI (RTP, RTSP) JDBC Query interface: automatic pre-selection guided by domain knowledge interactive, visual feedback Client-Server structure: fully distributed JMF media presentation RMI-based interaction Client Objects Query Engine Objects MetadataAnnotations KeyframesMPEG Movies Lexica Ontology Media Server Objects

On-line HW Architecture efficient & reliable storage management (near-line capacity, media change, 2. Location) high storage capacity (n TB, 1 h MPEG1 = 1 GB) powerful media servers / powerful network RAID Tape Library FC Switch Media Server GB Switch Internet 1Gbps Gb-Switch Router

UI / Annotation UI optimization thumbnails not that informative which thumbnail? (several around time mark) automatic thumbnail adjustment? seamless operation for user lexicon/ontology guidance user driven input Manual annotation tools MediaTagger EUDICO

Gain - User Group What gets lost? Is it necessary? Potential: direct Internet Service, less dependencies Current ProcedureMUMIS Procedure Manual Video Annotation Automatic Video Annotation and DB Integration Integration Central DB Query via PC Results on PC And Select & Play Contact Video Archive Get Video Tapes Search on Tape on VCR Segment & Play

Acknowledgements UEFA DFB, FA, KNVB EBU, WDR, NOS, SWR Allez les Bleus!!