Stefan Evert, IMS - Uni Stuttgart Brigitte Krenn, ÖFAI Wien IMS Evaluation of Association Measures.

Präsentation zum Thema: "Stefan Evert, IMS - Uni Stuttgart Brigitte Krenn, ÖFAI Wien IMS Evaluation of Association Measures."—  Präsentation transkript:

1 Stefan Evert, IMS - Uni Stuttgart Brigitte Krenn, ÖFAI Wien IMS Evaluation of Association Measures

2 Stefan Evert, IMS - Uni Stuttgart Brigitte Krenn, ÖFAI Wien IMS Want to identify the practical feasibility of a certain AM for identifying collocations ?which types of collocation ?which corpora (domain, size) ?high frequency versus low frequency data compare the outcomes of different association measures

3 Stefan Evert, IMS - Uni Stuttgart Brigitte Krenn, ÖFAI Wien IMS We have differently ranked collocation candidates We need true collocation data for comparison, e.g collocation lexica list of true collocations occurring in the extraction corpus

4 Stefan Evert, IMS - Uni Stuttgart Brigitte Krenn, ÖFAI Wien IMS Problems & Inconveniences using collocation lexica for evaluation will not tell us how well an AM worked on a particular corpus it only tells us that some of the reference collocations also occur in in our base data and the AM has found them

5 Stefan Evert, IMS - Uni Stuttgart Brigitte Krenn, ÖFAI Wien IMS Problems & Inconveniences Using a list of true collocations occurring in the extraction corpus requires a good deal of hand- annotation requires objective criteria for the distinction of collocational and noncollocational word combinations in our candidate list

6 Stefan Evert, IMS - Uni Stuttgart Brigitte Krenn, ÖFAI Wien IMS Our Approach Evaluation of lexical association measures AMs against a manually identified reference corpus of true collocations (TPs) Evaluation based on the full reference set Precise, linguistically motivated definition of TPs The evaluation of results based on recall and precision graphs

7 Stefan Evert, IMS - Uni Stuttgart Brigitte Krenn, ÖFAI Wien IMS For Further Discussion Testing for significance of AMs is an important but still open question There is a potential for fine- tuning of AMs given a specific data set and a particular type of collocations to be extracted (Krenn, Evert 2001)

8 Stefan Evert, IMS - Uni Stuttgart Brigitte Krenn, ÖFAI Wien IMS Evaluation Experiments

9 Stefan Evert, IMS - Uni Stuttgart Brigitte Krenn, ÖFAI Wien IMS Data Extraction corpora newspaper: 8 million words Frankfurter Rundschau Corpus (ECI Multilingual Corpus 1) newsgroup: 10 million words FLAG corpus (LT-DFKI)

10 Stefan Evert, IMS - Uni Stuttgart Brigitte Krenn, ÖFAI Wien IMS Data Base data: list of PP-verb pairs ~ (PN,V)-combinations Collocation types: support verb constructions FVG figurative expressions figur

11 Stefan Evert, IMS - Uni Stuttgart Brigitte Krenn, ÖFAI Wien IMS Examples

12 Stefan Evert, IMS - Uni Stuttgart Brigitte Krenn, ÖFAI Wien IMS Support Verb Constructions FVG verb-object collocation function as predicates can be paraphrased by main verbs NP-verb or PP-verb verbal collocate (function verb / light verb / support verb) main verb conveys Aktionsart and causativity

13 Stefan Evert, IMS - Uni Stuttgart Brigitte Krenn, ÖFAI Wien IMS Support Verb Constructions FVG nominal collocate abstract noun often de-verbal or de-adjectival contributes the core meaning (prepositional collocate) verbal and nominal collocate together determine the argument structure of the collocation

14 Stefan Evert, IMS - Uni Stuttgart Brigitte Krenn, ÖFAI Wien IMS FVG Examples pred. phrase verbActionsartcaustranslation in Betriebgehenincho-go into operation nehmenincho+put into operation setzenincho+start up seinneutral-be running bleibencontin-keep on running lassencontin+keep (sth) running

15 Stefan Evert, IMS - Uni Stuttgart Brigitte Krenn, ÖFAI Wien IMS FVG Examples pred. phrase verbActionsartcaustranslation ausser Betrieb gehentermin-go out of sevice nehmentermin+take out of sevice setzentermin+stop seinneutral-be out of order bleibencontin-stay out of order lassencontin+keep out of order

16 Stefan Evert, IMS - Uni Stuttgart Brigitte Krenn, ÖFAI Wien IMS Figurative Expressions figur not restricted to NP/PP-verb figurative reinterpretation of literal meaning required (e.g., unter die Haut gehen (get under ones skin) nouns: conrete verbs: often causative-noncausative alternation e.g., auf Eis legen (put on ice) auf Eis liegen (be on ice)

17 Stefan Evert, IMS - Uni Stuttgart Brigitte Krenn, ÖFAI Wien IMS Decision Tree: FVG versus figur

18 Stefan Evert, IMS - Uni Stuttgart Brigitte Krenn, ÖFAI Wien IMS Frequency Distributions

19 Stefan Evert, IMS - Uni Stuttgart Brigitte Krenn, ÖFAI Wien IMS Frequency Distributions

20 Stefan Evert, IMS - Uni Stuttgart Brigitte Krenn, ÖFAI Wien IMS Frequency Distributions

21 Stefan Evert, IMS - Uni Stuttgart Brigitte Krenn, ÖFAI Wien IMS Combination of Properties in the Candidate Lists newspaper f >= 3 FVG newspaper f >= 3 figur newsgroup f >= 5 FVG,figur newspaper f >= 3 FVG,figur

22 Stefan Evert, IMS - Uni Stuttgart Brigitte Krenn, ÖFAI Wien IMS Evaluation Procedure Source Corpus 1.992 1.992 1.986 1.578 1.652 1.672 2.440 1.596 1.731 1.992 2.947 1.999 1.705 1.717 1.719 1.723 1.724 1.998 1.999 2.449... ab Dienstag bietet ab Donnerstag bietet ab Freitag bietet ab Jahren beginnt ab Jahren bietet ab Jahren eingeladen ab Jahren geeignet ab Jahren heißt ab Jahren käthi ab Jahren tanzen ab Jahren treffen ab Juni restauriert ab Mark finden ab Mark kostet ab Mark zu_finden ab März bietet+an ab Mittwoch bietet ab Notierungen nutzen ab Notierungen zu_nutzen ab November einladen... t-scoreCandidate pair candidate list

23 Stefan Evert, IMS - Uni Stuttgart Brigitte Krenn, ÖFAI Wien IMS Evaluation Procedure 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. 16. 17. 18. 19. 20.... Rank 19.218 13.523 12.751 11.724 11.147 10.465 10.008 9.782 9.700 9.473 8.978 8.931 8.613 8.600 8.423 8.395 8.298 8.289 8.282 8.269... um Uhr beginnt bis Uhr geöffnet zur Verfügung stehen zur Verfügung gestellt zur Verfügung stellen ums Leben gekommen zur Verfügung steht auf Programm stehen in Anspruch genommen auf Tagesordnung stehen am Dienstag sagte am Montag sagte auf Seite lesen auf Kürzungen behält vor auf Programm steht im Mittelpunkt steht in Regionalausgabe erscheint an Stelle melden auf Seite zeigen zur Verfügung zu_stellen... t-scoreCandidate pair significance list

24 Stefan Evert, IMS - Uni Stuttgart Brigitte Krenn, ÖFAI Wien IMS Evaluation Procedure: N-best Lists 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. 16. 17. 18. 19. 20.... Rank 19.218 13.523 12.751 11.724 11.147 10.465 10.008 9.782 9.700 9.473 8.978 8.931 8.613 8.600 8.423 8.395 8.298 8.289 8.282 8.269... um Uhr beginnt bis Uhr geöffnet zur Verfügung stehen zur Verfügung gestellt zur Verfügung stellen ums Leben gekommen zur Verfügung steht auf Programm stehen in Anspruch genommen auf Tagesordnung stehen am Dienstag sagte am Montag sagte auf Seite lesen auf Kürzungen behält vor auf Programm steht im Mittelpunkt steht in Regionalausgabe erscheint an Stelle melden auf Seite zeigen zur Verfügung zu_stellen... t-scoreCandidate pair 9 false positives 11 true positives precision: 11/20 = 55% total: 1280 TPs

25 Stefan Evert, IMS - Uni Stuttgart Brigitte Krenn, ÖFAI Wien IMS Precision Graph: PNV full forms

26 Stefan Evert, IMS - Uni Stuttgart Brigitte Krenn, ÖFAI Wien IMS Base Line: Random Selection

27 Stefan Evert, IMS - Uni Stuttgart Brigitte Krenn, ÖFAI Wien IMS Precision Graphs

28 Stefan Evert, IMS - Uni Stuttgart Brigitte Krenn, ÖFAI Wien IMS Precision Graphs

29 Stefan Evert, IMS - Uni Stuttgart Brigitte Krenn, ÖFAI Wien IMS Precision Graphs

30 Stefan Evert, IMS - Uni Stuttgart Brigitte Krenn, ÖFAI Wien IMS Recall Graphs

31 Stefan Evert, IMS - Uni Stuttgart Brigitte Krenn, ÖFAI Wien IMS Precision/Recall

32 Stefan Evert, IMS - Uni Stuttgart Brigitte Krenn, ÖFAI Wien IMS Precision Graphs: Newspaper, FVG + figur

33 Stefan Evert, IMS - Uni Stuttgart Brigitte Krenn, ÖFAI Wien IMS Precision Graphs: Newspaper FVG figur

34 Stefan Evert, IMS - Uni Stuttgart Brigitte Krenn, ÖFAI Wien IMS Precision Graphs: AdjN

35 Stefan Evert, IMS - Uni Stuttgart Brigitte Krenn, ÖFAI Wien IMS Precision Graphs: AdjN

36 Stefan Evert, IMS - Uni Stuttgart Brigitte Krenn, ÖFAI Wien IMS Precision/Recall: AdjN

37 Stefan Evert, IMS - Uni Stuttgart Brigitte Krenn, ÖFAI Wien IMS Frequency Layers: AdjN Data f 5 2 f < 5

38 Stefan Evert, IMS - Uni Stuttgart Brigitte Krenn, ÖFAI Wien IMS Frequency Layers: PNV Data f 10 3 f < 5

39 Stefan Evert, IMS - Uni Stuttgart Brigitte Krenn, ÖFAI Wien IMS Lemmas vs. Word Forms (PNV) lemmas f 3 word forms f 3

40 Stefan Evert, IMS - Uni Stuttgart Brigitte Krenn, ÖFAI Wien IMS Text Type and Domain (PNV) news group discussions newspaper comparison for non-lemmatised candidates

41 Stefan Evert, IMS - Uni Stuttgart Brigitte Krenn, ÖFAI Wien IMS The MI Mystery (FVG) region of high "local precision" for 4.0 < MI < 7.5

42 Stefan Evert, IMS - Uni Stuttgart Brigitte Krenn, ÖFAI Wien IMS Further particularities of the newspaper data candidates with MI > 7.5 are more frequent than expected under independence assumption but very few FVG among them data do not support the counter- MI argument of overestimation of data with low-frequency joint and marginal distributions

43 Stefan Evert, IMS - Uni Stuttgart Brigitte Krenn, ÖFAI Wien IMS optimized MI | MI - 5.75 | account for the FVG concentration among 4.0 = 7.5 in the newspaper test data

44 Stefan Evert, IMS - Uni Stuttgart Brigitte Krenn, ÖFAI Wien IMS Summary of Results Best measures: t-score / frequency best for identifying PP-verb collocations (FVG, figur) log-likelihood, t-score, Fisher, binominal and multinominal p value work well for AdjN

45 Stefan Evert, IMS - Uni Stuttgart Brigitte Krenn, ÖFAI Wien IMS Summary of Results Reproducibility of results for different text types: Precision results from newsgroup data comparable to newspaper data Strong evidence that identical classes of collocations are similarly distributed in different types of corpora

46 Stefan Evert, IMS - Uni Stuttgart Brigitte Krenn, ÖFAI Wien IMS Summary of Results Differences in suitability of AMs to identify particular collocation types: (PN,V)-candidates with high MI score are less likely to be FVG Log-likelihood not well suited for identifying FVG but better suited for identifying figur

47 Stefan Evert, IMS - Uni Stuttgart Brigitte Krenn, ÖFAI Wien IMS Summary of Results Experimental results based either on a small number of best- scoring candidates or on more than the first 50 % of the SLs are unreliable

48 Stefan Evert, IMS - Uni Stuttgart Brigitte Krenn, ÖFAI Wien IMS Conclusion on AMs Optimal results do not necessarily come from a statistical discussion but from tuning on a particular data set

49 Stefan Evert, IMS - Uni Stuttgart Brigitte Krenn, ÖFAI Wien IMS Vast Land: Lowest-frequency Data lowest-frequency data (hapax legomena, dis legomena,...) are a serious challenge for all statistical approaches typical solution: cut-off thresholds Evert/Krenn used cut-off thresholds in evaluation to reduce manual annotation work need to estimate number of TPs among excluded lowest-frequency candidates