Präsentation herunterladen
Die Präsentation wird geladen. Bitte warten
Veröffentlicht von:Sibylle Laudick Geändert vor über 10 Jahren
1
Stefan Evert, IMS - Uni Stuttgart Brigitte Krenn, ÖFAI Wien IMS Evaluation of Association Measures
2
Stefan Evert, IMS - Uni Stuttgart Brigitte Krenn, ÖFAI Wien IMS Want to identify the practical feasibility of a certain AM for identifying collocations ?which types of collocation ?which corpora (domain, size) ?high frequency versus low frequency data compare the outcomes of different association measures
3
Stefan Evert, IMS - Uni Stuttgart Brigitte Krenn, ÖFAI Wien IMS We have differently ranked collocation candidates We need true collocation data for comparison, e.g collocation lexica list of true collocations occurring in the extraction corpus
4
Stefan Evert, IMS - Uni Stuttgart Brigitte Krenn, ÖFAI Wien IMS Problems & Inconveniences using collocation lexica for evaluation will not tell us how well an AM worked on a particular corpus it only tells us that some of the reference collocations also occur in in our base data and the AM has found them
5
Stefan Evert, IMS - Uni Stuttgart Brigitte Krenn, ÖFAI Wien IMS Problems & Inconveniences Using a list of true collocations occurring in the extraction corpus requires a good deal of hand- annotation requires objective criteria for the distinction of collocational and noncollocational word combinations in our candidate list
6
Stefan Evert, IMS - Uni Stuttgart Brigitte Krenn, ÖFAI Wien IMS Our Approach Evaluation of lexical association measures AMs against a manually identified reference corpus of true collocations (TPs) Evaluation based on the full reference set Precise, linguistically motivated definition of TPs The evaluation of results based on recall and precision graphs
7
Stefan Evert, IMS - Uni Stuttgart Brigitte Krenn, ÖFAI Wien IMS For Further Discussion Testing for significance of AMs is an important but still open question There is a potential for fine- tuning of AMs given a specific data set and a particular type of collocations to be extracted (Krenn, Evert 2001)
8
Stefan Evert, IMS - Uni Stuttgart Brigitte Krenn, ÖFAI Wien IMS Evaluation Experiments
9
Stefan Evert, IMS - Uni Stuttgart Brigitte Krenn, ÖFAI Wien IMS Data Extraction corpora newspaper: 8 million words Frankfurter Rundschau Corpus (ECI Multilingual Corpus 1) newsgroup: 10 million words FLAG corpus (LT-DFKI)
10
Stefan Evert, IMS - Uni Stuttgart Brigitte Krenn, ÖFAI Wien IMS Data Base data: list of PP-verb pairs ~ (PN,V)-combinations Collocation types: support verb constructions FVG figurative expressions figur
11
Stefan Evert, IMS - Uni Stuttgart Brigitte Krenn, ÖFAI Wien IMS Examples
12
Stefan Evert, IMS - Uni Stuttgart Brigitte Krenn, ÖFAI Wien IMS Support Verb Constructions FVG verb-object collocation function as predicates can be paraphrased by main verbs NP-verb or PP-verb verbal collocate (function verb / light verb / support verb) main verb conveys Aktionsart and causativity
13
Stefan Evert, IMS - Uni Stuttgart Brigitte Krenn, ÖFAI Wien IMS Support Verb Constructions FVG nominal collocate abstract noun often de-verbal or de-adjectival contributes the core meaning (prepositional collocate) verbal and nominal collocate together determine the argument structure of the collocation
14
Stefan Evert, IMS - Uni Stuttgart Brigitte Krenn, ÖFAI Wien IMS FVG Examples pred. phrase verbActionsartcaustranslation in Betriebgehenincho-go into operation nehmenincho+put into operation setzenincho+start up seinneutral-be running bleibencontin-keep on running lassencontin+keep (sth) running
15
Stefan Evert, IMS - Uni Stuttgart Brigitte Krenn, ÖFAI Wien IMS FVG Examples pred. phrase verbActionsartcaustranslation ausser Betrieb gehentermin-go out of sevice nehmentermin+take out of sevice setzentermin+stop seinneutral-be out of order bleibencontin-stay out of order lassencontin+keep out of order
16
Stefan Evert, IMS - Uni Stuttgart Brigitte Krenn, ÖFAI Wien IMS Figurative Expressions figur not restricted to NP/PP-verb figurative reinterpretation of literal meaning required (e.g., unter die Haut gehen (get under ones skin) nouns: conrete verbs: often causative-noncausative alternation e.g., auf Eis legen (put on ice) auf Eis liegen (be on ice)
17
Stefan Evert, IMS - Uni Stuttgart Brigitte Krenn, ÖFAI Wien IMS Decision Tree: FVG versus figur
18
Stefan Evert, IMS - Uni Stuttgart Brigitte Krenn, ÖFAI Wien IMS Frequency Distributions
19
Stefan Evert, IMS - Uni Stuttgart Brigitte Krenn, ÖFAI Wien IMS Frequency Distributions
20
Stefan Evert, IMS - Uni Stuttgart Brigitte Krenn, ÖFAI Wien IMS Frequency Distributions
21
Stefan Evert, IMS - Uni Stuttgart Brigitte Krenn, ÖFAI Wien IMS Combination of Properties in the Candidate Lists newspaper f >= 3 FVG newspaper f >= 3 figur newsgroup f >= 5 FVG,figur newspaper f >= 3 FVG,figur
22
Stefan Evert, IMS - Uni Stuttgart Brigitte Krenn, ÖFAI Wien IMS Evaluation Procedure Source Corpus 1.992 1.992 1.986 1.578 1.652 1.672 2.440 1.596 1.731 1.992 2.947 1.999 1.705 1.717 1.719 1.723 1.724 1.998 1.999 2.449... ab Dienstag bietet ab Donnerstag bietet ab Freitag bietet ab Jahren beginnt ab Jahren bietet ab Jahren eingeladen ab Jahren geeignet ab Jahren heißt ab Jahren käthi ab Jahren tanzen ab Jahren treffen ab Juni restauriert ab Mark finden ab Mark kostet ab Mark zu_finden ab März bietet+an ab Mittwoch bietet ab Notierungen nutzen ab Notierungen zu_nutzen ab November einladen... t-scoreCandidate pair candidate list
23
Stefan Evert, IMS - Uni Stuttgart Brigitte Krenn, ÖFAI Wien IMS Evaluation Procedure 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. 16. 17. 18. 19. 20.... Rank 19.218 13.523 12.751 11.724 11.147 10.465 10.008 9.782 9.700 9.473 8.978 8.931 8.613 8.600 8.423 8.395 8.298 8.289 8.282 8.269... um Uhr beginnt bis Uhr geöffnet zur Verfügung stehen zur Verfügung gestellt zur Verfügung stellen ums Leben gekommen zur Verfügung steht auf Programm stehen in Anspruch genommen auf Tagesordnung stehen am Dienstag sagte am Montag sagte auf Seite lesen auf Kürzungen behält vor auf Programm steht im Mittelpunkt steht in Regionalausgabe erscheint an Stelle melden auf Seite zeigen zur Verfügung zu_stellen... t-scoreCandidate pair significance list
24
Stefan Evert, IMS - Uni Stuttgart Brigitte Krenn, ÖFAI Wien IMS Evaluation Procedure: N-best Lists 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. 16. 17. 18. 19. 20.... Rank 19.218 13.523 12.751 11.724 11.147 10.465 10.008 9.782 9.700 9.473 8.978 8.931 8.613 8.600 8.423 8.395 8.298 8.289 8.282 8.269... um Uhr beginnt bis Uhr geöffnet zur Verfügung stehen zur Verfügung gestellt zur Verfügung stellen ums Leben gekommen zur Verfügung steht auf Programm stehen in Anspruch genommen auf Tagesordnung stehen am Dienstag sagte am Montag sagte auf Seite lesen auf Kürzungen behält vor auf Programm steht im Mittelpunkt steht in Regionalausgabe erscheint an Stelle melden auf Seite zeigen zur Verfügung zu_stellen... t-scoreCandidate pair 9 false positives 11 true positives precision: 11/20 = 55% total: 1280 TPs
25
Stefan Evert, IMS - Uni Stuttgart Brigitte Krenn, ÖFAI Wien IMS Precision Graph: PNV full forms
26
Stefan Evert, IMS - Uni Stuttgart Brigitte Krenn, ÖFAI Wien IMS Base Line: Random Selection
27
Stefan Evert, IMS - Uni Stuttgart Brigitte Krenn, ÖFAI Wien IMS Precision Graphs
28
Stefan Evert, IMS - Uni Stuttgart Brigitte Krenn, ÖFAI Wien IMS Precision Graphs
29
Stefan Evert, IMS - Uni Stuttgart Brigitte Krenn, ÖFAI Wien IMS Precision Graphs
30
Stefan Evert, IMS - Uni Stuttgart Brigitte Krenn, ÖFAI Wien IMS Recall Graphs
31
Stefan Evert, IMS - Uni Stuttgart Brigitte Krenn, ÖFAI Wien IMS Precision/Recall
32
Stefan Evert, IMS - Uni Stuttgart Brigitte Krenn, ÖFAI Wien IMS Precision Graphs: Newspaper, FVG + figur
33
Stefan Evert, IMS - Uni Stuttgart Brigitte Krenn, ÖFAI Wien IMS Precision Graphs: Newspaper FVG figur
34
Stefan Evert, IMS - Uni Stuttgart Brigitte Krenn, ÖFAI Wien IMS Precision Graphs: AdjN
35
Stefan Evert, IMS - Uni Stuttgart Brigitte Krenn, ÖFAI Wien IMS Precision Graphs: AdjN
36
Stefan Evert, IMS - Uni Stuttgart Brigitte Krenn, ÖFAI Wien IMS Precision/Recall: AdjN
37
Stefan Evert, IMS - Uni Stuttgart Brigitte Krenn, ÖFAI Wien IMS Frequency Layers: AdjN Data f 5 2 f < 5
38
Stefan Evert, IMS - Uni Stuttgart Brigitte Krenn, ÖFAI Wien IMS Frequency Layers: PNV Data f 10 3 f < 5
39
Stefan Evert, IMS - Uni Stuttgart Brigitte Krenn, ÖFAI Wien IMS Lemmas vs. Word Forms (PNV) lemmas f 3 word forms f 3
40
Stefan Evert, IMS - Uni Stuttgart Brigitte Krenn, ÖFAI Wien IMS Text Type and Domain (PNV) news group discussions newspaper comparison for non-lemmatised candidates
41
Stefan Evert, IMS - Uni Stuttgart Brigitte Krenn, ÖFAI Wien IMS The MI Mystery (FVG) region of high "local precision" for 4.0 < MI < 7.5
42
Stefan Evert, IMS - Uni Stuttgart Brigitte Krenn, ÖFAI Wien IMS Further particularities of the newspaper data candidates with MI > 7.5 are more frequent than expected under independence assumption but very few FVG among them data do not support the counter- MI argument of overestimation of data with low-frequency joint and marginal distributions
43
Stefan Evert, IMS - Uni Stuttgart Brigitte Krenn, ÖFAI Wien IMS optimized MI | MI - 5.75 | account for the FVG concentration among 4.0 = 7.5 in the newspaper test data
44
Stefan Evert, IMS - Uni Stuttgart Brigitte Krenn, ÖFAI Wien IMS Summary of Results Best measures: t-score / frequency best for identifying PP-verb collocations (FVG, figur) log-likelihood, t-score, Fisher, binominal and multinominal p value work well for AdjN
45
Stefan Evert, IMS - Uni Stuttgart Brigitte Krenn, ÖFAI Wien IMS Summary of Results Reproducibility of results for different text types: Precision results from newsgroup data comparable to newspaper data Strong evidence that identical classes of collocations are similarly distributed in different types of corpora
46
Stefan Evert, IMS - Uni Stuttgart Brigitte Krenn, ÖFAI Wien IMS Summary of Results Differences in suitability of AMs to identify particular collocation types: (PN,V)-candidates with high MI score are less likely to be FVG Log-likelihood not well suited for identifying FVG but better suited for identifying figur
47
Stefan Evert, IMS - Uni Stuttgart Brigitte Krenn, ÖFAI Wien IMS Summary of Results Experimental results based either on a small number of best- scoring candidates or on more than the first 50 % of the SLs are unreliable
48
Stefan Evert, IMS - Uni Stuttgart Brigitte Krenn, ÖFAI Wien IMS Conclusion on AMs Optimal results do not necessarily come from a statistical discussion but from tuning on a particular data set
49
Stefan Evert, IMS - Uni Stuttgart Brigitte Krenn, ÖFAI Wien IMS Vast Land: Lowest-frequency Data lowest-frequency data (hapax legomena, dis legomena,...) are a serious challenge for all statistical approaches typical solution: cut-off thresholds Evert/Krenn used cut-off thresholds in evaluation to reduce manual annotation work need to estimate number of TPs among excluded lowest-frequency candidates
Ähnliche Präsentationen
© 2024 SlidePlayer.org Inc.
All rights reserved.