Summary of the paper
| Title | Evaluating Lexical Substitution: Analysis and New Measures |
|---|---|
| Authors | Sanaz Jabbari, Mark Hepple and Louise Guthrie |
| Abstract | Lexical substitution is the task of finding areplacement for a target word in a sentence so as topreserve, as closely as possible, the meaning of theoriginal sentence. It has been proposed that lexicalsubstitution be used as a basis for assessing theperformance of word sense disambiguation systems, anidea realised in the English Lexical Substitution Taskof SemEval-2007. In this paper, we examine theevaluation metrics used for the English LexicalSubstitution Task and identify some problems that arisefor them. We go on to propose some alternative measuresfor this purpose, that avoid these problems, and whichin turn can be seen as redefining the key tasks thatlexical substitution systems should be expected toperform. We hope that these new metrics will betterserve to guide the development of lexical substitutionsystems in future work. One of the new metricsaddresses how effective systems are in rankingsubstitution candidates, a key ability for lexicalsubstitution systems, and we report some resultsconcerning the assessment of systems produced by thismeasure as compared to the relevant measure fromSemEval-2007. |
| Language | Textual Entailment and Paraphrasing |
| Topics | Word Sense Disambiguation, Evaluation methodologies, Textual Entailment and Paraphrasing |
Full paper  |
Evaluating Lexical Substitution: Analysis and New Measures |
| Bibtex | @InProceedings{JABBARI10.782, author = {Sanaz Jabbari, Mark Hepple and Louise Guthrie}, title = {Evaluating Lexical Substitution: Analysis and New Measures}, booktitle = {Proceedings of the Seventh conference on International Language Resources and Evaluation (LREC'10)}, year = {2010}, month = {may}, date = {19-21}, address = {Valletta, Malta}, editor = {Nicoletta Calzolari (Conference Chair), Khalid Choukri, Bente Maegaard, Joseph Mariani, Jan Odjik, Stelios Piperidis, Mike Rosner, Daniel Tapias}, publisher = {European Language Resources Association (ELRA)}, isbn = {2-9517408-6-7}, language = {english} } |