Summary of the paper
| Title | A Question-answer Distance Measure to Investigate QA System Progress |
|---|---|
| Authors | Guillaume Bernard, Sophie Rosset, Martine Adda-Decker and Olivier Galibert |
| Abstract | The performance of question answering system is evaluated throughsuccessive evaluations campaigns. A set of questions are given to theparticipating systems which are to find the correct answer in acollection of documents. The creation process of the questions maychange from one evaluation to the next. This may entail an uncontroledquestion difficulty shift. For the QAst 2009evaluation campaign, a new procedure was adopted to build the questions.Comparing results of QAst 2008 and QAst 2009 evaluations, a strongperformance loss could be measured in 2009 for French and English,while the Spanish systems globally made progress. The measured lossmight be related to this new way of elaborating questions. The general purposeof this paper is to propose a measure to calibrate the difficulty of a question set. In particular, a reasonable measure should output higher values for 2009 than for 2008.The proposed measure relies on a distance measure between the critical elementsofa question and those of the associated correct answer. An increase of theproposeddistance measure for French and English 2009 evaluations as compared to 2008could be established.This increase correlates with the previously observed degraded performances.We conclude on the potential of this evaluation criterion: the importance ofsucha measure for the elaboration of new question corpora for questions answeringsystemsand a tool to control the level of difficulty for successive evaluationcampaigns. |
| Language | |
| Topics | Question Answering |
Full paper  |
A Question-answer Distance Measure to Investigate QA System Progress |
| Bibtex | @InProceedings{BERNARD10.518, author = {Guillaume Bernard, Sophie Rosset, Martine Adda-Decker and Olivier Galibert}, title = {A Question-answer Distance Measure to Investigate QA System Progress}, booktitle = {Proceedings of the Seventh conference on International Language Resources and Evaluation (LREC'10)}, year = {2010}, month = {may}, date = {19-21}, address = {Valletta, Malta}, editor = {Nicoletta Calzolari (Conference Chair), Khalid Choukri, Bente Maegaard, Joseph Mariani, Jan Odjik, Stelios Piperidis, Mike Rosner, Daniel Tapias}, publisher = {European Language Resources Association (ELRA)}, isbn = {2-9517408-6-7}, language = {english} } |