Summary of the paper
| Title | Fine-grained Linguistic Evaluation of Question Answering Systems |
|---|---|
| Authors | Sarra El Ayari, Brigitte Grau and Anne-Laure Ligozat |
| Abstract | Question answering systems are complex systems using natural languageprocessing. Some evaluation campaigns are organized to evaluate such systems inorder to propose a classification of systems based on final results (number ofcorrect answers). Nevertheless, teams need to evaluate more precisely theresults obtained by their systems if they want to do a diagnostic evaluation.There are no tools or methods to do these evaluations systematically. Wepresent REVISE, a tool for glass box evaluation based on diagnostic of questionanswering system results. |
| Language | Evaluation methodologies |
| Topics | Question Answering, Corpus (creation, annotation, etc.), Evaluation methodologies |
Full paper  |
Fine-grained Linguistic Evaluation of Question Answering Systems |
| Bibtex | @InProceedings{ELAYARI10.520, author = {Sarra El Ayari, Brigitte Grau and Anne-Laure Ligozat}, title = {Fine-grained Linguistic Evaluation of Question Answering Systems}, booktitle = {Proceedings of the Seventh conference on International Language Resources and Evaluation (LREC'10)}, year = {2010}, month = {may}, date = {19-21}, address = {Valletta, Malta}, editor = {Nicoletta Calzolari (Conference Chair), Khalid Choukri, Bente Maegaard, Joseph Mariani, Jan Odjik, Stelios Piperidis, Mike Rosner, Daniel Tapias}, publisher = {European Language Resources Association (ELRA)}, isbn = {2-9517408-6-7}, language = {english} } |