Summary of the paper
| Title | Evaluation Protocol and Tools for Question-Answering on Speech Transcripts |
|---|---|
| Authors | Nicolas Moreau, Olivier Hamon, Djamel Mostefa, Sophie Rosset, Olivier Galibert, Lori Lamel, Jordi Turmo, Pere R. Comas, Paolo Rosso, Davide Buscaldi and Khalid Choukri |
| Abstract | Question Answering (QA) technology aims at providing relevant answers tonatural language questions. Most Question Answering research has focused onmining document collections containing written texts to answer writtenquestions. In addition to written sources, a large (and growing) amount ofpotentially interesting information appears in spoken documents, such asbroadcast news, speeches, seminars, meetings or telephone conversations. TheQAST track (Question-Answering on Speech Transcripts) was introduced in CLEF toinvestigate the problem of question answering in such audio documents. Thispaper describes in detail the evaluation protocol and tools designed anddeveloped for the CLEF-QAST evaluation campaigns that have taken place between2007 and 2009. We first remind the data, question sets, and submissionprocedures that were produced or set up during these three campaigns. As forthe evaluation procedure, the interface that was developed to ease theassessors’ work is described. In addition, this paper introduces amethodology for a semi-automatic evaluation of QAST systems based on time slotcomparisons. Finally, the QAST Evaluation Package 2007-2009 resulting fromthese evaluation campaigns is also introduced. |
| Language | Information Extraction, Information Retrieval |
| Topics | Question Answering, Evaluation methodologies, Information Extraction, Information Retrieval |
Full paper  |
Evaluation Protocol and Tools for Question-Answering on Speech Transcripts |
| Bibtex | @InProceedings{MOREAU10.372, author = {Nicolas Moreau, Olivier Hamon, Djamel Mostefa, Sophie Rosset, Olivier Galibert, Lori Lamel, Jordi Turmo, Pere R. Comas, Paolo Rosso, Davide Buscaldi and Khalid Choukri}, title = {Evaluation Protocol and Tools for Question-Answering on Speech Transcripts}, booktitle = {Proceedings of the Seventh conference on International Language Resources and Evaluation (LREC'10)}, year = {2010}, month = {may}, date = {19-21}, address = {Valletta, Malta}, editor = {Nicoletta Calzolari (Conference Chair), Khalid Choukri, Bente Maegaard, Joseph Mariani, Jan Odjik, Stelios Piperidis, Mike Rosner, Daniel Tapias}, publisher = {European Language Resources Association (ELRA)}, isbn = {2-9517408-6-7}, language = {english} } |