Summary of the paper
| Title | A Comprehensive Resource to Evaluate Complex Open Domain Question Answering |
|---|---|
| Authors | Silvia Quarteroni and Alessandro Moschitti |
| Abstract | Complex Question Answering is a discipline that involves a deep understandingof question/answer relations, such as those characterizing definition andprocedural questions and their answers. To contribute to the improvement of this technology, we deliver two question and answer corpora for complex questions, WEB-QA andTREC-QA, extracted by the same Question Answering system, YourQA, from the Web and from the AQUAINT-6 data collection respectively. We believe that such corpora can be useful resources to address a type of QA that is far from being efficiently solved. WEB-QA and TREC-QA are available in two formats: judgment files and training/testing files. Judgment files contain a ranked list of candidate answers to TREC-10 complex questions, extracted using YourQA as a baseline system and manually labelled according to a Likert scale from 1 (completely incorrect) to 5 (totally correct). Training and testing files contain learning instances compatible withSVM-light; these are useful for experimenting with shallow and complex structural features such as parsetrees and semantic role labels. Our experiments with the above corpora have allowed to prove that structured information representation is useful to improve the accuracy of complex QA systems and to re-rank answers. |
| Language | Evaluation methodologies |
| Topics | Question Answering, Corpus (creation, annotation, etc.), Evaluation methodologies |
Full paper  |
A Comprehensive Resource to Evaluate Complex Open Domain Question Answering |
| Bibtex | @InProceedings{QUARTERONI10.188, author = {Silvia Quarteroni and Alessandro Moschitti}, title = {A Comprehensive Resource to Evaluate Complex Open Domain Question Answering}, booktitle = {Proceedings of the Seventh conference on International Language Resources and Evaluation (LREC'10)}, year = {2010}, month = {may}, date = {19-21}, address = {Valletta, Malta}, editor = {Nicoletta Calzolari (Conference Chair), Khalid Choukri, Bente Maegaard, Joseph Mariani, Jan Odjik, Stelios Piperidis, Mike Rosner, Daniel Tapias}, publisher = {European Language Resources Association (ELRA)}, isbn = {2-9517408-6-7}, language = {english} } |