Summary of the paper
| Title | Paragraph Acquisition and Selection for List Question Using Amazon’s Mechanical Turk |
|---|---|
| Authors | Fang Xu and Dietrich Klakow |
| Abstract | Creating more fine-grained annotated data than previously relevent documentsets is important for evaluating individual components in automatic questionanswering systems. In this paper, we describe using the Amazon's MechanicalTurk (AMT) to judge whether paragraphs in relevant documents answercorresponding list questions in TREC QA track 2004. Based on AMT results, webuild a collection of 1300 gold-standard supporting paragraphs for listquestions. Our online experiments suggested that recruiting more people pertask assures better annotation quality. In order to learning true labels fromAMT annotations, we investigated three approaches on two datasets withdifferent levels of annotation errors. Experimental studies show that the NaiveBayesian model and EM-based GLAD model can generate results highly agreeingwith gold-standard annotations, and dominate significantly over the majorityvoting method for true label learning. We also suggested setting higher HITapproval rate to assure better online annotation quality, which leads to betterperformance of learning methods. |
| Language | Statistical and machine learning methods |
| Topics | Corpus (creation, annotation, etc.), Question Answering, Statistical and machine learning methods |
Full paper  |
Paragraph Acquisition and Selection for List Question Using Amazon’s Mechanical Turk |
| Bibtex | @InProceedings{XU10.241, author = {Fang Xu and Dietrich Klakow}, title = {Paragraph Acquisition and Selection for List Question Using Amazon’s Mechanical Turk}, booktitle = {Proceedings of the Seventh conference on International Language Resources and Evaluation (LREC'10)}, year = {2010}, month = {may}, date = {19-21}, address = {Valletta, Malta}, editor = {Nicoletta Calzolari (Conference Chair), Khalid Choukri, Bente Maegaard, Joseph Mariani, Jan Odjik, Stelios Piperidis, Mike Rosner, Daniel Tapias}, publisher = {European Language Resources Association (ELRA)}, isbn = {2-9517408-6-7}, language = {english} } |