Summary of the paper
| Title | Appraise: An Open-Source Toolkit for Manual Phrase-Based Evaluation of Translations |
|---|---|
| Authors | Christian Federmann |
| Abstract | We describe a focused effort to investigate the performance of phrase-based,human evaluation of machine translation output achieving a high annotatoragreement. We define phrase-based evaluation and describe the implementation ofAppraise, a toolkit that supports the manual evaluation of machine translationresults. Phrase ranking can be done using either a fine-grained six-way scoringscheme that allows to differentiate between "much better" and "slightlybetter", or a reduced subset of ranking choices. Afterwards we discuss kappavalues for both scoring models from several experiments conducted with humanannotators. Our results show that phrase-based evaluation can be used for fastevaluation obtaining significant agreement among annotators. The granularity ofranking choices should, however, not be too fine-grained as this seems toconfuse annotators and thus reduces the overall agreement. The work reported inthis paper confirms previous work in the field and illustrates that the usageof human evaluation in machine translation should be reconsidered. The Appraisetoolkit is available as open-source and can be downloaded from the author'swebsite. |
| Language | Evaluation methodologies |
| Topics | Tools, systems, applications, Machine Translation, SpeechToSpeech Translation, Evaluation methodologies |
Full paper  |
Appraise: An Open-Source Toolkit for Manual Phrase-Based Evaluation of Translations |
| Bibtex | @InProceedings{FEDERMANN10.197, author = {Christian Federmann}, title = {Appraise: An Open-Source Toolkit for Manual Phrase-Based Evaluation of Translations}, booktitle = {Proceedings of the Seventh conference on International Language Resources and Evaluation (LREC'10)}, year = {2010}, month = {may}, date = {19-21}, address = {Valletta, Malta}, editor = {Nicoletta Calzolari (Conference Chair), Khalid Choukri, Bente Maegaard, Joseph Mariani, Jan Odjik, Stelios Piperidis, Mike Rosner, Daniel Tapias}, publisher = {European Language Resources Association (ELRA)}, isbn = {2-9517408-6-7}, language = {english} } |