Summary of the paper
| Title | Evaluating Machine Translation Utility via Semantic Role Labels |
|---|---|
| Authors | Chi-kiu Lo and Dekai Wu |
| Abstract | We present the methodology that underlies mew metrics for semantic machinetranslation evaluation we are developing. Unlike widely-used lexical and n-grambased MT evaluation metrics, the aim of semantic MT evaluation is to measurethe utility of translations. We discuss the design of empirical studies toevaluate the utility of machine translation output by assessing the accuracyfor key semantic roles. These roles are from the English 5W templates (who,what, when, where, why) used in recent GALE distillation evaluations. Recentwork by Wu and Fung (2009) introduced semantic role labeling into statisticalmachine translation to enhance the quality of MT output. However, this approachhas so far only been evaluated using lexical and n-gram based SMT evaluationmetrics like BLEU which are not aimed at evaluating the utility of MT output.Direct data analysis are still needed to understand how semantic models can beleveraged to evaluate the utility of MT output. In this paper, we discuss a newmethodology for evaluating the utility of the machine translation output, byassessing the accuracy with which human readers are able to complete theEnglish 5W templates. |
| Language | Semantics |
| Topics | Machine Translation, SpeechToSpeech Translation, Evaluation methodologies, Semantics |
Full paper  |
Evaluating Machine Translation Utility via Semantic Role Labels |
| Bibtex | @InProceedings{LO10.752, author = {Chi-kiu Lo and Dekai Wu}, title = {Evaluating Machine Translation Utility via Semantic Role Labels}, booktitle = {Proceedings of the Seventh conference on International Language Resources and Evaluation (LREC'10)}, year = {2010}, month = {may}, date = {19-21}, address = {Valletta, Malta}, editor = {Nicoletta Calzolari (Conference Chair), Khalid Choukri, Bente Maegaard, Joseph Mariani, Jan Odjik, Stelios Piperidis, Mike Rosner, Daniel Tapias}, publisher = {European Language Resources Association (ELRA)}, isbn = {2-9517408-6-7}, language = {english} } |