Summary of the paper
| Title | Mining the Correlation between Human and Automatic Evaluation at Sentence Level |
|---|---|
| Authors | Yanli Sun |
| Abstract | Automatic evaluation metrics are fast and cost-effective measurements of thequality of a Machine Translation (MT) system. However, as humans are theend-user of MT output, human judgement is the benchmark to assess theusefulness of automatic evaluation metrics. While most studies report thecorrelation between human evaluation and automatic evaluation at corpus level,our study examines their correlation at sentence level. In addition to thestatistical correlation scores, such as Spearman's rank-order correlationcoefficient, a finer-grained and detailed examination of the sensitivity ofautomatic metrics compared to human evaluation is also reported in this study.The results show that the threshold for human evaluators to agree with thejudgements of automatic metrics varies with the automatic metrics at sentencelevel. While the automatic scores for two translations are greatly different,human evaluators may consider the translations to be qualitatively similar andvice versa. The detailed analysis of the correlation between automatic andhuman evaluation allows us determine with increased confidence whether anincrease in the automatic scores will be agreed by human evaluators or not. |
| Language | Tools, systems, applications |
| Topics | Machine Translation, SpeechToSpeech Translation, Evaluation methodologies, Tools, systems, applications |
Full paper  |
Mining the Correlation between Human and Automatic Evaluation at Sentence Level |
| Bibtex | @InProceedings{SUN10.87, author = {Yanli Sun}, title = {Mining the Correlation between Human and Automatic Evaluation at Sentence Level}, booktitle = {Proceedings of the Seventh conference on International Language Resources and Evaluation (LREC'10)}, year = {2010}, month = {may}, date = {19-21}, address = {Valletta, Malta}, editor = {Nicoletta Calzolari (Conference Chair), Khalid Choukri, Bente Maegaard, Joseph Mariani, Jan Odjik, Stelios Piperidis, Mike Rosner, Daniel Tapias}, publisher = {European Language Resources Association (ELRA)}, isbn = {2-9517408-6-7}, language = {english} } |