Summary of the paper
| Title | Ways of Evaluation of the Annotators in Building the Prague Czech-English Dependency Treebank |
|---|---|
| Authors | Marie Mikulová and Jan Štěpánek |
| Abstract | In this paper, we present several ways to measure and evaluate the annotationand annotators, proposed and used during the building of the Czech part of thePrague Czech-English Dependency Treebank. At first, the basic principles of thetreebank annotation project are introduced (division to three layers:morphological, analytical and tectogrammatical). The main part of the paperdescribes in detail one of the important phases of the annotation process:three ways of evaluation of the annotators - inter-annotator agreement, errorrate and performance. The measuring of the inter-annotator agreement iscomplicated by the fact that the data contain added and deleted nodes, makingthe alignment between annotations non-trivial. The error rate is measured by aset of automatic checking procedures that guard the validity of some invariantsin the data. The performance of the annotators is measured by a booking webapplication. All three measures are later compared and related to each other. |
| Language | LR national/international projects, organizational/policy issues |
| Topics | Corpus (creation, annotation, etc.), Evaluation methodologies, LR national/international projects, organizational/policy issues |
Full paper  |
Ways of Evaluation of the Annotators in Building the Prague Czech-English Dependency Treebank |
| Bibtex | @InProceedings{MIKULOV10.388, author = {Marie Mikulová and Jan Štěpánek}, title = {Ways of Evaluation of the Annotators in Building the Prague Czech-English Dependency Treebank}, booktitle = {Proceedings of the Seventh conference on International Language Resources and Evaluation (LREC'10)}, year = {2010}, month = {may}, date = {19-21}, address = {Valletta, Malta}, editor = {Nicoletta Calzolari (Conference Chair), Khalid Choukri, Bente Maegaard, Joseph Mariani, Jan Odjik, Stelios Piperidis, Mike Rosner, Daniel Tapias}, publisher = {European Language Resources Association (ELRA)}, isbn = {2-9517408-6-7}, language = {english} } |