Summary of the paper
| Title | Fred’s Reusable Evaluation Device: Providing Support for Quick and Reliable Linguistic Annotation |
|---|---|
| Authors | Hannah Copperman and Christopher R. Walker |
| Abstract | This paper describes an interface that was developed for processing largeamounts of human judgments of linguistically annotated data. Fred’s ReusableEvaluation Device (“Fred”) provides administrators with a tool to submitlinguistic evaluation tasks to judges. Each evaluation task is then presentedto exactly two judges, who can submit their judgments at their own leisure.Fred then provides several metrics to administrators. The most important metricis precision, which is provided for each evaluation task and each annotator.Administrators can look at precision for a given data set over time, as well asby evaluation type, data set, or annotator. Inter-annotator agreement is alsoreported, and that can be tracked over time as well. The interface wasdeveloped to provide a tool for evaluating semantically marked up text. Thetypes of evaluations Fred has been used for so far include things likecorrectness of subject-relation identification, and correctness of temporalrelations. However, Fred’s full versatility has not yet been fully exploited. |
| Language | Semantics |
| Topics | Tools, systems, applications, Corpus (creation, annotation, etc.), Semantics |
Full paper  |
Fred’s Reusable Evaluation Device: Providing Support for Quick and Reliable Linguistic Annotation |
| Bibtex | @InProceedings{COPPERMAN10.444, author = {Hannah Copperman and Christopher R. Walker}, title = {Fred’s Reusable Evaluation Device: Providing Support for Quick and Reliable Linguistic Annotation}, booktitle = {Proceedings of the Seventh conference on International Language Resources and Evaluation (LREC'10)}, year = {2010}, month = {may}, date = {19-21}, address = {Valletta, Malta}, editor = {Nicoletta Calzolari (Conference Chair), Khalid Choukri, Bente Maegaard, Joseph Mariani, Jan Odjik, Stelios Piperidis, Mike Rosner, Daniel Tapias}, publisher = {European Language Resources Association (ELRA)}, isbn = {2-9517408-6-7}, language = {english} } |