Summary of the paper
| Title | Determining Reliability of Subjective and Multi-label Emotion Annotation through Novel Fuzzy Agreement Measure |
|---|---|
| Authors | Plaban Kr. Bhowmick, Anupam Basu and Pabitra Mitra |
| Abstract | The paper presents a new fuzzy agreement measure $\gamma_f$ for determining theagreement in multi-label and subjective annotation task. In this annotationframework, one data item may belong to a category or a class with a beliefvalue denoting the degree of confidence of an annotator in assigning the dataitem to that category. We have provided a notion of disagreement based on thebelief values provided by the annotators with respect to a category. The fuzzyagreement measure $\gamma_f$ has been proposed by defining different fuzzyagreement sets based on the distribution of difference of belief valuesprovided by the annotators. The fuzzy agreement has been computed by studyingthe average agreement over all the data items and annotators. Finally, weelaborate on the computation $\gamma_f$ measure with a case study on emotiontext data where a data item (sentence) may belong to more than one emotioncategory with varying belief values. |
| Language | Validation of LRs |
| Topics | Corpus (creation, annotation, etc.), Emotion Recognition/Generation, Validation of LRs |
Full paper  |
Determining Reliability of Subjective and Multi-label Emotion Annotation through Novel Fuzzy Agreement Measure |
| Bibtex | @InProceedings{BHOWMICK10.67, author = {Plaban Kr. Bhowmick, Anupam Basu and Pabitra Mitra}, title = {Determining Reliability of Subjective and Multi-label Emotion Annotation through Novel Fuzzy Agreement Measure}, booktitle = {Proceedings of the Seventh conference on International Language Resources and Evaluation (LREC'10)}, year = {2010}, month = {may}, date = {19-21}, address = {Valletta, Malta}, editor = {Nicoletta Calzolari (Conference Chair), Khalid Choukri, Bente Maegaard, Joseph Mariani, Jan Odjik, Stelios Piperidis, Mike Rosner, Daniel Tapias}, publisher = {European Language Resources Association (ELRA)}, isbn = {2-9517408-6-7}, language = {english} } |