Summary of the paper
| Title | Dialogues in Context: An Objective User-Oriented Evaluation Approach for Virtual Human Dialogue |
|---|---|
| Authors | Susan Robinson, Antonio Roque and David Traum |
| Abstract | As conversational agents are now being developed to encounter more complexdialogue situations it is increasingly difficult to find satisfactory methodsfor evaluating these agents. Task-based measures are insufficient where thereis no clearly defined task. While user-based evaluation methods may give ageneral sense of the quality of an agent's performance, they shed little lighton the relative quality or success of specific features of dialogue that arenecessary for system improvement. This paper examines current dialogue agentevaluation practices and motivates the need for a more detailed approach fordefining and measuring the quality of dialogues between agent and user. Wepresent a framework for evaluating the dialogue competence of artificial agentsinvolved in complex and underspecified tasks when conversing with people. Amulti-part coding scheme is proposed that provides a qualitative analysis ofhuman utterances, and rates the appropriateness of the agent's responses tothese utterances. The scheme is outlined, and then used to evaluate Staff DutyOfficer Moleno, a virtual guide in Second Life. |
| Language | Tools, systems, applications |
| Topics | Evaluation methodologies, Dialogue, Tools, systems, applications |
Full paper  |
Dialogues in Context: An Objective User-Oriented Evaluation Approach for Virtual Human Dialogue |
| Bibtex | @InProceedings{ROBINSON10.674, author = {Susan Robinson, Antonio Roque and David Traum}, title = {Dialogues in Context: An Objective User-Oriented Evaluation Approach for Virtual Human Dialogue}, booktitle = {Proceedings of the Seventh conference on International Language Resources and Evaluation (LREC'10)}, year = {2010}, month = {may}, date = {19-21}, address = {Valletta, Malta}, editor = {Nicoletta Calzolari (Conference Chair), Khalid Choukri, Bente Maegaard, Joseph Mariani, Jan Odjik, Stelios Piperidis, Mike Rosner, Daniel Tapias}, publisher = {European Language Resources Association (ELRA)}, isbn = {2-9517408-6-7}, language = {english} } |