Summary of the paper
| Title | An Evaluation Framework for Natural Language Understanding in Spoken Dialogue Systems |
|---|---|
| Authors | Joshua B. Gordon and Rebecca J. Passonneau |
| Abstract | We present an evaluation framework to enable developers of information seeking,transaction based spoken dialogue systems to compare the robustness of naturallanguage understanding (NLU) approaches across varying levels of word errorrate and contrasting domains. We develop statistical and semantic parsing basedapproaches to dialogue act identification and concept retrieval. Voice searchis used in each approach to ultimately query the database. Included in theframework is a method for developers to bootstrap a representativepseudo-corpus, which is used to estimate NLU performance in a new domain. Weillustrate the relative merits of these NLU techniques by contrasting ourstatistical NLU approach with a semantic parsing method over two contrastingapplications, our CheckItOut library system and the deployed Let’s Go Public!system, across four levels of word error rate. We find that with respect toboth dialogue act identification and concept retrieval, our statistical NLUapproach is more likely to robustly accommodate the freer form, lessconstrained utterances of CheckItOut at higher word error rates than ispossible with semantic parsing. |
| Language | Speech Recognition/Understanding |
| Topics | Evaluation methodologies, Dialogue, Speech Recognition/Understanding |
Full paper  |
An Evaluation Framework for Natural Language Understanding in Spoken Dialogue Systems |
| Bibtex | @InProceedings{GORDON10.928, author = {Joshua B. Gordon and Rebecca J. Passonneau}, title = {An Evaluation Framework for Natural Language Understanding in Spoken Dialogue Systems}, booktitle = {Proceedings of the Seventh conference on International Language Resources and Evaluation (LREC'10)}, year = {2010}, month = {may}, date = {19-21}, address = {Valletta, Malta}, editor = {Nicoletta Calzolari (Conference Chair), Khalid Choukri, Bente Maegaard, Joseph Mariani, Jan Odjik, Stelios Piperidis, Mike Rosner, Daniel Tapias}, publisher = {European Language Resources Association (ELRA)}, isbn = {2-9517408-6-7}, language = {english} } |