Summary of the paper
| Title | Ad-hoc Evaluations Along the Lifecycle of Industrial Spoken Dialogue Systems: Heading to Harmonisation? |
|---|---|
| Authors | Marianne Laurent, Philippe Bretier and Carole Manquillet |
| Abstract | With a view to rationalise the evaluation process within the Orange Labs spokendialogue system projects, a field audit has been realised among the variousrelated professionals. The article presents the study's main conclusions anddraws work perspectives to enhance the evaluation process in such a complexorganisation. We first present the typical spoken dialogue system projectlifecycle and the involved communities of stakeholders. We then sketch a map ofindicators used across the teams. It shows that each professional categorydesigns its evaluation metrics according to a case-by-case strategy, each onetargeting different goals and methodologies. And last, we identify weaknessesin the evaluation process is handled by the various teams. Among others, wemention:the dependency on the design and exploitation tools that may not be suitablefor an adequate collection of relevant indicators, the need to refine someindicators' definition and analysis to obtain valuable information for systemenhancement, the sharing issue that advocates for a common definition ofindicators across the teams and, as a consequence, the need for sharedapplications that support and encourage such a rationalisation. |
| Language | Corpus (creation, annotation, etc.) |
| Topics | Evaluation methodologies, Dialogue, Corpus (creation, annotation, etc.) |
Full paper  |
Ad-hoc Evaluations Along the Lifecycle of Industrial Spoken Dialogue Systems: Heading to Harmonisation? |
| Bibtex | @InProceedings{LAURENT10.133, author = {Marianne Laurent, Philippe Bretier and Carole Manquillet}, title = {Ad-hoc Evaluations Along the Lifecycle of Industrial Spoken Dialogue Systems: Heading to Harmonisation?}, booktitle = {Proceedings of the Seventh conference on International Language Resources and Evaluation (LREC'10)}, year = {2010}, month = {may}, date = {19-21}, address = {Valletta, Malta}, editor = {Nicoletta Calzolari (Conference Chair), Khalid Choukri, Bente Maegaard, Joseph Mariani, Jan Odjik, Stelios Piperidis, Mike Rosner, Daniel Tapias}, publisher = {European Language Resources Association (ELRA)}, isbn = {2-9517408-6-7}, language = {english} } |