Summary of the paper
| Title | Building a Textual Entailment Suite for the Evaluation of Automatic Content Scoring Technologies |
|---|---|
| Authors | Jana Z. Sukkarieh and Eleanor Bolge |
| Abstract | Automatic content scoring for free-text responses has started to emerge as anapplication of Natural Language Processing in its own right, much like questionanswering or machine translation. The task, in general, is reduced to comparinga student’s answer to a model answer. Although a considerable amount of workhas been done, common benchmarks and evaluation measures for this applicationdo not currently exist. It is yet impossible to perform a comparativeevaluation or progress tracking of this application across systems ― anapplication that we view as a textual entailment task. This paper concentrateson introducing an Educational Testing Service-built test suite that makes astep towards establishing such a benchmark. The suite can be used as regressionand performance evaluations both intra-c-rater® or inter automatic contentscoring technologies. It is important to note that existing textual entailmenttest suites like PASCAL RTE or FraCas, though beneficial, are not suitable forour purposes since we deal with atypical naturally-occurring student responsesthat need to be categorized in order to serve as regression test cases. |
| Language | Evaluation methodologies |
| Topics | Textual Entailment and Paraphrasing, Corpus (creation, annotation, etc.), Evaluation methodologies |
Full paper  |
Building a Textual Entailment Suite for the Evaluation of Automatic Content Scoring Technologies |
| Bibtex | @InProceedings{SUKKARIEH10.310, author = {Jana Z. Sukkarieh and Eleanor Bolge}, title = {Building a Textual Entailment Suite for the Evaluation of Automatic Content Scoring Technologies}, booktitle = {Proceedings of the Seventh conference on International Language Resources and Evaluation (LREC'10)}, year = {2010}, month = {may}, date = {19-21}, address = {Valletta, Malta}, editor = {Nicoletta Calzolari (Conference Chair), Khalid Choukri, Bente Maegaard, Joseph Mariani, Jan Odjik, Stelios Piperidis, Mike Rosner, Daniel Tapias}, publisher = {European Language Resources Association (ELRA)}, isbn = {2-9517408-6-7}, language = {english} } |