Summary of the paper
| Title | Efficient Minimal Perfect Hash Language Models |
|---|---|
| Authors | David Guthrie, Mark Hepple and Wei Liu |
| Abstract | The availability of large collections of text have made it possible to buildlanguage models that incorporate counts of billions of n-grams. This paperproposes two new methods of efficiently storing large language models thatallow O(1) random access and use significantly less space than all knownapproaches. We introduce two novel data structures that take advantage of thedistribution of n-grams in corpora and make use of various numbers of minimalperfect hashes to compactly store language models containing full frequencycounts of billions of n-grams using 2.5 Bytes per n-gram and language models ofquantized probabilities using 2.26 Bytes per n-gram. These methods allowlanguage processing applications to take advantage of much larger languagemodels than previously was possible using the same hardware and we additionallydescribe how they can be used in a distributed environment to store even largermodels. We show that our approaches are simple to implement and can easily becombined with pruning and quantization to achieve additional reductions in thesize of the language model. |
| Language | Machine Translation, SpeechToSpeech Translation |
| Topics | Language modelling, Speech Recognition/Understanding, Machine Translation, SpeechToSpeech Translation |
Full paper  |
Efficient Minimal Perfect Hash Language Models |
| Bibtex | @InProceedings{GUTHRIE10.860, author = {David Guthrie, Mark Hepple and Wei Liu}, title = {Efficient Minimal Perfect Hash Language Models}, booktitle = {Proceedings of the Seventh conference on International Language Resources and Evaluation (LREC'10)}, year = {2010}, month = {may}, date = {19-21}, address = {Valletta, Malta}, editor = {Nicoletta Calzolari (Conference Chair), Khalid Choukri, Bente Maegaard, Joseph Mariani, Jan Odjik, Stelios Piperidis, Mike Rosner, Daniel Tapias}, publisher = {European Language Resources Association (ELRA)}, isbn = {2-9517408-6-7}, language = {english} } |