Similarity-Based Estimation of Word Cooccurrence Probabilities
| dc.creator | Dagan, Ido | |
| dc.creator | Pereira, Fernando | |
| dc.creator | Lee, Lillian | |
| dc.date | 1994-05-02 | |
| dc.date.accessioned | 2026-07-07T09:08:14Z | |
| dc.date.available | 2026-07-07T09:08:14Z | |
| dc.description | In many applications of natural language processing it is necessary to determine the likelihood of a given word combination. For example, a speech recognizer may need to determine which of the two word combinations ``eat a peach'' and ``eat a beach'' is more likely. Statistical NLP methods determine the likelihood of a word combination according to its frequency in a training corpus. However, the nature of language is such that many word combinations are infrequent and do not occur in a given corpus. In this work we propose a method for estimating the probability of such previously unseen word combinations using available information on ``most similar'' words. We describe a probabilistic word association model based on distributional word similarity, and apply it to improving probability estimates for unseen word bigrams in a variant of Katz's back-off model. The similarity-based method yields a 20% perplexity improvement in the prediction of unseen bigrams and statistically significant reductions in speech-recognition error. | |
| dc.description | 13 pages, to appear in proceedings of ACL-94 | |
| dc.identifier | https://arxiv.org/abs/cmp-lg/9405001 | |
| dc.identifier | http://arxiv.org/abs/cmp-lg/9405001 | |
| dc.identifier.uri | http://salesiana.dossiersoluciones.com/handle/123456789/150649 | |
| dc.subject | Computation and Language | |
| dc.title | Similarity-Based Estimation of Word Cooccurrence Probabilities | |
| dc.type | text |