Learning to Create and Reuse Words in Open-Vocabulary Language Modeling

Add to your list(s) Download to your calendar using vCal

• Kazuya Kawakami (DeepMind)
• Thursday 23 November 2017, 11:00-12:00
• SR-24, English Faculty Building, 9 West Road (Sidgwick Site).

If you have a question about this talk, please contact Dimitri Kartsaklis.

Fixed-vocabulary language models fail to account for one of the most characteristic statistical facts of natural language: the frequent creation and reuse of new word types. Although character-level language models offer a partial solution in that they can create word types not attested in the training corpus, they do not capture the “bursty” distribution of such words. In this talk, we discuss a hierarchical LSTM language model that generates sequences of word tokens character by character with a caching mechanism that learns to reuse previously generated words. To validate our model we construct a new open-vocabulary language modeling corpus (the Multilingual Wikipedia Corpus; MWC ) from comparable Wikipedia articles in 7 typologically diverse languages and demonstrate the effectiveness of our model across this range of languages.

This talk is part of the Language Technology Lab Seminars series.

This talk is included in these lists:

• SR-24, English Faculty Building, 9 West Road (Sidgwick Site)
• Cambridge Centre for Data-Driven Discovery (C2D3)
• Cambridge Forum of Science and Humanities
• Cambridge Language Sciences
• Cambridge talks
• Chris Davis' list
• Guy Emerson's list
• Interested Talks
• Language Sciences for Graduate Students
• Language Technology Lab Seminars
• Simon Baker's List
• Trust & Technology Initiative - interesting events
• bld31
• ndk22's list
• ob366-ai4er
• rp587

Note that ex-directory lists are not shown.