Abstract

Machine learning models work well on a dataset given enough training examples, but they often fail to isolate and reuse previously acquired knowledge when the data distribution shifts (e.g., when presented with a new dataset or very long context). In contrast, humans are able to learn incrementally and accumulate persistent knowledge to facilitate faster learning of new skills without forgetting old ones.

In this talk, I will argue that obtaining such an ability for a language model requires significant advances in how to represent, store, and reuse knowledge acquired from textual data. I will present a semiparametric language model framework that separates computation (information processing) in a large parametric neural network and memory storage in a non-parametric component. I will show two instantiations of such a model. First, I will discuss how to use it to allow a language model to continually learn new tasks without forgetting old ones. Second, I will present a language model architecture that adaptively combines local context and global context to make more accurate predictions.