Abstract

The recent success of neural networks has raised questions about the role of linguistic structure in NLP models, but also opens up new opportunities. In this talk I’ll show how neural networks enable us to predict richer linguistic representations than previously feasible, and explore how to incorporate linguistically-informed structural biases in language generation models. First, I’ll present a robust end-to-end parser for Minimal Recursion Semantics (MRS), a framework for compositional semantics implemented in high-precision computational grammars. This task presents several challenges for structure prediction models; I’ll show how they can be addressed through generalizing transition-based parsing in the framework of encoder-decoder recurrent neural networks, and using pointer networks. Results show that incorporating structure into the neural architecture improves performance over attention-based baselines by a large margin on both MRS and Abstract Meaning Representation parsing. Second, I’ll present a generative dependency parser which also serves as a neural syntactic language model. A feed-forward architecture and a decoding algorithm based on particle filtering enable efficient and accurate inference, while unsupervised fine-tuning improves language modelling perplexity. Finally, I’ll present a sequence-to-sequence model in which the alignment between the input and output is a latent variable marginalized through dynamic programming. The model is structured to learn mostly monotone alignments, which makes it applicable to many transduction tasks while enabling online inference.