Abstract

Autoregressive sequence-to-sequence models with attention mechanisms have achieved state-of-the-art performance in various tasks including Neural Machine Translation (NMT), Automatic Speech Recognition (ASR) and Text-To-Speech (TTS). This talk introduces attention forcing, a group of training approaches, to address a training-inference mismatch. For autoregressive models, the most standard training approach, teacher forcing, guides a model with the reference output history. However during inference the generated output history must be used. To reduce the mismatch, attention forcing guides the model with the generated output history and reference attention. Extensions of this general framework will be introduced for more challenging applications. For example, most approaches addressing the training-inference mismatch are incompatible with parallel training, which is essential for Transformer models. In contrast, the parallel version of attention forcing supports parallel training, and hence Transformer models. The effectiveness of attention forcing will be demonstrated by the experiments in TTS and NMT .