Abstract

Training generative AI is not a one-step process. In the case of large language models (LLMs), self-supervision is often followed by supervised and reinforcement learning stages to improve instruction following, safety, and other desirable qualities. This multi-stage process that has emerged in the last 3 years has led to huge leaps in model capabilities. It has also led to new challenges and risks. In this talk, I will overview some of our group’s work to identify and address such challenges by focusing on the training data used at different stages. First, I will discuss the problem of adapting LLMs to new, specialized domains and the role that synthetic, i.e. LLM -generated, training data can play. Then, I will share some of our work showing how mismatches in training data at different stages can lead to safety alignment risks. In one case, LLMs with inadequate safety training can be more likely to respond to harmful queries when presented in languages with less abundant data like Swahili or Scots Gaelic. In another case, LLMs trained to reason about solving math problems can then deploy those same reasoning skills to reason out of their own safety guardrails. Together, these findings highlight the importance of careful training data management at all stages of AI development.

Bio: Stephen Bach is the Eliot Horowitz Assistant Professor of Computer Science at Brown University. His latest research is on improving the processes by which humans teach and instruct computers. That includes learning to generalize from fewer examples, with methods like zero-shot and few-shot learning, as well as engineering training data, with methods like synthetic data generation and programmatic weak supervision. He was a core contributor to the Snorkel framework, which was recognized with a Best of VLDB 2018 award. Snorkel has been used in production at numerous Fortune 500 companies for programmatic training data curation. He also co-led the team that developed the T0 family of large language models. The team was also one of the proposers of instruction tuning, which is the process of fine-tuning language models with supervised training to follow instructions. Instruction tuning is now a standard part of training large language models. Stephen is also an advisor to Snorkel AI, a company that provides software and services for data-centric AI.