Abstract

We present a generalization of score-based diffusion models to function space by perturbing functional data via a Gaussian process at multiple scales. We obtain an appropriate notion of score by defining densities with respect to Guassian measures and generalize denoising score matching. We then define the generative process by integrating a function-valued Langevin dynamic. We show that the corresponding discretized algorithm generates samples at a fixed cost that is independent of the data discretization. As an application for such a model, we formulate video generation as a sequence of joint inpainting and interpolation problems defined by frame deformations. We train an image diffusion model using Gaussian process inputs and use it to solve the video generation problem by enforcing equivariance with respect to frame deformations. Our results are state-of-the-art for video generation using models trained only on image data.