Abstract

Diffusion models have emerged as a powerful framework for generative modeling that relies on score matching to learn gradients of the data distribution’s log-density. A key element for the success of diffusion models is that the optimal score function is not identified when solving the denoising score matching problem. In fact, the optimal score in both unconditioned and conditioned settings leads to a diffusion model that returns to the training samples and effectively memorizes the data distribution. In this presentation, we study the dynamical system associated with the optimal score and describe its long-term behavior relative to the training samples. Lastly, we show the effect of three forms of score function regularization on avoiding memorization: restricting the score’s approximation space, early stopping of the training process, and early stopping of the diffusion process during sample generation. Moreover, we establish a connection between early stopping of the diffusion and explicit Tikhonov regularization of the score matching problem. These results are numerically validated using distributions with and without densities including image-based problems.