BEGIN:VCALENDAR
VERSION:2.0
PRODID:-//Talks.cam//talks.cam.ac.uk//
X-WR-CALNAME:Talks.cam
BEGIN:VEVENT
SUMMARY:Denoising Levy Probabilistic Models - Alain Oliviero-Durmus (Écol
 e Polytechnique)
DTSTART:20240703T093000Z
DTEND:20240703T110000Z
UID:TALK218362@talks.cam.ac.uk
DESCRIPTION:Investigating the noise distribution beyond Gaussians for nois
 e injection in diffusion generative models remains an open problem. The Ga
 ussian case has been a large success experimentally and theoretically\, ad
 mitting a unified stochastic differential equation (SDE) framework\, encom
 passing score-based and denoising formulations. Recent studies have invest
 igated the potential of heavy-tailed noise distributions to mitigate mode 
 collapse and effectively manage datasets exhibiting class imbalance\, heav
 y tails\, or prominent outliers. Very recently\, Yoon et al. (NeurIPS 2023
 )\, presented the Levy-Ito model (LIM) that directly extended the SDE-base
 d framework\, to a class of heavy-tailed SDEs\, where the injected noise f
 ollowed an -stable distribution -- a rich class of heavy-tailed distributi
 ons. Despite its theoretical elegance and performance improvements\, LIM r
 elies on highly involved mathematical techniques\, which may limit its acc
 essibility and hinder its broader adoption and further development. In thi
 s study\, we take a step back\, and instead of starting from the SDE formu
 lation\, we extend the denoising diffusion probabilistic model (DDPM) by d
 irectly replacing the Gaussian noise with -stable noise. We show that\, by
  using only elementary proof techniques\, the proposed approach\, denoisin
 g Levy probabilistic model (DLPM) algorithmically boils down to running va
 nilla DDPM with minor modifications\, hence allowing the use of existing i
 mplementations with minimal changes. Remarkably\, as opposed to the Gaussi
 an case\, DLPM and LIM yield different backward processes leading to disti
 nct sampling algorithms. This fundamental difference translates favorably 
 for the performance of DLPM in various aspects: our experiments show that 
 DLPM achieves better coverage of the tails of the data distribution\, bett
 er generation of unbalanced datasets\, and improved computation times requ
 iring significantly smaller number of backward steps.If time permits I wil
 l also discussScore diffusion models without early stopping: finite Fisher
  information is all you needDiffusion models are a new class of generative
  models that revolve around the estimation of the score function associate
 d with a stochastic differential equation. Subsequent to its acquisition\,
  the approximated score function is then harnessed to simulate the corresp
 onding time-reversal process\, ultimately enabling the generation of appro
 ximate data samples. Despite their evident practical significance these mo
 dels carry\, a notable challenge persists in the form of a lack of compreh
 ensive quantitative results\, especially in scenarios involving non-regula
 r scores and estimators. In almost all reported bounds in Kullback Leibler
  (KL) divergence\, it is assumed that either the score function or its app
 roximation is Lipschitz uniformly in time. However\, this condition is ver
 y restrictive in practice or appears to be difficult to establish.To circu
 mvent this issue\, previous works mainly focused on establishing convergen
 ce bounds in KL for an early stopped version of the diffusion model and a 
 smoothed version of the data distribution\, or assuming that the data dist
 ribution is supported on a compact manifold. These explorations have lead 
 to interesting bounds in either Wasserstein or Fortet-Mourier metrics. How
 ever\, the question remains about the relevance of such early-stopping pro
 cedure or compactness conditions. In particular\, if there exist a natural
  and mild condition ensuring explicit and sharp convergence bounds in KL.I
 n this article\, we tackle the aforementioned limitations by focusing on s
 core diffusion models with fixed step size stemming from the Ornstein-Ulhe
 nbeck semigroup and its kinetic counterpart. Our study provides a rigorous
  analysis\, yielding simple\, improved and sharp convergence bounds in KL 
 applicable to any data distribution with finite Fisher information with re
 spect to the standard Gaussian distribution.
LOCATION:External
END:VEVENT
END:VCALENDAR