Taming Time and Dimension: Advanced Scientific Computing for Next-Generation Diffusion Models
Project Description
Existing background work
Diffusion models have emerged as a powerful class of generative models, with applications in image, audio, and 3D synthesis. This PhD project aims to advance the computational efficiency of diffusion models by addressing two primary challenges: expensive time integration during sampling and the curse of dimensionality in data representation. We propose to develop novel scientific computing methods combining advanced time integration schemes with high-dimensional approximation techniques. This interdisciplinary approach will bridge state-of-the-art generative AI with classical numerical analysis and modern tensor approximation methods, potentially unlocking major efficiency gains while maintaining or improving generation quality.
Existing background work
Recent advancements in diffusion model sampling have focused on either improving time integration or tackling high-dimensional representations. On the time integration front, methods like DPM-Solver and DEIS have leveraged exponential integrators and sophisticated ODE solvers. For high-dimensional approximation, techniques such as tensor decompositions and sparse grids have shown promise in related fields. However, there remains a significant gap in approaches that effectively combine these two aspects for diffusion models. This project builds upon these foundations while aiming to create a unified framework that addresses both challenges simultaneously.
Relevant references: Sit: Exploring flow and diffusion-based generative models with scalable interpolant transformers Nanye Ma, Mark Goldstein, Michael S Albergo, Nicholas M Boffi, Eric Vanden-Eijnden, Saining Xie arXiv preprint arXiv:2401.08740
Stable generative modeling using Schroedinger bridges Georg Gottwald, Fengyi Li, Youssef Marzouk, Sebastian Reich arXiv preprint arXiv:2401.04372
Generative Modelling with Tensor Train approximations of Hamilton–Jacobi–Bellman equations David Sommer, Robert Gruhlke, Max Kirstein, Martin Eigel, Claudia Schillings arXiv preprint arXiv:2402.15285
Improved Order Analysis and Design of Exponential Integrator for Diffusion Models Sampling Qinsheng Zhang, Jiaming Song, Yongxin Chen arXiv preprint arXiv:2308.02157
Main objectives of the project
The primary objectives of this project are to: (1) Develop novel time integration schemes tailored for diffusion model ODEs/SDEs, incorporating adaptive stepping and error control. (2) Investigate high-dimensional approximation techniques, including tensor decompositions and sparse grids, for efficient representation of diffusion model states and operators. (3) Design algorithms that combine advanced time integration with high-dimensional approximation, including multi-level and multi-fidelity approaches. (4) Conduct rigorous theoretical analysis of the proposed methods, deriving stability conditions, convergence rates, and error bounds.
Details of Software/Data Deliverables
The project will produce a suite of scientific computing algorithms specifically designed for diffusion models. These will include novel time integration schemes, high-dimensional approximation techniques, and hybrid methods that combine both approaches. The algorithms will be built with a focus on both scalability to large-scale diffusion models and modularity, enabling easy integration with existing diffusion model implementations.
