Background

In score-based diffusion models , the authors proposed a unified framework that connects the score-based model NCSN and DDPM through the perspective of Stochastic Differential Equations (SDE). They interpret the forward process (adding noise) and the backward process (denoising sampling) as SDE and reverse SDE, respectively.

Data Perturbation with Itô SDE

The diffusion process ${{\mathit{x}}_t}_{t=0}^T$ from the original input ${\mathit{x}}_0$ to Gaussian noise ${\mathit{x}}_T$ with continuous time variable $t\in [0,T]$ can be modeled by the Itô SDE

• ${\mathit{w}}_t$ is the standard Wiener process. ${\mathit{w}}_t\sim \mathcal{N}(0,t)\to {\mathit{w}}_{t+\mathrm{\Delta }t}-{\mathit{w}}_t\sim \mathcal{N}(0,\mathrm{\Delta }t)\to d{\mathit{w}}_t\sim \sqrt{\mathrm{\Delta }t}\mathit{ϵ}\mathit{ϵ}\sim \mathcal{N}(0,\mathit{I})$
• $\mathbf{f}(\mathit{x},t):{\mathbb{R}}^d\to {\mathbb{R}}^d$ are the drift coefficients and $d$ is the dimension of $\mathit{x}$. It is always affine, resulting in $f(\mathit{x},t)=f(t)\mathit{x}$ where $f(\cdot ):\mathbb{R}\to \mathbb{R}$.
• $\mathit{G}(\mathit{x},t):{\mathbb{R}}^d\to {\mathbb{R}}^{d\times d}$ are the diffusion coefficients at time $t$.

We consider the case $\mathit{G}(\mathit{x},t)=g(t)\mathit{I}$ which is independent of $\mathit{x}$. Then, Eq. (1) can be rewritten as

The perturbation kernel of this SDE have the form

where

Here, I’ll use $s_t$ and $s(t)$, and ${\sigma }_t$ and $\sigma (t)$ interchangeably for simplicity. The corresponding marginal distribution is obtained by

The Fokker-Plank equation describes the evolution of the marginal distribution $p_t(\mathit{x})$, interchangeable with $p({\mathit{x}}_t)$, over time under the effect of drift forces and random (or noise) forces. It can be written as

where

If we consider the case $\mathit{G}(\mathit{x},t)=g(t)\mathit{I}$ which is independent of $\mathit{x}$, then Eq. $\text{13}$ can be rewritten as

Probability Flow ODE

According to the Fokker-Plank equation, there exists an ODE which shares the same marginal distribution $p(\mathit{x})$ as the SDE. From Eq. $\text{15}$, the corresponding SDE is reduced to ODE given by

The ODE is named as the probability flow ODE (PF ODE).

If we further build the ODE according to $s_t$ and ${\sigma }_t$, we have

The derivation can be found in EDM paper (Eq. 28 and 34). Then, Eq. $\text{17}$ can be rewritten as

where the marginal $p_t(\mathit{x})=s_t^{-1}p({\mathit{x}}_t{s}_t^{-1};{\sigma }_t)$ and

as shown in Eq. $\text{9}$.

Connection Between PF ODE and SDE

According to the Eq. (102) in , the authors derived a family of SDE for any choice of $g(t)$. The SDE is given by

The PF ODE is a special case of the SDE when $g(t)=0$. The derivation is given in the EDM paper (Appendix B.5). It is a little bit long but not difficult to follow. Furthermore, the authors parameterized $g(t)=\sqrt{2\beta (t)}{\sigma }_t$ and Eq. $\text{23}$ becomes

where $\beta (t)$ is a free function.

Reverse SDE

The reverse diffusion process from ${\mathit{x}}_T$ to ${\mathit{x}}_0$ can also be modeled by Itô SDE according to :

Unified Framework of Score Diffusion Models

NCSN (VE SDE)

DDPM (VP SDE)

DDIM

Preconditioning

Sampling

Related Topics

Flow Matching

Consistency Model