3 6 1 Kullback Leibler Divergence Between Gaussian Distributions

In this exercise, we write $\mathbb{P}_{f, \Sigma}$ for the Gaussian distribution $\mathcal{N}(f, \Sigma)$ and we compute the Kullback-Leibler divergence $K L\left(\mathbb{P}_{f_0, \Sigma_0}, \mathbb{P}_{f_1, \Sigma_1}\right)$ for $f_0, f_1 \in \mathbb{R}^n$ and $\Sigma_0, \Sigma_1$ two $n \times n$ symmetric positive definite matrices.

We will denote: $\|x\|_{\Sigma} = \sqrt{x^\top \Sigma^{-1} x}$ the norm associated to a symmetric positive definite matrix $\Sigma$ on $\mathbb{R}^n$.

Also, let us denote:

(1)

\begin{align} p_{f, \Sigma} : x \in \mathbb{R}^n \mapsto (2 \pi)^{-k/2} |\Sigma|^{-1/2} \exp \left( - \frac{1}{2} \| x - f \|_\Sigma^2 \right) \end{align}

the probability density function of the Gaussian distribution $\mathcal{N}(f, \Sigma)$.

1. As a warm-up, let us start with the spherical case where $\Sigma_0=\Sigma_1=\sigma^2 I_n$. Then:

(2)

\begin{aligned} KL \left(\mathbb{P}_{f_0, \sigma^2 I_n}, \mathbb{P}_{f_1, \sigma^2 I_n}\right) &= \mathbb{E}_{f_0, \sigma^2 I_n} \left[ \log \frac{d\mathbb{P}_{f_0, \sigma^2 I_n}}{d\mathbb{P}_{f_1, \sigma^2 I_n}} (Y) \right] \\ &= \mathbb{E}_{f_0, \sigma^2 I_n} \left[ \log \frac{p_{f_0, \sigma^2 I_n} (Y)}{p_{f_1, \sigma^2 I_n} (Y)} \right] \\ &= \mathbb{E}_{f_0, \sigma^2 I_n} \left[ \log \left(\frac{\exp \left(-\left\|Y-f_0\right\|^2 /\left(2 \sigma^2\right)\right)}{\exp \left(-\left\|Y-f_1\right\|^2 /\left(2 \sigma^2\right)\right)}\right)\right] \\ &= \frac{1}{2 \sigma^2} \mathbb{E}_{f_0, \sigma^2 I_n}\left[\left\|Y-f_1\right\|^2-\left\|Y-f_0\right\|^2\right] \\ &= \frac{ \left\| \mathbb{E}_{f_0, \sigma^2 I_n} \left[ Y - f_1 \right] \right\|^2 - \left\| \mathbb{E}_{f_0, \sigma^2 I_n} \left[ Y - f_0 \right] \right\|^2 }{2 \sigma^2} \\ &=\frac{\left\|f_0-f_1\right\|^2}{2 \sigma^2} \end{aligned}

where we used the simple formula: $\mathbb{E} [ \| X \|^2 ] = \operatorname{Tr} \operatorname{Cov} X + \| \mathbb{E} X \|^2$.

2. Following the same lines, for any $f_0, f_1 \in \mathbb{R}^n$ and $\Sigma_0, \Sigma_1$ symmetric positive definite matrices:

(3)

\begin{aligned} K L\left(\mathbb{P}_{f_0, \Sigma_0}, \mathbb{P}_{f_1, \Sigma_1}\right) &= \mathbb{E}_{f_0, \Sigma_0} \left[ \log \frac{p_{f_0, \Sigma_0} (Y)}{p_{f_1, \Sigma_1} (Y)} \right] \\ &= \mathbb{E}_{f_0, \Sigma_0} \left[ \log \left( \frac{ |\Sigma_0|^{-1/2} \exp \left(- \left\| Y - f_0 \right\|_{\Sigma_0}^2 / 2 \right)}{ |\Sigma_1|^{-1/2} \exp \left(-\left\| Y - f_1 \right\|_{\Sigma_1}^2 / 2 \right)} \right) \right] \\ &= \frac{1}{2} \left[ \log \left| \Sigma_1 \Sigma_0^{-1} \right| + \mathbb{E}_{f_0, \Sigma_0} \left[ \left\| Y - f_1 \right \|_{\Sigma_1}^2 - \left\| Y - f_0 \right\|_{\Sigma_0}^2 \right] \right] \\ &= \frac{1}{2} \left[ \log \left| \Sigma_1 \Sigma_0^{-1} \right| + \operatorname{Tr} ( \Sigma_1^{-1} \Sigma_0) + \| f_0 - f_1 \|_{\Sigma_1}^2 - \operatorname{Tr} I_n \right] \\ &= \frac{1}{2} \left[ \log \left| \Sigma_1 \Sigma_0^{-1} \right| + \left(f_0-f_1\right)^T \Sigma_1^{-1}\left(f_0-f_1\right)+\operatorname{Tr}\left(\Sigma_1^{-1} \Sigma_0-I_n\right) \right] \end{aligned}

where we used the following generalisation of the previous formula:

(4)

\begin{align} \mathbb{E} [ \| X \|_{\Sigma}^2 ] = \operatorname{Tr} ( \Sigma^{-1} \operatorname{Cov} X ) + \| \mathbb{E} X \|_{\Sigma}^2 \end{align}

simply obtained by applying the previous formula to $\Sigma^{-1/2} X$:

(5)

\begin{aligned} \mathbb{E} [ \| \Sigma^{-1/2} X \|^2 ] &= \operatorname{Tr} \operatorname{Cov} (\Sigma^{-1/2} X) + \| \mathbb{E} [ \Sigma^{-1/2} X ] \|^2 \\ &= \operatorname{Tr} ( \Sigma^{-1/2} \operatorname{Cov} X \, \Sigma^{-1/2}) + \| \Sigma^{-1/2} \mathbb{E} X \|^2 \end{aligned}

and using: $\| x \|_\Sigma = \| \Sigma^{-1/2} x \|$ for any $x \in \mathbb{R}^n$.