2 8 3 Collections Of Nested Models

1. With theorem 2.2, the estimator defined by (2.9) fulfills the following inequality for all $m\in \{1,\dots,M\}$:

(1)

\begin{align} \mathbb{E}\left[\|\hat{f}-f^*\|^2\right] \leq C_K \left(\mathbb{E}\left[\|\hat{f_m}-f^*\|^2\right]+\sigma^2\left(\log\left(\pi_m^{-1}\right)+1\right)\right), \end{align}

for some constant $C_K>1$ depending only on $K$.

Let us fix some $m\in \{1,\dots,M\}$.
First, we remind that the random variable $\|\text{Proj}_{S_m}\epsilon\|^2$ follows a chi-square distribution of $\dim S_m = m$ degrees of freedom (see Lemma A.3), so

(2)

\begin{align} \mathbb{E}\left[\|\hat{f_m}-f^*\|^2\right]\geq\mathbb{E}\left[\|\text{Proj}_{S_m}\epsilon\|^2\right]=m\sigma^2. \end{align}

We compute

(3)

\begin{align} \log\left(\pi_m^{-1}\right) &= \alpha m + \log\left(\frac{1-e^{-\alpha M}}{e^\alpha -1}\right)\\ &= m \left( \alpha + \frac{1}{m} \log\left(\frac{1-e^{-\alpha M}}{e^\alpha -1}\right) \right)\\ & \leq m \left( \alpha + \left[\log\left(\frac{1-e^{-\alpha M}}{e^\alpha -1}\right)\right]_+ \right). \end{align}

We denote $C_{\alpha}= \alpha + \left[\log\left(\frac{1-e^{-\alpha M}}{e^\alpha -1}\right)\right]_+$, Then we have :

(4)

\begin{align} \sigma^2\left(\log\left(\pi_m^{-1}\right)+1\right)&\leq \sigma^2\left(mC_{\alpha} +1 \right) \\ & \leq \sigma^2m(C_{\alpha} +1)\\ & \leq \mathbb{E}\left[\|\hat{f_m}-f^*\|^2\right](C_{\alpha} +1). \end{align}

Finally, for all $m\in\{1,\dots,M\}$,

(5)

\begin{align} \boxed{\mathbb{E}\left[\|\hat{f}-f^*\|^2\right] \leq C_{K,\alpha}\mathbb{E}\left[\|\hat{f_m}-f^*\|^2\right]} \end{align}

with $C_{K,\alpha} = (C_{\alpha} +2)C_K$.

2. (a) We have $\log (\pi_m^{-1}) = (\sqrt{2/K}-1)^2m$, so

(6)

\begin{align} \text{pen}(m) &= K\left(\sqrt{m}+\sqrt{ (\sqrt{2/K}-1)^2m}\right)^2,\\ &= Km\left(1+|\sqrt{2/K}-1|\right)^2. \end{align}

We know that $K\in ]1,2[$, so $\sqrt{2/K}-1>0$, and

(7)

\begin{align} \text{pen}(m) = Km \frac{2}{K} = 2m, \end{align}

we recognize a linear penalty (AIC Criterion).

(b) We compute the sum:

(8)

\begin{align} \sum_{m\in\{1,\dots,M\}}\pi_m = \sum_{m=1}^M e^{-\alpha m} = e^{-\alpha}\frac{1-e^{-\alpha M}}{1-e^{-\alpha}}. \end{align}

We recognize the normalization factor for the probability distribution used in (a). With $K=1.1$ and $M\rightarrow\infty$, we have

(9)

\begin{align} \sum_{m\in\{1,\dots,M\}}\pi_m \approx 7.75. \end{align}

Solution proposed by Marin