1 6 5 Basics Of Linear Regression

1) Let's define the function: $F(B)= |XB-Y|$ with B in $\mathbb{R}^{p}$.

Let $t \in [0,1]$, we have

(1)

\begin{eqnarray} ||F(Bt+(1-t)A)|| & = & ||X(Bt+(1-t)A)-Y||,\\ & = & ||tXB+(1-t)XA-(1-t)Y-tY||,\\ & \leqslant & t||XB-Y||+(1-t)||XA-Y||,\\ & \leqslant & tF(B)+(1-t)F(A).\\ \end{eqnarray}

So the function F is convex.
Moreover, $x\mapsto x^{2}$ is convex and increasing on $\mathbb{R}^{+}$, so by composition f is convex.

(On peut aussi remarquer que la Hessienne de F, obtenue en prolongeant le développement limité à l'ordre 2, $(X^{t} X)$, est positive, ce qui donne la convexité.)

Next,

(2)

\begin{eqnarray} f(B+h)& = & ||X(B+h)-Y||^{2},\\ & = & <X(B+h)-Y,X(B+h)-Y>, \\ & = & <XB-Y+Xh,XB+Xh-Y>, \\ &= & ||XB-Y||^{2}+2<Xh,XB-Y> + O(h^{2}).\\ \end{eqnarray}

So $D_{B}f(h)=<Xh,2XB-2Y> = <h,2^{t}XXB-2^{t}XY>$, then $Grad_Bf=2^{t}XXB-2^{t}XY$.

2) Using the previous question, f is convex and can only have a minimum reach in $\hat{B}$, one that cancels the gradient, that is to say which verifies:
$Grad_\hat{B}f=2^{t}XX\hat{B}-2^{t}XY=0$.

Moreover, rank(X)=p so X is injective. Let's define Y as $^{t}XXY=0$, so $^{t}Y^{t}XXY=0$ then $XY=0$ and finally $Y=0$ by injectivity, so: rank($^{t}XX$)=p.
We are now allowed to say that: $^{t}XX$ is inversible because his rank is p .

To conclude: $\hat{B}=(^{t}XX)^{-1}. ^{t}XY$.

$\\$
$\\$

3)Let $a_{ij}=(^{t}AA)_{ij}$, so we have: $^{t}(A\varepsilon)A\varepsilon = ||A\varepsilon||^{2}=\sum\limits_{i\in 1..n}\sum\limits_{k \in 1..n}a_{ik}\varepsilon_{k}\varepsilon_{i}$.

Noticing that: $\forall k \neq i$, $\mathbb{E}[\varepsilon_{k}\varepsilon_{i}]=\mathbb{E}[\varepsilon_{i}]\mathbb{E}[\varepsilon_{k}]=0$, by independence.

Then $\mathbb{E}[||A\varepsilon||^{2}]=\mathbb{E}[^{t}(A\varepsilon)A\varepsilon]= \mathbb{E}[\sum\limits_{i \in 1..n}a_{ii}\varepsilon_{i}^{2}]=\sum\limits_{i \in 1..n}a_{ii}\sigma^{2}=Tr(^{t}AA)\sigma^{2}$.

$\\$

$\\$

4) Knowing that:

(3)

\begin{eqnarray} ||\hat{B}-B^{*}||^{2} & = & ||(^{t}XX)^{-1}. ^{t}XY-B^{*}||^{2},\\ & = & ||(^{t}XX)^{-1}. ^{t}X(XB^{*}+\varepsilon)-B^{*}||^{2},\\ & = & ||(^{t}XX)^{-1}.^{t}X\varepsilon)||^{2}.\\ \end{eqnarray}

Using the previous question with $A=(^{t}XX)^{-1}.^{t}X$ and noticing that: Tr(AB)=Tr(BA) and $(^{t}A)^{-1}=^{t}(A^{-1})$, we obtain that: $\mathbb{E}[||\hat{B}-B^{*}||^{2}]=\mathbb{E}[||(^{t}XX)^{-1}.^{t}X\varepsilon)||^{2}]=Tr((^{t}XX)^{-1}.^{t}XX.(^{t}XX)^{-1})\sigma^{2}=Tr((^{t}XX)^{-1})\sigma^{2}$.