11 5 1 Linear Discriminant Analysis

1. With the same notations as the exercise, the distribution of $X$ is:

(1)

\begin{align} \mathbb{P}(X\in dx) &= \mathbb{P}(X\in dx\ \lvert \ Y=1)\mathbb{P}(Y=1)+\mathbb{P}(X\in dx\ \lvert \ Y=-1)\mathbb{P}(Y=-1),\\ &= g(x)dx, \end{align}

with the density $\boxed{g(x) = \pi_1g_1(x) + \pi_{-1}g_{-1}(x).}$

2. The Bayes classifier is given by $h_*(x)=\text{sign}(\mathbb{E}[Y\ \lvert \ X=x])$. We compute

(2)

\begin{align} \mathbb{E}[Y\ \lvert \ X=x] &= \mathbb{P}(Y=1\ \lvert \ X=x)-\mathbb{P}(Y=-1\ \lvert \ X=x)\\ &= \pi_1g_1(x) - \pi_{-1}g_{-1}(x) \end{align}

So $\boxed{h_*(x)=\text{sign}(\pi_1g_1(x) - \pi_{-1}g_{-1}(x)).}$

3. We also assume that the matrices $\Sigma_1$ and $\Sigma_{-1}$ are positive-definite symetric. This hypothesis was not in the exercice but if it is not true then $g_k$ is not a density of a probability distribution. Let $x\in\mathbb{R}$.

• We remark that $g_k(x)>0$ so $\pi_1g_1(x) > \pi_{-1}g_{-1}(x)$ is equivalent to

(3)

\begin{align} \log{\left(\frac{g_1(x)}{g_{-1}(x)}\right)}>\log{\left(\frac{\pi_{-1}}{\pi_1}\right)}. \end{align}

We now compute the left term of this inequality.

• $\Sigma_1= \Sigma_{-1}$ thus $\det\Sigma_1^{-1}= \det\Sigma_{-1}^{-1}$, and therefore

(4)

\begin{align} \log{\left(\frac{g_1(x)}{g_{-1}(x)}\right)} = \frac{1}{2}(x-\mu_{-1})^T\Sigma^{-1}(x-\mu_{-1})-\frac{1}{2}(x-\mu_{1})^T\Sigma^{-1}(x-\mu_{1}). \end{align}

For all $a,\ b\in\mathbb{R}^d$ let us define the scalar product $\langle a,b\rangle_{\Sigma^{-1}} = a^T\Sigma^{-1}b,$ and the associated norm
$\| a\| _{\Sigma^{-1}} = \sqrt{\langle a,a\rangle_{\Sigma^{-1}}}$. It is a scalar product because $\Sigma$ is symmetric positive-definite, and thus $\Sigma^{-1}$ too.
Let $\delta = \frac{\mu_1-\mu_{-1}}{2}$ and $x' = x-\frac{\mu_1+\mu_{-1}}{2}$.
Then

(5)

\begin{align} \log{\left(\frac{g_1(x)}{g_{-1}(x)}\right)} = \frac{1}{2}\left(\| x'+\delta\| _{\Sigma^{-1}}^2-\| x'-\delta\| _{\Sigma^{-1}}^2\right), \end{align}

and the polarization identity immediately gives

(6)

\begin{align} \log{\left(\frac{g_1(x)}{g_{-1}(x)}\right)} = 2\langle x',\delta\rangle_{\Sigma^{-1}} = \langle 2\delta,x'\rangle_{\Sigma^{-1}}. \end{align}

So

(7)

\begin{align} \log{\left(\frac{g_1(x)}{g_{-1}(x)}\right)} = (\mu_1-\mu_{-1})^T\Sigma^{-1}\left(x-\frac{\mu_1+\mu_{-1}}{2}\right). \end{align}

• Finally, the condition $\pi_1g_1(x) > \pi_{-1}g_{-1}(x)$ is equivalent to

(8)

\begin{align} \boxed{(\mu_1-\mu_{-1})^T\Sigma^{-1}\left(x-\frac{\mu_1+\mu_{-1}}{2}\right)>\log{\left(\frac{\pi_{-1}}{\pi_1}\right)}.} \end{align}

4. Let $w = \Sigma^{-1}(\mu_1-\mu_{-1})$ and $c = \frac{1}{2}w^T(\mu_1+\mu_{-1})+\log{(\frac{\pi_{-1}}{\pi_1})}$.
Then $\{h_*=1\} =\{x\in\mathbb{R}^d\ \lvert \ w^T x>c\}$ and $\{h_*=-1\} =\{x\in\mathbb{R}^d\ \lvert \ w^T x\leq c\}$ so the frontier between those two half-spaces is the affine hyperplane $\{x\in\mathbb{R}^d\ \lvert \ w^T x=c\}$.

5. We compute

(9)

\begin{align} \mathbb{P}\left(h_*(X) = 1\ \lvert \ Y=1\right) = \mathbb{P}\left((\mu_1-\mu_{-1})^T\Sigma^{-1}\left(x-\frac{\mu_1+\mu_{-1}}{2}\right)>0\ \bigg| \ Y=1\right) \end{align}

We remark that conditionally on $\{Y = 1\}$, the variable $\Sigma^{-1/2}(X-\mu_1)$ follows a standard normal distribution $\mathcal{N}(0,I_d)$.
Let $N\sim \mathcal{N}(0,I_d)$. Let $\gamma = \Sigma^{-1/2}\left(\frac{\mu_1-\mu_{-1}}{2}\right)$ and $\gamma' = \frac{\gamma}{\| \gamma\| }$.

(10)

\begin{align} \mathbb{P}(h_*(X) = 1\ \lvert \ Y=1) &= \mathbb{P}(\gamma^T N>\| \gamma\| ^2)\\ &= \mathbb{P}(\gamma'^T N>\| \gamma\| )\\ &= \Phi(\| \gamma\| ). \end{align}

We have $\| \gamma\| = d(\mu_1,\mu_{-1})$, thus $\boxed{\mathbb{P}(h_*(X) = 1\ \lvert \ Y=1) = \Phi(d(\mu_1,\mu_{-1})).}$

6. The frontier is the kernel of a quadratic form, which is a quadric surface.

Solution proposed by Marin