Variance & Covariance

Variance

Definition

$$V(X)=E[(X-\mu {)}^2]$$

Properties

$$\begin{array}{rl}V(aX+bY)& =E[((aX+bY)-(a{\mu }_x+b{\mu }_y){)}^2]\\ & =E[(a(X-{\mu }_x)+b(Y-{\mu }_y){)}^2]\\ & =E[a^2(X-{\mu }_x{)}^2+b^2(Y-{\mu }_y{)}^2+2ab(X-{\mu }_x)(Y-{\mu }_y)]\\ & =a^{2}E[(X-{\mu }_x{)}^2]+\dots \\ & =a^{2}V(X)+b^{2}V(Y)+2abCov(X,Y)\\ & =a^2{\sigma }_x+b^2{\sigma }_y+2ab{\sigma }_{12}\end{array}$$

Covariance

If $x_i,x_j$ are independent then ${\Sigma }_{ij}=0$. Converse is not true in general. But if both follow independent normal distributions, then ${\Sigma }_{ij}⟹\text{ independence }$.

$$\begin{array}{rl}Cov(x_1,x_2)& =E[(x_1-{\mu }_1)(x_2-{\mu }_2)]\\ & =E[x_1{x}_2]-E[x_1]E[x_2]\end{array}$$

Remark

Always keep in mind that $Var(x_1)=Cov(x_1,x_1)$

Properties

Observe that

$$\begin{array}{rl}Cov(aX,bY)& =E[a(X-{\mu }_X)b(Y-{\mu }_Y)]\\ & =abE[(X-{\mu }_X)(Y-{\mu }_Y)]\\ & =abCov(X,Y)\end{array}$$

Standard Deviation

$$\sigma =\sqrt{\text{variance}}=\sqrt{V(x)}=\sqrt{E[X-\mu {]}^2}$$

Variance-Covariance Matrix

We denote variance-covariance in a matrix of size $p\times p$ by

$$Cov(X)=\Sigma =E[(x-\mu )(x-\mu {)}^T]$$ $$\Sigma =E\left[\begin{array}{cccc}(x_1-{\mu }_1{)}^2& (x_1-{\mu }_1)(x_2-{\mu }_2)& \dots & (x_1-{\mu }_1)(x_p-{\mu }_p)\\ ⋮& \ddots & & \\ (x_p-{\mu }_p)(x_1-{\mu }_1)& (x_p-{\mu }_p)(x_2-{\mu }_2)& \dots & (x_p-{\mu }_p{)}^2\end{array}\right]$$ $$\Sigma =\left[\begin{array}{cccc}E[(x_1-{\mu }_1{)}^2]& E[(x_1-{\mu }_1)(x_2-{\mu }_2)]& \dots & E[(x_1-{\mu }_1)(x_p-{\mu }_p)]\\ ⋮& \ddots & & \\ E[(x_p-{\mu }_p)(x_1-{\mu }_1)]& E[(x_p-{\mu }_p)(x_2-{\mu }_2)]& \dots & E[(x_p-{\mu }_p{)}^2]\end{array}\right]$$ $$\Sigma =\left[\begin{array}{cccc}{\sigma }_{11}& {\sigma }_{12}& \dots & {\sigma }_{1p}\\ ⋮& \ddots & & \\ {\sigma }_{p1}& {\sigma }_{p2}& \dots & {\sigma }_{pp}\end{array}\right]$$

Here any $s_{ii}$ denote the variance of variable $i$ and any $s_{ij}$ denote the covariance between variable $i$ and $j$.

${\Sigma }_{p\times p}$ is a symmetric and positive semi-definite.

Standard Deviation Matrix

$$V^{1/2}=\left[\begin{array}{cccc}\sqrt{{\sigma }_{11}}& 0& \dots & 0\\ 0& \sqrt{{\sigma }_{22}}& \dots & 0\\ ⋮& \ddots & \ddots & 0\\ 0& 0& 0& \sqrt{{\sigma }_{pp}}\end{array}\right]=diag(\sqrt{{\sigma }_{11}},\dots ,\sqrt{{\sigma }_{pp}})$$

Relation with variance-covariance matrix

$$\Sigma =v^{1/2}P{v}^{1/2}$$

thus,

$$P=(v^{1/2}{)}^{-1}\Sigma (v^{1/2}{)}^{-1}$$

Think about univariate case:

$$\begin{array}{rl}\rho & =\frac{{\sigma }_{12}}{\sqrt{{\sigma }_{11}}\sqrt{{\sigma }_{22}}}\\ & ={\sigma }_{12}{\sigma }_{11}^{-1/2}{\sigma }_{22}^{-1/2}\\ & ={\sigma }_{11}^{-1/2}{\sigma }_{12}{\sigma }_{22}^{-1/2}\end{array}$$

Linear combination of variables

Let $C=\left[\begin{array}{cc}a& b\end{array}\right]$. Then the linear combination $a{X}_1+b{X}_2$ can be written as

$$CX=a{X}_1+b{X}_2$$

Then the Expected Value becomes

$$E[CX]=CE[X]=C{\mu }_X$$

And the variance we use Quadratic form to construct the variance matrix:

$$V(a{X}_1,b{X}_2)=V(CX)=C\Sigma C^T$$

Observe that

$$\begin{array}{rl}C\Sigma C^T& =\left[\begin{array}{cc}a& b\end{array}\right]\left[\begin{array}{cc}{\sigma }_{11}& {\sigma }_{12}\\ {\sigma }_{21}& {\sigma }_{22}\end{array}\right]\left[\begin{array}{c}a\\ b\end{array}\right]\\ & ={\sigma }_{11}{a}^2+{\sigma }_{22}{b}^2+2{\sigma }_{12}ab\end{array}$$

Extension to linear systems

Let

$$z_i=c_{i1}{x}_1+c_{i_2}{x}_2+\cdots +c_{ip}{x}_p$$

where $i\in 1\dots q$. One can construct $C_{q\times p}$ matrix so that system becomes

$$Z_{q\times 1}=C_{q\times p}{X}_{p\times 1}=\left[\begin{array}{cccc}{c}_{11}& c_{12}& \dots & c_{1p}\\ ⋮& ⋮& ⋮& ⋮\\ c_{q1}& c_{q2}& \dots & c_{qp}\end{array}\right]\left[\begin{array}{c}{x}_1\\ x_2\\ ⋮\\ x_p\end{array}\right]$$

then,

$${\mu }_Z=E[CX]=CE[X]=C{\mu }_X$$ $${\Sigma }_Z=Cov(Z)=Cov(CX)=C^{T}Cov(X)C=C{\Sigma }_X{C}^T$$