Multiple linear regression

Check Simple linear regression for basics.

Formulation is as follows

$$y_i={\beta }_0+\sum _{i=1}^k{\beta }_i{x}_i+{\epsilon }_i$$

where $i=1,2\dots ,k$ is the number of regressors.

$$LSE:\frac{\partial }{\partial {\beta }_i}SSE=0$$

Let $X\in R^{1\times k}$ and $B\in R^{k+1\times 1}$. Then we need $k+1$ equations.

Now follow:

$$\begin{array}{rl}{y}_i& ={\beta }_0+\sum _{i=1}^k{\beta }_i{x}_i+{\epsilon }_i\\ \widehat{y_i}& ={\beta }_0+\sum _{i=1}^k{\beta }_i{x}_i\end{array}$$

Then we replace $y$ with $Y$ vector, combining all the equations

$$Y=\left[\begin{array}{c}{y}_1\\ y_2\\ ⋮\\ y_n\end{array}\right]\text{ , }X=\left[\begin{array}{cc}1& x_1\\ 1& x_2\\ ⋮& ⋮\\ 1& x_n\end{array}\right]\text{ , }B=\left[\begin{array}{c}{\beta }_0\\ {\beta }_1\end{array}\right]\text{ , }ϵ=\left[\begin{array}{c}{ϵ}_1\\ {ϵ}_2\\ ⋮\\ {ϵ}_n\end{array}\right]$$

then

$$Y=XB+ϵ$$

is our multiple linear model.

$SSE$ here is

$$SSE=(Y-XB{)}^T(Y-XB)={ϵ}^Tϵ$$

and LSE is

$$\begin{array}{rl}\frac{d}{dB}SSE& =0\\ \frac{d}{dB}(Y-XB{)}^T(Y-XB)& =0\\ -2X^T(Y-XB)& =0\\ X^{T}XB-X^{T}Y& =0\\ (X^{T}X)B& =X^{T}Y\\ {\hat{B}}_{LSE}& =(X^{T}X{)}^{-1}{X}^{T}Y\end{array}$$

Here ${\hat{B}}_{LSE}$ is Best Linear Unbiased Estimator for $B$.

$$\begin{array}{rl}E[\hat{B}]& =B\\ V(\hat{B})& =\left[\begin{array}{cc}V({\hat{\beta }}_0)& Cov({\hat{\beta }}_0,{\hat{\beta }}_1)\\ Cov({\hat{\beta }}_1,{\hat{\beta }}_0))& V({\hat{\beta }}_1)\end{array}\right]\\ & ={\sigma }^2(X^{T}X{)}^{-1}\\ SE(\hat{B})& =\sqrt{{\sigma }^2\text{diag}(X^{T}X{)}^{-1}}\end{array}$$

And sums of square are

$$\begin{array}{rl}SST& =Y^{T}Y-\frac{1}{n}{Y}^{T}JY\text{ , }J=({\left[\begin{array}{cccc}1& 1& \dots & 1\end{array}\right]}^T{)}_{n\times 1}\\ SSE& ={\epsilon }^T\epsilon =(Y-X\hat{B}{)}^T(Y-X\hat{B})\\ SSR& =SST-SSE\end{array}$$

R

In R use

1. solve(A) to find $A^{-1}$.
2. t(A) for $A^T$
3. lm(y~x1+x2+...xn) for multiple linear regression