Simple Linear Regression

Definition

Our objective is to explore the effect of regressor $x$ on the response variable $y$. In this context we define our model as the following:

$$y=h(x,\epsilon )$$

Here, y is the dependent and observable variable.

All of our assumptions are based on the $\epsilon$, which is the error term in the model. It is also called as residual. Our assumptions are:

1. $E[\epsilon ]=0$
2. $V(\epsilon )={\sigma }^2$
3. $Cov({\epsilon }_i,{\epsilon }_j)=0$, $i\ne j$
4. $\epsilon \sim NID(0,{\sigma }^2)$

Observe that $y$ is a function of a random variable, $\epsilon$. Thus $y$ itself is a random variable.

In the case of SLR we have one scalar response variable and one regressor. We define our model as following:

$$y={\beta }_0+{\beta }_{1}x+\epsilon$$

One can interpret ${\beta }_0$ as intercept and ${\beta }_1$ as the slope of the fitted line.

At the time we construct our model, we do not know parameters ${\beta }_0$ and ${\beta }_1$ and our aim is to estimate these from the data using Least Squares Estimation.

Model Analysis

Estimation by LSE

We estimate ${\beta }_0$, ${\beta }_1$ and ${\sigma }^2$. To do so we say that data is paired as $(x_i,y_i)$. Observe that

$${\epsilon }_i=y_i-({\beta }_0+{\beta }_{1}x)$$

We define sum of squared errors as

$$SSE=\sum _{i=1}^n{\epsilon }_i^2$$

LSE for ${\beta }_0$

$$\frac{d}{d{\beta }_0}SSE=0$$ $$\frac{d}{d{\beta }_0}\sum _{i=1}^n{\epsilon }^2=0$$ $$\frac{d}{d{\beta }_0}\sum _{i=1}^n(y_i-{\beta }_0-{\beta }_1{x}_i{)}^2=0$$

Derivative of a sum is sum of the derivatives.

$$\sum _{i=1}^n\frac{d}{d{\beta }_0}(y_i-{\beta }_0-{\beta }_1{x}_i{)}^2=0$$ $$\sum _{i=1}^n-2(y_i-{\beta }_0-{\beta }_1{x}_i)=0$$ $$-2\sum _{i=1}^n{y}_i-{\beta }_0-{\beta }_1{x}_i=0$$ $$\sum _{i=1}^n{y}_i-\sum _{i=1}^n{\beta }_0-\sum _{i=1}^n{\beta }_1{x}_i=0$$ $$\sum _{i=1}^n{y}_i-\sum _{i=1}^n{\beta }_1{x}_i=n{\beta }_0$$ $${\beta }_0=\sum _{i=1}^n\frac{y_i}{n}-\sum _{i=1}^n\frac{{\beta }_1{x}_i}{n}$$ $$\widehat{{\beta }_0}=\bar{y}-\widehat{{\beta }_1}\bar{x}$$

LSE for ${\beta }_1$

$$\sum _{i=1}^n\frac{d}{d{\beta }_1}(y_i-{\beta }_0-{\beta }_1{x}_i{)}^2=0$$ $$\sum _{i=1}^n-2x_i(y_i-{\beta }_0-{\beta }_1{x}_i)=0$$ $$\sum _{i=1}^n{x}_i{y}_i-x_i{\beta }_0-{\beta }_1{x}_i^2=0$$ $$\sum _{i=1}^n{x}_i{y}_i-{\beta }_0\sum _{i=1}^n{x}_i-{\beta }_1\sum _{i=1}^n{x}_i^2=0$$ $$\sum _{i=1}^n{x}_i{y}_i-(\bar{y}-\widehat{{\beta }_1}\bar{x})\sum _{i=1}^n{x}_i-{\beta }_1\sum _{i=1}^n{x}_i^2=0$$ $$\sum _{i=1}^n{x}_i{y}_i-\bar{y}\sum _{i=1}^n{x}_i+\widehat{{\beta }_1}\bar{x}\sum _{i=1}^n{x}_i-{\beta }_1\sum _{i=1}^n{x}_i^2=0$$ $$\widehat{{\beta }_1}(\bar{x}\sum _{i=1}^n{x}_i-\sum _{i=1}^n{x}_i^2)=\bar{y}\sum _{i=1}^n{x}_i-\sum _{i=1}^n{x}_i{y}_i$$ $$\widehat{{\beta }_1}=\frac{\left(\bar{y}{\sum }_{i=1}^n{x}_i-{\sum }_{i=1}^n{x}_i{y}_i\right)}{(\bar{x}{\sum }_{i=1}^n{x}_i-{\sum }_{i=1}^n{x}_i^2)}$$

Substitute $\sum x_i=\bar{x}n$

$$\widehat{{\beta }_1}=\frac{{\sum }_{i=1}^n(x_i-\bar{x})(y_i-\bar{y})}{{\sum }_{i=1}^n(x_i-\bar{x}{)}^2}$$ $$\widehat{{\beta }_1}=\frac{Cov(x,y)}{V(x)}$$ $$\widehat{{\beta }_1}=\frac{S_{XY}}{S_{XX}}$$

where

$$S_{XY}=\sum _{i=1}^n(x_i-\bar{x})(y_i-\bar{y})$$

LSE for ${\sigma }^2$

$$\widehat{{\sigma }^2}=\frac{SSE}{n-2}$$

Here $n-2$ is the degrees of freedom.

Distribution of least squares estimates

We have estimated ${\beta }_0$, ${\beta }_1$ and ${\sigma }^2$ using random samples from the data. Thus, they are random variables too.

Since the LSE is BLUE we have:

${\beta }_0$

1. $E[{\beta }_0]={\beta }_0$
2. $V({\beta }_0)=\left(\frac{\widehat{{\sigma }^2}}{S_{XX}}\frac{{\sum }^n{x}_i^2}{n}\right)$
3. $SE({\beta }_0)=\sqrt{\left(\frac{\widehat{{\sigma }^2}}{S_{XX}}\frac{{\sum }^n{x}_i^2}{n}\right)}$ Then

$${\beta }_0\sim N\left({\beta }_0,\sqrt{\left(\frac{\widehat{{\sigma }^2}}{S_{XX}}\frac{{\sum }^n{x}_i^2}{n}\right)}\right)$$

${\beta }_1$

1. $E[{\beta }_1]={\beta }_1$
2. $V({\beta }_1)=\frac{\widehat{{\sigma }^2}}{S_{XX}}$
3. $SE({\beta }_1)=\sqrt{\frac{\widehat{{\sigma }^2}}{S_{XX}}}$ Then

$${\beta }_1\sim N\left({\beta }_1,\sqrt{\frac{\widehat{{\sigma }^2}}{S_{XX}}}\right)$$

t-values

$$T_i=\frac{\widehat{{\beta }_i}-\overline{{\beta }_i}}{SE({\beta }_i)}\sim t_{(n-2)}$$

Goodness-of-fit

Question

We question whether the data match the model or not.

One would say that the model is a good fit for the data if

$$\widehat{y_i}\approx y_i$$

Thus

$${\epsilon }_i\approx 0$$

Sums of squares

• $SST={\sum }^n(y_i-\bar{y}{)}^2$, total deviation in $y$.
• $SSE={\sum }^n(y_i-\widehat{y_i}{)}^2$, sum of residuals.
• $SSR=SST-SSE$, deviation caused by regression.

If $\widehat{y_i}\approx y_i$ then $SSE\approx 0$.

Coefficient of determination

It is defined as:

$$R^2=1-\frac{SSE}{SST}$$

$R^2$ represents the share due to $x$ in total variation in $y$. So $R^2=0.95$ means that 95% variation in $y$ is due to $x$ and 5% of the variation is due to model residuals. Such a model is considered as a good fit.

Observe that $SSE\sim 0⟹R^2\sim 1$ Thus we say

• $R^2\sim 1⟹$ model is a good-fit
• $R^2\sim 0⟹$ model may not be a poor-fit. You need to conduct goodness-of-fit test.

Goodness-of-fit Test

Hypothesis

$H_0$: ${\beta }_1=0$, means that $x$ has no effect on $y$. $H_1$: ${\beta }_1{\ne }_0$, $x$ has effect on $y$. We test this hypothesis with ANOVA.

We construct our ANOVA table:

Source	DoF	Sum of Squares	Mean of SS	$F^{\ast }$
Regression	1	$SSR$	$MSR=\frac{SSR}{1}$	$\frac{MSR}{MSE}$
Error	$n-2$	$SSE$	$MSE=\frac{SSE}{n-2}$
Total	$n-1$	$SST$
Thus one can deduct that:

$$F^{\ast }=\frac{SSR}{\frac{SSE}{n-2}}=\frac{(n-2)(SST-SSE)}{SSE}$$ $$F^{\ast }\sim F_{(\alpha ,1,n-2)}$$

Usual $\alpha$ (significance level) values are:

• 0.01
• 0.05
• 0.10

Rule of decision

Let $cv=F_{(\alpha ,1,n-2})$

• Reject $H_0$ if $F^{\ast }\ge cv$. Thus conclude that $x$ has statistically significant effect on $y$ at $\alpha$ significance level.
• Fail to reject $H_0$ if $F^{\ast }<cv$. Therefore say that there is not enough evidence to state the effect of $x$ on $y$.

Test of hypothesis