10.5 Comparing between models (PMA6 9.4)
The goal: Find the subset of independent variables that optimizes (either minimize or maximize) a certain criteria. In other words, the goal is to find the optimal model.
How do we measure “optimal”?
First we need to look at two quantities:
10.5.1 RSS: Residual Sum of Squares
Recall the method of least squares introduced in section 9 minimizes the residual sum of squares around the regression plane. This value is central to all following model comparison. How “far away” are the model estimates from the observed?
\[ \sum(Y - \bar{Y})^{2}(1-R^{2}) \]
10.5.2 General F Test
See also Section 10.2.
Two nested models are similar if the p-value for the General F-test is non-significant at a .15 level. Nested: The list of variables in one model is a subset of the list of variables from a bigger model. Similar to all other ANOVA models, you are essentially comparing the difference in RSS between nested models.
# Full model
full.employ.model <- lm(cesd ~ age + chronill + employ, data=depress)
# Reduced model
reduced.employ.model <- lm(cesd ~ age, data=depress)
anova(reduced.employ.model, full.employ.model)
## Analysis of Variance Table
##
## Model 1: cesd ~ age
## Model 2: cesd ~ age + chronill + employ
## Res.Df RSS Df Sum of Sq F Pr(>F)
## 1 292 22197
## 2 285 20036 7 2161.4 4.3921 0.0001197 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1anova() not aov().
Other references: https://online.stat.psu.edu/stat501/lesson/6/6.2
10.5.3 Likelihood function
What is the likelihood that we observed the data \(x\), given parameter values \(\theta\). \[ \mathcal{L}(\theta \mid x)=p_{\theta }(x)=P_{\theta }(X=x) \]
- For strictly convenient mathematical matters, we tend to work with the log-likelihood (LL).
- Great because \(log\) is a monotonic increasing function, maximizing the LL = maximizing the likelihood function.
- We can compare between models using functions based off the LL.
There are several measures we can use to compare between competing models.
10.5.4 Multiple \(R^{2}\)
If the model explains a large amount of variation in the outcome that’s good right? So we could consider using \(R^{2}\) as a selection criteria and trying to find the model that maximizes this value.
- Problem: The multiple \(R^{2}\) always increases as predictors are added to the model.
- Ex. 1: N = 100, P = 1, E(\(R^{2}\)) = 0.01
- Ex. 2: N = 21, P = 10, E(\(R^{2}\)) = 0.5
- Problem: \(R^{2} = 1-\frac{Model SS}{Total SS}\) is biased: If population \(R^{2}\) is really zero, then E(\(R^{2}\)) = P/(N-1).
Reference PMA6 Figure 9.1
10.5.5 Adjusted \(R^{2}\)
To alleviate bias use Mean squares instead of SS.
\(R^{2} = 1-\frac{Model MS}{Total MS}\)
equivalently,
\(R^{2}_{adj} = R^{2} - \frac{p(1-R^{2})}{n-p-1}\)
Now Adjusted \(R^{2}\) is approximately unbiased and won’t inflate as \(p\) increases.
10.5.6 Mallows \(C_{p}\)
\[ C_{p} = (N-P-1)\left(\frac{RMSE}{\hat{\sigma}^{2}} -1 \right) + (P+1) \]
where \(RMSE = \frac{RSS}{N-P-1}\).
- Smaller is better
- When all variables are chosen, \(P+1\) is at it’s maximum but the other part of \(C_{p}\) is zero since \(RMSE\)==\(\hat{\sigma}^{2}\)
10.5.7 Akaike Information Criterion (AIC)
- A penalty is applied to the deviance that increases as the number of parameters \(p\) increase.
- Tries to find a parsimonious model that is closer to the “truth”.
- Uses an information function, e.g., the likelihood function \((LL)\).
\[ AIC = -2LL + 2p\]
- Smaller is better
- Can also be written as a function of the residual sum of squares (RSS) (in book)
- Estimates the information in one model relative to other models
- So if all models suck, your AIC will just tell you which one sucks less.
- Built in
AIC()function in R - Rule of thumb: Model 1 and Model 2 are considered to have significantly different fit if the difference in AIC values is greater than 2.
\[\mid AIC_{1} - AIC_{2}\mid > 2\]