This page discusses the algorithm behind stepwise regression methods, which is one of the most widely used variable screening methods, and their implementation in software

Bidirectional elimination

1. Identifies the response $y$, and set of potentially important predictors $x_1,x_2,\dots, x_k$ to use.
2. Test the parameter of all possible one-predictor models $y=\beta_0 + \beta_1x_i$.
3. Pick a predictor $x_1$ which yields the largest t-statistic out of all the tests on step $(2)$.
4. Test the remaining predictors to fit into a two-predictor model, and picks $x_2$ which yields largest t-statistics.
5. Test the first predictor to see if $\hat\beta_1$ is still significant.
6. If not $\hat \beta_1$ is no longer significant, remove $x_1$. Search a predictor with largest t-statistic in the presence of the term $\hat\beta_2 x_2$.
7. Continue searching for predictors, until no further predictors can be found that yield significant t-statistics (at $\alpha$ level).

Forward selection

Similar to bidirectional elimination. The difference is that this algorithm doesn’t test previously added predictors like on step $(4)$

Backward elimination

Instead of starting by fitting all possible one-predictor model like bidirectional elimination, this algorithm starts with all predictors included in the initial model.

The algorithm then tests for the least significant predictor, and removes them, the complete opposite of the forward elimination algorithm.

Caution