Forward Feature Selection
In forward feature selection, it starts with empty feature and on every step it add the one feature that has the most affect on prediction (depending on the model's evaluation score).
Steps:
- Initialization with an empty subset
- Each feature paired with previous set and evaluated a pre-defined evaluation metric. The feature with the highest performance is selected
- Iterated through step 2, unless the pre-defined number of features have not reached
Cons:
- It doesn't consider multiple feature interaction (Feature A might not have enough boost on performance, but feature (A, B, D) can boost the performance significantly)
- Solution: Backward Feature Elimination