Gini Impurity

#machine-learning #interview

Gini impurity is used to calculate impurity of a leaf node in Decision Tree.

Equation of Gini Impurity

$Gini impurity of Leaf = 1 - \sum_{c}^{C} {(Probability of Class c)}^{2}$ $Total Gini Impurity of Tree = \sum_{i} \frac{{# of elements in branch}_{i}}{Total # of elements in all branches} {Gini Impurity of Branch}_{i}$

Advantages of GINI Impurity

SImple to calculate
Interpretable
Robust to overfitting

Disadvantages of GINI Impurity

Not as effective as Entropy and Information Gain when classes are imbalanced
Less sensitive to noise

Example

For the left tree

\begin{aligned} G I (H u m i d i t y == h i g h) & = 1 - \frac{3}{7} - \frac{4}{7} \\ G I (H u m i d i t y == n o r m a l) & = 1 - \frac{1}{7} - \frac{6}{7} \end{aligned}

G I (H u m i d i t y) = \frac{7}{14} G I (H u m i d i t y == h i g h) + \frac{7}{14} G I (H u m i d i t y == n o r m a l)

Related Notes

Multivariate Linear Regression
Neural Network Normalization
Prior Probability
RMSProp
Likelihood