Entropy and Information Gain

#machine-learning #interview

Entropy and Information Gain is used to calculate the information gain during splitting the tree into branches.

Entropy & Information Gain Formula

$Entropy = - \sum_{i} p_{i} l o g_{2} p_{i}$ $I G = E (p a r e n t) - P (c h i l d_{1}) E (c h i l d_{1}) - P (c h i l d_{2}) E (c h i l d_{2})$

Advantages of Information Gain

Disadvantages of Information Fain

Example

For the above example,
For the left tree,

\begin{aligned} P (+) & = \frac{9}{14} \\ P (-) & = \frac{5}{14} \\ E (H u m i d i t y) & = - P (+) * l o g_{2} P (+) - P (-) * l o g_{2} P (-) \end{aligned}

\begin{aligned} P (+) & = \frac{3}{7} \\ P (-) & = \frac{4}{7} \\ E (H u m i d i t y = H i g h) & = - P (+) * l o g_{2} P (+) - P (-) * l o g_{2} P (-) \end{aligned}

\begin{aligned} P (+) & = \frac{6}{7} \\ P (-) & = \frac{1}{7} \\ E (H u m i d i t y == N o r m a l) & = - P (+) * l o g_{2} P (+) - P (-) * l o g_{2} P (-) \end{aligned}

\begin{array}{r} Information Gain = E (H u m i d t y) - \frac{7}{14} E (H u m i d i t y == h i g h) - \frac{7}{14} E (H u m i d i t y == l o w) \end{array}

Get the information gain in the same way for the right tree and then take the split with more information gain