Entropy

#machine-learning #interview #loss-in-ml

\begin{aligned} E (s u r p r i s e) & = \sum l o g_{2} \frac{1}{P (x)} P (x) \\ = \sum P (x) (l o g_{2} 1 - l o g_{2} P (x)) \\ = \sum P (x) (0 - l o g_{2} P (x)) \\ = \sum - P (x) l o g_{2} P (x) \\ = E n t r o p y \end{aligned}

Entropy Formula

$E n t r o p y = \sum - P (x) l o g_{2} P (x)$

How entropy can be used?

Entropy is used to quantify similarity or differentiate in the number of classes

Low entropy means high difference in the class probabilities, i.e., 0.01 class A, 0.99 class B
- so Low Entropy means High Confidence
High entropy means low difference in the class probabilities, i.e., 0.49 class A, 0.51 class B
- so High Entropy mens Less Confidence

Related Notes