Mutual Information

Mutual Information (MI) shows how closely two distributions are related
$M I (i n p u t_{i}, o u t p u t)$ shows how $i n p u t_{i}$ has more contribution to the output than other
More MI means more contribution
If the data is continuous, use Binning or Bucketing to make them discrete
Range: $[0, \infty]$

Mutual Information = ?

$\begin{aligned} M I & = \sum_{x, y} P (x, y) l o g \frac{P (x, y)}{P (x) P (y)} \\ = \sum_{x} \sum_{y} P (x, y) l o g \frac{P (x, y)}{P (x) P (y)} \end{aligned}$

Related Notes

Continuous Bag of Words
K-means vs. Hierarchical
Oversampling
Multi Class Cross Entropy
Odds