Softmax

#machine-learning #interview #activation-function

Softmax

$\begin{aligned} S o f t m a x & = \frac{e^{x}}{\sum_{y} e^{y}} \\ = \frac{e^{x}}{e^{x} + e^{y}} \end{aligned}$

Range: $[0, 1]$
It give a probability distribution. So,
- The summation of softmax layer is always 1
- Every value is positive
- Every value is inside [0, 1]
Softmax is used for Multi Class Cross Entropy

Derivation of Softmax

Derivative w.r.t. x

\begin{aligned} f (x) & = \frac{e^{x}}{e^{x} + e^{y}} \\ \frac{d f (x)}{d x} & = \frac{(e^{x} + e^{y}) \frac{d}{d x} e^{x} - e^{x} \frac{d}{d x} (e^{x} + e^{y})}{(e^{x} + e^{y})^{2}} \\ = \frac{(e^{x} + e^{y}) e^{x} - e^{x} e^{x}}{(e^{x} + e^{y})^{2}} \\ = \frac{e^{x}}{(e^{x} + e^{y})} (\frac{(e^{x} + e^{y})}{(e^{x} + e^{y})} - \frac{e^{x}}{(e^{x} + e^{y})}) \\ = f (x) (1 - f (x)) \end{aligned}

Derivative w.r.t y

\begin{aligned} f (x) & = \frac{e^{x}}{e^{x} + e^{y}} \\ \frac{d f (x)}{d y} & = \frac{(e^{x} + e^{y}) \frac{d}{d y} e^{x} - e^{x} \frac{d}{d y} (e^{x} + e^{y})}{(e^{x} + e^{y})^{2}} \\ = \frac{(e^{x} + e^{y}) 0 - e^{x} e^{y}}{(e^{x} + e^{y})^{2}} \\ = - \frac{e^{x}}{e^{x} + e^{y}} . \frac{e^{y}}{e^{x} + e^{y}} \\ = - f (x) . f (y) \end{aligned}

Related Notes