Leaky ReLU

#deep-learning #activation-function ##interview

Leaky ReLU is a variant of ReLU activation function that introduces a small, non-zero slope for negative inputs. This was introduced to solve the Dying ReLU problem where the gradient doesn't improve for the negative values.

Leaky ReLU Formula

$Leaky ReLU (x) = m a x (α x, x)$

Here $α$ is a hyperparameter which is usually a very small value ([0, 1]).

References

Related Notes

Nesterov Accelerated Gradient (NAG)
Layer Normalization
Perplexity
ML Interview
Greedy Decoding