Gumbel Softmax

#deep-learning #interview

Proposed in this paper

Gumbel Softmax is used mostly in the models where we need to sample discrete variables and also need to back-propagate through them. If we do usual decoding process, we would have used the argmax, which is not differentiable so we can't back-propagate through it.

The idea of the gumbel softmax is to add some noise which is sampled from $G u m b e l (0, 1)$ and sample from that noise. A temperature parameter is used to control how much discrete we want the output distribution.

As we can see from the fig,

For low temperature ( $τ$ ), gumbel softmax distribution gives One Hot Vector like output
On the other side, for high temperature, it gives an Uniform Distribution

Gumbel softmax is differentiable because it is continuous for all values >= 0

Gumbel Softmax distribution formula:

y_{i} = \frac{e x p ((l o g (π_{i}) + g_{i}) / τ)}{\sum_{j}^{j = n} e x p ((l o g (π_{j}) + g_{j}) / τ)}

π is the probability and g is the sampled gumbel τ is the temperature

Gumbel Softmax

Gumbel Softmax distribution formula:

References

Related Notes