Multi-Head Attention

Multi head attention is a either of Self-Attention or Masked Self-Attention depending on its encoder or decoder nature.

The main reason of having multiple head rather one single is that by multiple head we can look at multiple relationship. For example, "dog sat on the bench", one attention can embed the relationship between the dog and sat and another attention can embed the relation between sat and bench.

This helps to increase the capacity and generalization ability of the model.

Attentioni=QKTdkdk=dmodel/hMultiHeadAttention=concat(Attentioni)W

References


Related Notes