As we have seen in Positional Encoding in Transformer, one of the ways of encoding order is to use fixed embedding like sine or cosine function

The main benefits of trigonometric functions are:

1. Numerically stable
2. Need no training
3. Predictable as it has a specific pattern
4. No extra parameters

References