• SGD with momentum is a variation of Stochastic Gradient Descent (SGD)
• AKA Nesterov Momentum
• It adds a momentum which push the gradient to the right direction
  • It makes the noisy gradient smoother
• The momentum term is typically set to a value between 0 and 1

Intuition

When we are using momentum, we are actually pushing a ball from the hill. As like a ball, it will accumulate acceleration and goes faster by time.

In case of gradient, it will also accumulates gradient and the update of gradient will be more and more by iteration.

Pros:

1. It gives smoother gradient than Stochastic Gradient Descent (SGD)
2. Improve convergence time

Cons:

1. It can overshoot good solutions and settle for suboptimal ones if the momentum is high
2. Requires tuning of momentum hyperparameter

References