• RMSProp = Root Mean Square Propagation
• RMSProp updates the AdaGrad algorithm by using exponentially decaying average of the squares of the gradients rather than the sum
  • This helps to reduce the monotonic decaying of learning Rate
  • Converges fasters

Pros:

1. Work well will sparse data
2. Automatically adjusts learning rates based on parameter updates
3. Can converge faster than AdaGrad

Cons:

1. It can still converge too slowly
2. Requires tuning of the decay rate hyperparameter

References