Optimizers

#deep-learning #need-review ##interview

There are many optimizers:

  1. Gradient Descent
  2. Stochastic Gradient Descent (SGD)
  3. Mini Batch SGD
  4. Stochastic Gradient Descent with Momentum
  5. Nesterov Accelerated Gradient (NAG)
  6. AdaGrad
  7. RMSProp
  8. AdaDelta
  9. Adam
  10. AdaMax
  11. NADAM
  12. AMSGrad

References

  1. https://www.ruder.io/optimizing-gradient-descent
  2. https://www.scaler.com/topics/deep-learning/optimizers-in-deep-learning/
  3. https://medium.com/mlearning-ai/optimizers-in-deep-learning-7bf81fed78a0
  4. https://towardsdatascience.com/optimizers-for-training-neural-network-59450d71caf6

Related Notes