Second Order Derivative or Hessian Matrix
Second order derivative is the Derivative of the derivative.
They are also known as Hessian Matrix
The optimization algorithms that use second order derivatives are,
- Newton's Method
- BFGS
- L-BFGS
Pros
- Better estimate of the curvature of the loss function
- Faster Convergence
- Better Solution
Cons
- Computational cost is huge, though Quasi Newton Method aims to solve that issues
- Memory requirement for saving the Hessian Matrix
- Can be down to Saddle Points