Conjugate-Gradient and Quasi-Newton Methods
Conjugate-Gradient and Quasi-Newton Methods: We now will discuss two gradient-optimization methods commonly used in geophysical inversion: the conjugate-gradient (CG) method and the quasi-Newton (QN) method. Unlike the Newton method, these two methods do not explicitly compute the inverse to the Hessian; instead, they iteratively move along descent directions that reduce the data residual. Each iteration costs only O(N2) operations of a matrix-vector multiply. Another strength is that, in the case of CG and low-memory QN methods, no Hessian matrix needs to be stored or inverted explicitly. Their weakness is that fast convergence is not guaranteed. However, they are generally faster than the nonpreconditioned steepest-descent method.