• As we know in gradient descent method, we shift our parameters against descent direction. In this post, we denote as:

where $\mathbf{x}^{(k)}$ is current point, and we have update parameters formula:

where $\alpha$ called learning rate.

• If we assume that our function $f(\mathbf{x})$ is convex, we can find optimal learning rate to convergence faster. For the convenience of finding optimal $\alpha^{(k)}$ value, we normalize the descent direction:

So on, we have next problem:

• Because we consider $f(\mathbf{x})$ is convex, so we can solve $(4)$ by set:

Reference:

1. Algorithms for Optimization | Mykel J. Kochenderfer - Tim A. Wheeler