5.1.3.1 Infinite-horizon problem

The reader may find it somewhat discouraging that even for simple examples like the ones we just presented, solving the HJB equation analytically is a challenging task. Nevertheless, the HJB equation gives a considerable insight into the problem, and one can often apply numerical techniques to obtain approximate solutions. We now discuss one important situation in which the HJB equation takes a simpler form. Suppose that both the control system and the cost functional are time-invariant, i.e., $f=f(x,u)$ and $L=L(x,u)$ , and that there is no terminal cost ($K\equiv0$ ). Keeping the final state free, we let the final time $t_1$ approach $\infty$ ; in the limit, the cost functional becomes $J(u)=\int_{t_0}^\infty L(x(t),u(t))dt$ and we have what is called an infinite-horizon problem.^5.2 It is clear that in this scenario, the cost does not depend on the initial time, hence the value function depends on $x$ only: $V=V(x)$ . Thus the partial derivative ${V}_{t}$ vanishes and the HJB equation (5.10) reduces to

$$0=\inf_{u\in U} \left\{L(x,u)+\left\langle {V}_{x}(x),f(x,u)\right\rangle \right\}.$$ (5.15)

We let the reader think of other problem formulations for which the HJB equation simplifies in the same way.

The PDE (5.19) may still be difficult to solve, but it is certainly more tractable than the general HJB equation (5.10). In the special case when $x$ is scalar, (5.19) is actually an ODE. Let us consider the infinite-horizon version of Example 5.1. The HJB equation becomes ${x^4-3\big(\textstyle\frac14{V}_{x}(x)\big)^{4/3}}=0$ from which we derive ${V}_{x}(x)=\pm\big(\frac13\big)^{3/4}4x^3$ . We must choose the plus sign because $V$ should be positive definite (since $L$ is positive definite). We do not even need to solve for $V$ because the optimal feedback law is obtained from ${V}_{x}$ directly: $u^*(t)=-\big(\frac14{V}_{x}(x^*(t))\big)^{1/3}=-\big(\frac13\big)^{1/4}x^*(t)$ .