5.2.1 Example: nondifferentiable value function

For the scalar system

$$\dot x=xu$$

with $x\in\mathbb{R}$ and $u\in[-1,1]$ , consider a fixed-time, free-endpoint problem with the cost $J(u)=x(t_1)$ . The optimal solution is easily found by inspection: if $x_0>0$ then apply $u\equiv -1$ which results in $\dot x=-x$ , hence the cost is $x(t_1)=e^{-(t_1-t_0)}x_0$ ; if $x_0<0$ then use $u\equiv 1$ which gives $\dot x=x$ and the cost is $x(t_1)=e^{t_1-t_0}x_0$ ; finally, if $x_0=0$ then $x\equiv 0$ for all $u$ and the cost is 0. We see that the value function is given by

$$V(t,x)=\begin{cases}e^{-(t_1-t)}x \quad&\text{ if }\ x>0\\ e^{t_1-t}x\quad&\text{ if }\ x<0\\ 0\quad&\text{ if }\ x=0 \end{cases}$$ (5.26)

Away from $x=0$ it indeed satisfies the HJB equation for this example, which is $-{V}_{t}=\inf_{u}\left\{{V}_{x}\, xu\right\}=-\left\vert{V}_{x}\, x\right\vert$ , with the boundary condition $V(t_1,x)=x$ . For a fixed $t<t_1$ , the graph of $V$ as a function of $x$ is plotted in Figure 5.4. At $x=0$ this function is Lipschitz but not $\mathcal C^1$ . It can actually be shown that the above HJB equation does not admit any $\mathcal C^1$ solution.

Figure: Value function nondifferentiable at $x=0$

It turns out that this state of affairs is not an exception; in fact, it is quite typical for problems with bounded controls and terminal cost to have nondifferentiable value functions. On the other hand, the local Lipschitz property--which the function (5.30) does possess--is a known attribute of value functions for some reasonably general classes of optimal control problems (we will say more on this below).

The above example clarifies why we cannot derive the maximum principle from the HJB equation. There really is no ``easy" proof of the maximum principle (except in settings much less general than the one we considered). More importantly, the difficulty that we just exposed has implications not only for relating the HJB equation and the maximum principle, but for the HJB theory itself. Namely, we need to reconsider the assumption that $V\in\mathcal C^1$ and instead work with some generalized concept of a solution to the HJB partial differential equation.^5.3Because of this difficulty, the theory of dynamic programming did not become rigorous until the early 1980s when, after a series of related developments, the notion of a viscosity solution was introduced by Crandall and Lions; that work completes the historical timeline of key contributions listed in Section 5.1.5. (The maximum principle, on the other hand, was on solid technical ground from the beginning.) We turn to viscosity solutions in the next section, postponing a discussion of further links between the HJB equation and the maximum principle until Section 7.2.