7.2.2 Canonical equations as characteristics of the HJB equation

We will now show that Hamilton's canonical equations arise as the characteristics--or, more precisely, characteristic strips--of the HJB equation. With some abuse of notation, let us write the HJB equation as

$${V}_{t}(t,x)-H(t,x,-{V}_{x}(t,x))=0.$$ (7.15)

Note that we did not include the control $u$ as an argument in $H$ ; instead, we assume that the control that maximizes the Hamiltonian has already been plugged in. (This formulation also covers the calculus of variations and mechanics settings, where the velocity variable is eliminated via the Legendre transformation.) We know that we can bring the PDE (7.15) to the form (7.9) by introducing the extra state variable $x_{n+1}:=t$ (cf. Section 5.3.3). We can thus write down the characteristic equations (7.14) for this particular PDE. Since the time $t$ is present in (7.15), we can use it as the independent variable and write $\dot x_1$ , etc. We also define the costates $p_i:=-\xi_i=-{v}_{x_i}$ , which is consistent with our earlier convention (see Section 5.2). Then the equations (7.14) immediately yield

$$\dot x_i={H}_{p_i},\qquad i=1,\dots,n$$

(plus one additional equation $\dot x_{n+1}=1$ which is redundant) as well as

$$\dot p_i=-{H}_{x_i},\qquad i=1,\dots,n+1$$

where we used the fact that $H$ does not depend on $V$ (hence the term ${F}_{v}$ in (7.14) vanishes). Thus we have indeed recovered the familiar canonical differential equations.

Finally, there is one more equation in (7.14) which describes the evolution of $v$ . Here it tells us that the value function satisfies

$$\dot V=\sum_{i=1}^n \xi_i \dot x_i+\xi_{n+1}=H-\sum_{i=1}^n p_i{H}_{p_i}$$ (7.16)

where we applied the HJB equation (7.15) one more time to arrive at the second equality. (Actually, it is also easy to obtain this result directly by writing $\dot V={V}_{t}+\langle {V}_{x},\dot x\rangle$ .) It is the equation (7.16) that enables a solution of the HJB equation via the method of characteristics. We know that the Cauchy problem for the HJB equation is given by $V(t_1,x)=K(x)$ , where $K$ is the terminal cost and $t_1$ is the terminal time (this initial value problem involves flowing backward in time). To calculate the value $V(t,x)$ for some specific time $t$ and state $x$ , we must first find a point $\bar x$ such that $x=x(t)$ where $(x(\cdot),p(\cdot))$ is the solution of the system of canonical equations with the boundary conditions $x(t_1)=\bar x$ , $p(t_1)=-{K}_{x}(\bar x)$ . Then, $V(t,x)$ is computed by integration along the corresponding characteristic:

$$V(t,x(t))=K(\bar x)-\int_t^{t_1}\Big(H(s,x(s),p(s))-\sum_{i=1}^n p_i(s) {H}_{p_i}(s,x(s),p(s))\Big)ds.$$

Again, the fact that $H$ does not depend on $V$ is helpful here. Figure 7.3 illustrates the procedure.

Figure: Solving the HJB equation with the help of characteristics