3.4.4 Some comments

We remark that we could define the Hamiltonian and the adjoint vector using a different sign convention, as follows:

$$\widehat H(t,x,u,p):=\langle p,f(t,x,u)\rangle +L(t,x,u),\qquad \hat p:=-p^*.$$

Then the function

$$\widehat H(t,x^*(t),\cdot,\hat p(t))= -\langle p^*(t),f(t,x^*(t),\cdot)\rangle +L(t,x^*(t),\cdot)=-H(t,x^*(t),\cdot,p^*(t))$$

would have a minimum at $u^*(t)$ , while $x^*$ , $\hat p$ would still satisfy the correct canonical equations with respect to $\widehat H$ :

$$\dot x^*=f(t,x^*,u^*)={\widehat H}_{p}(t,x^*,u^*)$$

and

$$\dot{\hat p}=-\dot p^*={{H}_{x}}(t,x^*,u^*,p^*)= \big.{\left({f}_... ...t _*\hat p-\left.{L}_{x}\right\vert _{*}=- {\widehat H}_{x}(t,x^*,u^*,\hat p).$$

At first glance, this reformulation in terms of Hamiltonian minimization (rather than maximization) might seem more natural, because we are solving the minimization problem for the cost functional $J$ . However, our problem is equivalent to the maximization problem for the functional $-J$ (defined by the running cost $-L$ and the terminal cost $-K$ ). So, whether we arrive at a minimum principle or a maximum principle is determined just by the sign convention, and has nothing to do with whether the cost functional is being minimized or maximized. There is no consensus in the literature on this choice of sign. The convention we follow here is consistent with our definition of the Hamiltonian in calculus of variations and its mechanical interpretation as the total energy of the system; see Sections 2.4.1 and 2.4.3. (In general, however, the cost in the optimal control problem is artificial from the physical point of view and is not related to Hamilton's action integral.)

Note that the necessary conditions for optimality from Section 3.4.3 are formulated as an existence statement for the adjoint vector $p^*$ which arises directly as a solution of the second differential equation in (3.40); this is in contrast with Section 2.4.1 where the momentum $p$ was first defined by the formula (2.28) and then a differential equation for it was obtained from the Euler-Lagrange equation. In the present setting, (3.39) and (3.40) encode all the necessary information about $p^*$ , and we will find this approach more fruitful in optimal control. Observe that in the special case of the system $\dot x=u$ which corresponds to the unconstrained calculus of variations setting, we immediately obtain from (3.29) and (3.39) that $p^*$ must be given by $p^*={L}_{u}(t,x^*,u^*)$ , and the momentum definition is recovered (up to the change of notation). This implies, in particular, that the Weierstrass necessary condition must also hold, in view of the calculation at the end of Section 3.1.2 (again modulo the change of notation).

The total derivative of the Hamiltonian with respect to time along an optimal trajectory is given by

$${\frac{d}{dt}}\left.H\right\vert _{*}= \left.{H}_{t}\right\vert _... ...ft.{H}_{u}\right\vert _{*},\dot u^*\right\rangle =\left.{H}_{t}\right\vert _{*}$$ (3.42)

because the canonical equations (3.40) and the Hamiltonian stationarity condition (3.39) guarantee that the first two inner products cancel each other and the third one equals 0. In particular, if the problem is time-invariant in the sense that both $f$ and $L$ are independent of $t$ , then ${H}_{t}=0$ and we conclude that $H(x^*(t),u^*(t),p^*(t))$ must remain constant. If we want to think of $H$ as the system's energy, the last statement says that this energy must be conserved along optimal trajectories.

We know that, at least in principle, we can obtain a second-order sufficient condition for optimality if we make appropriate assumptions to ensure that the second variation $\left.\delta^2 J\right\vert _{u^*}$ is positive definite and dominates terms of order $o(\alpha^2)$ in $J(u^*+\alpha\xi)-J(u^*)$ . While in general these assumptions take some work to write down and verify, the next exercise points to a case in which such a sufficient condition is easily established and applied (and the necessary condition becomes more tractable as well). This is the case when the system is linear and the cost is quadratic; we will study this class of problems in detail in Chapter 6.