3.4.3 Second variation

To better understand the behavior of $H$ as a function of $u$ along the optimal trajectory, let us bring in the second variation. To do this, we must first augment the description (3.25) of the perturbed state trajectory with the second-order term in $\alpha$ , writing $x=x^*+\alpha\eta+\alpha^2\zeta +o(\alpha^2).$ We then need to go back to the expressions (3.32)-(3.34) and expand them by adding second-order terms in $\alpha$ . With $p$ already set equal to $p^*$ as defined above, it is relatively straightforward to check that the $\zeta$ -dependent terms drop out--exactly in the same way as the $\eta$ -dependent terms dropped out of the first variation (3.35)--and that the second variation is given by

$$\left.\delta^2 J\right\vert _{u^*}(\xi)= -\frac12\int_{t_0}^{t_1}... ...in{pmatrix}\eta \\ \xi \end{pmatrix}dt+\frac12 \eta^T{K}_{{x}{x}}(x^*(t_1))\eta$$ (3.40)

where we recall that $\eta$ is obtained as the state of the system (3.27) driven by $\xi$ with the initial condition $\eta(t_0)=0$ .

We know from the second-order necessary condition for optimality (see Section 1.3.3) that we must have $\left.\delta^2 J\right\vert _{u^*}(\xi)\ge 0$ for all $\xi$ . Let us concentrate on the integrand in (3.41) and ask the following question: does one term in the Hessian matrix of $H$ dominate the other terms? If yes, then this term should be nonpositive to ensure the correct sign of $\left.\delta^2 J\right\vert _{u^*}(\xi)$ . We encountered a very similar issue in Section 2.6.1 in relation to the inequality (2.61). We found there that for the overall integral to be nonnegative, the function multiplying $(\eta'(x))^2$ must be nonnegative, because $\eta'$ may be large while $\eta$ itself is small. The present situation is a bit different since $\xi$ is not just the derivative of $\eta$ , i.e., the system relating the two is not a simple integrator. However, the corresponding conclusion is still valid: $\xi^T\left.{H}_{{ u}{u}}\right\vert _{*}\xi$ is the dominant term because we may have a large $\xi$ producing a small $\eta$ (but not vice versa), as illustrated in Figure 3.5. Thus, in order for the second variation (3.41) to be nonnegative, we must have $\xi^T\left.{H}_{{ u}{u}}\right\vert _{*}\xi\le 0$ for all $\xi$ , which can only happen if the matrix $\left.{H}_{{ u}{u}}\right\vert _{*}$ is negative semidefinite:

$${H}_{{u}{u}}(t,x^*(t),u^*(t),p^*(t))\le 0\qquad \forall\,t\in[t_0,t_1].$$ (3.41)

This condition is known as the Legendre-Clebsch condition (in its Hamiltonian formulation).

Figure: Large $\xi$ can produce small $\eta$

We already know from (3.39) that for each $t\in[t_0,t_1]$ , the function $H(t,x^*(t),\cdot,p^*(t)))$ must have a stationary point at $u^*(t)$ . Now the Legendre-Clebsch condition (3.42) tells us that if this stationary point is an extremum, then it is necessarily a maximum. Even though we have not proved that the stationary point must indeed be an extremum, it is tempting to conjecture that this Hamiltonian maximization property is true. In other words, our findings up to this point are very suggestive of the following (not yet proved) necessary conditions for optimality: If $u^*(\cdot)$ is an optimal control and $x^*(\cdot)$ is the corresponding optimal state trajectory, then there exists an adjoint vector (costate) $p^*(\cdot)$ such that:

1. $x^*$ and $p^*$ satisfy the canonical equations (3.40) with the boundary conditions $x^*(t_0)=x_0$ , $p^*(t_1)=-{K}_{x}(x^*(t_1))$ .

2. For each fixed $t$ , the function $u\mapsto H(t,x^*(t),u,p^*(t))$ has a (local) maximum at $u=u^*(t)$ :
   $$H(t,x^*(t),u^*(t),p^*(t))\ge H(t,x^*(t),u,p^*(t)) \qquad \forall\,u$$    near $$u^*(t),\ \forall\,t\in[t_0,t_1].$$