3.4.2 First variation

We want to compute and analyze the first variation $\left.\delta J\right\vert _{u^*}$ of the cost functional $J$ at the optimal control function $u^*$ . To do this, in view of the definition (1.33), we must isolate the first-order terms with respect to $\alpha$ in the cost difference between the perturbed control (3.24) and the optimal control $u^*$ :

$$J(u)-J(u^*)=J(u^*+\alpha\xi)-J(u^*)=\left.\delta J\right\vert _{u^*}(\xi)\alpha+o(\alpha).$$ (3.30)

The formula (3.30) suggests regarding the difference $J(u)-J(u^*)$ as being composed of three distinct terms, which we now inspect in more detail. We will let $\approx$ denote equality up to terms of order $o(\alpha)$ . For the terminal cost, we have

$$K(x(t_1))-K(x^*(t_1))=K(x^*(t_1)+\alpha\eta(t_1)+o(\alpha))-K(x^*(t_1)) \approx\left\langle {{K}_{x}}(x^*(t_1)),\alpha\eta(t_1)\right\rangle.$$ (3.31)

For the Hamiltonian, omitting the $t$ -arguments inside $x$ and $u$ for brevity, we have

$$\begin{split}H(t,x,u,p)-H(t,x^*,u^*,p)&=H(t,x^*+\alpha\eta+o(... ...angle{{H}_{u}}(t,x^*,u^*,p),\alpha\xi\right\rangle. \end{split}$$ (3.32)

As for the inner product $\langle p,\dot x-\dot x^*\rangle$ , we use integration by parts as we did several times in calculus of variations:

$$\begin{split}\int_{t_0}^{t_1}\langle p(t),\dot x(t)-\dot x^*(... ..._0}^{t_1}\langle \dot p(t), \alpha\eta(t)\rangle dt \end{split}$$ (3.33)

where we used the fact that $x(t_0)=x^*(t_0)$ . Combining the formulas (3.30)-(3.34), we readily see that the first variation is given by

$$\left.\delta J\right\vert _{u^*}(\xi)= -\int_{t_0}^{t_1} \big(\le... ...ight\rangle \big)dt+\left\langle{K}_{x}(x^*(t_1))+p(t_1),\eta(t_1)\right\rangle$$ (3.34)

where $\eta$ is related to $\xi$ via the differential equation (3.27) with the zero initial condition.

The familiar first-order necessary condition for optimality (from Section 1.3.2) says that we must have $\left.\delta J\right\vert _{u^*}(\xi)=0$ for all $\xi$ . This condition is true for every function $p$ , but becomes particularly revealing if we make a special choice of $p$ . Namely, let $p$ be the solution of the differential equation

$$\dot p=-{{H}_{x}}(t,x^*,u^*,p)$$ (3.35)

satisfying the boundary condition

$$p(t_1)=-{K}_{x}(x^*(t_1)).$$ (3.36)

Note that this boundary condition specifies the value of $p$ at the end of the interval $[t_0,t_1]$ , i.e., it is a final (or terminal) condition rather than an initial condition. In case of no terminal cost we treat $K$ as being equal to 0, which corresponds to $p(t_1)=0$ . We label the function $p$ defined by (3.36) and (3.37) as $p^*$ from now on, to reflect the fact that it is associated with the optimal trajectory. We also extend the notation $\left.\right\vert _{*}$ to mean evaluation along the optimal trajectory with $p=p^*$ , so that, for example, $\left. H\right\vert _{*}(t)=H(t,x^*(t),u^*(t),p^*(t))$ . Setting $p=p^*$ and using the equations (3.36) and (3.37) to simplify the right-hand side of (3.35), we are left with

$$\left.\delta J\right\vert _{u^*}(\xi)=-\int_{t_0}^{t_1}\left\langle \left.{H}_{u}\right\vert _{*},\xi\right\rangle dt=0$$ (3.37)

for all $\xi$ . We know from Lemma 2.1^3.3 that this implies $\left.{H}_{u}\right\vert _{*}\equiv0$ or, in more detail,

$${H}_{u}(t,x^*(t),u^*(t),p^*(t))= 0\qquad \forall\,t\in[t_0,t_1].$$ (3.38)

The meaning of this condition is that the Hamiltonian has a stationary point as a function of $u$ along the optimal trajectory. More precisely, the function $H(t,x^*(t),\cdot,p^*(t)))$ has a stationary point at $u^*(t)$ for all $t$ . This is just a reformulation of the property already discussed in Section 2.4.1 in the context of calculus of variations.

In light of the definition (3.29) of the Hamiltonian, we can rewrite our control system (3.18) more compactly as $\dot x={H}_{p}(t,x,u)$ . Thus the joint evolution of $x^*$ and $p^*$ is governed by the system

$$\begin{split}\dot x^*&=\left.{H}_{p}\right\vert _{*}\\ \dot p^*&=\left.-{H}_{x}\right\vert _{*} \end{split}$$ (3.39)

which the reader will recognize as the system of Hamilton's canonical equations (2.30) from Section 2.4.1. Let us examine the differential equation for $p^*$ in (3.40) in more detail. We can expand it with the help of (3.29) as

$$\dot p^*=\big.{-\left({f}_{x}\right)^T}\big\vert _* p^*+\left.{L}_{x}\right\vert _{*}$$

where we recall that ${f}_{x}$ is the Jacobian matrix of $f$ with respect to $x$ . This is a linear time-varying system of the form $\dot p^*=-A_*^T(t)p^*+\left.{L}_{x}\right\vert _{*}$ where $A_*(\cdot)$ is the same as in the differential equation (3.28) derived earlier for the first-order state perturbation $\eta$ . Two linear systems $\dot x=Ax$ and $\dot z=-A^Tz$ are said to be adjoint to each other, and for this reason $p$ is called the adjoint vector; the system-theoretic significance of this concept will become clearer in Section 4.2.8. Note also that we can think of $p$ as acting on the state or, more precisely, on the state velocity vector, since it always appears inside inner products such as $\langle p,\dot x\rangle$ ; for this reason, $p$ is also called the costate (this notion will be further explored in Section 7.1).

The purpose of the next two exercises is to recover earlier conditions from calculus of variations, namely, the Euler-Lagrange equation and the Lagrange multiplier condition (for multiple degrees of freedom and several constraints) from the preliminary necessary conditions for optimality derived so far, expressed by the existence of an adjoint vector $p^*$ satisfying (3.39) and (3.40).

We caution the reader that the transition between the Hamiltonian and the Lagrangian requires some care, especially in the context of Exercise 3.6. The Lagrangian explicitly depends on $\dot x$ (and the partial derivatives ${L}_{\dot x_i}$ appear in the Euler-Lagrange equations), whereas the Hamiltonian should not (it should be a function of $t$ , $x$ , $u$ , and $p$ ). The differential equations can be used to put $H$ into the right form. A consequence of this, in Exercise 3.6, is that $x$ will appear inside $H$ in several different places. The Lagrange multipliers $\lambda_i^*$ are related to the components of the adjoint vector $p^*$ , but they are not the same.