We want to compute and analyze the first variation of the cost functional at the optimal control function . To do this, in view of the definition (1.33), we must isolate the first-order terms with respect to in the cost difference between the perturbed control (3.24) and the optimal control :
The familiar first-order necessary condition for optimality (from Section 1.3.2) says that we must have for all . This condition is true for every function , but becomes particularly revealing if we make a special choice of . Namely, let be the solution of the differential equation
In light of the definition (3.29) of the Hamiltonian, we can rewrite our control system (3.18) more compactly as . Thus the joint evolution of and is governed by the system
where we recall that is the Jacobian matrix of with respect to . This is a linear time-varying system of the form where is the same as in the differential equation (3.28) derived earlier for the first-order state perturbation . Two linear systems and are said to be adjoint to each other, and for this reason is called the adjoint vector; the system-theoretic significance of this concept will become clearer in Section 4.2.8. Note also that we can think of as acting on the state or, more precisely, on the state velocity vector, since it always appears inside inner products such as ; for this reason, is also called the costate (this notion will be further explored in Section 7.1).
The purpose of the next two exercises is to recover earlier conditions from calculus of variations, namely, the Euler-Lagrange equation and the Lagrange multiplier condition (for multiple degrees of freedom and several constraints) from the preliminary necessary conditions for optimality derived so far, expressed by the existence of an adjoint vector satisfying (3.39) and (3.40).
We caution the reader that the transition between the Hamiltonian and the Lagrangian requires some care, especially in the context of Exercise 3.6. The Lagrangian explicitly depends on (and the partial derivatives appear in the Euler-Lagrange equations), whereas the Hamiltonian should not (it should be a function of , , , and ). The differential equations can be used to put into the right form. A consequence of this, in Exercise 3.6, is that will appear inside in several different places. The Lagrange multipliers are related to the components of the adjoint vector , but they are not the same.