In the principle of optimality (5.4) the value function appears on both sides with different arguments. We can thus think of (5.4) as describing a dynamic relationship among the optimal values of the costs (5.1) for different and , which we declared earlier to be our goal. However, this relationship is rather clumsy and not very convenient to use in its present form. What we will now do is pass to its more compact infinitesimal version, which will take the form of a partial differential equation (PDE). The steps that follow rely on first-order Taylor expansions; the reader will recall that we used somewhat similar calculations when deriving the maximum principle. First, write appearing on the right-hand side of (5.4) as
where we remembered that . This allows us to express as
The two terms cancel out (because the one inside the infimum does not depend on and can be pulled outside), which leaves us with
Note that the terminal cost appears only in the boundary condition and not in the HJB equation itself. In fact, the specifics about the terminal cost and terminal time did not play a role in our derivation of the HJB equation. For different target sets, the boundary condition changes (as we already discussed) but the HJB equation remains the same. However, the HJB equation will not hold for just like it does not hold at in the fixed-time case, because the principle of optimality is not valid there.
We can apply one more transformation in order to rewrite the HJB equation in a simpler--and also more insightful--way. It is easy to check that (5.10) is equivalent to
We see that the expression inside the supremum in (5.11) is nothing but the Hamiltonian, with playing the role of the costate. This brings us to the Hamiltonian form of the HJB equation:
So far, the existence of an optimal control has not been assumed. When an optimal (in the global sense) control does exist, the infimum in the previous calculations can be replaced by a minimum and this minimum is achieved when is plugged in. In particular, the principle of optimality (5.4) yields