Hamilton-Jacobi equations & Dynamic Programming

Consider the control problem on a Manifold \(\mathcal{M}\) wherein one is required to go from the current \(q\) to a final \(q_f\) in time \(\tau\), all whilst minimizing the "action", \[S_f(\tau, q; \varphi) = \int_{-\tau}^{0} \mathtt{d} t\; L(\varphi(t), \dot{\varphi}(t));\quad \varphi: \langle-\tau, 0\rangle \mapsto \langle q, q_f\rangle.\] \[V_f(\tau, q) = \min_{\begin{array}{c}\varphi : [-\tau, 0] \rightarrow \mathcal{M},\\ \varphi: \langle-\tau, 0\rangle \mapsto \langle q, q_f\rangle.\end{array}} S(\tau, q; \varphi).\]

Approximating by finite-difference, \[V_f(\tau, q) = \min_{\dot{q}} \left[L(q, \dot{q}) \delta t + V(\tau - \delta t, q - \dot{q} \delta t)\right] + o(\delta t) .\]

• Note: We're going "backwards-in-time", away from \(q_f\), in contrast with HJB tradition.

Hence, \[\partial_0 V_f(\tau, q) = \min_{\dot{q}} L(q, \dot{q}) - \partial_1 V(\tau, q) \dot{q}.\]

If \(\dot{q} \mapsto L(q, \dot{q})\) is "convex" \(\forall q \in \mathcal{M}\) (read as "has no local minima") then the term inside the minimization is simply the Legendre-transform of \(L\), \[\mathbb{D}_2[L](q, p) = \sup_{\dot{q}} p \dot{q} - L(q, \dot{q}).\] Hence, \[\partial_0 V_f(\tau, q) = - H(q, p := \partial_1 V_f(\tau, q)).\] and we get the famed Hamilton-Jacobi equation. If the functions involved are smooth, then \(\partial_2 L(q, \dot{q}^{(*)}) = \partial_1 V_f(\tau, q)\), where \(\dot{q}^{(*)}\) is the minimizer.

If we can find value functions \(V_f\) which gives a "map" for reaching \(q_f, \forall q_f\), then the system is completely integrable (and probably boring). This also means that solving Boundary-value problems for such systems is trivial.

• Digression: I haven't thought much about the subtleties of the co-ordinate invariant viewpoint. Folk in Differential Geometry, use what is known as the "Fiber derivative" over the Tangent-bundle: one simply assumes \(p^{(*)} = \mathbb{F}[L](\dot{q}) \equiv \partial_2 L(q, \dot{q})\). The latter viewpoint is probably okay for local-variational-stationarity, but completely obscures momentum by relegating it as a construct from Euler-Lagrange. This is also the approach taken by SICM. It appears however, that the "no-local-minimum" condition is topologically invariant due to Morse-theory (probably), and also equivalent to the fiber-derivative being a diffeomorphism.

Because Legendre transform is involutive, we can recover \(\dot{q}\), \[L(q, \dot{q}) = \mathbb{D}_2[H](q, p) = \sup_{p} \dot{q} p - H(q, p).\] The value attaining the latter maximum given (for smooth functions) by \(\dot{q}^{(*)} = \partial_2 H(q, p)\).

Now for \(\dot{p}\). \[\begin{array}{l}\dot{p} = \partial^2_{01} V(\tau, q) + \partial^2_{1} V(\tau, q) \dot{q}\\ \quad = \partial_1 (- \lambda \tau, q . H(q, \partial_1 V(\tau, q))) + \partial^2_{1} V(\tau, q) \partial_2 H(q, p)\\ \quad = - \partial_1 H(q, p).\end{array}\] Consequently the Hamiltonian is a first integral of motion, \(\dot{H} = 0\).