Quick Derivation of Hamilton-Jacobi-Bellman Equation
1. Settings
Objective:
\[J(T)=\int_{t_0}^T L(x(t),u(t))dt+\phi(x_T)\]Dynamic programming:
\[V(x,t)=\min_{u(t)}\left\{\int_t^{t+h}L(x(\tau),u(\tau))d\tau+V(x(t+h),t+h)\right\}\]System:
\[\dot{x}=f(x,u)\]2. Approximation
Zero-order approximation:
\[\int_t^{t+h}L(x(\tau),u(\tau))d\tau \approx hL(x(t),u(t))\]First-order approximation (Taylor expansion):
\[\begin{aligned}V(x(t+h),t+h)&\approx V(x(t),t)+h\frac{dV}{dt}\\&= V(x(t),t)+h\frac{\partial V}{\partial x}\frac{\partial x}{\partial t}+h\frac{\partial V}{\partial t} \\ &= V(x(t),t)+h\frac{\partial V}{\partial x}f(x,u)+h\frac{\partial V}{\partial t} \end{aligned}\]3. HJB Equation
Substitue the approximations for the exact representations in the dynamic programming.
\[\begin{aligned} V(x,t)&=\min_{u(t)}\left\{hL(x(t),u(t))+V(x,t)+h\frac{\partial V}{\partial x}f(x,u)+h\frac{\partial V}{\partial t}\right\} \\ &= V(x,t)+\min_{u(t)}\left\{hL(x(t),u(t))+h\frac{\partial V}{\partial x}f(x,u)+h\frac{\partial V}{\partial t}\right\} \end{aligned}\] \[0 =\min_{u(t)}\left\{hL(x(t),u(t))+h\frac{\partial V}{\partial x}f(x,u)+h\frac{\partial V}{\partial t}\right\}\] \[\therefore\; 0 =\frac{\partial V}{\partial t}+\min_{u}\left\{L(x,u)+\frac{\partial V}{\partial x}f(x,u)\right\}\]