Ng et al. 1999 showed that a potential function $\mathrm{\Psi }$ can shape a reward function without changing the optimal policy, $$\underset{\text{True reward}}{\underbrace{r(\mathbf{s},\mathbf{a})}}=\underset{\text{Shaped reward}}{\underbrace{\tilde{r}(\mathbf{s},\mathbf{a})}}+\underset{\text{Potential function shaping}}{\underbrace{\mathrm{\Psi }(\mathbf{s})-\gamma {\mathbb{E}}_{{\mathbf{s}}^{\prime }\sim \mathcal{P}(\cdot \mid \mathbf{s},\mathbf{a})}[\mathrm{\Psi }({\mathbf{s}}^{\prime })]}}.$$ $$\underset{\text{True critic}}{\underbrace{\mathcal{Q}(\mathbf{s},\mathbf{a})}}=\underset{\text{Shaped critic}}{\underbrace{\tilde{\mathcal{Q}}(\mathbf{s},\mathbf{a})}}+\underset{\text{Potential}}{\underbrace{\mathrm{\Psi }(\mathbf{s})}}.$$ In entropy-regularized RL, the optimal policy take the form of a pseudo-posterior, $$\underset{\text{Cloned policy}}{\underbrace{q_{\alpha }(\mathbf{a}\mid \mathbf{s})}}=\underset{\text{Unknown advantage function}}{\underbrace{\exp (\frac{1}{\alpha }(\mathcal{Q}(\mathbf{s},\mathbf{a})-{\mathcal{V}}_{\alpha }(\mathbf{s})))}}\underset{\text{Policy prior}}{\underbrace{p(\mathbf{a}\mid \mathbf{s})}}.$$ Rearranging terms, we can express the critic in terms of the log policy ratio and soft value function $$\underset{\text{Unknown critic}}{\underbrace{\mathcal{Q}(\mathbf{s},\mathbf{a})}}=\underset{\text{Log policy ratio}}{\underbrace{\alpha \log \frac{q_{\alpha }(\mathbf{a}\mid \mathbf{s})}{p(\mathbf{a}\mid \mathbf{s})}}}+\underset{\text{Soft value function}}{\underbrace{{\mathcal{V}}_{\alpha }(\mathbf{s})}}.$$ Comparing this result with that of Ng et al., we show that the log policy ratio is a shaped reward function, $$\text{Shaping theory:}\underset{\text{True reward}}{\underbrace{r(\mathbf{s},\mathbf{a})}}=\underset{\text{Shaped reward}}{\underbrace{\tilde{r}(\mathbf{s},\mathbf{a})}}+\underset{\text{Potential function shaping}}{\underbrace{\mathrm{\Psi }(\mathbf{s})-\gamma {\mathbb{E}}_{{\mathbf{s}}^{\prime }\sim \mathcal{P}(\cdot \mid \mathbf{s},\mathbf{a})}[\mathrm{\Psi }({\mathbf{s}}^{\prime })]}}.$$ $$\text{Coherent reward:}\underset{\text{True reward}}{\underbrace{r(\mathbf{s},\mathbf{a})}}=\underset{\text{Coherent reward}}{\underbrace{\alpha \log \frac{q_{\alpha }(\mathbf{a}\mid \mathbf{s})}{p(\mathbf{a}\mid \mathbf{s})}}}+\underset{\text{Soft value function shaping}}{\underbrace{{\mathcal{V}}_{\alpha }(\mathbf{s})-\gamma {\mathbb{E}}_{{\mathbf{s}}^{\prime }\sim \mathcal{P}(\cdot \mid \mathbf{s},\mathbf{a})}[{\mathcal{V}}_{\alpha }({\mathbf{s}}^{\prime })]}}.$$ To leverage coherency, behavioral cloning amounts to KL-regularized heteroscedastic regression, $$\underset{\theta }{max}{\mathbb{E}}_{\mathbf{a},\mathbf{s}\sim \mathcal{D}}[\log q_{\theta }(\mathbf{a}\mid \mathbf{s})]-\lambda {\mathbb{D}}_{\text{KL}}[q_{\theta }(\mathbf{w})\mid \mid p(\mathbf{w})],$$ $$\text{ where }{q}_{\theta }(\mathbf{a}\mid \mathbf{s})=\int p(\mathbf{a}\mid \mathbf{s},\mathbf{w})q_{\theta }(\mathbf{w})\mathrm{d}\mathbf{w},p(\mathbf{a}\mid \mathbf{s})=\int p(\mathbf{a}\mid \mathbf{s},\mathbf{w})p(\mathbf{w})\mathrm{d}\mathbf{w}.$$

Using this regularized regression objective, the cloned policy can be used to define the shaped reward using the log policy ratio, in contrast to prior work which learn the reward using classification.