Global Convergence Theory Established for Wasserstein Policy Gradient in Entropy-Regularized Reinforcement Learning

Wasserstein policy gradient is a policy optimization approach for reinforcement learning that uses optimal-transport geometry to update action distributions, making it attractive for continuous-control tasks. Until now, its global convergence properties were poorly understood because standard Langevin diffusion analyses do not transfer directly to the RL setting, where the objective depends on the policy through the Bellman recursion rather than a static convex functional. The authors resolve this by showing that the soft Bellman residual admits a statewise KL representation relative to a Gibbs policy, that Bellman contraction links this residual to the global optimality gap, and that a Bellman resolvent identity connects value improvement to relative Fisher information. Together with a uniform log-Sobolev inequality for the evolving Gibbs family, these tools yield a distributional Polyak–Łojasiewicz condition, which replaces the convexity assumption typically required for global convergence proofs. The analysis also establishes the regularity and uniform bounds needed to control discretization error, yielding geometric contraction up to a discretization bias. Conceptually, the paper demonstrates that entropy-regularized RL, while not convex in the classical sense, possesses a favorable PL-type geometry induced by the Bellman structure that is sufficient to guarantee global convergence of WPG.