Reverse Flow Matching: A Unified Framework for Training Diffusion and Flow Policies in Online Reinforcement Learning

Diffusion and flow-based policies have become increasingly prominent in online reinforcement learning due to their expressive representational capacity, but training them efficiently has remained a significant open challenge. A core difficulty is that online RL lacks direct samples from the target Boltzmann distribution defined by the Q-function, which standard generative modeling assumes. Prior work had proposed two seemingly unrelated solution families: noise-expectation methods, which use weighted averages of noise as training targets, and gradient-expectation methods, which use weighted averages of Q-function gradients. The proposed RFM framework unifies these by framing the training objective as a posterior mean estimation problem given an intermediate noisy sample, and introduces Langevin Stein operators to construct zero-mean control variates, yielding a broad class of estimators that subsume both prior families. This unification enables two concrete advances: it extends Boltzmann distribution targeting from diffusion models to flow policies, and it allows principled combination of Q-value and Q-gradient information into a single, more effective estimator. Empirical results on continuous-control benchmarks show that an RFM-instantiated flow policy outperforms diffusion policy baselines in training efficiency and stability. The paper has been accepted as a Spotlight presentation at ICML 2026, with code publicly available.