New Training Framework Improves Multimodal AI Models' Ability to Integrate Text and Images

Researchers at arXiv have identified a fundamental failure mode in multimodal AI systems that combine text reasoning with visual generation. In complex reasoning tasks, these models experience "Modal Isolation," where generated images diverge from textual context while subsequent text ignores visual evidence, preventing the modalities from genuinely informing each other. The researchers attribute this problem to compounding information loss at the boundaries where the model transitions between text and image processing. To address this, they developed MoTiF (Modality Transition Fidelity), a two-stage training framework that directly optimizes modality transitions by decomposing reasoning cycles into atomic operations and quantifying cross-modal hallucination and visual utilization deficits. The framework uses Reflective Supervised Fine-Tuning to help models detect and recover from erroneous visual outputs, followed by Flow-GRPO, a reinforcement learning approach that improves image generation fidelity. Testing across four visual puzzle benchmarks demonstrated that this transition-level supervision substantially improves both cross-modal coherence and final task accuracy, suggesting that effective interleaved reasoning requires explicit structural supervision at modality boundaries rather than simply scaling models or optimizing for end-task accuracy.