FOVI-Dex: Foveation enables efficient perceptive learning of dexterous manipulation in simulation

Advanced physics simulators and rendering engines promise to support the next-generation of robotic reinforcement learning (RL), yet training perceptive policies remains challenging due to the high-costs of rendering and perception. We propose foveation as a solution to this challenge, coupling a novel foveated rendering solution with foveated perception, alongside a careful approach for ensuring transfer to foveated sampling of real uniform cameras. We train a language-conditioned visuotactile policy controlling a dexterous arm-hand to find and grasp a target object in clutter (FOVI-Dex), via online RL without demonstrations. In comparison to a high-resolution fixed-camera control, FOVI-Dex uses 8.9 less vision tokens, supporting 6 more parallel environments and 4.4 higher training throughput. At matched training budgets, this efficiency enables FOVI-Dex to outperform both the high-resolution baseline and a matched uniform active camera baseline, in both sim and real, with an outsized advantage in the real world.