A new technical report on arXiv addresses a question that matters increasingly for embodied AI: how do frontier multimodal large language models (MLLMs), which can already control robots without task-specific training, actually compare to purpose-built robotic policies such as vision-language-action (VLA) models or world action models (WAM)? To investigate this, the authors introduce EBench, an evaluation protocol that places an agent called GPT-6-Astra under the same visual and proprioceptive observation conditions used to assess post-trained policies.
The study examines two distinct configurations. In direct control, the agent itself generates waypoint commands for the robot to follow. In the harness with policy setup, the agent retains the option to call on an external VLA or WAM policy to execute specific actions. This hybrid design makes it possible to assess how much a general-purpose agent benefits from delegating to specialized components rather than handling every step independently.
The findings show that under direct control, the MLLM-based agent reaches performance broadly comparable to post-trained policies overall. It performs best on short-horizon tasks and shows the most consistent results across different generalization conditions. However, it falls behind specialized policies on long-horizon tasks, tabletop manipulation, and high-precision, contact-rich actions, areas where dedicated training still provides a clear edge.
When the agent is allowed to invoke an external policy, overall task success rises to match the strongest post-trained policies tested, while significantly cutting the computational token cost involved. The authors caution, though, that results still fall short of each component's individual ceiling, due to suboptimal timing of policy invocation, the lack of intermediate subgoals, and disruptions to object grasping when the agent resumes direct control. Taken together, the report offers a detailed picture of where frontier MLLM agents currently stand in robotic manipulation, and points to concrete directions for refining agent-policy harness designs going forward.