The team behind XStack starts from a straightforward observation: machine learning code today remains largely tied to the ecosystem in which it was first written, typically PyTorch running on NVIDIA GPUs or JAX running on Google TPUs. Moving a repository from one setup to the other requires rewriting and validating code across both a different framework and different hardware, a demanding process even for experienced engineers.
The authors turn this practical pain point into a benchmark. In XStack, an agent is given a PyTorch repository along with a clean TPU machine and must produce a working JAX version. Success is determined by grading criteria built from the source repository's published results, which the authors authenticated and calibrated themselves. The task spans entire repositories rather than isolated code snippets, making it a long-horizon, end-to-end evaluation.
Current results suggest the task remains far from solved. Claude Opus 5 leads with a 75% pass rate, followed by GPT-5.6-sol at 45%, while every other evaluated agent tops out at 40% or below. That gap between the leading model and the rest of the field points to the depth of reasoning and debugging skill required for reliable cross-framework migration, capabilities most current systems have yet to master.
By releasing XStack, the researchers are less interested in showcasing today's scores than in offering a measurable yardstick for future progress. The benchmark fits into a broader push to test AI agents on realistic, large-scale software engineering work rather than narrow, synthetic problems. As portability of ML code across hardware ecosystems becomes more strategically important, having an objective way to track this ability is a useful step, even as current numbers show it remains largely unresolved.