XStack benchmark evaluates ML repository migration between PyTorch and JAX
Original title: XStack: A Benchmark for ML Repo Migration
Best agent passes 75% of migrations while others perform worse, showing migration remains unreliable.
Machine-learning code is locked to specific frameworks and hardware. XStack tests agents on migrating entire PyTorch repositories to JAX on TPU machines. Agents must deliver a valid JAX repository that reproduces the source's published results. Grading criteria are calibrated against the original papers. Claude Opus 5 passes 75% of cases while GPT-5.6-sol reaches 45% and all other agents at most 40%.
- Claude Opus 5 achieves 75% success rate
- GPT-5.6-sol passes 45%
- Benchmark covers whole repositories
- Grading based on published results
- Migration by agents remains unreliable
From the original article
1 min read at arxiv.orgML code should run where researchers choose, not where it was first written. Today, machine learning code mainly lives in two ecosystems, PyTorch on NVIDIA GPUs and JAX on Google TPUs. Migrating repositories between them means building and testing across frameworks and hardware.
By Linrong Cai · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.