In a short post on Mastodon, Fields Medal-winning mathematician Terence Tao flagged a trend he considers worth watching closely: the growing use of open, unsolved mathematical problems as raw material for training and evaluating AI systems. Though brief, his comment touches on a methodological issue with real consequences for how the field measures progress.
Tao's core point rests on a simple but consequential distinction. Unlike renewable computational resources, the world's supply of open math problems is finite. Once such a problem is solved by an AI system, or once a solution becomes public and is potentially absorbed into the training data of subsequent models, it can no longer serve as a genuine test of a system's ability to reason through something truly novel. In effect, the problem has been mined and depleted as a benchmark.
The observation feeds into a broader, ongoing debate in machine learning research about benchmark contamination and saturation. Models trained on vast web-scraped corpora may end up with direct or indirect exposure to solutions of supposedly hard problems, which undermines claims about their reasoning abilities. Recent high-profile demonstrations of AI systems tackling International Mathematical Olympiad questions have brought renewed attention to this concern.
Tao's remark raises a practical question for both mathematicians and AI researchers: how to keep a supply of sufficiently difficult, uncontaminated problems available to honestly assess AI progress, as systems increasingly work through the existing stock of open questions. The comment is short, but it points to a need for the mathematical community to think collectively about how such problems are curated, disclosed, or safeguarded, particularly as technology companies place growing emphasis on showcasing model performance against hard mathematical challenges.