Scientific discovery often starts with an unexpected observation that triggers a new line of inquiry. Researchers set out to test whether large language models, left to their own devices in a world where certain physical laws differ from our own, could notice an abnormal behavior, such as heavier objects falling more slowly than lighter ones, and then work out the underlying rule through experimentation, without being told what to look for.
To evaluate this, the authors introduce LongChang, an automated framework that generates simulated worlds in which one or more physical laws have been altered. The LLMs being tested act as explorers: they manipulate objects, design tools, and run experiments driven entirely by their own questions, with no assigned research task and no hints about what might be different. Separate LLM agents operate alongside them to generate and review scenes, check tool designs, and assess whether the conclusions reached are correct and backed by experimental evidence, judged against the actual altered laws and the exploration record. This setup lets scenes evolve based on what has already been explored, allowing the entire cycle to run without human oversight.
Among the twelve models evaluated, only one, referred to as GPT-6-Astra, managed to correctly identify the altered laws, reaching a discovery rate of 16.7%. The remaining eleven models scored zero. Exploration logs show that anomalies frequently go uninvestigated, and even when unusual behavior is noticed, the models typically arrive at only a partial understanding of the actual rule change.
The findings point to a clear gap between what current LLMs can do when answering well-defined questions and their capacity for open-ended scientific inquiry, noticing that something is off, forming a hypothesis, and designing experiments to test it without external guidance. Benchmarks of this kind remain uncommon, and this one suggests that autonomous scientific discovery is still a largely unresolved challenge for today's AI systems.