Sebastian Raschka, a researcher and writer known for his accessible breakdowns of language model architectures, devotes his latest piece to several intertwined topics currently circulating in the AI community. The first concerns hints and rumors about a possible successor to GPT-4/5, sometimes referred to internally or in leaks under the codename 'Astra', though OpenAI has not confirmed details about its architecture or release timeline.
The second thread examines 'looped transformers', a research direction where the same computational block is reused multiple times within a network rather than stacking distinct layers. This approach could, in theory, increase a model's effective computational depth without proportionally increasing parameter count, an idea several academic groups have explored to address scaling limitations of standard transformer designs.
Finally, Raschka discusses 'hidden reasoning', the mechanisms by which some recent models generate internal reasoning steps that are not shown to the end user, similar to what OpenAI's o1/o3 model family reportedly does. This design choice raises both technical questions, such as how to train and evaluate this latent reasoning, and transparency concerns, since users often cannot see the model's full chain of thought.
The piece fits within Raschka's broader body of work aimed at clarifying complex technical subjects for practitioners and researchers. While it does not reveal confirmed details about a GPT-6, the article offers a useful snapshot of current research directions addressing efficiency gains and the growing opacity of large language models.