A post describing experiments with GPT-6 Astra makes claims that do not hold up under close reading, according to a detailed critique of the work. The post's abstract says Astra can look at a network's raw weights and determine what it was trained to do. It also says Astra can write, from scratch, small game-playing networks that outperform networks trained with PPO for 100 million steps.
What did the Astra weight-editing experiments show?
The post's most eye-catching result involves Qwen3.5-0.8B, a language model. Asked to improve the model's math performance without gradients, Astra identified a problem in the weights that was causing reasoning loops. Qwen's own model card warns about this kind of problem. Astra fixed it by natively editing 0.016% of the weights. The post says that change raised the model's score on MATH-500 from 7.9% to 48.1% under the same sampling settings, and that the improvement carried over to instruction following, tripling its IFEval score.
The critique argues that the Qwen experiment does not show what the abstract implies. It does not work just by looking at raw weights. The Qwen test, along with the other experiments involving real language models, relied on Python. The Qwen result also let the model run on up to 50 batches of practice problems.
Did Astra read raw weights directly in its experiments?
The critic, who laid out the experiments in a table, found that the "just by looking at the raw weights" setting, with raw numbers in context and no tools or running of the network, appears in only three experiments. Those tests involved networks of between 17,000 and 172,000 weights. In one of them, the model was given a single recorded episode of a failure and, in one try, changed 3.5% of the weights to fix a stuck car policy. Each of the three tests that read weights directly came back correct or successful.
Did Astra's game-writing networks beat PPO?
The other experiments follow a different pattern. The 9×9 Go network and the Qwen maths task both allowed Astra to write Python and run the model. A companion test on Gemma 3 270M, which used the same approach, failed. The game-writing experiment, which the abstract presents as a clear win, also needs qualifying. Its networks start with zero weights, so there is nothing to read. The critic noted that the networks beat PPO-500M only on Breakout, scoring 517 to 25. They lost to it on Seaquest, 53 to 89, and on Asterix, 53 to 55.
The critic does not dismiss the work. The Qwen result, in the critic's view, shows that Astra can be prompted to devise and carry out tasks that a human machine learning engineer could handle. Engineers routinely write code that analyzes or modifies network weights directly, and doing so well takes skill. The critic said AI models from a year or two ago would have failed at all of these tasks.
The objection is to how the results are packaged. Putting experiments that read weights directly in the same post as those that write Python and run the model suggests a single technique with broad reach. Such a technique could write a small network from scratch and also improve a large existing model. The critic argues that this would indicate a deep understanding of the weights. The post, however, relies on several different methods, and the results do not show that one method can do all of it.
Cet article a été rédigé avec l'assistance de l'IA.
News Factory APP - actualités agentiques pour booster votre SEO et AEO.