Researchers have developed a method for distilling a standard Transformer model into Mamba, a State Space Model that does not use Attention blocks. The resulting model keeps most of the teacher's performance and carries over to downstream tasks, though some tasks that rely more heavily on in-context recall show gaps.
Why move from Transformers to Mamba at all?
State Space Models such as Mamba have gained traction as an alternative to Transformers. They use less memory and deliver higher throughput during generation than their Attention-based counterparts. The catch is that the machine learning community has spent years building up know-how for training Transformers, and many pretrained Transformer models are already available. Switching architectures means either giving up that investment or finding a way to carry it over.
That is the problem the team set out to solve. Earlier work on cross-architecture distillation found that a straightforward attempt to transfer knowledge from a Transformer into Mamba failed to preserve the original teacher's performance. Developers often worked around the problem with hybrid designs that combine Attention and SSM blocks.
The researchers argue that the problem can be addressed through a better starting point. By giving Mamba a principled initialization, they say, the cross-architecture distillation process can produce better results without the hybrid approach.
How does the Transformer-to-Mamba distillation recipe work?
Their method runs in two stages. First, knowledge is distilled from a traditional Transformer into a linearized version of Attention, using an adaptation of the kernel trick. Second, that linearized model is distilled into an adapted Mamba model that contains no Attention blocks.
Can a Mamba model match Transformer performance?
The team tested the approach using Pythia-1B as the teacher. The distilled Mamba model reached a perplexity of 14.01, close to the teacher's 13.86. The researchers said the performance largely carried over to downstream tasks, but some gaps appeared on tasks that depend more on in-context recall.
To evaluate the recipe, the researchers ran ablation studies at the 1-billion-parameter scale using 10 billion tokens. They varied the sequence mixer architecture, analyzed how performance scaled with model size and the total number of distillation tokens, and tested how sensitive results were to how tokens were split between the two stages.
For developers weighing whether to move to SSMs, the work offers a path to reuse existing Transformer checkpoints rather than train new models from scratch. The results still show a trade-off, particularly on recall-heavy tasks, and the paper's findings are limited to the scale and model tested.
This article was written with the assistance of AI.
News Factory APP - agentic news to boost your SEO & AEO.