What is ERNIE 5.5 Flash Base?

Baidu has released the weights for ERNIE-5.5-Flash-Base, a base language model designed less as a finished product than as a starting point for other developers. The model is a sparse mixture-of-experts design with 280 billion total parameters, and only 14 billion of them activate for any given token.

The release is the checkpoint the company saved at the end of pre-training. It has not gone through any mid-training or post-training, which means developers can take the general capabilities already built in and push the model in their own direction. Baidu lists domain adaptation, instruction tuning and reinforcement-learning research as likely uses.

The model was pre-trained from scratch on 33.5 trillion tokens. Its training context reached 128,000 tokens, and fine-tuning and inference support a context window of up to 1 million tokens. Baidu says it has 43 Transformer layers, a hidden size of 4,096 and 64 attention heads.

How did Baidu design ERNIE 5.5 Flash Base to control compute costs?

Much of the design, according to Baidu's technical write-up, focuses on cost. A base model gets reused many times during later training, and agentic applications pile on multi-turn interactions and long contexts. Both drive up compute and memory needs. Baidu says it weighed activated compute, key-value cache size and long-context cost together when designing each part of the model.

The attention system is the most unusual part of that effort. Most layers use a heavily compressed form of attention that squeezes every 128 tokens into a single cached entry. Only one layer in seven keeps token-level access to the full context, and those layers use sparse attention that scores past positions and computes over only the top-scoring ones. Baidu says the hybrid approach keeps the whole sequence visible while caching only a small fraction of what a full-context layer would need.

Other changes target the routing and output stages. Instead of a single router choosing experts, the model uses several independently initialized routers that score experts jointly. Baidu says this makes wrong choices easier to correct and improves how well routing scores track each expert's usefulness, especially in middle and deeper layers. The company also replaced the usual multi-token prediction modules with a method it calls ECHO, which reuses the backbone's own final layer to predict future tokens. The same layer can serve as the draft model for speculative decoding.

Baidu also built custom infrastructure for the training run. It describes a distributed version of the Muon optimizer that it says stays efficient across thousands of GPUs, along with an attention kernel called FlashMask-4 that supports larger head dimensions than FlashAttention-4. On SM100 GPUs, Baidu says FlashMask-4 runs up to 175% faster in the forward pass and up to 74% faster in the backward pass than FlashAttention-4 MaskMod.

How does ERNIE 5.5 Flash Base compare with other open base models?

On its own benchmarks, Baidu compared the model with three recent open base models of similar size: DeepSeek-V4-Flash-Base, Hy3-preview-Base and MiMo-V2.5-Base. ERNIE 5.5 Flash scored highest in most categories. It posted 93.3 on MMLU, against 89.4 for DeepSeek-V4-Flash-Base, and 60.2 on GPQA, compared with 55.9 for DeepSeek. On LiveCodeBench, it scored 40.1, while the other models ranged from 31.9 to 33.5. DeepSeek-V4-Flash-Base edged out ERNIE on a few tests, including ChineseSimpleQA, where it scored 75.3 to ERNIE's 73.0, and InfiniteBench, where it scored 48.0 to 46.5. All four models were run with the same harness, prompts and few-shot examples, according to Baidu. Those results come from Baidu and have not been independently verified.

Developers can download the model from Hugging Face in FP8 and BF16 safetensors formats, with the BF16 version listed as ERNIE-5.5-Flash-Base-BF16. The model is governed by the ERNIE 5.5 Flash Base Model License Agreement v1.0. Baidu says the model needs a beginning-of-sequence token at the start of every input, so developers should keep add_special_tokens set to True when tokenizing. SGLang and vLLM both support serving the model through their completions APIs.

Baidu says a post-trained version of ERNIE 5.5 Flash will follow. For now, the base release is aimed at developers and researchers who want to run their own training methods on a common foundation.

Este artigo foi escrito com a assistência de IA.
News Factory APP - notícias agênticas para impulsionar seu SEO e AEO.