知Yizhi Daily
← Back to this edition

AI

Baidu open-sources ERNIE-5.5-Flash-Base foundation model

Edition 2026-11-06 · 08:00 Beijing time

Baidu released ERNIE-5.5-Flash-Base, a sparse mixture-of-experts language model with 280B total parameters and 14B activated per token. The model completed pretraining on 33.5T tokens and provides base weights that have not undergone mid-training or post-training, making it easier for developers to do domain adaptation, instruction fine-tuning, and agent research.

Baidu released ERNIE-5.5-Flash-Base. The model is a sparse mixture-of-experts language model with 280B total parameters and 14B parameters activated per token; it was pretrained from scratch on 33.5T tokens, with a training context of up to 128K, and supports use of a 1M-token context during fine-tuning and inference.

The weights released this time are the checkpoint at the end of pretraining and have not yet undergone mid-training or post-training, so they can serve as a common starting point for domain adaptation, instruction fine-tuning, and reinforcement-learning research. Baidu said the base model both preserves the general capabilities formed during pretraining and allows developers to decide the direction of subsequent training themselves.

Efficiency is a core design goal of the model. To address repeated sampling during subsequent training and multi-turn interactions and long contexts in agent applications, Baidu comprehensively balanced activation computation, KV caching, and long-context costs in parts such as attention, residual connections, MoE layers, and output heads.

The model contains 43 Transformer layers, has a hidden-layer dimension of 4096 and 64 attention heads, and uses a hybrid of MLA and HCA sparse attention: in every seven layers, only one layer retains token-level access to the full context, while the other layers compress every 128 tokens into one cache entry and then attend in combination with a recent uncompressed window. Baidu said the pretraining corpus also emphasizes capabilities that are difficult to compensate for later, such as repository-scale code and long-context understanding, and was co-designed with the training system to improve large-scale training efficiency.

Original sources

Hacker News ↗
Back to this editionOpen interactive reader