Training a large Mixture-of-Experts model at long context or with a big batch size fails as soon as any one part of the process needs more memory than the device can hold. A new paper argues that the real target is not the average memory footprint but the single highest spike across every component, all at once.
The paper, posted to the arXiv preprint server on Sept. 13, 2026, was written by Shrey Pandit, Xuan-Phi Nguyen, Yiran Zhao and Shafiq Joty. The authors identify four parts of training that the parallelism plans in common use leave unbounded, and each one grows in its own way. Expert dispatch depends on the routing matrix. The vocabulary projection scales with tokens times vocabulary size. Gradient checkpoint boundaries scale with depth times sequence length. Optimizer state scales with the parameter count.
Fixing the worst one does not solve the problem. Which component runs out of memory first depends on the model, the context length and the number of devices. Lower the largest peak, and another one becomes the bottleneck.
To address all four, the researchers built schedules whose GPU working set is fixed when a job launches. The first, called PipelinedLLEP, extends least-loaded expert parallelism by capping how many tokens each source contributes to a dispatch chunk. The second, Ring-DTP, handles the vocabulary projection by circulating activations or weight shards around a ring of devices and folding each block of logits into an online log-sum-exp calculation. The third, selective checkpoint offload, or SCO, keeps the one long-lived tensor at each checkpoint boundary in CPU memory. The fourth, OffloadStreamAdamW, converts the serial CPU Adam update used in optimizer offload into a pipeline of buckets.
The authors say these methods change only the order and granularity of computation and data movement, not the math itself, so the loss and gradients remain exact. That point matters for anyone weighing speed gains against changes in model behavior.
In tests of individual components, the methods cut the MoE dispatch peak by as much as 59.3 percent without reducing throughput. They reduced the vocabulary projection peak by 86.6 percent and made the offloaded optimizer step 2.05 times faster.
When the methods were combined and run on MoE models ranging from 120 billion to 667 billion parameters, the authors report that they trained at a context length of 1 million tokens. That is 8 to 32 times the reach of a tuned FSDP2 baseline. The combined system also reached up to 10.4 times the baseline's throughput.
The results are reported in the paper's abstract and the authors' own benchmarks. The full paper is listed under the arXiv category for Distributed, Parallel, and Cluster Computing.
Este artigo foi escrito com a assistência de IA.
News Factory APP - notícias agênticas para impulsionar seu SEO e AEO.