ArXiv▲ 43

Training Communication-Efficient Mixture-of-Experts Language Models with Layer Re-Configuration

ArXiv|Friday, August 28, 2026|3 days ago|Simeng Sun

No AI summary available for this article.

Why It Matters

When training Mixture-of-Experts (MoE) language models with expert parallelism, all-to-all token dispatch and combine collectives can consume a substantial fraction of end-to-end training time.

Provenance

Discovered via ArXiv and published by ArXiv.

Open original ↗Source profile →

Key Claims

In this work, we study communication-efficient MoE models (CE-MoE), in which we adopt a heterogeneous layer pattern that decouples token-mixing and channel-mixing depth.
Compared to conventional models which interleave MoE layers after each token-mixing layer (e.g., attention, Mamba-2), CE-MoE models concentrate expert capacity in a select few routed MoE layers, while maintaining depth by adding additio...

Original description

When training Mixture-of-Experts (MoE) language models with expert parallelism, all-to-all token dispatch and combine collectives can consume a substantial fraction of end-to-end training time. In this work, we study communication-efficient MoE models (CE-MoE), in which we adopt a heterogeneous layer pattern that decouples token-mixing and channel-mixing depth. Compared to conventional models which interleave MoE layers after each token-mixing layer (e.g., attention, Mamba-2), CE-MoE models concentrate expert capacity in a select few routed MoE layers, while maintaining depth by adding additio...

arxiv cs.AI publisher:arxiv tier:research

Read on ArXiv ↗← Back to feed

Discovered via ArXiv

Research papers and preprints from arXiv.

Publisher: arxiv.org

ID: http://arxiv.org/abs/2608.28511v1 · Indexed about 2 hours ago