ArXiv

OrderMoE: An expert similarity driven distributed edge MoE inference

2026-07-19 · 09:20 UTC ·Xin Yuan, Ning Li, Quan Chen... ·1 min read
Authors
Xin Yuan, Ning Li, Quan Chen...
Categories
cs.NI, cs.DC, cs.LG
arXiv
https://arxiv.org/abs/2607.17154v1
PDF
https://arxiv.org/pdf/2607.17154v1

Brief

OrderMoE is an expert-similarity driven framework for distributed MoE inference on bandwidth- and resource-constrained edge clusters. It derives expert similarity from router-induced logits, groups and allocates experts to improve local similarity coverage, and uses a quality- and trajectory-aware runtime selector to substitute local experts for remote ones. Real-edge testbed results show large reductions in latency, traffic, and remote invocations with only modest quality degradation.

Why it matters

OrderMoE (Xin Yuan et al., arXiv 2026-07-19) constructs an expert-similarity model from router-induced logits to partition MoE experts into similarity groups and perform similarity-aware grouping and deployment across edge servers.

Key details

  • The system adds a quality-aware and trajectory-aware runtime server-expert selection algorithm that can decide whether to invoke a remote target expert or use a feasible local substitute, trading small, controllable inference-quality degradation for reduced cross-server token transmission.
  • On a real distributed edge testbed the paper reports that OrderMoE significantly reduces average and tail latency, cross-server traffic, and remote expert invocation ratio while introducing only small, controllable quality loss (paper: 17 pages, 12 figures).
Source evidence

Abstract

Although mixture-of-experts, MoE, models have been increasingly adopted to scale large language models with moderate computation cost, it remains challenging to deploy MoE inference over resource-constrained and bandwidth-limited edge infrastructures. Existing distributed MoE serving methods mainly rely on exact expert placement, caching, replication, or communication scheduling, while overlooking the functional similarity among experts, which provides an opportunity to reduce cross-server token transmission. Therefore, this paper introduces a similarity-aware expert allocation and distributed deployment framework, dubbed OrderMoE, which aims to accelerate edge MoE inference while balancing inference latency, communication overhead, server workload, and inference quality. OrderMoE first constructs an expert similarity model based on router-induced logits representations and partitions experts in each MoE layer into multiple similarity groups. Then, it develops a similarity-aware expert grouping and deployment strategy to improve local similarity coverage across edge servers. Since reducing remote expert invocation and preserving exact inference quality are conflicting objectives, OrderMoE further designs a quality-aware and trajectory-aware runtime server-expert selection algorithm to decide whether a token should invoke its remote target expert or use a feasible local substitute expert. Experimental results on a real distributed edge testbed show that OrderMoE significantly reduces average latency, tail latency, cross-server traffic, and remote expert invocation ratio, while introducing only small and controllable inference quality degradation.

Comment: 17 pages, 12 figures