High-Performance Expert Parallelism Kernels for Large Language Model Inference

The article provides a detailed technical breakdown of how expert parallelism kernels work in distributed LLM inference systems. Unlike tensor parallelism or pipeline parallelism where communication patterns are predetermined, expert parallelism requires dynamic routing: a router network decides which tokens should be processed by which experts based on learned logits, and these routing decisions vary with each forward pass. The article uses a concrete example of 8 GPUs across 2 nodes with 16 experts (2 per GPU) and 2 routed experts per token to illustrate the dispatch and combine operations. It references DeepSeek's production deployment achieving 2.2k tokens/second per GPU on H200 clusters using wide expert parallelism combined with data-parallel attention, and notes that DeepSeek's DeepEP library established the modern architecture for these kernels. The post promises to build up both high-throughput and low-latency kernel designs.