Home / AI Research / Tideway: Phase-Aware Tiered Memory Management for MoE Inference on Consumer Devices
TIAI Research
Tideway: Phase-Aware Tiered Memory Management for MoE Inference on Consumer Devices
by DukeUniversity · Yifei Wang, Tsun-yu Yang, Yechen Xu, Hancheng Ye, Qinsi Wang, Baijun Zhou
Large language model (LLM) agents are rapidly becoming a dominant paradigm for AI applications. Recently, there is a growing demand to deploy large-scale Mixture-of-Experts (MoE) models locally on consumer PCs to balance data privacy, API costs, and inference efficiency. However, unlike datacenter servers with abundant DRAM, consumer devices are tightly constrained in both GPU VRAM and host RAM, forcing massive MoE weights to spill across VRAM, host RAM, and SSD. Under such spilling, we observe that existing systems fail to fully utilize this memory hierarchy, treating the tiers as independent caches and thrashing in inference. We present TIDEWAY, a system for efficient inference of largeā¦
Upvotes67
Comments3
Published9 Nov 2026