AI-SOVEREIGN
Home / AI Research / Tideway: Phase-Aware Tiered Memory Management for MoE Inference on Consumer Devices
TIAI Research

Tideway: Phase-Aware Tiered Memory Management for MoE Inference on Consumer Devices

by DukeUniversity · Yifei Wang, Tsun-yu Yang, Yechen Xu, Hancheng Ye, Qinsi Wang, Baijun Zhou

Large language model (LLM) agents are rapidly becoming a dominant paradigm for AI applications. Recently, there is a growing demand to deploy large-scale Mixture-of-Experts (MoE) models locally on consumer PCs to balance data privacy, API costs, and inference efficiency. However, unlike datacenter servers with abundant DRAM, consumer devices are tightly constrained in both GPU VRAM and host RAM, forcing massive MoE weights to spill across VRAM, host RAM, and SSD. Under such spilling, we observe that existing systems fail to fully utilize this memory hierarchy, treating the tiers as independent caches and thrashing in inference. We present TIDEWAY, a system for efficient inference of large…

Upvotes67
Comments3
Published9 Nov 2026