Home / AI Research / Chorus: Shared N-gram Embeddings with Layer-wise Adaptation
CHAI Research
Chorus: Shared N-gram Embeddings with Layer-wise Adaptation
· Stephen Chung, Wenyu Du
Language models can gain capacity at little computational cost from n-gram embedding tables: large tables of learned vectors for short sequences of consecutive tokens, which are looked up and added to the model's input or to its intermediate layers. Recent designs commonly use a separate n-gram embedding table for each layer, which makes the tables costly in memory and prevents layers from sharing what they learn. Sharing a single table, however, gives every layer the same embedding and leaves no room for layer-wise preferences, and it performs poorly in practice. We propose Chorus, in which one set of n-gram embedding tables serves the whole network and every layer reads it in its own…
Upvotes5
Comments2
Published16 Nov 2026