A growing number of recent open-weight language models rely on n-gram embedding tables, which store learned representations of short token sequences, such as pairs or triples, retrieved by the model during inference. This mechanism improves performance but comes with a steep memory cost: DeepSeek-V4.1-Flash, for instance, dedicates 196 billion parameters to its n-gram memory component, called Engram.
A new study addresses this issue by asking whether that memory overhead can be reduced without losing the performance benefits the tables provide. The authors propose storing n-gram embeddings at reduced numerical precision and systematically test various quantization schemes and bit allocations across tables. Experiments on models trained from scratch in the nanochat, Llama, and GPT-2 families show that four-bit tables save about 75% of table memory while preserving more than 99% of the validation-loss improvement the tables originally delivered.
Pushing precision lower leads to a larger performance drop, but the researchers find that a brief fine-tuning phase recovers much of the lost ground. More notably, they show that the memory freed up can be reinvested into larger tables, allowing the model to surpass the performance of the original full-precision tables while still using less memory overall.
The approach also holds up on existing large-scale models: tested on DeepSeek-V4.1-Flash and Qwen3.8-Flash-Next, low-bit tables cut memory usage by 47 to 55%, with perplexity increasing by less than 1.2% and downstream task performance largely intact. The findings point to low-bit quantization as a practical lever for managing the memory demands of n-gram-based architectures, a concern that is likely to grow as these tables scale up in production systems.