arXiv 2026
GAAC: Lossless KV Compression for Bandwidth-Efficient LLM Inference on Personal Devices
University of Notre Dame
Paper (arXiv 2610.38210) Code on GitHub
Abstract
Weight-only, lossy quantization substantially reduces the storage and memory requirements for serving large language models (LLMs), but the growing key–value (KV) cache remains a bottleneck for long-context inference on personal devices. We present Group-Adaptive Additive Codec (GAAC), a lossless KV compression and serving system that exploits exponent redundancy to reduce storage and accelerate attention without introducing KV approximation. GAAC encodes BF16 exponents using calibrated per-channel base-plus-offset codes over contiguous ranges, and adapts code widths across small groups while preserving KV values bit-for-bit. A fused attention kernel reconstructs KV tiles on the fly, avoiding full-cache decompression. Block-wise compression and fixed-capacity slots support incremental cache updates and paged serving.

Key results
- 1.41×bit-exact BF16 KV compression
- −26%resident KV memory
- 1.27×faster decode attention (64K, DGX Spark)
- 1.15×faster end-to-end W4A16 decode step
- +29%serving throughput at equal KV memory
- 1.25×faster decode attention on Apple M4
Comparison
| Model | Context | KV budget | BF16 KV batch | BF16 KV tok/s | GAAC batch | GAAC tok/s | Gain |
|---|---|---|---|---|---|---|---|
| Qwen3-8B | 16K | 10 GB | 4 | 60.3 | 5 | 68.1 | +13% |
| Qwen3-8B | 64K | 20 GB | 2 | 19.0 | 3 | 22.8 | +20% |
| Qwen3-32B | 64K | 35 GB | 2 | 8.47 | 3 | 10.51 | +24% |
| Llama-3.1-8B | 64K | 18 GB | 2 | 20.0 | 3 | 24.9 | +25% |
| Qwen3.8-27B | 64K | 14 GB | 3 | 23.3 | 4 | 29.6 | +27% |
| Qwen3.6-35B-A3B | 64K | 5 GB | 3 | 62.9 | 4 | 81.3 | +29% |
FAQ
How can I speed up long-context LLM decoding on a memory-bandwidth-bound device?
Read fewer KV bytes per decode step without changing any values. GAAC keeps the BF16 KV cache compressed in device memory and reconstructs each tile inside the attention kernel, so attention reads about 1.4× fewer bytes. On NVIDIA DGX Spark this makes 64K-context decode attention 1.19–1.27× faster than FlashAttention-2.
Is GAAC lossy?
No. Sign and mantissa bits are stored unchanged and every exponent is reconstructed exactly, so the KV cache is bitwise identical to BF16. Calibration only affects the compression ratio: exponents outside the calibrated windows fall back to raw mode.
How does GAAC differ from SplitZip?
SplitZip compresses KV for transfer between prefill and decode GPUs and decompresses it on arrival. GAAC keeps KV compressed while it sits in memory and decodes it inside attention on every step. To keep that decode cheap, it replaces the exponent codebook with a per-channel base plus a 3- or 4-bit offset, so reconstruction is a single integer addition.
Does it work with paged KV caches and serving engines like vLLM?
Yes. Each block is filled in BF16 and compressed once when full. Compressed blocks go into fixed-capacity slots, and blocks that do not fit stay in BF16, so the engine manages only two storage classes. In vLLM 0.17.1 this fits one more long-context sequence in the same KV-memory budget and raises throughput by 13–29%.
Does it work with an FP8 KV cache?
Yes. GAAC adds 1.09–1.14× lossless compression on top of FP8 E4M3 and 1.25–1.30× on top of E5M2, for 2.18–2.60× total versus BF16 with no error beyond the FP8 conversion.
Which hardware and models were tested?
NVIDIA DGX Spark (GB10) and Apple M4 (Metal). Models: Qwen3-4B, Qwen3-8B, Qwen3-32B, Llama-3.1-8B, hybrid-attention Qwen3.8-27B and MoE Qwen3.6-35B-A3B, all with AWQ W4A16 weights.
Citation
@misc{guo2026gaaclosslesskv,
title={GAAC: Lossless KV Compression for Bandwidth-Efficient LLM Inference on Personal Devices},
author={Yipin Guo and Arun M George and Siddharth Joshi},
year={2026},
eprint={2610.38210},
archivePrefix={arXiv},
primaryClass={cs.AI},
url={https://arxiv.org/abs/2610.38210},
}