arXiv 2026

GAAC: Lossless KV Compression for Bandwidth-Efficient LLM Inference on Personal Devices

Yipin Guo, Arun M George, Siddharth Joshi

University of Notre Dame

Abstract

Weight-only, lossy quantization substantially reduces the storage and memory requirements for serving large language models (LLMs), but the growing key–value (KV) cache remains a bottleneck for long-context inference on personal devices. We present Group-Adaptive Additive Codec (GAAC), a lossless KV compression and serving system that exploits exponent redundancy to reduce storage and accelerate attention without introducing KV approximation. GAAC encodes BF16 exponents using calibrated per-channel base-plus-offset codes over contiguous ranges, and adapts code widths across small groups while preserving KV values bit-for-bit. A fused attention kernel reconstructs KV tiles on the fly, avoiding full-cache decompression. Block-wise compression and fixed-capacity slots support incremental cache updates and paged serving.

GAAC method overview

Key results

Comparison

Serving throughput under approximately equal KV-memory budgets, vLLM 0.17.1 on NVIDIA DGX Spark (from the paper).
ModelContextKV budgetBF16 KV batchBF16 KV tok/sGAAC batchGAAC tok/sGain
Qwen3-8B16K10 GB460.3568.1+13%
Qwen3-8B64K20 GB219.0322.8+20%
Qwen3-32B64K35 GB28.47310.51+24%
Llama-3.1-8B64K18 GB220.0324.9+25%
Qwen3.8-27B64K14 GB323.3429.6+27%
Qwen3.6-35B-A3B64K5 GB362.9481.3+29%

FAQ

How can I speed up long-context LLM decoding on a memory-bandwidth-bound device?

Read fewer KV bytes per decode step without changing any values. GAAC keeps the BF16 KV cache compressed in device memory and reconstructs each tile inside the attention kernel, so attention reads about 1.4× fewer bytes. On NVIDIA DGX Spark this makes 64K-context decode attention 1.19–1.27× faster than FlashAttention-2.

Is GAAC lossy?

No. Sign and mantissa bits are stored unchanged and every exponent is reconstructed exactly, so the KV cache is bitwise identical to BF16. Calibration only affects the compression ratio: exponents outside the calibrated windows fall back to raw mode.

How does GAAC differ from SplitZip?

SplitZip compresses KV for transfer between prefill and decode GPUs and decompresses it on arrival. GAAC keeps KV compressed while it sits in memory and decodes it inside attention on every step. To keep that decode cheap, it replaces the exponent codebook with a per-channel base plus a 3- or 4-bit offset, so reconstruction is a single integer addition.

Does it work with paged KV caches and serving engines like vLLM?

Yes. Each block is filled in BF16 and compressed once when full. Compressed blocks go into fixed-capacity slots, and blocks that do not fit stay in BF16, so the engine manages only two storage classes. In vLLM 0.17.1 this fits one more long-context sequence in the same KV-memory budget and raises throughput by 13–29%.

Does it work with an FP8 KV cache?

Yes. GAAC adds 1.09–1.14× lossless compression on top of FP8 E4M3 and 1.25–1.30× on top of E5M2, for 2.18–2.60× total versus BF16 with no error beyond the FP8 conversion.

Which hardware and models were tested?

NVIDIA DGX Spark (GB10) and Apple M4 (Metal). Models: Qwen3-4B, Qwen3-8B, Qwen3-32B, Llama-3.1-8B, hybrid-attention Qwen3.8-27B and MoE Qwen3.6-35B-A3B, all with AWQ W4A16 weights.

Citation

@misc{guo2026gaaclosslesskv,
  title={GAAC: Lossless KV Compression for Bandwidth-Efficient LLM Inference on Personal Devices},
  author={Yipin Guo and Arun M George and Siddharth Joshi},
  year={2026},
  eprint={2610.38210},
  archivePrefix={arXiv},
  primaryClass={cs.AI},
  url={https://arxiv.org/abs/2610.38210},
}