Low-Rank Tensor Decomposition for Key-Value Cache Compression in Large Language Models
For my registered MSc thesis at Universitaet Trier, I am developing JoLT for memory-efficient LLM inference. It combines partial Tucker decomposition with a compact low-bit correction term (a rotated residual) and an optimization step (Lagrangian budget allocation) that decides how much to compress each part of the cache under a fixed memory budget. The related preprint is A JoLT for the KV Cache: https://arxiv.org/abs/2607.12550.
Near-lossless 2x to 3x KV cache compression with accuracy within 2 percentage points of the uncompressed model
KV caches can become a major memory cost during long-context LLM inference. In my thesis, I combine partial Tucker decomposition, a low-rank factorization over token and feature modes, with a rotated low-bit residual that cheaply corrects the approximation.
I use a Lagrangian dual to jointly choose the compression settings (decomposition ranks and residual bit widths) under a storage budget. Across five models and four model families, JoLT achieved near-lossless 2x to 3x compression while keeping accuracy within 2 percentage points of the uncompressed model. I evaluated it on 64K long-context retrieval and GSM8K reasoning.