Skip to content (access key 's')
Logo of Technion
Logo of CS Department
Events

The Taub Faculty of Computer Science Events and Talks

Exploiting Redundancies for Memory-Efficient LLM Training and Inference
event speaker icon
Nitzan Hodos (Ph.D. Thesis Seminar)
event date icon
Tuesday, 24.11.2026, 09:00
event speaker icon
Advisor: Prof. Assaf Schuster

Training LLMs at scale is limited by GPU memory. While most memory-efficient methods focus on compressing model weights, gradients and optimizer states, we show that the activations stored for the backward pass are the real bottleneck, as they scale with batch size and context length. In CompAct [1], we show that these activations are so redundant that storing low-rank random projections of them, instead of the full tensors, reduces peak GPU memory by 25-30% in pretraining and by 50% in fine-tuning, without hurting model performance. Then, in PAMM [2], we show that the activations of the attention projection layers specifically can be compressed by up to 512x by exploiting their redundancy along the token dimension: a small random subset of tokens is kept, and every other token is represented as a scaled copy of its most similar kept token, matching or even improving perplexity for LLaMA models of up to 7B parameters. Finally, we present AnchorKV [3], where we show how the redundancy exploited by PAMM can also be used to compress the KV cache, the memory bottleneck in LLM inference. By storing a few anchor tokens exactly, representing every other token through its nearest anchor, and adding quantized residuals where they matter most, AnchorKV compresses the cache 20x without discarding a single token, retaining 99.4% of full-cache accuracy on Llama-3.1-70B. Together, these works show that the intermediate tensors that dominate LLM memory are far more redundant than they appear, and can be compressed aggressively while maintaining performance.

[1] Y. Shamshoum*, N. Hodos*, Y. Sieradzki, A. Schuster. CompAct: Compressed Activations for Memory-Efficient LLM Training. NAACL 2025.
[2] M. Khalaf, Y. Shamshoum, N. Hodos, Y. Sieradzki, A. Schuster. QKV Projections Require a Fraction of Their Memory. ICLR 2026.
[3] M. Khalaf, Y. Shamshoum, N. Hodos, Y. Sieradzki, A. Schuster. AnchorKV: Anchor-Residual KV Cache Compression. Under review at NAACL 2027.