דלג לתוכן (מקש קיצור 's')
אירועים

אירועים והרצאות בפקולטה למדעי המחשב ע"ש הנרי ומרילין טאוב

ניצול יתירויות לאימון והסקה חסכוניים בזיכרון של מודלי שפה גדולים
event speaker icon
ניצן חודוס (הרצאה סמינריונית לדוקטורט)
event date icon
יום שלישי, 24.11.2026, 09:00
event speaker icon
מנחה: פרופ' אסף שוסטר

Training LLMs at scale is limited by GPU memory. While most memory-efficient methods focus on compressing model weights, gradients and optimizer states, we show that the activations stored for the backward pass are the real bottleneck, as they scale with batch size and context length. In CompAct [1], we show that these activations are so redundant that storing low-rank random projections of them, instead of the full tensors, reduces peak GPU memory by 25-30% in pretraining and by 50% in fine-tuning, without hurting model performance. Then, in PAMM [2], we show that the activations of the attention projection layers specifically can be compressed by up to 512x by exploiting their redundancy along the token dimension: a small random subset of tokens is kept, and every other token is represented as a scaled copy of its most similar kept token, matching or even improving perplexity for LLaMA models of up to 7B parameters. Finally, we present AnchorKV [3], where we show how the redundancy exploited by PAMM can also be used to compress the KV cache, the memory bottleneck in LLM inference. By storing a few anchor tokens exactly, representing every other token through its nearest anchor, and adding quantized residuals where they matter most, AnchorKV compresses the cache 20x without discarding a single token, retaining 99.4% of full-cache accuracy on Llama-3.1-70B. Together, these works show that the intermediate tensors that dominate LLM memory are far more redundant than they appear, and can be compressed aggressively while maintaining performance.

[1] Y. Shamshoum*, N. Hodos*, Y. Sieradzki, A. Schuster. CompAct: Compressed Activations for Memory-Efficient LLM Training. NAACL 2025.
[2] M. Khalaf, Y. Shamshoum, N. Hodos, Y. Sieradzki, A. Schuster. QKV Projections Require a Fraction of Their Memory. ICLR 2026.
[3] M. Khalaf, Y. Shamshoum, N. Hodos, Y. Sieradzki, A. Schuster. AnchorKV: Anchor-Residual KV Cache Compression. Under review at NAACL 2027.