LLMs are known to exhibit highly sparse computation patterns during training and inference, with multiple works exploiting these for improved model performance or reduced compute costs.
We present PAMM, a simple matrix-multiply approximation algorithm which enables runtime memory compression during both training and inference of LLMs.
Using PAMM we show that the QKV linear maps of the attention mechanism are extremely sparse during model pretraining, enabling as high as 512x memory compression for these modules.
We also demonstrate that PAMM can be used to compress the KV cache of an LLM during inference, by up to 20x with near-zero induced error on the model's performance.
