Unified KV Cache Compression Methods for Auto-Regressive Models
-
Updated
Aug 13, 2026 - Python
#
kv-cache-compression
Here are 58 public repositories matching this topic...
LLM KV cache compression made easy
python transformers inference pytorch kv-cache large-language-models llm long-context kv-cache-compression
-
Updated
Oct 6, 2026 - Python
Awesome-LLM-KV-Cache: A curated list of 📙Awesome LLM KV Cache Papers with Codes.
-
Updated
Jun 17, 2026
llama.cpp fork with TurboQuant WHT-rotated KV cache & weight compression + Gemma 4 MTP and Qwen 3.6 NextN speculative decoding (+30-50% throughput).
metal cpp vulkan cuda quantization mtp multimodal apple-silicon low-bit-quantization llama-cpp local-llm llm-inference gguf speculative-decoding kv-cache-compression qwen3 multi-token-prediction turboquant gemma4 nextn
-
Updated
Oct 12, 2026 - C++
[NeurIPS'25 Oral] Query-agnostic KV cache eviction: 3–4× reduction in memory and 2× decrease in latency (Qwen3/2.5, Gemma3, LLaMA3)
-
Updated
Feb 11, 2026 - Python
Block Transformer: Global-to-Local Language Modeling for Fast Inference (NeurIPS 2024)
-
Updated
Apr 13, 2025 - Python
[ICLR 2025] Palu: Compressing KV-Cache with Low-Rank Projection
-
Updated
Feb 20, 2025 - Python
First open-source implementation of Google TurboQuant (ICLR 2026) -- near-optimal KV cache compression for LLM inference. 5x compression with near-zero quality loss.
machine-learning compression deep-learning pytorch transformer attention quantization iclr vector-quantization memory-optimization kv-cache google-research llm vllm llm-inference kv-cache-compression
-
Updated
May 25, 2026 - Python
Pytorch implementation for "Compressed Context Memory For Online Language Model Interaction" (ICLR'24)
-
Updated
Apr 18, 2024 - Python
Native Windows vLLM: 0.27.1 Latest; 0.29.0 Python 3.14/CUDA 13.2 release (CPU TorchAudio), plus Python 3.13/CUDA 13.0 prerelease. PyTorch 2.13, FlashAttention/Rust, OpenAI serving, 10 KV formats, Multi-TurboQuant, experimental CPU/NVMe KV offload. No WSL or Docker.
windows gpu cuda pytorch nvidia triton msvc quantization kv-cache awq llm llm-serving vllm llm-inference flash-attention qwen kv-cache-compression turboquant vllm-windows multi-turboquant
-
Updated
Oct 10, 2026 - Python
xKV: Cross-Layer SVD for KV-Cache Compression [ICML 2026]
-
Updated
Jul 7, 2026 - Python
This is the official repo of "QuickLLaMA: Query-aware Inference Acceleration for Large Language Models"
-
Updated
Jul 16, 2024 - Python
(ACL2025 oral) SCOPE: Optimizing KV Cache Compression in Long-context Generation
-
Updated
May 28, 2025 - Jupyter Notebook
Accurate and fast KV cache compression with a gating mechanism
-
Updated
Jul 27, 2026 - Python
Fast KV-cache quantization for Apple Silicon (MLX) — 43 research-adapted compression methods with Metal kernels
python macos compression metal inference kv quantization mlx model-compression ondeviceai on-device-ai kv-cache apple-silicon llm local-llm llm-inference context-length kv-cache-compression mlx-lm metal-kernels
-
Updated
Oct 12, 2026 - Python
First open-source KVTC implementation (NVIDIA, ICLR 2026) -- 8-32x KV cache compression via PCA + adaptive quantization + entropy coding
compression pytorch nvidia transformer pca attention dynamic-programming quantization deflate entropy-coding memory-optimization kv-cache llm llm-inference kv-cache-compression iclr-2026
-
Updated
Apr 17, 2026 - Python
A list of awesome papers on compression and acceleration of Large Language Models (LLMs) and Multimodal Large Language Models (MLLMs).
acceleration compression survey pruning quantization knowledge-distillation awesome-papers large-language-models multimodal-large-language-models speculative-decoding kv-cache-compression
-
Updated
May 12, 2026
A high-throughput LLM serving engine with non-uniform KV cache compression, built on vLLM
-
Updated
Sep 9, 2026 - Python
Official repo for AMD hybrid models training and inference workflow
amd attention-mechanism mamba mla hybrid-models kv-cache large-language-models llm kv-cache-compression zebra-llama
-
Updated
Sep 11, 2026 - Python
The Unified Latent-State Memory Fabric (UL-SMF) is a hardware-software co-designed memory compression fabric that solves the memory bottleneck in long-context Transformer inference. By combining FSQ with dynamic 16-dimensional latent mapping, UL-SMF compresses Key-Value (KV) cache tensors by up to 384x while maintaining >94% semantic retention.
machine-learning compression pytorch artificial-intelligence llama gemma compression-methods compression-implementations mistral vram pytorch-implementation kv-cache llm llms vllm qwen kv-cache-quantization kv-cache-compression vram-optimization vram-optimizer
-
Updated
Aug 23, 2026 - Python
Add this topic to your repo
To associate your repository with the kv-cache-compression topic, visit your repo's landing page and select "manage topics."