TurboQuant: KV Cache Compression for Fast Inference
Optimizing large language model inference through learned KV cache quantization. Reducing memory bandwidth bottlenecks in transformer attention without sacrificing output quality.
llama-turbo uses an interface called kv_cache and we use it to swap out the default kv cache implementation with our own quantized version. This allows us to implement KV cache quantization to accelerate inference on large language models. Instead of storing full-precision key-value pairs in attention, we learn to compress them dynamically, reducing memory pressure during token generation.
Baseline Performance (Sprint 0): 7.05 tokens/sec on Mistral-7B with 8K context window
Target (Sprint 1+): 2-3x improvement via KV cache compression
- KV Cache Quantization: Quantizing Large Language Models for Inference — Google Research on efficient KV storage
- Base Framework: llama.cpp — Efficient inference in C++
- Model Distribution: Ollama — Simple LLM deployment
- Model Hub: Hugging Face — Model & dataset repository
Baseline Models:
- Mistral-7B-Instruct-v0.1 (Q4_0) — 7B parameters, 4-bit quantization
- stories15M-q4_0 — 15M parameter tiny model for smoke tests
Datasets:
- WikiText-103-v1 — Real-world corpus for baseline stress tests (1810 words context)
llama-turbo/
├── llama.cpp/ # Submodule: upstream inference engine
├── scripts/
│ ├── run-on-mac.sh # Local macOS test harness (download, validate, benchmark)
│ ├── smoke-test.sh # Lightweight inference validation
│ └── verify-release.sh # Check release artifacts
├── .github/workflows/
│ ├── ci.yml # Linux + macOS builds on every push/PR
│ ├── benchmark.yml # Inference smoke tests on main
│ └── release.yml # Package artifacts + benchmark results
├── benchmark/results/ # Baseline measurements (JSON)
├── tests/ # Upstream llama.cpp tests
└── SPRINT_0_REPORT.md # Sprint 0 completion report
| Sprint | Goal | Status | Report |
|---|---|---|---|
| Sprint 0 | Establish testing infrastructure & baseline metrics | ✅ Complete | SPRINT_0_REPORT.md |
| Sprint 1 | Implement TurboQuant KV cache compression | 🔄 In Progress | — |
| Sprint 2+ | Optimize for multi-GPU, edge devices | 📋 Planned | — |
./scripts/run-on-mac.shDownloads latest release, validates binaries, runs smoke test, then optionally runs baseline stress test with 7B model.
Requirements:
- macOS or Linux
- 10GB free RAM (for 7B baseline)
ghCLI (GitHub Actions integration) - logged in- ollama installed and logged in
- hf cli installed and logged in
./scripts/verify-release.shChecks latest GitHub release for binaries and test artifacts.
./scripts/smoke-test.sh --build-dir binQuick 10-token inference test (18MB model, ~2 seconds).
Architect: Mickey Perlstein
Core Components:
- Inference Engine: llama.cpp (C++ with Metal GPU support)
- KV Compression: Learned quantization during attention
- Testing: Real-world datasets (WikiText-103), multi-part reasoning tasks
- CI/CD: GitHub Actions (automated builds, benchmarks, releases)
Memory-Bound Stress Test
- Model: Mistral-7B-Instruct-v0.1 (Q4_0)
- Context: 1810 words from WikiText-103-v1
- Task: Multi-part summarization + analysis (1000 tokens)
- Configuration: 8K context window
- Result: 7.05 tokens/sec
- Status: Memory-bound (KV cache is bottleneck)
Why this baseline matters: Large context + large output = massive KV cache. Compression gains will be dramatic.
# Configure (Metal GPU on macOS)
cmake -B build -S llama.cpp \
-DGGML_METAL=ON \
-DLLAMA_BUILD_TESTS=ON \
-DLLAMA_BUILD_EXAMPLES=ON \
-DCMAKE_BUILD_TYPE=Release
# Build
cmake --build build --config Release -j$(nproc)
# Test
ctest --test-dir build --output-on-failureBased on llama.cpp (MIT License)
- llama.cpp — Efficient inference framework
- Ollama — Model serving
- Hugging Face — Models & datasets
- KV Cache Quantization Research — Google Research
- WikiText-103 Dataset — Real-world text corpus