fraQtl — KV-cache compression with receipts

2.3× the KV cache capacity of fp16 (1.2× fp8) at parity decode speed, in stock vLLM on A100 — every number needle-in-a-haystack-gated, and independently reproducible for ≈$12 with one command.

We publish a combination of receipts no one else in KV-cache compression publishes: three arms (fp16 / fp8-KV / compressed) run same-command on the same engine, retrieval-gated per user, with preemption-free concurrency counts — and a one-command independent reproduction kit. Engine vendors publish serving throughput without a compressed arm or a retrieval gate; compression papers publish accuracy tables without a serving engine. We publish both, together, on every number. If a number is on one of our cards, there is a receipt JSON behind it and a script that regenerates it on a rented A100. (Why gate on retrieval? A shipped fp8-KV configuration in vLLM once collapsed 128K needle accuracy from 91% to 13% — throughput numbers without a retrieval gate can be meaningless.)

Collections

• KV-Cache Receipts — 3× the 128K users per A100 (vLLM)
• Hi-Fi Quants — calibrated GGUFs with KLD receipts

Pick your lane

How the KV compression works (concept)

A compression membrane between the model and vLLM's paged KV cache: writes store K/V in compressed pages (a small calibrated protected subspace at high precision + an INT4 tail); attention reads the compressed pages directly in-kernel at tensor-core speed — no decompress-then-attend, so capacity doesn't cost decode latency. Calibration is per-layer, per-KV-head on wikitext-2; the calibrated sidecars are published, the calibration method is not.

Honesty rules (every card, every number)

• Retrieval-verified via needle-in-a-haystack grids — we never say "lossless"
• fp8-KV is the honest competitor and appears in every table
• Batch size and context stated on every number
• Reproduce it yourself: one command, a Modal account, and ≈$12 of A100 time

Verification requests and model requests: open a discussion on any repo.