Can't wait to try this out on my LuceBox! 2 weeks to go!
Sandro (@pupposandro)
Excited to launch Luce KVFlash. We've been working harder than ever with @davideciffa to bring better DX for local AI.
Today, long context has a second memory bill nobody budgets for: the KV cache.
On Qwen3.6-27B at 256K it costs 4.6 GiB of VRAM and drags decode down to 13 tok/s, because every new token reads the whole thing.
KVFlash keeps a small pool of KV on the GPU, auto-sized to your VRAM, and pages cold 64-token chunks to host RAM, bit-exact and recallable.
decode holds a flat 38.6 tok/s from 64K to the native 256K on a 3090, 2.9x the full cache at 256K, 72 MiB resident and benchmark accuracy unchanged.
— https://nitter.net/pupposandro/status/2066212966458933565#m