Loading…
Each step sends the system prompt, the tool schemas and the whole history so far. While nothing earlier is rewritten, every step is a strict prefix extension of the last, so the engine reuses the KV blocks it already holds and prefills only the new tail. That is why cached should climb across steps while new stays small.
Anything that rewrites history, summarising, dropping a tool result, reordering, invalidates the cache from the first changed token onward. You see it as a step whose cached share collapses and whose latency jumps. That is the trade this rig exists to make visible.
A hosted API reports one number, cached_tokens, and only sometimes. Hit
rate, KV utilisation, queue depth and eviction need your own engine. Point
BIOAGENTS_BASE_URL at a local vLLM or SGLang and the KV tab fills in.