How Orb Reuses the LLM's KV Cache

An LLM builds a KV cache as it reads a prompt. A later request can reuse that work when its prompt starts with the same tokens. The cache is prefix-based: changing an early token invalidates everything after it, while appending text at the end is cheap.

Orb's prompt architecture keeps stable conversation context at the front and puts turn-specific instructions at the end.

Animation: KV cache animation walks through two turns and the reasoning-mode caveat.

The prompt shape

Think of a call as a stack. The bottom is shared; only the top changes:

system prompt       ← stable
conversation history ← stable for this exchange
trailing request     ← varies by pass and speaker

CachedBase captures the stable part once per turn and endpoint. Director, Writer, Editor, and workflow calls extend that base with their own trailing messages.

The three passes

Pass	Adds to the base	Output
Director	OOC instruction and the current user message	A forced direct_scene tool call
Writer	Selected lore, scene direction, and the user message	Streaming prose
Editor	The Writer request, the evolving draft, and an edit instruction	Audit fixes, the subject fixation edit, post-processing patches, or feedback

The Editor extends the Writer request instead of rebuilding a new conversation. Its first call can therefore reuse the Writer's history and draft; later ReAct iterations only add the newest instruction and tool result.

Director output, keyword lore, and other per-turn values stay in the trailing request. They must not change the system prompt or the shared history.

Rules that preserve reuse

Build the base once

The system prompt and history are assembled once in backend/pipeline/config.py and held in an immutable CachedBase from backend/inference/cached_call.py. Passes call base.complete(...); they do not assemble their own prefix.

Deterministic message and context construction lives in backend/prompting/. Pass-specific instruction prose lives beside its pass under backend/pipeline/passes/; provider and cache execution remain in backend/inference/.

The system prompt includes stable card, persona, scenario, constant lore, and scene instructions. History is shared byte-for-byte, including attachment encoding. A macro that changes those bytes, such as an unseeded {{roll}}, breaks reuse; persisted message text is used for later turns.

Open each request with one message

Every pass starts its trailing request with a single user message directly after the base. On models whose cache cannot be rewound to any position, such as sliding-window Gemma 4 and hybrid-attention Qwen3.8, llama.cpp keeps restore points at the start of each request's last user message. A one-message request therefore leaves a restore point at the end of the history, and the next turn's Director resumes there. Text shared inside a request, such as the lore block the solo Director and Writer both open with, is not reused: a request resumes at the message start, unless the previous call ended within a few hundred tokens of the divergence.

Measured on Qwen3.8 27B: moving the lore into a user message of its own let the Writer reuse it (about 1,250 tokens), but left no restore point at the end of the history. The next Director then re-read about 6,000 tokens. Text mode has no message boundaries to place restore points at, so on these models it re-reads 400 to 1,200 more tokens per call than chat mode.

Keep group context consistent

In a group, history labels each assistant message with its member. The selected character context mode determines whether card text is in the shared body or in the active speaker's trailing message. Every call for a speaker uses the same mode and speaker, so the Editor never audits a draft against a different cast.

Swap mode creates a cache lane per speaker because the active card appears before history. Private perspective keeps the card after history, leaving one common trunk. Shared dossier has one common body containing every member's dossier.

The Director runs before a speaker is selected, so it uses the neutral group base. The first speaker must still rebuild when the mode makes the prefix speaker-specific.

Treat tools as part of the prompt

Orb sends a stable, ordered tool list through the base, even to passes that do not call a tool. The pass selects its behavior with tool_choice:

• Director and Editor force one tool.
• Writer uses tool_choice="none" so it writes prose.
• Workflow tools add their schemas to the same per-turn list.

backend/prompting/tool_catalog.py owns the list. Built-ins are emitted in the explicit BUILTIN_TOOL_ORDER; workflow tools append in registration order, and re-registering one preserves its position. Schema property order and compact, insertion-order-preserving JSON bytes are part of the cache contract.

forced_tool_call appends the forced schema when the per-turn map does not already carry it, and an appended schema is a diverging tools region: it renders ahead of history, so it evicts the whole conversation from the server's prefix cache rather than costing only its own bytes. A call that forces a tool therefore has to pick a side. Either it rides the turn's lane, and the tool must be in the map before resolve_pipeline_config freezes it into a CachedBase — what the Agent's own tools do, and what a workflow would do by yielding enable_tools from a pre-pipeline hook. Or it rides its own lane, and shares nothing with the turn: its own short prefix, and enabled_tools=None so forced_tool_call ships the forced tool alone. What it cannot do is force a tool against the turn's blob without being in it.

The second option is usually cheaper when the call is closed over its explicit input. format_consistency's voice rewrite restates one draft against the newest reply already in the target voice and does not need the rest of the conversation; sending the history would only add prompt cost. Servers can retain multiple cached sequences, so this short lane can coexist with the conversation lane.

Inference servers may render only the forced tool, or no tools for none. As a result, the three passes can share the conversation body without sharing the entire rendered prefix. Think in cache lanes: each distinct rendered shape warms and reuses its own lane across turns.

The exact cache hit is provider-specific. Provider usage is the source of truth; Orb's local tracker is only a diagnostic signal.

The Agent's own tools follow the same rule. Whenever the Agent is on, the blob offers direct_scene, editor_apply_patch, editor_rewrite, and editor_find_replace, and select_lorebook whenever Agentic Lorebook is enabled. The Director toggle, the output auditor toggle, the length guard, and whether any entry is pickable gate the passes through PipelineConfig.active_tools and the lorebook turn; they never change the schemas. PipelineConfig.enabled_tools is the offered map that workflow calls rebuild the blob from.

Fragment cooldowns follow the same rule as the speaking-plan roster: their volatile availability is stated in the trailing Director prompt and enforced server-side, never by changing the tool schema.

Enabling or disabling a fragment follows it too. direct_scene, give_feedback, and update_state are built from every fragment the user and the cast's cards define, enabled or not, in an order no toggle moves. direct_scene lists every fragment flagged required as top-level required, enabled or not, because upstreams that decode against the tool schema follow it and ignore a "Required:" line in the trailing request; a disabled required field is filled and dropped. Per-fragment Director mode leaves required empty, since each step fills one field. give_feedback and update_state carry no required, and give_feedback joins the list once any feedback fragment is defined. A fragment-built property carries its name and type only. The blob reaches every call on the lane, the Writer's included in single-model mode, so a description there would put a disabled fragment's instructions in front of the reply. Code-authored properties (moods, speaking_plan, retire) and tool descriptions keep their text. Each call's trailing request lists only its live fields with their descriptions and requiredness, names the offered-but-unavailable ones, and passes that live view, descriptions restored, as the per-call json_schema, which narrows text-mode grammars and structured-output endpoints while leaving the prompt bytes alone. Values for fields the call was not offered live are dropped. Editing a fragment's description leaves the blob unchanged; creating or deleting a fragment, renaming its id, flagging it required, or moving it to another tool or between a single value and a list still rewrites it, as an authoring change rather than a toggle.

The stakes, measured on an 8k-token Director call: a tools list that changes on a toggle rebuilds the whole prefix wherever tools render ahead of the conversation. llama.cpp with Gemma 4 keeps 0 of 7.8k tokens (8.8 s of prefill against 1.2 s), and Claude through OpenRouter reads 0% and rewrites 11k tokens. Providers that render only the forced tool (NanoGPT, OpenRouter's Together upstream) ignore a tool appearing, but still lose the conversation when direct_scene itself changes.

Endpoints with structured_tool_calls use a response schema and omit tool schemas from every chat request. Native text-completion mode also omits tools from the prompt and uses a grammar or client-side parsing. These modes keep the same stable system/history contract with a simpler prefix.

Separate model lanes

When the Agent uses a different model from the Writer, the models cannot share a KV cache. The Writer sends no Agent tools; Director and Editor share the Agent model's base instead.

Keep optional work on the same path

Agentic Lorebook selection is one more trailing request on the Agent lane's base, between the Director and the Writer. Its catalog omits constant entries; the constants stay in the shared system prompt. A selection prefix of its own would diverge inside the system prompt, ahead of the whole history: on a single-slot llama.cpp server with Gemma 4, that re-prefilled the conversation twice per turn, once for selection and once for the Writer after it.

Feedback, state updates, and document auditing extend the relevant prompt in the same way as the Editor. Image prompting rebuilds the neutral scene prefix through the shared cast resolver, so off-turn calls use the same group history and context rules as a turn. Images an image prompt adds -- each review's render, and the chat's earlier picture when prompter reference is on -- ride in the compose tail and never in the prefix, so the selector and compose calls share it and each review extends the compose request byte for byte.

One turn, briefly

For a user message such as I draw my sword:

1. Build the base from the system prompt and prior history.
2. Ask the Director for scene direction.
3. Ask the Writer for prose, adding direction and selected lore to the tail.
4. If needed, extend that request with the draft and let the Editor apply audit fixes, then the subject fixation edit, then each post-processing fragment in order.
5. Run Feedback and secondary workflows against the resulting draft.

The first three calls may have different tool-rendered lanes, but each call keeps its own reusable prefix. The Editor's follow-up calls extend the prompt it just used.

Across turns

The next turn appends the saved user and assistant messages to the old history:

turn N:   system + history + writer request
turn N+1: system + history + user N + assistant N + director request

That shared beginning is why a long conversation does not require a complete prefill on every call. A group exchange follows the same rule as speakers add their replies; lore activation itself remains frozen until the next exchange.

Batch lanes

A few passes are not conversation turns at all: the library auto-tagger sends one call per card, and every call carries the same system message and the same tools blob, with only the card's text moving. That is a prefix shared across the whole run rather than across a conversation, and it is why the run is sequential — parallel requests land in separate llama.cpp slots with separate KV caches, so the instruction block gets paid for once per slot instead of once per run.

The test-time checker knows about these: a batch pass groups its calls by the pass name instead of by the conversation, or N calls would land in N groups of one and be skipped. See _BATCH_PASSES in tests/integration/_llm_mock.py.

Reasoning can create another lane

Common chat templates rewrite the top of the system turn on the thinking flag: Gemma 4 inserts <|think|>, and Qwen3.8 inserts a reasoning-effort preamble. A pass that reasons and a pass that does not therefore render different prefixes from the first bytes.

In text mode Orb renders the prompt itself, so it keeps one render per lane. lane_template_thinking turns the lane's thinking on when any pass sharing that server reasons, and CachedBase.template_thinking carries it to every call on the lane. A call with reasoning off keeps the thinking-on bytes and closes the thought at the tail with the template's own empty span. forced_tool_call uses the Agent lane's value.

In chat mode the server renders the template, so differing reasoning_enabled_passes values still split the lane. llama.cpp's host-memory prompt cache restores each side on hybrid-attention models such as Qwen3.8, but not on sliding-window models such as Gemma 4. Keep the setting uniform when cross-pass reuse matters on chat endpoints.

Providers that cache only where asked

llama.cpp, DeepSeek, OpenAI, and most routed open models reuse any matching prefix on their own. Claude (natively or behind OpenRouter and NanoGPT), and Gemini and Qwen behind OpenRouter, cache nothing unless the request marks where a reusable prefix ends, and they find an entry again only at a position an earlier request marked. A single marker on the last block, which is what top-level automatic caching does, never matches in Orb: the trailing request differs on every call. Measured on Claude, that layout wrote the whole prompt on every call and read none of it.

backend/inference/prompt_cache.py therefore marks three positions on every chat request:

Anchor	TTL	Reused by
End of the system prompt	1 h	Every call on the lane, including edited histories and batch lanes
Last message of the CachedBase prefix	1 h	Every pass of the turn, and the next turn
Final block	5 min	The Editor extending the Writer's request, and its ReAct iterations

The experimental Claude Code CLI marks its system prompt and the final block of its input itself. Orb sends the transcript as JSON data in one CLI user message, split into one text block per entry, and marks the last CachedBase entry with the CLI's 1 h TTL; the CLI keeps that marker on the wire. Every earlier base end is a block boundary, so each turn and regeneration reads the previous call's base and writes only the entries after it.

A structured call reaches the model as the CLI's StructuredOutput tool, and tools precede the system prompt in the cached prefix. The adapter therefore builds that schema from the lane's tool list alone, one key per tool plus none, so every agent pass of a turn sends the same tool. The pass's tool choice rides in the final block as an instruction, and the adapter rejects an answer under another key than the forced one.

CachedBase.complete passes its prefix length as cache_prefix_len, so the base anchor sits exactly where the frozen base ends. Calls without a base get only the system and final anchors. The 1 h TTL on the base covers a reader's pause between turns: a miss there rewrites the whole conversation, while the longer TTL only doubles the price of each turn's new messages.

Servers that do their own prefix caching ignore the markers. A marked message travels as a one-part text list instead of a string, and that renders to the same bytes (llama.cpp's template output and hosted token counts agree), so the marker moving between messages never changes the rendered prefix. A server whose schema refuses the list shape gets one unmarked retry; if that retry is accepted, the pair is sent no markers for the rest of the process.

Each request also carries an x-session-id derived from the model and the lane's first message. OpenRouter spreads a model across upstreams with separate caches, and without a session id it only becomes sticky after a cache hit, which a lane that changes upstream on every call may never get. Servers that do not read the header ignore it.

Configuration wins over both: a top-level cache_control in the endpoint's extra body replaces the breakpoints, and an x-session-id in its extra headers (any casing) replaces the derived one.

What the tracker tells you

Orb records two kinds of cache information:

• Provider usage: the number of prompt tokens actually served from cache.
• Local estimate: message-prefix overlap and whether the sent tool blob changed from the previous same-label call.

The local values cannot know where a provider's template renders tools, so they are kept separate. A high message overlap with a low provider hit can indicate different tool rendering or reasoning lanes.

For Anthropic usage, the report totals input_tokens, cache_read_input_tokens, and cache_creation_input_tokens; input_tokens alone is only the uncached portion. The Claude Code CLI transport reports its provider cache reads, but its adapted response schema is not sent as an Orb tool array, so the tracker does not compare tool arrays for that transport.

The tracker compares a call with the previous call on its own lane, falling back to the last same-label call of the previous turn — which makes the first call of a new turn useful rather than an unhelpful zero baseline.

A tracker lane is (server, model, shape). Server and model identify the physical cache owner, matching the first two fields used by the test-time checker. Shape is empty for the shared conversation prompt and has a stable name for an intentionally independent rendered prefix. For example, format consistency's self-contained voice rewrite records format_consistency:voice_rewrite, so a group exchange's next Director call skips that short prefix and compares with the last conversation call instead. forced_tool_call accepts this as cache_shape and passes it to the tracker.

The shape is a semantic discriminator, not a fingerprint of the current message or tools bytes. Calls in one intended family must remain comparable so an accidental change still appears in the report. Keying on the model name alone is also insufficient: dual-model runs the writer and the agent on different servers, and the two routinely answer to the same name. When a turn spans more than one lane the report prints a lanes: legend and tags each row, so a pass with no predecessor reads as "first call on its lane" rather than as a broken prefix.

In short

Keep system prompt and history stable. Put changing direction, lore, and tool choices at the tail. Extend an existing prompt for Editor and workflow work. Use provider usage to judge the result, and expect separate lanes when model, tool rendering, speaker context, or reasoning mode changes.