Deep research you can follow — claim by claim, source by source.
The Cache Miss After the Gap
The aggregate cache hit rate is the wrong instrument for agent serving. Coding agents hit 95.7% of their input tokens . They get there because 88% of their calls are tool-result continuations that could hardly miss . The call that does miss is the first one after an idle gap: a human message arriving minutes later, hitting at about 84.4% against 97.5% for a continuation . An upper-bound counterfactual in the same trace prices that class at $5,189 . That is 12.8% of final cost . It is a small class of call carrying a large share of the bill.
Agents run on two clocks and serving systems give them one timer. Tool gaps are seconds; the gap between human requests has a median of about 1.4 minutes . Its mean is 46.7 minutes . Holding state across the longer clock works, and it is not free. In an idealized sweep of the trace, raising the eviction timeout from one minute to one hour lifts the achievable hit rate from 85.4% to 98.6% . It costs roughly seven times more suspended KV . Every mechanism here is a policy about that pause: how long to keep the state, where, in which tier, and who may delete it.
Latency targets inherit the same averaging mistake. Enforcing them costs 60% of throughput on an agent workload at 64K context . A human-interactive control at a different SLO setting gives up 1.9% . That is the cost of caring about latency for a caller who is a loop.
The spine here is one lab's instrumented trace of its own Claude Code and Codex sessions, and the harness above the server can void any of it. Compaction already rewrites the context in 9.7% of sessions , and most of that is not a person's decision: 86.5% of those compactions are tool-initiated .
The SyFI coding trace contains approximately 4,300 sessions and 55B tokens in total, collected from daily use of Claude Code and Codex in the SyFI lab. Each user request takes an average of 8.8 self-directed steps (LLM-tools cycles) and issues 10.8 tool calls before giving the final answer. End-to-end response time per request is heavy-tailed: median ~38 s, mean ~4 min, and p99 nearly 44 min; the gap between requests averages 46.7 min with a median of ~1.4 min. Across the trace, models read 52.56 B cached input tokens and prefill 2.34 B new ones, yet generate just 186.9 M output tokens—inputs outnumber outputs by 294×.
Existing model-quality benchmarks such as Terminal-Bench and SWE-bench are poorly suited to modeling serving-system performance because they involve relatively few tool calls and focus on single, isolated tasks. Coding agents do not answer in one shot; for each user request, the agent executes an autonomous loop of model generations and tool calls before returning a final answer, with 88% of all LLM rounds responding to a tool result rather than a human. Prefix caching is effective but not optim
The public SYFI pool contains 357,161 agent steps across Claude (140,338, 39%) and Codex (216,823, 61%). Total input tokens: 54.9B; Cached-read tokens: 52.6B; Append input tokens: 2.34B; Total output tokens: 186.9M. Median prefix tokens: 126K (Claude) / 116K (Codex); Median append tokens: 857; Median output tokens: 252 (Claude) / 184 (Codex). Sessions with ≥1 compaction: 9.7%; tool-initiated compactions: 86.5%; average cost per session: $9.36; prefix share of cost: 61.7%. TraceLab collected 4,265 sessions (2,676 Claude Code sessions and 1,589 Codex sessions) from 43 developers between September 2025 and June 2026, containing 357,161 LLM steps and 432,510 tool calls.
TraceLab provides real-world coding agent traces collected from Claude Code and Codex, capturing multi-step agent sessions, LLM requests, tool calls, cache behavior, and latency distributions. The dataset shows that a large majority of input tokens are served from the prefix cache, with 52.6B cached-read tokens out of 54.9B total input tokens.
The paper uses Qwen-2.5-32B in FP16 on an 8-GPU NVIDIA H20 node with tensor parallelism. The primary workload is a closed-loop benchmark derived from MoonCake ToolAgent traces: 1000 sessions of 10 rounds each, with context lengths in {32768, 65536} tokens, plus shorter-context experiments at 8192 tokens. At 8K context, the throughput gap between the tightest SLO (C=512) and the human-less baseline is 15% (6400 vs. 5440 tok/s); at 32K the gap is 40% (5820 vs. 3490 tok/s); at 64K it reaches 60%. For the human-interactive workload (Poisson-arrival with 2s mean inter-arrival time) at 64K context, the sacrifice at production SLO settings (C=8192) is 1.9% (6030 vs. 5920 tok/s).
Every major LLM serving system is designed to meet TTFT and TPOT SLOs, but long-horizon AI tasks (tool-use agents, coding pipelines, multi-round reasoning chains) call LLMs programmatically in tight loops where no human observes TTFT or TPOT, so serving systems sacrifice throughput to meet SLAs these workloads never need. The throughput sacrifice (the 'human tax') grows with context length: at 8K context the gap between the tightest SLO and the human-less baseline is 15%, at 32K it widens to 40%
Every report belongs to one of three, by the kind of question it answers.