Appendix: how we measured the cost of waiting with and without MCP Events
The appendix to "MCP Events beat polling on reliability and token cost".
This appendix was generated by an LLM (Claude). A script copied its tables from the studies' results files; Claude wrote the text around them from the same files, and it was checked against the data but not written by hand.
This page holds what the post leaves out: each study's design, every per-server and per-approach table, the tests in full, the exploratory results and every departure from the plan. Servers are named by category because their vendors haven't reviewed the results.
Dollars throughout are API list price per million tokens. Opus 5.5: $4 input, $5 (5-minute) or $8 (1-hour) cache write, $0.20 cache read, $20 output. GPT-6.1 Sol: $2.50 uncached input (priced as a cache write, since Codex signed in with a subscription reports no cache writes), $0.10 cached input, $10 output, with a higher tier above 272k tokens. On a subscription plan the same requests count against usage limits, so request counts sit beside the dollars. Both studies ran Claude Code 2.1.321 on Opus 5.5 and Codex 0.167.1 on GPT-6.1 Sol on one machine. Wakeshift versions: study 1's round 2 ran plugin v1.4.50 and v1.4.51, the long holds v1.4.52, and its API-key run v1.4.54 and v1.4.55; study 1's round-1 sessions, which include every events arm, mostly don't record their version in the data (512 of 525; the run logs have it); study 2 ran v1.4.54 (Codex connector 2026.11.14.1515, with a fix to how it holds its connection). Dates (UTC): study 1, November 12 to 14; study 2, November 14 to 16, the API-key run on November 16.
The two studies measure different things. The first measures the cost of one wait for a job; the second measures cost per idle hour and missed replies over a two-hour window. Their numbers are never comparable directly.
Study 1: Long-running jobs
Design
The question: an agent starts a job on a vendor's MCP server that runs longer than the tool call. What does it spend in model requests before it tells the user the result, when it keeps checking (polling) and when the server tells it the job finished (an event)?
Arms:
- A, as shipped: the vendor's MCP server at its upstream base commit, untouched. Round 2 re-measured some servers with progress streamed through the runner's proxy (arm A2); round 1's arm is A1. Where both exist, A2 replaces A1 without pooling, and the comparison against round 1's events arm is labeled "across rounds".
- B, events: the same server emitting an event when the job finishes, delivered through Wakeshift's bridge as a stand-in for native MCP Events support (SEP-3415). The tool subscribes the conversation and returns at once only when Wakeshift confirms it can wake that conversation and the agent didn't choose a wait the client can hold; otherwise it behaves exactly like arm A.
- L, long-poll: arm A's server with the status tool made to block until the job ends or a deadline of 50 s (Claude Code) or 280 s (Codex). The runner's proxy repeats the agent's own status call every 5 s and returns the first answer showing the job finished, or the last one at the deadline. The tool's description gains one sentence: "This call waits up to N seconds for the job to finish; if it's still running, call it again." The deadlines are our settings, chosen under each client's limit for one call (Claude Code's 60 s for a call that sends nothing; Codex's 300 s).
- E, equalized: for the AI model runner only, arm A with every tool description identical to arm B's apart from the wake, so B differs from E only by the event.
Wakeshift's plugin is installed in every round-2 arm. Each session is a fresh conversation in which a simulated user asks for the job and its result in one message. If the agent hasn't told the result 120 s after the job ends, the user asks once.
Measures, per session:
- The waiting episode: every model request after the one that issued the job call, through the request that produced the final answer, or to the session's end when the agent never tells it. Claude Code's turn-end helper requests count (measured in round 2, imputed in round 1; the imputation runs +4.1% against Claude Code's own record).
- Requests per wait: model requests in the waiting episode, helpers included.
- Per conversation: every model request of the session.
- Had to ask and never told: counts per cell.
Statistics: medians with interquartile ranges. For each pair of cells with n ≥ 5, a 95% bootstrap interval on the difference of medians (10,000 resamples, seed 1113) and a one-sided permutation test. A saving is detected when p < 0.05 and the interval's lower end is above zero; otherwise there is no detectable difference. Nothing is pooled across servers or harnesses. Cells with n = 2 are shown as values only.
The analysis plan for round 2 was written and committed on 2026-11-13, before any round-2 run and before the round-1 data was re-scored under it. Round 1 itself was not pre-registered; three adversarial reviews of it (methods, harness and a hostile reader) found the problems the plan fixes, including the buffering proxy described under the deviations.
Discards apply to both arms alike: a bench failure shown by the logs, a vendor failure (the job never started or failed at the vendor), or loss of network on the test machine. The plan counted failures of Wakeshift (a failed subscribe, a late or missed wake, a server error) as the events arm's outcome. Two later rules, made after the results were known and applied to every arm, changed that: sessions that failed during an outage of infrastructure unrelated to the approach are excluded (deviation 12), and so are sessions run before the fix of a bug we found during the study (deviation 14). Every exclusion rule is described here with its effect, and the sessions it removed are listed or counted. Sessions that crossed a change of the Wakeshift version under test are kept and flagged (the Straddlers column in Every cell); in some events cells they are the majority, for example 6 of 8 in web crawling on Claude Code and 7 of 12 in AI video generation on Codex.
Scope. The study ran six servers with jobs the agent starts, longer than a minute, checked through a status tool, plus an agent communication server (the agent waits for an email reply sent about three minutes after its own message), a crawling API that holds its call and sends nothing, a research API that holds its call while streaming, short jobs (20 to 45 s) as controls, and 200k tokens of earlier context at n = 2 per arm, values only. The agent communication server (email replies) is a wait the agent didn't start, so we moved it out of study 1's conclusion on 2026-11-16; the second study covers that kind of wait, and its rows stay in every table here. The post's conclusion covers the remaining cells where holding the request open doesn't fit; the scope and the cells it leaves out are under "Outside the conclusion's scope" below, with the short controls (events cost $0.001 to $0.06 more per job there).
Left out of the study: a web data collection service and a file encoding service, whose jobs usually finished inside the tool's own wait; a site-migration service, whose credits ran out; and one server whose use case didn't fit, removed after the analysis (our decision). Apart from the removed server, every excluded cell is in the published dataset.
Spend: round 2's model total was about $36 at list price, the long-hold extension $1.92, and the API-key extension $11.30 on the API key.
Main results: fresh context, subscription
"Saved" is median as shipped minus median events, with the bootstrap 95% interval. "Asked" and "Never told" give arms A / L / B.
This table counts every session that ran, before the exclusion rules below. With them applied, as in the post, the research API in Claude Code costs 14% more with events instead of 22%, and no events user had to ask for a result on it or on the second crawling API in Claude Code (see "Sessions excluded for infrastructure failures").
| Server | Harness | Sign-in | As shipped (arm) | n | Requests per wait | Wait $ as shipped | Long-poll $ (n) | Events $ (n) | Saved | p | Verdict | Long-poll → events | $ per conversation | Asked A / L / B | Never told A / L / B |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Web crawling | Claude Code | subscription | A1 | 8/8 | 21.0 → 4.0 | 0.318 (0.302–0.342) | 0.117 (8), 6.5 req | 0.078 (0.076–0.093) | 76% ($0.240; CI 0.208 to 0.271) | 0.002 | detected | 34%, detected | 0.418 → 0.188 (55%, detected) | 4 / 0 / 0 | 0 / 0 / 0 |
| Web crawling | Codex | subscription | A1 | 8/8 | 17.0 → 4.0 | 0.078 (0.073–0.104) | 0.059 (8), 10.0 req | 0.036 (0.036–0.042) | 53% ($0.041; CI 0.031 to 0.075) | 0.006 | detected | 38%, detected | 0.104 → 0.072 (31%, detected) | 0 / 0 / 0 | 0 / 0 / 0 |
| Video editing | Claude Code | subscription | A1 | 8/10 | 23.0 → 4.0 | 0.308 (0.293–0.337) | – | 0.089 (0.089–0.091) | 71% ($0.219; CI 0.171 to 0.255) | 0.001 | detected | – | 0.419 → 0.197 (53%, detected) | 0 / – / 0 | 0 / – / 0 |
| Video editing | Codex | subscription | A1 | 8/10 | 17.0 → 3.0 | 0.102 (0.087–0.111) | – | 0.038 (0.038–0.056) | 63% ($0.064; CI 0.035 to 0.080) | 0.001 | detected | – | 0.125 → 0.066 (47%, detected) | 0 / – / 0 | 0 / – / 0 |
| Web scraping | Claude Code | subscription | A2 (across rounds) | 8/9 | 14.5 → 4 | 0.331 (0.307–0.374) | – | 0.080 (0.077–0.164) | 76% ($0.251; CI 0.138 to 0.314) | 0.002 | detected | – | 0.522 → 0.289 (45%, detected) | 0 / – / 0 | 0 / – / 0 |
| Web scraping | Codex | subscription | A1 | 8/9 | 19.0 → 3 | 0.140 (0.116–0.154) | 0.089 (8), 15.0 req | 0.045 (0.041–0.046) | 68% ($0.094; CI 0.060 to 0.115) | 0.002 | detected | 49%, detected | 0.229 → 0.145 (36%, detected) | 0 / 0 / 0 | 0 / 0 / 0 |
| Agent communication (email reply) | Claude Code | subscription | A1 | 8/9 | 15.0 → 4 | 0.203 (0.194–0.205) | – | 0.059 (0.059–0.060) | 71% ($0.143; CI 0.134 to 0.147) | 0.002 | detected | – | 0.323 → 0.176 (46%, detected) | 0 / – / 1 | 0 / – / 0 |
| Agent communication (email reply) | Codex | subscription | A1 | 8/8 | 12.5 → 3.5 | 0.085 (0.056–0.104) | – | 0.038 (0.034–0.042) | 55% ($0.047; CI 0.014 to 0.096) | 0.002 | detected | – | 0.136 → 0.095 (30%, detected) | 0 / – / 0 | 0 / – / 0 |
| CI builds | Claude Code | subscription | A1 | 8/11 | 7.0 → 4 | 0.139 (0.135–0.148) | – | 0.097 (0.093–0.120) | 30% ($0.041; CI -0.003 to 0.053) | 0.078 | no detectable difference | – | 0.352 → 0.347 (2%, no detectable difference) | 0 / – / 0 | 0 / – / 0 |
| CI builds | Codex | subscription | A1 | 8/8 | 13.0 → 3.0 | 0.088 (0.060–0.101) | – | 0.036 (0.036–0.036) | 59% ($0.052; CI 0.022 to 0.078) | 0.002 | detected | – | 0.131 → 0.091 (31%, detected) | 0 / – / 0 | 0 / – / 0 |
| AI video generation | Claude Code | subscription | A2 (across rounds) | 8/8 | 4.0 → 4.0 | 0.046 (0.045–0.046) | 0.047 (8), 4.0 req | 0.058 (0.057–0.058) | -27% ($-0.012; CI -0.013 to -0.011) | 0.984 | events cost more | -23%, events cost more | 0.146 → 0.180 (-24%, events cost more) | 0 / 0 / 0 | 0 / 0 / 0 |
| AI video generation | Codex | subscription | A1 | 8/12 | 11.0 → 3.0 | 0.046 (0.045–0.059) | 0.071 (8), 16.0 req | 0.033 (0.032–0.033) | 30% ($0.014; CI 0.011 to 0.040) | 0.000 | detected | 55%, detected | 0.096 → 0.083 (14%, no detectable difference) | 0 / 0 / 0 | 0 / 0 / 0 |
| AI model runner | Claude Code | subscription | A2 (across rounds) | 8/9 | 4.0 → 4 | 0.053 (0.053–0.133) | – | 0.063 (0.062–0.064) | -18% ($-0.010; CI -0.011 to 0.079) | 0.991 | events cost more | – | 0.196 → 0.228 (-16%, events cost more) | 0 / – / 0 | 0 / – / 0 |
| AI model runner | Codex | subscription | A2 (across rounds) | 8/11 | 13.0 → 4 | 0.116 (0.094–0.134) | – | 0.046 (0.045–0.052) | 60% ($0.069; CI 0.036 to 0.088) | 0.003 | detected | – | 0.215 → 0.136 (37%, detected) | 0 / – / 0 | 0 / – / 0 |
| Crawling API that holds the call | Claude Code | subscription | A2 (across rounds) | 8/8 | 6.0 → 5.0 | 0.094 (0.092–0.094) | – | 0.097 (0.096–0.097) | -3% ($-0.003; CI -0.006 to -0.002) | 1.000 | events cost more | – | 0.200 → 0.218 (-9%, events cost more) | 8 / – / 1 | 8 / – / 0 |
| Crawling API that holds the call | Codex | subscription | A1 | 7/7 | 22 → 3 | 0.141 (0.124–0.171) | – | 0.032 (0.031–0.032) | 77% ($0.109; CI 0.081 to 0.141) | 0.006 | detected | – | 0.187 → 0.083 (56%, detected) | 7 / – / 0 | 6 / – / 0 |
| Research API | Claude Code | subscription | A1 | 8/8 | 4.5 → 5.5 | 0.179 (0.152–0.184) | – | 0.218 (0.191–0.240) | -22% ($-0.040; CI -0.097 to -0.007) | 0.983 | events cost more | – | 0.283 → 0.327 (-15%, events cost more) | 0 / – / 2 | 0 / – / 0 |
| Research API | Codex | subscription | A1 | 8/8 | 21.0 → 3.0 | 0.114 (0.105–0.120) | – | 0.076 (0.071–0.094) | 33% ($0.038; CI 0.013 to 0.053) | 0.002 | detected | – | 0.148 → 0.111 (25%, detected) | 4 / – / 0 | 4 / – / 0 |
The AI model runner's A2 → B comparison is confounded by job length: round 2's jobs ran a median of 210 s (Claude Code) and 268 s (Codex), against 139 to 174 s in round 1. The like-for-like comparison is the equalized arm:
| Server | Harness | E2 $ (n) | requests | Events $ | E2 → events | p | Verdict |
|---|---|---|---|---|---|---|---|
| AI model runner | Claude Code | 0.101 (8) | 7.0 | 0.063 | 38% | 0.009 | no detectable difference |
| AI model runner | Codex | 0.065 (8) | 8.5 | 0.046 | 28% | 0.016 | no detectable difference |
With matched descriptions, the model runner showed no detectable difference in either harness. For the other five servers the events branch differs from as shipped only by the wake passage appended to the tool descriptions, which is part of the event mechanism. Job lengths differ between arms in some cells (web scraping in Claude Code: 562 s as shipped against 431 s with events; the AI model runner in Codex: 268 s against 134 s), which favors events there. The per-cell tests are not corrected for multiple comparisons; two long-poll comparisons sit at p = 0.033 and 0.047.
Outside the conclusion's scope
The post's conclusion leaves out servers that already hold the request open and send progress, for tasks that reliably finish in under about five minutes: holding the call that way works in both harnesses, since Codex ends calls at 300 seconds. It has costs of its own: Claude Code is tied up for its first couple of minutes, and Codex narrates while it waits (see Holding the call). The rule was set on 2026-11-16, after every result here was known. Two Claude Code cells are out under it: AI video generation (as-shipped jobs 188 to 231 s) and the AI model runner (139 to 362 s; one of 8 jobs ran past five minutes, and it was left out by our decision). The research API in Claude Code also holds the call while streaming, but its as-shipped jobs ran 222 to 414 s, past five minutes in 4 of 8 sessions, so it fails "reliably under five minutes" and is in scope. The second crawling API in Claude Code is in scope: its held call sends no progress, its crawls take about six and a half minutes, and as shipped it never delivered the result (8 of 8). The Codex counterparts stay in scope: there the agent made 11 to 22 requests per wait, narrating while it waited. The short controls, whose jobs end within the tool's own wait, are below.
| Server | Harness | n (as shipped / events) | As shipped $ | Events $ | Difference | Verdict | Had to ask (as shipped / events) | Never told (as shipped / events) |
|---|---|---|---|---|---|---|---|---|
| AI video generation | Claude Code | 8 / 8 | 0.046 | 0.058 | events 27% more | detected | 0 / 0 | 0 / 0 |
| AI model runner | Claude Code | 8 / 9 | 0.053 | 0.063 | events 18% more | not detected (CI −0.011 to 0.079) | 0 / 0 | 0 / 0 |
Short controls (jobs of 20 to 45 seconds, fresh, subscription; medians from final.json; no user had to ask and none went untold in any of them):
| Server | Harness | Job (s) | n (as shipped / events) | As shipped $ | Events $ | Events cost more by |
|---|---|---|---|---|---|---|
| Web crawling | Claude Code | 45 | 6 / 6 | 0.084 | 0.088 | $0.004 |
| Web crawling | Codex | 45 | 6 / 6 | 0.033 | 0.043 | $0.010 |
| Web scraping | Claude Code | 45 | 5 / 5 | 0.032 | 0.033 | $0.001 |
| Web scraping | Codex | 45 | 5 / 5 | 0.015 | 0.042 | $0.028 |
| CI builds | Claude Code | 20 | 5 / 10 | 0.059 | 0.118 | $0.060 |
| CI builds | Codex | 20 | 5 / 10 | 0.017 | 0.040 | $0.023 |
| AI video generation | Claude Code | 30 | 5 / 5 | 0.013 | 0.013 | $0.001 |
| AI video generation | Codex | 30 | 5 / 5 | 0.005 | 0.037 | $0.033 |
| AI model runner | Claude Code | 45 | 5 / 5 | 0.019 | 0.061 | $0.042 |
| AI model runner | Codex | 45 | 5 / 5 | 0.006 | 0.042 | $0.036 |
The CI build short controls have 5 straddlers in each arm. An earlier summary gave "up to $0.04 more per job"; the medians above give up to $0.06.
Sessions excluded for infrastructure failures
The rule (our decision, 2026-11-16, after the results were known): a session is excluded when its failure was caused by infrastructure unrelated to the approach under test, in every arm and both studies. That covers Wakeshift's bridge or core outages, vendor-side outages, errors and timeouts, harness crashes, and bench or runner faults. Model behavior, the agent giving up, and an approach's own design limits (Codex ending a call at 300 seconds, Claude Code dropping a call that sends nothing) are not infrastructure and stay in.
How we looked. Study 1: every one of 736 sessions (all arms, contexts and sign-ins) was scanned for error markers in its steps, its tool results and its terminal log, and every session that ended with the user asking or never told (47) was traced to a cause. Study 2: every kept session's scores, toy-server log and transcript was scanned for tool, transport and harness errors, and every early stop was checked.
Study 1, removed (all four in the events arm, all during an outage of Wakeshift's pre-release server on 2026-11-12, about 20:36 to 20:58 UTC):
| Server | Harness | Session | Evidence |
|---|---|---|---|
| Research API | Claude Code | run 20261112-123332-3724, cc-b-300-2 | Server log: block_failed (APIError) on its agent_run call at 20:44:47 UTC, subscription rolled back; the tool's own text still promised a wake; no model request between the end of the first turn and the user's question |
| Research API | Claude Code | same run, cc-b-300-3 | Server log: block_failed at 20:53:57 UTC; same pattern |
| Crawling API that holds the call | Claude Code | run 20261112-120337-1e54, cc-b-480-4 | Job 20:45:30 to 20:51:47 UTC, inside the outage; no wake and no request until the user asked 120 s after the job ended, while the other 7 Claude Code events sessions were woken 1.3 to 4.1 s after it ended; the vendor's run notes record the outage during this run. The server log for this run wasn't retained, so the cause rests on the timing and the missing wake. |
| Email replies | Claude Code | run 20261112-124413-dff7, cc-b-180-1 | The vendor's run notes: Wakeshift's subscribe endpoint answered 500 and the SDK logged block_failed at 20:50:34 UTC, rolling back the subscription; the session was rerun later and the rerun was woken |
Study 1, examined and kept: the failures in the polling arms are the agent giving up while the job ran (web crawling in Claude Code), Claude Code dropping a held call that sent nothing (the crawling API that holds the call), Codex ending calls at 300 s (the crawling and research APIs in Codex; the log reads "timed out awaiting tools/call after 300s"), and the held-call test server's designed holds. One held-call test session logged an Anthropic API retry ("API error · Retrying in 1s"), which recovered without a failure. Sessions earlier left out for vendor rate limits and bench faults stay out, as already listed.
Study 2, removed: none. Examined and kept: a transient transport error in Codex's MCP client ("Transport send error … EventNotificationTransport") appeared once in each of six Codex sessions (three long-poll, three events) and recovered on the agent's next call. No reply was missed because of it. In the one session stopped early near such an error (long-poll, schedule 5: error at 7.5 minutes, runaway stop at 8.7), the agent's request rate was the same before the error (about 4 a minute) and the runaway rule fired on its own polling cadence. Toy-server "not found" errors (48 calls in 36 sessions) were the agent asking for IDs the test server had never listed; one, CHECKOUT-API-3F, appears as an example in the real error-alert tool's own text, which the test server copies. That is model behavior, not a fault in the test server. P1 to P5 are therefore unchanged.
With and without the excluded sessions (study 1, Claude Code; as-shipped arms unchanged):
| Server | As shipped $ (n) | Events, all sessions: $ (n), had to ask | Saved, all | Events, excluding infrastructure failures: $ (n), had to ask | Saved, excluding | Verdict, excluding |
|---|---|---|---|---|---|---|
| Research API | 0.179 (8) | 0.218 (8), 2 | 22% more | 0.204 (6), 0 | 14% more | events cost more (CI −0.081 to −0.002) |
| Crawling API that holds the call | 0.094 (8) | 0.097 (8), 1 | 3% more | 0.096 (7), 0 | 3% more | events cost more (CI −0.006 to −0.002) |
| Email replies | 0.203 (8) | 0.059 (9), 1 | 71% | 0.059 (8), 0 | 71% | detected |
In the conclusion's 14 cells, events: 0 of 124 had to ask excluding the three in-scope sessions, 3 of 127 counting them, and 10 of 134 had to ask and 6 were never told counting these and the 7 sessions the fixed-bug rule removed (deviation 14); as shipped: 23 of 111 had to ask and 18 were never told, unchanged by the rule.
Long-poll
| Server | Harness | As shipped | Long-poll (n = 8) | Events | Events vs long-poll |
|---|---|---|---|---|---|
| Web crawling | Claude Code | $0.318 (A1), 21 requests | $0.117, 6.5 requests | $0.078, 4 requests | 34%, detected |
| Web crawling | Codex | $0.078 (A1), 17 | $0.059, 10 | $0.036, 4 | 38%, detected |
| Web scraping | Codex | $0.140 (A1), 19 | $0.089, 15 | $0.045, 3 | 49%, detected |
| Web scraping | Claude Code | as shipped is already a 45 s long-poll: $0.331 (A2), 14.5 | – | $0.080, 4 | 76%, detected |
| AI video generation | Claude Code | $0.046 (A2), 4 | $0.047, 4 | $0.058, 4 | events cost more |
| AI video generation | Codex | $0.046 (A1), 11 | $0.071, 16 | $0.033, 3 | 55%, detected |
Long-poll against events crosses rounds and versions (round 2's arm L against round 1's arm B). Users asked in no long-poll session. Codex holds a silent long-poll for up to 280 s and narrates while it waits, so its long-poll can cost more than as shipped (AI video generation). The web scraping server's status tool already waits 45 s as shipped, so its Claude Code cell is the long-poll case; its agent still called the tool about every 47 seconds.
The API key: the cache check
Run on 2026-11-14 after the main results, at our request, and registered before its first session. Claude Code on an Anthropic API key (a 5-minute prompt cache), fresh, n = 8 per arm.
| Server | Harness | Sign-in | As shipped (arm) | n | Requests per wait | Wait $ as shipped | Long-poll $ (n) | Events $ (n) | Saved | p | Verdict | Long-poll → events | $ per conversation | Asked A / L / B | Never told A / L / B |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Web crawling | Claude Code | api-key | A2k | 8/8 | 6.5 → 5.0 | 0.099 (0.096–0.102) | – | 0.063 (0.063–0.068) | 36% ($0.036; CI 0.025 to 0.044) | 0.002 | detected | – | 0.150 → 0.119 (21%, detected) | 0 / – / 0 | 0 / – / 0 |
| Video editing | Claude Code | api-key | A2k | 8/8 | 18.0 → 5.0 | 0.165 (0.154–0.179) | – | 0.071 (0.069–0.072) | 57% ($0.094; CI 0.062 to 0.138) | 0.002 | detected | – | 0.223 → 0.129 (42%, detected) | 0 / – / 0 | 0 / – / 0 |
| Web scraping | Claude Code | api-key | A2k | 8/8 | 10.0 → 6.0 | 0.195 (0.138–0.216) | – | 0.090 (0.064–0.112) | 54% ($0.105; CI 0.029 to 0.148) | 0.002 | detected | – | 0.314 → 0.200 (36%, detected) | 0 / – / 0 | 0 / – / 0 |
| Server | Job s, A2k / B2k (median, range) | Subscription job s, A / B (median) | Events wakes later than 5 min after the last request | Wake: cache write / read tokens (B2k median) | Wake $ (B2k median) |
|---|---|---|---|---|---|
| Web crawling | 42 (34–107) / 36 (14–51) | 149 / 184 | 0 of 8 | 671 / 24,108 | 0.011 |
| Web scraping | 298 (250–321) / 263 (245–277) | 562 (A2) / 431 | 0 of 8 (gaps 241–274 s) | 773 / 31,599 | 0.013 |
| Video editing | 255 (181–298) / 232 (182–396) | 306 / 278 | 1 of 8 (392 s) | 4,414 / 24,818 | 0.032 |
Against the subscription (cross-round, cross-version, and the jobs ran shorter, so values only):
| Server | As shipped: subscription → API key | Events: subscription (B1) → API key (B2k) | Saved: subscription → API key |
|---|---|---|---|
| Web crawling | $0.318 (21 req) → $0.099 (6.5) | $0.078 → $0.063 | 76% → 36% |
| Web scraping | $0.331 (A2, 14.5) → $0.195 (10) | $0.080 → $0.090 (no detectable difference) | 76% → 54% |
| Video editing | $0.308 (23) → $0.165 (18) | $0.089 → $0.071 | 71% → 57% |
The saving is smaller than on the subscription because polling made fewer checks on shorter jobs; the events waits cost about the same. Only 1 of the 24 events waits sat idle past 5 minutes (a 396 s video job): its wake wrote 10,102 tokens to the cache against 780 to 4,470 in the other waits, and the wait cost $0.117 against the cell's median $0.071, still under every as-shipped wait in that cell ($0.121 to $0.219). Two of the web scraping events sessions polled anyway and are in the medians. Idle waits well past 5 minutes on an API key were not measured in this set.
Holding the call: the held-call experiment
A test server whose build_report tool holds the call for 1, 5 or 15 minutes, with or without a progress notification every 15 s, then returns a total the agent must tell. Partway through, the user asks an unrelated question ("what's 17 times 23?") to see whether the session stays usable. Pre-registered on 2026-11-13 before its runs; n = 3 per cell, 54 sessions; no events arm.
| Client | Sign-in | Hold | Progress | The call | Backgrounded | Told unasked / asked / never | Waiting $ median (range) | Requests | Interjection answered after (s) |
|---|---|---|---|---|---|---|---|---|---|
| Claude Code | subscription | 1 min | off | cut at 60 s (3/3) | 0 | 3 / 0 / 0 | 0.029 (0.028–0.029) | 3 | 31.0 |
| Claude Code | subscription | 1 min | on | held (3/3) | 0 | 3 / 0 / 0 | 0.020 (0.018–0.020) | 2 | 30.2 |
| Claude Code | subscription | 5 min | off | cut at 60 s (3/3) | 0 | 3 / 0 / 0 | 0.146 (0.139–0.178) | 15 | 1.7 |
| Claude Code | subscription | 5 min | on | held (3/3) | 3 | 3 / 0 / 0 | 0.055 (0.055–0.056) | 6 | 2.2 |
| Claude Code | subscription | 15 min | off | cut at 60 s (3/3) | 0 | 0 / 3 / 0 | 0.249 (0.092–0.304) | 25 | 1.7 |
| Claude Code | subscription | 15 min | on | held 15 min (3/3) | 3 | 3 / 0 / 0 | 0.056 (0.053–0.058) | 6 | 1.6 |
| Claude Code | API key | 1 min | off | cut at 60 s (3/3) | 0 | 3 / 0 / 0 | 0.022 (0.021–0.022) | 3 | 31.1 |
| Claude Code | API key | 1 min | on | held (3/3) | 0 | 3 / 0 / 0 | 0.014 (0.014–0.016) | 2 | 29.9 |
| Claude Code | API key | 5 min | off | cut at 60 s (3/3) | 0 | 1 / 2 / 0 | 0.122 (0.085–0.131) | 15 | 2.1 |
| Claude Code | API key | 5 min | on | held (3/3) | 3 | 3 / 0 / 0 | 0.051 (0.041–0.052) | 6 | 1.7 |
| Claude Code | API key | 15 min | off | cut at 60 s (3/3) | 0 | 0 / 3 / 0 | 0.186 (0.112–0.201) | 20 | 1.8 |
| Claude Code | API key | 15 min | on | held 15 min (3/3) | 3 | 3 / 0 / 0 | 0.076 (0.070–0.078) | 6 | 1.7 |
| Codex | subscription | 1 min | off | held (3/3) | – | 3 / 0 / 0 | 0.024 (0.020–0.024) | 7 | 3.7 |
| Codex | subscription | 1 min | on | held (3/3) | – | 3 / 0 / 0 | 0.021 (0.014–0.052) | 6 | 3.5 |
| Codex | subscription | 5 min | off | held to 300 s, its limit¹ (3/3) | – | 3 / 0 / 0 | 0.102 (0.060–0.124) | 19 | 3.7 |
| Codex | subscription | 5 min | on | the same¹ (3/3) | – | 3 / 0 / 0 | 0.106 (0.087–0.124) | 20 | 2.7 |
| Codex | subscription | 15 min | off | cut at 300 s (3/3) | – | 2 / 1 / 0 | 0.214 (0.178–0.296) | 36 | 3.1 |
| Codex | subscription | 15 min | on | cut at 300 s (3/3) | – | 1 / 2 / 0 | 0.232 (0.199–0.258) | 39 | 2.5 |
¹ The server returned at 300.00 s, racing Codex's limit; the agents confirmed with a status call within 4 s.
Long holds (registered before their runs; Claude Code on a subscription, progress on, n = 3 each, all nine run at once on separate servers):
| Hold | The call | Progress sent | Backgrounded at | Told unasked / asked / never | Told after the end (s) | Waiting $ median (range) | Wait requests | Wait tokens (median) | Interjection answered after (s) | Timeout messages |
|---|---|---|---|---|---|---|---|---|---|---|
| 15 min (sidebar, above) | held (3/3) | 59 | ~2 min (3/3) | 3 / 0 / 0 | <1 | 0.056 (0.053–0.058) | 6 | – | 1.6 | – |
| 30 min | held 1800 s (3/3) | 119 | 121–123 s (3/3) | 3 / 0 / 0 | 1.6–2.3 | 0.045 (0.045–0.055) | 5–6 | 161k | 1.7–2.0 | none |
| 60 min | held 3600 s (3/3) | 239 | 121–122 s (3/3) | 3 / 0 / 0 | 1.5–2.4 | 0.046 (0.045–0.057) | 5–7 | 161k | 1.4–1.9 | none |
| 120 min | held 7200 s (3/3) | 479 | 122 s (3/3) | 3 / 0 / 0 | 1.5–2.4 | 0.108 (0.108–0.109) | 5 | 161k | 1.7–1.9 | none |
No call was cut, and no timeout message appeared, up to 2 hours. At 120 minutes the subscription's one-hour cache had expired, so the resume rewrote about 8.3k tokens. On a real crawling server that held its stream open but sent nothing, Claude Code backgrounded the call at about 2 minutes and dropped it at about 5 minutes of silence.
Job length (exploratory)
The planned run at about 20-minute jobs did not happen: neither vendor could produce a legitimate 20-minute job at small vendor spend. In its place, within each fresh cell with n ≥ 8, the Spearman correlation of requests per wait with job length: polling arms rise with job length in 16 of 17 cells where the agent checks a status tool and the jobs varied (+0.43 to +0.99; the exception is web crawling in Claude Code, −0.44). Events stay at 3 to 5.5 requests across cells whose median jobs run 134 to 431 s. Nothing past about 11 minutes was measured.
200k tokens of earlier context (n = 2, values only)
The AI video generation and AI model runner Claude Code as-shipped cells at 200k were measured through round 1's buffering proxy and are not a fair baseline.
| Server | Harness | Sign-in | As shipped (arm) | n | Requests per wait | Wait $ as shipped | Long-poll $ (n) | Events $ (n) | Saved | p | Verdict | Long-poll → events | $ per conversation | Asked A / L / B | Never told A / L / B |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Web crawling | Claude Code | subscription | A1 | 2/2 | 17.0 → 4.0 | 0.826 (0.806–0.847) [0.7856 0.8670] | – | 0.206 (0.206–0.206) [0.2057 0.2063] | 75% ($0.620; CI – to –) | – | n < 5: values only | – | 2.316 → 1.699 (27%, n<5) | 1 / – / 0 | 0 / – / 0 |
| Web crawling | Codex | subscription | A1 | 2/2 | 25.5 → 4.0 | 0.527 (0.419–0.635) [0.3111 0.7434] | – | 0.105 (0.105–0.105) [0.1053 0.1054] | 80% ($0.422; CI – to –) | – | n < 5: values only | – | 1.002 → 0.586 (42%, n<5) | 0 / – / 0 | 0 / – / 0 |
| Video editing | Claude Code | subscription | A1 | 2/2 | 23.5 → 4.0 | 1.136 (1.129–1.143) [1.1222 1.1496] | – | 0.223 (0.223–0.224) [0.2230 0.2238] | 80% ($0.912; CI – to –) | – | n < 5: values only | – | 2.638 → 1.722 (35%, n<5) | 0 / – / 0 | 0 / – / 0 |
| Video editing | Codex | subscription | A1 | 2/2 | 16.0 → 3.0 | 0.349 (0.327–0.370) [0.3055 0.3916] | – | 0.090 (0.090–0.090) [0.0900 0.0901] | 74% ($0.259; CI – to –) | – | n < 5: values only | – | 0.822 → 0.568 (31%, n<5) | 0 / – / 0 | 0 / – / 0 |
| Web scraping | Claude Code | subscription | A1 | 2/2 | 11.0 → 4.0 | 0.652 (0.652–0.653) [0.6511 0.6536] | – | 0.210 (0.210–0.210) [0.2100 0.2105] | 68% ($0.442; CI – to –) | – | n < 5: values only | – | 2.283 → 1.844 (19%, n<5) | 0 / – / 0 | 0 / – / 0 |
| Web scraping | Codex | subscription | A1 | 2/2 | 16.0 → 3.0 | 0.384 (0.381–0.387) [0.3780 0.3899] | – | 0.095 (0.094–0.096) [0.0932 0.0972] | 75% ($0.289; CI – to –) | – | n < 5: values only | – | 0.957 → 0.674 (30%, n<5) | 0 / – / 0 | 0 / – / 0 |
| Agent communication (email reply) | Claude Code | subscription | A1 | 2/2 | 14.0 → 4.0 | 0.637 (0.594–0.679) [0.5520 0.7219] | – | 0.195 (0.195–0.195) [0.1945 0.1947] | 69% ($0.442; CI – to –) | – | n < 5: values only | – | 2.148 → 1.709 (20%, n<5) | 0 / – / 0 | 0 / – / 0 |
| Agent communication (email reply) | Codex | subscription | A1 | 2/4 | 13.0 → 4.5 | 0.282 (0.273–0.291) [0.2636 0.3001] | – | 0.129 (0.116–0.261) [0.1161 0.1164 0.1409 0.6213] | 54% ($0.153; CI – to –) | – | n < 5: values only | – | 0.799 → 0.632 (21%, n<5) | 0 / – / 0 | 0 / – / 0 |
| CI builds | Claude Code | subscription | A1 | 2/2 | 7.5 → 4.0 | 0.411 (0.388–0.433) [0.3651 0.4562] | – | 0.226 (0.223–0.230) [0.2198 0.2329] | 45% ($0.184; CI – to –) | – | n < 5: values only | – | 2.195 → 1.909 (13%, n<5) | 0 / – / 0 | 0 / – / 0 |
| CI builds | Codex | subscription | A1 | 2/2 | 12.5 → 3.0 | 0.270 (0.265–0.275) [0.2593 0.2808] | – | 0.088 (0.088–0.088) [0.0880 0.0880] | 67% ($0.182; CI – to –) | – | n < 5: values only | – | 0.789 → 0.612 (22%, n<5) | 0 / – / 0 | 0 / – / 0 |
| AI video generation | Claude Code | subscription | A1 ⚠ buffered baseline | 2/2 | 7.0 → 4.0 | 0.328 (0.306–0.349) [0.2849 0.3706] | – | 0.190 (0.189–0.191) [0.1888 0.1910] | 42% ($0.138; CI – to –) | – | n < 5: values only | – | 1.862 → 1.737 (7%, n<5) | 0 / – / 0 | 0 / – / 0 |
| AI video generation | Codex | subscription | A1 | 2/2 | 11.5 → 3.0 | 0.240 (0.228–0.253) [0.2149 0.2660] | – | 0.083 (0.083–0.084) [0.0821 0.0842] | 65% ($0.157; CI – to –) | – | n < 5: values only | – | 0.750 → 0.609 (19%, n<5) | 0 / – / 0 | 0 / – / 0 |
| AI model runner | Claude Code | subscription | A1 ⚠ buffered baseline | 2/2 | 4.5 → 4.0 | 0.247 (0.220–0.275) [0.1931 0.3016] | – | 0.195 (0.194–0.195) [0.1939 0.1953] | 21% ($0.053; CI – to –) | – | n < 5: values only | – | 1.833 → 1.781 (3%, n<5) | 0 / – / 0 | 0 / – / 0 |
| AI model runner | Codex | subscription | A1 | 2/2 | 6.0 → 4.0 | 0.136 (0.135–0.136) [0.1353 0.1359] | – | 0.119 (0.118–0.120) [0.1180 0.1201] | 12% ($0.017; CI – to –) | – | n < 5: values only | – | 0.727 → 0.702 (3%, n<5) | 0 / – / 0 | 0 / – / 0 |
Every cell
Sessions in the published data, per cell (wait in dollars with IQR; "Straddlers" are sessions that crossed a change of the Wakeshift version under test, kept and flagged).
| Server | Harness | Job | Context | Arm | n | Wait $ (IQR) | Requests | $ per conversation | Asked | Never told | Tell (s) | Job median (s) | Straddlers |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Web crawling | Claude Code | 600 | 200k | A1 | 2 | 0.8263 (0.8059–0.8467) | 17.0 | 2.3164 | 1 | 0 | 20.985 | 150.984 | 1 |
| Web crawling | Claude Code | 600 | 200k | B1 | 2 | 0.206 (0.2058–0.2061) | 4.0 | 1.6991 | 0 | 0 | 6.0735 | 149.393 | 0 |
| Web crawling | Claude Code | 600 | fresh | A1 | 8 | 0.318 (0.302–0.3416) | 21.0 | 0.418 | 4 | 0 | 39.034 | 148.6445 | 3 |
| Web crawling | Claude Code | 600 | fresh | A2k | 8 | 0.099 (0.0964–0.1016) | 6.5 | 0.1503 | 0 | 0 | 26.623 | 42.1195 | 0 |
| Web crawling | Claude Code | 600 | fresh | B1 | 8 | 0.0777 (0.0762–0.0926) | 4.0 | 0.1876 | 0 | 0 | 6.7335 | 183.959 | 6 |
| Web crawling | Claude Code | 600 | fresh | B2k | 8 | 0.0629 (0.0626–0.068) | 5.0 | 0.1187 | 0 | 0 | 12.0935 | 36.3075 | 0 |
| Web crawling | Claude Code | 600 | fresh | L2 | 8 | 0.1174 (0.1063–0.1322) | 6.5 | 0.2116 | 0 | 0 | 23.8475 | 129.0075 | 2 |
| Web crawling | Claude Code | 45 | fresh | A1 | 6 | 0.0842 (0.063–0.1393) | 3.0 | 0.1824 | 0 | 0 | 2.763 | 10.4295 | 0 |
| Web crawling | Claude Code | 45 | fresh | B1 | 6 | 0.0878 (0.0861–0.089) | 5.0 | 0.1902 | 0 | 0 | 6.908 | 16.793 | 0 |
| Web crawling | Codex | 600 | 200k | A1 | 2 | 0.5273 (0.4192–0.6353) | 25.5 | 1.0024 | 0 | 0 | 17.6745 | 196.758 | 0 |
| Web crawling | Codex | 600 | 200k | B1 | 2 | 0.1053 (0.1053–0.1053) | 4.0 | 0.5859 | 0 | 0 | 11.079 | 150.9745 | 0 |
| Web crawling | Codex | 600 | fresh | A1 | 8 | 0.0777 (0.0732–0.1042) | 17.0 | 0.1039 | 0 | 0 | 10.812 | 127.751 | 3 |
| Web crawling | Codex | 600 | fresh | B1 | 8 | 0.0365 (0.0363–0.0418) | 4.0 | 0.0721 | 0 | 0 | 9.923 | 148.028 | 3 |
| Web crawling | Codex | 600 | fresh | L2 | 8 | 0.0588 (0.0532–0.0742) | 10.0 | 0.0945 | 0 | 0 | 25.3325 | 121.111 | 2 |
| Web crawling | Codex | 45 | fresh | A1 | 6 | 0.0334 (0.0302–0.0476) | 1.0 | 0.0607 | 0 | 0 | 2.794 | 15.953 | 0 |
| Web crawling | Codex | 45 | fresh | B1 | 6 | 0.0429 (0.0429–0.0429) | 4.0 | 0.069 | 0 | 0 | 9.8595 | 34.1205 | 0 |
| Video editing | Claude Code | 600 | 200k | A1 | 2 | 1.1359 (1.129–1.1427) | 23.5 | 2.6382 | 0 | 0 | 47.469 | 287.4465 | 0 |
| Video editing | Claude Code | 600 | 200k | B1 | 2 | 0.2234 (0.2232–0.2236) | 4.0 | 1.7216 | 0 | 0 | 3.724 | 217.6245 | 2 |
| Video editing | Claude Code | 600 | fresh | A1 | 8 | 0.3079 (0.2929–0.3368) | 23.0 | 0.4189 | 0 | 0 | 42.624 | 306.2345 | 2 |
| Video editing | Claude Code | 600 | fresh | A2k | 8 | 0.1654 (0.1544–0.1792) | 18.0 | 0.2225 | 0 | 0 | 31.954 | 255.3445 | 1 |
| Video editing | Claude Code | 600 | fresh | B1 | 10 | 0.0893 (0.089–0.0908) | 4.0 | 0.1971 | 0 | 0 | 3.6085 | 277.572 | 2 |
| Video editing | Claude Code | 600 | fresh | B2k | 8 | 0.0711 (0.0689–0.0723) | 5.0 | 0.1293 | 0 | 0 | 3.157 | 231.9675 | 0 |
| Video editing | Codex | 600 | 200k | A1 | 2 | 0.3486 (0.327–0.3701) | 16.0 | 0.8218 | 0 | 0 | 22.7205 | 248.341 | 0 |
| Video editing | Codex | 600 | 200k | B1 | 2 | 0.09 (0.09–0.0901) | 3.0 | 0.5684 | 0 | 0 | 5.2005 | 276.002 | 2 |
| Video editing | Codex | 600 | fresh | A1 | 8 | 0.1018 (0.0866–0.1107) | 17.0 | 0.1249 | 0 | 0 | 17.474 | 269.4835 | 4 |
| Video editing | Codex | 600 | fresh | B1 | 10 | 0.0381 (0.038–0.056) | 3.0 | 0.0662 | 0 | 0 | 4.6225 | 238.388 | 4 |
| Web scraping | Claude Code | 600 | 200k | A1 | 2 | 0.6524 (0.6518–0.653) | 11.0 | 2.2829 | 0 | 0 | 10.758 | 364.441 | 0 |
| Web scraping | Claude Code | 600 | 200k | B1 | 2 | 0.2103 (0.2101–0.2104) | 4.0 | 1.844 | 0 | 0 | 6.853 | 396.3315 | 0 |
| Web scraping | Claude Code | 600 | fresh | A1 | 8 | 0.2803 (0.2304–0.3108) | 12.0 | 0.4874 | 0 | 0 | 9.9525 | 400.3745 | 4 |
| Web scraping | Claude Code | 600 | fresh | A2 | 8 | 0.3308 (0.3066–0.3744) | 14.5 | 0.522 | 0 | 0 | 12.9965 | 561.7045 | 0 |
| Web scraping | Claude Code | 600 | fresh | A2k | 8 | 0.1951 (0.1382–0.2164) | 10.0 | 0.3142 | 0 | 0 | 13.5725 | 297.5625 | 0 |
| Web scraping | Claude Code | 600 | fresh | B1 | 9 | 0.0802 (0.0768–0.1639) | 4 | 0.2888 | 0 | 0 | 7.099 | 430.69 | 4 |
| Web scraping | Claude Code | 600 | fresh | B2k | 8 | 0.0896 (0.0644–0.1121) | 6.0 | 0.1996 | 0 | 0 | 10.0405 | 263.409 | 0 |
| Web scraping | Claude Code | 45 | fresh | A1 | 5 | 0.0324 (0.0322–0.0329) | 2 | 0.1569 | 0 | 0 | 3.352 | 10.95 | 0 |
| Web scraping | Claude Code | 45 | fresh | B1 | 5 | 0.0334 (0.0333–0.0341) | 2 | 0.1629 | 0 | 0 | 3.159 | 11.263 | 0 |
| Web scraping | Codex | 600 | 200k | A1 | 2 | 0.384 (0.381–0.3869) | 16.0 | 0.9566 | 0 | 0 | 20.3225 | 323.0445 | 0 |
| Web scraping | Codex | 600 | 200k | B1 | 2 | 0.0952 (0.0942–0.0962) | 3.0 | 0.6744 | 0 | 0 | 10.2365 | 510.466 | 0 |
| Web scraping | Codex | 600 | fresh | A1 | 8 | 0.1397 (0.116–0.1536) | 19.0 | 0.2287 | 0 | 0 | 17.728 | 424.1195 | 3 |
| Web scraping | Codex | 600 | fresh | B1 | 9 | 0.0454 (0.0412–0.0462) | 3 | 0.1455 | 0 | 0 | 11.798 | 407.045 | 5 |
| Web scraping | Codex | 600 | fresh | L2 | 8 | 0.0891 (0.0845–0.0925) | 15.0 | 0.1827 | 0 | 0 | 12.956 | 288.946 | 2 |
| Web scraping | Codex | 45 | fresh | A1 | 5 | 0.0146 (0.0145–0.0147) | 2 | 0.1064 | 0 | 0 | 3.645 | 10.46 | 0 |
| Web scraping | Codex | 45 | fresh | B1 | 5 | 0.0423 (0.0422–0.0425) | 3 | 0.1351 | 0 | 0 | 7.998 | 8.445 | 0 |
| Agent communication | Claude Code | 180 | 200k | A1 | 2 | 0.6369 (0.5944–0.6794) | 14.0 | 2.1481 | 0 | 0 | 59.6415 | 199.0105 | 0 |
| Agent communication | Claude Code | 180 | 200k | B1 | 2 | 0.1946 (0.1946–0.1946) | 4.0 | 1.7091 | 0 | 0 | 4.5335 | 184.3805 | 1 |
| Agent communication | Claude Code | 180 | fresh | A1 | 8 | 0.2026 (0.1938–0.2055) | 15.0 | 0.3229 | 0 | 0 | 105.799 | 184.096 | 2 |
| Agent communication | Claude Code | 180 | fresh | B1 | 9 | 0.0594 (0.059–0.0596) | 4 | 0.1757 | 1 | 0 | 4.8895 | 183.763 | 3 |
| Agent communication | Codex | 180 | 200k | A1 | 2 | 0.2819 (0.2727–0.291) | 13.0 | 0.7994 | 0 | 0 | 16.269 | 189.1995 | 0 |
| Agent communication | Codex | 180 | 200k | B1 | 4 | 0.1287 (0.1163–0.261) | 4.5 | 0.6323 | 0 | 0 | 9.9255 | 184.623 | 2 |
| Agent communication | Codex | 180 | fresh | A1 | 8 | 0.0849 (0.0559–0.1038) | 12.5 | 0.136 | 0 | 0 | 21.1345 | 184.0235 | 4 |
| Agent communication | Codex | 180 | fresh | B1 | 8 | 0.0382 (0.0342–0.0421) | 3.5 | 0.0945 | 0 | 0 | 7.7995 | 183.971 | 1 |
| CI builds | Claude Code | 240 | 200k | A1 | 2 | 0.4106 (0.3878–0.4334) | 7.5 | 2.1946 | 0 | 0 | 3.1595 | 242.5695 | 0 |
| CI builds | Claude Code | 240 | 200k | B1 | 2 | 0.2263 (0.2231–0.2296) | 4.0 | 1.909 | 0 | 0 | 6.5455 | 244.7425 | 0 |
| CI builds | Claude Code | 240 | fresh | A1 | 8 | 0.1387 (0.1345–0.1479) | 7.0 | 0.3523 | 0 | 0 | 4.7705 | 243.718 | 0 |
| CI builds | Claude Code | 240 | fresh | B1 | 11 | 0.0974 (0.0927–0.1199) | 4 | 0.3467 | 0 | 0 | 7.469 | 243.471 | 3 |
| CI builds | Claude Code | 20 | fresh | A1 | 5 | 0.0587 (0.0567–0.0629) | 2 | 0.3622 | 0 | 0 | 3.894 | 23.05 | 5 |
| CI builds | Claude Code | 20 | fresh | B1 | 10 | 0.1184 (0.0936–0.1427) | 5.0 | 0.3647 | 0 | 0 | 5.778 | 22.8385 | 5 |
| CI builds | Codex | 240 | 200k | A1 | 2 | 0.27 (0.2646–0.2754) | 12.5 | 0.7886 | 0 | 0 | 4.822 | 243.4605 | 0 |
| CI builds | Codex | 240 | 200k | B1 | 2 | 0.088 (0.088–0.088) | 3.0 | 0.6116 | 0 | 0 | 8.96 | 244.7945 | 0 |
| CI builds | Codex | 240 | fresh | A1 | 8 | 0.0881 (0.0602–0.1011) | 13.0 | 0.1309 | 0 | 0 | 5.26 | 244.112 | 1 |
| CI builds | Codex | 240 | fresh | B1 | 8 | 0.036 (0.0359–0.0365) | 3.0 | 0.0907 | 0 | 0 | 8.0125 | 243.289 | 1 |
| CI builds | Codex | 20 | fresh | A1 | 5 | 0.0168 (0.0152–0.0199) | 2 | 0.0839 | 0 | 0 | 4.686 | 22.289 | 5 |
| CI builds | Codex | 20 | fresh | B1 | 10 | 0.0398 (0.0361–0.0646) | 3.0 | 0.1092 | 0 | 0 | 7.7935 | 23.877 | 5 |
| AI video generation | Claude Code | 300 | 200k | A1 | 2 | 0.3277 (0.3063–0.3491) | 7.0 | 1.8618 | 0 | 0 | 3.211 | 198.1675 | 1 |
| AI video generation | Claude Code | 300 | 200k | B1 | 2 | 0.1899 (0.1894–0.1905) | 4.0 | 1.7369 | 0 | 0 | 5.784 | 208.0905 | 0 |
| AI video generation | Claude Code | 300 | fresh | A1 | 8 | 0.1001 (0.0857–0.1059) | 7.5 | 0.2107 | 0 | 0 | 3.024 | 198.0165 | 2 |
| AI video generation | Claude Code | 300 | fresh | A2 | 8 | 0.0456 (0.0454–0.0458) | 4.0 | 0.1456 | 0 | 0 | 1.3945 | 210.632 | 0 |
| AI video generation | Claude Code | 300 | fresh | B1 | 8 | 0.0577 (0.0571–0.0581) | 4.0 | 0.1804 | 0 | 0 | 5.2065 | 194.559 | 3 |
| AI video generation | Claude Code | 300 | fresh | L2 | 8 | 0.0469 (0.0466–0.0476) | 4.0 | 0.1471 | 0 | 0 | 1.7845 | 209.2645 | 4 |
| AI video generation | Claude Code | 30 | fresh | A1 | 5 | 0.0127 (0.0127–0.0129) | 1 | 0.1129 | 0 | 0 | 1.055 | 15.023 | 0 |
| AI video generation | Claude Code | 30 | fresh | B1 | 5 | 0.0133 (0.0132–0.0133) | 1 | 0.1178 | 0 | 0 | 2.027 | 14.515 | 0 |
| AI video generation | Codex | 300 | 200k | A1 | 2 | 0.2405 (0.2277–0.2532) | 11.5 | 0.7495 | 0 | 0 | 3.6045 | 200.9415 | 1 |
| AI video generation | Codex | 300 | 200k | B1 | 2 | 0.0832 (0.0827–0.0837) | 3.0 | 0.6086 | 0 | 0 | 7.1225 | 201.009 | 1 |
| AI video generation | Codex | 300 | fresh | A1 | 8 | 0.0461 (0.045–0.0595) | 11.0 | 0.0964 | 0 | 0 | 2.5155 | 214.8295 | 6 |
| AI video generation | Codex | 300 | fresh | B1 | 12 | 0.0325 (0.0322–0.0332) | 3.0 | 0.0828 | 0 | 0 | 7.6065 | 216.412 | 7 |
| AI video generation | Codex | 300 | fresh | L2 | 8 | 0.0715 (0.064–0.0817) | 16.0 | 0.1117 | 0 | 0 | 3.8725 | 218.556 | 3 |
| AI video generation | Codex | 30 | fresh | A1 | 5 | 0.0045 (0.0043–0.0048) | 1 | 0.0451 | 0 | 0 | 2.633 | 18.993 | 0 |
| AI video generation | Codex | 30 | fresh | B1 | 5 | 0.0371 (0.0371–0.0372) | 4 | 0.0815 | 0 | 0 | 9.178 | 20.721 | 0 |
| AI model runner | Claude Code | 600 | 200k | A1 | 2 | 0.2474 (0.2202–0.2745) | 4.5 | 1.8329 | 0 | 0 | 4.317 | 144.066 | 0 |
| AI model runner | Claude Code | 600 | 200k | B1 | 2 | 0.1946 (0.1943–0.1949) | 4.0 | 1.7813 | 0 | 0 | 6.393 | 127.339 | 0 |
| AI model runner | Claude Code | 600 | fresh | A1 | 8 | 0.0853 (0.0846–0.1002) | 5.0 | 0.2385 | 0 | 0 | 2.8125 | 153.8425 | 2 |
| AI model runner | Claude Code | 600 | fresh | A2 | 8 | 0.0535 (0.0532–0.1334) | 4.0 | 0.1964 | 0 | 0 | 2.859 | 209.966 | 2 |
| AI model runner | Claude Code | 600 | fresh | B1 | 9 | 0.0631 (0.0616–0.0642) | 4 | 0.2282 | 0 | 0 | 5.189 | 139.226 | 2 |
| AI model runner | Claude Code | 600 | fresh | E2 | 8 | 0.1012 (0.0538–0.161) | 7.0 | 0.2427 | 0 | 0 | 2.7075 | 292.044 | 1 |
| AI model runner | Claude Code | 45 | fresh | A1 | 5 | 0.0191 (0.0183–0.0191) | 1 | 0.1599 | 0 | 0 | 3.25 | 12.571 | 0 |
| AI model runner | Claude Code | 45 | fresh | B1 | 5 | 0.061 (0.0605–0.0628) | 4 | 0.2169 | 0 | 0 | 6.898 | 16.999 | 0 |
| AI model runner | Codex | 600 | 200k | A1 | 2 | 0.1356 (0.1354–0.1358) | 6.0 | 0.7265 | 0 | 0 | 3.809 | 120.252 | 1 |
| AI model runner | Codex | 600 | 200k | B1 | 2 | 0.1191 (0.1185–0.1196) | 4.0 | 0.7018 | 0 | 0 | 10.9395 | 200.667 | 1 |
| AI model runner | Codex | 600 | fresh | A1 | 7 | 0.0443 (0.0417–0.05) | 8 | 0.1353 | 0 | 0 | 5.897 | 173.927 | 2 |
| AI model runner | Codex | 600 | fresh | A2 | 8 | 0.1155 (0.0944–0.1337) | 13.0 | 0.2149 | 0 | 0 | 6.5695 | 268.177 | 1 |
| AI model runner | Codex | 600 | fresh | B1 | 11 | 0.0464 (0.0449–0.0518) | 4 | 0.1355 | 0 | 0 | 10.676 | 134.435 | 2 |
| AI model runner | Codex | 600 | fresh | E2 | 8 | 0.0647 (0.0359–0.1059) | 8.5 | 0.1731 | 0 | 0 | 5.041 | 163.5015 | 0 |
| AI model runner | Codex | 45 | fresh | A1 | 5 | 0.0064 (0.0061–0.0073) | 1 | 0.0923 | 0 | 0 | 3.609 | 16.547 | 0 |
| AI model runner | Codex | 45 | fresh | B1 | 5 | 0.042 (0.0399–0.0449) | 3 | 0.1326 | 0 | 0 | 7.81 | 17.537 | 0 |
| Crawling API that holds the call | Claude Code | 480 | fresh | A1 | 8 | 0.0474 (0.0469–0.0488) | 3.0 | 0.166 | 8 | 8 | – | 403.248 | 4 |
| Crawling API that holds the call | Claude Code | 480 | fresh | A2 | 8 | 0.094 (0.0918–0.0941) | 6.0 | 0.1999 | 8 | 8 | – | 385.422 | 0 |
| Crawling API that holds the call | Claude Code | 480 | fresh | B1 | 8 | 0.0965 (0.0957–0.0972) | 5.0 | 0.2185 | 1 | 0 | 12.125 | 389.1885 | 3 |
| Crawling API that holds the call | Codex | 480 | fresh | A1 | 7 | 0.1407 (0.124–0.1711) | 22 | 0.1874 | 7 | 6 | – | 401.998 | 2 |
| Crawling API that holds the call | Codex | 480 | fresh | B1 | 7 | 0.0317 (0.0314–0.0318) | 3 | 0.0828 | 0 | 0 | 10.773 | 409.173 | 3 |
| Research API | Claude Code | 300 | fresh | A1 | 8 | 0.1789 (0.1516–0.1841) | 4.5 | 0.2831 | 0 | 0 | 27.709 | 297.4005 | 3 |
| Research API | Claude Code | 300 | fresh | B1 | 8 | 0.2184 (0.1911–0.2404) | 5.5 | 0.3268 | 2 | 0 | 37.305 | 282.785 | 3 |
| Research API | Codex | 300 | fresh | A1 | 8 | 0.114 (0.1047–0.1197) | 21.0 | 0.1477 | 4 | 4 | 43.6975 | 249.767 | 3 |
| Research API | Codex | 300 | fresh | B1 | 8 | 0.0763 (0.0709–0.0939) | 3.0 | 0.111 | 0 | 0 | 37.384 | 274.7615 | 2 |
| Held-call test server | Claude Code | 900 | fresh | A2 | 6 | 0.075 (0.0561–0.2101) | 8.0 | 0.1549 | 3 | 0 | 0.054 | 900.9145 | 0 |
| Held-call test server | Claude Code | 900 | fresh | A2k | 6 | 0.0951 (0.0767–0.1674) | 8.5 | 0.1416 | 3 | 0 | 0.451 | 901.2265 | 3 |
| Held-call test server | Claude Code | 300 | fresh | A2 | 6 | 0.0971 (0.0551–0.1442) | 10.0 | 0.1733 | 0 | 0 | 38.0335 | 300.5165 | 0 |
| Held-call test server | Claude Code | 300 | fresh | A2k | 6 | 0.0688 (0.0512–0.1131) | 9.0 | 0.1128 | 2 | 0 | 0.9975 | 300.148 | 0 |
| Held-call test server | Claude Code | 60 | fresh | A2 | 6 | 0.0239 (0.0195–0.0284) | 2.5 | 0.1005 | 0 | 0 | 0.1765 | 60.3955 | 0 |
| Held-call test server | Claude Code | 60 | fresh | A2k | 6 | 0.0183 (0.0142–0.0215) | 2.5 | 0.0627 | 0 | 0 | 0.379 | 60.7845 | 3 |
| Held-call test server | Codex | 900 | fresh | A2 | 6 | 0.223 (0.2031–0.2512) | 37.5 | 0.264 | 3 | 0 | 24.928 | 904.3515 | 0 |
| Held-call test server | Codex | 300 | fresh | A2 | 6 | 0.104 (0.0909–0.1195) | 19.5 | 0.1407 | 0 | 0 | 4.0225 | 300.0045 | 0 |
| Held-call test server | Codex | 60 | fresh | A2 | 6 | 0.0223 (0.0204–0.0241) | 6.0 | 0.0599 | 0 | 0 | 1.592 | 60.005 | 0 |
Study 2: External events
Design
Pre-registered as design v3.1, commit beb1b5c (tag prereg-v1), 2026-11-14 05:47 UTC, before any session.
Test servers. Each copies one real server's tool names, titles, descriptions, input schemas, annotations and server instructions word for word: an inbound email service, a code review service and an error alert service. Responses come from real captures (email) or the API documentation's examples (the other two). Output schemas and icons are not served; neither harness is known to put them in the model's context. The tool lists are checked word for word against the captures (1,047 checks). One server's license keeps its text private, so for it we publish the extraction script and content hashes.
Tasks. The blocked task (T1): reply to customers as they write back, over a 2-hour window, with releases on 10 schedules drawn in advance from a fixed seed, none in the last 20 minutes; at the end the user asks once whether anything came in. Two schedules drew no replies (known before any session), so 6 schedules carry the 45 replies each approach had to answer. The three-stream task (T3): reviews on three pull requests, an inbox and an error feed for the same two hours, exploratory.
Approaches (arms). G0, the server as shipped, silent about waiting. G3, a long-poll read tool holding each call up to 50 s (Claude Code) or 280 s (Codex) while sending progress. S, the user's 15-minute check: /loop 15m in Claude Code; in Codex, which has no scheduler, a plain request to check every 15 minutes. G4, a wait tool held open until the next event while sending progress every 15 s (Claude Code only; Codex cuts every call at 300 s). G5, events through Wakeshift (plugin v1.4.54; Codex connector 2026.11.14.1515). The screen also ran G1 ("poll every 5 to 10 seconds", from another product's server), G2 ("check back in about 10 minutes") and GB (a wording written by an agent blind to the study).
Measures. A reply is missed when the agent handled it only after the user asked, or never. Idle cost is model spend per hour outside acting spans, where an acting span runs from an item's first fetch to the earlier of the turn's end and the agent's next check after acting on it. Every approach ran on every schedule in the same wave.
Primary tests. P1 to P5 compare each approach with events on the same schedules: an exact one-sided sign-flip test on the sum of paired differences (P5 two-sided), Holm-corrected over the five. A test detects a difference only when its Holm-adjusted rejection and a 95% interval excluding zero agree. Everything else is exploratory.
The bar (falsifiers). F1 (as shipped, long-poll): at most $0.10 and 10 requests an idle hour more than events, and at most a fifth of replies missed. F2 (held call): at most $0.05 an hour more than events, and at most 5 points more missed. F3 (the 15-minute check): F1's cost lines, at most a tenth missed and 90% handled within 15 minutes. F4 (events don't deliver): events miss more than a tenth of replies in any harness and task, or cost more idle money than as shipped in 2 or more of the 4 harness and task cells. F5: the test server must match the real email server within ±30% on cost per check and checks per hour. An approach that clears its bar falsifies the claim that agents need events for this kind of wait.
Runaway rule and caps. Each session has a cost cap. The runaway rule stops a session looping at 10 s or less (the median gap of its last 10 requests) whose trailing 10-minute spend rate is above $4 an hour (Claude Code) or $2 (Codex). Replies after a stop are unobserved and excluded.
Stages. A screen of 104 one-hour sessions on a pre-release Wakeshift version, varying the wording; the main stage of 180 kept sessions on the released version; and the real-server check (Part C) of 16 pairs. Total spend $327.57 at list price, the API-key run included.
The combined bar (blocked task)
Missed share is the median over the 6 schedules with replies; extras are median paired differences against G5. Absolute medians (idle $/h, idle requests/h): Claude Code G0 0.69 / 32.9, G3 0.07 / 5.2, S 0.33 / 15.0, G4 0.10 / 4.8, G5 0.07 / 2.5; Codex G0 0.08 / 11.5, G3 0.16 / 22.2, S 0.11 / 13.0, G5 0.015 / 2.0.
| Harness | Arm | Extra idle $/h vs G5 | Extra idle requests/h | Missed share | Events missed | Handled ≤ 5 / ≤ 15 min | Bar |
|---|---|---|---|---|---|---|---|
| Claude Code | G0 as shipped (silent) | +0.59 | +27.6 | 0.00 | 0/45 | 38 / 40 of 45 | fails: cost (F1) |
| Claude Code | G3 long-poll, 50 s | +0.04 | +3.5 | 0.96 | 39/45 | 6 / 6 of 45 | fails: misses (F1) |
| Claude Code | S / |
+0.24 | +13.2 | 0.00 | 0/45 | 10 / 44 of 45 | fails: cost (F3; 98% within 15 min) |
| Claude Code | G4 held call | +0.01 | +1.2 | 0.00 | 0/45 | 45 / 45 of 45 | passes (F2 met) |
| Claude Code | G5 events | 0 | 0 | 0.00 | 0/45 | 45 / 45 of 45 | (reference) |
| Codex | G0 as shipped (silent) | +0.05 | +9.5 | 0.88 | 22/45 | 23 / 23 of 45 | fails: misses (F1) |
| Codex | G3 long-poll, 280 s | +0.15 | +20.5 | 0.88 | 21/33* | 12 / 12 of 33 | fails: both (F1) |
| Codex | S "every 15 min" | +0.08 | +11.0 | 0.94 | 28/34* | 6 / 6 of 34 | fails: misses (F3) |
| Codex | G5 events | 0 | 0 | 0.00 | 0/45 | 45 / 45 of 45 | (reference) |
* Codex G3 and S stopped early in 2 and 1 sessions: one G3 session at its cost cap, the other G3 session and the S session by the runaway rule (12 and 11 replies unobserved). Counting every unobserved reply as handled, they missed 21 and 28 of 45. In Codex as shipped, 8 of 10 sessions stopped checking within about 17 minutes; the other two watched for more than 80 minutes and missed 1 reply between them. Events in Codex include 24 replies released more than an hour after the subscribe.
The primary tests
| Test | Measure | Contrast | Harness | n | Median diff | 95% CI (bootstrap; exact) | p | Holm threshold | Detected |
|---|---|---|---|---|---|---|---|---|---|
| P1 | idle $/h | G3 − G5 | Claude Code | 10 | +$0.038 | [−0.071, 0.051]; [−0.042, 0.048] | 0.46 | 0.025 | no |
| P2 | idle $/h | G3 − G5 | Codex | 10 | +$0.147 | [0.090, 0.642]; [0.096, 0.838] | 0.001 | 0.010 | yes |
| P3 | missed share | G0 − G5 | Claude Code | 6 | 0.00 | [0, 0]; [0, 0] | 1.00 | 0.050 | no |
| P4 | missed share | G0 − G5 | Codex | 6 | +0.875 | [0.06, 1.00]; [0.005, 0.995] | 0.031 | 0.0125 | no (effect large; misses Holm at n = 6) |
| P5 | idle $/h | G4 − G5 (two-sided) | Claude Code | 10 | +$0.011 | [−0.001, 0.099]; [−0.009, 0.089] | 0.11 | 0.017 | no (near tie) |
Sensitivities. Under the design's literal acting-span rule: P1 +$0.006 (p 0.76), P2 +$0.147 (p 0.001), P5 +$0.011 (p 0.25), no change in any decision. Bounds on missed share for stopped sessions leave P3 and P4 unchanged. Without the 51 sessions that straddled a production deploy, P2 has 6 schedules, +$0.108 (p 0.016, all 6 in one direction) and can't clear its threshold, so its detection depends on keeping them, as the design specifies; the others without straddlers: P1 n 6, +$0.006 (p 0.61); P3 n 4, 0; P4 n 4, +1.0 (p 0.06); P5 n 6, +$0.004 (p 1.0). With 6 schedules, the smallest attainable p is 1/64, above P4's threshold of 0.0125 once only P2 had been rejected.
The outcome under the pre-registration: P2 detected; P1, P3, P4, P5 not; F2 met in Claude Code on the blocked task, meaning the held call passed its bar; F1, F3, F4 not met; F5 met, meaning the test-server check failed. The registered statements that apply: holding the call ties or wins in Claude Code, so events are framed as the answer for Codex, for several streams and for clients that can't hold a call; mixed results, so claims are per harness and task; and the test server diverges from the real one, so its numbers are comparisons within the test server.
Total spend per session on the blocked task (median, subscription, idle and acting time together): Claude Code as shipped $1.74, long-poll $0.47, /loop 15m $0.89, held call $0.37, events $0.37; Codex as shipped $0.25, long-poll $0.40, the 15-minute request $0.30, events $0.14. The registered measure is idle cost.
Three streams (exploratory)
| Harness | Arm | Idle $/h | Idle requests/h | Missed | ≤ 15 min | Extra idle $/h vs G5 [95% CI] |
|---|---|---|---|---|---|---|
| Claude Code | G0 | 1.37 | 39.1 | 0/45 | 45 | +1.29 [1.10, 1.35] |
| Claude Code | G3 | 1.34 | 42.9 | 2/45 | 43 | +1.24 |
| Claude Code | S | 0.48 | 18.1 | 0/45 | 45 | +0.41 [0.35, 0.50] |
| Claude Code | G4 | 0.15 | 4.8 | 3/17 | 14 | +0.06 (3 runaway stops) |
| Claude Code | G5 | 0.09 | 3.5 | 0/45 | 45 | – |
| Codex | G0 | 0.89 | 120.4 | 0/45 | 45 | +0.87 [0.77, 0.95] |
| Codex | G3 | 0.55 | 73.0 | 2/45 | 43 | +0.51 |
| Codex | S | 0.54 | 74.2 | 2/45 | 43 | +0.51 [0.48, 0.58] |
| Codex | G5 | 0.024 | 2.3 | 0/45 | 45 | – |
Polling across three streams cost 5.2 to 36.9 times events' idle cost (Claude Code G0 14.8×, G3 14.5×, S 5.2×; Codex G0 36.9×, G3 22.6×, S 22.3×); the held call 1.6×. Ten polling sessions stopped at their cost cap (Claude Code G0 1, G3 4; Codex G0 5), and their cost stops counting there, which favors polling. The held call needs one call per stream; in 3 of 10 sessions the agent called the wait tools under two seconds apart and the runaway rule stopped them within four minutes, so 17 of its 45 releases were observed.
The screen (one-hour sessions, exploratory)
8 sessions per wording and harness. G2 ("check back in about 10 minutes") kept both harnesses reliable: Claude Code 1 of 12 missed, Codex 0 of 11. G1 ("poll every 5 to 10 seconds") missed 6 of 11 in each harness, and one Codex session polled 815 times before its cost cap. GB, the blind wording that told the agent not to check in a tight loop and to end its turn, missed 7 of 12 in Claude Code and 11 of 11 in Codex. G4 missed 0 of 11; G5 1 of 12 and 0 of 12.
The real-server check (F5)
| Harness | Level | Pairs | Cost per check real / toy | Checks per hour real / toy | Pass |
|---|---|---|---|---|---|
| Claude Code | G0 | 4 | $0.029 / $0.027 | 7.5 / 3.0 | no (checks) |
| Claude Code | G2 | 4 | $0.035 / $0.041 | 7.5 / 6.5 | yes |
| Codex | G0 | 4 | $0.012 / $0.008 | 17.5 / 12.5 | no (both, by ~40%) |
| Codex | G2 | 4 | $0.015 / $0.014 | 14.0 / 16.0 | yes |
As shipped, agents checked the real server more often than the test server; with G2's wording they matched. With 4 pairs per cell this is a check, not a measurement. It covers only the email service, one-hour sessions and those two wordings, and it compared checking, not missed replies.
The cost projection (illustration only)
A model projecting cost from the measured per-check costs and intervals was to be validated against the measured cells within ±25%. It missed in 14 of 14 cells on the first attempt (using each cell's median gap between checks) and 13 of 14 on the second (using each cell's effective interval, which reflects giving up); the input changed between attempts. Under the registered rule it is an illustration only. Measured inputs: per check on a warm cache, Claude Code adds about 960 tokens and writes about 180; Codex about 230 and 50; about 1.1 to 1.2 requests per check.
The API key (exploratory)
First declined (deviation 32), then run after the primary results were known, at our request: Claude Code, the blocked task, schedules 1 to 4, n = 4 per approach. Effect sizes only.
| Arm | Idle $/h, API key | Idle $/h, subscription (same 4 schedules) | Idle requests/h (API) | $ per session (API) | Missed | ≤ 5 / ≤ 15 min |
|---|---|---|---|---|---|---|
| G0 as shipped | 0.50 | 0.69 | 22.2 | 2.14 | 1/24 | 23 / 23 |
S / |
0.55 | 0.31 | 13.4 | 1.80 | 0/24 | 9 / 23 |
| G4 held call | 0.17 | 0.11 | 5.8 | 0.68 | 0/24 | 24 / 24 |
| G5 events | 0.17 | 0.11 | 4.3 | 0.80 | 0/24 | 24 / 24 |
Paired against events on the API key: /loop 15m +$0.40 an hour (all 4 schedules positive), held call +$0.018 (per schedule −$0.03 to +$0.03), as shipped +$0.27 (one schedule −$0.04, another +$1.37).
Behavior coding and context
Two model coders (Sonnet 5.5 and GPT-6.1 Sol, masked to the approach) labeled each session; a label counts only when both agree. Agreement (κ) on the main stage: asked the user 0.61, claimed done early 0.53, mentioned a wake 0.83, false all-clear 0.65. On the blocked task, Claude Code's long-poll asked the user to tell it about replies in 5 of 10 sessions; events sessions never did. No polling session carried Wakeshift's note or files. Wakeshift's plugin adds about 120 to 220 tokens to an events session's first request.
Drift and environment
Per-wave medians are stable across the main stage's waves, apart from the held call's runaway-driven outliers on three streams. Production deployed twice during the main stage, so 51 sessions straddle a deploy; they are kept and flagged, as the design specifies for production. The events arm used three production apps, one per surface.
Deviations
Study 1
Each was written before the data it affects was seen, except where noted.
-
One server was swapped out of the API-key and long-poll sets before round 2, because its backend was no longer available: the long-poll set used the agent communication server instead, and the API-key set a server later removed from the study.
-
The Wakeshift version under test was pinned at the round's first run (v1.4.50); later moves are recorded below, with no cell mixing versions.
-
The AI model runner's equalized arm gains one sentence from the events branch: "Leave
waitandtimeout_secondsat their defaults." -
The runner's proxy had its own 300-second timeout in round 1; from round 2 it has none. Which round-1 sessions it could have cut is checked in the re-scoring.
-
The version moved to v1.4.51 mid-round. Plugin and connector files were unchanged; treated as a boundary, and cells compared across it are labeled.
-
The web scraping account's monthly limit allowed long-poll only in Codex. Its Claude Code as-shipped arm already waits 45 s in the server, which is what the long-poll does.
-
Three held-call cells were rerun on fresh test servers after bench failures (a sign-in failure and stale reports from an earlier run).
-
The agent communication server's long-poll never engaged (its "job" is an inbox, not a job ID); its 4 sessions are a bench failure, and its long-poll was not measured.
-
The API-key set was cut short for the draft's deadline; that partial run belonged to the removed server and is dropped with it.
-
The long-hold extension (30, 60 and 120 minutes) was added after the 15-minute result, registered before its runs, and ran nine sessions at once on separate servers, on v1.4.52 (plugin files unchanged).
-
Two extensions were registered on 2026-11-14 before their sessions, after the main results: the API-key cache check, run in full, and longer jobs, not run. During it the web crawling vendor's crawls became 3 to 4 times faster than in rounds 1 and 2 (checked on sites the bench had never crawled), so that cell barely tests the cache. The web scraping events arm was rerun after a bench fault (no job started), and the version moved to v1.4.54 and then v1.4.55 mid-extension; the version of every session is recorded.
-
The infrastructure-failure rule (2026-11-16, after the results; our decision): sessions whose failure was caused by infrastructure unrelated to the approach are excluded, in every arm and both studies. It removed four study-1 events sessions hit by the Wakeshift server outage of 2026-11-12 (research API cc-b-300-2 and cc-b-300-3, crawling API cc-b-480-4, email replies cc-b-180-1, all Claude Code). Effect: in Claude Code the research API's events cost moves from 22% more to 14% more, the crawling API's stays 3% more, and in-scope events asks go from 3 of 127 to 0 of 124. Details and evidence are under "Sessions excluded for infrastructure failures".
-
The conclusion's scope (2026-11-16, after the results; our decision): out are servers that already hold the request open and send progress on tasks that reliably finish in under about five minutes (AI video generation and the AI model runner in Claude Code); the research API and the silent crawling API in Claude Code are in.
-
The fixed-bug rule (2026-11-16, after the results; our decision): sessions affected by a bug we found and fixed during the study, in Wakeshift, its connector, an integration or our test runner, are excluded in every arm, since they measure the bug rather than the approach. Sessions that ran before a fix but that the bug couldn't have changed are kept: for example, Codex sessions that ran before the test runner's proxy timeout was removed, since Codex cuts every call at 300 s on its own. The rule removed 7 Codex events sessions on the web scraping server, run before a fix to Wakeshift's Codex connector (it didn't accept hyphenated tool names): in all 7 the user had to ask, and 6 never got the result.
Analysis notes from the results: round 2's as-shipped arm replaces round 1's without pooling; tests sort each arm's values first (intervals and p can differ in the third decimal from session order; medians don't); the never-told pattern was widened to catch "I don't know … page count"; job length is reported, not normalized; the API-key rows are kept out of the main table.
Corrections after the first round. Reviews found that the runner's proxy buffered servers' progress updates, which cut off waits that stream progress. After the fix and a rerun, round 1's 42% saving on AI video generation disappeared (events now cost 27% more in Claude Code) and its 26% saving on the AI model runner was no longer supported. The one server removed after the analysis is recorded as a deviation in the published method.
Study 2
Entries 1 to 17 were made while building, before any session ran, and are committed with the pre-registration.
-
A schedule draw before 2 minutes is moved to 2 minutes (affects two schedules).
-
Two main-stage schedules drew no replies. Nothing was redrawn. P3 and P4 have n = 6, so the smallest one-sided p is 1/64, above Holm's first threshold; they can be detected only if two other tests are rejected first.
-
The "poll every 5 to 10 seconds" wording is adapted to each surface.
-
The wordings change each surface's two list or check tools; fetch-by-ID tools are unchanged.
-
What the test servers serve (names, titles, descriptions, schemas, annotations, instructions; not icons or output schemas).
-
Default tool sets follow each server's own list; triage and deprecated tools are excluded for the error alert service.
-
The prompts were revised after an outside reader, before commit.
-
The 15-minute request is filled per task and sent once the first turn ends.
-
The held call's and events' tool wordings are adapted per surface from the email service's wording.
-
Wake texts carry IDs only, so acting needs a fetch.
-
The screen's balanced table of tasks, schedules and wordings.
-
Idle time and "gave up" as implemented.
-
The runaway rule's test.
-
A wave's worst-case cost counts polling lanes at their caps.
-
Isolation additions for each session's environment.
-
The two model coders and their shared prompt.
-
The real-server check's inbox and setup.
-
The blind wording's writer got an abridged brief in places.
-
The real-server check ran one real lane at a time (no spare inbox), roughly doubling its wall time.
-
The screen's events plugin could not be pinned to a commit and was logged per wave instead.
-
Before any study data, the end of an acting span was changed: Codex often polls inside one long turn, so "to the end of that turn" left almost no idle time. The literal rule is reported as a sensitivity (P1 +$0.006 instead of +$0.038; no decision changes).
-
How subscription usage is read.
-
The sign-flip test's statistic is the sum of paired differences, which the design's power statement assumes.
-
Pre-flight found that Wakeshift's Codex connector stopped listening after an hour without a turn. The screen's releases ended inside the hour; the main stage waited for a released fix.
-
In the screen, the three-stream events sessions used one Wakeshift app for three servers, which Wakeshift refuses; two sessions were discarded and rerun, and the error alert surface then used another vendor's existing test app, a confound limited to that cell.
-
The main stage's events arm used three production apps, one per surface; production deploys during the main stage are flagged, not excluded (51 sessions).
-
An unrelated built-in tool server appeared intermittently in Claude Code's tool list; the tool-list guard failed those sessions (8 three-stream sessions and one real-server pair), and they were rerun with it disabled.
-
A runner bug aborted one wave (no data, $9.50 counted in spend).
-
The pre-registered cut order was applied at $228.87 spent: optional sets, the remaining real-server pairs, and later three-stream blocks were cut. The blocked task's data, and P1 to P5, were complete.
-
About an hour after P1 to P5 were computed (04:25 UTC on November 15), we raised the cap from $300 to $600 (about 05:20 UTC) and the cuts were reversed: the remaining three-stream blocks ran 3 to 8 hours after the first waves, and the real-server pairs ran. These runs were decided after the main result was known.
-
A descriptive interim look at the main stage, at our request, while runs continued: medians and counts only, no tests. Nothing in the plan changed.
-
The optional sets were first declined.
-
The cost projection's validation input was changed after a first failure (14 of 14 cells off); it failed again (13 of 14).
-
The API-key run (optional set 1) was run after all, after the primary results, at our request. Exploratory; changes no test.
-
The infrastructure-failure rule (2026-11-16, after the results; our decision; applied to both studies): sessions whose failure was caused by infrastructure unrelated to the approach are excluded, in every arm. It removed no study-2 session; P1 to P5 are unchanged. The sessions examined are listed in the appendix section on infrastructure failures.
Correction C1 (from the earlier skeptic review of the results files): two of the three early stops on the blocked task were by the runaway rule, not the cost cap; "none of the straddlers change a conclusion" was wrong, since P2 depends on them; and the three-stream range "5 to 40 times" was replaced by the computed 5.2 to 36.9.