Appendix: how we measured the cost of waiting with and without MCP Events

The appendix to "MCP Events beat polling on reliability and token cost".

This appendix was generated by an LLM (Claude). A script copied its tables from the studies' results files; Claude wrote the text around them from the same files, and it was checked against the data but not written by hand.

This page holds what the post leaves out: each study's design, every per-server and per-approach table, the tests in full, the exploratory results and every departure from the plan. Servers are named by category because their vendors haven't reviewed the results.

Dollars throughout are API list price per million tokens. Opus 5.5: $4 input, $5 (5-minute) or $8 (1-hour) cache write, $0.20 cache read, $20 output. GPT-6.1 Sol: $2.50 uncached input (priced as a cache write, since Codex signed in with a subscription reports no cache writes), $0.10 cached input, $10 output, with a higher tier above 272k tokens. On a subscription plan the same requests count against usage limits, so request counts sit beside the dollars. Both studies ran Claude Code 2.1.321 on Opus 5.5 and Codex 0.167.1 on GPT-6.1 Sol on one machine. Wakeshift versions: study 1's round 2 ran plugin v1.4.50 and v1.4.51, the long holds v1.4.52, and its API-key run v1.4.54 and v1.4.55; study 1's round-1 sessions, which include every events arm, mostly don't record their version in the data (512 of 525; the run logs have it); study 2 ran v1.4.54 (Codex connector 2026.11.14.1515, with a fix to how it holds its connection). Dates (UTC): study 1, November 12 to 14; study 2, November 14 to 16, the API-key run on November 16.

The two studies measure different things. The first measures the cost of one wait for a job; the second measures cost per idle hour and missed replies over a two-hour window. Their numbers are never comparable directly.

Study 1: Long-running jobs

Design

The question: an agent starts a job on a vendor's MCP server that runs longer than the tool call. What does it spend in model requests before it tells the user the result, when it keeps checking (polling) and when the server tells it the job finished (an event)?

Arms:

Wakeshift's plugin is installed in every round-2 arm. Each session is a fresh conversation in which a simulated user asks for the job and its result in one message. If the agent hasn't told the result 120 s after the job ends, the user asks once.

Measures, per session:

Statistics: medians with interquartile ranges. For each pair of cells with n ≥ 5, a 95% bootstrap interval on the difference of medians (10,000 resamples, seed 1113) and a one-sided permutation test. A saving is detected when p < 0.05 and the interval's lower end is above zero; otherwise there is no detectable difference. Nothing is pooled across servers or harnesses. Cells with n = 2 are shown as values only.

The analysis plan for round 2 was written and committed on 2026-11-13, before any round-2 run and before the round-1 data was re-scored under it. Round 1 itself was not pre-registered; three adversarial reviews of it (methods, harness and a hostile reader) found the problems the plan fixes, including the buffering proxy described under the deviations.

Discards apply to both arms alike: a bench failure shown by the logs, a vendor failure (the job never started or failed at the vendor), or loss of network on the test machine. The plan counted failures of Wakeshift (a failed subscribe, a late or missed wake, a server error) as the events arm's outcome. Two later rules, made after the results were known and applied to every arm, changed that: sessions that failed during an outage of infrastructure unrelated to the approach are excluded (deviation 12), and so are sessions run before the fix of a bug we found during the study (deviation 14). Every exclusion rule is described here with its effect, and the sessions it removed are listed or counted. Sessions that crossed a change of the Wakeshift version under test are kept and flagged (the Straddlers column in Every cell); in some events cells they are the majority, for example 6 of 8 in web crawling on Claude Code and 7 of 12 in AI video generation on Codex.

Scope. The study ran six servers with jobs the agent starts, longer than a minute, checked through a status tool, plus an agent communication server (the agent waits for an email reply sent about three minutes after its own message), a crawling API that holds its call and sends nothing, a research API that holds its call while streaming, short jobs (20 to 45 s) as controls, and 200k tokens of earlier context at n = 2 per arm, values only. The agent communication server (email replies) is a wait the agent didn't start, so we moved it out of study 1's conclusion on 2026-11-16; the second study covers that kind of wait, and its rows stay in every table here. The post's conclusion covers the remaining cells where holding the request open doesn't fit; the scope and the cells it leaves out are under "Outside the conclusion's scope" below, with the short controls (events cost $0.001 to $0.06 more per job there).

Left out of the study: a web data collection service and a file encoding service, whose jobs usually finished inside the tool's own wait; a site-migration service, whose credits ran out; and one server whose use case didn't fit, removed after the analysis (our decision). Apart from the removed server, every excluded cell is in the published dataset.

Spend: round 2's model total was about $36 at list price, the long-hold extension $1.92, and the API-key extension $11.30 on the API key.

Main results: fresh context, subscription

"Saved" is median as shipped minus median events, with the bootstrap 95% interval. "Asked" and "Never told" give arms A / L / B.

This table counts every session that ran, before the exclusion rules below. With them applied, as in the post, the research API in Claude Code costs 14% more with events instead of 22%, and no events user had to ask for a result on it or on the second crawling API in Claude Code (see "Sessions excluded for infrastructure failures").

Server Harness Sign-in As shipped (arm) n Requests per wait Wait $ as shipped Long-poll $ (n) Events $ (n) Saved p Verdict Long-poll → events $ per conversation Asked A / L / B Never told A / L / B
Web crawling Claude Code subscription A1 8/8 21.0 → 4.0 0.318 (0.302–0.342) 0.117 (8), 6.5 req 0.078 (0.076–0.093) 76% ($0.240; CI 0.208 to 0.271) 0.002 detected 34%, detected 0.418 → 0.188 (55%, detected) 4 / 0 / 0 0 / 0 / 0
Web crawling Codex subscription A1 8/8 17.0 → 4.0 0.078 (0.073–0.104) 0.059 (8), 10.0 req 0.036 (0.036–0.042) 53% ($0.041; CI 0.031 to 0.075) 0.006 detected 38%, detected 0.104 → 0.072 (31%, detected) 0 / 0 / 0 0 / 0 / 0
Video editing Claude Code subscription A1 8/10 23.0 → 4.0 0.308 (0.293–0.337) – 0.089 (0.089–0.091) 71% ($0.219; CI 0.171 to 0.255) 0.001 detected – 0.419 → 0.197 (53%, detected) 0 / – / 0 0 / – / 0
Video editing Codex subscription A1 8/10 17.0 → 3.0 0.102 (0.087–0.111) – 0.038 (0.038–0.056) 63% ($0.064; CI 0.035 to 0.080) 0.001 detected – 0.125 → 0.066 (47%, detected) 0 / – / 0 0 / – / 0
Web scraping Claude Code subscription A2 (across rounds) 8/9 14.5 → 4 0.331 (0.307–0.374) – 0.080 (0.077–0.164) 76% ($0.251; CI 0.138 to 0.314) 0.002 detected – 0.522 → 0.289 (45%, detected) 0 / – / 0 0 / – / 0
Web scraping Codex subscription A1 8/9 19.0 → 3 0.140 (0.116–0.154) 0.089 (8), 15.0 req 0.045 (0.041–0.046) 68% ($0.094; CI 0.060 to 0.115) 0.002 detected 49%, detected 0.229 → 0.145 (36%, detected) 0 / 0 / 0 0 / 0 / 0
Agent communication (email reply) Claude Code subscription A1 8/9 15.0 → 4 0.203 (0.194–0.205) – 0.059 (0.059–0.060) 71% ($0.143; CI 0.134 to 0.147) 0.002 detected – 0.323 → 0.176 (46%, detected) 0 / – / 1 0 / – / 0
Agent communication (email reply) Codex subscription A1 8/8 12.5 → 3.5 0.085 (0.056–0.104) – 0.038 (0.034–0.042) 55% ($0.047; CI 0.014 to 0.096) 0.002 detected – 0.136 → 0.095 (30%, detected) 0 / – / 0 0 / – / 0
CI builds Claude Code subscription A1 8/11 7.0 → 4 0.139 (0.135–0.148) – 0.097 (0.093–0.120) 30% ($0.041; CI -0.003 to 0.053) 0.078 no detectable difference – 0.352 → 0.347 (2%, no detectable difference) 0 / – / 0 0 / – / 0
CI builds Codex subscription A1 8/8 13.0 → 3.0 0.088 (0.060–0.101) – 0.036 (0.036–0.036) 59% ($0.052; CI 0.022 to 0.078) 0.002 detected – 0.131 → 0.091 (31%, detected) 0 / – / 0 0 / – / 0
AI video generation Claude Code subscription A2 (across rounds) 8/8 4.0 → 4.0 0.046 (0.045–0.046) 0.047 (8), 4.0 req 0.058 (0.057–0.058) -27% ($-0.012; CI -0.013 to -0.011) 0.984 events cost more -23%, events cost more 0.146 → 0.180 (-24%, events cost more) 0 / 0 / 0 0 / 0 / 0
AI video generation Codex subscription A1 8/12 11.0 → 3.0 0.046 (0.045–0.059) 0.071 (8), 16.0 req 0.033 (0.032–0.033) 30% ($0.014; CI 0.011 to 0.040) 0.000 detected 55%, detected 0.096 → 0.083 (14%, no detectable difference) 0 / 0 / 0 0 / 0 / 0
AI model runner Claude Code subscription A2 (across rounds) 8/9 4.0 → 4 0.053 (0.053–0.133) – 0.063 (0.062–0.064) -18% ($-0.010; CI -0.011 to 0.079) 0.991 events cost more – 0.196 → 0.228 (-16%, events cost more) 0 / – / 0 0 / – / 0
AI model runner Codex subscription A2 (across rounds) 8/11 13.0 → 4 0.116 (0.094–0.134) – 0.046 (0.045–0.052) 60% ($0.069; CI 0.036 to 0.088) 0.003 detected – 0.215 → 0.136 (37%, detected) 0 / – / 0 0 / – / 0
Crawling API that holds the call Claude Code subscription A2 (across rounds) 8/8 6.0 → 5.0 0.094 (0.092–0.094) – 0.097 (0.096–0.097) -3% ($-0.003; CI -0.006 to -0.002) 1.000 events cost more – 0.200 → 0.218 (-9%, events cost more) 8 / – / 1 8 / – / 0
Crawling API that holds the call Codex subscription A1 7/7 22 → 3 0.141 (0.124–0.171) – 0.032 (0.031–0.032) 77% ($0.109; CI 0.081 to 0.141) 0.006 detected – 0.187 → 0.083 (56%, detected) 7 / – / 0 6 / – / 0
Research API Claude Code subscription A1 8/8 4.5 → 5.5 0.179 (0.152–0.184) – 0.218 (0.191–0.240) -22% ($-0.040; CI -0.097 to -0.007) 0.983 events cost more – 0.283 → 0.327 (-15%, events cost more) 0 / – / 2 0 / – / 0
Research API Codex subscription A1 8/8 21.0 → 3.0 0.114 (0.105–0.120) – 0.076 (0.071–0.094) 33% ($0.038; CI 0.013 to 0.053) 0.002 detected – 0.148 → 0.111 (25%, detected) 4 / – / 0 4 / – / 0

The AI model runner's A2 → B comparison is confounded by job length: round 2's jobs ran a median of 210 s (Claude Code) and 268 s (Codex), against 139 to 174 s in round 1. The like-for-like comparison is the equalized arm:

Server Harness E2 $ (n) requests Events $ E2 → events p Verdict
AI model runner Claude Code 0.101 (8) 7.0 0.063 38% 0.009 no detectable difference
AI model runner Codex 0.065 (8) 8.5 0.046 28% 0.016 no detectable difference

With matched descriptions, the model runner showed no detectable difference in either harness. For the other five servers the events branch differs from as shipped only by the wake passage appended to the tool descriptions, which is part of the event mechanism. Job lengths differ between arms in some cells (web scraping in Claude Code: 562 s as shipped against 431 s with events; the AI model runner in Codex: 268 s against 134 s), which favors events there. The per-cell tests are not corrected for multiple comparisons; two long-poll comparisons sit at p = 0.033 and 0.047.

Outside the conclusion's scope

The post's conclusion leaves out servers that already hold the request open and send progress, for tasks that reliably finish in under about five minutes: holding the call that way works in both harnesses, since Codex ends calls at 300 seconds. It has costs of its own: Claude Code is tied up for its first couple of minutes, and Codex narrates while it waits (see Holding the call). The rule was set on 2026-11-16, after every result here was known. Two Claude Code cells are out under it: AI video generation (as-shipped jobs 188 to 231 s) and the AI model runner (139 to 362 s; one of 8 jobs ran past five minutes, and it was left out by our decision). The research API in Claude Code also holds the call while streaming, but its as-shipped jobs ran 222 to 414 s, past five minutes in 4 of 8 sessions, so it fails "reliably under five minutes" and is in scope. The second crawling API in Claude Code is in scope: its held call sends no progress, its crawls take about six and a half minutes, and as shipped it never delivered the result (8 of 8). The Codex counterparts stay in scope: there the agent made 11 to 22 requests per wait, narrating while it waited. The short controls, whose jobs end within the tool's own wait, are below.

Server Harness n (as shipped / events) As shipped $ Events $ Difference Verdict Had to ask (as shipped / events) Never told (as shipped / events)
AI video generation Claude Code 8 / 8 0.046 0.058 events 27% more detected 0 / 0 0 / 0
AI model runner Claude Code 8 / 9 0.053 0.063 events 18% more not detected (CI −0.011 to 0.079) 0 / 0 0 / 0

Short controls (jobs of 20 to 45 seconds, fresh, subscription; medians from final.json; no user had to ask and none went untold in any of them):

Server Harness Job (s) n (as shipped / events) As shipped $ Events $ Events cost more by
Web crawling Claude Code 45 6 / 6 0.084 0.088 $0.004
Web crawling Codex 45 6 / 6 0.033 0.043 $0.010
Web scraping Claude Code 45 5 / 5 0.032 0.033 $0.001
Web scraping Codex 45 5 / 5 0.015 0.042 $0.028
CI builds Claude Code 20 5 / 10 0.059 0.118 $0.060
CI builds Codex 20 5 / 10 0.017 0.040 $0.023
AI video generation Claude Code 30 5 / 5 0.013 0.013 $0.001
AI video generation Codex 30 5 / 5 0.005 0.037 $0.033
AI model runner Claude Code 45 5 / 5 0.019 0.061 $0.042
AI model runner Codex 45 5 / 5 0.006 0.042 $0.036

The CI build short controls have 5 straddlers in each arm. An earlier summary gave "up to $0.04 more per job"; the medians above give up to $0.06.

Sessions excluded for infrastructure failures

The rule (our decision, 2026-11-16, after the results were known): a session is excluded when its failure was caused by infrastructure unrelated to the approach under test, in every arm and both studies. That covers Wakeshift's bridge or core outages, vendor-side outages, errors and timeouts, harness crashes, and bench or runner faults. Model behavior, the agent giving up, and an approach's own design limits (Codex ending a call at 300 seconds, Claude Code dropping a call that sends nothing) are not infrastructure and stay in.

How we looked. Study 1: every one of 736 sessions (all arms, contexts and sign-ins) was scanned for error markers in its steps, its tool results and its terminal log, and every session that ended with the user asking or never told (47) was traced to a cause. Study 2: every kept session's scores, toy-server log and transcript was scanned for tool, transport and harness errors, and every early stop was checked.

Study 1, removed (all four in the events arm, all during an outage of Wakeshift's pre-release server on 2026-11-12, about 20:36 to 20:58 UTC):

Server Harness Session Evidence
Research API Claude Code run 20261112-123332-3724, cc-b-300-2 Server log: block_failed (APIError) on its agent_run call at 20:44:47 UTC, subscription rolled back; the tool's own text still promised a wake; no model request between the end of the first turn and the user's question
Research API Claude Code same run, cc-b-300-3 Server log: block_failed at 20:53:57 UTC; same pattern
Crawling API that holds the call Claude Code run 20261112-120337-1e54, cc-b-480-4 Job 20:45:30 to 20:51:47 UTC, inside the outage; no wake and no request until the user asked 120 s after the job ended, while the other 7 Claude Code events sessions were woken 1.3 to 4.1 s after it ended; the vendor's run notes record the outage during this run. The server log for this run wasn't retained, so the cause rests on the timing and the missing wake.
Email replies Claude Code run 20261112-124413-dff7, cc-b-180-1 The vendor's run notes: Wakeshift's subscribe endpoint answered 500 and the SDK logged block_failed at 20:50:34 UTC, rolling back the subscription; the session was rerun later and the rerun was woken

Study 1, examined and kept: the failures in the polling arms are the agent giving up while the job ran (web crawling in Claude Code), Claude Code dropping a held call that sent nothing (the crawling API that holds the call), Codex ending calls at 300 s (the crawling and research APIs in Codex; the log reads "timed out awaiting tools/call after 300s"), and the held-call test server's designed holds. One held-call test session logged an Anthropic API retry ("API error · Retrying in 1s"), which recovered without a failure. Sessions earlier left out for vendor rate limits and bench faults stay out, as already listed.

Study 2, removed: none. Examined and kept: a transient transport error in Codex's MCP client ("Transport send error … EventNotificationTransport") appeared once in each of six Codex sessions (three long-poll, three events) and recovered on the agent's next call. No reply was missed because of it. In the one session stopped early near such an error (long-poll, schedule 5: error at 7.5 minutes, runaway stop at 8.7), the agent's request rate was the same before the error (about 4 a minute) and the runaway rule fired on its own polling cadence. Toy-server "not found" errors (48 calls in 36 sessions) were the agent asking for IDs the test server had never listed; one, CHECKOUT-API-3F, appears as an example in the real error-alert tool's own text, which the test server copies. That is model behavior, not a fault in the test server. P1 to P5 are therefore unchanged.

With and without the excluded sessions (study 1, Claude Code; as-shipped arms unchanged):

Server As shipped $ (n) Events, all sessions: $ (n), had to ask Saved, all Events, excluding infrastructure failures: $ (n), had to ask Saved, excluding Verdict, excluding
Research API 0.179 (8) 0.218 (8), 2 22% more 0.204 (6), 0 14% more events cost more (CI −0.081 to −0.002)
Crawling API that holds the call 0.094 (8) 0.097 (8), 1 3% more 0.096 (7), 0 3% more events cost more (CI −0.006 to −0.002)
Email replies 0.203 (8) 0.059 (9), 1 71% 0.059 (8), 0 71% detected

In the conclusion's 14 cells, events: 0 of 124 had to ask excluding the three in-scope sessions, 3 of 127 counting them, and 10 of 134 had to ask and 6 were never told counting these and the 7 sessions the fixed-bug rule removed (deviation 14); as shipped: 23 of 111 had to ask and 18 were never told, unchanged by the rule.

Long-poll

Server Harness As shipped Long-poll (n = 8) Events Events vs long-poll
Web crawling Claude Code $0.318 (A1), 21 requests $0.117, 6.5 requests $0.078, 4 requests 34%, detected
Web crawling Codex $0.078 (A1), 17 $0.059, 10 $0.036, 4 38%, detected
Web scraping Codex $0.140 (A1), 19 $0.089, 15 $0.045, 3 49%, detected
Web scraping Claude Code as shipped is already a 45 s long-poll: $0.331 (A2), 14.5 – $0.080, 4 76%, detected
AI video generation Claude Code $0.046 (A2), 4 $0.047, 4 $0.058, 4 events cost more
AI video generation Codex $0.046 (A1), 11 $0.071, 16 $0.033, 3 55%, detected

Long-poll against events crosses rounds and versions (round 2's arm L against round 1's arm B). Users asked in no long-poll session. Codex holds a silent long-poll for up to 280 s and narrates while it waits, so its long-poll can cost more than as shipped (AI video generation). The web scraping server's status tool already waits 45 s as shipped, so its Claude Code cell is the long-poll case; its agent still called the tool about every 47 seconds.

The API key: the cache check

Run on 2026-11-14 after the main results, at our request, and registered before its first session. Claude Code on an Anthropic API key (a 5-minute prompt cache), fresh, n = 8 per arm.

Server Harness Sign-in As shipped (arm) n Requests per wait Wait $ as shipped Long-poll $ (n) Events $ (n) Saved p Verdict Long-poll → events $ per conversation Asked A / L / B Never told A / L / B
Web crawling Claude Code api-key A2k 8/8 6.5 → 5.0 0.099 (0.096–0.102) – 0.063 (0.063–0.068) 36% ($0.036; CI 0.025 to 0.044) 0.002 detected – 0.150 → 0.119 (21%, detected) 0 / – / 0 0 / – / 0
Video editing Claude Code api-key A2k 8/8 18.0 → 5.0 0.165 (0.154–0.179) – 0.071 (0.069–0.072) 57% ($0.094; CI 0.062 to 0.138) 0.002 detected – 0.223 → 0.129 (42%, detected) 0 / – / 0 0 / – / 0
Web scraping Claude Code api-key A2k 8/8 10.0 → 6.0 0.195 (0.138–0.216) – 0.090 (0.064–0.112) 54% ($0.105; CI 0.029 to 0.148) 0.002 detected – 0.314 → 0.200 (36%, detected) 0 / – / 0 0 / – / 0
Server Job s, A2k / B2k (median, range) Subscription job s, A / B (median) Events wakes later than 5 min after the last request Wake: cache write / read tokens (B2k median) Wake $ (B2k median)
Web crawling 42 (34–107) / 36 (14–51) 149 / 184 0 of 8 671 / 24,108 0.011
Web scraping 298 (250–321) / 263 (245–277) 562 (A2) / 431 0 of 8 (gaps 241–274 s) 773 / 31,599 0.013
Video editing 255 (181–298) / 232 (182–396) 306 / 278 1 of 8 (392 s) 4,414 / 24,818 0.032

Against the subscription (cross-round, cross-version, and the jobs ran shorter, so values only):

Server As shipped: subscription → API key Events: subscription (B1) → API key (B2k) Saved: subscription → API key
Web crawling $0.318 (21 req) → $0.099 (6.5) $0.078 → $0.063 76% → 36%
Web scraping $0.331 (A2, 14.5) → $0.195 (10) $0.080 → $0.090 (no detectable difference) 76% → 54%
Video editing $0.308 (23) → $0.165 (18) $0.089 → $0.071 71% → 57%

The saving is smaller than on the subscription because polling made fewer checks on shorter jobs; the events waits cost about the same. Only 1 of the 24 events waits sat idle past 5 minutes (a 396 s video job): its wake wrote 10,102 tokens to the cache against 780 to 4,470 in the other waits, and the wait cost $0.117 against the cell's median $0.071, still under every as-shipped wait in that cell ($0.121 to $0.219). Two of the web scraping events sessions polled anyway and are in the medians. Idle waits well past 5 minutes on an API key were not measured in this set.

Holding the call: the held-call experiment

A test server whose build_report tool holds the call for 1, 5 or 15 minutes, with or without a progress notification every 15 s, then returns a total the agent must tell. Partway through, the user asks an unrelated question ("what's 17 times 23?") to see whether the session stays usable. Pre-registered on 2026-11-13 before its runs; n = 3 per cell, 54 sessions; no events arm.

Client Sign-in Hold Progress The call Backgrounded Told unasked / asked / never Waiting $ median (range) Requests Interjection answered after (s)
Claude Code subscription 1 min off cut at 60 s (3/3) 0 3 / 0 / 0 0.029 (0.028–0.029) 3 31.0
Claude Code subscription 1 min on held (3/3) 0 3 / 0 / 0 0.020 (0.018–0.020) 2 30.2
Claude Code subscription 5 min off cut at 60 s (3/3) 0 3 / 0 / 0 0.146 (0.139–0.178) 15 1.7
Claude Code subscription 5 min on held (3/3) 3 3 / 0 / 0 0.055 (0.055–0.056) 6 2.2
Claude Code subscription 15 min off cut at 60 s (3/3) 0 0 / 3 / 0 0.249 (0.092–0.304) 25 1.7
Claude Code subscription 15 min on held 15 min (3/3) 3 3 / 0 / 0 0.056 (0.053–0.058) 6 1.6
Claude Code API key 1 min off cut at 60 s (3/3) 0 3 / 0 / 0 0.022 (0.021–0.022) 3 31.1
Claude Code API key 1 min on held (3/3) 0 3 / 0 / 0 0.014 (0.014–0.016) 2 29.9
Claude Code API key 5 min off cut at 60 s (3/3) 0 1 / 2 / 0 0.122 (0.085–0.131) 15 2.1
Claude Code API key 5 min on held (3/3) 3 3 / 0 / 0 0.051 (0.041–0.052) 6 1.7
Claude Code API key 15 min off cut at 60 s (3/3) 0 0 / 3 / 0 0.186 (0.112–0.201) 20 1.8
Claude Code API key 15 min on held 15 min (3/3) 3 3 / 0 / 0 0.076 (0.070–0.078) 6 1.7
Codex subscription 1 min off held (3/3) – 3 / 0 / 0 0.024 (0.020–0.024) 7 3.7
Codex subscription 1 min on held (3/3) – 3 / 0 / 0 0.021 (0.014–0.052) 6 3.5
Codex subscription 5 min off held to 300 s, its limit¹ (3/3) – 3 / 0 / 0 0.102 (0.060–0.124) 19 3.7
Codex subscription 5 min on the same¹ (3/3) – 3 / 0 / 0 0.106 (0.087–0.124) 20 2.7
Codex subscription 15 min off cut at 300 s (3/3) – 2 / 1 / 0 0.214 (0.178–0.296) 36 3.1
Codex subscription 15 min on cut at 300 s (3/3) – 1 / 2 / 0 0.232 (0.199–0.258) 39 2.5

¹ The server returned at 300.00 s, racing Codex's limit; the agents confirmed with a status call within 4 s.

Long holds (registered before their runs; Claude Code on a subscription, progress on, n = 3 each, all nine run at once on separate servers):

Hold The call Progress sent Backgrounded at Told unasked / asked / never Told after the end (s) Waiting $ median (range) Wait requests Wait tokens (median) Interjection answered after (s) Timeout messages
15 min (sidebar, above) held (3/3) 59 ~2 min (3/3) 3 / 0 / 0 <1 0.056 (0.053–0.058) 6 – 1.6 –
30 min held 1800 s (3/3) 119 121–123 s (3/3) 3 / 0 / 0 1.6–2.3 0.045 (0.045–0.055) 5–6 161k 1.7–2.0 none
60 min held 3600 s (3/3) 239 121–122 s (3/3) 3 / 0 / 0 1.5–2.4 0.046 (0.045–0.057) 5–7 161k 1.4–1.9 none
120 min held 7200 s (3/3) 479 122 s (3/3) 3 / 0 / 0 1.5–2.4 0.108 (0.108–0.109) 5 161k 1.7–1.9 none

No call was cut, and no timeout message appeared, up to 2 hours. At 120 minutes the subscription's one-hour cache had expired, so the resume rewrote about 8.3k tokens. On a real crawling server that held its stream open but sent nothing, Claude Code backgrounded the call at about 2 minutes and dropped it at about 5 minutes of silence.

Job length (exploratory)

The planned run at about 20-minute jobs did not happen: neither vendor could produce a legitimate 20-minute job at small vendor spend. In its place, within each fresh cell with n ≥ 8, the Spearman correlation of requests per wait with job length: polling arms rise with job length in 16 of 17 cells where the agent checks a status tool and the jobs varied (+0.43 to +0.99; the exception is web crawling in Claude Code, −0.44). Events stay at 3 to 5.5 requests across cells whose median jobs run 134 to 431 s. Nothing past about 11 minutes was measured.

200k tokens of earlier context (n = 2, values only)

The AI video generation and AI model runner Claude Code as-shipped cells at 200k were measured through round 1's buffering proxy and are not a fair baseline.

Server Harness Sign-in As shipped (arm) n Requests per wait Wait $ as shipped Long-poll $ (n) Events $ (n) Saved p Verdict Long-poll → events $ per conversation Asked A / L / B Never told A / L / B
Web crawling Claude Code subscription A1 2/2 17.0 → 4.0 0.826 (0.806–0.847) [0.7856 0.8670] – 0.206 (0.206–0.206) [0.2057 0.2063] 75% ($0.620; CI – to –) – n < 5: values only – 2.316 → 1.699 (27%, n<5) 1 / – / 0 0 / – / 0
Web crawling Codex subscription A1 2/2 25.5 → 4.0 0.527 (0.419–0.635) [0.3111 0.7434] – 0.105 (0.105–0.105) [0.1053 0.1054] 80% ($0.422; CI – to –) – n < 5: values only – 1.002 → 0.586 (42%, n<5) 0 / – / 0 0 / – / 0
Video editing Claude Code subscription A1 2/2 23.5 → 4.0 1.136 (1.129–1.143) [1.1222 1.1496] – 0.223 (0.223–0.224) [0.2230 0.2238] 80% ($0.912; CI – to –) – n < 5: values only – 2.638 → 1.722 (35%, n<5) 0 / – / 0 0 / – / 0
Video editing Codex subscription A1 2/2 16.0 → 3.0 0.349 (0.327–0.370) [0.3055 0.3916] – 0.090 (0.090–0.090) [0.0900 0.0901] 74% ($0.259; CI – to –) – n < 5: values only – 0.822 → 0.568 (31%, n<5) 0 / – / 0 0 / – / 0
Web scraping Claude Code subscription A1 2/2 11.0 → 4.0 0.652 (0.652–0.653) [0.6511 0.6536] – 0.210 (0.210–0.210) [0.2100 0.2105] 68% ($0.442; CI – to –) – n < 5: values only – 2.283 → 1.844 (19%, n<5) 0 / – / 0 0 / – / 0
Web scraping Codex subscription A1 2/2 16.0 → 3.0 0.384 (0.381–0.387) [0.3780 0.3899] – 0.095 (0.094–0.096) [0.0932 0.0972] 75% ($0.289; CI – to –) – n < 5: values only – 0.957 → 0.674 (30%, n<5) 0 / – / 0 0 / – / 0
Agent communication (email reply) Claude Code subscription A1 2/2 14.0 → 4.0 0.637 (0.594–0.679) [0.5520 0.7219] – 0.195 (0.195–0.195) [0.1945 0.1947] 69% ($0.442; CI – to –) – n < 5: values only – 2.148 → 1.709 (20%, n<5) 0 / – / 0 0 / – / 0
Agent communication (email reply) Codex subscription A1 2/4 13.0 → 4.5 0.282 (0.273–0.291) [0.2636 0.3001] – 0.129 (0.116–0.261) [0.1161 0.1164 0.1409 0.6213] 54% ($0.153; CI – to –) – n < 5: values only – 0.799 → 0.632 (21%, n<5) 0 / – / 0 0 / – / 0
CI builds Claude Code subscription A1 2/2 7.5 → 4.0 0.411 (0.388–0.433) [0.3651 0.4562] – 0.226 (0.223–0.230) [0.2198 0.2329] 45% ($0.184; CI – to –) – n < 5: values only – 2.195 → 1.909 (13%, n<5) 0 / – / 0 0 / – / 0
CI builds Codex subscription A1 2/2 12.5 → 3.0 0.270 (0.265–0.275) [0.2593 0.2808] – 0.088 (0.088–0.088) [0.0880 0.0880] 67% ($0.182; CI – to –) – n < 5: values only – 0.789 → 0.612 (22%, n<5) 0 / – / 0 0 / – / 0
AI video generation Claude Code subscription A1 ⚠ buffered baseline 2/2 7.0 → 4.0 0.328 (0.306–0.349) [0.2849 0.3706] – 0.190 (0.189–0.191) [0.1888 0.1910] 42% ($0.138; CI – to –) – n < 5: values only – 1.862 → 1.737 (7%, n<5) 0 / – / 0 0 / – / 0
AI video generation Codex subscription A1 2/2 11.5 → 3.0 0.240 (0.228–0.253) [0.2149 0.2660] – 0.083 (0.083–0.084) [0.0821 0.0842] 65% ($0.157; CI – to –) – n < 5: values only – 0.750 → 0.609 (19%, n<5) 0 / – / 0 0 / – / 0
AI model runner Claude Code subscription A1 ⚠ buffered baseline 2/2 4.5 → 4.0 0.247 (0.220–0.275) [0.1931 0.3016] – 0.195 (0.194–0.195) [0.1939 0.1953] 21% ($0.053; CI – to –) – n < 5: values only – 1.833 → 1.781 (3%, n<5) 0 / – / 0 0 / – / 0
AI model runner Codex subscription A1 2/2 6.0 → 4.0 0.136 (0.135–0.136) [0.1353 0.1359] – 0.119 (0.118–0.120) [0.1180 0.1201] 12% ($0.017; CI – to –) – n < 5: values only – 0.727 → 0.702 (3%, n<5) 0 / – / 0 0 / – / 0

Every cell

Sessions in the published data, per cell (wait in dollars with IQR; "Straddlers" are sessions that crossed a change of the Wakeshift version under test, kept and flagged).

Server Harness Job Context Arm n Wait $ (IQR) Requests $ per conversation Asked Never told Tell (s) Job median (s) Straddlers
Web crawling Claude Code 600 200k A1 2 0.8263 (0.8059–0.8467) 17.0 2.3164 1 0 20.985 150.984 1
Web crawling Claude Code 600 200k B1 2 0.206 (0.2058–0.2061) 4.0 1.6991 0 0 6.0735 149.393 0
Web crawling Claude Code 600 fresh A1 8 0.318 (0.302–0.3416) 21.0 0.418 4 0 39.034 148.6445 3
Web crawling Claude Code 600 fresh A2k 8 0.099 (0.0964–0.1016) 6.5 0.1503 0 0 26.623 42.1195 0
Web crawling Claude Code 600 fresh B1 8 0.0777 (0.0762–0.0926) 4.0 0.1876 0 0 6.7335 183.959 6
Web crawling Claude Code 600 fresh B2k 8 0.0629 (0.0626–0.068) 5.0 0.1187 0 0 12.0935 36.3075 0
Web crawling Claude Code 600 fresh L2 8 0.1174 (0.1063–0.1322) 6.5 0.2116 0 0 23.8475 129.0075 2
Web crawling Claude Code 45 fresh A1 6 0.0842 (0.063–0.1393) 3.0 0.1824 0 0 2.763 10.4295 0
Web crawling Claude Code 45 fresh B1 6 0.0878 (0.0861–0.089) 5.0 0.1902 0 0 6.908 16.793 0
Web crawling Codex 600 200k A1 2 0.5273 (0.4192–0.6353) 25.5 1.0024 0 0 17.6745 196.758 0
Web crawling Codex 600 200k B1 2 0.1053 (0.1053–0.1053) 4.0 0.5859 0 0 11.079 150.9745 0
Web crawling Codex 600 fresh A1 8 0.0777 (0.0732–0.1042) 17.0 0.1039 0 0 10.812 127.751 3
Web crawling Codex 600 fresh B1 8 0.0365 (0.0363–0.0418) 4.0 0.0721 0 0 9.923 148.028 3
Web crawling Codex 600 fresh L2 8 0.0588 (0.0532–0.0742) 10.0 0.0945 0 0 25.3325 121.111 2
Web crawling Codex 45 fresh A1 6 0.0334 (0.0302–0.0476) 1.0 0.0607 0 0 2.794 15.953 0
Web crawling Codex 45 fresh B1 6 0.0429 (0.0429–0.0429) 4.0 0.069 0 0 9.8595 34.1205 0
Video editing Claude Code 600 200k A1 2 1.1359 (1.129–1.1427) 23.5 2.6382 0 0 47.469 287.4465 0
Video editing Claude Code 600 200k B1 2 0.2234 (0.2232–0.2236) 4.0 1.7216 0 0 3.724 217.6245 2
Video editing Claude Code 600 fresh A1 8 0.3079 (0.2929–0.3368) 23.0 0.4189 0 0 42.624 306.2345 2
Video editing Claude Code 600 fresh A2k 8 0.1654 (0.1544–0.1792) 18.0 0.2225 0 0 31.954 255.3445 1
Video editing Claude Code 600 fresh B1 10 0.0893 (0.089–0.0908) 4.0 0.1971 0 0 3.6085 277.572 2
Video editing Claude Code 600 fresh B2k 8 0.0711 (0.0689–0.0723) 5.0 0.1293 0 0 3.157 231.9675 0
Video editing Codex 600 200k A1 2 0.3486 (0.327–0.3701) 16.0 0.8218 0 0 22.7205 248.341 0
Video editing Codex 600 200k B1 2 0.09 (0.09–0.0901) 3.0 0.5684 0 0 5.2005 276.002 2
Video editing Codex 600 fresh A1 8 0.1018 (0.0866–0.1107) 17.0 0.1249 0 0 17.474 269.4835 4
Video editing Codex 600 fresh B1 10 0.0381 (0.038–0.056) 3.0 0.0662 0 0 4.6225 238.388 4
Web scraping Claude Code 600 200k A1 2 0.6524 (0.6518–0.653) 11.0 2.2829 0 0 10.758 364.441 0
Web scraping Claude Code 600 200k B1 2 0.2103 (0.2101–0.2104) 4.0 1.844 0 0 6.853 396.3315 0
Web scraping Claude Code 600 fresh A1 8 0.2803 (0.2304–0.3108) 12.0 0.4874 0 0 9.9525 400.3745 4
Web scraping Claude Code 600 fresh A2 8 0.3308 (0.3066–0.3744) 14.5 0.522 0 0 12.9965 561.7045 0
Web scraping Claude Code 600 fresh A2k 8 0.1951 (0.1382–0.2164) 10.0 0.3142 0 0 13.5725 297.5625 0
Web scraping Claude Code 600 fresh B1 9 0.0802 (0.0768–0.1639) 4 0.2888 0 0 7.099 430.69 4
Web scraping Claude Code 600 fresh B2k 8 0.0896 (0.0644–0.1121) 6.0 0.1996 0 0 10.0405 263.409 0
Web scraping Claude Code 45 fresh A1 5 0.0324 (0.0322–0.0329) 2 0.1569 0 0 3.352 10.95 0
Web scraping Claude Code 45 fresh B1 5 0.0334 (0.0333–0.0341) 2 0.1629 0 0 3.159 11.263 0
Web scraping Codex 600 200k A1 2 0.384 (0.381–0.3869) 16.0 0.9566 0 0 20.3225 323.0445 0
Web scraping Codex 600 200k B1 2 0.0952 (0.0942–0.0962) 3.0 0.6744 0 0 10.2365 510.466 0
Web scraping Codex 600 fresh A1 8 0.1397 (0.116–0.1536) 19.0 0.2287 0 0 17.728 424.1195 3
Web scraping Codex 600 fresh B1 9 0.0454 (0.0412–0.0462) 3 0.1455 0 0 11.798 407.045 5
Web scraping Codex 600 fresh L2 8 0.0891 (0.0845–0.0925) 15.0 0.1827 0 0 12.956 288.946 2
Web scraping Codex 45 fresh A1 5 0.0146 (0.0145–0.0147) 2 0.1064 0 0 3.645 10.46 0
Web scraping Codex 45 fresh B1 5 0.0423 (0.0422–0.0425) 3 0.1351 0 0 7.998 8.445 0
Agent communication Claude Code 180 200k A1 2 0.6369 (0.5944–0.6794) 14.0 2.1481 0 0 59.6415 199.0105 0
Agent communication Claude Code 180 200k B1 2 0.1946 (0.1946–0.1946) 4.0 1.7091 0 0 4.5335 184.3805 1
Agent communication Claude Code 180 fresh A1 8 0.2026 (0.1938–0.2055) 15.0 0.3229 0 0 105.799 184.096 2
Agent communication Claude Code 180 fresh B1 9 0.0594 (0.059–0.0596) 4 0.1757 1 0 4.8895 183.763 3
Agent communication Codex 180 200k A1 2 0.2819 (0.2727–0.291) 13.0 0.7994 0 0 16.269 189.1995 0
Agent communication Codex 180 200k B1 4 0.1287 (0.1163–0.261) 4.5 0.6323 0 0 9.9255 184.623 2
Agent communication Codex 180 fresh A1 8 0.0849 (0.0559–0.1038) 12.5 0.136 0 0 21.1345 184.0235 4
Agent communication Codex 180 fresh B1 8 0.0382 (0.0342–0.0421) 3.5 0.0945 0 0 7.7995 183.971 1
CI builds Claude Code 240 200k A1 2 0.4106 (0.3878–0.4334) 7.5 2.1946 0 0 3.1595 242.5695 0
CI builds Claude Code 240 200k B1 2 0.2263 (0.2231–0.2296) 4.0 1.909 0 0 6.5455 244.7425 0
CI builds Claude Code 240 fresh A1 8 0.1387 (0.1345–0.1479) 7.0 0.3523 0 0 4.7705 243.718 0
CI builds Claude Code 240 fresh B1 11 0.0974 (0.0927–0.1199) 4 0.3467 0 0 7.469 243.471 3
CI builds Claude Code 20 fresh A1 5 0.0587 (0.0567–0.0629) 2 0.3622 0 0 3.894 23.05 5
CI builds Claude Code 20 fresh B1 10 0.1184 (0.0936–0.1427) 5.0 0.3647 0 0 5.778 22.8385 5
CI builds Codex 240 200k A1 2 0.27 (0.2646–0.2754) 12.5 0.7886 0 0 4.822 243.4605 0
CI builds Codex 240 200k B1 2 0.088 (0.088–0.088) 3.0 0.6116 0 0 8.96 244.7945 0
CI builds Codex 240 fresh A1 8 0.0881 (0.0602–0.1011) 13.0 0.1309 0 0 5.26 244.112 1
CI builds Codex 240 fresh B1 8 0.036 (0.0359–0.0365) 3.0 0.0907 0 0 8.0125 243.289 1
CI builds Codex 20 fresh A1 5 0.0168 (0.0152–0.0199) 2 0.0839 0 0 4.686 22.289 5
CI builds Codex 20 fresh B1 10 0.0398 (0.0361–0.0646) 3.0 0.1092 0 0 7.7935 23.877 5
AI video generation Claude Code 300 200k A1 2 0.3277 (0.3063–0.3491) 7.0 1.8618 0 0 3.211 198.1675 1
AI video generation Claude Code 300 200k B1 2 0.1899 (0.1894–0.1905) 4.0 1.7369 0 0 5.784 208.0905 0
AI video generation Claude Code 300 fresh A1 8 0.1001 (0.0857–0.1059) 7.5 0.2107 0 0 3.024 198.0165 2
AI video generation Claude Code 300 fresh A2 8 0.0456 (0.0454–0.0458) 4.0 0.1456 0 0 1.3945 210.632 0
AI video generation Claude Code 300 fresh B1 8 0.0577 (0.0571–0.0581) 4.0 0.1804 0 0 5.2065 194.559 3
AI video generation Claude Code 300 fresh L2 8 0.0469 (0.0466–0.0476) 4.0 0.1471 0 0 1.7845 209.2645 4
AI video generation Claude Code 30 fresh A1 5 0.0127 (0.0127–0.0129) 1 0.1129 0 0 1.055 15.023 0
AI video generation Claude Code 30 fresh B1 5 0.0133 (0.0132–0.0133) 1 0.1178 0 0 2.027 14.515 0
AI video generation Codex 300 200k A1 2 0.2405 (0.2277–0.2532) 11.5 0.7495 0 0 3.6045 200.9415 1
AI video generation Codex 300 200k B1 2 0.0832 (0.0827–0.0837) 3.0 0.6086 0 0 7.1225 201.009 1
AI video generation Codex 300 fresh A1 8 0.0461 (0.045–0.0595) 11.0 0.0964 0 0 2.5155 214.8295 6
AI video generation Codex 300 fresh B1 12 0.0325 (0.0322–0.0332) 3.0 0.0828 0 0 7.6065 216.412 7
AI video generation Codex 300 fresh L2 8 0.0715 (0.064–0.0817) 16.0 0.1117 0 0 3.8725 218.556 3
AI video generation Codex 30 fresh A1 5 0.0045 (0.0043–0.0048) 1 0.0451 0 0 2.633 18.993 0
AI video generation Codex 30 fresh B1 5 0.0371 (0.0371–0.0372) 4 0.0815 0 0 9.178 20.721 0
AI model runner Claude Code 600 200k A1 2 0.2474 (0.2202–0.2745) 4.5 1.8329 0 0 4.317 144.066 0
AI model runner Claude Code 600 200k B1 2 0.1946 (0.1943–0.1949) 4.0 1.7813 0 0 6.393 127.339 0
AI model runner Claude Code 600 fresh A1 8 0.0853 (0.0846–0.1002) 5.0 0.2385 0 0 2.8125 153.8425 2
AI model runner Claude Code 600 fresh A2 8 0.0535 (0.0532–0.1334) 4.0 0.1964 0 0 2.859 209.966 2
AI model runner Claude Code 600 fresh B1 9 0.0631 (0.0616–0.0642) 4 0.2282 0 0 5.189 139.226 2
AI model runner Claude Code 600 fresh E2 8 0.1012 (0.0538–0.161) 7.0 0.2427 0 0 2.7075 292.044 1
AI model runner Claude Code 45 fresh A1 5 0.0191 (0.0183–0.0191) 1 0.1599 0 0 3.25 12.571 0
AI model runner Claude Code 45 fresh B1 5 0.061 (0.0605–0.0628) 4 0.2169 0 0 6.898 16.999 0
AI model runner Codex 600 200k A1 2 0.1356 (0.1354–0.1358) 6.0 0.7265 0 0 3.809 120.252 1
AI model runner Codex 600 200k B1 2 0.1191 (0.1185–0.1196) 4.0 0.7018 0 0 10.9395 200.667 1
AI model runner Codex 600 fresh A1 7 0.0443 (0.0417–0.05) 8 0.1353 0 0 5.897 173.927 2
AI model runner Codex 600 fresh A2 8 0.1155 (0.0944–0.1337) 13.0 0.2149 0 0 6.5695 268.177 1
AI model runner Codex 600 fresh B1 11 0.0464 (0.0449–0.0518) 4 0.1355 0 0 10.676 134.435 2
AI model runner Codex 600 fresh E2 8 0.0647 (0.0359–0.1059) 8.5 0.1731 0 0 5.041 163.5015 0
AI model runner Codex 45 fresh A1 5 0.0064 (0.0061–0.0073) 1 0.0923 0 0 3.609 16.547 0
AI model runner Codex 45 fresh B1 5 0.042 (0.0399–0.0449) 3 0.1326 0 0 7.81 17.537 0
Crawling API that holds the call Claude Code 480 fresh A1 8 0.0474 (0.0469–0.0488) 3.0 0.166 8 8 – 403.248 4
Crawling API that holds the call Claude Code 480 fresh A2 8 0.094 (0.0918–0.0941) 6.0 0.1999 8 8 – 385.422 0
Crawling API that holds the call Claude Code 480 fresh B1 8 0.0965 (0.0957–0.0972) 5.0 0.2185 1 0 12.125 389.1885 3
Crawling API that holds the call Codex 480 fresh A1 7 0.1407 (0.124–0.1711) 22 0.1874 7 6 – 401.998 2
Crawling API that holds the call Codex 480 fresh B1 7 0.0317 (0.0314–0.0318) 3 0.0828 0 0 10.773 409.173 3
Research API Claude Code 300 fresh A1 8 0.1789 (0.1516–0.1841) 4.5 0.2831 0 0 27.709 297.4005 3
Research API Claude Code 300 fresh B1 8 0.2184 (0.1911–0.2404) 5.5 0.3268 2 0 37.305 282.785 3
Research API Codex 300 fresh A1 8 0.114 (0.1047–0.1197) 21.0 0.1477 4 4 43.6975 249.767 3
Research API Codex 300 fresh B1 8 0.0763 (0.0709–0.0939) 3.0 0.111 0 0 37.384 274.7615 2
Held-call test server Claude Code 900 fresh A2 6 0.075 (0.0561–0.2101) 8.0 0.1549 3 0 0.054 900.9145 0
Held-call test server Claude Code 900 fresh A2k 6 0.0951 (0.0767–0.1674) 8.5 0.1416 3 0 0.451 901.2265 3
Held-call test server Claude Code 300 fresh A2 6 0.0971 (0.0551–0.1442) 10.0 0.1733 0 0 38.0335 300.5165 0
Held-call test server Claude Code 300 fresh A2k 6 0.0688 (0.0512–0.1131) 9.0 0.1128 2 0 0.9975 300.148 0
Held-call test server Claude Code 60 fresh A2 6 0.0239 (0.0195–0.0284) 2.5 0.1005 0 0 0.1765 60.3955 0
Held-call test server Claude Code 60 fresh A2k 6 0.0183 (0.0142–0.0215) 2.5 0.0627 0 0 0.379 60.7845 3
Held-call test server Codex 900 fresh A2 6 0.223 (0.2031–0.2512) 37.5 0.264 3 0 24.928 904.3515 0
Held-call test server Codex 300 fresh A2 6 0.104 (0.0909–0.1195) 19.5 0.1407 0 0 4.0225 300.0045 0
Held-call test server Codex 60 fresh A2 6 0.0223 (0.0204–0.0241) 6.0 0.0599 0 0 1.592 60.005 0

Study 2: External events

Design

Pre-registered as design v3.1, commit beb1b5c (tag prereg-v1), 2026-11-14 05:47 UTC, before any session.

Test servers. Each copies one real server's tool names, titles, descriptions, input schemas, annotations and server instructions word for word: an inbound email service, a code review service and an error alert service. Responses come from real captures (email) or the API documentation's examples (the other two). Output schemas and icons are not served; neither harness is known to put them in the model's context. The tool lists are checked word for word against the captures (1,047 checks). One server's license keeps its text private, so for it we publish the extraction script and content hashes.

Tasks. The blocked task (T1): reply to customers as they write back, over a 2-hour window, with releases on 10 schedules drawn in advance from a fixed seed, none in the last 20 minutes; at the end the user asks once whether anything came in. Two schedules drew no replies (known before any session), so 6 schedules carry the 45 replies each approach had to answer. The three-stream task (T3): reviews on three pull requests, an inbox and an error feed for the same two hours, exploratory.

Approaches (arms). G0, the server as shipped, silent about waiting. G3, a long-poll read tool holding each call up to 50 s (Claude Code) or 280 s (Codex) while sending progress. S, the user's 15-minute check: /loop 15m in Claude Code; in Codex, which has no scheduler, a plain request to check every 15 minutes. G4, a wait tool held open until the next event while sending progress every 15 s (Claude Code only; Codex cuts every call at 300 s). G5, events through Wakeshift (plugin v1.4.54; Codex connector 2026.11.14.1515). The screen also ran G1 ("poll every 5 to 10 seconds", from another product's server), G2 ("check back in about 10 minutes") and GB (a wording written by an agent blind to the study).

Measures. A reply is missed when the agent handled it only after the user asked, or never. Idle cost is model spend per hour outside acting spans, where an acting span runs from an item's first fetch to the earlier of the turn's end and the agent's next check after acting on it. Every approach ran on every schedule in the same wave.

Primary tests. P1 to P5 compare each approach with events on the same schedules: an exact one-sided sign-flip test on the sum of paired differences (P5 two-sided), Holm-corrected over the five. A test detects a difference only when its Holm-adjusted rejection and a 95% interval excluding zero agree. Everything else is exploratory.

The bar (falsifiers). F1 (as shipped, long-poll): at most $0.10 and 10 requests an idle hour more than events, and at most a fifth of replies missed. F2 (held call): at most $0.05 an hour more than events, and at most 5 points more missed. F3 (the 15-minute check): F1's cost lines, at most a tenth missed and 90% handled within 15 minutes. F4 (events don't deliver): events miss more than a tenth of replies in any harness and task, or cost more idle money than as shipped in 2 or more of the 4 harness and task cells. F5: the test server must match the real email server within ±30% on cost per check and checks per hour. An approach that clears its bar falsifies the claim that agents need events for this kind of wait.

Runaway rule and caps. Each session has a cost cap. The runaway rule stops a session looping at 10 s or less (the median gap of its last 10 requests) whose trailing 10-minute spend rate is above $4 an hour (Claude Code) or $2 (Codex). Replies after a stop are unobserved and excluded.

Stages. A screen of 104 one-hour sessions on a pre-release Wakeshift version, varying the wording; the main stage of 180 kept sessions on the released version; and the real-server check (Part C) of 16 pairs. Total spend $327.57 at list price, the API-key run included.

The combined bar (blocked task)

Missed share is the median over the 6 schedules with replies; extras are median paired differences against G5. Absolute medians (idle $/h, idle requests/h): Claude Code G0 0.69 / 32.9, G3 0.07 / 5.2, S 0.33 / 15.0, G4 0.10 / 4.8, G5 0.07 / 2.5; Codex G0 0.08 / 11.5, G3 0.16 / 22.2, S 0.11 / 13.0, G5 0.015 / 2.0.

Harness Arm Extra idle $/h vs G5 Extra idle requests/h Missed share Events missed Handled ≤ 5 / ≤ 15 min Bar
Claude Code G0 as shipped (silent) +0.59 +27.6 0.00 0/45 38 / 40 of 45 fails: cost (F1)
Claude Code G3 long-poll, 50 s +0.04 +3.5 0.96 39/45 6 / 6 of 45 fails: misses (F1)
Claude Code S /loop 15m +0.24 +13.2 0.00 0/45 10 / 44 of 45 fails: cost (F3; 98% within 15 min)
Claude Code G4 held call +0.01 +1.2 0.00 0/45 45 / 45 of 45 passes (F2 met)
Claude Code G5 events 0 0 0.00 0/45 45 / 45 of 45 (reference)
Codex G0 as shipped (silent) +0.05 +9.5 0.88 22/45 23 / 23 of 45 fails: misses (F1)
Codex G3 long-poll, 280 s +0.15 +20.5 0.88 21/33* 12 / 12 of 33 fails: both (F1)
Codex S "every 15 min" +0.08 +11.0 0.94 28/34* 6 / 6 of 34 fails: misses (F3)
Codex G5 events 0 0 0.00 0/45 45 / 45 of 45 (reference)

* Codex G3 and S stopped early in 2 and 1 sessions: one G3 session at its cost cap, the other G3 session and the S session by the runaway rule (12 and 11 replies unobserved). Counting every unobserved reply as handled, they missed 21 and 28 of 45. In Codex as shipped, 8 of 10 sessions stopped checking within about 17 minutes; the other two watched for more than 80 minutes and missed 1 reply between them. Events in Codex include 24 replies released more than an hour after the subscribe.

The primary tests

Test Measure Contrast Harness n Median diff 95% CI (bootstrap; exact) p Holm threshold Detected
P1 idle $/h G3 − G5 Claude Code 10 +$0.038 [−0.071, 0.051]; [−0.042, 0.048] 0.46 0.025 no
P2 idle $/h G3 − G5 Codex 10 +$0.147 [0.090, 0.642]; [0.096, 0.838] 0.001 0.010 yes
P3 missed share G0 − G5 Claude Code 6 0.00 [0, 0]; [0, 0] 1.00 0.050 no
P4 missed share G0 − G5 Codex 6 +0.875 [0.06, 1.00]; [0.005, 0.995] 0.031 0.0125 no (effect large; misses Holm at n = 6)
P5 idle $/h G4 − G5 (two-sided) Claude Code 10 +$0.011 [−0.001, 0.099]; [−0.009, 0.089] 0.11 0.017 no (near tie)

Sensitivities. Under the design's literal acting-span rule: P1 +$0.006 (p 0.76), P2 +$0.147 (p 0.001), P5 +$0.011 (p 0.25), no change in any decision. Bounds on missed share for stopped sessions leave P3 and P4 unchanged. Without the 51 sessions that straddled a production deploy, P2 has 6 schedules, +$0.108 (p 0.016, all 6 in one direction) and can't clear its threshold, so its detection depends on keeping them, as the design specifies; the others without straddlers: P1 n 6, +$0.006 (p 0.61); P3 n 4, 0; P4 n 4, +1.0 (p 0.06); P5 n 6, +$0.004 (p 1.0). With 6 schedules, the smallest attainable p is 1/64, above P4's threshold of 0.0125 once only P2 had been rejected.

The outcome under the pre-registration: P2 detected; P1, P3, P4, P5 not; F2 met in Claude Code on the blocked task, meaning the held call passed its bar; F1, F3, F4 not met; F5 met, meaning the test-server check failed. The registered statements that apply: holding the call ties or wins in Claude Code, so events are framed as the answer for Codex, for several streams and for clients that can't hold a call; mixed results, so claims are per harness and task; and the test server diverges from the real one, so its numbers are comparisons within the test server.

Total spend per session on the blocked task (median, subscription, idle and acting time together): Claude Code as shipped $1.74, long-poll $0.47, /loop 15m $0.89, held call $0.37, events $0.37; Codex as shipped $0.25, long-poll $0.40, the 15-minute request $0.30, events $0.14. The registered measure is idle cost.

Three streams (exploratory)

Harness Arm Idle $/h Idle requests/h Missed ≤ 15 min Extra idle $/h vs G5 [95% CI]
Claude Code G0 1.37 39.1 0/45 45 +1.29 [1.10, 1.35]
Claude Code G3 1.34 42.9 2/45 43 +1.24
Claude Code S 0.48 18.1 0/45 45 +0.41 [0.35, 0.50]
Claude Code G4 0.15 4.8 3/17 14 +0.06 (3 runaway stops)
Claude Code G5 0.09 3.5 0/45 45 –
Codex G0 0.89 120.4 0/45 45 +0.87 [0.77, 0.95]
Codex G3 0.55 73.0 2/45 43 +0.51
Codex S 0.54 74.2 2/45 43 +0.51 [0.48, 0.58]
Codex G5 0.024 2.3 0/45 45 –

Polling across three streams cost 5.2 to 36.9 times events' idle cost (Claude Code G0 14.8×, G3 14.5×, S 5.2×; Codex G0 36.9×, G3 22.6×, S 22.3×); the held call 1.6×. Ten polling sessions stopped at their cost cap (Claude Code G0 1, G3 4; Codex G0 5), and their cost stops counting there, which favors polling. The held call needs one call per stream; in 3 of 10 sessions the agent called the wait tools under two seconds apart and the runaway rule stopped them within four minutes, so 17 of its 45 releases were observed.

The screen (one-hour sessions, exploratory)

8 sessions per wording and harness. G2 ("check back in about 10 minutes") kept both harnesses reliable: Claude Code 1 of 12 missed, Codex 0 of 11. G1 ("poll every 5 to 10 seconds") missed 6 of 11 in each harness, and one Codex session polled 815 times before its cost cap. GB, the blind wording that told the agent not to check in a tight loop and to end its turn, missed 7 of 12 in Claude Code and 11 of 11 in Codex. G4 missed 0 of 11; G5 1 of 12 and 0 of 12.

The real-server check (F5)

Harness Level Pairs Cost per check real / toy Checks per hour real / toy Pass
Claude Code G0 4 $0.029 / $0.027 7.5 / 3.0 no (checks)
Claude Code G2 4 $0.035 / $0.041 7.5 / 6.5 yes
Codex G0 4 $0.012 / $0.008 17.5 / 12.5 no (both, by ~40%)
Codex G2 4 $0.015 / $0.014 14.0 / 16.0 yes

As shipped, agents checked the real server more often than the test server; with G2's wording they matched. With 4 pairs per cell this is a check, not a measurement. It covers only the email service, one-hour sessions and those two wordings, and it compared checking, not missed replies.

The cost projection (illustration only)

A model projecting cost from the measured per-check costs and intervals was to be validated against the measured cells within ±25%. It missed in 14 of 14 cells on the first attempt (using each cell's median gap between checks) and 13 of 14 on the second (using each cell's effective interval, which reflects giving up); the input changed between attempts. Under the registered rule it is an illustration only. Measured inputs: per check on a warm cache, Claude Code adds about 960 tokens and writes about 180; Codex about 230 and 50; about 1.1 to 1.2 requests per check.

The API key (exploratory)

First declined (deviation 32), then run after the primary results were known, at our request: Claude Code, the blocked task, schedules 1 to 4, n = 4 per approach. Effect sizes only.

Arm Idle $/h, API key Idle $/h, subscription (same 4 schedules) Idle requests/h (API) $ per session (API) Missed ≤ 5 / ≤ 15 min
G0 as shipped 0.50 0.69 22.2 2.14 1/24 23 / 23
S /loop 15m 0.55 0.31 13.4 1.80 0/24 9 / 23
G4 held call 0.17 0.11 5.8 0.68 0/24 24 / 24
G5 events 0.17 0.11 4.3 0.80 0/24 24 / 24

Paired against events on the API key: /loop 15m +$0.40 an hour (all 4 schedules positive), held call +$0.018 (per schedule −$0.03 to +$0.03), as shipped +$0.27 (one schedule −$0.04, another +$1.37).

Behavior coding and context

Two model coders (Sonnet 5.5 and GPT-6.1 Sol, masked to the approach) labeled each session; a label counts only when both agree. Agreement (κ) on the main stage: asked the user 0.61, claimed done early 0.53, mentioned a wake 0.83, false all-clear 0.65. On the blocked task, Claude Code's long-poll asked the user to tell it about replies in 5 of 10 sessions; events sessions never did. No polling session carried Wakeshift's note or files. Wakeshift's plugin adds about 120 to 220 tokens to an events session's first request.

Drift and environment

Per-wave medians are stable across the main stage's waves, apart from the held call's runaway-driven outliers on three streams. Production deployed twice during the main stage, so 51 sessions straddle a deploy; they are kept and flagged, as the design specifies for production. The events arm used three production apps, one per surface.

Deviations

Study 1

Each was written before the data it affects was seen, except where noted.

  1. One server was swapped out of the API-key and long-poll sets before round 2, because its backend was no longer available: the long-poll set used the agent communication server instead, and the API-key set a server later removed from the study.

  2. The Wakeshift version under test was pinned at the round's first run (v1.4.50); later moves are recorded below, with no cell mixing versions.

  3. The AI model runner's equalized arm gains one sentence from the events branch: "Leave wait and timeout_seconds at their defaults."

  4. The runner's proxy had its own 300-second timeout in round 1; from round 2 it has none. Which round-1 sessions it could have cut is checked in the re-scoring.

  5. The version moved to v1.4.51 mid-round. Plugin and connector files were unchanged; treated as a boundary, and cells compared across it are labeled.

  6. The web scraping account's monthly limit allowed long-poll only in Codex. Its Claude Code as-shipped arm already waits 45 s in the server, which is what the long-poll does.

  7. Three held-call cells were rerun on fresh test servers after bench failures (a sign-in failure and stale reports from an earlier run).

  8. The agent communication server's long-poll never engaged (its "job" is an inbox, not a job ID); its 4 sessions are a bench failure, and its long-poll was not measured.

  9. The API-key set was cut short for the draft's deadline; that partial run belonged to the removed server and is dropped with it.

  10. The long-hold extension (30, 60 and 120 minutes) was added after the 15-minute result, registered before its runs, and ran nine sessions at once on separate servers, on v1.4.52 (plugin files unchanged).

  11. Two extensions were registered on 2026-11-14 before their sessions, after the main results: the API-key cache check, run in full, and longer jobs, not run. During it the web crawling vendor's crawls became 3 to 4 times faster than in rounds 1 and 2 (checked on sites the bench had never crawled), so that cell barely tests the cache. The web scraping events arm was rerun after a bench fault (no job started), and the version moved to v1.4.54 and then v1.4.55 mid-extension; the version of every session is recorded.

  12. The infrastructure-failure rule (2026-11-16, after the results; our decision): sessions whose failure was caused by infrastructure unrelated to the approach are excluded, in every arm and both studies. It removed four study-1 events sessions hit by the Wakeshift server outage of 2026-11-12 (research API cc-b-300-2 and cc-b-300-3, crawling API cc-b-480-4, email replies cc-b-180-1, all Claude Code). Effect: in Claude Code the research API's events cost moves from 22% more to 14% more, the crawling API's stays 3% more, and in-scope events asks go from 3 of 127 to 0 of 124. Details and evidence are under "Sessions excluded for infrastructure failures".

  13. The conclusion's scope (2026-11-16, after the results; our decision): out are servers that already hold the request open and send progress on tasks that reliably finish in under about five minutes (AI video generation and the AI model runner in Claude Code); the research API and the silent crawling API in Claude Code are in.

  14. The fixed-bug rule (2026-11-16, after the results; our decision): sessions affected by a bug we found and fixed during the study, in Wakeshift, its connector, an integration or our test runner, are excluded in every arm, since they measure the bug rather than the approach. Sessions that ran before a fix but that the bug couldn't have changed are kept: for example, Codex sessions that ran before the test runner's proxy timeout was removed, since Codex cuts every call at 300 s on its own. The rule removed 7 Codex events sessions on the web scraping server, run before a fix to Wakeshift's Codex connector (it didn't accept hyphenated tool names): in all 7 the user had to ask, and 6 never got the result.

Analysis notes from the results: round 2's as-shipped arm replaces round 1's without pooling; tests sort each arm's values first (intervals and p can differ in the third decimal from session order; medians don't); the never-told pattern was widened to catch "I don't know … page count"; job length is reported, not normalized; the API-key rows are kept out of the main table.

Corrections after the first round. Reviews found that the runner's proxy buffered servers' progress updates, which cut off waits that stream progress. After the fix and a rerun, round 1's 42% saving on AI video generation disappeared (events now cost 27% more in Claude Code) and its 26% saving on the AI model runner was no longer supported. The one server removed after the analysis is recorded as a deviation in the published method.

Study 2

Entries 1 to 17 were made while building, before any session ran, and are committed with the pre-registration.

  1. A schedule draw before 2 minutes is moved to 2 minutes (affects two schedules).

  2. Two main-stage schedules drew no replies. Nothing was redrawn. P3 and P4 have n = 6, so the smallest one-sided p is 1/64, above Holm's first threshold; they can be detected only if two other tests are rejected first.

  3. The "poll every 5 to 10 seconds" wording is adapted to each surface.

  4. The wordings change each surface's two list or check tools; fetch-by-ID tools are unchanged.

  5. What the test servers serve (names, titles, descriptions, schemas, annotations, instructions; not icons or output schemas).

  6. Default tool sets follow each server's own list; triage and deprecated tools are excluded for the error alert service.

  7. The prompts were revised after an outside reader, before commit.

  8. The 15-minute request is filled per task and sent once the first turn ends.

  9. The held call's and events' tool wordings are adapted per surface from the email service's wording.

  10. Wake texts carry IDs only, so acting needs a fetch.

  11. The screen's balanced table of tasks, schedules and wordings.

  12. Idle time and "gave up" as implemented.

  13. The runaway rule's test.

  14. A wave's worst-case cost counts polling lanes at their caps.

  15. Isolation additions for each session's environment.

  16. The two model coders and their shared prompt.

  17. The real-server check's inbox and setup.

  18. The blind wording's writer got an abridged brief in places.

  19. The real-server check ran one real lane at a time (no spare inbox), roughly doubling its wall time.

  20. The screen's events plugin could not be pinned to a commit and was logged per wave instead.

  21. Before any study data, the end of an acting span was changed: Codex often polls inside one long turn, so "to the end of that turn" left almost no idle time. The literal rule is reported as a sensitivity (P1 +$0.006 instead of +$0.038; no decision changes).

  22. How subscription usage is read.

  23. The sign-flip test's statistic is the sum of paired differences, which the design's power statement assumes.

  24. Pre-flight found that Wakeshift's Codex connector stopped listening after an hour without a turn. The screen's releases ended inside the hour; the main stage waited for a released fix.

  25. In the screen, the three-stream events sessions used one Wakeshift app for three servers, which Wakeshift refuses; two sessions were discarded and rerun, and the error alert surface then used another vendor's existing test app, a confound limited to that cell.

  26. The main stage's events arm used three production apps, one per surface; production deploys during the main stage are flagged, not excluded (51 sessions).

  27. An unrelated built-in tool server appeared intermittently in Claude Code's tool list; the tool-list guard failed those sessions (8 three-stream sessions and one real-server pair), and they were rerun with it disabled.

  28. A runner bug aborted one wave (no data, $9.50 counted in spend).

  29. The pre-registered cut order was applied at $228.87 spent: optional sets, the remaining real-server pairs, and later three-stream blocks were cut. The blocked task's data, and P1 to P5, were complete.

  30. About an hour after P1 to P5 were computed (04:25 UTC on November 15), we raised the cap from $300 to $600 (about 05:20 UTC) and the cuts were reversed: the remaining three-stream blocks ran 3 to 8 hours after the first waves, and the real-server pairs ran. These runs were decided after the main result was known.

  31. A descriptive interim look at the main stage, at our request, while runs continued: medians and counts only, no tests. Nothing in the plan changed.

  32. The optional sets were first declined.

  33. The cost projection's validation input was changed after a first failure (14 of 14 cells off); it failed again (13 of 14).

  34. The API-key run (optional set 1) was run after all, after the primary results, at our request. Exploratory; changes no test.

  35. The infrastructure-failure rule (2026-11-16, after the results; our decision; applied to both studies): sessions whose failure was caused by infrastructure unrelated to the approach are excluded, in every arm. It removed no study-2 session; P1 to P5 are unchanged. The sessions examined are listed in the appendix section on infrastructure failures.

Correction C1 (from the earlier skeptic review of the results files): two of the three early stops on the blocked task were by the runaway rule, not the cost cap; "none of the straddlers change a conclusion" was wrong, since P2 depends on them; and the three-stream range "5 to 40 times" was replaced by the computed 5.2 to 36.9.

Wake your users' agents when your events happen

Define your events and call emit, and Wakeshift wakes the agents your users already run, Claude Code and Codex included.