Coding-agent harness benchmarks

Compares coding-agent harnesses while holding the model and task fixed. Each board is one result-sealed bundle.

8 harnesses4 models5 result-sealed bundles771 valid result rows699 matched result rowsupdated 2026-07-21

Hide boards with disclosed caveats

Scores are not comparable across bundles: task sets, trial counts, and timeout caps differ.

OpenBench: gpt-5.6 across 7 harnesses

kind releasedate 2026-07-21denominators matched (task, trial)matched rows 294common task/trials 42results SHA 546f167576aatask-set SHA d949d46e8aae

gpt-5.6-solcaveats disclosed

1 caveat(s) from the release page

Archived task definitions differ from or are unavailable in the current checkout for add-feature, make-ci-green, taskflow, terminal-bench/schemelike-metacircular-eval, webcore. The result seal and bundled evidence still verify; recorded task digests remain the run-time fingerprints.

Harness↕	Solve rate · Wilson 95%↓ 050100%	Solved↕	Median wall↕	Fresh tokens/solve↕	Uncached input/solve↕	Output/solve↕	Cache-read/solve↕	Cache-write/solve↕	Telemetry source / basis	Telemetry coverage
cursor	85.7% 72.2%–93.3%	36/42	42.9s	5,519	72	5,446	407,067	45,776	nativeharness_reported	proxy 0/42 · native 42/42
devin	81.0% 66.7%–90.0%	34/42	68.4s	—	—	—	—	—	unavailable	proxy 0/42 · native 0/42
grokbuild	81.0% 66.7%–90.0%	34/42	86.3s	46,216	38,727	7,489	368,079	0	proxyproxy_measured	proxy 42/42 · native 42/42
opencode	81.0% 66.7%–90.0%	34/42	60.5s	42,634	36,708	5,927	313,916	0	nativevendor_split	proxy 0/42 · native 42/42
claude	76.2% 61.5%–86.5%	32/42	58.8s	40,255	34,455	5,800	153,488	0	proxyproxy_measured	proxy 42/42 · native 42/42
pi	76.2% 61.5%–86.5%	32/42	39.7s	37,602	32,916	4,686	111,408	0	proxyproxy_measured	proxy 42/42 · native 42/42
codex	73.8% 58.9%–84.7%	31/42	94.6s	117,107	102,174	14,933	1,164,511	0	proxyproxy_measured	proxy 42/42 · native 39/42

results.jsonlrelease page

OpenBench: deepseek across 5 harnesses

kind releasedate 2026-07-20denominators matched (task, trial)matched rows 185common task/trials 37results SHA ec6b9d58fd3atask-set SHA 1c4805e7fcd0

deepseek-v4-flashcaveats disclosed

5 caveat(s) from the release page

Archived first-party matrix published without rerunning models. The source contains 225 rows across 5 harnesses, 15 tasks, and 3 trials.
The bundled results SHA is sealed. Current-tree task checks report expected post-run definition drift; recorded task digests are retained as run-time fingerprints.
The unified board ranks only task/trial cells shared by every harness. The historical release page retains its older per-arm countable view, so its denominators differ.
Harness and host/container version stamps vary in the archive. Token columns use the complete counting-proxy lane for every displayed harness.
Archived task definitions differ from or are unavailable in the current checkout for add-feature, make-ci-green, taskflow, terminal-bench/feal-differential-cryptanalysis, terminal-bench/llm-inference-batching-scheduler, terminal-bench/schemelike-metacircular-eval, webcore. The result seal and bundled evidence still verify; recorded task digests remain the run-time fingerprints.

Harness↕	Solve rate · Wilson 95%↓ 050100%	Solved↕	Median wall↕	Fresh tokens/solve↕	Uncached input/solve↕	Output/solve↕	Cache-read/solve↕	Telemetry source / basis	Telemetry coverage
claude	81.1% 65.8%–90.5%	30/37	47.6s	78,508	33,949	44,558	1,918,016	proxyproxy_measured	proxy 37/37 · native 36/37
pi	75.7% 59.9%–86.6%	28/37	24.5s	62,946	33,190	29,756	1,601,774	proxyproxy_measured	proxy 37/37 · native 34/37
opencode	70.3% 54.2%–82.5%	26/37	40.3s	68,354	37,065	31,288	1,529,383	proxyproxy_measured	proxy 37/37 · native 36/37
grokbuild	67.6% 51.5%–80.4%	25/37	27.7s	120,124	74,115	46,010	1,579,745	proxyproxy_measured	proxy 37/37 · native 37/37
codex	64.9% 48.8%–78.2%	24/37	27.2s	86,414	36,972	49,442	1,119,568	proxyproxy_measured	proxy 37/37 · native 32/37

results.jsonlrelease page

OpenBench: grok-4.5 across 4 harnesses

kind releasedate 2026-07-20denominators matched (task, trial)matched rows 156common task/trials 39results SHA 3905ac339eeatask-set SHA 1c4805e7fcd0

grok-4.5caveats disclosed

5 caveat(s) from the release page

Archived first-party matrix published without rerunning models. The source contains 180 rows across 4 harnesses, 15 tasks, and 3 trials.
The bundled results SHA is sealed. Current-tree task checks report expected post-run definition drift; recorded task digests are retained as run-time fingerprints.
The unified board ranks only task/trial cells shared by every harness. The historical release page retains its older per-arm countable view, so its denominators differ.
No counting-proxy telemetry was retained for this run. Complete native split telemetry is available only where the board reports 100% native coverage.
Archived task definitions differ from or are unavailable in the current checkout for add-feature, make-ci-green, taskflow, terminal-bench/feal-differential-cryptanalysis, terminal-bench/llm-inference-batching-scheduler, terminal-bench/schemelike-metacircular-eval, webcore. The result seal and bundled evidence still verify; recorded task digests remain the run-time fingerprints.

Harness↕	Solve rate · Wilson 95%↓ 050100%	Solved↕	Median wall↕	Fresh tokens/solve↕	Uncached input/solve↕	Output/solve↕	Cache-read/solve↕	Cache-write/solve↕	Telemetry source / basis	Telemetry coverage
cursor	89.7% 76.4%–95.9%	35/39	25.9s	36,707	28,600	8,106	247,504	0	nativeharness_reported	proxy 0/39 · native 39/39
grokbuild	87.2% 73.3%–94.4%	34/39	42.3s	58,603	49,845	8,758	212,932	—	nativeunknown	proxy 0/39 · native 39/39
pi	79.5% 64.5%–89.2%	31/39	25.3s	—	—	—	—	—	unavailable	proxy 0/39 · native 38/39
opencode	76.9% 61.7%–87.4%	30/39	21.2s	55,008	45,494	9,514	250,735	0	nativevendor_split	proxy 0/39 · native 39/39

results.jsonlrelease page

OpenBench showcase: BYO aider vs pi vs opencode (deepseek-v4-flash)

kind communitydate 2026-07-20denominators matched (task, trial)matched rows 24common task/trials 8results SHA 677642cc0649task-set SHA 8b757f8abf38

deepseek-v4-flashcaveats disclosed

4 caveat(s) from the release page

Harness mode is not apples-to-apples: aider ran as a one-shot --message invocation, while pi and opencode ran full agentic loops. Token comparison reflects that difference in harness mode, not a pure like-for-like agent loop contest.
One aider cell excluded: 1 of 9 aider cells was infrastructure-classified and is excluded from solve-rate denominators (aider reported as 8/8). pi and opencode are 9/9.
Uniform total-token basis: totals include uncached input, output, and cache reads from split fields; the vendor aggregate is never used. Values are 6,192 vs 48,262 vs 111,380 tokens/solve (aider vs pi vs opencode).
Verifiable bundle: this release includes results.jsonl and provenance.json. Re-check digests with obench verify docs/releases/2026-07-20-aider-showcase.

Harness↕	Solve rate · Wilson 95%↓ 050100%	Solved↕	Median wall↕	Fresh tokens/solve↕	Uncached input/solve↕	Output/solve↕	Cache-read/solve↕	Telemetry source / basis	Telemetry coverage
aider	100.0% 67.6%–100.0%	8/8	39.0s	5,856	2,308	3,548	336	proxyproxy_measured	proxy 8/8 · native 0/8
opencode	100.0% 67.6%–100.0%	8/8	31.7s	13,669	10,328	3,342	91,600	proxyproxy_measured	proxy 8/8 · native 8/8
pi	100.0% 67.6%–100.0%	8/8	26.4s	7,742	4,928	2,814	35,456	proxyproxy_measured	proxy 8/8 · native 8/8

results.jsonlrelease page

OpenBench: GLM 5.2 quarantine-safe archive

kind releasedate 2026-07-09denominators matched (task, trial)matched rows 40common task/trials 8results SHA 5434f3f8aa66task-set SHA 46075fa27b81

glm-5.2caveats disclosed

5 caveat(s) from the release page

Quarantine-safe derived bundle: all 15 terminal-bench/cancel-async-tasks rows were excluded because its load-sensitive checker is binding-quarantined. No model was rerun.
The bundled results SHA is sealed. Current-tree task checks report expected post-run definition drift; recorded task digests are retained as run-time fingerprints.
The retained archive lacks image digests, version-source labels, and an explicit per-row timeout field; contemporaneous version-stamp drift is documented in the bundle README.
The unified board ranks only task/trial cells shared by every harness. The release page also shows the per-arm countable view, so its denominators differ.
Archived task definitions differ from or are unavailable in the current checkout for terminal-bench/feal-differential-cryptanalysis, terminal-bench/llm-inference-batching-scheduler, terminal-bench/schemelike-metacircular-eval. The result seal and bundled evidence still verify; recorded task digests remain the run-time fingerprints.

Harness↕	Solve rate · Wilson 95%↓ 050100%	Solved↕	Median wall↕	Fresh tokens/solve↕	Uncached input/solve↕	Output/solve↕	Cache-read/solve↕	Cache-write/solve↕	Telemetry source / basis	Telemetry coverage
pi	100.0% 67.6%–100.0%	8/8	693.5s	—	—	—	—	—	unavailable	proxy 0/8 · native 6/8
opencode	87.5% 52.9%–97.8%	7/8	661.2s	94,197	54,762	39,435	973,166	0	nativevendor_split	proxy 0/8 · native 8/8
claude	62.5% 30.6%–86.3%	5/8	569.0s	—	—	—	—	—	unavailable	proxy 0/8 · native 4/8
grokbuild	62.5% 30.6%–86.3%	5/8	267.6s	—	—	—	—	—	unavailable	proxy 0/8 · native 0/8
codex	50.0% 21.5%–78.5%	4/8	801.1s	—	—	—	—	—	unavailable	proxy 0/8 · native 4/8

results.jsonlrelease page

Not ranked (2)

No result-sealed results.jsonl.

2026-07-02-m3
no results.jsonl (HTML-only release page)
2026-07-20-kimi-k3
no results.jsonl (HTML-only release page)

AI gateway benchmarks

Compares request latency, throughput, and reliability across AI gateways under separately scheduled cold and warm conditions. Select a model below; each bundle keeps its own request counts and matched-block denominators.

4 published bundles4 benchmarked modelsmodel-specific denominators belowupdated 2026-08-20

Gateway Bench measures request-level transport and serving telemetry. Cold and warm denominators are separate and are never merged.

GPT-5.6 Sol Gateway Bench, 100 requests per route

date 2026-08-20model match rolling_aliascold blocks 50/50warm blocks 50/50requests 700verified commit 73c569ebf06aexperiment 960f93e76135

Evidence depth: 50 cold + 50 warm matched blocks per route.

Run note: Five routes, each with 50 cold and 50 warm measured requests. Warm measurements include a separate primer. All routes use GPT-5.6 Sol with medium reasoning; temperature and top_p are omitted so provider defaults apply. Gateway routes lock to OpenAI with fallbacks and response caching disabled. Cloudflare returned HTTP 402 for 25 of 100 measured requests; availability is included in its composite score.

Supplemental run note: Ramp Router adds 50 cold and 50 warm measured requests collected on 2026-08-20. Its direct-relative deltas use the fresh direct OpenAI control from the same supplemental run, not the 2026-08-18 control. Ramp returns the served GPT-5.6 Sol model ID but not an upstream-provider field; OpenBench derives OpenAI ownership from the admitted model identity. Supplemental evidence (2026-08-20): results.jsonl; verified commit b7606d1f450f; experiment 86d9f9d74624

Gateway leaderboard

OpenBench Composite compares TTFT and output throughput with the matched direct-provider arm; the previous score used fixed absolute ceilings. Direct performance equals 75 before request success is applied; higher is better, cost is excluded, and Direct OpenAI is an unranked reference.

OpenRouter

100% request success

77.1composite

Vercel

100% request success

75.1composite

Concentrate

100% request success

72.1composite

Cloudflare

75% request success

56.3composite

Ramp Router

100% request success

51.6composite

Cold requests

Complete blocks: 50/50. TTFT begins when the measured request is sent; connection setup is reported separately.

Route↕	TTFT p50 / p95↕	Stream total p50 / p95↕	Response headers p50 / p95↕	First body byte p50 / p95↕	Throughput tok/s p50 / p95↕	Total / cached / cache-write tokens p50 / p95
Direct OpenAI	1.146s / 2.406s	1.920s / 3.722s	0.361s / 0.785s	0.361s / 0.833s	51.2 / 69.8	total 84.5 / 95.6 cached 0.0 / 0.0 cache write 0.0 / 0.0
Cloudflare	1.394s / 2.339s	2.289s / 3.518s	0.708s / 1.197s	0.708s / 1.198s	51.5 / 63.7	total 85.0 / 96.9 cached 0.0 / 0.0 cache write 0.0 / 0.0
Concentrate	1.177s / 2.598s	2.025s / 3.674s	0.626s / 0.999s	0.626s / 0.999s	51.0 / 65.1	total 84.5 / 96.5 cached 0.0 / 0.0 cache write 0.0 / 0.0
OpenRouter	1.040s / 1.931s	1.887s / 2.894s	0.375s / 0.774s	0.503s / 0.858s	51.4 / 60.4	total 85.5 / 97.0 cached 0.0 / 0.0 cache write — / — (0/50)
Ramp Router	1.269s / 6.410s	2.755s / 7.027s	1.268s / 6.408s	1.268s / 6.409s	37.5 / 67.1	total 83.5 / 94.1 cached 0.0 / 0.0 cache write 0.0 / 0.0
Vercel	1.044s / 2.490s	1.970s / 3.295s	1.019s / 2.480s	1.043s / 2.488s	50.9 / 66.5	total 85.0 / 95.1 cached 0.0 / 0.0 cache write 0.0 / 0.0

Warm requests

Complete blocks: 50/50. TTFT begins when the measured request is sent; connection setup is reported separately.

Route↕	TTFT p50 / p95↕	Stream total p50 / p95↕	Response headers p50 / p95↕	First body byte p50 / p95↕	Throughput tok/s p50 / p95↕	Total / cached / cache-write tokens p50 / p95
Direct OpenAI	1.022s / 2.321s	1.909s / 3.979s	0.351s / 0.683s	0.351s / 0.683s	47.9 / 70.5	total 85.0 / 93.5 cached 0.0 / 0.0 cache write 0.0 / 0.0
Cloudflare	1.044s / 1.817s	1.966s / 2.995s	0.505s / 0.743s	0.505s / 0.744s	49.6 / 69.0	total 86.0 / 96.0 cached 0.0 / 0.0 cache write 0.0 / 0.0
Concentrate	1.159s / 2.713s	2.069s / 3.607s	0.557s / 0.932s	0.557s / 0.933s	49.7 / 64.2	total 87.5 / 93.5 cached 0.0 / 0.0 cache write 0.0 / 0.0
OpenRouter	1.068s / 2.436s	1.884s / 3.558s	0.410s / 0.799s	0.514s / 0.799s	54.0 / 73.7	total 85.0 / 95.5 cached 0.0 / 0.0 cache write — / — (0/50)
Ramp Router	1.693s / 6.233s	2.947s / 6.790s	1.692s / 6.233s	1.693s / 6.233s	40.1 / 85.6	total 83.0 / 94.0 cached 0.0 / 0.0 cache write 0.0 / 0.0
Vercel	1.087s / 2.386s	1.912s / 3.782s	1.074s / 2.289s	1.087s / 2.295s	51.3 / 69.0	total 84.0 / 93.0 cached 0.0 / 0.0 cache write 0.0 / 0.0

Cold setup

Connection setup phases for cold requests only.

Route↕	DNS p50 / p95↕	TCP p50 / p95↕	TLS p50 / p95↕
Direct OpenAI	0.013s / 0.173s	0.070s / 0.218s	0.078s / 0.216s
Cloudflare	0.002s / 0.060s	0.072s / 0.219s	0.082s / 0.228s
Concentrate	0.002s / 0.004s	0.068s / 0.202s	0.082s / 0.211s
OpenRouter	0.002s / 0.078s	0.070s / 0.193s	0.080s / 0.199s
Ramp Router	0.002s / 0.003s	0.020s / 0.037s	0.028s / 0.043s
Vercel	0.003s / 0.157s	0.071s / 0.201s	0.089s / 0.208s

Paired request deltas

Every delta is gateway minus Direct OpenAI. These are latency metrics, so positive means slower/worse and negative means faster/better. Medians use complete paired blocks with bootstrap 95% intervals.

Gateway route↕	Condition↕	Δ response headers↕	Δ TTFT↕
Cloudflare	cold	+0.307s 95% CI +0.241s to +0.393s · paired 42/50	+0.179s 95% CI -0.033s to +0.494s · paired 42/50
Cloudflare	warm	+0.159s 95% CI +0.140s to +0.221s · paired 33/50	+0.116s 95% CI -0.017s to +0.198s · paired 33/50
Concentrate	cold	+0.234s 95% CI +0.189s to +0.291s · paired 50/50	+0.072s 95% CI -0.149s to +0.256s · paired 50/50
Concentrate	warm	+0.205s 95% CI +0.157s to +0.224s · paired 50/50	+0.162s 95% CI -0.029s to +0.433s · paired 50/50
OpenRouter	cold	+0.019s 95% CI -0.004s to +0.039s · paired 50/50	-0.139s 95% CI -0.252s to +0.061s · paired 50/50
OpenRouter	warm	+0.061s 95% CI +0.015s to +0.087s · paired 50/50	+0.038s 95% CI -0.128s to +0.165s · paired 50/50
Vercel	cold	+0.613s 95% CI +0.573s to +0.745s · paired 50/50	+0.008s 95% CI -0.219s to +0.089s · paired 50/50
Vercel	warm	+0.720s 95% CI +0.641s to +0.855s · paired 50/50	+0.088s 95% CI -0.080s to +0.308s · paired 50/50
Ramp Router	cold	+0.812s 95% CI +0.584s to +1.336s · paired 50/50	+0.254s 95% CI +0.036s to +0.542s · paired 50/50
Ramp Router	warm	+1.295s 95% CI +0.886s to +1.541s · paired 50/50	+0.546s 95% CI +0.279s to +0.877s · paired 50/50

Completion integrity

Provider-reported completion reasons. Natural stop means the response ended normally; length means the provider reported a length-based termination, commonly an output or context limit; missing means no finish reason was reported; other covers any remaining explicit reason. Warm-primer natural stop shows whether the separate connection-warming request ended normally.

Route↕	Condition	Measured missing	Warm-primer natural-stop
Direct OpenAI	Cold	50	—
Direct OpenAI	Warm	50	0/50
Cloudflare	Cold	50	—
Cloudflare	Warm	50	0/50
Concentrate	Cold	50	—
Concentrate	Warm	50	0/50
OpenRouter	Cold	50	—
OpenRouter	Warm	50	0/50
Ramp Router	Cold	50	—
Ramp Router	Warm	50	0/50
Vercel	Cold	50	—
Vercel	Warm	50	0/50

results.jsonl

DeepSeek V4 Flash Gateway Bench, 100 requests per route

date 2026-08-03model match rolling_aliascold blocks 50/50warm blocks 50/50requests 500verified commit e0e5dd2978c7experiment 34873bc948a0

Evidence depth: 50 cold + 50 warm matched blocks per route.

Run note: Five routes, each with 50 cold and 50 warm measured requests. Warm measurements include a separate primer. Gateway routes lock to DeepSeek with fallbacks and response caching disabled. Token counts are route-reported; Cloudflare reported a different input-token denominator from the other routes.

Gateway leaderboard

OpenRouter

100% request success

72.5composite

Concentrate

100% request success

68.2composite

Cloudflare

100% request success

56.4composite

Vercel

100% request success

38.2composite

Cold requests

Complete blocks: 50/50. TTFT begins when the measured request is sent; connection setup is reported separately.

Route↕	TTFT p50 / p95↕	Stream total p50 / p95↕	Response headers p50 / p95↕	First body byte p50 / p95↕	Throughput tok/s p50 / p95↕	Total / cached / cache-write tokens p50 / p95
Direct DeepSeek	0.853s / 1.144s	1.272s / 1.601s	0.345s / 0.424s	0.356s / 0.424s	79.6 / 97.3	total 149.5 / 163.1 cached 0.0 / 0.0 cache write — / — (0/50)
Cloudflare	1.022s / 4.238s	1.619s / 5.156s	1.022s / 4.237s	1.022s / 4.238s	100.8 / 352.3	total 83.5 / 96.5 cached 0.0 / 0.0 cache write — / — (0/50)
Concentrate	1.105s / 1.539s	1.491s / 2.014s	0.558s / 0.735s	0.558s / 0.735s	80.8 / 103.1	total 148.0 / 162.6 cached 0.0 / 0.0 cache write 0.0 / 0.0
OpenRouter	0.978s / 1.212s	1.418s / 1.760s	0.446s / 0.556s	0.522s / 0.579s	76.9 / 103.1	total 147.5 / 171.6 cached 0.0 / 0.0 cache write 0.0 / 0.0
Vercel	1.563s / 6.838s	1.960s / 7.135s	0.502s / 0.626s	0.513s / 0.631s	81.9 / 106.9	total 149.0 / 167.6 cached 0.0 / 0.0 cache write — / — (0/50)

Warm requests

Complete blocks: 50/50. TTFT begins when the measured request is sent; connection setup is reported separately.

Route↕	TTFT p50 / p95↕	Stream total p50 / p95↕	Response headers p50 / p95↕	First body byte p50 / p95↕	Throughput tok/s p50 / p95↕	Total / cached / cache-write tokens p50 / p95
Direct DeepSeek	0.864s / 1.153s	1.274s / 1.630s	0.338s / 0.417s	0.346s / 0.424s	81.8 / 102.5	total 147.5 / 171.6 cached 0.0 / 0.0 cache write — / — (0/50)
Cloudflare	1.002s / 2.793s	1.665s / 5.138s	1.002s / 2.792s	1.002s / 2.792s	85.2 / 306.6	total 82.0 / 119.2 cached 0.0 / 0.0 cache write — / — (0/50)
Concentrate	1.036s / 1.262s	1.426s / 1.680s	0.478s / 0.603s	0.478s / 0.603s	80.6 / 105.2	total 148.0 / 158.0 cached 0.0 / 0.0 cache write 0.0 / 0.0
OpenRouter	0.917s / 1.172s	1.390s / 1.653s	0.379s / 0.477s	0.462s / 0.480s	78.1 / 105.8	total 149.0 / 176.6 cached 0.0 / 0.0 cache write 0.0 / 0.0
Vercel	1.328s / 8.706s	1.786s / 9.123s	0.428s / 0.527s	0.437s / 0.531s	81.6 / 187.6	total 147.5 / 164.6 cached 0.0 / 0.0 cache write — / — (0/50)

Cold setup

Connection setup phases for cold requests only.

Route↕	DNS p50 / p95↕	TCP p50 / p95↕	TLS p50 / p95↕
Direct DeepSeek	0.002s / 0.031s	0.006s / 0.007s	0.009s / 0.013s
Cloudflare	0.001s / 0.021s	0.012s / 0.013s	0.016s / 0.019s
Concentrate	0.001s / 0.002s	0.012s / 0.014s	0.017s / 0.022s
OpenRouter	0.001s / 0.021s	0.011s / 0.014s	0.015s / 0.019s
Vercel	0.002s / 0.043s	0.006s / 0.008s	0.023s / 0.027s

Paired request deltas

Every delta is gateway minus Direct DeepSeek. These are latency metrics, so positive means slower/worse and negative means faster/better. Medians use complete paired blocks with bootstrap 95% intervals.

Gateway route↕	Condition↕	Δ response headers↕	Δ TTFT↕
Cloudflare	cold	+0.676s 95% CI +0.582s to +0.811s · paired 50/50	+0.145s 95% CI +0.053s to +0.357s · paired 50/50
Cloudflare	warm	+0.668s 95% CI +0.459s to +0.865s · paired 50/50	+0.168s 95% CI -0.047s to +0.301s · paired 50/50
Concentrate	cold	+0.203s 95% CI +0.185s to +0.224s · paired 50/50	+0.248s 95% CI +0.159s to +0.350s · paired 50/50
Concentrate	warm	+0.143s 95% CI +0.123s to +0.156s · paired 50/50	+0.172s 95% CI +0.076s to +0.218s · paired 50/50
OpenRouter	cold	+0.098s 95% CI +0.082s to +0.112s · paired 50/50	+0.129s 95% CI +0.073s to +0.216s · paired 50/50
OpenRouter	warm	+0.038s 95% CI +0.032s to +0.050s · paired 50/50	+0.058s 95% CI -0.038s to +0.113s · paired 50/50
Vercel	cold	+0.132s 95% CI +0.121s to +0.160s · paired 50/50	+0.712s 95% CI +0.286s to +1.318s · paired 50/50
Vercel	warm	+0.088s 95% CI +0.069s to +0.109s · paired 50/50	+0.445s 95% CI +0.226s to +1.229s · paired 50/50

Completion integrity

Route↕	Condition	Measured natural-stop	Warm-primer natural-stop
Direct DeepSeek	Cold	50	—
Direct DeepSeek	Warm	50	50/50
Cloudflare	Cold	50	—
Cloudflare	Warm	50	50/50
Concentrate	Cold	50	—
Concentrate	Warm	50	50/50
OpenRouter	Cold	50	—
OpenRouter	Warm	50	50/50
Vercel	Cold	50	—
Vercel	Warm	50	50/50

results.jsonl

Kimi K3 Gateway Bench, 100 requests per route

date 2026-07-28model match rolling_aliascold blocks 50/50warm blocks 50/50requests 500verified commit c45441997c81experiment 9748aef6256d

Evidence depth: 50 cold + 50 warm matched blocks per route.

Run note: Five routes, each with 50 cold and 50 warm measured requests. Warm measurements include a separate primer. Routes use rolling Kimi K3 aliases.

Gateway leaderboard

OpenRouter

100% request success

75.3composite

Vercel

100% request success

72.3composite

Concentrate

100% request success

72.2composite

Cloudflare

100% request success

69.5composite

Cold requests

Complete blocks: 50/50. TTFT begins when the measured request is sent; connection setup is reported separately.

Route↕	TTFT p50 / p95↕	Stream total p50 / p95↕	Response headers p50 / p95↕	First body byte p50 / p95↕	Throughput tok/s p50 / p95↕	Total / cached / cache-write tokens p50 / p95
Direct Moonshot	2.257s / 3.493s	6.932s / 12.391s	2.256s / 3.493s	2.256s / 3.493s	34.6 / 40.9	total 263.0 / 370.1 cached 115.0 / 117.5 cache write — / — (0/50)
Cloudflare	2.831s / 5.256s	6.788s / 13.251s	2.831s / 5.256s	2.831s / 5.256s	34.8 / 48.0	total 227.5 / 379.6 cached 116.0 / 117.5 (30/50) cache write — / — (0/50)
Concentrate	2.699s / 3.839s	5.707s / 10.723s	2.699s / 3.839s	2.699s / 3.839s	36.2 / 45.3	total 224.0 / 351.0 cached 114.0 / 117.0 cache write 0.0 / 0.0
OpenRouter	2.568s / 3.392s	6.011s / 10.152s	2.537s / 3.377s	2.538s / 3.377s	35.1 / 48.0	total 237.0 / 353.3 cached 0.0 / 117.0 cache write 0.0 / 0.0
Vercel	2.734s / 4.504s	6.541s / 11.249s	2.726s / 4.494s	2.734s / 4.504s	34.7 / 48.1	total 244.5 / 351.6 cached 113.0 / 117.5 cache write — / — (0/50)

Warm requests

Complete blocks: 50/50. TTFT begins when the measured request is sent; connection setup is reported separately.

Route↕	TTFT p50 / p95↕	Stream total p50 / p95↕	Response headers p50 / p95↕	First body byte p50 / p95↕	Throughput tok/s p50 / p95↕	Total / cached / cache-write tokens p50 / p95
Direct Moonshot	2.351s / 3.710s	6.798s / 11.104s	2.350s / 3.710s	2.351s / 3.710s	34.2 / 56.6	total 256.0 / 372.1 cached 115.0 / 119.0 (30/50) cache write — / — (0/50)
Cloudflare	2.344s / 4.020s	5.966s / 9.987s	2.344s / 4.020s	2.344s / 4.020s	34.9 / 44.0	total 237.0 / 342.0 cached 116.0 / 119.0 (42/50) cache write — / — (0/50)
Concentrate	2.514s / 3.691s	6.677s / 10.043s	2.513s / 3.691s	2.514s / 3.691s	34.9 / 44.4	total 251.5 / 357.9 cached 115.0 / 118.0 cache write 0.0 / 0.0
OpenRouter	2.436s / 2.988s	6.580s / 9.581s	2.423s / 2.982s	2.423s / 2.982s	34.1 / 41.1	total 260.5 / 347.1 cached 0.0 / 118.0 cache write 0.0 / 0.0
Vercel	2.459s / 3.138s	6.320s / 10.413s	2.459s / 3.132s	2.459s / 3.138s	35.1 / 46.0	total 252.0 / 361.2 cached 115.0 / 119.0 cache write — / — (0/50)

Cold setup

Connection setup phases for cold requests only.

Route↕	DNS p50 / p95↕	TCP p50 / p95↕	TLS p50 / p95↕
Direct Moonshot	0.002s / 0.037s	0.012s / 0.013s	0.018s / 0.022s
Cloudflare	0.002s / 0.021s	0.011s / 0.013s	0.016s / 0.020s
Concentrate	0.001s / 0.056s	0.011s / 0.012s	0.018s / 0.021s
OpenRouter	0.001s / 0.022s	0.011s / 0.012s	0.016s / 0.019s
Vercel	0.002s / 0.135s	0.005s / 0.007s	0.024s / 0.053s

Paired request deltas

Every delta is gateway minus Direct Moonshot. These are latency metrics, so positive means slower/worse and negative means faster/better. Medians use complete paired blocks with bootstrap 95% intervals.

Gateway route↕	Condition↕	Δ response headers↕	Δ TTFT↕
Cloudflare	cold	+0.413s 95% CI +0.222s to +0.674s · paired 50/50	+0.412s 95% CI +0.222s to +0.674s · paired 50/50
Cloudflare	warm	+0.067s 95% CI -0.068s to +0.159s · paired 50/50	+0.067s 95% CI -0.068s to +0.159s · paired 50/50
Concentrate	cold	+0.249s 95% CI +0.086s to +0.393s · paired 50/50	+0.251s 95% CI +0.077s to +0.393s · paired 50/50
Concentrate	warm	+0.236s 95% CI +0.110s to +0.281s · paired 50/50	+0.236s 95% CI +0.110s to +0.303s · paired 50/50
OpenRouter	cold	+0.259s 95% CI +0.071s to +0.409s · paired 50/50	+0.271s 95% CI +0.090s to +0.420s · paired 50/50
OpenRouter	warm	+0.187s 95% CI -0.065s to +0.347s · paired 50/50	+0.193s 95% CI -0.077s to +0.378s · paired 50/50
Vercel	cold	+0.297s 95% CI +0.118s to +0.492s · paired 50/50	+0.306s 95% CI +0.127s to +0.498s · paired 50/50
Vercel	warm	+0.117s 95% CI -0.056s to +0.271s · paired 50/50	+0.122s 95% CI -0.060s to +0.282s · paired 50/50

Completion integrity

Route↕	Condition	Measured natural-stop	Warm-primer natural-stop
Direct Moonshot	Cold	50	—
Direct Moonshot	Warm	50	50/50
Cloudflare	Cold	50	—
Cloudflare	Warm	50	50/50
Concentrate	Cold	50	—
Concentrate	Warm	50	50/50
OpenRouter	Cold	50	—
OpenRouter	Warm	50	50/50
Vercel	Cold	50	—
Vercel	Warm	50	50/50

results.jsonl

GPT-4o mini Gateway Bench

date 2026-07-27model match rolling_aliascold blocks 30/30warm blocks 30/30requests 300verified commit 6d1de84d6c96experiment e3649dbc6dc4

Evidence depth: 30 cold + 30 warm matched blocks per route.

Gateway leaderboard

OpenRouter

100% request success

85.7composite

Cloudflare

100% request success

75.9composite

Concentrate

100% request success

71.2composite

Vercel

100% request success

62.2composite

Cold requests

Complete blocks: 30/30. TTFT begins when the measured request is sent; connection setup is reported separately.

Route↕	TTFT p50 / p95↕	Stream total p50 / p95↕	Response headers p50 / p95↕	First body byte p50 / p95↕	Throughput tok/s p50 / p95↕	Total / cached / cache-write tokens p50 / p95
Direct OpenAI	0.527s / 1.356s	0.735s / 1.763s	0.232s / 0.887s	0.232s / 0.888s	92.3 / 123.2	total 51.0 / 54.0 cached 0.0 / 0.0 cache write 0.0 / 0.0
Cloudflare	0.656s / 1.055s	0.812s / 1.215s	0.341s / 0.767s	0.341s / 0.767s	84.6 / 113.7	total 51.0 / 53.5 cached 0.0 / 0.0 cache write 0.0 / 0.0
Concentrate	0.763s / 0.946s	0.930s / 1.803s	0.404s / 0.538s	0.404s / 0.538s	84.9 / 123.0	total 51.0 / 54.0 cached 0.0 / 0.0 cache write 0.0 / 0.0
OpenRouter	0.471s / 0.675s	0.662s / 0.845s	0.459s / 0.668s	0.470s / 0.668s	100.6 / 156.9	total 54.0 / 56.5 cached 0.0 / 0.0 cache write — / — (0/30)
Vercel	0.789s / 2.494s	0.951s / 2.719s	0.773s / 2.485s	0.783s / 2.490s	87.9 / 345.3	total 51.0 / 54.0 cached 0.0 / 0.0 cache write 0.0 / 0.0

Warm requests

Complete blocks: 30/30. TTFT begins when the measured request is sent; connection setup is reported separately.

Route↕	TTFT p50 / p95↕	Stream total p50 / p95↕	Response headers p50 / p95↕	First body byte p50 / p95↕	Throughput tok/s p50 / p95↕	Total / cached / cache-write tokens p50 / p95
Direct OpenAI	0.504s / 0.977s	0.656s / 1.221s	0.184s / 0.436s	0.184s / 0.437s	95.1 / 127.2	total 52.0 / 54.5 cached 0.0 / 0.0 cache write 0.0 / 0.0
Cloudflare	0.561s / 0.739s	0.754s / 0.893s	0.260s / 0.410s	0.260s / 0.410s	96.6 / 126.4	total 52.0 / 54.5 cached 0.0 / 0.0 cache write 0.0 / 0.0
Concentrate	0.704s / 0.933s	0.896s / 1.096s	0.361s / 0.436s	0.361s / 0.436s	85.3 / 131.6	total 52.0 / 54.5 cached 0.0 / 0.0 cache write 0.0 / 0.0
OpenRouter	0.433s / 0.655s	0.587s / 0.764s	0.423s / 0.485s	0.432s / 0.632s	105.9 / 415.5	total 54.5 / 57.0 cached 0.0 / 0.0 cache write — / — (0/30)
Vercel	0.572s / 1.605s	0.726s / 2.458s	0.559s / 1.571s	0.572s / 1.601s	94.4 / 449.2	total 52.5 / 55.0 cached 0.0 / 0.0 cache write 0.0 / 0.0

Cold setup

Connection setup phases for cold requests only.

Route↕	DNS p50 / p95↕	TCP p50 / p95↕	TLS p50 / p95↕
Direct OpenAI	0.002s / 0.003s	0.011s / 0.012s	0.016s / 0.023s
Cloudflare	0.001s / 0.002s	0.011s / 0.012s	0.017s / 0.023s
Concentrate	0.001s / 0.002s	0.011s / 0.014s	0.018s / 0.023s
OpenRouter	0.001s / 0.002s	0.011s / 0.012s	0.016s / 0.020s
Vercel	0.002s / 0.148s	0.006s / 0.007s	0.024s / 0.028s

Paired request deltas

Gateway route↕	Condition↕	Δ response headers↕	Δ TTFT↕
Cloudflare	cold	+0.090s 95% CI +0.072s to +0.120s · paired 30/30	+0.095s 95% CI +0.011s to +0.163s · paired 30/30
Cloudflare	warm	+0.081s 95% CI +0.050s to +0.112s · paired 30/30	+0.037s 95% CI +0.003s to +0.116s · paired 30/30
Concentrate	cold	+0.169s 95% CI +0.143s to +0.190s · paired 30/30	+0.193s 95% CI +0.097s to +0.235s · paired 30/30
Concentrate	warm	+0.168s 95% CI +0.143s to +0.204s · paired 30/30	+0.202s 95% CI +0.142s to +0.230s · paired 30/30
OpenRouter	cold	+0.233s 95% CI +0.212s to +0.244s · paired 30/30	-0.088s 95% CI -0.176s to -0.009s · paired 30/30
OpenRouter	warm	+0.230s 95% CI +0.202s to +0.252s · paired 30/30	-0.054s 95% CI -0.109s to -0.018s · paired 30/30
Vercel	cold	+0.508s 95% CI +0.440s to +0.596s · paired 30/30	+0.227s 95% CI +0.171s to +0.297s · paired 30/30
Vercel	warm	+0.348s 95% CI +0.328s to +0.465s · paired 30/30	+0.058s 95% CI +0.014s to +0.214s · paired 30/30

results.jsonl

Not published (1)

Did not pass public Gateway Bench verification.

2026-07-28-kimi-k3-managed-30
bundle verification failed: public manifest does not match schema

OpenBench benchmark results

Digest-verified benchmark releases, methodology, and project information.

coding-agent harnessesAI gateway routes

Releases

First-party bundles.

OpenBench M3 — Harness Comparison
2026-07-02 · gpt-5.5-medium
OpenBench: GLM 5.2 quarantine-safe archive
2026-07-09 · glm-5.2
OpenBench: grok-4.5 across 4 harnesses
2026-07-20 · grok-4.5
OpenBench: kimi-k3 across 3 harnesses
2026-07-20 · kimi-k3
OpenBench: deepseek across 5 harnesses
2026-07-20 · deepseek-v4-flash
OpenBench: gpt-5.6 across 7 harnesses
2026-07-21 · gpt-5.6-sol

Packs

Versioned task and harness packs.

openbench/core-smoke@1.0.0 tasks
Apache-2.0 · data/packs/openbench-core-smoke · 3b1e576398a8
Tiny polarity-checked smoke tasks (make-it-run, fix-failing-test)

OpenBench benchmark results

Digest-verified benchmark releases, methodology, and project information.

coding-agent harnessesAI gateway routes

What is being measured

OpenBench runs two benchmark families. They share a task contract and a checker, and no denominators.

Harness Bench

Varies the coding-agent harness — the CLI that wraps a model in a run loop, tool set, and permission policy — while holding the model and task fixed. An arm is (harness, model). A task is solved when its checker.sh exits 0; the harness's own claim of success is never trusted.

Gateway Bench

Measures one model request at a time under separately scheduled cold and warm transport conditions. It reports request success, route verification, transport and stream phase timing, throughput, usage, and per-request cost. It is not a coding-agent outcome benchmark. Gateway Bench requests and Harness Bench cells are never pooled or compared as one denominator.

The Gateway Bench leaderboard scores each gateway relative to the matched direct-provider arm for the same model; the previous score used fixed absolute latency and throughput ceilings. Direct performance anchors each metric at 75, with 25% cold TTFT median, 20% cold TTFT p95, 25% warm TTFT median, 20% warm TTFT p95, and 10% warm median output throughput; the weighted result is multiplied by request success. TTFT starts when the measured request is sent; cold DNS, TCP, and TLS setup are reported separately. Cost is excluded, the direct-provider arm is unranked, and the detailed measurements below the score remain the factual record.

Denominators and intervals

Harness Bench denominators are countable cells. Infrastructure and rate-limit failures are excluded; other failures, including timeouts, stay in the denominator.
Harness Bench uses Wilson 95% intervals over matched (task, trial) cells whenever a bundle has two or more arms.
Gateway Bench displays complete cold and warm block counts separately. Availability uses a Wilson 95% interval over every attempted measured request, so gateway errors such as HTTP 429 responses and timeouts remain in that denominator. Phase summaries use successful, route-verified requests and retain metric-specific coverage. Paired deltas use complete gateway/direct blocks and bootstrap 95% intervals.

Efficiency and cost

Median wall time is taken among solved cells only.
Each Harness Bench arm uses one complete split-token lane across all matched result rows, preferring proxy telemetry and otherwise using native telemetry. Fresh tokens are uncached input plus output; cache reads and cache writes remain separate. Incomplete lanes produce no token metrics and report their row coverage instead.
Each per-solve token figure sums traffic from every matched attempt, including failed attempts, then divides by the number of solved cells. It measures attempted traffic required per solve, not the average size of successful attempts alone.
Harness $/solve appears only for models with a configured price.
Gateway Bench response headers are the time until HTTP response headers. First body byte and semantic TTFT are reported separately; response headers are not labeled TTFB.
Gateway Bench measured cost is the frozen-list request estimate. Charged cost is separately reported billing evidence. Each retains its own request coverage, as do total, cached-input, and cache-write token readings.
Harness defaults are not clamped.

Comparability

Cells from different bundles are never blended. Each board is one bundle; cross-bundle ranking on different task sets is not supported.
Every ranked bundle ships results.jsonl plus a provenance digest and is re-verified before it appears here. Digests show tamper-evidence, not absence of cherry-picking.
Results cover only the included tasks, trials, model deployments, harness versions, and timeout caps.

Reproducing a board

Every board links its results.jsonl. Re-check a bundle with obench verify <bundle> (harness) or obench gateway probe verify <bundle> (gateway), and rebuild this page with obench site build.

OpenBench benchmark results

Digest-verified benchmark releases, methodology, and project information.

coding-agent harnessesAI gateway routes

Contact

Want to add a gateway or harness, submit results, report a problem, or share an idea? Reach Matthew through either channel.

Open a GitHub issue Message @mattlam_ on X