Five coding-agent harnesses on the same model
(gpt-5.5-medium), 3 tasks × 5 trials each, run locally on 2026-07-02.
A shakedown of the benchmark pipeline on a real 75-run spend — read it as “the harness works and here is what it found,” not a leaderboard.
The one axis on which the harnesses actually separate. Bars are sorted fastest→slowest; whiskers are the 95% confidence interval of the mean (n=15). Where whiskers overlap, the difference is not resolved at this sample size.
Every real harness solved every task on every trial. Identical intervals mean they are
statistically indistinguishable on success; only the null control
(which makes no edits) separates — confirming the checkers actually discriminate.
Same five harnesses, one panel per task, shared time axis. pi and
cursor stay fast everywhere; the slow trio (devin, codex,
opencode) reshuffles task to task — the visual signature of a cluster that
isn’t cleanly ordered.
All 15 raw timings per harness. pi and cursor are tight;
codex and opencode are wide (sd ≈ 23 s), which is why their
means blur together. The vertical tick marks each harness’s mean.
The matrix re-run (3 trials) with token capture on every adapter. Correctness is still a tie (all 9/9), so the story is cost per solved task. The token tax spans ~8× — wider than the wall-time spread — and the token order is not the time order.
Each harness placed by mean wall-time (x) and tokens per solve (y). If the two axes agreed, the points would fall on a line; they don’t.
opencode and codex sit at nearly the same wall-time yet
codex spends ~2× the tokens; devin is the slowest but not the
heaviest. pi alone is both fast and lean.
Three harder, partial-credit tasks (make-ci-green, add-feature, misleading-error), 3 trials. The correctness ceiling held: the four clean harnesses each scored 9/9, mean-score 1.0 — separation on correctness was not achieved at gpt-5.5-medium, and a pilot predicted it (every pilot cell hit 1.0, missing the 20–80% band high). So M4.5 is again an efficiency story — this time on genuinely multi-step tasks.
Four open models (first-party APIs) on the same three hard tasks GPT-5.5-medium saturated in M4.5, via
pi + opencode. The question: does a weaker model finally break the ceiling and show
partial scores? Answer: 3 of 4 reach parity — glm-5.2,
deepseek-v4-flash, and kimi-k2.7-code (via pi) all score a clean 1.00, matching the
frontier. Only glm-4.7-flash (free, small) drops below with genuine partial
credit.
The efficiency thesis in one number: all 72 real agent runs (4 open models × 2 harnesses × 3 hard tasks × 3 trials) cost ~$1.02 total — frontier-comparable open-model coding for about a dollar.
The dashboard below puts GPT-5.5-medium in the same panels as the reference. It is a
legitimate head-to-head: the frontier rows come from the M4.5 run on the same three tasks, through
the same pi harness, at the same 3×3 trial grid (n=9 per model). GPT-5.5
is drawn in neutral ink as the baseline; the open models carry their own hues.
opencode.py’s 900 s timeout handler
(concatenates bytes stdout with str). They’re 7× glm-4.7-flash:opencode
+ 1× kimi:opencode — opencode runs these models so slowly (glm-4.7-flash 789 s vs pi’s
102 s; kimi 371 vs 68) that they time out constantly and hit the crash. So the “glm-flash pi 0.47
vs opencode 0.22” gap and the kimi:opencode 0.67 are largely infra artifacts. kimi is fully capable
(kimi:pi = 1.00). The 8 cells are excluded from the capability scores below; the bug is filed.