Agentic Coding Benchmark: Kimi Code vs Codex vs Claude Code Results

What Happened

A community-contributed harness leaderboard was published comparing agentic coding tools — Kimi Code, Hermes, Pi, Claude Code, and Codex — on two standardized evaluation suites: Terminal Bench 2.1 and Deep SWE 1.1. The post aggregates results across harness configurations rather than single-model outputs, positioning the comparison around the full agent loop (scaffolding, tool use, and model) rather than raw model capability. No vendor-supplied figures are cited as primary; results are drawn from community harness runs.

Why It Matters

Harness-level benchmarks address a persistent evaluation gap: teams selecting coding agents have had to choose between vendor-published model cards and informal anecdotal reports, neither of which reflects the end-to-end system a developer actually deploys. Terminal Bench 2.1 and Deep SWE 1.1 measure task completion under realistic terminal and software-engineering conditions, which correlates more directly with production utility than aggregate code-generation scores. For operators running multi-agent pipelines, the harness — not the underlying model — is the unit of procurement, so differential harness performance on identical tasks is the decision-relevant variable. Community-maintained leaderboards also introduce adversarial pressure on vendor claims: results are reproducible and falsifiable, which shifts the burden of proof toward published methodology. Teams that adopt harness-relative selection criteria gain a defensible basis for tooling decisions that survives model churn.

Technical Details

Terminal Bench 2.1 evaluates agents on multi-step command-line tasks requiring file manipulation, process management, and error recovery within a sandboxed shell environment. Deep SWE 1.1 targets repository-scale software engineering tasks — issue resolution, test-driven fixes, and multi-file refactors — scored against held-out test suites. The leaderboard compares harness configurations, meaning the scaffolding (prompting strategy, tool-call parsing, context management, retry logic) is a scored variable alongside the model. This complicates direct model comparison: a strong model under a weak harness can underperform a weaker model under a tuned harness, and the leaderboard's rankings reflect that composite. Limitations include undisclosed task sampling, potential harness-tuning bias toward specific suites, and the absence of cost-per-task normalization, which matters operationally since agentic runs can vary by an order of magnitude in token spend.

Operational Impact

Selection workflows shift from model-card comparison to harness-level bake-offs against internal task distributions; teams can now calibrate a shortlist using public benchmark rankings before committing to private evaluation. The composite scoring means operators should expect to tune scaffolding — context window management, tool schemas, retry policies — as a first-class lever, potentially recovering performance without switching vendors. Cost-per-completed-task becomes the natural complement to benchmark rank, since a higher-ranked harness that burns 3x tokens may lose on unit economics. Procurement cycles shorten where public harness results align with internal workloads, and lengthen where they diverge, pushing evaluation effort toward the divergence cases. Vendors face pressure to publish harness configurations and reproduce community results, raising the documentation bar for agent tooling generally.