Nonobench Releases Open Benchmark of 49 LLMs on Nonogram Puzzles

What Happened

A [P] submission to r/MachineLearning announced Nonobench, an open-source benchmark evaluating 49 large language models on nonogram puzzles. The release includes public code and a results table, allowing external reproduction across model versions. Nonograms require solvers to satisfy row and column run-length constraints simultaneously, which forces iterative deduction rather than single-pass pattern matching.

Why It Matters

Most reasoning benchmarks either rely on private held-out sets or on static datasets that saturate within a few model generations, which makes longitudinal comparison expensive and noisy. Nonobench addresses both problems by publishing the harness alongside results, so any operator can re-run the same evaluation against a new checkpoint, fine-tune, or inference configuration. For teams deciding between model versions, quantization levels, or prompt strategies, this creates a low-cost discriminator that isolates constraint-satisfaction behavior from general language fluency. The strategic value is not the puzzle itself but the reproducibility contract: frozen task set, open scoring code, and a model roster broad enough to anchor comparisons. Builders running internal evals can now calibrate their own harnesses against a public reference instead of relying on vendor-reported numbers.

Technical Details

Nonograms (picross) encode a grid with per-row and per-column run-length clues; solving requires propagating partial constraints, backtracking on contradictions, and maintaining global consistency across intersecting lines. The benchmark covers 49 models spanning major open-weight and closed families, with outputs scored on grid-level correctness rather than token-level similarity. Because scoring is deterministic and the task generator is bundled, results are reproducible without API-side randomness, provided operators fix temperature and seeding. Primary limitations: puzzle-scale ceiling is bounded by context length and output token budget, and the current release does not separate reasoning failures from formatting failures unless the harness normalizes final grid extraction. Performance variance across the 49 models is the signal — absolute accuracy is secondary to the ordering and to failure-mode distribution.

Operational Impact

Reasoning evals previously required either bespoke task construction or paid access to private benchmark suites; Nonobench reduces the marginal cost of a new evaluation to a container run against a fixed dataset. This makes it practical to gate model upgrades on constraint-reasoning performance rather than on aggregate chat quality, which historically masks regressions in multi-step tasks. Teams can now A/B a candidate checkpoint against a production baseline on identical puzzles and get a directional read within hours, not weeks. Expect prompt-engineering and inference-config sweeps to shift toward structured-output enforcement, since grid extraction errors inflate apparent reasoning failures. For operators shipping agentic pipelines, Nonobench provides a cheap pre-deployment probe for whether a model can maintain state across dependent steps — a proxy for tool-calling and plan-execution reliability.