freeze-bench

Most coding benchmarks accidentally hand the model the answer key. It can run git log, grep the commit message, or read the tests added by the patch and repeat the answer. The run looks like reasoning, but it is retrieval.

An LLM judge makes the problem worse: a fluent explanation can outscore a correct patch. So I built freeze-bench around a reconstruction problem instead. The model gets the repository before the fix, with no history, no answer key, and no network. Grading runs the patch and checks the result.

Every task goes through the same five stages:

1. Freeze. Mine a real bug and record the base commit, fix commit, dates, and the issue text as it appeared before the fix.
2. Sanitize. Export the base commit to a detached worktree with no git history, no reference solution, no new tests, and no network.
3. Run. Use the same worktree and prompt for every model under comparison.
4. Grade. The repro must flip fail-to-pass, regressions must stay green, and every cheat check must hold. A run that reads git history or the rubric scores zero.
5. Date. Tag every task with its fixed date so it can be filtered against a model's cutoff.

Tasks come in three types: known-answer (a fix exists), open-limitation (no fix exists yet), and synthetic (a fix is reverted on top of HEAD). Known-answer tasks carry the highest leak risk and the strongest signal.

32 tasks

Active execution-graded set, drawn from 38 mined tasks. October 2026.

The set is built from real work, including FMMS, Glassbox, OpenLMlib, and CLOE. Each known-answer task ships with a repro command and a regression command, so grading is just a zero/non-zero exit code instead of an LLM opinion.

Build a sanitized worktree, then run one model across three tasks:

scripts/make-worktree.sh ISSUE-009
scripts/run-batch.sh "provider/model" ISSUE-009 ISSUE-011 ISSUE-039

Grade the reports in runs/ against the matching rubric. The rubric never enters the evaluated repository.

I want to expand the active set beyond 32 tasks, publish comparable model runs with cutoff dates, and package the harness so the results can be reproduced outside this machine.