On ARC-AGI-3, the same model goes from 30% to 96% by changing harness
On 12 August Jeremy Berman published the code of a harness that has an agent play the 25 games of the ARC-AGI-3 benchmark. The repository reports 30.2% for the model on its own, 96.2% with the harness — 24 games won out of 25, 99.3% pass@2 with all 25 won, for roughly $540.
The harness is almost nothing ARC-specific: Claude Code, six pre-approved tools (Bash, Read, Write, Edit, Grep, Glob), no MCP server, no subagents, no plugins. In a single pass the model wrote 269 programs, about 12,700 lines, all discarded when their game ended. The same harness with Codex: 73.7%, and 26,971 actions against 8,789.