cli-implement-01core-12pythonDifficulty tier 3/5
Implement a log summariser from its specification alone
specificationclihidden-tests-onlyexit-codesoutput-format
Task parameters
- Reference steps
- 10
- Step ceiling
- 40
- Runs
- 15
- Solved
- 14 of 15
- Models
- 5
What is broken, and what fixed means
The workspace holds a complete written specification for a command line log summariser and a stub that raises NotImplementedError. Nothing else. This is the only task in the suite with no visible tests, which is the point: it measures whether an agent reads a specification or pattern matches on one. SPEC.md fixes the three subcommands and their two options, the exit code for every failure mode, the exact column widths and ordering of every line of output, the tie break rule, how blank and malformed lines are treated, and what happens on empty input. It carries four worked examples with exact expected output. Fixed means the hidden tests, which are derived from the specification rather than from the examples, all pass. The shortcuts that make the examples come out right while the specification does not hold are splitting a record on every pipe, reading the span off the first and last line of the file, skipping malformed lines instead of failing on them, and using fixed column widths.
Results by model
One group per model. The solve rate carries its spread across seeds, and every run below it links to the full step by step replay.
stub:hasty
66.7%+/- 47.1% over 3 seeds
3 runs, seeds 0, 1, 2
- seed 0solved 10/10 hidden testscompleteddockerreplay
- steps
- 7 of 40
- partial credit
- 100%
- step efficiency
- 1.000
- recovery
- n/a
- cost
- $0.00
- wall clock
- 1.52 s
- started
- 16 Sep 2026, 18:45 UTC
- seed 1not solved 0/10 hidden testscompleteddockerpremature finishF03replay
- steps
- 5 of 40
- partial credit
- 0%
- step efficiency
- n/a
- recovery
- n/a
- cost
- $0.00
- wall clock
- 921 ms
- started
- 16 Sep 2026, 18:45 UTC
- seed 2solved 10/10 hidden testscompleteddockerreplay
- steps
- 8 of 40
- partial credit
- 100%
- step efficiency
- 1.000
- recovery
- n/a
- cost
- $0.00
- wall clock
- 1.59 s
- started
- 16 Sep 2026, 18:45 UTC
stub:methodical
100.0%+/- 0.0% over 3 seeds
3 runs, seeds 0, 1, 2
- seed 0solved 10/10 hidden testscompleteddockerreplay
- steps
- 10 of 40
- partial credit
- 100%
- step efficiency
- 1.000
- recovery
- n/a
- cost
- $0.00
- wall clock
- 2.18 s
- started
- 16 Sep 2026, 18:43 UTC
- seed 1solved 10/10 hidden testscompleteddockerreplay
- steps
- 10 of 40
- partial credit
- 100%
- step efficiency
- 1.000
- recovery
- n/a
- cost
- $0.00
- wall clock
- 3.15 s
- started
- 16 Sep 2026, 18:43 UTC
- seed 2solved 10/10 hidden testscompleteddockerreplay
- steps
- 10 of 40
- partial credit
- 100%
- step efficiency
- 1.000
- recovery
- n/a
- cost
- $0.00
- wall clock
- 1.77 s
- started
- 16 Sep 2026, 18:43 UTC
stub:reckless
100.0%+/- 0.0% over 3 seeds
3 runs, seeds 0, 1, 2
- seed 0solved 10/10 hidden testscompleteddockerF08F09replay
- steps
- 13 of 40
- partial credit
- 100%
- step efficiency
- 0.769
- recovery
- 100%
- cost
- $0.00
- wall clock
- 1.76 s
- started
- 16 Sep 2026, 18:51 UTC
- seed 1solved 10/10 hidden testscompleteddockerF08F09replay
- steps
- 13 of 40
- partial credit
- 100%
- step efficiency
- 0.769
- recovery
- 100%
- cost
- $0.00
- wall clock
- 1.75 s
- started
- 16 Sep 2026, 18:51 UTC
- seed 2solved 10/10 hidden testscompleteddockerF08F09replay
- steps
- 13 of 40
- partial credit
- 100%
- step efficiency
- 0.769
- recovery
- 100%
- cost
- $0.00
- wall clock
- 1.76 s
- started
- 16 Sep 2026, 18:51 UTC
stub:sloppy
100.0%+/- 0.0% over 3 seeds
3 runs, seeds 0, 1, 2
- seed 0solved 10/10 hidden testscompleteddockerF01F02F05replay
- steps
- 13 of 40
- partial credit
- 100%
- step efficiency
- 0.769
- recovery
- 100%
- cost
- $0.00
- wall clock
- 1.74 s
- started
- 16 Sep 2026, 18:49 UTC
- seed 1solved 10/10 hidden testscompleteddockerF01F02F05replay
- steps
- 13 of 40
- partial credit
- 100%
- step efficiency
- 0.769
- recovery
- 100%
- cost
- $0.00
- wall clock
- 1.75 s
- started
- 16 Sep 2026, 18:49 UTC
- seed 2solved 10/10 hidden testscompleteddockerF01F02F05replay
- steps
- 13 of 40
- partial credit
- 100%
- step efficiency
- 0.769
- recovery
- 100%
- cost
- $0.00
- wall clock
- 1.75 s
- started
- 16 Sep 2026, 18:49 UTC
stub:thrasher
100.0%+/- 0.0% over 3 seeds
3 runs, seeds 0, 1, 2
- seed 0solved 10/10 hidden testscompleteddockerF04replay
- steps
- 15 of 40
- partial credit
- 100%
- step efficiency
- 0.667
- recovery
- n/a
- cost
- $0.00
- wall clock
- 2.05 s
- started
- 16 Sep 2026, 18:46 UTC
- seed 1solved 10/10 hidden testscompleteddockerF04replay
- steps
- 15 of 40
- partial credit
- 100%
- step efficiency
- 0.667
- recovery
- n/a
- cost
- $0.00
- wall clock
- 1.92 s
- started
- 16 Sep 2026, 18:46 UTC
- seed 2solved 10/10 hidden testscompleteddockerF04replay
- steps
- 15 of 40
- partial credit
- 100%
- step efficiency
- 0.667
- recovery
- n/a
- cost
- $0.00
- wall clock
- 2.25 s
- started
- 16 Sep 2026, 18:46 UTC