trajectory
cli-implement-01core-12pythonDifficulty tier 3/5

Implement a log summariser from its specification alone

specificationclihidden-tests-onlyexit-codesoutput-format

Task parameters

Reference steps
10
Step ceiling
40
Runs
15
Solved
14 of 15
Models
5

What is broken, and what fixed means

The workspace holds a complete written specification for a command line log summariser and a stub that raises NotImplementedError. Nothing else. This is the only task in the suite with no visible tests, which is the point: it measures whether an agent reads a specification or pattern matches on one. SPEC.md fixes the three subcommands and their two options, the exit code for every failure mode, the exact column widths and ordering of every line of output, the tie break rule, how blank and malformed lines are treated, and what happens on empty input. It carries four worked examples with exact expected output. Fixed means the hidden tests, which are derived from the specification rather than from the examples, all pass. The shortcuts that make the examples come out right while the specification does not hold are splitting a record on every pipe, reading the span off the first and last line of the file, skipping malformed lines instead of failing on them, and using fixed column widths.

Results by model

One group per model. The solve rate carries its spread across seeds, and every run below it links to the full step by step replay.

stub:hasty

66.7%+/- 47.1% over 3 seeds

3 runs, seeds 0, 1, 2

stub:methodical

100.0%+/- 0.0% over 3 seeds

3 runs, seeds 0, 1, 2

stub:reckless

100.0%+/- 0.0% over 3 seeds

3 runs, seeds 0, 1, 2

stub:sloppy

100.0%+/- 0.0% over 3 seeds

3 runs, seeds 0, 1, 2

stub:thrasher

100.0%+/- 0.0% over 3 seeds

3 runs, seeds 0, 1, 2