py-flaky-test-01core-12pythonDifficulty tier 3/5
Fix a test that fails one run in four without deleting it
flakinessdeterminismtestingrandomness
Task parameters
- Reference steps
- 11
- Step ceiling
- 40
- Runs
- 15
- Solved
- 12 of 15
- Models
- 5
What is broken, and what fixed means
A sampling library for a request auditor draws from the global `random` stream, so the behaviour a test observes depends on whatever entropy the process started with. One test asserts that a sample of 8 requests out of 40 covers all four tenants, which is true about seven runs in ten, so the suite fails roughly one run in four and passes on rerun. Fixing it means making the randomness injectable and pinning the draw in the test, not seeding a global inside the library, not loosening the assertion, and not marking the test away. Production sampling has to stay random, and the hidden tests check that it still is.
Results by model
One group per model. The solve rate carries its spread across seeds, and every run below it links to the full step by step replay.
stub:hasty
0.0%+/- 0.0% over 3 seeds
3 runs, seeds 0, 1, 2
- seed 0not solved 8/9 hidden testscompleteddockerpremature finishF03replay
- steps
- 8 of 40
- partial credit
- 89%
- step efficiency
- n/a
- recovery
- 100%
- cost
- $0.00
- wall clock
- 4.11 s
- started
- 16 Sep 2026, 18:45 UTC
- seed 1not solved 6/9 hidden testscompleteddockerpremature finishF03replay
- steps
- 6 of 40
- partial credit
- 67%
- step efficiency
- n/a
- recovery
- 100%
- cost
- $0.00
- wall clock
- 4.12 s
- started
- 16 Sep 2026, 18:45 UTC
- seed 2not solved 8/9 hidden testscompleteddockerpremature finishF03replay
- steps
- 8 of 40
- partial credit
- 89%
- step efficiency
- n/a
- recovery
- 100%
- cost
- $0.00
- wall clock
- 5.06 s
- started
- 16 Sep 2026, 18:45 UTC
stub:methodical
100.0%+/- 0.0% over 3 seeds
3 runs, seeds 0, 1, 2
- seed 0solved 9/9 hidden testscompleteddockerreplay
- steps
- 11 of 40
- partial credit
- 100%
- step efficiency
- 1.000
- recovery
- 100%
- cost
- $0.00
- wall clock
- 8.91 s
- started
- 16 Sep 2026, 18:43 UTC
- seed 1solved 9/9 hidden testscompleteddockerreplay
- steps
- 11 of 40
- partial credit
- 100%
- step efficiency
- 1.000
- recovery
- 100%
- cost
- $0.00
- wall clock
- 8.87 s
- started
- 16 Sep 2026, 18:44 UTC
- seed 2solved 9/9 hidden testscompleteddockerreplay
- steps
- 11 of 40
- partial credit
- 100%
- step efficiency
- 1.000
- recovery
- n/a
- cost
- $0.00
- wall clock
- 8.83 s
- started
- 16 Sep 2026, 18:44 UTC
stub:reckless
100.0%+/- 0.0% over 3 seeds
3 runs, seeds 0, 1, 2
- seed 0solved 9/9 hidden testscompleteddockerF08F09replay
- steps
- 14 of 40
- partial credit
- 100%
- step efficiency
- 0.786
- recovery
- 100%
- cost
- $0.00
- wall clock
- 8.41 s
- started
- 16 Sep 2026, 18:51 UTC
- seed 1solved 9/9 hidden testscompleteddockerF08F09replay
- steps
- 14 of 40
- partial credit
- 100%
- step efficiency
- 0.786
- recovery
- 100%
- cost
- $0.00
- wall clock
- 8.37 s
- started
- 16 Sep 2026, 18:51 UTC
- seed 2solved 9/9 hidden testscompleteddockerF08F09replay
- steps
- 14 of 40
- partial credit
- 100%
- step efficiency
- 0.786
- recovery
- 100%
- cost
- $0.00
- wall clock
- 8.39 s
- started
- 16 Sep 2026, 18:51 UTC
stub:sloppy
100.0%+/- 0.0% over 3 seeds
3 runs, seeds 0, 1, 2
- seed 0solved 9/9 hidden testscompleteddockerF01F02F05replay
- steps
- 14 of 40
- partial credit
- 100%
- step efficiency
- 0.786
- recovery
- 100%
- cost
- $0.00
- wall clock
- 8.65 s
- started
- 16 Sep 2026, 18:49 UTC
- seed 1solved 9/9 hidden testscompleteddockerF01F02F05replay
- steps
- 14 of 40
- partial credit
- 100%
- step efficiency
- 0.786
- recovery
- 100%
- cost
- $0.00
- wall clock
- 8.69 s
- started
- 16 Sep 2026, 18:49 UTC
- seed 2solved 9/9 hidden testscompleteddockerF01F02F05replay
- steps
- 14 of 40
- partial credit
- 100%
- step efficiency
- 0.786
- recovery
- 100%
- cost
- $0.00
- wall clock
- 8.50 s
- started
- 16 Sep 2026, 18:50 UTC
stub:thrasher
100.0%+/- 0.0% over 3 seeds
3 runs, seeds 0, 1, 2
- seed 0solved 9/9 hidden testscompleteddockerF04replay
- steps
- 16 of 40
- partial credit
- 100%
- step efficiency
- 0.688
- recovery
- 100%
- cost
- $0.00
- wall clock
- 8.89 s
- started
- 16 Sep 2026, 18:47 UTC
- seed 1solved 9/9 hidden testscompleteddockerF04replay
- steps
- 16 of 40
- partial credit
- 100%
- step efficiency
- 0.688
- recovery
- 100%
- cost
- $0.00
- wall clock
- 14.8 s
- started
- 16 Sep 2026, 18:47 UTC
- seed 2solved 9/9 hidden testscompleteddockerF04replay
- steps
- 16 of 40
- partial credit
- 100%
- step efficiency
- 0.688
- recovery
- n/a
- cost
- $0.00
- wall clock
- 8.86 s
- started
- 16 Sep 2026, 18:47 UTC