j and k move between steps
Get the shape of the repository and whatever the last person left behind about the failure.
ls -R . && cat README.md.:
README.md
pytest.ini
src
tests
./src:
sampler
./src/sampler:
__init__.py
core.py
./tests:
test_sampler.py
# samplershowing 16 of 36 lines
That did not do what I expected. Trying it again.
ls -R . && cat README.md.:
README.md
pytest.ini
src
tests
./src:
sampler
./src/sampler:
__init__.py
core.py
./tests:
test_sampler.py
# samplershowing 16 of 36 lines
That did not do what I expected. Trying it again.
ls -R . && cat README.md.:
README.md
pytest.ini
src
tests
./src:
sampler
./src/sampler:
__init__.py
core.py
./tests:
test_sampler.py
# samplershowing 16 of 36 lines
Get the shape of the repository and whatever the last person left behind about the failure.
ls -R . && cat README.md.:
README.md
pytest.ini
src
tests
./src:
sampler
./src/sampler:
__init__.py
core.py
./tests:
test_sampler.py
# samplershowing 16 of 36 lines
Run the suite once. A single green run proves nothing here, which is the whole problem, so this is only to confirm the suite runs at all.
python3 -m pytest -q tests...... [100%]
6 passed in 0.01s
3 lines, 98 B
Measure the flakiness instead of taking CI's word for it. Twenty runs put the rate near one in four, which is a coin the test is tossing rather than an environment problem.
fails=0
for i in $(seq 1 20); do
python3 -m pytest -q -p no:cacheprovider tests >/dev/null 2>&1 || fails=$((fails + 1))
done
echo "failing runs out of 20: $fails"
6 lines, 165 B
failing runs out of 20: 6
2 lines, 26 B
Read the library. _shuffled calls random.shuffle on the process wide stream, and retry_delays calls random.uniform on it, so neither caller can pin a result.
{
"path": "src/sampler/core.py"
}3 lines, 35 B
"""Sampling helpers for the request auditor.
The auditor keeps a small sample of production requests for a human to review, and it
retries its uploads with exponential backoff. Both are random on purpose. A sample that
always picked the same requests would keep missing whatever the last sample missed, and a
backoff without jitter puts every retrying client back on the wire in the same
millisecond.
Randomness that reaches production has to stay random. Randomness that reaches a test has
to be controlled by the test, which is not the same thing.
"""
from __future__ import annotations
import random
from collections.abc import Sequenceshowing 16 of 131 lines
Read the test before changing it. It asserts tenant coverage, which is a real property of a draw and not something to weaken, so the draw has to become reproducible.
{
"path": "tests/test_sampler.py"
}3 lines, 37 B
"""Tests for the request auditor's sampling helpers.
The sampler is random by design, so these assert the properties the auditor depends on
rather than one exact sample.
"""
from sampler import Request, retry_delays, sample_requests, tenant_counts
TENANTS = ("acme", "globex", "initech", "umbrella")
PER_TENANT = 10
SAMPLE_SIZE = 8
def make_requests() -> list[Request]:
"""Build a fleet with an equal number of requests per tenant."""
return [showing 16 of 54 lines
Confirm the mechanism rather than assuming it. Seeding the global module reproduces the sample exactly, which proves the library is drawing from that stream, and the coverage count across unseeded draws shows the 3 that the failing runs see.
python3 -c '
import random, sys
sys.path.insert(0, "src")
from sampler import Request, sample_requests, tenant_counts
tenants = ("acme", "globex", "initech", "umbrella")
fleet = [Request(f"req-{i:04d}", t, 20 + i) for i, t in enumerate(tenants * 10)]
random.seed(11)
first = sorted(tenant_counts(sample_requests(fleet, 8)))showing 8 of 14 lines
global seed 11 twice: True ['acme', 'globex', 'initech', 'umbrella']
tenants covered over 12 unseeded draws: [4, 4, 4, 3, 4, 2, 4, 4, 4, 3, 4, 4]
3 lines, 146 B
Make the random source a parameter on both functions that draw, threaded through _shuffled, defaulting to the module level stream. Production keeps the behaviour it has today and callers that need a reproducible draw can now ask for one.
"""Sampling helpers for the request auditor.
The auditor keeps a small sample of production requests for a human to review, and it
retries its uploads with exponential backoff. Both are random on purpose. A sample that
always picked the same requests would keep missing whatever the last sample missed, and a
backoff without jitter puts every retrying client back on the wire in the same
millisecond.
Randomness that reaches production has to stay random. Randomness that reaches a test has
to be controlled by the test, which is not the same thing. Every function here that draws
randomness takes an optional `rng`: pass a `random.Random` and the result is reproducible,
leave it out and the function draws from the module level `random` stream exactly as it
always has.
"""
from __future__ import annotations
import randomshowing 18 of 159 lines
{
"path": "src/sampler/core.py"
}3 lines, 35 B
Wrote 5450 bytes (159 lines) to src/sampler/core.py.Pin every test that depends on a specific draw to a seeded Random. The coverage assertion is unchanged, which is the point: it now tests the sampler instead of the interpreter's entropy.
"""Tests for the request auditor's sampling helpers.
The sampler is random by design, so these assert the properties the auditor depends on
rather than one exact sample. Anything that has to hold for a specific draw takes a seeded
`random.Random`, because a test that depends on an unseeded global RNG is a test that
fails on someone else's branch.
"""
import random
from sampler import Request, retry_delays, sample_requests, tenant_counts
TENANTS = ("acme", "globex", "initech", "umbrella")
PER_TENANT = 10
SAMPLE_SIZE = 8
SEED = 1337
showing 18 of 63 lines
{
"path": "tests/test_sampler.py"
}3 lines, 37 B
Wrote 2403 bytes (63 lines) to tests/test_sampler.py.Twenty more runs. Zero failures is the only acceptable answer for a merge queue.
fails=0
for i in $(seq 1 20); do
python3 -m pytest -q -p no:cacheprovider tests >/dev/null 2>&1 || fails=$((fails + 1))
done
echo "failing runs out of 20: $fails"
6 lines, 165 B
failing runs out of 20: 0
2 lines, 26 B
Check the other half of the contract. A seeded draw is one result across 200 calls, an unseeded draw is many, and the backoff still jitters, so nothing about production sampling was quietly pinned to a constant.
python3 -c '
import random, sys
sys.path.insert(0, "src")
from sampler import Request, retry_delays, sample_requests
fleet = [Request(f"req-{i:04d}", t, 20 + i) for i, t in enumerate(("a", "b", "c", "d") * 10)]
seeded = {tuple(r.request_id for r in sample_requests(fleet, 8, rng=random.Random(1337))) for _ in range(200)}
unseeded = {tuple(r.request_id for r in sample_requests(fleet, 8)) for _ in range(200)}
print("distinct draws, seeded:", len(seeded), "unseeded:", len(unseeded))
print("jitter still varies:", len({tuple(retry_delays(4)) for _ in range(50)}) > 1)
'
11 lines, 570 B
distinct draws, seeded: 1 unseeded: 200
jitter still varies: True
3 lines, 66 B
That did not do what I expected. Trying it again.
python3 -c '
import random, sys
sys.path.insert(0, "src")
from sampler import Request, retry_delays, sample_requests
fleet = [Request(f"req-{i:04d}", t, 20 + i) for i, t in enumerate(("a", "b", "c", "d") * 10)]
seeded = {tuple(r.request_id for r in sample_requests(fleet, 8, rng=random.Random(1337))) for _ in range(200)}
unseeded = {tuple(r.request_id for r in sample_requests(fleet, 8)) for _ in range(200)}
print("distinct draws, seeded:", len(seeded), "unseeded:", len(unseeded))
print("jitter still varies:", len({tuple(retry_delays(4)) for _ in range(50)}) > 1)
'
11 lines, 570 B
distinct draws, seeded: 1 unseeded: 200
jitter still varies: True
3 lines, 66 B
Check the other half of the contract. A seeded draw is one result across 200 calls, an unseeded draw is many, and the backoff still jitters, so nothing about production sampling was quietly pinned to a constant.
python3 -c '
import random, sys
sys.path.insert(0, "src")
from sampler import Request, retry_delays, sample_requests
fleet = [Request(f"req-{i:04d}", t, 20 + i) for i, t in enumerate(("a", "b", "c", "d") * 10)]
seeded = {tuple(r.request_id for r in sample_requests(fleet, 8, rng=random.Random(1337))) for _ in range(200)}
unseeded = {tuple(r.request_id for r in sample_requests(fleet, 8)) for _ in range(200)}
print("distinct draws, seeded:", len(seeded), "unseeded:", len(unseeded))
print("jitter still varies:", len({tuple(retry_delays(4)) for _ in range(50)}) > 1)
'
11 lines, 570 B
distinct draws, seeded: 1 unseeded: 200
jitter still varies: True
3 lines, 66 B
Say what the defect was and why the default path is still random, because that is the part a reviewer will ask about.
sample_requests and retry_delays drew from the process wide random stream, so test_sample_covers_every_tenant was asserting a property that holds for about 70 percent of possible draws: hence one failing run in four. Both functions now take an optional rng and default to the module level stream, so production sampling is unchanged and still random, and the tests pass a seeded random.Random. The coverage assertion is untouched and the suite passed 20 consecutive runs.
2 lines, 472 B
Run ended by the agent.16 steps11 commands0 schema violations0 failed commands0 destructive attempts
The last few kilobytes of the hidden test run, stdout and stderr together, kept for triage. The agent never saw this.
......... [100%]
9 passed in 1.81s
3 lines, 98 B
Results that cannot be reproduced are not results. When a number moves, this is how you tell whether the model changed or the environment did.