The dropped-tail lease check waits until the slot is handed on, not a fixed heartbeat + lease + 1 s #137

Merged

agra merged 2 commits from task/3734-tests-step-concurrency-sx-s-dropped-tail into master

2026-10-25 03:17:44 +03:00

agra commented

2026-10-24 20:51:30 +03:00

Owner

Tracker: #3734

What this delivers

tests/step_concurrency.sx check_tail_capacity checks "the slot a dropped tail left behind is handed on once its lease lapses". It used to turn the server for a fixed tail.HEARTBEAT_MS + tail.LEASE_MS + 1000 ms (8 s) after dropping a held tail, then send one tail and expect it admitted. The slot comes back LEASE_MS after the dropped tail's last renewal. That renewal is the heartbeat after the server loop has seen the dropped connection, so it depends on how fast the server turns. Whenever it came more than about 2 s after the drop, the one tail was refused 503 and the check failed, though the slot came back moments later.

Now, after the drop, the check sends a probe tail for the spare stream and turns the server until the gate answers it. An admitted probe sends its retry: 500 greeting; a refused one gets a whole 503. A refused probe is dropped, and another is sent PROBE_MS (100 ms) later. The check passes once a probe is admitted, and fails only if none is admitted within 3 * (tail.HEARTBEAT_MS + tail.LEASE_MS) (21 s). A pass now ends when the slot comes back, about 7.2 s after the drop on an idle host, instead of after the full 8 s.

Only the test changes. No file under src/ or runner/ changes, and the check's name stays as it was.

Evidence

All runs were on m5, with binaries built by the gate cache's sx (300086027f) from git archive arms of this branch's base 8cda22f, which already has the clock-timed heartbeat (fa0995c).

Where the slot comes back. In an instrumented arm, the dropped tail's last heartbeat came 1.16 s after the drop, and the first admitted probe 7.23 s after the drop. So the old fixed probe at 8 s had about 0.8 s of slack for everything between the drop and that renewal. Under taskpolicy -b the figures were 1.08 s and 7.16 s.
Fail before, pass after. In both arms, a 2 s stall (2 * tail.HEARTBEAT_MS) was added to the server loop right after the check's clock starts. This is the "server's turns fall behind the clock" case the task describes. The old check failed 3 of 3 (FAIL the slot a dropped tail left behind is handed on once its lease lapses, CONCURRENCY FAIL (1 checks)). The new check passed 3 of 3, with the slot found 8.1–8.3 s after the drop.
Under parallel load. There were three rounds of 4 old and 4 new copies of the suite side by side, with 18 busy loops, at a one-minute load rising to 19. The rounds ran only while no other gate was open on the runner. Old and new each passed 12 of 12. With the clock-timed heartbeat on master, CPU load alone no longer pushes the renewal past the old slack, but a loop that falls behind does.
Gate history. Since fa0995c reached master (04:13Z on 2026-10-24), no full gate job has failed this check. The last one that did was job 91, on head 3ee9597, which lacked fa0995c. This change removes the fixed wait that left the check about 0.8 s of slack.

Gates

At head 529f3f1, on m5. That commit only rewords the comment above the probe loop, so the code is as reviewed at 7092fbd:

full: tracker gate run 375 (this run's #1), passed, 23:47–00:10 EEST. Every tests/*.sx file passed, tests/step_concurrency.sx included. Release and step9 passed. The main Playwright config passed 740 with 1 skipped in 3.9 min, and the runner config passed 10.

Earlier, at head 7092fbd, on m5:

fast: tracker gate run 353, passed, 20:51–20:52 EEST.
full: tracker gate run 355, passed, 20:52–21:13 EEST. Every tests/*.sx file passed, tests/step_concurrency.sx included; release and step9 passed; the main Playwright config passed 740 in 3.5 min; the runner config passed 10.
fast: tracker gate run 360, passed, 22:08–22:09 EEST.

Tracker: [#3734](https://sak.swipelab.com/tasks/3734) ## What this delivers `tests/step_concurrency.sx` `check_tail_capacity` checks "the slot a dropped tail left behind is handed on once its lease lapses". It used to turn the server for a fixed `tail.HEARTBEAT_MS + tail.LEASE_MS + 1000` ms (8 s) after dropping a held tail, then send one tail and expect it admitted. The slot comes back `LEASE_MS` after the dropped tail's last renewal. That renewal is the heartbeat after the server loop has seen the dropped connection, so it depends on how fast the server turns. Whenever it came more than about 2 s after the drop, the one tail was refused 503 and the check failed, though the slot came back moments later. Now, after the drop, the check sends a probe tail for the spare stream and turns the server until the gate answers it. An admitted probe sends its `retry: 500` greeting; a refused one gets a whole 503. A refused probe is dropped, and another is sent `PROBE_MS` (100 ms) later. The check passes once a probe is admitted, and fails only if none is admitted within `3 * (tail.HEARTBEAT_MS + tail.LEASE_MS)` (21 s). A pass now ends when the slot comes back, about 7.2 s after the drop on an idle host, instead of after the full 8 s. Only the test changes. No file under `src/` or `runner/` changes, and the check's name stays as it was. ## Evidence All runs were on m5, with binaries built by the gate cache's sx (300086027f) from `git archive` arms of this branch's base 8cda22f, which already has the clock-timed heartbeat (fa0995c). - **Where the slot comes back.** In an instrumented arm, the dropped tail's last heartbeat came 1.16 s after the drop, and the first admitted probe 7.23 s after the drop. So the old fixed probe at 8 s had about 0.8 s of slack for everything between the drop and that renewal. Under `taskpolicy -b` the figures were 1.08 s and 7.16 s. - **Fail before, pass after.** In both arms, a 2 s stall (`2 * tail.HEARTBEAT_MS`) was added to the server loop right after the check's clock starts. This is the "server's turns fall behind the clock" case the task describes. The old check failed 3 of 3 (`FAIL the slot a dropped tail left behind is handed on once its lease lapses`, `CONCURRENCY FAIL (1 checks)`). The new check passed 3 of 3, with the slot found 8.1–8.3 s after the drop. - **Under parallel load.** There were three rounds of 4 old and 4 new copies of the suite side by side, with 18 busy loops, at a one-minute load rising to 19. The rounds ran only while no other gate was open on the runner. Old and new each passed 12 of 12. With the clock-timed heartbeat on master, CPU load alone no longer pushes the renewal past the old slack, but a loop that falls behind does. - **Gate history.** Since fa0995c reached master (04:13Z on 2026-10-24), no `full` gate job has failed this check. The last one that did was job 91, on head 3ee9597, which lacked fa0995c. This change removes the fixed wait that left the check about 0.8 s of slack. ## Gates At head 529f3f1, on m5. That commit only rewords the comment above the probe loop, so the code is as reviewed at 7092fbd: - `full`: tracker gate run 375 (this run's #1), passed, 23:47–00:10 EEST. Every `tests/*.sx` file passed, `tests/step_concurrency.sx` included. Release and step9 passed. The main Playwright config passed 740 with 1 skipped in 3.9 min, and the runner config passed 10. Earlier, at head 7092fbd, on m5: - `fast`: tracker gate run 353, passed, 20:51–20:52 EEST. - `full`: tracker gate run 355, passed, 20:52–21:13 EEST. Every `tests/*.sx` file passed, `tests/step_concurrency.sx` included; release and step9 passed; the main Playwright config passed 740 in 3.5 min; the runner config passed 10. - `fast`: tracker gate run 360, passed, 22:08–22:09 EEST.

agra added 1 commit

2026-10-24 20:51:31 +03:00

The dropped-tail lease check probes until the slot is handed on, not once after a fixed wait 7092fbdf8b

check_tail_capacity dropped a held tail, turned the server for a fixed
HEARTBEAT_MS + LEASE_MS + 1000 ms, then sent one tail and expected it
admitted. When the server's turns fell behind the clock, the dropped tail's
last renewal came later than that wait allowed, so the one tail was refused
503 although the slot came back moments later. The check now sends a probe
tail every PROBE_MS until one is admitted, within a bound of three heartbeats
plus leases.

agra added 1 commit

2026-10-25 00:10:47 +03:00

The dropped-tail lease check's comment says the renewal comes later the slower the server turns 529f3f1b04

agra merged commit 1640bfa0e8 into master

2026-10-25 03:17:44 +03:00

agra referenced this pull request from a commit

2026-10-25 03:17:45 +03:00

Merge pull request 'The dropped-tail lease check waits until the slot is handed on, not a fixed heartbeat + lease + 1 s' (#137) from task/3734-tests-step-concurrency-sx-s-dropped-tail into master