runner_core's gate re-attach check kills the first sak-gate only once the gate's own `begun` line is in its output #148

Merged

agra merged 1 commit from task/3807-runner-core-s-gate-re-attach-check-gives into master

2026-10-25 12:41:36 +02:00

agra commented

2026-10-25 08:04:56 +02:00

Owner

Tracker: #3807

check_gate_reattach in tests/runner_core.sx starts sak-gate held in the background and kills that call once the gate has printed begun. That is how it shows a killed call printed only the gate's start. Its script had two ways to kill the call before the gate's own begun:

grep -q begun also matched the $ echo begun; … line. sak-gate prints that line for the command before the command runs (run_gate, runner/gate.sx), and the gate server hands it to the client at once. So the kill could land in the gap between that line and the gate shell's own begun. Under load the shell starts more slowly and the gap widens. This is the failure seen without any patch, below.
It waited at most 100 × 0.1 s for begun, then killed the call anyway. On m5 that came to 24 s today, and 30 s under taskpolicy -b. A gate slow to start lost the check.

Both give the signature of gate job 166 on m5 (#3785 at 640d09d onto ad26ff3): only "a sak-gate killed while its gate runs has printed only the gate's start" fails, while the re-attach and the gate's end pass.

The script now waits with until grep -qx begun "$o/first.out"; do sleep 0.1; done. It matches only the gate's own begun line and has no cap of its own. If a gate never starts, the runner's wall timeout (60 s for this runner) ends the run, and the check fails.

Only tests/runner_core.sx changes.

Fail before, pass after

Each arm is a git archive of the commit. It ran sx run tests/runner_core.sx with the gate cache's sx (pin 300086027f) under taskpolicy -b on m5. The arms printed what the killed call wrote to first.out, using one added print after the reattach run.

Probe	Base `9b00833`	Head `c449b77`
The held gate's command starts with `sleep 1;` (its shell is slow to echo `begun`)	`RUNNER CORE FAIL (1 checks)`, the reattach check. `first.out` holds the id and the `$ sleep 1; echo begun; …` line, and no `begun`	`RUNNER CORE OK`. `first.out` ends `begun`
The held gate gets the setup `sleep 40` (it is slow to start), and the expected re-attach output starts with `$ sleep 40`	`RUNNER CORE FAIL (1 checks)`, the reattach check. `first.out` holds the id and `$ sleep 40`: the cap had run out	`RUNNER CORE OK`

Under load

These rounds ran with no probe patch, on m5 between 08:12 and 08:35 EET. Run 784's sak full gates were running beside them on the same host, and every arm ran under taskpolicy -b. Three arms ran side by side at one-minute loads of 3.9–10.9.

Base 9b00833: 2 of 6 rounds RUNNER CORE FAIL (1 checks), each on the reattach check alone. Three of the six rounds added only the first.out print. The failing one of those (round 5) shows first.out holding the id and the $ echo begun; … line, and no begun. Its gate started and ended within the same second, far inside the cap, so the word match was the cause.
Head c449b77: 9 of 9 rounds RUNNER CORE OK, in two arms.

Gates

fast #1, tracker gate run 430, at c449b77 with a clean tree: passed.
full #2, tracker gate run 434, at c449b77 with a clean tree: passed (08:38–08:59 EET on m5). That covers every tests/*.sx file, runner_core included, then release with step9, the deploy contract, the main Playwright config (740 passed) and the runner config (10 passed).

Tracker: [#3807](https://sak.swipelab.com/tasks/3807) `check_gate_reattach` in `tests/runner_core.sx` starts `sak-gate held` in the background and kills that call once the gate has printed `begun`. That is how it shows a killed call printed only the gate's start. Its script had two ways to kill the call before the gate's own `begun`: 1. `grep -q begun` also matched the `$ echo begun; …` line. sak-gate prints that line for the command before the command runs (`run_gate`, runner/gate.sx), and the gate server hands it to the client at once. So the kill could land in the gap between that line and the gate shell's own `begun`. Under load the shell starts more slowly and the gap widens. This is the failure seen without any patch, below. 2. It waited at most 100 × 0.1 s for `begun`, then killed the call anyway. On m5 that came to 24 s today, and 30 s under `taskpolicy -b`. A gate slow to start lost the check. Both give the signature of gate job 166 on m5 (#3785 at 640d09d onto ad26ff3): only "a sak-gate killed while its gate runs has printed only the gate's start" fails, while the re-attach and the gate's end pass. The script now waits with `until grep -qx begun "$o/first.out"; do sleep 0.1; done`. It matches only the gate's own `begun` line and has no cap of its own. If a gate never starts, the runner's wall timeout (60 s for this runner) ends the run, and the check fails. Only `tests/runner_core.sx` changes. ## Fail before, pass after Each arm is a `git archive` of the commit. It ran `sx run tests/runner_core.sx` with the gate cache's sx (pin 300086027f) under `taskpolicy -b` on m5. The arms printed what the killed call wrote to `first.out`, using one added `print` after the reattach run. | Probe | Base 9b00833 | Head c449b77 | |---|---|---| | The held gate's command starts with `sleep 1;` (its shell is slow to echo `begun`) | `RUNNER CORE FAIL (1 checks)`, the reattach check. `first.out` holds the id and the `$ sleep 1; echo begun; …` line, and no `begun` | `RUNNER CORE OK`. `first.out` ends `begun` | | The held gate gets the setup `sleep 40` (it is slow to start), and the expected re-attach output starts with `$ sleep 40` | `RUNNER CORE FAIL (1 checks)`, the reattach check. `first.out` holds the id and `$ sleep 40`: the cap had run out | `RUNNER CORE OK` | ## Under load These rounds ran with no probe patch, on m5 between 08:12 and 08:35 EET. Run 784's sak `full` gates were running beside them on the same host, and every arm ran under `taskpolicy -b`. Three arms ran side by side at one-minute loads of 3.9–10.9. - Base 9b00833: 2 of 6 rounds `RUNNER CORE FAIL (1 checks)`, each on the reattach check alone. Three of the six rounds added only the `first.out` print. The failing one of those (round 5) shows `first.out` holding the id and the `$ echo begun; …` line, and no `begun`. Its gate started and ended within the same second, far inside the cap, so the word match was the cause. - Head c449b77: 9 of 9 rounds `RUNNER CORE OK`, in two arms. ## Gates - `fast` #1, tracker gate run 430, at c449b77 with a clean tree: passed. - `full` #2, tracker gate run 434, at c449b77 with a clean tree: passed (08:38–08:59 EET on m5). That covers every `tests/*.sx` file, runner_core included, then release with step9, the deploy contract, the main Playwright config (740 passed) and the runner config (10 passed).

agra added 1 commit

2026-10-25 08:04:56 +02:00

The gate re-attach check kills the first sak-gate once the gate's own begun line is in its output, however long the gate takes to start c449b77c32

agra changed title from ~~runner_core's gate re-attach check waits for the held gate's start as long as the gate takes to start~~ to runner_core's gate re-attach check kills the first sak-gate only once the gate's own begun line is in its output

2026-10-25 09:00:19 +02:00

agra merged commit 3744adecf7 into master

2026-10-25 12:41:36 +02:00

agra referenced this pull request from a commit

2026-10-25 12:41:37 +02:00

Merge pull request 'runner_core's gate re-attach check kills the first sak-gate only once the gate's own `begun` line is in its output' (#148) from task/3807-runner-core-s-gate-re-attach-check-gives into master