A gate job waits for its step's runner only while that runner is free, keeps off a runner already gating its project, and names the runner that ran it #138

Merged

agra merged 5 commits from task/3785-a-gate-job-runs-only-on-the-runner-of-th into master

2026-10-25 16:26:27 +02:00

agra commented

2026-10-24 21:17:00 +03:00

Owner

Tracker: #3785

A gate job prefers the runner of its flow run's latest step, for its warm caches. Until now that runner kept the job to itself the whole time it held its channel. It did so even with every slot taken, or with another gate of the same project running there, while another runner that meets the job sat idle (jobs 57 and 104).

What this PR delivers

The step's runner is preferred only while it is free. To be preferred, it must hold its channel, meet the job, hold fewer run requests and jobs than its capacity, and hold no other gate job of the job's project, because a runner runs a project's gates one at a time.
A runner already gating the project is passed over. When the step's runner is not preferred, the job goes to the connected runners that meet it and hold no gate job of its project. Only when no such runner is connected does it go to every runner that meets it. So the step's runner, passed over because its project lock is held, no longer races the free runner for the job, a race it could win and then wait on its own lock with a slot held. Its claim answers 409 runner_cannot_run while a free runner holds its channel.
store.gate_runners makes this decision once, for both the channel's offers (offer_jobs) and the claim (claim_job), from the same list of connected runners (channel.connected_runners), so the two always agree.
The offers follow the runners' slots. A claim of a run request or of a job now offers the queued jobs again. So a job that preferred the claiming runner goes to the others as soon as that runner fills up, without waiting for some later event.
The flow run says where its gate ran. The text a gate step hands the next step now reads "The project's gate full failed at <head> on runner . Its output: …". So when a host fault (a held port, a full disk) stops a gate, the flow run names the host it stopped on.
docs/api.md (Flow runs, Jobs, the channel's job offer) and runner/README.md describe the rule.
A flow run dispatched on a named runner still sends its gate jobs to that runner alone. Nothing changes on the runners: they already claim an offered job only once they have a slot free.

What it leaves

Item 2 of the task's Next is not built, by Agra's choice. That item asked to move a gate job that failed on a host-fault signature to another runner, or to record the host as the reason. Retrying the job on another runner would work around the fault and leave it on the host for every later job. Each fault is instead removed where it arises:
- the fixed e2e ports by #3776, on master since 1681a1c;
- a full disk by #3784, where the runner stops claiming below a free-space bound;
- leaked processes by #3670.
The task's other ask, that the flow run says where a host fault stopped it, is met by the carried text naming the runner. When the step's runner is free but broken (job 112: s32 with port 18988 held), the job still runs there and fails, and the flow run goes failed → Implement, with the next step's prompt naming that runner.
sak learns of a run's own sak-gate runs only after they end. So a runner whose project lock is held by a run's in-run full (job 104's 10.6 min wait) still counts as free, provided it has a slot. Only gate jobs count toward the lock. A claimed job waiting on that lock says waiting on its stream, and its claimed_by names the runner.
#3784 (on master since 9b00833) makes a runner short of free space let an offered job go unclaimed. That check runs on the runner, and sak is not told why the job was let go. So a step's runner that is short of space but still holds its channel and a free slot stays the one runner the job goes to. The job waits until the runner reports room again, and its runner_id names that runner.

Regression

tests/named_runner.sx check_busy connects alpha and beta. It makes gate jobs whose step alpha ran, and whose Orchestrator beta ran:

While alpha has a slot free, beta's claim is refused 409 runner_cannot_run. This was already so.
Once alpha claims a run that fills its capacity (1), the job is offered on beta's channel, and beta's claim takes it. Beta's gate job then ends.
With alpha at capacity 3, holding that run and one claimed gate job of the project, alpha's claim of the next gate job its step queued answers 409 runner_cannot_run, though alpha has a slot free, and beta's claim takes it.
With both alpha and beta holding a gate job of the project, alpha claims the next one its step queued.

Each arm is a git archive tree holding this head's tests/named_runner.sx, run with the gate's pinned sx.

At 8a5679e's src/ (this PR's previous head), the step's runner passed over for the lock still races beta for the job:

FAIL while alpha holds a gate job of the project and beta none, alpha's claim of the next one its step queued -> 409 runner_cannot_run, though alpha has a slot free
FAIL and beta claims it
NAMED RUNNER FAIL (2 checks)

At master 8cda22f's src/:

FAIL once alpha claims a run that fills its capacity, the gate job is offered to beta
FAIL and beta's claim takes it
FAIL beta's gate job ends
FAIL while alpha holds a gate job of the project and beta none, alpha's claim of the next one its step queued -> 409 runner_cannot_run, though alpha has a slot free
FAIL and beta claims it
NAMED RUNNER FAIL (5 checks)

At this head it passes (NAMED RUNNER OK). One mutation arm per clause of gate_runners, and of the re-offer, fails only the checks that pin it:

dropping the step's runner's slot clause fails check 2 (the offer to beta, beta's claim, and so the end of beta's job);
dropping the step's runner's project-gate clause fails check 3, alpha's 409 and beta's claim. Alpha then holds three jobs and runs, so check 4's job goes to beta alone and fails too;
dropping the project-gate clause of the fallback to the connected runners fails check 3 alone;
dropping the fallback to every runner that meets the job, when none of them is free of a gate of its project, fails check 4 alone;
dropping the re-offer after a run claim fails check 2's offer to beta alone.

tests/runner_core.sx expects the Implement prompt after the failed probe gate to read "failed at <head> on runner runner-core-other".

Golden set

None. No file under web/ changes.

Gates at the head

a563eb5 merges master 3744ade into the branch. On master, claim_job takes the claiming process's slot and answers a repeated claim by the process that holds the job again. The merge keeps both, and a fresh claim of a gate job still goes through gate_runners. Gate step job 177 found 640d09d conflicting with master c76bb26, after Merge's job 176 saw the base move.
full at a563eb5 on s32: tracker gate run 461 (this run's #1), passed (11:20–11:51Z on 2026-10-25). That covers every tests/*.sx and the release with step9 and the deploy contract. It also covers both Playwright configs: the main one with 740 passed and 1 skipped, and the runner one with 10 passed.
Before gate run 461, tests/named_runner.sx and tests/runner_lost.sx (which drives the slotted job claims) passed on a563eb5's tree. With master 3744ade's src/, this head's tests/named_runner.sx fails the same 5 checks listed under Regression for master 8cda22f.
Earlier rounds, at 640d09d: full tracker gate runs 373 and 381 passed on s32. Gate step jobs 161 (m5) and 174 passed. Gate step jobs 140 and 166 failed only #3807's reattach flake in tests/runner_core.sx, which master now holds the fix for (3744ade).

Tracker: [#3785](https://sak.swipelab.com/tasks/3785) A gate job prefers the runner of its flow run's latest step, for its warm caches. Until now that runner kept the job to itself the whole time it held its channel. It did so even with every slot taken, or with another gate of the same project running there, while another runner that meets the job sat idle (jobs 57 and 104). ## What this PR delivers - **The step's runner is preferred only while it is free.** To be preferred, it must hold its channel, meet the job, hold fewer run requests and jobs than its capacity, and hold no other gate job of the job's project, because a runner runs a project's gates one at a time. - **A runner already gating the project is passed over.** When the step's runner is not preferred, the job goes to the connected runners that meet it and hold no gate job of its project. Only when no such runner is connected does it go to every runner that meets it. So the step's runner, passed over because its project lock is held, no longer races the free runner for the job, a race it could win and then wait on its own lock with a slot held. Its claim answers `409 runner_cannot_run` while a free runner holds its channel. - `store.gate_runners` makes this decision once, for both the channel's offers (`offer_jobs`) and the claim (`claim_job`), from the same list of connected runners (`channel.connected_runners`), so the two always agree. - **The offers follow the runners' slots.** A claim of a run request or of a job now offers the queued jobs again. So a job that preferred the claiming runner goes to the others as soon as that runner fills up, without waiting for some later event. - **The flow run says where its gate ran.** The text a gate step hands the next step now reads "The project's gate `full` failed at <head> on runner <name>. Its output: …". So when a host fault (a held port, a full disk) stops a gate, the flow run names the host it stopped on. - `docs/api.md` (Flow runs, Jobs, the channel's job offer) and `runner/README.md` describe the rule. - A flow run dispatched on a named runner still sends its gate jobs to that runner alone. Nothing changes on the runners: they already claim an offered job only once they have a slot free. ## What it leaves - **Item 2 of the task's Next is not built, by Agra's choice.** That item asked to move a gate job that failed on a host-fault signature to another runner, or to record the host as the reason. Retrying the job on another runner would work around the fault and leave it on the host for every later job. Each fault is instead removed where it arises: - the fixed e2e ports by #3776, on master since 1681a1c; - a full disk by #3784, where the runner stops claiming below a free-space bound; - leaked processes by #3670. The task's other ask, that the flow run says where a host fault stopped it, is met by the carried text naming the runner. When the step's runner is free but broken (job 112: s32 with port 18988 held), the job still runs there and fails, and the flow run goes `failed → Implement`, with the next step's prompt naming that runner. - sak learns of a run's own `sak-gate` runs only after they end. So a runner whose project lock is held by a run's in-run `full` (job 104's 10.6 min wait) still counts as free, provided it has a slot. Only gate jobs count toward the lock. A claimed job waiting on that lock says `waiting` on its stream, and its `claimed_by` names the runner. - #3784 (on master since 9b00833) makes a runner short of free space let an offered job go unclaimed. That check runs on the runner, and sak is not told why the job was let go. So a step's runner that is short of space but still holds its channel and a free slot stays the one runner the job goes to. The job waits until the runner reports room again, and its `runner_id` names that runner. ## Regression `tests/named_runner.sx` `check_busy` connects alpha and beta. It makes gate jobs whose step alpha ran, and whose Orchestrator beta ran: 1. While alpha has a slot free, beta's claim is refused `409 runner_cannot_run`. This was already so. 2. Once alpha claims a run that fills its capacity (1), the job is offered on beta's channel, and beta's claim takes it. Beta's gate job then ends. 3. With alpha at capacity 3, holding that run and one claimed gate job of the project, alpha's claim of the next gate job its step queued answers `409 runner_cannot_run`, though alpha has a slot free, and beta's claim takes it. 4. With both alpha and beta holding a gate job of the project, alpha claims the next one its step queued. Each arm is a `git archive` tree holding this head's `tests/named_runner.sx`, run with the gate's pinned sx. At 8a5679e's `src/` (this PR's previous head), the step's runner passed over for the lock still races beta for the job: ``` FAIL while alpha holds a gate job of the project and beta none, alpha's claim of the next one its step queued -> 409 runner_cannot_run, though alpha has a slot free FAIL and beta claims it NAMED RUNNER FAIL (2 checks) ``` At master 8cda22f's `src/`: ``` FAIL once alpha claims a run that fills its capacity, the gate job is offered to beta FAIL and beta's claim takes it FAIL beta's gate job ends FAIL while alpha holds a gate job of the project and beta none, alpha's claim of the next one its step queued -> 409 runner_cannot_run, though alpha has a slot free FAIL and beta claims it NAMED RUNNER FAIL (5 checks) ``` At this head it passes (`NAMED RUNNER OK`). One mutation arm per clause of `gate_runners`, and of the re-offer, fails only the checks that pin it: - dropping the step's runner's slot clause fails check 2 (the offer to beta, beta's claim, and so the end of beta's job); - dropping the step's runner's project-gate clause fails check 3, alpha's 409 and beta's claim. Alpha then holds three jobs and runs, so check 4's job goes to beta alone and fails too; - dropping the project-gate clause of the fallback to the connected runners fails check 3 alone; - dropping the fallback to every runner that meets the job, when none of them is free of a gate of its project, fails check 4 alone; - dropping the re-offer after a run claim fails check 2's offer to beta alone. `tests/runner_core.sx` expects the Implement prompt after the failed probe gate to read "failed at <head> on runner runner-core-other". ## Golden set None. No file under `web/` changes. ## Gates at the head - a563eb5 merges master 3744ade into the branch. On master, `claim_job` takes the claiming process's `slot` and answers a repeated claim by the process that holds the job again. The merge keeps both, and a fresh claim of a gate job still goes through `gate_runners`. Gate step job 177 found 640d09d conflicting with master c76bb26, after Merge's job 176 saw the base move. - `full` at a563eb5 on s32: tracker gate run 461 (this run's #1), passed (11:20–11:51Z on 2026-10-25). That covers every `tests/*.sx` and the release with step9 and the deploy contract. It also covers both Playwright configs: the main one with 740 passed and 1 skipped, and the runner one with 10 passed. - Before gate run 461, `tests/named_runner.sx` and `tests/runner_lost.sx` (which drives the slotted job claims) passed on a563eb5's tree. With master 3744ade's `src/`, this head's `tests/named_runner.sx` fails the same 5 checks listed under Regression for master 8cda22f. - Earlier rounds, at 640d09d: `full` tracker gate runs 373 and 381 passed on s32. Gate step jobs 161 (m5) and 174 passed. Gate step jobs 140 and 166 failed only #3807's reattach flake in `tests/runner_core.sx`, which master now holds the fix for (3744ade).

agra added 1 commit

2026-10-24 21:17:00 +03:00

A gate job waits for its step's runner only while that runner has a slot free and runs no other gate of the project 971d2bea66

The gate job is otherwise offered to, and may be claimed by, every connected
runner that meets it, and a claim offers the queued jobs again so that a job
goes to the others once its runner fills up. The text a gate step hands the
next step names the runner its gate ran on.

agra added 1 commit

2026-10-24 21:48:18 +03:00

docs: a gate job prefers its step's runner only while that runner is free, and its gate step names the runner it ran on 8a5679ec84

agra changed title from ~~A gate job waits for its step's runner only while that runner is free, and its failure names the runner that ran it~~ to A gate job waits for its step's runner only while that runner is free, keeps off a runner already gating its project, and names the runner that ran it

2026-10-24 23:58:30 +03:00

agra added 2 commits

2026-10-24 23:58:31 +03:00