Every flow run goes through its queue, and a paused run lends its slot to the runs that unblock it #205

Merged

agra merged 4 commits from task/3998-every-flow-run-goes-through-its-queue-an into master 2026-10-31 19:40:01 +02:00

Conversation 0 Commits 4 Files changed 33 +886 -323

agra commented 2026-10-31 18:18:28 +02:00

Owner

Copy link

Tracker: #3998

Step 1 of plan #3997.

What it delivers

• begin_flow(db, id, lent_by, queued) has one caller, advance_queues. take_out and dispatch_step are gone, and so is flow_run.from_queue: the column, FlowRun.from_queue, the API member and the web type.
• A Wait dispatches each undispatched blocker into its queue through enqueue_flow and begins none; a blocker already queued stays where it is, and reevaluate_wait begins none either. Run steps reads the step's latest flow run, begun or not: with none it queues one and holds, it holds while that run is queued, running or blocked, and it concludes a step needs you while the run waits on Agra or once it was stopped, begun or not. A blocker's or a step's run waits in its queue on what its task waits on, as every queued run does.
• The kernel, in src/store.sx: one SQL reading of done (done_sql, which task_done reads and done_by goes through); one recursive walk, unblocks, over blocks, follows and parent (parent only while the parent's live flow run is at Run steps), each kind of edge a recursive SELECT of its own that joins relation from the reached task by its primary key or relation_other; going (begun and not ended, QUEUE_GOING_SQL); paused; and lent slots, a largest matching of a queue's going runs to the paused runs they unblock (queue_lending, augment).
• advance_queues takes each queue in turn (begin_next): a queued run whose task waits on nothing begins on a lent slot when an augmenting path from it reaches a paused run lending to no going run, while the going runs that are not paused are fewer than slots; with no such path, on a slot of its own while the going runs left unmatched are fewer. 0 slots begins nothing, and a queue whose going runs that are not paused fill its slots begins nothing before any walk (queue_going). dequeued records lent_by: the paused run of its queue the run began on, else null.
• Every going run is in its queue's going and answers behind; done_today holds every run of the queue that began and ended today; GET /api/queues and GET /api/projects/:id/queues list a plan's branch while a step's run is queued or going on it.
• The Wait's blockers give each blocker its live flow run, else its latest started one (BlockingTask.flow_run), so a blocker waiting its turn answers its queued run. The comments of BlockingTask, blockers_json, web/src/lib/types.ts, web/src/lib/flow-runs.ts and web/src/lib/FlowCard.svelte say so.
• MIGRATION_59 rewrites every stored dequeued event to {"lent_by": …} (the flow run its flow_run_id named when that one is of its run's project and base, else null) and drops flow_run.from_queue. tests/support/schema.sx gains BACK_TO_58, and back_to_57() and back_to_55() go through it.
• docs/api.md: Flow runs, the history table, Plans, The Wait and Queues.

Regressions (fail on master, pass at the head)

The tests are in tests/flow_queue.sx and tests/blockers.sx; began in tests/flow_queue.sx reads lent_by. Built against master's src/, tests/flow_queue.sx ends FLOW QUEUE FAIL (38 checks) and tests/blockers.sx BLOCKERS FAIL (9 checks). Every began check fails on master because master writes no lent_by, so the behaviour master shows for each regression was read with a probe that drives the same scenarios through the API, built against master's src/ and against the head:

regression	master	head
A Wait's blocker in its holder's queue at 1 slot	going [1], blocker behind null, dequeued {"from_queue":false,"flow_run_id":1}	going [1,2], holder and blocker behind 0, dequeued {"lent_by":1}
Five blockers in the holder's queue at 1 slot	the five runs running at once	one running, four queued; each begins on the lent slot as the one before ends done, and the holder resumes after the fifth
The holder's blockers	each answers running, begun outside the slots	the four waiting answer queued, undispatched false, the fifth its queued run and not its stopped one
Bounded by the going runs that are not paused (P1, P2 at 1 slot)	B1 running as P1's Wait is reached, B2 running as P2's is; going [10,11]	B1 queued while P2 runs, then running on P1's lent slot once P2 is held, B2 queued until P1 ends
A blocker in another project's queue	running while that queue's slot is held, dequeued {"from_queue":false,"flow_run_id":15}, absent from going [14]	queued until the slot frees, then dequeued {"lent_by":null}
Run steps, a step's run whose task is blocked	the step running at once, absent from going [17]	the step queued, its waits_on naming the blocking task; stopped, the plan asks Agra; dispatched again, the plan holds; the task done, the step begins on the plan's lent slot
Run steps, plan with a branch	GET /api/projects/:id/queues lists master only	it lists task/19-the-plan with the step going
A database at 58	reopened at 58, from_queue kept, each dequeued still {"from_queue":…,"flow_run_id":…}	59, from_queue gone, the run begun outside the slots going, pragma_foreign_key_check empty, {"lent_by":1} for the Wait's blocker of its own queue and {"lent_by":null} for the blocker in another project, the step on its branch and a run its queue began
No flow run answers from_queue	a going run answers … no from_queue FAILs	passes

The other cases (the lent slot lent on by a blocker held at its own Wait, lending only to runs that unblock the paused run, the largest matching, 0 slots, done_today, a blocks relation made to a held run's task, a blocker whose task is itself blocked) fail on master as listed in the runs above and pass at the head.

waitOf in web/src/lib/flow-runs.test.ts pins the Wait panel's reading of a blocker whose flow_run is queued: standing queued, with its flow. The SPA's code does not change, so it passes on master too.

The walk's cost

A probe database (4016 tasks, 2000 done by status; 12,018 relations: blocks i→(7i mod 4000)+1, follows i→(13i mod 4000)+1, parent i→i/10+1, plus a blocks chain of 8 and a follows chain of 8 crossing it; 3000 ended runs; a queue at 2 slots with one blocked and one running run and 50 queued runs that wait on nothing; a running and a needs_you run at Run steps and a blocked one at a Wait on another base), timed through store.unblocks and store.advance_queues:

	64f99701	head
one walk, reach 1	2.79 ms	0.10 ms
one walk, reach 9	2.81 ms	0.12 ms
a pass that begins nothing	148 ms	7.9 ms
the same pass at 1 slot, the queue full	6.5 ms	0.86 ms

At 64f99701 the plan reads relation in full in each recursive step (a UNION ALL subquery of every edge); at the head each arm SEARCHes sqlite_autoindex_relation_1 (task_id=?) or relation_other (other_task_id=?). The old and new walks give the same set for each of the 4016 tasks, reaches up to 16.

Golden set

None. The diff under web/e2e touches no snapshot.

Gates at the head

• sak-gate fast passed while iterating.
• sak-gate full (run 2) passed at ea298b03: the sx suite (73 tests), the release check, the deploy contract, vitest (474), Playwright (799) and the runner's Playwright config (10).

Tracker: [#3998](https://sak.swipelab.com/tasks/3998) Step 1 of plan [#3997](https://sak.swipelab.com/tasks/3997). ## What it delivers - `begin_flow(db, id, lent_by, queued)` has one caller, `advance_queues`. `take_out` and `dispatch_step` are gone, and so is `flow_run.from_queue`: the column, `FlowRun.from_queue`, the API member and the web type. - A Wait dispatches each undispatched blocker into its queue through `enqueue_flow` and begins none; a blocker already queued stays where it is, and `reevaluate_wait` begins none either. Run steps reads the step's latest flow run, begun or not: with none it queues one and holds, it holds while that run is `queued`, `running` or `blocked`, and it concludes `a step needs you` while the run waits on Agra or once it was stopped, begun or not. A blocker's or a step's run waits in its queue on what its task waits on, as every queued run does. - The kernel, in `src/store.sx`: one SQL reading of done (`done_sql`, which `task_done` reads and `done_by` goes through); one recursive walk, `unblocks`, over `blocks`, `follows` and `parent` (parent only while the parent's live flow run is at Run steps), each kind of edge a recursive SELECT of its own that joins `relation` from the reached task by its primary key or `relation_other`; going (begun and not ended, `QUEUE_GOING_SQL`); paused; and lent slots, a largest matching of a queue's going runs to the paused runs they unblock (`queue_lending`, `augment`). - `advance_queues` takes each queue in turn (`begin_next`): a queued run whose task waits on nothing begins on a lent slot when an augmenting path from it reaches a paused run lending to no going run, while the going runs that are not paused are fewer than `slots`; with no such path, on a slot of its own while the going runs left unmatched are fewer. 0 slots begins nothing, and a queue whose going runs that are not paused fill its slots begins nothing before any walk (`queue_going`). `dequeued` records `lent_by`: the paused run of its queue the run began on, else null. - Every going run is in its queue's `going` and answers `behind`; `done_today` holds every run of the queue that began and ended today; `GET /api/queues` and `GET /api/projects/:id/queues` list a plan's branch while a step's run is queued or going on it. - The Wait's `blockers` give each blocker its live flow run, else its latest started one (`BlockingTask.flow_run`), so a blocker waiting its turn answers its `queued` run. The comments of `BlockingTask`, `blockers_json`, `web/src/lib/types.ts`, `web/src/lib/flow-runs.ts` and `web/src/lib/FlowCard.svelte` say so. - MIGRATION_59 rewrites every stored `dequeued` event to `{"lent_by": …}` (the flow run its `flow_run_id` named when that one is of its run's project and base, else null) and drops `flow_run.from_queue`. `tests/support/schema.sx` gains `BACK_TO_58`, and `back_to_57()` and `back_to_55()` go through it. - docs/api.md: Flow runs, the history table, Plans, The Wait and Queues. ## Regressions (fail on master, pass at the head) The tests are in `tests/flow_queue.sx` and `tests/blockers.sx`; `began` in `tests/flow_queue.sx` reads `lent_by`. Built against master's `src/`, `tests/flow_queue.sx` ends `FLOW QUEUE FAIL (38 checks)` and `tests/blockers.sx` `BLOCKERS FAIL (9 checks)`. Every `began` check fails on master because master writes no `lent_by`, so the behaviour master shows for each regression was read with a probe that drives the same scenarios through the API, built against master's `src/` and against the head: | regression | master | head | | --- | --- | --- | | A Wait's blocker in its holder's queue at 1 slot | `going [1]`, blocker `behind null`, `dequeued {"from_queue":false,"flow_run_id":1}` | `going [1,2]`, holder and blocker `behind 0`, `dequeued {"lent_by":1}` | | Five blockers in the holder's queue at 1 slot | the five runs `running` at once | one `running`, four `queued`; each begins on the lent slot as the one before ends `done`, and the holder resumes after the fifth | | The holder's `blockers` | each answers `running`, begun outside the slots | the four waiting answer `queued`, `undispatched` false, the fifth its queued run and not its stopped one | | Bounded by the going runs that are not paused (P1, P2 at 1 slot) | B1 `running` as P1's Wait is reached, B2 `running` as P2's is; `going [10,11]` | B1 `queued` while P2 runs, then `running` on P1's lent slot once P2 is held, B2 `queued` until P1 ends | | A blocker in another project's queue | `running` while that queue's slot is held, `dequeued {"from_queue":false,"flow_run_id":15}`, absent from `going [14]` | `queued` until the slot frees, then `dequeued {"lent_by":null}` | | Run steps, a step's run whose task is blocked | the step `running` at once, absent from `going [17]` | the step `queued`, its `waits_on` naming the blocking task; stopped, the plan asks Agra; dispatched again, the plan holds; the task done, the step begins on the plan's lent slot | | Run steps, plan with a branch | `GET /api/projects/:id/queues` lists `master` only | it lists `task/19-the-plan` with the step going | | A database at 58 | reopened at 58, `from_queue` kept, each `dequeued` still `{"from_queue":…,"flow_run_id":…}` | 59, `from_queue` gone, the run begun outside the slots going, `pragma_foreign_key_check` empty, `{"lent_by":1}` for the Wait's blocker of its own queue and `{"lent_by":null}` for the blocker in another project, the step on its branch and a run its queue began | | No flow run answers `from_queue` | `a going run answers … no from_queue` FAILs | passes | The other cases (the lent slot lent on by a blocker held at its own Wait, lending only to runs that unblock the paused run, the largest matching, 0 slots, `done_today`, a `blocks` relation made to a held run's task, a blocker whose task is itself blocked) fail on master as listed in the runs above and pass at the head. `waitOf` in `web/src/lib/flow-runs.test.ts` pins the Wait panel's reading of a blocker whose `flow_run` is `queued`: standing `queued`, with its flow. The SPA's code does not change, so it passes on master too. ## The walk's cost A probe database (4016 tasks, 2000 done by status; 12,018 relations: `blocks` i→(7i mod 4000)+1, `follows` i→(13i mod 4000)+1, `parent` i→i/10+1, plus a `blocks` chain of 8 and a `follows` chain of 8 crossing it; 3000 ended runs; a queue at 2 slots with one `blocked` and one `running` run and 50 queued runs that wait on nothing; a `running` and a `needs_you` run at Run steps and a `blocked` one at a Wait on another base), timed through `store.unblocks` and `store.advance_queues`: | | `64f99701` | head | | --- | --- | --- | | one walk, reach 1 | 2.79 ms | 0.10 ms | | one walk, reach 9 | 2.81 ms | 0.12 ms | | a pass that begins nothing | 148 ms | 7.9 ms | | the same pass at 1 slot, the queue full | 6.5 ms | 0.86 ms | At `64f99701` the plan reads `relation` in full in each recursive step (a `UNION ALL` subquery of every edge); at the head each arm SEARCHes `sqlite_autoindex_relation_1 (task_id=?)` or `relation_other (other_task_id=?)`. The old and new walks give the same set for each of the 4016 tasks, reaches up to 16. ## Golden set None. The diff under `web/e2e` touches no snapshot. ## Gates at the head - `sak-gate fast` passed while iterating. - `sak-gate full` (run 2) passed at `ea298b03`: the sx suite (73 tests), the release check, the deploy contract, vitest (474), Playwright (799) and the runner's Playwright config (10).

agra added 3 commits 2026-10-31 18:18:28 +02:00

Every flow run goes through its queue, and a paused run lends its slot to the runs of its queue that unblock it ... 267207dfc7

begin_flow has one caller, advance_queues, and records in dequeued the
paused run whose lent slot its run began on. A Wait dispatches its
undispatched blockers into their queues and Run steps queues its step,
each beginning none. Migration 59 drops flow_run.from_queue and rewrites
every stored dequeued event to lent_by.

A blocker's flow run is its flow run that has not ended, else its latest started one, and a flow run answers no from_queue df6895d702

The API reference reads going, paused and lent slots in Queues, and a Wait's blockers and Run steps' steps waiting their turn there 64f99701cf

agra added 1 commit 2026-10-31 19:14:55 +02:00

The unblocks walk joins relation by index from each task it reaches, and a queue whose running runs fill its slots begins nothing before any walk ea298b030a

agra merged commit 90624baa12 into master 2026-10-31 19:40:01 +02:00

agra referenced this pull request from a commit 2026-10-31 19:40:02 +02:00

Merge pull request 'Every flow run goes through its queue, and a paused run lends its slot to the runs that unblock it' (#205) from task/3998-every-flow-run-goes-through-its-queue-an into master

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

bug

Something is not working

  

duplicate

This issue or pull request already exists

  

enhancement

New feature

  

help wanted

Need some help

  

invalid

Something is wrong

  

question

More information is needed

  

wontfix

This won't be fixed

No labels

bug

duplicate

enhancement

help wanted

invalid

question

wontfix

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

sx/sak!205

No description provided.