sx/sak
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

A run lives through a sak restart: its tracker calls are tried again, each lands once, and an unreported end is logged #141

Merged
agra merged 14 commits from task/3724-a-run-lives-through-a-sak-restart-its-tr into task/3694-plan-a-flow-run-waits-on-agra-only-for-a 2026-10-25 04:13:16 +02:00
Conversation 0 Commits 14 Files changed 42 +1238 -251
agra commented 2026-10-24 22:11:46 +03:00
Owner
Copy link

Tracker: #3724

Step 4 of plan #3694. This branch merged origin/master (the 0.3.6 release and #3731's fix of the two web tests that time out under load, 8cda22f) before its own commits, so the PR's diff against the plan's branch also shows those master commits. The merge was clean: master's #3731 change was already on the plan's branch, carried byte-identical as 952a11e.

Why runs 249 and 289 were lost

Both were lost to a sak restart, not to the runner's hand-over. Run 289's stream 7470 stops at 07:18:24Z on 2026-10-23, when sak restarted. On s32, runner.log lines 1546–1561 show 19:43:53 run 249 started, then "sak-runner 0.3.3: sak asks for 0.3.4" and the hand-over to slot b, and no finished line for run 249 ever. A deploy restarts sak (scripts/deploy_nas.sh:45 runs docker compose up -d sak) and Update all follows it (comment 17696), so run 249 died in the same restart, 29 s after it started.

The hand-over did not drop it: Channel.finishing stops only once runs is empty, a ticket settles only in retire after its pump thread ends, and check_hand_over already pins a step in flight ending done across a hand-over. What lost it was the runner's single attempt at each tracker call: the pump killed the run tracker_fault on the first event batch sak did not take, execute then raised on close_stream and complete, which sak could not take either, and in daemon mode nothing logged it. Its request stayed running until it was closed by hand.

What this delivers

sak takes each call once. A call the runner sends again after its answer was lost is answered with what the first one made:

  • POST /api/runs/:id/claim and POST /api/jobs/:id/claim by the runner that holds the request or job claimed answer 200 with that claim again. A run's is built from the account and harness the request recorded when it was claimed, not by resolving the account again.
  • POST /api/tasks/:id/streams takes an optional run_id or job_id, which the stream records. An open naming a run or job whose stream still runs answers 200 with that stream. A request queued again (account_limited) opens a stream of its own, since its first stream is closed.
  • POST /api/streams/:id/events and POST /api/streams/:id/messages take an optional key. A key the stream or inbox already took writes nothing and answers 200 with that batch's last_seq or that message.
  • POST /api/runs/:id/gate-runs takes number (required), the gate run's number in its run's claim, and records the stream the run started on with it. A number the run already holds on that stream answers 200 with that gate run. A request queued again (account_limited) starts its next claim on a stream of its own, and that claim's gate runs, numbered from 1 again, are recorded as new ones. A record is taken only while the run is running: a request that is only claimed has no stream yet, and answers 409 run_not_held, where the base took it.
  • POST /api/memories naming a run whose text that run already saved answers 200 with that memory.
  • Migration 45 adds the stream's run and job, the batch and message keys, and the gate run's number and stream. It numbers the stored gate runs within each run in id order and gives them no stream; no runner sends one of them again, since a runner older than this PR sends no number.

The runner tries its calls again.

  • --tracker-bound <seconds> (default 300, also a config file key) bounds how long a run's or a job's call is tried again, with a backoff from 250 ms doubling to 4 s, while sak is unreachable or answers 5xx. That covers the claim, the stream open, start, the gate feeds' batches and records, the forbidden-word message, renew, remember, report_usage, the close and complete, and a job's claim, open, start and completion. A 5xx past the bound now raises Unavailable rather than Refused.
  • The claim now runs on the run's pump thread, so a claim being tried again holds that run alone, not the main thread. A run or job whose pump thread cannot start is therefore not claimed and stays queued. Before, it was claimed and completed failed · pump_unstarted.
  • The pump tries its batches and acks once a tick. What sak does not take is held, in order, while the pump keeps reading the harness. The run is killed tracker_fault once the bound has passed since the last batch or ack sak took, or at once when sak refuses one.
  • Every batch carries a key: pump:<n> for the pump's, gate:<number>:<n> for a gate run's feed, and forbidden:<n> for the forbidden-word messages. The open names the run or job.
  • A start, close or complete (and a job's start and completion) tried more than once that sak answers 409 is read back. The call is taken when sak shows its effect.
  • A failed renew, remember or report_usage keeps the home as before and is logged, but no longer stops the close and complete.
  • A run or job whose execute raises logs run <id> unreported <call>: <error> (or job <id>), the call named as the tracker client's method. A claim whose answer never came logs run <id> unreported claim: <error>.

docs/api.md (Streams, Runs, Jobs, Memories) and runner/README.md (the flag table, One run, the retry and unreported paragraphs) describe this.

Rollout. number is required on a gate-run record. Once sak runs this, a runner older than this PR has its gate-run records refused 400 (logged unrecorded) until Update all moves it. Its runs and gates still run and close as before.

Answers to review run 680

  • F1, a gate-run record keyed on (run, number) dropped a requeued run's next gates (76c3b9f). The runner numbers a run's gate runs from 1 on each claim, and a request that ends failed · account_limited is queued again under the same id, so the next claim's gate #1 was answered with the first claim's record and never recorded. A gate run now carries the stream its run started on, and the repeat and the unique index are on (run, stream, number). Since a claimed request has no stream until it starts, a record is taken only while the run is running. docs/api.md says so. The other keys the runner counts per execution, pump:<n>, gate:<number>:<n> and forbidden:<n>, are each known within their stream, and a new claim opens its own, so they need no change. A memory stays known by its run and text across claims. One case is left open, read from the code and not probed: a new claim's open names its run, and is answered with the first claim's stream if that stream is still running. That happens only when the first claim's close failed past its bound while its complete (account_limited) landed.
  • F2, the pump's bound counted from the first failure while later calls landed (ff11275). Each batch or ack that lands now clears the pump's record of when sak stopped taking its calls, and resets the backoff, so the bound counts from the last call sak took. runner/README.md says so.

Regressions, failing at the step's base and passing at its head

Each check below was run from a git archive arm of the step's base, the plan's branch at c139ad1, holding this PR's test files. The arm used the gate cache's sx and, for tests/runner_daemon.sx, a dist/sak-runner built from the base's runner. The base's tracker client has no key argument and no bound, so the base copy of runner_daemon.sx drops the trailing null of post_message and the lines that set tracker_bound_ms. The head passed every one of these files, in arms and in tracker gate runs 371 (a379400) and 394 (ff11275): RUNNER DAEMON OK, RUNNER CORE OK, STREAMS API OK, STREAM INBOX OK, MEMORIES OK, GATES OK, RUN QUEUE OK, BLOCKERS OK.

tests/runner_daemon.sx (7 checks fail at the base, and the two of check_flaky at a379400):

  • a stand-in drops the answer to every second batch of a run's stream for twice a 4 s --tracker-bound, sak taking each batch sent again (check_flaky, added for F2). At ff11275 the stand-in dropped 25 answers, the run ended done, and its stream holds all 42 lines once, in order; RUNNER DAEMON OK. At a379400, from a git archive arm holding this test: the run was killed tracker_fault 4 s after it started, 5 answers dropped, and FAIL the run ends done, FAIL every line the harness wrote is on its stream once, in order (7 lines), RUNNER DAEMON FAIL (2 checks). Every other check passed there.
  • sak stopped for 10 s while a stub run writes events, then served again on the same port (check_outage, the stub stream-json harness's new chatter mode). At the head the run ends done with every line it wrote on the stream once, in order (46 and 52 lines in two head runs). At the base: FAIL the run ends done, FAIL every line the harness wrote is on its stream once, in order (1 lines), FAIL the stream closes done. The base's log for that run reads run 5 claimed, run 5 started stream 5, and then nothing: its request stays running and no line logs its end.
  • sak stopped past a 4 s --tracker-bound and served again within the end calls' bound (check_bound). At the head the run is killed, its request reads failed · tracker_fault, and its stream closes killed. At the base: FAIL the run is killed, and its request reads failed · tracker_fault, FAIL its stream closes killed with tracker_fault.
  • a stand-in drops the answer to the claim, the stream open, an event batch, the gate-run record and complete once each, then to a remember (check_stand_in). The test's dispatch wrapper lets each named call reach sak, then answers it with a body that faults before its first byte, so the server closes the connection with no answer. At the head all 5 are dropped and sak holds the run's end as the runner recorded it (done, claimed by alpha), one stream, each event once, one gate-run record and each of the 3 memories once, and the runner's log holds no unreported line. At the base: FAIL alpha runs the request, its claim's and its stream open's answers dropped. Its log reads run 7 received and never claimed: the lost answer left the request claimed by alpha, which never started it.
  • sak gone for good while a run writes events (check_gone). At the head the runner logs run 3 unreported close_stream: TrackerErr.Transport once the end calls' bound passes. At the base: FAIL once the end calls' bound passes, the runner logs the close sak never took.

tests/step_streams_api.sx:

  • a second open naming a run whose stream still runs -> 200 with that stream, and opens none and a batch whose key the stream already took -> 200 with that batch's last_seq, and writes nothing fail at the base.
  • an open naming a run whose stream has closed, as a request queued again does -> 201 with a stream of its own passes at the base. It fails at e69779a, this branch's first server commit, whose open matched a closed stream too. The first head daemon run found that: a request queued again after account_limited was handed its first, closed stream, and every batch was refused 409 stream_closed. 548852c fixes it.

tests/step_stream_inbox.sx: a second post with a key the inbox already took -> 200 with the first message, and stores none fails at the base.

tests/memories.sx: a second memory of a run's text that run already saved -> 200 with that memory, and saves none fails at the base. Two later checks fail with it, because the base saves a second memory.

tests/gates.sx: a record of a number the run already holds -> 200 with that gate run, and records none fails at the base, along with the record's number, the 400 for no number from 1, and the list of the run's gate runs.

tests/gates.sx check_requeued (added for F1): a run claimed, started, its gate #1 recorded, then ended failed · account_limited, claimed again and started on a new stream. At ff11275 all pass (GATES OK). At a379400, from an arm holding this test, GATES FAIL (3 checks):

  • FAIL a run claimed and not yet started takes no gate run -> 409 run_not_held (a379400 takes a record from a claimed request, and here answered 200 with the first claim's);
  • FAIL its next claim's gate of the same number -> 201 with a gate run of its own (a379400 answered 200 with the first claim's record);
  • FAIL the run lists the gate runs of both claims (it listed one).

tests/step_run_queue.sx: a claim by the runner that holds the request claimed -> 200 with its claim again, on the account it recorded and not the one first by position now fails at the base. Before the repeat, the test moves a second serving account to position 1.

tests/blockers.sx: a claim by the runner that holds the job claimed -> 200 with its claim again fails at the base.

At the base, the schema-version checks of these files fail too (migration reaches the schema head, 44 against 45). Every test that pins the schema version moves to 45.

tests/runner_core.sx check_renewal_unstored now pins the new end rule: a renewal sak does not store keeps the renewed login in the run's home, and the run still closes and ends. Before, it pinned the run failing with Refused.

Gates

  • sak-gate full #1, tracker gate run 369, at 61ae1c4: failed in its tests/*.sx step. Every runner and API test passed, including runner_daemon, runner_core, runner_handover, runner_view, runner_launchd, blockers, gates, memories and step_run_queue. Three tests failed on migration 45 itself:

    • tests/file_keys.sx "a database at 17 migrates" and tests/tools.sx "it upgrades". Each winds a database back (to 17 and to 40) by dropping what the later migrations added. Neither dropped migration 45's stream columns, so ALTER TABLE stream ADD COLUMN run_request_id failed on the way up again.
    • tests/step_stream_links.sx "the upgrade keeps an index over a task's attempts" counted every index on stream, and migration 45 adds two.

    a379400 makes the wind-backs drop migration 45's additions and counts the attempts index by its name. All three pass at a379400.

  • sak-gate full #2, tracker gate run 371, at a379400 (clean tree): passed. Every tests/*.sx file passed, then release with step9_release, the deploy contract, the main Playwright config (740 passed, 1 skipped, 3.8 min) and the runner Playwright config (10 passed).

  • sak-gate full, tracker gate run 394, at ff11275 (clean tree), after the answers to review run 680: passed. Every tests/*.sx file passed, then release with step9_release, the deploy contract, the main Playwright config (740 passed, 1 skipped, 3.5 min) and the runner Playwright config (10 passed).

Golden set: none.

Tracker: [#3724](https://sak.swipelab.com/tasks/3724) Step 4 of plan #3694. This branch merged `origin/master` (the 0.3.6 release and #3731's fix of the two web tests that time out under load, 8cda22f) before its own commits, so the PR's diff against the plan's branch also shows those master commits. The merge was clean: master's #3731 change was already on the plan's branch, carried byte-identical as 952a11e. ## Why runs 249 and 289 were lost Both were lost to a sak restart, not to the runner's hand-over. Run 289's stream 7470 stops at 07:18:24Z on 2026-10-23, when sak restarted. On s32, `runner.log` lines 1546–1561 show `19:43:53 run 249 started`, then "sak-runner 0.3.3: sak asks for 0.3.4" and the hand-over to slot b, and no `finished` line for run 249 ever. A deploy restarts sak (`scripts/deploy_nas.sh:45` runs `docker compose up -d sak`) and Update all follows it (comment 17696), so run 249 died in the same restart, 29 s after it started. The hand-over did not drop it: `Channel.finishing` stops only once `runs` is empty, a ticket settles only in `retire` after its pump thread ends, and `check_hand_over` already pins a step in flight ending `done` across a hand-over. What lost it was the runner's single attempt at each tracker call: the pump killed the run `tracker_fault` on the first event batch sak did not take, `execute` then raised on `close_stream` and `complete`, which sak could not take either, and in daemon mode nothing logged it. Its request stayed `running` until it was closed by hand. ## What this delivers **sak takes each call once.** A call the runner sends again after its answer was lost is answered with what the first one made: - `POST /api/runs/:id/claim` and `POST /api/jobs/:id/claim` by the runner that holds the request or job `claimed` answer `200` with that claim again. A run's is built from the account and harness the request recorded when it was claimed, not by resolving the account again. - `POST /api/tasks/:id/streams` takes an optional `run_id` or `job_id`, which the stream records. An open naming a run or job whose stream still runs answers `200` with that stream. A request queued again (`account_limited`) opens a stream of its own, since its first stream is closed. - `POST /api/streams/:id/events` and `POST /api/streams/:id/messages` take an optional `key`. A key the stream or inbox already took writes nothing and answers `200` with that batch's `last_seq` or that message. - `POST /api/runs/:id/gate-runs` takes `number` (required), the gate run's number in its run's claim, and records the stream the run started on with it. A number the run already holds on that stream answers `200` with that gate run. A request queued again (`account_limited`) starts its next claim on a stream of its own, and that claim's gate runs, numbered from 1 again, are recorded as new ones. A record is taken only while the run is `running`: a request that is only `claimed` has no stream yet, and answers `409 run_not_held`, where the base took it. - `POST /api/memories` naming a run whose text that run already saved answers `200` with that memory. - Migration 45 adds the stream's run and job, the batch and message keys, and the gate run's number and stream. It numbers the stored gate runs within each run in id order and gives them no stream; no runner sends one of them again, since a runner older than this PR sends no `number`. **The runner tries its calls again.** - `--tracker-bound <seconds>` (default 300, also a config file key) bounds how long a run's or a job's call is tried again, with a backoff from 250 ms doubling to 4 s, while sak is unreachable or answers 5xx. That covers the claim, the stream open, start, the gate feeds' batches and records, the forbidden-word message, `renew`, `remember`, `report_usage`, the close and `complete`, and a job's claim, open, start and completion. A 5xx past the bound now raises `Unavailable` rather than `Refused`. - The claim now runs on the run's pump thread, so a claim being tried again holds that run alone, not the main thread. A run or job whose pump thread cannot start is therefore not claimed and stays queued. Before, it was claimed and completed `failed · pump_unstarted`. - The pump tries its batches and acks once a tick. What sak does not take is held, in order, while the pump keeps reading the harness. The run is killed `tracker_fault` once the bound has passed since the last batch or ack sak took, or at once when sak refuses one. - Every batch carries a key: `pump:<n>` for the pump's, `gate:<number>:<n>` for a gate run's feed, and `forbidden:<n>` for the forbidden-word messages. The open names the run or job. - A start, close or `complete` (and a job's start and completion) tried more than once that sak answers `409` is read back. The call is taken when sak shows its effect. - A failed `renew`, `remember` or `report_usage` keeps the home as before and is logged, but no longer stops the close and `complete`. - A run or job whose `execute` raises logs `run <id> unreported <call>: <error>` (or `job <id>`), the call named as the tracker client's method. A claim whose answer never came logs `run <id> unreported claim: <error>`. `docs/api.md` (Streams, Runs, Jobs, Memories) and `runner/README.md` (the flag table, One run, the retry and unreported paragraphs) describe this. **Rollout.** `number` is required on a gate-run record. Once sak runs this, a runner older than this PR has its gate-run records refused `400` (logged `unrecorded`) until Update all moves it. Its runs and gates still run and close as before. ## Answers to review run 680 - **F1, a gate-run record keyed on (run, number) dropped a requeued run's next gates** (76c3b9f). The runner numbers a run's gate runs from 1 on each claim, and a request that ends `failed · account_limited` is queued again under the same id, so the next claim's gate #1 was answered with the first claim's record and never recorded. A gate run now carries the stream its run started on, and the repeat and the unique index are on (run, stream, number). Since a claimed request has no stream until it starts, a record is taken only while the run is `running`. `docs/api.md` says so. The other keys the runner counts per execution, `pump:<n>`, `gate:<number>:<n>` and `forbidden:<n>`, are each known within their stream, and a new claim opens its own, so they need no change. A memory stays known by its run and text across claims. One case is left open, read from the code and not probed: a new claim's open names its run, and is answered with the first claim's stream if that stream is still `running`. That happens only when the first claim's close failed past its bound while its `complete` (`account_limited`) landed. - **F2, the pump's bound counted from the first failure while later calls landed** (ff11275). Each batch or ack that lands now clears the pump's record of when sak stopped taking its calls, and resets the backoff, so the bound counts from the last call sak took. `runner/README.md` says so. ## Regressions, failing at the step's base and passing at its head Each check below was run from a `git archive` arm of the step's base, the plan's branch at c139ad1, holding this PR's test files. The arm used the gate cache's sx and, for `tests/runner_daemon.sx`, a `dist/sak-runner` built from the base's runner. The base's tracker client has no key argument and no bound, so the base copy of `runner_daemon.sx` drops the trailing `null` of `post_message` and the lines that set `tracker_bound_ms`. The head passed every one of these files, in arms and in tracker gate runs 371 (a379400) and 394 (ff11275): `RUNNER DAEMON OK`, `RUNNER CORE OK`, `STREAMS API OK`, `STREAM INBOX OK`, `MEMORIES OK`, `GATES OK`, `RUN QUEUE OK`, `BLOCKERS OK`. `tests/runner_daemon.sx` (7 checks fail at the base, and the two of `check_flaky` at a379400): - **a stand-in drops the answer to every second batch of a run's stream for twice a 4 s `--tracker-bound`, sak taking each batch sent again** (`check_flaky`, added for F2). At ff11275 the stand-in dropped 25 answers, the run ended `done`, and its stream holds all 42 lines once, in order; `RUNNER DAEMON OK`. At a379400, from a `git archive` arm holding this test: the run was killed `tracker_fault` 4 s after it started, 5 answers dropped, and `FAIL the run ends done`, `FAIL every line the harness wrote is on its stream once, in order (7 lines)`, `RUNNER DAEMON FAIL (2 checks)`. Every other check passed there. - **sak stopped for 10 s while a stub run writes events, then served again on the same port** (`check_outage`, the stub stream-json harness's new `chatter` mode). At the head the run ends `done` with every line it wrote on the stream once, in order (46 and 52 lines in two head runs). At the base: `FAIL the run ends done`, `FAIL every line the harness wrote is on its stream once, in order (1 lines)`, `FAIL the stream closes done`. The base's log for that run reads `run 5 claimed`, `run 5 started stream 5`, and then nothing: its request stays `running` and no line logs its end. - **sak stopped past a 4 s `--tracker-bound` and served again within the end calls' bound** (`check_bound`). At the head the run is killed, its request reads `failed · tracker_fault`, and its stream closes `killed`. At the base: `FAIL the run is killed, and its request reads failed · tracker_fault`, `FAIL its stream closes killed with tracker_fault`. - **a stand-in drops the answer to the claim, the stream open, an event batch, the gate-run record and `complete` once each, then to a `remember`** (`check_stand_in`). The test's dispatch wrapper lets each named call reach sak, then answers it with a body that faults before its first byte, so the server closes the connection with no answer. At the head all 5 are dropped and sak holds the run's end as the runner recorded it (`done`, claimed by alpha), one stream, each event once, one gate-run record and each of the 3 memories once, and the runner's log holds no `unreported` line. At the base: `FAIL alpha runs the request, its claim's and its stream open's answers dropped`. Its log reads `run 7 received` and never `claimed`: the lost answer left the request `claimed` by alpha, which never started it. - **sak gone for good while a run writes events** (`check_gone`). At the head the runner logs `run 3 unreported close_stream: TrackerErr.Transport` once the end calls' bound passes. At the base: `FAIL once the end calls' bound passes, the runner logs the close sak never took`. `tests/step_streams_api.sx`: - `a second open naming a run whose stream still runs -> 200 with that stream, and opens none` and `a batch whose key the stream already took -> 200 with that batch's last_seq, and writes nothing` fail at the base. - `an open naming a run whose stream has closed, as a request queued again does -> 201 with a stream of its own` passes at the base. It fails at e69779a, this branch's first server commit, whose open matched a closed stream too. The first head daemon run found that: a request queued again after `account_limited` was handed its first, closed stream, and every batch was refused `409 stream_closed`. 548852c fixes it. `tests/step_stream_inbox.sx`: `a second post with a key the inbox already took -> 200 with the first message, and stores none` fails at the base. `tests/memories.sx`: `a second memory of a run's text that run already saved -> 200 with that memory, and saves none` fails at the base. Two later checks fail with it, because the base saves a second memory. `tests/gates.sx`: `a record of a number the run already holds -> 200 with that gate run, and records none` fails at the base, along with the record's `number`, the `400` for no number from 1, and the list of the run's gate runs. `tests/gates.sx` `check_requeued` (added for F1): a run claimed, started, its gate #1 recorded, then ended `failed · account_limited`, claimed again and started on a new stream. At ff11275 all pass (`GATES OK`). At a379400, from an arm holding this test, `GATES FAIL (3 checks)`: - `FAIL a run claimed and not yet started takes no gate run -> 409 run_not_held` (a379400 takes a record from a `claimed` request, and here answered `200` with the first claim's); - `FAIL its next claim's gate of the same number -> 201 with a gate run of its own` (a379400 answered `200` with the first claim's record); - `FAIL the run lists the gate runs of both claims` (it listed one). `tests/step_run_queue.sx`: `a claim by the runner that holds the request claimed -> 200 with its claim again, on the account it recorded and not the one first by position now` fails at the base. Before the repeat, the test moves a second serving account to position 1. `tests/blockers.sx`: `a claim by the runner that holds the job claimed -> 200 with its claim again` fails at the base. At the base, the schema-version checks of these files fail too (`migration reaches the schema head`, 44 against 45). Every test that pins the schema version moves to 45. `tests/runner_core.sx` `check_renewal_unstored` now pins the new end rule: a renewal sak does not store keeps the renewed login in the run's home, and the run still closes and ends. Before, it pinned the run failing with `Refused`. ## Gates - `sak-gate full` #1, tracker gate run 369, at 61ae1c4: **failed** in its `tests/*.sx` step. Every runner and API test passed, including `runner_daemon`, `runner_core`, `runner_handover`, `runner_view`, `runner_launchd`, `blockers`, `gates`, `memories` and `step_run_queue`. Three tests failed on migration 45 itself: - `tests/file_keys.sx` "a database at 17 migrates" and `tests/tools.sx` "it upgrades". Each winds a database back (to 17 and to 40) by dropping what the later migrations added. Neither dropped migration 45's `stream` columns, so `ALTER TABLE stream ADD COLUMN run_request_id` failed on the way up again. - `tests/step_stream_links.sx` "the upgrade keeps an index over a task's attempts" counted every index on `stream`, and migration 45 adds two. a379400 makes the wind-backs drop migration 45's additions and counts the attempts index by its name. All three pass at a379400. - `sak-gate full` #2, tracker gate run 371, at a379400 (clean tree): **passed**. Every `tests/*.sx` file passed, then release with `step9_release`, the deploy contract, the main Playwright config (740 passed, 1 skipped, 3.8 min) and the runner Playwright config (10 passed). - `sak-gate full`, tracker gate run 394, at ff11275 (clean tree), after the answers to review run 680: **passed**. Every `tests/*.sx` file passed, then release with `step9_release`, the deploy contract, the main Playwright config (740 passed, 1 skipped, 3.5 min) and the runner Playwright config (10 passed). Golden set: none.
VERSION is 0.3.6, so LATER takes the next release version and MISLABELLED moves past it. The values are the ones task #3779 names.
agra changed title from sak takes a repeated call of a run once: a claim, a stream open, a batch, a gate-run record, a memory and an inbox post answer what the first one made to A run lives through a sak restart: its tracker calls are tried again, each lands once, and an unreported end is logged 2026-10-24 23:36:32 +03:00 
A claim by the runner that holds a run or job claimed answers that claim
again, on the account the request recorded. A stream open may name its run
or job, a batch of events and an inbox post may carry a key, a gate-run
record carries its number in its run, and a memory names its run: a repeat
of any of them answers 200 with what the first made, and makes nothing.
Migration 45 adds the stream's run and job, the batch and message keys, and
the gate run's number, numbering the stored gate runs within each run in id
order.
A request that ends account_limited is queued again under the same id, and
its next claim opened a stream that the open answered with the first,
closed one. The open now matches the run's or job's running stream, and the
unique indexes of migration 45 cover running streams only.
A run's and a job's calls are tried again with a backoff, from 250 ms
doubling to 4 s, while sak is unreachable or answers 5xx, until the tracker
bound passes. The claim moves onto the run's pump thread. The pump tries its
batches and acks once a tick and holds what sak does not take, in order,
while it reads the harness; the run is killed tracker_fault once sak has
taken none of it for the bound. Every batch, the forbidden-word message and
the stream open carry what sak knows a repeat of them by, and a start, close
or end that sak answers 409 after a retry is read back. A failed renewal,
memory or usage report no longer stops the close and the end, and a run
whose end never reached sak logs it as unreported.
file_keys and tools take a database back to 17 and 40 by dropping what the
later migrations added; they now drop migration 45's stream columns, its
batch table and the message key too, so the migration applies again. The
stream-links upgrade counts the attempts index by name, since migration 45
adds two partial indexes on stream.
A runner numbers a run's gate runs from 1 on each claim, and a request that
ends failed for its account's limit is queued again under the same id. Keyed
on (run, number), the next claim's first gate was answered with the first
claim's record and never recorded. A gate run now carries the stream its run
started on, the repeat and the unique index are on (run, stream, number), and
a record is taken only while the run is running on that stream.
A batch or an ack that landed while a later one did not left the pump's
missing time at the first failure, so a run whose calls sak kept taking in
between was killed tracker_fault once the bound had passed since then. Each
batch or ack that lands now clears it and resets the backoff.
agra merged commit d860685c31 into task/3694-plan-a-flow-run-waits-on-agra-only-for-a 2026-10-25 04:13:16 +02:00
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
bug
Something is not working

  
duplicate
This issue or pull request already exists

  
enhancement
New feature

  
help wanted
Need some help

  
invalid
Something is wrong

  
question
More information is needed

  
wontfix
This won't be fixed

No labels
bug
duplicate
enhancement
help wanted
invalid
question
wontfix
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
sx/sak!141
No description provided.