A gate job's output, a run's stderr and a gate's stream pieces are cut on a character boundary, and sak stores only well-formed UTF-8 #143

Merged

agra merged 2 commits from task/3799-a-gate-job-cuts-its-output-tail-on-a-byt into master

2026-10-25 08:55:47 +02:00

agra commented

2026-10-25 01:27:53 +03:00

Owner

Tracker: #3799

A failed gate's output longer than 64 KiB was kept from a byte offset, so it could start inside a multi-byte character. sak stored that byte in the job's output, copied it into the next run's step instructions, and served invalid UTF-8 from /api/jobs/<id>, /api/runs/<id> and every page of /api/runs holding that run (job 131 and run 689 on the live tracker).

What this delivers

The runner cuts on characters.
- A gate job's kept tail starts at the first byte at or after len - OUTPUT_KEEP that is not a UTF-8 continuation byte (util.char_from, runner/job.sx).
- The two other byte-budget cuts of agent or gate text end before a character they would split (util.whole_len): the first 4 KiB of a harness's stderr on the stream's close (runner/run.sx), and a gate's stream piece once 16 KiB are pending (runner/gate.sx Feed.put). Before, the piece could end on a character's first byte and the next piece start on its second.
sak takes only well-formed UTF-8. object_body (src/api.sx) replaces each maximal ill-formed UTF-8 subsequence of a JSON body with U+FFFD before parsing it, so every string of every JSON body is well-formed whichever runner sent it. The JSON parser already refuses lone surrogates in \u escapes, so raw bytes were the only way in.
Migration 45 repairs what is stored. It rewrites every text value of every TEXT column that is not well-formed UTF-8 the same way, inside one write transaction (apply_repair, src/db.sx). That covers run 689's step instructions and job 131's output, and any stream piece that was split.
The UTF-8 sequence rules move from src/logging.sx to src/util.sx, which the runner also imports, as lead_of, agreeing and utf8_sequence_len, beside well_formed, char_from and whole_len.
docs/api.md and runner/README.md say what is cut where.

Migration 45 is also the number #3724's branch takes. Whichever of the two merges second renumbers its migration to 46.

Regressions, fail-before and pass-after

Each arm is a copy of the branch's files at b376cf1 with one change reverted, run on m5 with the gate cache's sx (300086027f).

tests/runner_core.sx, three new checks in check_gate_job_tail:

A gate job whose failing gate prints its command line, then 22000 ✓ (66000 bytes). The last 65536 bytes start on a ✓'s third byte (0x93, the task's case).
A job a runner ends through POST /api/jobs/:id/complete with output starting with a lone 0x93.
A gate that prints 16383 bytes, then a ✓'s first byte, then its other two bytes half a second later.

Arm	tail cut inside a character starts at the next whole one	output starting with a lone continuation byte is kept with U+FFFD	stream piece at FLUSH_BYTES ends before the character it would split	file
head `b376cf1`	ok	ok	ok	380 ok, 1 FAIL (see below)
all three reverted (master's behaviour)	FAIL	FAIL	FAIL	378 ok, 3 FAIL
runner/job.sx byte cut restored	FAIL	ok	ok	380 ok, 1 FAIL
`object_body` repair removed	ok	FAIL	ok	380 ok, 1 FAIL
runner/gate.sx `flush(true)` restored at FLUSH_BYTES	ok	ok	FAIL	380 ok, 1 FAIL

The head's other FAIL in that by-hand round was "a sak-gate killed while its gate runs has printed only the gate's start" (check_gate_reattach). It is the timing check #3629 covers, and it failed only because five runner_core runs were going side by side. The four arms passed it in the same window, and the full gate's runner_core passed every check at this head.

tests/utf8_text.sx (new) covers four things:

well_formed: a lone continuation byte, a sequence cut short and an overlong pair each become U+FFFD, one per maximal subpart.
char_from and whole_len.
Migration 45 on a db set back to version 44 that holds ill-formed text.

The helpers do not exist at master. With apply_repair(db, 45) taken out of migrate, 3 checks fail: "the db opens at the schema head", "migration 45 reaches the schema head" and "it rewrites each ill-formed sequence it finds as U+FFFD".

The stderr cut is covered only through whole_len's checks.

Migration 45 took 3.6 s on m5, under load, over a 692 MB db of 500,000 stream events of about 1 KB each, and it repaired the one ill-formed event seeded among them.

The schema head moves to 45, so every test that asserts it now says 45.

No golden is added or changed: nothing under web/ changes.

Gates at the head

full (run #2, tracker gate run 387) at b376cf1: passed. Every step passed: the builds, npm run build (vitest 401/401), the libc grep, every tests/*.sx (runner_core and utf8_text among them, every check), release with step9, the deploy contract, the main Playwright config (740 passed, 1 skipped) and the runner config (10 passed).
fast (run #1, tracker gate run 388) at b376cf1: passed.

Tracker: [#3799](https://sak.swipelab.com/tasks/3799) A failed gate's output longer than 64 KiB was kept from a byte offset, so it could start inside a multi-byte character. sak stored that byte in the job's `output`, copied it into the next run's step instructions, and served invalid UTF-8 from `/api/jobs/<id>`, `/api/runs/<id>` and every page of `/api/runs` holding that run (job 131 and run 689 on the live tracker). ## What this delivers - **The runner cuts on characters.** - A gate job's kept tail starts at the first byte at or after `len - OUTPUT_KEEP` that is not a UTF-8 continuation byte (`util.char_from`, runner/job.sx). - The two other byte-budget cuts of agent or gate text end before a character they would split (`util.whole_len`): the first 4 KiB of a harness's stderr on the stream's close (runner/run.sx), and a gate's stream piece once 16 KiB are pending (runner/gate.sx `Feed.put`). Before, the piece could end on a character's first byte and the next piece start on its second. - **sak takes only well-formed UTF-8.** `object_body` (src/api.sx) replaces each maximal ill-formed UTF-8 subsequence of a JSON body with U+FFFD before parsing it, so every string of every JSON body is well-formed whichever runner sent it. The JSON parser already refuses lone surrogates in `\u` escapes, so raw bytes were the only way in. - **Migration 45 repairs what is stored.** It rewrites every text value of every TEXT column that is not well-formed UTF-8 the same way, inside one write transaction (`apply_repair`, src/db.sx). That covers run 689's step instructions and job 131's output, and any stream piece that was split. - The UTF-8 sequence rules move from src/logging.sx to src/util.sx, which the runner also imports, as `lead_of`, `agreeing` and `utf8_sequence_len`, beside `well_formed`, `char_from` and `whole_len`. - docs/api.md and runner/README.md say what is cut where. Migration 45 is also the number #3724's branch takes. Whichever of the two merges second renumbers its migration to 46. ## Regressions, fail-before and pass-after Each arm is a copy of the branch's files at b376cf1 with one change reverted, run on m5 with the gate cache's sx (300086027f). **tests/runner_core.sx**, three new checks in `check_gate_job_tail`: - A gate job whose failing gate prints its command line, then 22000 `✓` (66000 bytes). The last 65536 bytes start on a `✓`'s third byte (`0x93`, the task's case). - A job a runner ends through `POST /api/jobs/:id/complete` with `output` starting with a lone `0x93`. - A gate that prints 16383 bytes, then a `✓`'s first byte, then its other two bytes half a second later. | Arm | tail cut inside a character starts at the next whole one | output starting with a lone continuation byte is kept with U+FFFD | stream piece at FLUSH_BYTES ends before the character it would split | file | | --- | --- | --- | --- | --- | | head b376cf1 | ok | ok | ok | 380 ok, 1 FAIL (see below) | | all three reverted (master's behaviour) | FAIL | FAIL | FAIL | 378 ok, 3 FAIL | | runner/job.sx byte cut restored | FAIL | ok | ok | 380 ok, 1 FAIL | | `object_body` repair removed | ok | FAIL | ok | 380 ok, 1 FAIL | | runner/gate.sx `flush(true)` restored at FLUSH_BYTES | ok | ok | FAIL | 380 ok, 1 FAIL | The head's other FAIL in that by-hand round was "a sak-gate killed while its gate runs has printed only the gate's start" (`check_gate_reattach`). It is the timing check #3629 covers, and it failed only because five runner_core runs were going side by side. The four arms passed it in the same window, and the `full` gate's runner_core passed every check at this head. **tests/utf8_text.sx** (new) covers four things: - `well_formed`: a lone continuation byte, a sequence cut short and an overlong pair each become U+FFFD, one per maximal subpart. - `char_from` and `whole_len`. - Migration 45 on a db set back to version 44 that holds ill-formed text. The helpers do not exist at master. With `apply_repair(db, 45)` taken out of `migrate`, 3 checks fail: "the db opens at the schema head", "migration 45 reaches the schema head" and "it rewrites each ill-formed sequence it finds as U+FFFD". The stderr cut is covered only through `whole_len`'s checks. Migration 45 took 3.6 s on m5, under load, over a 692 MB db of 500,000 stream events of about 1 KB each, and it repaired the one ill-formed event seeded among them. The schema head moves to 45, so every test that asserts it now says 45. No golden is added or changed: nothing under `web/` changes. ## Gates at the head - `full` (run #2, tracker gate run 387) at b376cf1: **passed**. Every step passed: the builds, `npm run build` (vitest 401/401), the libc grep, every `tests/*.sx` (runner_core and utf8_text among them, every check), release with step9, the deploy contract, the main Playwright config (740 passed, 1 skipped) and the runner config (10 passed). - `fast` (run #1, tracker gate run 388) at b376cf1: **passed**.

agra added 2 commits

2026-10-25 01:27:53 +03:00