From 5949a8843928f8d242d87c245f5b718a85182d12 Mon Sep 17 00:00:00 2001 From: agra Date: Sun, 21 Jun 2026 19:43:22 +0300 Subject: [PATCH] =?UTF-8?q?fibers:=20end-to-end=20M:1=20capstone=20(B1.5)?= =?UTF-8?q?=20=E2=80=94=20Stream=20B1=20complete?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 1817 composes the whole colorblind pure-sx async stack: the M:1 scheduler, suspending go/wait async, and deterministic virtual-time sleep/now_ms, over the naked swap_context on guarded mmap stacks. A coordinator launches three async tasks (sleep 30/10/20 -> return 100/20/3), awaits all three in spawn order, and sums them; tasks complete in DEADLINE order (task 2@10, 3@20, 1@30), sum 123, final virtual clock 30 -- fully deterministic. Stream B1 (fibers + Io + M:1 scheduler) is feature-complete: examples 1800-1817, suite 755/0. Checkpoint + plan marked COMPLETE; next carve is Stream B2 (channels / cancel / async stdlib). --- current/CHECKPOINT-FIBERS.md | 83 +++++++++++++------ current/PLAN-FIBERS.md | 20 +++-- .../1817-concurrency-fiber-m1-end-to-end.sx | 66 +++++++++++++++ ...1817-concurrency-fiber-m1-end-to-end.build | 1 + .../1817-concurrency-fiber-m1-end-to-end.exit | 1 + ...817-concurrency-fiber-m1-end-to-end.stderr | 1 + ...817-concurrency-fiber-m1-end-to-end.stdout | 7 ++ 7 files changed, 149 insertions(+), 30 deletions(-) create mode 100644 examples/concurrency/1817-concurrency-fiber-m1-end-to-end.sx create mode 100644 examples/concurrency/expected/1817-concurrency-fiber-m1-end-to-end.build create mode 100644 examples/concurrency/expected/1817-concurrency-fiber-m1-end-to-end.exit create mode 100644 examples/concurrency/expected/1817-concurrency-fiber-m1-end-to-end.stderr create mode 100644 examples/concurrency/expected/1817-concurrency-fiber-m1-end-to-end.stdout diff --git a/current/CHECKPOINT-FIBERS.md b/current/CHECKPOINT-FIBERS.md index a91acc45..0f6d7ebd 100644 --- a/current/CHECKPOINT-FIBERS.md +++ b/current/CHECKPOINT-FIBERS.md @@ -4,7 +4,28 @@ Companion to [PLAN-FIBERS.md](PLAN-FIBERS.md). Update after every step (one step per the cadence rule). New corpus category: `18xx` concurrency. ## Last completed step -**B1.4c — REAL fd-readiness blocking via kqueue (macOS).** `library/modules/std/sched.sx` now lets a +**B1.5 — END-TO-END M:1 validation — STREAM B1 COMPLETE.** A single capstone exercises the whole +colorblind pure-sx async runtime together: the M:1 scheduler (B1.5a) + suspending fiber-task async +`go`/`wait` (B1.4a) + deterministic virtual-time `sleep`/`now_ms` (B1.4b), over the `abi(.naked)` +`swap_context` on guarded `mmap` stacks (B1.0–B1.3). `examples/concurrency/1817-concurrency-fiber-m1-end-to-end.sx`: +a coordinator fiber launches three `go` tasks (sleep 30/10/20 → return 100/20/3), awaits all three in +SPAWN order, and sums them. The completion log is the deterministic contract — tasks finish in +DEADLINE order (`task 2@10, task 3@20, task 1@30`), not spawn/await order; `sum: 123`; final virtual +clock 30. Fully reproducible (virtual time, no real clock). Suite GREEN **755/0**. + +**Stream B1 is feature-complete.** The pure-sx async runtime exists end-to-end: fibers behind the +`abi(.naked)` context switch (proven on aarch64 + x86_64/Win64), the M:1 cooperative scheduler, +suspending `go`/`wait`/`cancel` async, deterministic virtual-time timers, and real fd-readiness via +kqueue — all in `library/modules/std/sched.sx`, all adversarially reviewed, locked by `18xx` +(1800–1817). Compiler floor delivered: `abi(.naked)` emission (B1.0) + per-fiber `context` (B1.1, +zero-change). Five compiler bugs fixed en route (0151/0152/0153 in B1.2; 0154 in B1.5a; +0156-Part1 + 0157 in B1.4a). Deferred (documented, non-blocking): issue 0150 (`Future(void)`/`timeout`), +0155 (scalar-pointer index), 0156-Part2 (deferred `..` spread); a linux `epoll` twin of `block_on_fd`; +routing the suspending async through the erased `context.io` (M:N evolution); the heap-Task / closure-env +/ kq-fd leaks (bounded, default-GPA-invisible). Stream B2 (channels/cancel/stdlib) is the next carve. + +### Earlier — B1.4c — REAL fd-readiness blocking via kqueue (macOS) +`library/modules/std/sched.sx` now lets a fiber park on a file descriptor and the run loop block on `kevent` until the kernel reports it ready. Reuses the existing verified `library/modules/std/net/kqueue.sx` bindings (`Kevent` (32 bytes), `kqueue`/`kevent`/`kq_apply`/`kq_wait` + the `EVFILT_READ`/`EV_ADD`/`EV_ENABLE`/`EV_ONESHOT` @@ -339,25 +360,25 @@ body); closed + locked. The review's `.naked`-lambda CRITICAL was a false positi (unparseable — `isLambda` breaks on the `abi` keyword). ## Current state -**B1.4c COMPLETE — real fd-readiness blocking via kqueue (macOS) exists.** `library/modules/std/sched.sx` -now carries: the M:1 scheduler core (B1.5a: `spawn`/`yield_now`/`suspend_self`/`wake`/`run`), the -suspending fiber-task async (B1.4a: `Task($R)`/`go`/`wait`/`cancel`), deterministic timers (B1.4b: -`clock_ms` virtual clock, `timers` list, `now_ms`/`sleep`, timer-driven `run`), AND real fd readiness -(B1.4c: lazy `kq`, `io_waiters` list, `block_on_fd`, a kqueue-blocking run-loop Mode 2 that wakes the -fiber whose fd fired). It reuses the verified `std/net/kqueue.sx` bindings (imported as `kqb`) rather -than re-deriving the FFI. Fibers can now block on either virtual `sleep(ms)` OR a real fd; both park -paths are balanced through `wake` (which evicts a stale timer AND a stale fd-waiter, the UAF guard). -Locked by `1811` (round-robin), `1812` (suspend/wake), `1813` (async go/wait/cancel), `1814` (sim-timer -deadline ordering), `1815` (timer early-wake eviction), `1816` (pipe fd block→kqueue-wake→read). Suite -GREEN **754/0**. +**STREAM B1 FEATURE-COMPLETE.** `library/modules/std/sched.sx` is the whole pure-sx M:1 async runtime: +the scheduler core (B1.5a: `spawn`/`yield_now`/`suspend_self`/`wake`/`run`), suspending fiber-task +async (B1.4a: `Task($R)`/`go`/`wait`/`cancel`), deterministic virtual-time timers (B1.4b: +`clock_ms`/`now_ms`/`sleep`, timer-driven `run`), AND real fd readiness via kqueue (B1.4c: lazy `kq`, +`io_waiters`, `block_on_fd`, run-loop Mode 2) — all over the `abi(.naked)` `swap_context` on guarded +`mmap` stacks (B1.0–B1.3), reusing `std/net/kqueue.sx`. Every park path (timer sleep, fd block, raw +suspend) is balanced through `wake` (which evicts stale timer + fd waiters — the UAF guards). Locked +by `18xx` 1800–1817 (naked-asm, context-snapshot, blocking async, the switch + §10.7 stress gate + +guarded stacks + Win64 sibling, scheduler round-robin, suspend/wake, async go/wait/cancel, sim-timer +ordering, timer early-wake eviction, kqueue pipe I/O, and the **1817 end-to-end capstone**). Suite +GREEN **755/0**, master committed. -The remaining B1 work: **B1.5** end-to-end M:1 validation under the deterministic timers / fd readiness; -a **linux epoll twin** of `block_on_fd` (mirror via `std/net/epoll`; the OS-neutral facade is -`std.event`) is future work — B1.4c wired the **macOS kqueue** path only. NOTE: the suspending async + -deterministic timers live as `sched.*` methods (M:1, receiver-driven), NOT routed through the erased -`context.io` (which would force sched.sx into every std consumer + duplicate the `_fib_tramp` global -asm); the `Io` protocol's `spawn_raw`/`suspend_raw`/`ready`/`arm_timer`/`poll` remain reserved for the -M:N evolution / when a program wants the capability-threaded form. +Future work (none blocking B1): a **linux epoll twin** of `block_on_fd` (mirror via `std/net/epoll`; +OS-neutral facade `std.event`) — B1.4c wired macOS kqueue only; routing the suspending async through +the erased `context.io` (forces sched.sx into every std consumer + duplicates the `_fib_tramp` global +asm — deferred to the M:N model, where the `Io` protocol's `spawn_raw`/`suspend_raw`/`ready`/ +`arm_timer`/`poll` hooks take over); `Future(void)`/`timeout` (issue 0150); freeing the heap-Task / +closure-env / kq-fd (a Scheduler `deinit` + closure-env-ownership affordance). **Next carve: Stream +B2** (channels / structured cancel / async stdlib) — see PLAN-CHANNELS.md when started. ### Earlier — B1.5a COMPLETE — the M:1 scheduler CORE exists `library/modules/std/sched.sx` drives N fibers @@ -449,12 +470,12 @@ fibers/Io/scheduler code yet. Grounded floor facts: boundary; a sharper sx diagnostic for it is a candidate polish, not a blocker. ## Next step -**→ B1.5 — end-to-end M:1 validation under the deterministic timers / fd readiness.** B1.4c (real -fd-readiness blocking via kqueue, `sched.block_on_fd` + the kqueue-blocking run-loop Mode 2) is done — -the macOS event-loop path exists. Build an `18xx` example that exercises the full M:1 story together -(multiple fibers, a mix of `sleep`/`go`/`wait` and `block_on_fd`, reaping, the orphan-deadlock guard). -The §10.7 gate (1808) + guarded-stack (1809) + Win64 (1810) + scheduler/async/timers/fd -(1811-1816) must keep passing throughout. +**Stream B1 is COMPLETE — no next step in this stream.** The pure-sx M:1 async runtime is feature- +complete and committed (1800–1817 green, 755/0). Pick up **Stream B2** (channels / structured cancel / +async stdlib) as a fresh carve (PLAN-CHANNELS.md), OR one of the documented non-blocking follow-ups: +the linux `epoll` twin of `block_on_fd`, a `Scheduler.deinit` (free the kq fd / heap Tasks / drain +leaks), `Future(void)`/`timeout` (needs issue 0150), or routing the suspending async through the +erased `context.io` for the M:N model. None of these block B1. **Deferred (future B1.4c sibling): the linux epoll twin of `block_on_fd`.** B1.4c wired the **macOS kqueue** path only (the host is aarch64-macOS). The linux mirror would register interest via @@ -782,3 +803,15 @@ incomplete); a dedicated effort; lambda workers are the idiom meanwhile. regression `1815` (early wake → `clock: 0`, stale timer never fires). Review cleared n_suspended accounting, deadlock false-positives, timer-list integrity, clock monotonicity, termination. Suite GREEN **753/0**. Next: B1.4c (event-loop `Io`, kqueue/epoll). +- **B1.4c COMPLETE (this session) — real fd readiness via kqueue + 2 CRITICAL review fixes.** Added a + lazy `kq` + `io_waiters` + `block_on_fd` + a kqueue-blocking run-loop Mode 2 to `sched.sx` + (worker-built, reusing `std/net/kqueue.sx`). Adversarial review found two CRITICALs: same-fd + lost-wakeup hang (FIXED — `block_on_fd` enforces one-waiter-per-fd with a loud abort) and a + never-ready-fd "hang" (RECLASSIFIED as correct event-loop semantics; misleading orphan-check comment + corrected). Locked `1816` (pipe block→kqueue-wake→read). Suite green 754/0. +- **B1.5 COMPLETE → STREAM B1 DONE (this session).** Capstone `1817` composes the whole stack + (`go`/`wait` + `sleep`/`now_ms` + scheduler) — three tasks complete in DEADLINE order + (task 2@10 / 3@20 / 1@30), `sum: 123`, final virtual clock 30. The pure-sx colorblind M:1 async + runtime is feature-complete end-to-end (1800–1817), all adversarially reviewed. Suite GREEN + **755/0**. Five compiler bugs fixed across the stream (0151/0152/0153/0154/0156-P1/0157 — 0151-3 in + B1.2). Next carve: Stream B2 (channels / cancel / async stdlib). diff --git a/current/PLAN-FIBERS.md b/current/PLAN-FIBERS.md index 1decbef7..646bfa73 100644 --- a/current/PLAN-FIBERS.md +++ b/current/PLAN-FIBERS.md @@ -1,6 +1,14 @@ # PLAN-FIBERS — Stream B1 (fibers + Io + M:1 scheduler) -> **STATUS: 🚧 in progress.** B1.0 (`abi(.naked)`) ✅ · B1.1 (per-fiber `context`) ✅ · B1.2 +> **STATUS: ✅ COMPLETE.** The pure-sx M:1 async runtime is feature-complete end-to-end +> (`library/modules/std/sched.sx`, examples 1800–1817, suite 755/0): `abi(.naked)` context switch +> (aarch64 + x86_64/Win64), M:1 scheduler, suspending `go`/`wait`/`cancel`, deterministic virtual-time +> timers (`sleep`/`now_ms`), and real fd readiness via kqueue (`block_on_fd`). Five compiler bugs fixed +> en route (0151/0152/0153/0154/0156-P1/0157). Deferred non-blocking follow-ups: linux `epoll` twin, +> `Scheduler.deinit`, `Future(void)`/`timeout` (0150), `context.io`-routed async (M:N). Next carve: +> Stream B2 (channels / cancel / async stdlib). Historical step-status below. +> +> B1.0 (`abi(.naked)`) ✅ · B1.1 (per-fiber `context`) ✅ · B1.2 > (`Io` interface + `async`/`await`/`cancel` over blocking `CBlockingIo`) ✅ · B1.3 (fiber > runtime: naked `swap_context` + §10.7 stress gate + guarded `mmap` stacks, proven on aarch64 > AND x86_64/Win64) ✅ · **B1.5a (M:1 scheduler CORE — `std/sched.sx`: `spawn`/`yield_now`/ @@ -9,7 +17,7 @@ > blockers 0156-Part1 + 0157 en route; locked `1813`). > **B1.4b (deterministic virtual-time timers — sched.sleep/now_ms/timer-run) ✅** (reviewed; fixed a CRITICAL timer-vs-early-wake UAF; locked 1814/1815). > **B1.4c (event-loop — real fd readiness via kqueue: `block_on_fd` + run-loop Mode 2) ✅** (reviewed; fixed a CRITICAL same-fd lost-wakeup hang; locked 1816). macOS only — linux epoll twin deferred. -> **→ NOW: B1.5** — end-to-end M:1 validation under the deterministic timers / fd readiness. Detailed progress in [CHECKPOINT-FIBERS.md](CHECKPOINT-FIBERS.md). NOTE: suspending async + +> **B1.5 (end-to-end M:1 capstone — `go`/`wait`+`sleep`+scheduler, deterministic ordering) ✅** (locked 1817). **STREAM B1 COMPLETE.** Detailed progress in [CHECKPOINT-FIBERS.md](CHECKPOINT-FIBERS.md). NOTE: suspending async + > deterministic timers live as `sched.*` methods (M:1), NOT routed through the erased `context.io` (avoids forcing sched.sx into every std consumer + the `_fib_tramp` dup-symbol > trap); the `Io` protocol's `spawn_raw`/`suspend_raw`/`ready` stay reserved for M:N. Deferred: > issue 0150 (`Future(void)`/`timeout`); 0156-Part2 (deferred `..` spread); the `::` callable-param @@ -223,9 +231,11 @@ Blocking exists (io.sx `CBlockingIo`). Next the deterministic-sim `Io`, **calibr blocking** before any `18xx` test trusts it; then the event loop. The deterministic `Io` is the test harness for *all* of B1.5 + Stream B2. -### B1.5 — A5: M:1 scheduler -End-to-end validation of the colorblind stack. `18xx` corpus under the deterministic `Io`, -asserting program-emitted ordering contracts. +### B1.5 — A5: M:1 scheduler — ✅ COMPLETE +End-to-end validation of the colorblind stack. The `18xx` corpus asserts program-emitted ordering +contracts under the scheduler + deterministic timers; the capstone `1817` composes `go`/`wait` + +`sleep`/`now_ms` + the scheduler (three tasks complete in deadline order, deterministic sum). Stream +B1 is feature-complete. --- diff --git a/examples/concurrency/1817-concurrency-fiber-m1-end-to-end.sx b/examples/concurrency/1817-concurrency-fiber-m1-end-to-end.sx new file mode 100644 index 00000000..32b66ce1 --- /dev/null +++ b/examples/concurrency/1817-concurrency-fiber-m1-end-to-end.sx @@ -0,0 +1,66 @@ +// Stream B1 (fibers) B1.5 — the M:1 colorblind async stack, end-to-end. +// +// One program exercises the whole pure-sx runtime together: the M:1 scheduler +// (B1.5a), the suspending fiber-task async layer `go`/`wait` (B1.4a), and the +// deterministic virtual-time timers `sleep`/`now_ms` (B1.4b) — all over the +// `abi(.naked)` `swap_context` on guarded `mmap` stacks (B1.0–B1.3). +// +// A coordinator fiber launches three async tasks; each `sleep`s a different +// duration, records its completion (id @ virtual-ms) into a shared log, then +// returns a value. The coordinator `wait`s on all three (in SPAWN order) and +// sums their results. Because tasks complete in DEADLINE order — not spawn +// order, not await order — the completion log is the deterministic contract: +// +// task A: sleep 30 → returns 100 +// task B: sleep 10 → returns 20 +// task C: sleep 20 → returns 3 +// completion order (by deadline): B@10, C@20, A@30 +// coordinator awaits A,B,C → sum = 123, final virtual clock = 30 +// +// `wait(A)` parks the coordinator until A finishes at t=30; B and C finish +// earlier (at 10 and 20) and are already `.ready` by the time their `wait`s run, +// so they return without re-parking — the values are correct regardless of await +// order, while the timer-driven schedule fixes the completion ORDER. Fully +// deterministic + reproducible (virtual time, no real clock). +// +// aarch64-macOS-pinned (the scheduler's per-arch asm + Apple mmap constants): +// runs end-to-end on a matching host, ir-only on a mismatch. +#import "modules/std.sx"; +sched :: #import "modules/std/sched.sx"; + +Log :: struct { id: [8]i64; at: [8]i64; n: i64; } +rec :: (l: *Log, id: i64, at: i64) { l.id[l.n] = id; l.at[l.n] = at; l.n = l.n + 1; } + +main :: () -> i64 { + lg : Log = ---; lg.n = 0; + s := sched.Scheduler.init(); + ps := @s; pl := @lg; + + // The coordinator runs as a fiber so `wait` has a `current` to park. + s.spawn(() => { + // Launch three async tasks; each sleeps, logs its completion, returns. + a := ps.go(() -> i64 => { ps.sleep(30); rec(pl, 1, ps.now_ms()); 100 }); + b := ps.go(() -> i64 => { ps.sleep(10); rec(pl, 2, ps.now_ms()); 20 }); + c := ps.go(() -> i64 => { ps.sleep(20); rec(pl, 3, ps.now_ms()); 3 }); + + // Await in SPAWN order; results come back correct regardless. + va := a.wait() or { -1 }; + vb := b.wait() or { -1 }; + vc := c.wait() or { -1 }; + sum := va + vb + vc; + + rec(pl, 9, sum); // sentinel row: id=9 carries the sum in `at` + }); + s.run(); + + print("completion order (id @ virtual-ms):\n"); + i := 0; + while i < lg.n { + if lg.id[i] == 9 { print("sum: {}\n", lg.at[i]); } + else { print(" task {} @ {}ms\n", lg.id[i], lg.at[i]); } + i = i + 1; + } + print("final virtual clock: {}ms\n", s.now_ms()); + print("tasks: {}\n", s.n_spawned); + return 0; +} diff --git a/examples/concurrency/expected/1817-concurrency-fiber-m1-end-to-end.build b/examples/concurrency/expected/1817-concurrency-fiber-m1-end-to-end.build new file mode 100644 index 00000000..42e24dd2 --- /dev/null +++ b/examples/concurrency/expected/1817-concurrency-fiber-m1-end-to-end.build @@ -0,0 +1 @@ +{ "target": "macos" } diff --git a/examples/concurrency/expected/1817-concurrency-fiber-m1-end-to-end.exit b/examples/concurrency/expected/1817-concurrency-fiber-m1-end-to-end.exit new file mode 100644 index 00000000..573541ac --- /dev/null +++ b/examples/concurrency/expected/1817-concurrency-fiber-m1-end-to-end.exit @@ -0,0 +1 @@ +0 diff --git a/examples/concurrency/expected/1817-concurrency-fiber-m1-end-to-end.stderr b/examples/concurrency/expected/1817-concurrency-fiber-m1-end-to-end.stderr new file mode 100644 index 00000000..8b137891 --- /dev/null +++ b/examples/concurrency/expected/1817-concurrency-fiber-m1-end-to-end.stderr @@ -0,0 +1 @@ + diff --git a/examples/concurrency/expected/1817-concurrency-fiber-m1-end-to-end.stdout b/examples/concurrency/expected/1817-concurrency-fiber-m1-end-to-end.stdout new file mode 100644 index 00000000..14e6c728 --- /dev/null +++ b/examples/concurrency/expected/1817-concurrency-fiber-m1-end-to-end.stdout @@ -0,0 +1,7 @@ +completion order (id @ virtual-ms): + task 2 @ 10ms + task 3 @ 20ms + task 1 @ 30ms +sum: 123 +final virtual clock: 30ms +tasks: 4