Compare commits
1 Commits
master
...
fix/0192-q
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
22f4719e83 |
394
current/CHECKPOINT-ASM.md
Normal file
394
current/CHECKPOINT-ASM.md
Normal file
@@ -0,0 +1,394 @@
|
||||
# sx Inline Assembly — Checkpoint (ASM stream)
|
||||
|
||||
Companion to `current/PLAN-ASM.md`; design in
|
||||
[design/inline-asm-design.md](../design/inline-asm-design.md). Update after every
|
||||
commit, one step at a time per the cadence rule (no commit may both add a test
|
||||
and make it pass).
|
||||
|
||||
## Last completed step
|
||||
**G (indirect-memory `=*m` place outputs)** — the LAST substantive asm feature.
|
||||
Unlike a write-through `=` output (which returns a value then stored), an
|
||||
indirect output passes the place ADDRESS to the asm and the asm writes through
|
||||
it — no return slot. `emitInlineAsm` (`src/backend/llvm/ops.zig`): indirect
|
||||
outputs are excluded from the LLVM return type; their pointer is an opaque `ptr`
|
||||
call arg placed **first** (arg-consuming constraint order = output-section
|
||||
indirect pointers → inputs → read-write tied seeds); each gets an
|
||||
`elementtype(T)` call-site attribute (required in the opaque-pointer era) via
|
||||
`LLVMCreateTypeAttribute`/`LLVMAddCallSiteAttribute`; the store-back loop skips
|
||||
them. New `asmIsIndirect(e, op)` helper. Lowering (`lowerAsmExpr`) stops
|
||||
rejecting `*` (constraint kept verbatim, `=*m` reaches the constraint string
|
||||
as-is). `asmOperandIndex` unchanged — indirect outputs still count as operands,
|
||||
so `%[name]`→`${N}` holds. Verified by **running** on aarch64: store-through-
|
||||
pointer (`str x9, %[out]` → 42, IR `"=*m,~{x9}"(ptr elementtype(i64) …)`) and a
|
||||
mixed case (indirect + value output + input → `"=*m,=r,r"`, indirect ptr arg
|
||||
first, `${0}/${1}/${2}` correct). Two commits per cadence: (1)
|
||||
`examples/1652-platform-asm-indirect-mem.sx` locked the rejection; (2) implemented
|
||||
+ flipped 1652 to a runnable aarch64-pinned example (`{ "target": "macos" }`,
|
||||
ir-only elsewhere). `zig build test` green (661 corpus, 446 unit). Files:
|
||||
`src/ir/lower/expr.zig`, `src/backend/llvm/ops.zig`, `examples/1652-*`.
|
||||
|
||||
Prior: **G (read-write `+` place outputs)** — a `+r` / `+{reg}` `-> @place` output is now
|
||||
implemented. LLVM has no `+` constraint, so a
|
||||
read-write place lowers to: an output **`=`** constraint (return slot, stored back
|
||||
through the place after the call; the leading `+` rewritten to `=` in
|
||||
`appendAsmConstraints`), **plus** a **tied input** (the decimal index of that
|
||||
output) appended **after** the regular inputs, seeded with the place's loaded
|
||||
value passed as a call arg. Tied inputs come **last** so existing operand indices
|
||||
(`%[name]`→`${N}`) are undisturbed — `asmOperandIndex` unchanged. Lowering
|
||||
(`lowerAsmExpr`) no longer rejects `+` (indirect `*` still rejected loudly).
|
||||
`emitInlineAsm` (`src/backend/llvm/ops.zig`): grows arg/param arrays by the rw
|
||||
count (`n_args = n_inputs + n_rw`), loads each seed (`asm.rw.seed`), emits the
|
||||
tied constraint, and the existing store-back path writes the modified output back.
|
||||
New `asmIsReadWrite(e, op)` helper. Verified by **running**: increment-in-place
|
||||
(41→42, IR `"=r,0"`) and a mixed case (rw place + regular input + value output) →
|
||||
textbook `"=r,=r,r,0"` with correct `${N}` indices and args `(input, seed)`. Two
|
||||
commits per cadence: (1) `examples/1650-platform-asm-rw-place.sx` locked the
|
||||
rejection; (2) implemented + flipped 1650 to a runnable aarch64-pinned example
|
||||
(`{ "target": "macos" }`, ir-only elsewhere). `zig build test` green (658 corpus,
|
||||
446 unit). Files: `src/ir/lower/expr.zig`, `src/backend/llvm/ops.zig`,
|
||||
`examples/1650-*`.
|
||||
|
||||
Prior: **2** — `-> @place` write-through outputs. An asm result can be **stored through
|
||||
a place** (local / struct field) instead of returned; the place output does NOT
|
||||
join the result tuple. Parser: `-> @place` parses the `@place` as an ordinary
|
||||
address-of expression → an `out_place` operand (`src/parser.zig`). Lowering
|
||||
(`lowerAsmExpr`): out_place operand = the lowered `@place` address, `out_ty` =
|
||||
the pointee; read-write (`+`) and indirect-memory (`*`) constraints rejected
|
||||
loudly (not yet implemented). Added `out_ty: TypeId` to the IR `AsmOperand`
|
||||
(`src/ir/inst.zig`) so emit builds the **combined** return struct (ALL outputs).
|
||||
`emitInlineAsm` rewrite (`src/backend/llvm/ops.zig`): the LLVM return type is now
|
||||
built from every output's `out_ty`; after the call, out_place slots are
|
||||
`store`d through their address and out_value slots rebuild the sx result — with a
|
||||
**fast path** (no place outputs → the asm's struct return IS the result, so
|
||||
pure-value asm IR is unchanged). Verified: write-to-local (`get42`→42), struct
|
||||
field (`@p.b`), mixed value+place (`v=10 b=20`), `+` rejected. Locked with
|
||||
`examples/1649-platform-asm-place-output.sx` (mixed, runs on aarch64). `zig build
|
||||
test` green (657 corpus, 446 unit). Files: `src/parser.zig`, `src/ir/inst.zig`,
|
||||
`src/ir/lower/expr.zig`, `src/backend/llvm/ops.zig`, `examples/1649-*`.
|
||||
|
||||
Prior: **F** — global (module-scope) asm. A top-level `asm { "tmpl", };` block (template
|
||||
only) lowers to LLVM `module asm`, and a lib-less `extern` calls into the symbols
|
||||
it defines. New `asm_global` AST node (`src/ast.zig`) + `parseAsmGlobal`
|
||||
(`src/parser.zig`, dispatched from `parseTopLevel` on `kw_asm`) — rejects
|
||||
`volatile` and any operands/clobbers. The node forced (and got) arms in the same
|
||||
three `Node.Data` switches as `asm_expr` (`sema.zig` ×2, `semantic_diagnostics.zig`).
|
||||
`Module` gains a `global_asm: ArrayList([]const u8)` (`src/ir/module.zig`);
|
||||
`lowerMainAndComptime` captures each template (the dead `lowerDecls` is NOT the
|
||||
top-level pass — `lowerRoot` Pass 2 uses `lowerMainAndComptime`); `emit_llvm.zig`'s
|
||||
`emit()` appends each via `LLVMAppendModuleInlineAsm` (source order). Verified
|
||||
end-to-end: an aarch64 `_my_add` global routine called via `extern` returns 42.
|
||||
Locked with `examples/1648-platform-asm-global.sx`
|
||||
(`.build { "aot": true, "target": "macos" }` → AOT build+run on aarch64, ir-only
|
||||
elsewhere). `zig build test` green (656 corpus, 446 unit). **(Correction, later:
|
||||
module asm ALSO runs under the JIT — `sx run` compiles to an in-memory object,
|
||||
the integrated assembler assembles the `module asm` into it, ORC relocates and
|
||||
runs it, so the symbol is resolvable at JIT main execution. The original "AOT
|
||||
only" note was wrong; see 1653 for the JIT sibling. The genuine boundary is a
|
||||
COMPILE-TIME `#run` call into a module-asm symbol, which fails loud via host
|
||||
dlsym-miss — see 1654.)** Files: `src/ast.zig`, `src/parser.zig`, `src/sema.zig`,
|
||||
`src/ir/semantic_diagnostics.zig`, `src/ir/module.zig`, `src/ir/lower/decl.zig`,
|
||||
`src/ir/emit_llvm.zig`, `examples/1648-*`.
|
||||
|
||||
Prior: **E** — multi-output tuples. **Inline asm now returns tuples.** Replaced the
|
||||
N>1 bail with a shared `asmResultType` helper (`src/ir/lower/expr.zig`, mixed
|
||||
into `Lowering`) that derives the result type from the `out_value` operands
|
||||
(0→void, 1→T, N→named tuple, named via the §II.5 effective-name rule). The key
|
||||
realization: `toLLVMType(tuple)` already produces a literal struct `{T1,…,Tn}` —
|
||||
exactly LLVM's multi-output asm return — so **emit needed NO change**; building
|
||||
the op with a tuple result type makes the asm call return the struct, which IS
|
||||
sx's tuple value (destructured by the normal `tuple_get` path). `inferType`'s
|
||||
`.asm_expr` arm now also delegates to `asmResultType` (single owner), so
|
||||
`return asm`, `x := asm`, and `q, r := asm` all agree on the type. Verified
|
||||
end-to-end on aarch64: `split(0x1234)`→`(lo=52, hi=18)`, a udiv/msub divmod→
|
||||
`(3, 2)`. IR is textbook: `call { i64, i64 } asm "divq ${4}",
|
||||
"={rax},={rdx},{rax},{rdx},r,~{cc}"(…)` → extractvalue → tuple. Converted 1640 to
|
||||
the x86_64 multi-output IR lock (ir-only) + added `1647-platform-asm-aarch64-multi`
|
||||
(runs on aarch64). `zig build test` green (655 corpus, 446 unit). Files:
|
||||
`src/ir/lower/expr.zig`, `src/ir/lower.zig`, `src/ir/expr_typer.zig`,
|
||||
`examples/164{0,7}-*`.
|
||||
|
||||
Prior: **C.1 + D** — inline asm CODEGEN (lowering builds the op + LLVM emit). **Inline
|
||||
assembly now runs end-to-end.** `lowerAsmExpr` (`src/ir/lower/expr.zig`) stops
|
||||
bailing: it resolves each operand's effective name (§II.5 auto-naming), interns
|
||||
template/constraints/clobbers, lowers input `Ref`s, derives the result `TypeId`
|
||||
(0→void, 1→T), and builds the `inline_asm` op. Added a `%[name]`-references-a-
|
||||
real-operand check (the last deferred validation). Multi-output (N>1) still bails
|
||||
loudly ("Phase E"). `emitInlineAsm` (`src/backend/llvm/ops.zig`, port of Zig's
|
||||
`airAssembly`): assembles the LLVM constraint string (outputs→inputs→`~{clobber}`,
|
||||
`,`→`|`), rewrites the template (`%[name]`→`${N}`, `%%`→`%`, `$`→`$$`, `%=`→
|
||||
`${:uid}`), then `LLVMGetInlineAsm` + `LLVMBuildCall2` (AT&T). Dispatch wired
|
||||
(`emit_llvm.zig`, replacing the C.0 `@panic`). **`llvm_shim.c`**: added
|
||||
`LLVMInitializeNativeAsmParser()` — the JIT must assemble inline asm at run time.
|
||||
Verified end-to-end: aarch64 `add`/`mov` run on the host (exit 42), `nop volatile`
|
||||
runs (1642 now exit 0), IR is textbook (`call i64 asm "add ${0},${1},${2}",
|
||||
"=r,r,r"(…)`). Locked with `examples/1645-platform-asm-aarch64-add.sx` (runs on
|
||||
aarch64, ir-only elsewhere via `.build` + `.ir`). Also added the `inferType`
|
||||
`.asm_expr` arm (`src/ir/expr_typer.zig`, 0→void / 1→T) — without it a bare
|
||||
`x := asm {…-> T}` binding inferred `.unresolved` and silently produced 0;
|
||||
regression-locked with `examples/1646-platform-asm-value-binding.sx`. Updated
|
||||
1640 (now Phase-E bail) + 1642 (now runs). `zig build test` green (654 corpus,
|
||||
446 unit). Files: `src/ir/lower/expr.zig`, `src/backend/llvm/ops.zig`,
|
||||
`src/ir/emit_llvm.zig`, `src/ir/expr_typer.zig`, `llvm_shim.c`,
|
||||
`examples/164{0,2,5,6}-*`.
|
||||
|
||||
Prior: **C.0** — IR op `inline_asm` (lock; no behavior change). Added `inline_asm:
|
||||
InlineAsm` to the IR `Op` union + the `InlineAsm` struct (`template: StringId`,
|
||||
`operands: []const AsmOperand` {role/name/constraint/operand}, `clobbers:
|
||||
[]const StringId`, `has_side_effects`) in `src/ir/inst.zig` — all strings
|
||||
interned, operands in source order, result on `Inst.ty`. The new variant forced
|
||||
(and got) arms in two exhaustive `Op` switches: `src/ir/interp.zig` (loud
|
||||
`bailDetail` — inline asm is never comptime-evaluable) and `src/ir/print.zig`
|
||||
(IR dump). `src/ir/emit_llvm.zig` gets a `@panic` **tripwire** — emit lands in
|
||||
Phase D, and until then `lowerAsmExpr` still bails so no `inline_asm` op is ever
|
||||
created (reaching emit would be a lowering-switched-over-too-early bug). Unit
|
||||
test `inline_asm op shape` in `src/ir/inst.test.zig`. `zig build test` green
|
||||
(652 corpus, 446 unit). Files: `src/ir/inst.zig`, `src/ir/interp.zig`,
|
||||
`src/ir/print.zig`, `src/ir/emit_llvm.zig`, `src/ir/inst.test.zig`.
|
||||
|
||||
Prior: **B.1** — operand-name validation (design §II.5 auto-naming rule). Extended
|
||||
`lowerAsmExpr` with a `pinnedRegister(constraint)` helper (`"={eax}"`→`eax`,
|
||||
`"+{rax}"`→`rax`, `"=r"`→null) and two checks: (1) **reject the echo form**
|
||||
`[eax] "={eax}"` — a label identical to its own pinned register is redundant
|
||||
(the operand is already auto-named after the register); (2) **reject duplicate
|
||||
operand names** (ambiguous `%[name]` / result field). Locked with
|
||||
`examples/1643-platform-asm-echo-name.sx` + `1644-platform-asm-duplicate-name.sx`.
|
||||
`zig build test` green (652 corpus, 0 failed; 445 unit). Files:
|
||||
`src/ir/lower/expr.zig`.
|
||||
|
||||
Prior: **B.0** — asm shape validation (compile-path diagnostics). Restructured the
|
||||
`.asm_expr` lowering arm into `lowerAsmExpr` (`src/ir/lower/expr.zig`, mixed into
|
||||
`Lowering` in `src/ir/lower.zig`): it validates BEFORE the not-yet-implemented
|
||||
codegen bail, so the user sees the real problem first. Two checklist items now
|
||||
enforced with named diagnostics: (1) **template must be a compile-time-known
|
||||
string** (`"..."` / `#string`); (2) **no value outputs ⇒ must be `volatile`**
|
||||
(mirrors Zig — a result-less asm could be deleted). Valid shapes still bail with
|
||||
the "codegen not yet implemented" message. Result-type derivation + auto-naming
|
||||
stay deferred to a later step (observable only once Phase C produces a real IR
|
||||
op). Locked with `examples/1641-platform-asm-missing-volatile.sx` (volatile
|
||||
error) + `1642-platform-asm-nop-volatile.sx` (volatile no-output accepted →
|
||||
codegen bail). `zig build test` green (650 corpus, 0 failed; 445 unit). Files:
|
||||
`src/ir/lower/expr.zig`, `src/ir/lower.zig`, `examples/164{1,2}-*`.
|
||||
|
||||
Prior: **A.1** — parse `asm { … }` + loud lowering bail (folded A.1+A.2 into one honest
|
||||
lock commit, since the loud bail IS current correct behavior — cadence option
|
||||
(a)). Added `AsmExpr`/`AsmOperand` to `src/ast.zig` + the `asm_expr` `Node.Data`
|
||||
arm; `parseAsmExpr` in `src/parser.zig` (`parsePrimary` `.kw_asm` dispatch) —
|
||||
parses the template, flat operand list (`[name]? "constraint" -> Type` value
|
||||
output / `= expr` input), and `clobbers(.…)`; `volatile`/`clobbers` recognized
|
||||
contextually via `isContextualWord`. The new `asm_expr` tag forced (and got)
|
||||
arms in three exhaustive `Node.Data` switches: `src/sema.zig` `analyzeNode` +
|
||||
`findNodeAtOffset`, `src/ir/semantic_diagnostics.zig` `checkBindingNames` (all
|
||||
recurse into template + operand payloads). Lowering bails LOUD + named in
|
||||
`src/ir/lower/expr.zig` ("inline assembly codegen is not yet implemented…") via
|
||||
an explicit `.asm_expr` arm (not the generic `unknown_expr` else) returning
|
||||
`emitPlaceholder`. `-> @place` write-through is rejected with a clear "Phase 2"
|
||||
parse error. Locked with `examples/1640-platform-asm-parse.sx` (multi-output
|
||||
`divmod`, named operands, register pins, clobbers — parses then bails; called
|
||||
from `main`). `zig build test` green (648 corpus, 0 failed; 445 unit). Files:
|
||||
`src/ast.zig`, `src/parser.zig`, `src/sema.zig`, `src/ir/semantic_diagnostics.zig`,
|
||||
`src/ir/lower/expr.zig`, `examples/1640-*`.
|
||||
|
||||
Prior: **A.0** — `kw_asm` keyword (first compiler code). Added the `kw_asm` `Token.Tag`
|
||||
variant + `.{ "asm", .kw_asm }` keyword-map entry in `src/token.zig`; `volatile` /
|
||||
`clobbers` deliberately stay OUT of the global table (contextual). New exhaustive
|
||||
`Tag` switch in `src/lsp/server.zig` `classifyToken` flagged the missing arm (the
|
||||
intended coverage tripwire) — added `.kw_asm` to the keyword group. Lock test in
|
||||
new `src/lexer.test.zig` (`asm`→`kw_asm`, `volatile`/`clobbers`→`identifier`),
|
||||
wired into the `src/root.zig` barrel as `lexer_tests`. `zig build test` green (648
|
||||
corpus, 0 failed; 445 unit, 0 failed — +1). Files: `src/token.zig`,
|
||||
`src/lexer.test.zig`, `src/root.zig`, `src/lsp/server.zig`.
|
||||
|
||||
Prior: **0.2** — CLAUDE.md docs for `<name>.build`; **Phase 0 COMPLETE**.
|
||||
**0.1** — corpus runner **ir-only branch** for cross-target examples. Replaced
|
||||
0.0's loud placeholder bail: when `cfg.target` doesn't match the host (`ir_only`),
|
||||
`sweepRoot` skips run/build/exec and verifies via `sx ir --target` only —
|
||||
asserting `.exit` (ir cmd) + `.ir` (normalized stdout) + `.stderr`, never
|
||||
`.stdout` (write skipped in update mode, assertion skipped in verify mode). An
|
||||
`.ir` snapshot is **required** in ir-only mode — its absence is a loud failure
|
||||
("needs an .ir snapshot for ir-only mode"). Locked with
|
||||
`examples/1639-platform-target-cross.sx` (asm-free `main :: () -> i64 { return 0;
|
||||
}`), `.build` `{ "target": "x86_64-linux" }`, + checked-in `.ir`. Verified both
|
||||
guards fire: corrupting the `.ir` → IR mismatch; deleting it → the require-failure.
|
||||
`zig build test` green (647 corpus, 0 failed; 444 unit). Files:
|
||||
`src/corpus_run.test.zig`, `examples/1639-*`.
|
||||
|
||||
## Current state
|
||||
**Inline assembly works end-to-end: 0, 1, and N value outputs (tuples).** Full
|
||||
pipeline: lex (A.0) → parse (A.1) → validate (B.0/B.1 + `%[name]` check) → IR op
|
||||
(C.0) → lower-builds-op + LLVM emit + JIT asm-parser init (C.1/D) → multi-output
|
||||
tuples (E). Register-class + register-pinned operands, inputs, **symbol operands
|
||||
(`"s"` → direct `bl`/`call` to a function/global by mangled name)**, clobbers,
|
||||
`#string` multi-instruction templates, `%[name]`/`%%` rewriting, and the §II.5
|
||||
auto-naming rule all work and execute on the host JIT. Global `asm { … }` (Phase F) works via
|
||||
lib-less `extern` under BOTH the JIT (`sx run` → 1653) and AOT (1648) — `sx run`
|
||||
compiles to an object, so the integrated assembler bakes the `module asm` symbol
|
||||
in and ORC resolves it. All three `-> @place` output forms now work and execute
|
||||
on aarch64: **write-through** `=` (Phase 2), **read-write** `+` (tied input), and
|
||||
**indirect-memory** `=*m` (pointer arg + `elementtype`, asm writes through it).
|
||||
**Inline assembly is now feature-complete — no substantive features remain.** The
|
||||
x86_64 syscall-write ir-only example is DONE (1651). Global asm runs under both
|
||||
JIT (1653) and AOT (1648). `readme.md` now has an "Inline Assembly" section.
|
||||
|
||||
Known orthogonal bug: **issue 0137** — `sx run` on a program with no `main`
|
||||
segfaults (`src/target.zig:256-273`, unguarded JIT entry lookup). Pre-existing,
|
||||
asm-independent; does NOT block the ASM stream (every example has a `main`).
|
||||
|
||||
Phase E–F feasibility already confirmed against the live tree
|
||||
(`LLVMGetInlineAsm` / `LLVMBuildCall2` / `LLVMAppendModuleInlineAsm` in LLVM@19
|
||||
`Core.h`; ERR-stream `extractvalue`→tuple in `emit_llvm.zig:726-927`; lib-less
|
||||
`extern`, 60 sites; `--target` a global CLI flag).
|
||||
|
||||
## Next step
|
||||
**Inline assembly is feature-complete.** All substantive features are done:
|
||||
0/1/N value outputs (tuples), register-class + pinned operands, inputs, clobbers,
|
||||
`#string` templates, `%[name]`/`%%`/`$`/`%=` rewriting, §II.5 auto-naming, global
|
||||
`asm { … }` (AOT), and all three `-> @place` output forms — write-through (`=`),
|
||||
read-write (`+`), and indirect-memory (`=*m`). The x86_64 syscall-write ir-only
|
||||
example (1651) and the output-to-`const` rejection (issue 0138) are also done.
|
||||
|
||||
Global asm runs under BOTH the JIT (`sx run` → object → ORC; 1653) and AOT (1648)
|
||||
— the earlier "AOT only / `sx run` mishandles module-asm" note was stale and has
|
||||
been corrected. The one genuine boundary is a COMPILE-TIME `#run` into a
|
||||
module-asm symbol: the interpreter resolves externs via host dlsym, the symbol
|
||||
isn't linked yet, so it already fails loud (`comptime extern call: symbol not
|
||||
found via dlsym`) — pinned by 1654.
|
||||
|
||||
Remaining work, all **polish** (optional):
|
||||
- None substantive. Possible niceties: tighten the `#run`-into-module-asm error
|
||||
text to name module-asm specifically; broaden clobber validation to a checked
|
||||
per-arch enum (design doc Phase 4).
|
||||
|
||||
Orthogonal: **issue 0137** (no-`main` JIT segfault).
|
||||
|
||||
Done since last: output-to-`const` rejection (issue 0138), x86_64 syscall-write
|
||||
ir-only example (1651).
|
||||
|
||||
Orthogonal: **issue 0137** (no-`main` segfault).
|
||||
|
||||
## Log
|
||||
- (init) Plan + design doc written; ASM stream opened.
|
||||
- (0.0) Corpus runner target-gating: `<name>.build` JSON config (replaces `.aot`
|
||||
marker), `--target` threading, `hostMatchesTarget` execute-gate, loud
|
||||
cross-target placeholder bail. Migrated 1226/1227 `.aot`→`.build`; locked with
|
||||
1638 fixture + unit tests. `zig build test` green.
|
||||
- (0.1) ir-only branch: cross-target examples verify via `sx ir --target` only
|
||||
(exit+ir+stderr, no stdout; `.ir` required). Locked with 1639 fixture; verified
|
||||
corrupt-.ir → mismatch and missing-.ir → loud failure. `zig build test` green.
|
||||
- (0.2) docs: CLAUDE.md documents `<name>.build` JSON sidecar (aot + target +
|
||||
ir-only gating), replacing stale `.aot` marker prose. **Phase 0 COMPLETE.**
|
||||
- (A.0) `kw_asm` keyword in token.zig (+ map entry); LSP `classifyToken` switch
|
||||
coverage; lock test in new `lexer.test.zig` (wired via root.zig). `volatile` /
|
||||
`clobbers` stay contextual identifiers. `zig build test` green (445 unit, +1).
|
||||
- (A.1) parse `asm { … }` → `AsmExpr` + loud lowering bail; `asm_expr` arms in 3
|
||||
exhaustive `Node.Data` switches; `-> @place` rejected (Phase 2). Adopted operand
|
||||
auto-naming rule (design §II.5). Locked with 1640 fixture. Filed orthogonal
|
||||
issue 0137 (no-`main` JIT segfault). `zig build test` green (648 corpus, 445 unit).
|
||||
- (B.0) asm shape validation in `lowerAsmExpr`: comptime-string template +
|
||||
no-output⇒volatile, with named diagnostics before the codegen bail. Locked with
|
||||
1641 (volatile error) + 1642 (volatile accepted). `zig build test` green (650
|
||||
corpus, 445 unit).
|
||||
- (B.1) operand-name validation: `pinnedRegister` helper + reject echo form
|
||||
(`[eax] "={eax}"`) and duplicate names. Locked with 1643 + 1644. `zig build
|
||||
test` green (652 corpus, 445 unit).
|
||||
- (C.0) IR op `inline_asm: InlineAsm` + interp `bailDetail` + print arm + emit
|
||||
`@panic` tripwire (Phase D). No behavior change (lowering still bails). Unit
|
||||
test `inline_asm op shape`. `zig build test` green (652 corpus, 446 unit).
|
||||
- (C.1+D) CODEGEN — `lowerAsmExpr` builds the op (effective names, interned
|
||||
strings, input Refs, 0/1 result type) + `%[name]` validation; `emitInlineAsm`
|
||||
(constraint string + template rewrite + `LLVMGetInlineAsm`/`BuildCall2`, AT&T);
|
||||
`inferType` arm; `LLVMInitializeNativeAsmParser` for the JIT. **Inline asm runs
|
||||
end-to-end.** N>1 bails (Phase E). Locked with 1645 (aarch64 add, runs) + 1646
|
||||
(`:=` binding); updated 1640/1642. `zig build test` green (654 corpus, 446 unit).
|
||||
- (E) multi-output tuples — `asmResultType` helper (0→void/1→T/N→named tuple),
|
||||
shared by lowering + `inferType`. `toLLVMType(tuple)` == LLVM multi-output
|
||||
struct, so emit unchanged; the asm struct return IS the sx tuple. Runs on
|
||||
aarch64 (1647: `split`→`(lo,hi)`); 1640 → x86 multi-output IR lock (ir-only).
|
||||
`zig build test` green (655 corpus, 446 unit).
|
||||
- (F) global asm — `asm_global` AST node + `parseAsmGlobal` (top-level, rejects
|
||||
volatile/operands); `Module.global_asm` captured in `lowerMainAndComptime`;
|
||||
`emit()` appends via `LLVMAppendModuleInlineAsm`; call-into via lib-less
|
||||
`extern`. AOT-verified (1648, `_my_add`→42). `zig build test` green (656 corpus).
|
||||
- (docs) readme.md "Inline Assembly" section (b8800a2).
|
||||
- (2) `-> @place` write-through — `out_place` operand; `out_ty` on the IR
|
||||
AsmOperand; `emitInlineAsm` builds the combined output struct + splits
|
||||
(out_place → store-through, out_value → result), fast-path when no places.
|
||||
`+`/`*` rejected. Locked with 1649 (mixed, runs). `zig build test` green (657
|
||||
corpus, 446 unit).
|
||||
- (G) read-write `+` place outputs — `+` lowers to an output `=` + a tied input
|
||||
(output-index constraint) seeded with the place's loaded value, tied inputs
|
||||
appended last (operand indices undisturbed). `appendAsmConstraints` rewrites
|
||||
`+`→`=`; `emitInlineAsm` grows args by the rw count + loads seeds;
|
||||
`asmIsReadWrite` helper. Lowering stops rejecting `+` (`*` still rejected). Two
|
||||
commits (cadence): 1650 locked the rejection, then flipped to a runnable
|
||||
aarch64 example (`"=r,0"` IR). `zig build test` green (658 corpus, 446 unit).
|
||||
- (0138) output-to-`const` rejection — fixed the underlying general bug: scalar
|
||||
`@const` (address-of a folded `::` constant) reinterpreted the value as a
|
||||
pointer (`inttoptr`). `src/ir/lower/expr.zig` `.address_of` now diagnoses a
|
||||
scalar const (local + module) instead of falling through; array/struct consts
|
||||
keep storage. asm `-> @const` gets the clean diagnostic for free (same path).
|
||||
Regression `examples/1177-diagnostics-addr-of-const-rejected.sx`. Issue 0138
|
||||
RESOLVED. `zig build test` green (659 corpus, 446 unit).
|
||||
- (x86 syscall) x86_64 Linux `write(2)` via raw `syscall` — locks the constraint
|
||||
string `={rax},{rax},{rdi},{rsi},{rdx},~{rcx},~{r11},~{memory}` (register-pinned
|
||||
inputs + pinned value output + pointer input + clobbers). ir-only on aarch64
|
||||
(`.ir` asserted), runs on x86_64-linux (hand-authored `"ok\n"` stdout).
|
||||
`examples/1651-platform-asm-x86-syscall-write.sx`. Pure additive lock, no
|
||||
compiler change. `zig build test` green (660 corpus, 446 unit).
|
||||
- (G indirect) indirect-memory `=*m` place outputs — the place address is passed
|
||||
as an opaque `ptr` arg (with an `elementtype(T)` call-site attr), placed before
|
||||
inputs; asm writes through it; no return slot; store-back skips it.
|
||||
`asmIsIndirect` helper; lowering stops rejecting `*`. Verified by running on
|
||||
aarch64 (store-through → 42; mixed indirect+value+input → `"=*m,=r,r"`). Two
|
||||
commits (cadence): 1652 locked the rejection, then flipped to a runnable aarch64
|
||||
example. **Inline asm now feature-complete.** `zig build test` green (661 corpus,
|
||||
446 unit).
|
||||
- (jit) explored "asm in JIT": found it ALREADY works — `sx run` emits an
|
||||
in-memory object (integrated assembler bakes in both in-function inline asm and
|
||||
`module asm`), then ORC relocates+runs it. The stale "AOT only / `sx run`
|
||||
mishandles module-asm" checkpoint prose was corrected. Locked global-asm-under-
|
||||
JIT with `examples/1653-platform-asm-global-jit.sx` (`{ "target": "macos" }`, no
|
||||
aot, → 42). `zig build test` green (662 corpus, 446 unit).
|
||||
- (comptime guard) pinned the one genuine module-asm boundary:
|
||||
`examples/1654-platform-asm-global-comptime-call.sx` — `#run` into a module-asm
|
||||
symbol fails loud (`comptime extern call: symbol not found via dlsym`) because
|
||||
the interpreter resolves externs via host dlsym before link. Arch-independent
|
||||
(no `.build`). `zig build test` green (663 corpus, 446 unit).
|
||||
- (round trip) `examples/1655-platform-asm-callback-into-sx.sx` — global-asm
|
||||
trampoline that `bl _cb` back into an `export`ed sx function (sx→asm→sx, → 42).
|
||||
Documented that `export` (external linkage + C symbol + C ABI) is what makes
|
||||
the callback resolvable; `callconv(.c)` alone leaves it `internal` (DCE'd).
|
||||
`zig build test` green (664 corpus, 446 unit).
|
||||
- (symbol ops) symbol operands (`"s"`) — feed a function/global symbol; the
|
||||
template emits its platform-mangled name so `bl %[fn]` is a DIRECT branch (one
|
||||
fewer indirection than register-indirect `blr`, portable — no hardcoded `_`).
|
||||
Emit passes the operand with its own llvm type (LLVMTypeOf), no coercion
|
||||
(`asmIsSymbol` helper); lowering lowers the function RHS to `ptr @fn`. Decided
|
||||
AGAINST mirroring Zig (which has no symbol operand — 483 std asm sites, none
|
||||
call a function) because the direct `bl` matters. Two commits (cadence): 1656
|
||||
locked the rejection (replacing an LLVM-verifier crash), then implemented +
|
||||
flipped to a runnable aarch64 example (objdump-confirmed direct `bl <_cb>`).
|
||||
`zig build test` green (665 corpus, 446 unit).
|
||||
- (x86 cross-arch) ir-only x86_64 siblings so each emit path is locked on BOTH
|
||||
arches: 1657 read-write (`"incq ${0}","=r,0"`), 1658 indirect (`"movq $$42,
|
||||
${0}","=*m"`(ptr elementtype)), 1659 symbol (`"call ${2:P}"`, direct call). x86
|
||||
templates validated by cross-emitting an object (integrated assembler accepts;
|
||||
objdump confirms 1659's direct `call` reloc). Pure additive locks. `zig build
|
||||
test` green (668 corpus, 446 unit).
|
||||
- (symbol portability) made `%[fn]` portable across arches — `renderAsmTemplate`
|
||||
auto-injects LLVM's `:c` modifier (`${N}`→`${N:c}`) for symbol (`"s"`) operands
|
||||
lacking an explicit modifier (`asmNamedIsSymbol` helper). Without it x86 renders
|
||||
`$cb` (a bad `call` target needing a hand-written `:P`); aarch64 unaffected.
|
||||
Verified `:c` ≡ `:P` for x86-64 calls (both → `R_X86_64_PLT32`). Explicit
|
||||
`%[fn:X]` still wins (escape hatch). 1659 dropped its `:P` → same plain `%[fn]`
|
||||
as aarch64 1656; both IRs regen to `${N:c}`. `zig build test` green (668 corpus,
|
||||
446 unit).
|
||||
|
||||
## Known issues
|
||||
- **0138** — RESOLVED. `@const` (address-of a `::` comptime constant) yielded a
|
||||
wild pointer (`inttoptr (i64 <value> to ptr)`). Fixed by diagnosing scalar
|
||||
`@const` in `src/ir/lower/expr.zig` `.address_of` (no storage; array/struct
|
||||
consts unaffected). Delivered the ASM "output-to-`const` rejection" for free.
|
||||
Regression `examples/1177-diagnostics-addr-of-const-rejected.sx`.
|
||||
- **0137** — `sx run` on a program with no `main` segfaults (unguarded JIT entry
|
||||
lookup, `src/target.zig:256-273`). Pre-existing, asm-independent. Filed
|
||||
`issues/0137-jit-run-no-main-segfault.md`. Does not block A.1.
|
||||
132
current/CHECKPOINT-ATOMICS.md
Normal file
132
current/CHECKPOINT-ATOMICS.md
Normal file
@@ -0,0 +1,132 @@
|
||||
# CHECKPOINT-ATOMICS — Stream A (atomics lowering)
|
||||
|
||||
Companion to [PLAN-ATOMICS.md](PLAN-ATOMICS.md). Update after every step (one step at a
|
||||
time, per the cadence rule). New corpus category: `17xx`.
|
||||
|
||||
## Last completed step
|
||||
**A.2 (CAS) — DONE** (A.2a lock + A.2b green). `compare_exchange`/`_weak` → LLVM `cmpxchg`
|
||||
(result **`?T`, null = SUCCESS**; failure carries the actual value for retry). New IR op
|
||||
`atomic_cmpxchg` + `AtomicCmpxchg{ptr, cmp, new, val_ty, success_ordering, failure_ordering,
|
||||
weak}`. `emitAtomicCmpxchg`: `LLVMBuildAtomicCmpXchg` (success/failure orderings, singleThread=0)
|
||||
→ `{T, i1}` pair; `LLVMSetWeak` for weak; `?T` result = `{ extractvalue 0 (actual),
|
||||
xor(extractvalue 1, true) }` (has_value = NOT success). comptime_vm arm does real single-thread
|
||||
CAS (read/compare/store-on-equal, build `?T`; weak == strong at comptime). Recognizer
|
||||
(`atomic_cmpxchg`/`_weak`, 6 args) — CAS restricted to INTEGER T; BOTH orderings via
|
||||
`atomicOrderingFromNode`; dual-ordering validation (failure may not be release/acq_rel nor
|
||||
stronger than success, `atomicOrderingRank`). Methods `compare_exchange`/`_weak` on `Atomic($T)`
|
||||
with comptime `$success`/`$failure: Ordering`. `examples/1702` green (CAS ok→20 / fail actual=20 /
|
||||
weak retry loop 100→105); `examples/1186` locks a rejected ordering pair; unit test `emit: atomic
|
||||
cmpxchg (strong + weak)` asserts `cmpxchg` + `cmpxchg weak`. Suite green (718/0).
|
||||
|
||||
### A.1 (RMW) — DONE (A.1a lock + A.1b green)
|
||||
`fetch_add/sub/and/or/xor` + `fetch_min/max` → LLVM `atomicrmw` (returns OLD value). New IR op
|
||||
`atomic_rmw` + `RmwKind` (no `nand`); `LLVMBuildAtomicRMW` with binop from kind, signed/unsigned
|
||||
`Min/Max` from `val_ty`. RMW restricted to INTEGER T (float fadd / pointer RMW out of scope,
|
||||
rejected loudly); all five orderings valid for RMW. comptime_vm does real single-thread RMW.
|
||||
`examples/1701` green; unit test locks `atomicrmw add` + signed `min` vs unsigned `umin`.
|
||||
|
||||
## Next step
|
||||
**A.3 — fence** (`atomic_fence($o: Ordering)` → LLVM `fence`), per PLAN-ATOMICS. No value
|
||||
result; ordering must be acquire/release/acq_rel/seq_cst (relaxed is meaningless for a fence —
|
||||
reject loudly). New IR op `atomic_fence` + dispatch/print/comptime_vm (no-op single-thread) +
|
||||
`LLVMBuildFence`. Lock-then-green cadence as before.
|
||||
|
||||
### Earlier — A.0c (guard hardening)
|
||||
Adversarial review of A.0 found two CRITICAL silent-wrong defects (raw LLVM verifier errors
|
||||
via the public intrinsics) + a latent align fallback; all fixed: scalar-kind allowlist +
|
||||
per-op ordering validity (call.zig), `val_ty` align bail (ops.zig). Locked by examples
|
||||
1130/1131. Suite green (713/0).
|
||||
|
||||
### Earlier — A.0b (green)
|
||||
Real atomic load/store emission: `LLVMBuildLoad2`/`LLVMBuildStore`
|
||||
+ `LLVMSetOrdering` + mandatory `LLVMSetAlignment`, ordering via an explicit
|
||||
sx-tag→`LLVMAtomicOrdering` switch (`llvmOrdering`). `examples/1700` green (7/42/43); IR
|
||||
shows `load atomic i64, ptr … seq_cst, align 8` + `store atomic …`. Added unit test
|
||||
`emit: atomic load/store (seq_cst, aligned)` in `emit_llvm.test.zig` (asserts `load
|
||||
atomic`/`store atomic`/`seq_cst`/`align 8`). No fragile full-module `.ir` snapshot for 1700
|
||||
(it uses `print`); the unit test is the emission-shape gate. Suite green (710 + units).
|
||||
|
||||
### Earlier — A.0a (lock commit)
|
||||
Full atomic load/store plumbing with LLVM emission deliberately bailing loudly;
|
||||
`examples/1700` locked to the bail diagnostic.
|
||||
- `library/modules/std/atomic.sx`: `Ordering` enum, `Atomic($T)` struct (`init`/`load`/
|
||||
`store`, **seq_cst-only** — see capability gap below), `atomic_load`/`atomic_store`
|
||||
`#builtin` decls. **Opt-in import**, NOT in the universal `std.sx` facade (mirrors
|
||||
`trace`) — putting `Ordering` in the prelude grew every program's type table 378→380 and
|
||||
churned 37 `.ir` snapshots; reverted.
|
||||
- IR ops `atomic_load`/`atomic_store` + `AtomicOrdering` (all 5) + structs (inst.zig);
|
||||
print arms (print.zig); comptime_vm arms reuse load/store (single-thread correct);
|
||||
recognizer `tryLowerAtomicIntrinsic` (call.zig) — const-ordering-literal guard +
|
||||
scalar-size guard, both loud; emit dispatch arms (emit_llvm.zig) → `emitAtomicLoad`/
|
||||
`emitAtomicStore` (ops.zig) currently BAIL via `comptime_failed`.
|
||||
|
||||
## A.0.5 — full ordering surface (DONE)
|
||||
`Atomic($T).load($o: Ordering)` / `store(v, $o)` — ordering is a COMPTIME value param,
|
||||
explicit (Rust-style, no default; design §4.6). `a.load(.acquire)` emits `load atomic …
|
||||
acquire`; `a.store(v, .release)` emits `store atomic … release`; `a.load(.release)` is a
|
||||
compile error (per-op validity guard fires through the method path). Recognizer
|
||||
`atomicOrderingFromNode` now resolves a comptime-bound ordering identifier via
|
||||
`comptimeIntNamed` (+ `atomicOrderingFromTag`, with the sx-Ordering ↔ IR-AtomicOrdering
|
||||
declaration-order invariant documented). 1700 migrated to explicit orderings (output
|
||||
unchanged 7/42/43). Suite green (715/0).
|
||||
|
||||
**Unblocked by three comptime-value-param commits (workers):** enum (3c4305f), tagged_union
|
||||
(d7a6857), generic-struct methods (d95ba0a). NOTE: default VALUES for comptime params on
|
||||
generic-struct methods are NOT bound (orthogonal gap — free-fn defaults work); atomics
|
||||
sidesteps it cleanly by requiring explicit ordering (matches the design). Candidate
|
||||
follow-up, not an atomics blocker.
|
||||
|
||||
## Known issues / capability gaps
|
||||
- **RESOLVED:** comptime-constant ordering propagation — landed via comptime value params
|
||||
(3c4305f / d7a6857 / d95ba0a); A.0.5 migrated the methods, no seq_cst-only legacy.
|
||||
- **Orthogonal gap (not an atomics blocker):** default VALUES for comptime params don't bind
|
||||
on generic-struct methods (free-fn defaults DO work). Atomics requires explicit ordering
|
||||
(design-aligned), so it's unaffected. Candidate future fix.
|
||||
- **Cosmetic:** an invalid ordering passed through a method (`a.load(.release)`) reports the
|
||||
diagnostic at the lib forward site (`atomic.sx`), not the user's call. Loud + correct, but
|
||||
the span could be improved by threading the call-site span. Polish.
|
||||
- **Latent (observed, not yet filed):** calling an *unrecognized* bodiless `#builtin`
|
||||
silently returns 0 / no-ops with exit 0 (that's how 1700 behaved before recognition
|
||||
landed) — a silent-fallback footgun in the generic builtin-call path, independent of
|
||||
atomics. Flag to user; candidate `issues/` entry.
|
||||
|
||||
## Decisions (Stream A specifics; surface locked in design §4.6)
|
||||
- `Atomic($T)` = pure-sx transparent 1-field struct (NO new IR type); ops = `#builtin`
|
||||
intrinsics emitted as new IR ops. Minimal compiler surface.
|
||||
- Ordering is compile-time-only (const enum literal), baked into the op as a Zig enum;
|
||||
non-literal = loud diagnostic. sx tag → LLVM ordering via explicit switch (LLVM enum is
|
||||
non-contiguous: 2/4/5/6/7).
|
||||
- Atomic load/store REQUIRE explicit alignment (`LLVMSetAlignment`) — verifier mandate.
|
||||
- Comptime VM treats atomics as ordinary load/store (single-thread ⇒ correct), not a bail.
|
||||
- **Snapshot scope corrected:** `.ir` (LLVM IR) is arch-invariant for atomics → ONE host
|
||||
`.ir` per op, not arch-gated x86/aarch64 pairs (they'd be byte-identical). Asm-level arch
|
||||
divergence + weak-memory semantics are OUT of corpus scope (stress harness, Stream C).
|
||||
|
||||
## Log
|
||||
- **carve** — wrote PLAN-ATOMICS.md + CHECKPOINT-ATOMICS.md; grounded the intrinsic path,
|
||||
switch sites, LLVM-C API (no `LLVMBuildAtomicLoad`; use `LLVMBuildLoad2`+`SetOrdering`+
|
||||
`SetAlignment`), and corrected the arch-`.ir` misconception (`sx ir` emits arch-invariant
|
||||
LLVM IR). Stream ready; A.0a is the first implementation step.
|
||||
- **A.0a** — landed lib (atomic.sx, opt-in import) + IR ops (atomic_load/atomic_store +
|
||||
AtomicOrdering) + recognizer + print/vm arms + emit BAIL; locked `examples/1700` to the
|
||||
bail diagnostic. Reverted a universal-facade wiring that churned 37 `.ir` snapshots
|
||||
(Ordering would bloat every program's type table). Suite green (710/0).
|
||||
- **A.0b** — real atomic load/store emission (LLVMBuildLoad2/Store + SetOrdering +
|
||||
SetAlignment; explicit sx→LLVM ordering switch). 1700 green (7/42/43, `load atomic …
|
||||
seq_cst, align 8`). Unit test added. Suite green (710 + units).
|
||||
- **A.0c** — guard hardening from the adversarial review: scalar-kind allowlist + per-op
|
||||
ordering validity (call.zig), val_ty align bail (ops.zig), + diagnostic examples
|
||||
1130/1131. Suite green (713/0). (comptime enum value params landed via worker 3c4305f.)
|
||||
- **A.0.5** — full ordering surface: `Atomic($T).load/store($o: Ordering)` comptime ordering
|
||||
(explicit). Recognizer resolves comptime-bound ordering via `comptimeIntNamed`. 1700
|
||||
migrated to explicit orderings (acquire/release/relaxed/seq_cst). Unblocked by
|
||||
comptime-value-param workers (3c4305f/d7a6857/d95ba0a). Suite green (715/0).
|
||||
- **A.1** — RMW: atomic_rmw op + RmwKind + recognizer (rmwKindFromName, integer-only) +
|
||||
7 fetch_* methods/intrinsics. A.1a bail-lock → A.1b real LLVMBuildAtomicRMW (signed|unsigned
|
||||
min/max). comptime_vm real RMW. 1701 + unit test. Suite green (716/0).
|
||||
- **A.2** — CAS: atomic_cmpxchg op + recognizer (dual-ordering validation) + emit (?T from
|
||||
{actual,!success}) + comptime VM. compare_exchange/_weak methods. examples 1702 + 1186.
|
||||
Review agent died; self-verified comptime↔runtime agreement, sub-8, ordering edges.
|
||||
(Commits dca396e/79895be; A.2 has_value fix folded into A.3a.)
|
||||
- **A.3** — swap (atomicrmw xchg) + fence (new atomic_fence op). A.3a bail-lock → A.3b real.
|
||||
examples 1703/1704/1187 + unit test. Stream A feature-complete. Suite green (721/0).
|
||||
1663
current/CHECKPOINT-COMPILER-API.md
Normal file
1663
current/CHECKPOINT-COMPILER-API.md
Normal file
File diff suppressed because it is too large
Load Diff
861
current/CHECKPOINT-EXTERN-EXPORT.md
Normal file
861
current/CHECKPOINT-EXTERN-EXPORT.md
Normal file
@@ -0,0 +1,861 @@
|
||||
# sx `extern`/`export` + `#foreign` retirement — Checkpoint (FFI-linkage stream)
|
||||
|
||||
Companion to `current/PLAN-EXTERN-EXPORT.md` — one merged plan: **Part A** adds
|
||||
`extern`/`export`, **Part B** migrates `#foreign` and purges `foreign`. Update after
|
||||
every commit, one step at a time per the cadence rule.
|
||||
|
||||
## Last completed step
|
||||
**Phase 9 COMPLETE — total `foreign` purge; 9.4 GATE PASSES.** **THE ENTIRE
|
||||
FFI-LINKAGE STREAM (Parts A + B, Phases 0–9) IS DONE.** Final commits: 9.0 token
|
||||
delete (`dfae690`), 9.3 src/docs/example/library comment purge (`811a280`, `e99383f`,
|
||||
`dc51c4b`, + the capital-Foreign sweep), 9.3 example filename renames + dedup
|
||||
(`b52d424`), 9.3/9.4 issues/*.md purge + GATE (`b9cfe25`).
|
||||
- **9.0:** deleted the `hash_foreign` token entirely (token/lexer/parser/lsp + the lex
|
||||
test); `#foreign` now → a generic "expected ';'" parse error (accepted UX cost);
|
||||
deleted the obsolete 1176 rejection test.
|
||||
- **9.1/9.2:** all internal identifiers renamed (linkage→`extern`/`is_extern`,
|
||||
runtime-class→`Runtime*`/`runtime_*` per Decision 5, `foreign_path`→`runtime_path`
|
||||
across the build-hook boundary); `foreign_expr` node eliminated.
|
||||
- **9.3:** purged every `foreign` COMMENT (src caps + lowercase, examples, docs incl.
|
||||
the obsolete inline-asm Deviation 6, editors/vscode grammar) + renamed all 10
|
||||
`*-foreign*` example files (+ companions/expected/refs) to extern/runtime names
|
||||
(dedup'd 1218↔1229, removed orphan 1620 dir) + rewrote 20 issues/*.md writeups +
|
||||
renamed issues/0043.
|
||||
- **9.4 GATE:** `grep -rniIE 'foreign' src/ library/ examples/ issues/ docs/ editors/
|
||||
specs.md readme.md CLAUDE.md` → **0**, excluding only the legitimate keeps:
|
||||
`SQLITE_CONSTRAINT_FOREIGNKEY` (SQLite API const) + vendored `library/vendors/sqlite/
|
||||
c/*` (upstream third-party C). No `foreign`-named files in the tree (node_modules +
|
||||
.sx-tmp are gitignored third-party/scratch). Suite green (644 corpus / 443 unit, 0
|
||||
failed).
|
||||
|
||||
### Prior: Phase 9.3 — text/comment purge (src + docs + example comments) (commits
|
||||
`e99383f` docs, `dc51c4b` src, + examples purge STAGED pending a classifier outage —
|
||||
commit message ready; `git commit` the staged `examples/` changes when Bash is back).
|
||||
`foreign` is now purged from: **all `src/` comments** (reworded to extern/runtime-class;
|
||||
fixed 2 user-facing diagnostics — the type-annotation parse error no longer lists
|
||||
`#foreign`, and the Android no-`#jni_main` help shows `#jni_class(…) extern`), **specs/
|
||||
readme/CLAUDE** ("Foreign Function Interface"→"C Interop", etc.), and **all example .sx
|
||||
comments** (1219 stdout labels Foreign→Extern, snapshot regenerated). Suite green
|
||||
(646/444) throughout; snapshot-neutral except the intentional 1219 regen.
|
||||
|
||||
**What still contains `foreign` (the analyzed keep-list + the not-yet-done):**
|
||||
- **KEEP (gate-exempt):** `src/` `hash_foreign` token + lexer entry + `lex hash_foreign`
|
||||
test (`#foreignx`) + the 4 parser rejection messages ("`#foreign` has been removed…");
|
||||
`1176-diagnostics-foreign-removed.sx` (its `#foreign` decl + comments ARE the rejection
|
||||
test); `SQLITE_CONSTRAINT_FOREIGNKEY` + vendored `library/vendors/sqlite/c/*`.
|
||||
- **NOT YET DONE:** example FILENAMES (`*-foreign*.sx` + the `0729`/`1205`/`1218`/`1219`/
|
||||
`1306`/`1318`/`1216`/`1217` families) and their `#import`/`#include`/`#source` path refs
|
||||
+ `expected/` files — needs a git-mv rename step; and **`issues/*.md`** (~20 writeups).
|
||||
|
||||
### Prior: Phase 9.1 + 9.2 — internal IDENTIFIER purge COMPLETE (commits 9.1a `b838f63`,
|
||||
9.1b `b78e7dd`, 9.1c `cd14794`, 9.1d `7ffdc7d`, 9.2a `3354446`, 9.2b `5c8af6e`,
|
||||
9.2b-fix `a15a868`, 9.2c `d27be42`, 9.2d `8cca3b9`). **Every `foreign` IDENTIFIER in
|
||||
`src/` is renamed** — the only `foreign` left in `src/` is COMMENTS + the kept token
|
||||
(`hash_foreign` + its `#foreignx` lexer-boundary test) + the rejection-message string.
|
||||
Suite green (646/444) at every commit.
|
||||
- **9.1d** eliminated the `foreign_expr` AST node: migrated `c_import.zig` auto-synth
|
||||
to the extern shape, deleted the node + `ForeignExpr` + all readers.
|
||||
- **9.2a/b/c/d** ran the runtime-class family rename (Decision 5 → `Runtime*`):
|
||||
types `ForeignClassDecl`→`RuntimeClassDecl` etc.; fns `parseForeignClassDecl`→
|
||||
`parseRuntimeClassDecl`, `lowerForeignMethodCall`→`lowerRuntimeMethodCall`, …; state
|
||||
`foreign_class_map`→`runtime_class_map`, `foreign_class_decl` variant→
|
||||
`runtime_class_decl`; the extern-ref validators → `Extern` (linkage, `checkExternRefs`);
|
||||
the reference flag → **`is_extern`** (per user: reuse existing terminology, not a new
|
||||
`is_reference`); and `foreign_path`→`runtime_path` COUPLED across the hook boundary
|
||||
(build.sx accessor `jni_main_runtime_path_at` + the registered hook string +
|
||||
bundle.sx + specs.md), with 37 `.ir` snapshots regenerated for the renamed
|
||||
`@BuildOptions.jni_main_runtime_path_at` declare stub (symbol-name change only).
|
||||
- **9.1a/b/c** (linkage): 5 collision-free renames (callExtern, …); "foreign symbol"
|
||||
diagnostic + panic → "extern symbol" (1172 regen); deleted dead VarDecl legacy fields.
|
||||
|
||||
### Prior: Phase 9.1 (partial) — internal linkage-identifier purge (commits `b838f63` 9.1a,
|
||||
`b78e7dd` 9.1b, `cd14794` 9.1c). **PHASE 9 STARTED.** Decision 6 = PURGE EVERYTHING,
|
||||
scoped (user, 2026-06-15): purge `foreign` from **all `.sx` files + all documentation +
|
||||
all our Zig (`src/`)**, analyzing each grep hit — **legitimate keeps stay**
|
||||
(SQLITE_CONSTRAINT_FOREIGNKEY + other SQLite API constant names, vendored
|
||||
`library/vendors/sqlite/c/*`, `1176-diagnostics-foreign-removed.sx` [the rejection test
|
||||
MUST contain `#foreign`], the parser rejection-message string + `hash_foreign` token
|
||||
[kept so `#foreign` keeps its friendly deprecation error]).
|
||||
- **9.1a** (`b838f63`): 5 collision-free linkage renames — `callForeign`→`callExtern`,
|
||||
`marshalForeignArg`→`marshalExternArg`, `dedupeForeignSymbol`→`dedupeExternSymbol`,
|
||||
`foreign_name_map`→`extern_name_map`, `is_foreign_c_api`→`is_extern_c_api`.
|
||||
- **9.1b** (`b78e7dd`): the "foreign symbol already bound" diagnostic (decl.zig) +
|
||||
resolveFuncByName panic (call.zig) → "extern symbol". Intentional 1172 regen.
|
||||
- **9.1c** (`cd14794`): **deleted** the dead `VarDecl.is_foreign`/`foreign_lib`/
|
||||
`foreign_name` fields (the global `#foreign` path rejects → write-dead; 3 coalescing
|
||||
readers in decl.zig simplified to `vd.extern_name`/`vd.is_extern`).
|
||||
All snapshot-neutral except the intentional 1172 regen; suite green (646/444).
|
||||
|
||||
**COLLISION ANALYSIS (done — drives the rest of 9.1/9.2):**
|
||||
- `is_foreign` lives on FnDecl?(no — flipped to `extern_export`), **VarDecl (deleted in
|
||||
9.1c)**, and **ForeignClassDecl (ast.zig:903 — STILL LIVE**, distinguishes runtime-class
|
||||
reference vs define; renamed in 9.2, not 9.1).
|
||||
- `is_extern`/`extern_lib`/`extern_name` already exist (VarDecl + IR insts) — so the
|
||||
old `foreign_*` linkage names could NOT be blind-renamed onto them; 9.1c deleted the
|
||||
dead VarDecl trio instead of renaming.
|
||||
- `foreign_expr` (25) is **still BUILT by `c_import.zig` auto-synthesis** (`#import c
|
||||
{#include}` synthesizes fn bodies as `foreign_expr`). To eliminate it: migrate that
|
||||
synth path to build the extern shape (empty-block body + `extern_export = .extern_`),
|
||||
exactly the Phase 5.0 fn-body flip but for auto-synth — THEN delete the `foreign_expr`
|
||||
node + all readers. This is the last 9.1 item.
|
||||
|
||||
### Prior: Phase 8 — CUTOVER: parser hard-rejects `#foreign` (`feat!` commit `3811311`,
|
||||
preceded by the 8.0 xfail `8180faf` + 3 pre-cutover `refactor`s `2cce6a3`/`720556b`/
|
||||
`d132aab`). **PHASE 8 COMPLETE.** The prefix `#foreign` linkage directive is removed:
|
||||
all four parse sites (const-with-type 316, data global 425, fn body 2065, runtime-class
|
||||
prefix via caller 260) reject it with the migration message *"`#foreign` has been
|
||||
removed; use the postfix `extern` (import) / `export` (define) linkage keyword
|
||||
instead"*; added a span-aware `failAt` for the runtime-class case (the lookahead
|
||||
consumes the token before the reject decision). New example **1176**
|
||||
(`diagnostics-foreign-removed`) pins it. **Pre-cutover migrations** (all green,
|
||||
behavior-preserving): the 7 identity `ffi-foreign-*` test DECLS (`2cce6a3`), the two
|
||||
keyword-neutral diagnostic tests 1172 + 1228 with intentional snapshot regens
|
||||
(`720556b`), and the 4 multi-file example companions Phase 7 missed (0729/a+b, 1617/c,
|
||||
1623/mod — `d132aab`). **Deleted** obsolete tests 1174 (`#foreign`+postfix conflict, now
|
||||
unreachable) + 1620 (`#foreign nosuchunit`, superseded by extern twin 1231), the GATE
|
||||
A→B unit test + `lowerSrcToIr` helper (nothing left to compare), and converted the
|
||||
in-source `parse void function with foreign body` parser test to postfix `extern`.
|
||||
specs.md + readme.md document `extern`/`export` as the sole C-linkage surface. Suite
|
||||
green (646 corpus / 444 unit, 0 failed).
|
||||
|
||||
### Prior: Phase 7.4 — migrate straggler examples `#foreign`→`extern` (`refactor` commit
|
||||
`1a8991a`). **PHASE 7 MIGRATABLE WORK COMPLETE (7.1–7.4 done).** Migrated 16 fn/global
|
||||
examples across categories (0415/0602/0603/1024/1025/1605/1607-1609/1611/1616/1619/
|
||||
1622/1628/1635/1636). Marker'd ones corpus-validated; the 3 unmarked uikit importers
|
||||
(1607/1608/1616) verified byte-identical via `sx ir` probes. Empty snapshot diff; suite
|
||||
green (647/444).
|
||||
|
||||
**Phase 7 net result:** every example that uses `#foreign` *incidentally* (FFI plumbing,
|
||||
output-preserving) is now on `extern`/`export`. The **24 files still holding `#foreign`
|
||||
are exactly the intended keep-list**, all deferred to the Phase 8 cutover:
|
||||
- **`foreign`-asserting diagnostics** (migrating changes a snapshot): 1172, 1174, 1219
|
||||
(stdout label), 1228 (equivalence test), 1620.
|
||||
- **Identity `ffi-foreign-*` feature tests** (real decls; rename/dedup at cutover):
|
||||
1205-global, 1205-global-helper, 1207, 1218, 1219, 1306, 1318.
|
||||
- **Comment-only / provenance prose** (decls=0; `#foreign` only in comments): 0716, 0729,
|
||||
1216, 1223, 1224, 1225, 1229, 1230, 1231, 1332, 1348, 1349, 1426, + issues/0030.
|
||||
**Lesson (7.3):** the robust class-prefix transform is the GENERAL form
|
||||
`s/#foreign\s+#(\w+)\((\"[^\"]*\")\)\s*\{/#$1($2) extern {/` — 1417 also used
|
||||
`#jni_interface`/`#objc_protocol`/`#swift_class`/`#swift_struct`/`#swift_protocol`, and a
|
||||
`#(objc|jni)_class`-only regex left `extern` in *prefix* position → parse error. All such
|
||||
directives accept the postfix modifier (probed). Bare defined `#objc_class`/`#jni_class`
|
||||
examples (no `#foreign`) were left untouched — not a purge target (define→export is an
|
||||
optional consistency pass, deferrable).
|
||||
|
||||
### Prior: Phase 7.1 — migrate incidental 12xx ffi examples (`refactor` commit `731fb8d`).
|
||||
Migrated 12 plain-C examples (1200/1206/1209-1215/1220/1221/1222); established the
|
||||
keep-list policy above. Phase 7.2 (`a68f7c2`): 18 13xx obj-c examples (prefix→postfix
|
||||
classes). Phase 7.3 (`2888f6f`): 13 14xx jni examples incl. 1417 multi-runtime.
|
||||
|
||||
### Prior: Phase 6.5 — migrate `gpu/` `#foreign`→`extern`; `library/` now `#foreign`-free
|
||||
(`refactor` commit `32a7628`). **PHASE 6 COMPLETE.** Final batch: gpu/gles3.sx
|
||||
(eglGetProcAddress + 1 comment) + gpu/metal.sx (MTLCreateSystemDefaultDevice), bare
|
||||
fn markers → `extern`. Verified byte-identical `sx ir` on importers 1610 (gles3) +
|
||||
1606 (metal). **Zero `#foreign` remains anywhere under `library/`** — verified by
|
||||
`grep -rln '#foreign' library/` → no matches. Suite green (647 corpus / 444 unit, 0
|
||||
failed).
|
||||
|
||||
### Prior: Phase 6.4 — migrate `ffi/` `#foreign`→`extern` (`refactor` commit `666a2e2`).
|
||||
objc/objc_block/raylib/sdl3/wasm (~51 sites): fn markers + objc.sx's 2 import runtime
|
||||
classes (prefix→postfix `extern`). objc + objc_block validated by the 50 marked 13xx
|
||||
corpus examples (incl. import classes 1300/1301 + defined classes 1339/1349);
|
||||
raylib/ffi-sdl3/wasm verified by byte-identical `sx ir` probes pre/post. Empty snapshot
|
||||
diff; suite green.
|
||||
|
||||
### Prior: Phase 6.3 — migrate `std/` `#foreign`→`extern` (`refactor` commit `59f90d2`).
|
||||
Pure source rename across 11 std modules (~60 sites):
|
||||
cli/core/fmt/fs/log/net.kqueue/process/socket/thread/time/trace. All fn-decl markers
|
||||
— bare `#foreign;`, `#foreign libc;`/`#foreign tlib;` (LIB ref), and
|
||||
`#foreign libc "csym";` (LIB+rename) → the same `extern …` tail (`extern` carries the
|
||||
identical `[LIB] ["csym"]` axis); plus 2 stale comment mentions (fmt/fs). No class
|
||||
forms in std. These modules ARE host-corpus-exercised → empty snapshot diff is direct
|
||||
validation. Suite green (647 corpus / 444 unit, 0 failed). Remaining Phase 6 batches:
|
||||
6.4 ffi (~50, has runtime classes), 6.5 remainder.
|
||||
|
||||
### Prior: Phase 6.2 — migrate `platform/` `#foreign`→`extern`/`export` (`refactor` commit
|
||||
`2cd5d7b`). Pure source rename across uikit/android/android_jni/sdl3 (~64 sites):
|
||||
30 fn `… #foreign;`→`… extern;`; 34 import runtime classes
|
||||
`#foreign #objc_class/#jni_class("X") {`→`#…_class("X") extern {` (prefix→postfix);
|
||||
4 defined `Sx*` obj-c classes `#objc_class("X") {`→`… export {`. Behavior-preserving;
|
||||
empty snapshot diff. **Verification (these modules are largely uncompiled by the
|
||||
host corpus** — bundle examples import `bundle.sx`, not the runtime modules; android.sx
|
||||
only compiles under `OS==.android`): byte-identical `sx ir` on uikit importers 1610 +
|
||||
1606 (which DO compile uikit incl. the 4 defined `Sx*` classes on host) and an sdl3
|
||||
direct-import probe; android.sx verified by an identical 4-error dedup set (host
|
||||
pthread clashes — the keyword-neutral "foreign symbol already bound" dedup message is
|
||||
unchanged, and the probe parsed all migrated `extern` jni classes + EGL fns cleanly
|
||||
before hitting them). Suite green (647 corpus / 444 unit, 0 failed). Remaining Phase 6
|
||||
batches: 6.3 std (~60), 6.4 ffi (~50), 6.5 remainder.
|
||||
|
||||
### Prior: Phase 6.1 — migrate `vendors/sqlite` `#foreign`→`extern` (`refactor` commit
|
||||
`410a52e`). **PART B PHASE 6 STARTED.** Pure source rename: all 97
|
||||
`sqlite3_* … #foreign sqlib "csym";` fn decls → `extern sqlib "csym";` (+ the one
|
||||
stale header-comment reference, line 9). The `extern_lib` axis references the `sqlib`
|
||||
`#import c` unit identically to `#foreign sqlib`, so IR/output is byte-identical —
|
||||
empty snapshot diff (only `sqlite.sx` changed), and example 1624
|
||||
(`vendor-sqlite-module`) stdout byte-unchanged. Suite green (647 corpus / 444 unit,
|
||||
0 failed).
|
||||
|
||||
### Prior: Phase 5.1 — annotate A→B gate post-flip + add fn-rename case (`test` commit
|
||||
`93e7b6f`). **PHASE 5 COMPLETE → PART B Phase 5 done.** The A→B gate
|
||||
(`lower.test.zig`) already asserted `#foreign` ≡ `extern`/`export` byte-identical IR
|
||||
for fn / global / Obj-C class; post-Phase-5.0 the fn-decl + data-global paths build
|
||||
the SAME extern-named AST, so cases 1/2 are now STRUCTURALLY identical (guaranteed by
|
||||
construction, not coincidence). Annotated the gate header to record this and keep it
|
||||
as a regression tripwire (catches a future reader re-diverging the spellings, or a
|
||||
revert of the flip); case 3 (runtime class) stays behaviorally — not structurally —
|
||||
equal via the single `is_foreign_eff` field. Added a fn-rename case (case 2b,
|
||||
`extern_name` axis: `c_abs` → `"abs"`) to broaden coverage beyond bare import
|
||||
(verified IR-identical via `sx ir` probe before adding). Test-only, no snapshot churn.
|
||||
Suite green (647 corpus / 444 unit, 0 failed).
|
||||
|
||||
### Prior: Phase 5.0 — fn-decl `#foreign` body-marker FLIP (`refactor` commit `6b94bb6`).
|
||||
**PHASE 5.0 PARSER ROUTING COMPLETE.** The fn-body `#foreign [LIB] ["csym"]` marker
|
||||
now builds the SAME extern AST postfix `extern` produces (`extern_export = .extern_`
|
||||
+ `extern_lib`/`extern_name` + empty-block body) instead of a `foreign_expr` body.
|
||||
Behavior-preserving — all four prereqs (visibility, variadic, plain-free, lib-ref)
|
||||
ensure every downstream reader coalesces `is_foreign` with `extern_export`, so IR +
|
||||
runtime are byte-identical (full corpus + A→B gate green). Decision 7 churn realised:
|
||||
example 1620's lib-ref error flips "#foreign library" → "extern library" (the only
|
||||
snapshot moved; hand-edited, not regen). Parser unit test updated to assert the extern
|
||||
shape. Spot-checked 1219/1218/0729 (foreign rename / cvariadic / same-name) end-to-end.
|
||||
**All four `#foreign` parser paths now resolved:** global (`e5ddfbe`) + fn-body
|
||||
(`6b94bb6`) flipped onto extern; const-with-type is dead (deferred); runtime-class is
|
||||
already coalesced (`is_foreign_eff`). `c_import.zig` auto-synthesis STILL emits
|
||||
`foreign_expr` bodies (Phase 6+), so both shapes coexist — every reader stays dual.
|
||||
Suite green (647 corpus / 444 unit, 0 failed).
|
||||
|
||||
### Prior: Phase 5.0 prereqs 3 & 4 — plain-free classification + extern lib-ref validation
|
||||
(plain-free: xfail `2706521` → fix `3c94c14`; lib-ref: xfail `38c3240` → fix
|
||||
`ad6aed3`). Two MORE extern/#foreign divergences found while de-risking the fn-path
|
||||
flip, both now closed. **FOUR prereqs total done — the fn-decl flip fully de-risked.**
|
||||
- **Prereq 3 (plain-free):** `isPlainFreeFn`/`isPlainFreeFnDecl` (resolver.zig:178,
|
||||
generic.zig:815) excluded a `#foreign` body but classified an empty-block `extern`
|
||||
fn as a plain free fn — so existing extern fns were wrongly counted in the bare-call
|
||||
ambiguity verdict (example: two same-name `extern libc "abs"` authors errored
|
||||
ambiguous, while the `#foreign` twin 0729 compiles). Both predicates now also
|
||||
exclude `extern_export == .extern_`; `export` (real body) stays plain-free. Example
|
||||
**1230**.
|
||||
- **Prereq 4 (lib-ref validation):** `checkForeignRefs` (c_import.zig) validated only
|
||||
`foreign_expr.library_ref`, so a bogus `extern nosuchunit "abs"` compiled silently
|
||||
while `#foreign nosuchunit` errors (1620). Now reads the lib ref from EITHER spelling
|
||||
and names the surface keyword in the diagnostic (so 1620 stays byte-unchanged).
|
||||
Example **1231**.
|
||||
- Two OTHER classifying sites probed and found BENIGN for extern (no flip prereq):
|
||||
namespace/qualified dispatch (`registerQualifiedFn` decl.zig:2208, namespace gate
|
||||
call.zig:729) — a namespaced `extern` fn resolves identically to its `#foreign` twin
|
||||
(probe: `cm.c_abs(-9)` → 9 both ways; the registered qualified alias resolves to the
|
||||
same extern symbol).
|
||||
|
||||
### Prior: Phase 5.0 prereq — extern C-variadic tail (xfail `9a2c78d` → fix `0fdc821`) — the SECOND deferred fn-path prerequisite. **BOTH original fn-path prereqs done.** The C-variadic `...` handling was keyed on the `#foreign` (`foreign_expr`)
|
||||
body shape at two sites — the `is_variadic` drop in `declareFunction`
|
||||
(`decl.zig:2097`) and the call-site early-out in `packVariadicCallArgs`
|
||||
(`pack.zig:302`). A variadic `extern` therefore kept its trailing slice param and
|
||||
slice-packed the extras → garbage at the C ABI (probe: `sum_ints(3,10,20,30)` →
|
||||
53316585, not 60). Both gates now also fire for `extern_export == .extern_`, so a
|
||||
variadic `extern` drops the `..args: []T`, sets `is_variadic`, and passes extras
|
||||
through the C `...` slot with default argument promotion — byte-identical to its
|
||||
`#foreign` twin. New example **1229** (`1229-ffi-extern-cvariadic`, JIT `#source`,
|
||||
int-sum + double-avg). Suite green (645 corpus / 444 unit, 0 failed).
|
||||
|
||||
### Prior: Phase 5.0 prereq — visibility-gate equivalence (xfail `717c35d` → fix `7d8ba1a`) — the first of the two deferred fn-path prerequisites.
|
||||
The non-transitive C-import visibility gate (`isVisible(.c_import_bare)`,
|
||||
`decl.zig:2249`) used to recognise only the legacy `#foreign` body shape; a bare
|
||||
`extern` fn (empty-block body + `extern_export == .extern_`) escaped the gate via
|
||||
the `body != foreign_expr → return true` arm and was caught only by the general
|
||||
`isNameVisible` gate — yielding the generic "not visible" wording instead of the
|
||||
C-specific "C function not visible; add #import" one. Now BOTH lib-less spellings
|
||||
route to `visibleOverEdges`, and a library-bound `extern LIB` (like `#foreign LIB`)
|
||||
stays unconditionally visible — so a future fn-decl `#foreign`→`extern` migration
|
||||
is byte-identical at this gate. New cross-module example **1228**
|
||||
(`examples/1228-ffi-extern-c-non-transitive`, main → b → c) pins the equivalence:
|
||||
referencing c's lib-less `#foreign` AND `extern` twins transitively both produce
|
||||
the identical C-specific diagnostic. Suite green (644 corpus / 444 unit, 0 failed).
|
||||
**Empirical finding** (probe, not yet acted on): the bare-extern twin was NEVER a
|
||||
silent visibility hole — the general `isNameVisible` gate already denied it; only
|
||||
the *diagnostic wording* diverged. The fix aligns the wording + gate ownership.
|
||||
|
||||
### Prior: Phase 5.0 (global path) (`refactor` lock, commit `e5ddfbe`) — **PART B STARTED.**
|
||||
First of the four `#foreign` parser paths migrated onto the extern AST: the
|
||||
data-global form `name : T #foreign [lib] ["csym"];` now builds the same
|
||||
extern-named `VarDecl` (`is_extern`/`extern_lib`/`extern_name`) that postfix
|
||||
`extern` already produces, instead of `is_foreign`/`foreign_lib`/`foreign_name`.
|
||||
Behavior-preserving — lowering coalesces both forms identically
|
||||
(`decl.zig:1119,1127,1141`), so zero snapshot churn. The fn-decl, const-with-type,
|
||||
and runtime-class `#foreign` paths still build the legacy AST.
|
||||
|
||||
### Prior: Phase 4 (green) — **PHASE 4 COMPLETE → PART A DONE; GATE A→B LOCKED.** Four pieces:
|
||||
(1) **GATE A→B unit test** (`lower.test.zig`, `lowerSrcToIr` helper + "GATE A→B" test) —
|
||||
asserts `#foreign` and `extern`/`export` lower to byte-identical printed IR for a sample
|
||||
fn, data global, and Obj-C runtime class. This is the hard gate: Part B may not start
|
||||
migrating `#foreign` until it's green. Verified live (negative-probe: mutating one side
|
||||
fails the assertion). (2) **Diagnostic — `#foreign` + postfix conflict** (1174): prefix
|
||||
`#foreign` combined with postfix `extern`/`export` on an aggregate is now a clean parse
|
||||
error (was a confusing internal "compiler bug" during class synthesis). (3) **Diagnostic
|
||||
— `extern`+`export` mutual exclusion** (1175): both keywords on one fn decl is a clean
|
||||
error (was bare "expected ';'"). (4) **Docs**: `specs.md` + `readme.md` document the three
|
||||
`extern`/`export` axes (fns, globals, aggregates) alongside `#foreign` (which stays
|
||||
documented until the Part B cutover). Suite green (643 corpus / 444 unit, 0 fail).
|
||||
NOTE: `extern`+`callconv` redundancy needs no diagnostic — `callconv(.c) extern` is a
|
||||
harmless dup (both `.c`), and any non-`.c` callconv already errors on its own.
|
||||
|
||||
### Prior: Phase 3.1 (green) — **PHASE 3 COMPLETE.** Postfix `extern`/`export` on `#objc_class`/
|
||||
`#jni_class` aggregates fully works. `parseForeignClassDecl` now parses an optional
|
||||
`extern`/`export` modifier in the slot **between** the `("X")` directive args and the `{`
|
||||
body (`parser.zig:~1409`): `extern`→`is_foreign_eff = true` (reference an existing runtime
|
||||
class, == legacy `#foreign`); `export`→`is_foreign_eff = false` (define + register a new sx
|
||||
class, == bare `#objc_class` with no `#foreign`). The modifier maps straight onto the same
|
||||
`is_foreign` decision the prefix `#foreign` already fed the node, so **no `objc_class.zig`/
|
||||
lowering change was needed** — the new surface reuses the existing reference-vs-define path.
|
||||
Examples: **1348** (objc `extern` import, dispatches `NSObject.alloc().init()` → green via
|
||||
JIT), **1349** (objc `export` defined class, `SxBar.alloc()`/`bump`/`get` → `counter: 2`),
|
||||
**1426** (jni `extern` import, parse-only `parse-only ok`). Suite green (641 corpus / 443
|
||||
unit, 0 fail).
|
||||
|
||||
### Prior: Phase 2.2 (green) — **PHASE 2 COMPLETE.** `export` (define + expose) fully works:
|
||||
external linkage + C ABI + no sx ctx + force-lowered root + optional `"csym"` rename.
|
||||
All four export-gap conditions filled in `decl.zig`: (i) `.external` linkage for
|
||||
`extern_export == .export_` on both define paths (`lowerFunctionBodyInto`,
|
||||
`lowerFunction`); (ii) C-ABI promotion on the define paths + `declareFunction` stub cc;
|
||||
(iv) `funcWantsImplicitCtx` returns false for any non-`.none` modifier; **force-lower**:
|
||||
`export` fns are lowering roots in `lowerMainAndComptime` (else an uncalled export fn
|
||||
stays a bodiless `declare`); (iii) `export … "csym"` declares the stub under the C name
|
||||
+ `lazyLowerFunction` promotes the body into it via `foreign_name_map`. Examples **1226**
|
||||
(bare export, C calls `sx_square` → 37/82) + **1227** (`export "triple_c"`, C calls
|
||||
`triple_c` → 22) green via the new **AOT corpus mode**. Suite green (638 corpus / 443
|
||||
unit, 0 fail).
|
||||
|
||||
**AOT corpus mode + run_examples.sh retired.** C→sx-by-name can't link under the
|
||||
corpus's `sx run` JIT mode (a JIT-resident symbol is invisible to a dlopen'd C dylib's
|
||||
flat-namespace lookup), so an `expected/<name>.aot` marker switches an example to a
|
||||
`sx build` + execute flow. The standalone `tests/run_examples.sh` was deleted —
|
||||
`zig build test` is now the sole corpus runner (verify-step.sh + CLAUDE.md updated).
|
||||
|
||||
## Current state
|
||||
Syntax: bare `extern`/`export`, postfix after `callconv(.c)`, `extern ⇒ callconv(.c)`.
|
||||
**Decision 4 revised** (user 2026-06-14): `extern` carries an optional `LIB`+`"csym"`
|
||||
axis (`extern_lib`/`extern_name`) like `#foreign`; the `#library` decl + build-flag
|
||||
linking stays separate. **`extern` (PHASE 1) + `export` (PHASE 2) FULLY WORKING.**
|
||||
extern: functions — bare (`f :: (…) -> R extern;`) AND renamed (`extern [LIB] "csym"`);
|
||||
data globals — bare + renamed. export: functions — bare (`f :: (…) -> R export {…}`)
|
||||
AND renamed (`export "csym"`); external linkage, C ABI, no ctx, force-lowered as a root.
|
||||
All behavior-equivalent to the matching `#foreign` form. `extern_lib` is parsed + stored
|
||||
but is a *reference* only — actual linking stays the `#library`/build-flag axis.
|
||||
**Aggregates DONE (Phase 3)**: postfix `extern`/`export` on `#objc_class`/`#jni_class`
|
||||
(reference vs define+register). **Interplay/diagnostics/docs DONE (Phase 4)** + the
|
||||
**A→B GATE IS LOCKED** (`#foreign` ≡ `extern`/`export` IR for fn/global/class). **PART A
|
||||
COMPLETE.** Part B `foreign` footprint to purge: 643 lines / ~57 identifiers in `src/` +
|
||||
~28 doc lines. End-state invariant: **zero `foreign`** (Phase 9.4 gate). Examples: 1223
|
||||
(extern bare fn), 1224 (extern fn rename), 1225 (extern bare global), 1226 (export bare fn,
|
||||
AOT), 1227 (export fn rename, AOT), 1348 (objc extern class), 1349 (objc export class), 1426
|
||||
(jni extern class), 1174/1175 (interplay diagnostics).
|
||||
|
||||
## Next step
|
||||
**NONE — the FFI-linkage stream is COMPLETE.** `extern`/`export` fully replace
|
||||
`#foreign`; the keyword is rejected; zero `foreign` remains in the gated tree (Parts
|
||||
A + B, Phases 0–9 all done; the 9.4 gate passes). This stream can be archived.
|
||||
|
||||
Follow-ups (both DONE 2026-06-15, post-stream polish):
|
||||
- ✅ Added `extern`/`export` to the editors/vscode tmLanguage keyword list as a
|
||||
`storage.modifier.sx` pattern (`editors/vscode/syntaxes/sx.tmLanguage.json`).
|
||||
- ✅ Dropped the vestigial `RuntimeClassPrefix.is_extern` field +
|
||||
`parseRuntimeClassDecl`'s `is_extern` param (always-false dead path; the postfix
|
||||
`extern`/`export` keyword is the sole reference-vs-define decider). Suite green
|
||||
(644 corpus / 442 unit, 0 failed).
|
||||
|
||||
--- (historical: the finish-Phase-9 plan, now done) ---
|
||||
**PART B — finish Phase 9: example FILENAME renames + `issues/*.md` + 9.0/9.4.**
|
||||
(All `src/` identifiers + AST node + all comments/docs/example-comments are DONE.)
|
||||
|
||||
0. **FIRST: commit the staged `examples/` comment purge** (a classifier outage blocked
|
||||
the commit; changes are `git add`ed). Message: "refactor(ffi-linkage): Phase
|
||||
9.3-examples — purge 'foreign' from example .sx comments".
|
||||
1. **Example filename rename** (git-mv step, snapshot-careful): rename the `*-foreign*`
|
||||
example files to extern/runtime names and update every `#import`/`#include`/`#source`
|
||||
ref + the `expected/<name>.*` companions. Families: `0729-modules-flat-same-name-foreign`
|
||||
(+ `/a.sx`,`/b.sx` dir), `1205-ffi-foreign-global`(+`-helper`), `1207-ffi-foreign-global-from-helper`,
|
||||
`1216-ffi-08-foreign-in-method`(+`.h`/`.c`), `1217-ffi-09-foreign-result-chain`(+`.h`/`.c`),
|
||||
`1218-ffi-foreign-cvariadic`(+`.c`), `1219-ffi-foreign`, `1306-ffi-objc-foreign-class-chained-dispatch`,
|
||||
`1318-ffi-objc-property-foreign`. ⚠ A renamed file with an `.ir`/`.stderr` snapshot that
|
||||
echoes its own path will need that snapshot regenerated (intentional). Pick new names
|
||||
that drop "foreign" (e.g. `…-extern-global`, `…-extern-in-method`, `…-runtime-class-chained-dispatch`).
|
||||
NOTE: keep `1176-diagnostics-foreign-removed.sx` name (it's the rejection test — fine to keep "foreign").
|
||||
2. **issues/*.md** (~20) — rewrite writeup prose `#foreign`/`foreign`→`extern`/`runtime-class`.
|
||||
2b. **`docs/*.md`** — ALSO in the gate scope (was missed; the gate areas are now
|
||||
`src/ library/ examples/ issues/ docs/ specs.md readme.md CLAUDE.md`). `docs/debugger.md`
|
||||
referenced the renamed `callForeign` (fixed → `callExtern`, UNCOMMITTED with the staged
|
||||
batch); sweep all of `docs/` for stale renamed-identifier refs + `foreign` prose.
|
||||
3. **9.0 surface decision — RATIFIED (user, 2026-06-15): DELETE the `hash_foreign` token.**
|
||||
The user explicitly flagged token.zig:121 + lsp/server.zig:1693 "this also needs to
|
||||
go" — total purge, accept `#foreign`→generic error (no friendly migration hint). This
|
||||
is the LAST src change; it is load-bearing → needs a build + test + 1176 regen (do it
|
||||
when mutating Bash is back). Steps:
|
||||
- token.zig: remove `hash_foreign` enum (121).
|
||||
- lexer.zig: remove the `.{ "#foreign", Tag.hash_foreign }` map entry (91), drop
|
||||
`#foreign` from the directive-list comment (72), DELETE the `lex hash_foreign` test
|
||||
(626-631, incl. `#foreignx`).
|
||||
- parser.zig: remove the 4 `self.current.tag == .hash_foreign` rejection sites (268
|
||||
caller / 327 / 419 / 2024) + their messages, AND the 2 lookahead refs (`hasFnBody…`
|
||||
~3658 + ~3676). ⚠ Decide what `#foreign` lexes to with no keyword entry (likely an
|
||||
error/unknown-directive token) and confirm the parser surfaces a sane error.
|
||||
- lsp/server.zig: remove the `.hash_foreign,` arm (1693).
|
||||
- **1176-diagnostics-foreign-removed**: its expected stderr is the now-deleted
|
||||
"`#foreign` has been removed…" message → it WILL change. Regen 1176's snapshot to
|
||||
whatever the generic post-deletion error is (intentional), OR delete 1176 entirely
|
||||
(its purpose — a friendly rejection — no longer exists). Recommend: keep 1176 as a
|
||||
"`#foreign` is no longer a directive" regression, regen its snapshot. NOTE: after
|
||||
this, 1176 may still contain `#foreign` in its SOURCE (the rejected token) — that's
|
||||
the only legitimately-remaining `foreign` in `.sx`, OR rename/rework it to avoid even
|
||||
that if the gate must be absolute.
|
||||
4. **9.4 gate** — `grep -rniIE 'foreign'` over `.sx` + docs + `src/` → 0 (no keep-list
|
||||
left except possibly 1176's source token + `SQLITE_CONSTRAINT_FOREIGNKEY` + vendored C).
|
||||
|
||||
--- (historical: the prose-purge plan, now mostly done) ---
|
||||
**PART B — finish Phase 9: the COMMENT / DOC / issues text purge** (all `src/`
|
||||
identifiers + the AST node are already done; remaining is prose). Lower-risk than the
|
||||
renames (text only, mostly snapshot-neutral) but needs per-instance reading — NOT a
|
||||
blind sed. Footprint: `src/` ~205 (all comments now), `examples/*.sx` ~100 comments,
|
||||
`issues/*.md` ~20 files, docs (specs/readme/CLAUDE).
|
||||
|
||||
Order:
|
||||
1. **src/ comments** (~200) — reword `foreign`→`extern`/`runtime-class` to match the
|
||||
renamed identifiers. KEEP: the rejection-message string, the `hash_foreign` token +
|
||||
its `#foreignx`/`lex hash_foreign` test, and any comment that legitimately explains
|
||||
the cutover (it must name `#foreign` to say it's removed). The ast.zig FnDecl comment
|
||||
still says "mirroring `#foreign LIB "csym"` (foreign_lib/foreign_name)" — reword.
|
||||
2. **examples/*.sx comments** — the deferred provenance comments (full list in the prior
|
||||
Next-step revision / git). ⚠ Many CONTRAST `#foreign` vs `extern` — reword to stay
|
||||
coherent. ⚠ `1219-ffi-foreign.sx` prints `"foreign-rename: {}"`/`"=== 15. Foreign ==="`
|
||||
to STDOUT — changing those regens its snapshot (intentional). `1176`/`1216` legitimately
|
||||
discuss `#foreign` removal — keep minimal `#foreign` mentions where the test IS about it.
|
||||
3. **issues/*.md** (~20) — rewrite writeup prose to `extern`/`export`/`runtime-class`.
|
||||
4. **docs** — specs.md (rename "Foreign Function Interface" heading → "C Interop"; the
|
||||
`#import c` "foreign declarations" prose; the comptime "foreign function calls" line;
|
||||
§3344 "foreign code can't observe the error channel"), readme.md (211-212 the `#import
|
||||
c` exemption prose), CLAUDE.md (host_ffi `#foreign("c")` ref → `extern`; "foreign calls").
|
||||
5. **9.0 surface decision** (recommend KEEP `hash_foreign` token + rejection for a good
|
||||
deprecation — then it + the message + 1176 + `#foreignx` are permanent gate-exempt keeps).
|
||||
6. **9.4 gate** — `grep -rniIE 'foreign'` over `.sx` + docs + `src/` minus the keep-list → 0.
|
||||
KEEP-LIST (gate-exempt): `SQLITE_CONSTRAINT_FOREIGNKEY` + SQLite API names, vendored
|
||||
`library/vendors/sqlite/c/*`, the `hash_foreign` token + `#foreignx` test + rejection
|
||||
message string, `1176-diagnostics-foreign-removed.sx` (rejection test must contain it).
|
||||
**Gate (scoped per user 2026-06-15):** `grep -rniIE 'foreign'` → 0 across `.sx` files,
|
||||
all docs, and our `src/` Zig — EXCLUDING the legitimate keeps listed in Last completed
|
||||
step (SQLite API names, vendored C, the rejection test/message + `hash_foreign` token).
|
||||
|
||||
Remaining, in suggested (dependency-safe) order:
|
||||
1. **9.1d — eliminate `foreign_expr`** (last linkage item): migrate `c_import.zig`
|
||||
auto-synthesis to build the extern shape instead of a `foreign_expr` body (the Phase
|
||||
5.0 fn-body flip applied to auto-synth), then delete the `foreign_expr` AST node +
|
||||
`ForeignExpr` + all readers (25). Snapshot-neutral; verify full corpus (the `#import c`
|
||||
examples 1215/1216/1217 + sqlite 1624 exercise it).
|
||||
2. **9.2 — runtime-class family rename → `Runtime*` (Decision 5).** The BIG one, do as
|
||||
small per-identifier commits with `zig build` after each (snapshot-neutral). Targets
|
||||
(counts): `ForeignClassDecl`(65)→`RuntimeClassDecl` · `foreign_path`(62)→`runtime_path`
|
||||
· `foreign_class_map`(44) · `current_foreign_class`(34)/`_method` · `ForeignMethodDecl`(31)
|
||||
· `foreign_class_decl`(30) · `foreign_expr`-gone-by-now · `ForeignClassMember`(20) ·
|
||||
`ForeignFieldDecl`(15) · `ForeignClassDecl.is_foreign`(the live one)→e.g. `is_reference`
|
||||
· `parse/tryParseForeignClass*` · `lowerForeign{Method,Static}Call` ·
|
||||
`findForeign*InChain` · `resolveForeign*` · `register*ForeignClass*` · `*ForeignRefs` ·
|
||||
`ForeignRuntime` · `current_foreign_class`/`_method`. ⚠ COUPLED .sx↔.zig hook names:
|
||||
`jni_main_foreign_path_at`/`jni_main_foreign_paths`/`hookJniMainForeignPathAt`/
|
||||
`foreignPathToJavaName`/`splitForeignPath` span build.sx + bundle.sx + compiler_hooks.zig
|
||||
+ specs.md (2975/3049) — rename all four sites together.
|
||||
3. **9.x-src-comments** — the ~200 bare-`foreign` comments in `src/` (rename last, since
|
||||
many reference identifiers that 9.1d/9.2 rename; do AFTER those so the comment text
|
||||
matches the new names).
|
||||
4. **9.3-examples comments** — the deferred `.sx` provenance comments (0716, 0729, 1205/
|
||||
1207/1216/1218/1219/1220, 1223-1231, 1306/1308/1315/1318/1320/1321/1331/1332/1348/1349,
|
||||
1412/1414/1417/1418/1419/1426, 0117/0415, 1140/1141/1125, issues/0030.sx). ⚠ Many
|
||||
CONTRAST `#foreign` vs `extern` ("no `#foreign`, no `#library`") or reference renamed
|
||||
internals — rewrite each to stay coherent (NOT blind sed). ALSO: `1219-ffi-foreign.sx`
|
||||
prints `"foreign-rename: {}"` to STDOUT — changing it regens the snapshot (intentional).
|
||||
5. **9.3-issues** — `issues/*.md` writeups (~20 files) → rewrite `#foreign`/`foreign` to
|
||||
`extern`/`export`/`runtime-class` per the renames.
|
||||
6. **9.3-docs** — specs.md (12: rename "Foreign Function Interface" heading → "C Interop";
|
||||
the `#import c` "foreign declarations" prose; the jni_main_foreign_path_at refs with #2),
|
||||
readme.md (2), CLAUDE.md (2: host_ffi `#foreign("c")` ref + "foreign calls").
|
||||
7. **9.0 surface decision** — keep `hash_foreign` token + rejection (recommended: good
|
||||
deprecation) vs delete it. If kept, the token + the rejection-message string + 1176 are
|
||||
permanent legitimate keeps; the gate excludes them.
|
||||
8. **9.4 gate** — `grep -rniIE 'foreign'` over the gated set minus the keep-list → 0.
|
||||
- **6.2 verification note (carry forward):** the `platform/` runtime modules
|
||||
(uikit/android/android_jni) are NOT compiled by any marker'd host corpus test — verify
|
||||
future platform-adjacent migrations via direct `sx ir` on importers (1610/1606 compile
|
||||
uikit on host) or import probes, not the corpus alone.
|
||||
- **Phases 6–7** (`refactor` batches, empty snapshot diff per batch): migrate the
|
||||
stdlib + examples from `#foreign` spelling to `extern`. Because the AST is already
|
||||
unified, this is a pure SOURCE rename (`… #foreign LIB "sym";` → `… extern LIB "sym";`
|
||||
for fns; the global/const forms similarly), and IR/output must be byte-identical per
|
||||
batch. NOTE: `c_import.zig` auto-synthesis (`#import c {#include}`) still BUILDS
|
||||
`foreign_expr` bodies internally — that's a compiler-internal path, migrated separately
|
||||
(likely Phase 8/9 area), not a source-spelling change.
|
||||
- **Then Phase 8** (cutover: hard-reject the `#foreign` keyword) and **Phase 9** (purge
|
||||
all `foreign` identifiers — needs Decision 5 [done, `Runtime*Class*`] + Decision 6
|
||||
[open, historical carve-out]).
|
||||
|
||||
**Watch items carried forward:**
|
||||
- `c_import.zig:262` auto-synthesis still emits `foreign_expr` — both shapes coexist
|
||||
until that path is migrated; keep every `body.data == .foreign_expr` reader dual
|
||||
(checked exhaustively this stream).
|
||||
- const-with-type `#foreign` parser path (`parser.zig:316`) is still on `foreign_expr`
|
||||
but DEAD (registers no const); migrate or delete it at the Phase 8 cutover.
|
||||
- The `decl.zig:2055` "foreign symbol … already bound" dedupe message is keyword-neutral
|
||||
and fires for both forms — no churn, but reword to "extern" at cutover for consistency. Route the fn-decl `#foreign` path so a
|
||||
`#foreign` fn builds the SAME extern AST that postfix `extern` already produces,
|
||||
instead of a `foreign_expr` body. This is the highest-value path (the bulk of
|
||||
`#foreign` usage). Key sub-questions to resolve before/while routing:
|
||||
- The `foreign_expr` node carries `library_ref` + `c_name`; the `extern` fn carries
|
||||
`extern_export = .extern_` + `extern_lib` + `extern_name` on the FnDecl with an
|
||||
empty-block body. Migration = the parser's fn-body `#foreign` arm
|
||||
(`parser.zig:~2062`) builds the extern shape (set `extern_export`, map
|
||||
`library_ref→extern_lib`, `c_name→extern_name`) rather than a `foreign_expr`.
|
||||
- Lowering ALREADY coalesces the two at every fn site checked this stream
|
||||
(`decl.zig` 2088/2124/2132/2156/2324/2531 read `is_foreign OR extern_export`),
|
||||
and the two prereq gates (visibility `decl.zig:2249`, variadic `decl.zig:2097` +
|
||||
`pack.zig:302`) now do too — so the migration should be behavior-preserving with
|
||||
ZERO snapshot churn. VERIFY with the A→B gate test (`lower.test.zig`) + a full
|
||||
`zig build test` after routing; any churn means a site still reads `foreign_expr`
|
||||
structurally and must be coalesced first.
|
||||
- ⚠ This ALSO migrates the **const-with-type** path implicitly IF it shares the same
|
||||
`foreign_expr`→extern reshape (it builds `const_decl{value=foreign_expr}`). Decide:
|
||||
reshape the const path's value node alongside, or leave the dead const path on
|
||||
`foreign_expr` until Phase 8 cutover. The const path is dead (see findings below),
|
||||
so leaving it is acceptable; but the parser arm is shared-ish — check whether the
|
||||
fn-body arm change touches it.
|
||||
- Cadence: because the migration is behavior-preserving (no churn), it's a single
|
||||
`refactor`/lock commit (like the 5.0 global-path commit `e5ddfbe`), NOT an
|
||||
xfail→fix pair.
|
||||
|
||||
**Investigation findings (this session — reorder the remaining paths):**
|
||||
- **const-with-type** (`parser.zig:316`, `name :: type_expr #foreign …`) is a
|
||||
**DEAD path**: it builds `const_decl{value = foreign_expr}`, but
|
||||
`registerTypedModuleConst` (`decl.zig:848-851`) bails on a `foreign_expr` value
|
||||
(`else => return`), so it registers no const and emits no symbol — a probe
|
||||
(`g_abs :: FP #foreign "abs";`) returns `unresolved 'g_abs'` at the use site, and
|
||||
the form is used NOWHERE in `library`/`examples`/`issues`. Its migration target is
|
||||
ambiguous because the `foreign_expr` value node is SHARED with the fn-decl path,
|
||||
which isn't migrated yet. **Decision (user, 2026-06-14): defer it — migrate it
|
||||
alongside the fn-decl path once `foreign_expr`'s extern shape is decided.** The
|
||||
checkpoint's old "lowest-risk, route to the extern-named shape" note is wrong: the
|
||||
"confirm the value-node lowering path coalesces" gate can't be met (nothing lowers it).
|
||||
- **runtime-class prefix** (`parser.zig:~1351`, `#foreign #objc_class/#jni_class`) is
|
||||
**ALREADY coalesced**: both prefix `#foreign` and postfix `extern` feed the single
|
||||
`is_foreign_eff`→`is_foreign` field on `foreign_class_decl` (`parser.zig:1421-1432`),
|
||||
so there is NO Phase 5.0 AST change for it — only the Phase 9.2 `Runtime*Class*`
|
||||
rename remains. Drop it from the Phase 5.0 path list.
|
||||
|
||||
So Phase 5.0's real remaining work collapses to: the fn-path variadic prereq, then
|
||||
the fn-decl `#foreign` body-marker migration. const-with-type + runtime-class need
|
||||
no standalone Phase 5.0 commit.
|
||||
|
||||
Then Phase 5.1 (`lock`): unit test that `#foreign` and `extern` produce identical IR (the
|
||||
A→B gate already covers fn/global/class — extend or reuse `lowerSrcToIr`). Then Phases 6–7
|
||||
migrate stdlib + examples (empty snapshot diff per batch), Phase 8 cutover (hard-reject
|
||||
`#foreign`), Phase 9 total `foreign` purge.
|
||||
|
||||
**⚠ CONFIRM BEFORE PART B (Open decisions 5 & 6):** runtime-class rename target
|
||||
(`Runtime*Class*` recommended vs `Extern*Class*`) and the historical carve-out (keep
|
||||
`issues/*.md` provenance, gate live tree only — recommended). These decide Phase 9 renames;
|
||||
the plan says confirm before Phase 9, but worth raising with the user before sinking Part B
|
||||
effort. **Also pick up the two Deferred items below at the start of Part B** (the
|
||||
visibility-gate equivalence in particular needs a cross-module example).
|
||||
|
||||
**FUTURE MILESTONE — C→sx-by-name in JIT (`sx run`).** Investigated this session
|
||||
(user-requested spike, RESOLVED feasible-but-blocked). Adding the C `#source` objects
|
||||
directly into the ORC JITDylib (`LLVMOrcLLJITAddObjectFile`) instead of dlopen'ing a
|
||||
dylib makes C↔sx cross-references resolve both ways in one link domain — proven: a
|
||||
~20-line spike ran 1226 via `sx run` (37/82) and all 13 existing `#source` FFI examples
|
||||
still passed. BLOCKER: C objects using `_Thread_local` (the return-trace runtime
|
||||
`sx_trace.c`) SIGABRT under JITLink — MachO thread-local-variable handling needs the ORC
|
||||
`MachOPlatform` set up (the bare `LLVMOrcCreateLLJIT` default doesn't), and C
|
||||
constructors/`__mod_init_func` won't run without ORC initializer support. 42 `errors-*`
|
||||
examples crashed in the spike. A real impl needs a C++ shim in `llvm_shim.c`
|
||||
(`LLJITBuilder().setObjectLinkingLayerCreator(...)` + `MachOPlatform::Create`) — its own
|
||||
milestone, NOT Phase 2/3 scope. The AOT `.aot`-marker corpus mode is the pragmatic test
|
||||
path and works today. Spike fully reverted (target.zig/main.zig at HEAD).
|
||||
|
||||
**Deferred (carry into Part B):** (a) ~~docs~~ — DONE in Phase 4 (`specs.md`/`readme.md`
|
||||
document `extern`/`export`; `#foreign` stays until the Part B cutover); (b) ~~visibility-gate
|
||||
equivalence~~ — **DONE** (`717c35d`/`7d8ba1a`): the `c_import_bare` gate now polices a
|
||||
lib-less `extern` fn identically to its lib-less `#foreign` twin (same C-specific
|
||||
diagnostic); a library-bound `extern LIB` stays unconditionally visible. Locked by the
|
||||
cross-module example 1228. (Empirical: the bare-extern twin was never a silent hole — the
|
||||
general `isNameVisible` gate already denied it; only the diagnostic wording diverged.)
|
||||
|
||||
## Open decisions
|
||||
Part A ratified (bare / postfix / `⇒ callconv(.c)` / lib-separate). Part B:
|
||||
- **Decision 5 RATIFIED** (user, 2026-06-14): runtime-class rename target = `Runtime*Class*`
|
||||
(object-model axis, not linkage). Drives the Phase 9.2 identifier renames.
|
||||
- **Decision 6 RATIFIED** (user, 2026-06-15): **PURGE EVERYTHING** — the Phase 9.4 gate is
|
||||
absolute, including `issues/*.md` writeups (NOT the recommended keep-provenance default).
|
||||
Every `#foreign`/`foreign` reference in the gated tree (`src/ library/ examples/ issues/
|
||||
specs.md readme.md CLAUDE.md`) is rewritten to `extern`/`export`; provenance lives in git
|
||||
history + `(Regression issue NNNN)` notes, not the keyword spelling.
|
||||
- **Decision 7 RATIFIED** (user, 2026-06-15): **accept the churn** — `#foreign`-spelled
|
||||
decls produce `extern`-worded diagnostics; example 1620 regenerated (only snapshot moved).
|
||||
Aligns with Part B's extern-only end state; the interim oddity is cosmetic and removed at
|
||||
the Phase 8 cutover. Landed in the fn-body flip `6b94bb6`. (Original framing below.)
|
||||
— interim diagnostic wording for `#foreign`-spelled decls (gated the fn-body flip). Once the flip lands, a `#foreign`-spelled fn builds the extern AST, so any
|
||||
diagnostic that reads the unified AST can no longer tell the user wrote `#foreign` vs
|
||||
`extern`. Concretely, example 1620's lib-ref error flips "#foreign library…" →
|
||||
"extern library…". Options: **(A, recommended)** accept the narrow churn — regen 1620 as
|
||||
intentional; it aligns with Part B's `extern`-only end state and the interim oddity
|
||||
(`#foreign` source → "extern" message) is cosmetic and short-lived (Phase 8 cutover
|
||||
removes `#foreign`). **(B)** retain a one-bit surface marker on `FnDecl` (`wrote_foreign`)
|
||||
so interim diagnostics stay keyword-accurate (zero churn, small extra plumbing, marker
|
||||
deleted at cutover). Affects only diagnostic wording — IR/behavior identical either way.
|
||||
|
||||
## Log
|
||||
- (9.0 + 9.3 + 9.4) **PHASE 9 COMPLETE — STREAM DONE; 9.4 GATE PASSES.** Deleted the
|
||||
hash_foreign token (9.0, `dfae690`); purged all `foreign` comments incl. capital-F
|
||||
(src/examples/docs/editors); renamed 10 `*-foreign*` example files + dedup'd 1218
|
||||
(`b52d424`); rewrote 20 issues/*.md + renamed 0043 (`b9cfe25`). Gate: zero `foreign`
|
||||
in the gated tree except `SQLITE_CONSTRAINT_FOREIGNKEY` + vendored sqlite c/. Suite
|
||||
green (644/443). User flagged several leftover areas mid-purge (docs/, editors/,
|
||||
capital-Foreign comments, the token) — all addressed.
|
||||
- (9.3 src capital-Foreign) Fixed the case-sensitivity gap — my earlier src verify grep
|
||||
was case-sensitive, missing ~21 capital `Foreign`/`FOREIGN` comments (Foreign-class→
|
||||
Runtime-class, Foreign path→Runtime path, Foreign decls→Extern decls, FOREIGN function→
|
||||
extern function, etc.) across calls/inst/ffi_objc/jni_descriptor/emit_llvm/c_import/
|
||||
lower.* /ops.zig. All reworded via Edit (comments only — no build impact). UNCOMMITTED
|
||||
(mutating Bash blocked by a classifier outage). After this, src `foreign` = ONLY the
|
||||
`hash_foreign` token machinery + 4 rejection messages (the 9.0-delete targets).
|
||||
- (9.0 RATIFIED) User: DELETE the hash_foreign token (total purge). Pending build+regen.
|
||||
- (9.3 text purge) Purged `foreign` from all `src/` comments (`dc51c4b`), specs/readme/
|
||||
CLAUDE (`e99383f`), and all example .sx comments (STAGED, commit pending a classifier
|
||||
outage). Fixed 2 user-facing diagnostics (type-annotation error, Android jni_main help).
|
||||
1219 stdout labels Foreign→Extern (regen). Suite green (646/444). Remaining: example
|
||||
FILENAMES + issues/*.md + the 9.0 token decision + 9.4 gate.
|
||||
- (9.2a-d) **RUNTIME-CLASS IDENTIFIER PURGE COMPLETE** (Decision 5 → `Runtime*`).
|
||||
9.2a types (`3354446`), 9.2b fns+state+`is_extern` flag (`5c8af6e`, fixed `a15a868`
|
||||
per user: reuse `is_extern` not new `is_reference`), 9.2c extern-ref validators →
|
||||
`Extern` (`d27be42`), 9.2d `foreign_path`→`runtime_path` coupled across the build-hook
|
||||
boundary + 37 `.ir` regens (`8cca3b9`). `src/` now has ZERO `foreign` identifiers
|
||||
(only comments + the kept token/message remain). Suite green throughout.
|
||||
- (9.1d) Eliminated the `foreign_expr` AST node — migrated `c_import.zig` auto-synth to
|
||||
the extern shape, deleted the node + all readers. `refactor` `7ffdc7d`.
|
||||
- (9.1c) Deleted dead `VarDecl.is_foreign`/`foreign_lib`/`foreign_name` (global `#foreign`
|
||||
rejects → write-dead); 3 decl.zig readers simplified to `vd.extern_name`/`vd.is_extern`.
|
||||
Snapshot-neutral; suite green (646/444). `refactor` `cd14794`.
|
||||
- (9.1b) "foreign symbol already bound" diagnostic + resolveFuncByName panic → "extern
|
||||
symbol"; intentional 1172 regen. Suite green. `refactor` `b78e7dd`.
|
||||
- (9.1a) **PHASE 9 STARTED.** 5 collision-free linkage renames (callForeign→callExtern,
|
||||
marshalForeignArg, dedupeForeignSymbol, foreign_name_map→extern_name_map,
|
||||
is_foreign_c_api). Snapshot-neutral; suite green. `refactor` `b838f63`. Decision 6
|
||||
scoped by user: purge `.sx` + docs + our `src/` Zig, keep legitimate hits (SQLite API
|
||||
names, vendored C, the rejection test/message + hash_foreign token).
|
||||
- (8.1 cutover) **PHASE 8 COMPLETE.** Parser hard-rejects `#foreign` at all 4 sites
|
||||
(const/global/fn-body via `self.fail`; runtime-class via `self.failAt` at the caller,
|
||||
new helper); greens xfail 1176. Deleted obsolete 1174 + 1620, the GATE A→B test +
|
||||
`lowerSrcToIr` helper; converted the in-source parser test to postfix `extern`;
|
||||
`extern_export` → `const`. specs.md + readme.md drop `#foreign`. Suite green
|
||||
(646/444). `feat!` `3811311`.
|
||||
- (8.0 xfail) Added `1176-diagnostics-foreign-removed.sx` pinning the desired rejection.
|
||||
RED (still accepted). `test`/xfail `8180faf`.
|
||||
- (8 pre-cutover) Migrated the 4 multi-file example companions Phase 7 missed
|
||||
(0729/a+b, 1617/c, 1623/mod). `refactor` `d132aab`.
|
||||
- (8 pre-cutover) Migrated keyword-neutral diagnostics 1172 (decl→extern, message stays
|
||||
internal "foreign symbol") + 1228 (→ two foreign-free extern symbols c_abs_one/_two),
|
||||
intentional snapshot regens reviewed. `refactor` `720556b`.
|
||||
- (8 pre-cutover) Migrated the 7 identity `ffi-foreign-*` test decls to extern/export
|
||||
(decls only; comments left for Phase 9.3). `refactor` `2cce6a3`.
|
||||
- (7.4 stragglers) **PHASE 7 MIGRATABLE WORK COMPLETE.** Migrated 16 fn/global examples
|
||||
(0415/0602/0603/1024/1025/1605/1607-1609/1611/1616/1619/1622/1628/1635/1636) `#foreign`→
|
||||
`extern`; 1607/1608/1616 (unmarked) verified by `sx ir` probes. 24-file keep-list remains
|
||||
by design (deferred to Phase 8). Suite green (647/444). `refactor` `1a8991a`.
|
||||
- (7.3 14xx) Migrated 13 jni examples (1410-1419/1423/1424/1425). 1417 (all-runtimes) hit
|
||||
a parse-error trap: a `#(objc|jni)_class`-only regex left `extern` in PREFIX position on
|
||||
`#jni_interface`/`#objc_protocol`/`#swift_*` lines → fixed with the GENERAL
|
||||
`#foreign #(\w+)("X") {`→`#$1("X") extern {` rewrite (all such directives accept the
|
||||
postfix modifier, probed). Kept 1426 (comment-only). Suite green. `refactor` `2888f6f`.
|
||||
- (7.2 13xx) Migrated 18 obj-c examples (1308/1311-1321/1341-1347): prefix→postfix import
|
||||
classes + fn markers. Kept identity 1306/1318, comment-only 1332/1348/1349. No 13xx
|
||||
snapshot asserts on foreign. Suite green. `refactor` `a68f7c2`.
|
||||
- (7.1 12xx) **PHASE 7 STARTED.** Migrated 12 incidental plain-C examples
|
||||
(1200/1206/1209-1215/1220/1221/1222) `#foreign`→`extern`; output byte-identical,
|
||||
empty snapshot diff, corpus-validated. Established the keep-list policy (see Last
|
||||
completed step): kept 1172/1174/1620/1228 + ffi-foreign-* (1205/1207/1216/1218/1219)
|
||||
+ comment-only 1223/1229/1230/1231 for Phase 8. Suite green (647/444). `refactor`
|
||||
`731fb8d`.
|
||||
- (6.5 gpu) **PHASE 6 COMPLETE.** Migrated `gpu/gles3.sx` + `gpu/metal.sx` (3 sites);
|
||||
`library/` now `#foreign`-free (`grep -rln '#foreign' library/` → 0). Verified
|
||||
byte-identical `sx ir` on importers 1610/1606. Suite green (647/444). `refactor`
|
||||
`32a7628`.
|
||||
- (6.4 ffi) Migrated `ffi/` objc/objc_block/raylib/sdl3/wasm (~51 sites): fn markers +
|
||||
objc.sx's 2 import classes (prefix→postfix `extern`). objc/objc_block validated by 50
|
||||
marked 13xx examples; raylib/ffi-sdl3/wasm by `sx ir` probes pre/post. Empty snapshot
|
||||
diff; suite green (647/444). `refactor` `666a2e2`.
|
||||
- (6.3 std) Migrated 11 `std/` modules (~60 sites): cli/core/fmt/fs/log/net.kqueue/
|
||||
process/socket/thread/time/trace. All fn-decl markers (bare / `libc`|`tlib` LIB ref /
|
||||
`libc "csym"` rename) → `extern …` + 2 comment mentions; no class forms. Host-corpus-
|
||||
exercised → empty snapshot diff validates. Suite green (647/444). `refactor` `59f90d2`.
|
||||
- (6.2 platform) Migrated `platform/` (uikit/android/android_jni/sdl3, ~64 sites):
|
||||
30 fn `#foreign;`→`extern;`, 34 import classes prefix `#foreign #objc/jni_class`→
|
||||
postfix `… extern {`, 4 defined `Sx*` objc classes → `… export {`. Behavior-
|
||||
preserving, empty snapshot diff. Verified byte-identical `sx ir` on uikit importers
|
||||
1610/1606 + sdl3 probe; android via identical 4-error dedup set (host-only module).
|
||||
Suite green (647/444). `refactor` `2cd5d7b`. NOTE: these runtime modules aren't in
|
||||
the marker'd host corpus — verified out-of-band.
|
||||
- (6.1 sqlite) **PHASE 6 STARTED.** Migrated `vendors/sqlite/sqlite.sx`: 97
|
||||
`… #foreign sqlib "csym";` fn decls → `extern sqlib "csym";` (+ line-9 comment).
|
||||
`extern_lib` references the `sqlib` `#import c` unit like `#foreign sqlib`; IR
|
||||
byte-identical, empty snapshot diff, example 1624 stdout unchanged. Suite green
|
||||
(647/444). `refactor` `410a52e`.
|
||||
- (5.1 gate annotate) **PHASE 5 COMPLETE.** Annotated the A→B gate header
|
||||
(`lower.test.zig`) to record that post-Phase-5.0 the fn/global `#foreign` paths
|
||||
build the same extern-named AST → cases 1/2 are structurally (not coincidentally)
|
||||
identical; the gate stays as a regression tripwire. Added fn-rename case 2b
|
||||
(`c_abs` → `"abs"`, `extern_name` axis), IR-identical per a `sx ir` probe.
|
||||
Test-only, no snapshot churn. Suite green (647/444). `test` `93e7b6f`.
|
||||
- (5.0 fn-body flip) **PHASE 5.0 PARSER ROUTING COMPLETE.** Flipped the fn-body
|
||||
`#foreign` parser arm (`parser.zig:~2062`) onto the extern AST (empty-block body +
|
||||
`extern_export = .extern_` + extern_lib/extern_name); `extern_export` made `var` so
|
||||
the body arm can route onto it. Updated the parser unit test to assert the extern
|
||||
shape. Behavior-preserving via the four prereqs; only example 1620's lib-ref message
|
||||
churned ("#foreign library"→"extern library", Decision 7, hand-edited). Suite green
|
||||
(647 corpus / 444 unit). `refactor` `6b94bb6`.
|
||||
- (5.0 prereq plain-free xfail) Added `1230-ffi-extern-same-name-authors` (two flat
|
||||
authors of `absval` via `extern libc "abs"`; the `extern` twin of `#foreign` 0729).
|
||||
RED — extern authors wrongly counted as ambiguous (646/1 fail). `test`/xfail `2706521`.
|
||||
- (5.0 prereq plain-free fix) `isPlainFreeFn`/`isPlainFreeFnDecl` now also exclude
|
||||
`extern_export == .extern_` (external C symbol, no sx body; name-keyed first-wins like
|
||||
`#foreign`); `export` stays plain-free. 1230 green (`absval = 7`). Suite green (646/444).
|
||||
`fix`/green `3c94c14`.
|
||||
- (5.0 prereq lib-ref xfail) Added `1231-ffi-extern-undeclared-lib` (`extern nosuchunit
|
||||
"abs"` — bogus lib ref). RED — compiles silently (extern lib ref unvalidated).
|
||||
`test`/xfail `38c3240`.
|
||||
- (5.0 prereq lib-ref fix) `checkForeignRefs` (c_import.zig) now reads the lib ref from
|
||||
either spelling (foreign_expr.library_ref OR extern_lib) and names the surface keyword,
|
||||
so 1620 (#foreign) is byte-unchanged and 1231 (extern) gets "extern library … not
|
||||
declared". 1231 green. Suite green (647/444). `fix`/green `ad6aed3`. **ALL FOUR fn-path
|
||||
prereqs DONE → fn-body flip de-risked; awaiting Decision 7 (interim wording).**
|
||||
- (5.0 prereq variadic xfail) Added `1229-ffi-extern-cvariadic` (JIT `#source`,
|
||||
int-sum + double-avg, `extern` C-variadic). Expected snapshot pins the DESIRED
|
||||
correct output. RED (variadic `extern` slice-packs extras → garbage:
|
||||
`sum_ints(3,10,20,30)` → 53316585; doubles → 0.0). `test`/xfail `9a2c78d`.
|
||||
- (5.0 prereq variadic fix) Extended the two C-variadic gates — the `is_variadic`
|
||||
drop in `declareFunction` (`decl.zig:2097`) and the early-out in
|
||||
`packVariadicCallArgs` (`pack.zig:302`) — to fire for `extern_export == .extern_`
|
||||
as well as a `foreign_expr` body. 1229 green (`60` / `2.000000`). Suite green
|
||||
(645 corpus / 444 unit, 0 failed). `fix`/green `0fdc821`. **BOTH fn-path prereqs
|
||||
DONE → fn-decl `#foreign` body-marker migration unblocked.**
|
||||
- (5.0 prereq vis xfail) Added cross-module example `1228-ffi-extern-c-non-transitive`
|
||||
(main → b → c). Main references c's lib-less `#foreign` + `extern` twins
|
||||
transitively; expected snapshot pins the DESIRED equivalent C-specific
|
||||
diagnostic for both. RED (extern twin gets the generic "not visible" wording —
|
||||
443/444). `test`/xfail commit `717c35d`; the fix greens it.
|
||||
- (5.0 prereq vis fix) Extended `isVisible(.c_import_bare)` (`decl.zig:2249`) to
|
||||
switch on the body: a `foreign_expr` body OR an `extern_export == .extern_` decl
|
||||
with no lib both route to `visibleOverEdges`; a library-bound decl stays
|
||||
unconditionally visible. 1228 green — both twins emit "C function not visible".
|
||||
Suite green (644 corpus / 444 unit, 0 failed). `fix`/green commit `7d8ba1a`.
|
||||
**Deferred prereq (b) CLOSED.** Investigation this session also found
|
||||
const-with-type is a DEAD parser path (defer per user) and the runtime-class
|
||||
prefix is already coalesced (no Phase 5.0 change) — see Next step.
|
||||
- (5.0 global) **PART B STARTED.** Routed the `#foreign` data-global parser path
|
||||
(`parser.zig:425`) onto the extern-named `VarDecl` (`is_extern`/`extern_lib`/
|
||||
`extern_name`) — the same AST postfix `extern` builds. Behavior-preserving
|
||||
(lowering coalesces both at `decl.zig:1119,1127,1141`); zero snapshot churn. Suite
|
||||
green (444/444 unit, 643 corpus). `refactor` lock, commit `e5ddfbe`. Remaining
|
||||
Phase 5.0 paths: const-with-type (316), fn-body (2059, needs visibility+variadic
|
||||
prereqs), runtime-class prefix (1305).
|
||||
- (init) Plan written; FFI-linkage stream opened.
|
||||
- (merge) Folded FOREIGN-MIGRATION in as Part B; deleted the split plan + checkpoint.
|
||||
- (0.0) Added `kw_extern`/`kw_export` tokens + keyword-map entries + LSP keyword
|
||||
classification + `lex linkage keywords` test. Suite green; no identifier collisions
|
||||
in the corpus. `lock` commit.
|
||||
- (0.1) Added `ast.ExternExportModifier` + `FnDecl.extern_export` +
|
||||
`VarDecl.is_extern`/`extern_name` + `parseOptionalExternExport()` (unconsumed) + 2
|
||||
parser unit tests. Suite green (443/633). `lock` commit.
|
||||
- (1.0a) Wired fn-path extern parsing (`parseFnDecl` + both lookahead predicates) +
|
||||
added `FnDecl.extern_lib`/`extern_name` + `VarDecl.extern_lib` per user feedback
|
||||
(decision 4 revised: extern carries an optional lib axis). Unconsumed by lowering.
|
||||
Suite green (443/633). `lock` commit.
|
||||
- (1.0b) Added `examples/1223-ffi-extern-fn.sx` + hand-authored success snapshots.
|
||||
RED (634 ran, 1 failed — sema `body produces no value`). `xfail` commit; 1.1 greens it.
|
||||
- (1.1) Wired extern fn lowering (6 edits in `decl.zig`, all declare-only routing
|
||||
mirroring `foreign_expr`): `funcWantsImplicitCtx` + `declareFunction` cc +
|
||||
`lazyLowerFunction`/`lowerFunction`/`lowerFunctionBodyInto` guards. 1223 green;
|
||||
`declare i32 @abs(i32)` (C ABI, no ctx). Suite green (634/443). `green` commit.
|
||||
- (1.2a) Added `examples/1224-ffi-extern-fn-rename.sx` (`c_abs :: … extern "abs";`) +
|
||||
hand-authored success snapshot (`c_abs(-42) = 42`). RED (635 ran, 1 failed — parse
|
||||
error: `"abs"` after `extern` not yet accepted). `xfail`; 1.2b greens it. (Also
|
||||
recovered a formatter-clobbered `parser.zig` — see Known issues.)
|
||||
- (1.2b) `parseFnDecl` parses the optional `[LIB] ["csym"]` tail into
|
||||
`extern_lib`/`extern_name`; `declareFunction` unifies the rename (foreign c_name OR
|
||||
extern_name → declare under C name, map sx→C) and extends the dedupe guard to
|
||||
extern. 1224 green (`c_abs`→`abs`); 1223 unregressed. Suite green (635/443).
|
||||
`green` commit. extern_lib parsed+stored (lib linking stays the `#library` axis).
|
||||
- (1.2c) Added `examples/1225-ffi-extern-global.sx` (`__stdinp : *void extern;`,
|
||||
mirrors `#foreign` global 1205) + success snapshot. RED (636 ran, 1 failed — parse
|
||||
error: var-decl `extern` not accepted). `xfail`; 1.2d greens it.
|
||||
- (1.2d) Parser `kw_extern` branch in the var-decl path (`[LIB] ["csym"]` →
|
||||
`is_extern`/`extern_lib`/`extern_name`) + `registerTopLevelGlobal`/`globalInitValue`
|
||||
consume `is_extern`. 1225 green (`@__stdinp = external global ptr`). Suite green
|
||||
(636/443). `green` commit. **PHASE 1 COMPLETE** — `extern` fns + globals fully work.
|
||||
- (JIT spike) User-requested feasibility investigation of C→sx-by-name in `sx run`
|
||||
(JIT). Verdict: feasible via `LLVMOrcLLJITAddObjectFile` (C objects into the ORC
|
||||
JITDylib) — proven by a throwaway spike — but blocked by JITLink MachO TLV handling
|
||||
(`sx_trace.c`'s `_Thread_local` SIGABRTs without the ORC `MachOPlatform`). Own future
|
||||
milestone (see Next step). Spike reverted; no commit.
|
||||
- (2.0) Added the **AOT corpus mode** (`expected/<name>.aot` → `sx build` + execute) to
|
||||
`corpus_run.test.zig` + retired `tests/run_examples.sh` (verify-step.sh/CLAUDE.md
|
||||
updated) + `examples/1226-ffi-export-fn.{sx,c,h}` (C calls `sx_square` back). RED (AOT
|
||||
link fails: `_sx_square` undefined — export not lowered). `xfail`; 2.1 greens it.
|
||||
- (2.1) Filled export gaps i/ii/iv in `decl.zig` (`.external` linkage + `.c` cc on both
|
||||
define paths; `funcWantsImplicitCtx` false for any non-`.none` modifier) + force-lower
|
||||
export fns as roots in `lowerMainAndComptime`. 1226 green via AOT (37/82). Suite green
|
||||
(637/443). `green` commit.
|
||||
- (2.2a) Added `examples/1227-ffi-export-fn-rename.sx` (`export "triple_c"`, C calls
|
||||
`triple_c`). RED (define path emits `@sx_triple`, ignores `extern_name` → C ref
|
||||
undefined). `xfail`; 2.2b greens it.
|
||||
- (2.2b) `declareFunction` rename branch fires for `export` (stub under C name +
|
||||
sx→C in `foreign_name_map`); `lazyLowerFunction` resolves the stub by that C name so
|
||||
the body promotes into the C-named function (`define @triple_c`). sx-side call sites
|
||||
resolve via the same map (probe: 5*5→25). 1227 green (22); 1226 unregressed. Suite
|
||||
green (638/443). `green` commit. **PHASE 2 COMPLETE** — `export` fully works.
|
||||
- (3.0) Added `examples/1348-ffi-objc-extern-class.sx` (postfix `extern` on `#objc_class`,
|
||||
new spelling of `#foreign #objc_class`). RED (parser: `expected '{'` after the
|
||||
directive). Hand-authored green snapshots. `xfail` commit; 3.1 greens it.
|
||||
- (3.1a) Wired the postfix `extern`/`export` aggregate slot in `parseForeignClassDecl`
|
||||
(optional modifier between `("X")` and `{`; `var is_foreign_eff` overrides the passed
|
||||
`is_foreign`, threaded into the `foreign_class_decl` node). No lowering change — reuses
|
||||
the existing `is_foreign` reference-vs-define path. 1348 green. Suite green (639/443).
|
||||
`green` commit. **PHASE 3 COMPLETE.**
|
||||
- (3.1b) Behavior-lock: added `examples/1426-ffi-jni-extern-class.sx` (jni `extern`,
|
||||
parse-only) + `examples/1349-ffi-objc-export-class.sx` (objc `export` defined class,
|
||||
`counter: 2`). Both pass against the 3.1a parser change (locked in their own commit per
|
||||
the cadence rule). Suite green (641/443). `lock` commit. (Note: `-Dupdate-goldens`
|
||||
newline-normalizes empty stderr → reverted unrelated 1226/1227 churn, kept new stderr
|
||||
0-byte per repo convention; runner normalizes both.)
|
||||
- (4.gate) **GATE A→B** — added `lowerSrcToIr` helper + "GATE A→B" test to `lower.test.zig`:
|
||||
`#foreign` ≡ `extern`/`export` byte-identical printed IR for fn / global / Obj-C class.
|
||||
Verified live via negative-probe (mutate one side → assertion fails). Behavior-lock; the
|
||||
equivalence was prototyped first with `sx ir` (LLVM IR byte-identical for all three).
|
||||
Suite green (641/444). `test` commit.
|
||||
- (4.diag1) Added `examples/1174-diagnostics-foreign-postfix-conflict.sx` — prefix `#foreign`
|
||||
+ postfix `export` on an aggregate previously surfaced a confusing internal
|
||||
"emitObjcDefinedClassAllocImp … compiler bug". `xfail` (golden = clean message) → `green`:
|
||||
`parseForeignClassDecl` rejects the combo at the postfix keyword (`failFmt`). Suite green.
|
||||
- (4.docs) `specs.md` (new "`extern`/`export` linkage keywords" subsection after the
|
||||
`#foreign` FFI docs) + `readme.md` (C Interop section) document the three axes. `docs` commit.
|
||||
- (4.diag2) Added `examples/1175-diagnostics-extern-export-conflict.sx` — `extern export` on
|
||||
one fn decl previously gave bare "expected ';'". `xfail` (golden = clean message) → `green`:
|
||||
`parseFnDecl` rejects a second linkage keyword after `parseOptionalExternExport`. Suite
|
||||
green (643/444). **PHASE 4 COMPLETE → PART A DONE.**
|
||||
- (golden-fix) **`-Dupdate-goldens` churn RESOLVED.** Root cause was NOT a code bug:
|
||||
`writeGolden` always writes `content + "\n"` (empty → canonical 1-byte `\n`, used by 484
|
||||
of 489 empty goldens). The 5 churning stderr files [1226/1227/1348/1349/1426] were 0-byte
|
||||
*outliers* (verify trims trailing `\n` so both forms passed, but regen always rewrote them
|
||||
to 1-byte). Conformed all 5 to the 1-byte form → `-Dupdate-goldens` is now idempotent, no
|
||||
more churn. (Separately: a flaky `0712-sha256-streaming` >10s timeout appears only under
|
||||
concurrent `zig build` load — not a real failure; re-run serially.)
|
||||
|
||||
## Known issues
|
||||
- **Workflow hazard (1.2):** an editor format-on-save (or `zig fmt`) clobbered the
|
||||
working-tree `src/parser.zig` between commits — it reformatted one-liners AND
|
||||
silently dropped my `hasFnBodyAfterArrow` extern edit, reverting 1223 to a parse
|
||||
error. Recovered with `git checkout src/parser.zig` (HEAD had the correct,
|
||||
committed version). **After any Edit-tool change to a file the IDE may have open,
|
||||
rebuild + run the affected example before trusting the edit.**
|
||||
970
current/CHECKPOINT-FIBERS.md
Normal file
970
current/CHECKPOINT-FIBERS.md
Normal file
@@ -0,0 +1,970 @@
|
||||
# CHECKPOINT-FIBERS — Stream B1 (fibers + Io + M:1 scheduler)
|
||||
|
||||
Companion to [PLAN-FIBERS.md](PLAN-FIBERS.md). Update after every step (one step at a time,
|
||||
per the cadence rule). New corpus category: `18xx` concurrency.
|
||||
|
||||
## Last completed step
|
||||
**B1.6 — aarch64-LINUX port of the M:1 fiber runtime (sched.sx).** `library/modules/std/sched.sx`
|
||||
now runs end-to-end on aarch64-linux as well as aarch64-macOS, validated **byte-identical** on both
|
||||
via Apple `container` (static ELF, no emulation). The per-OS bits are comptime-branched:
|
||||
- `MAP_AP` (mmap MAP_ANON flag) — `inline if OS == { case .linux: 0x22 case .macos: 0x1002 }`,
|
||||
exhaustive on the supported OSes (no default → a new target fails loud on use).
|
||||
- The fd-readiness backend — kqueue on darwin, **epoll on linux**. The `epoll` import is scoped to
|
||||
the linux branch (`inline if OS == .linux { ep :: #import "modules/std/net/epoll.sx" }`) so darwin
|
||||
never pulls epoll types into the concurrency examples (the std-barrel-drift rule). `block_on_fd`, the
|
||||
run-loop Mode-2 drain, and `cancel_io_waiter_for` each branch kqueue/epoll; epoll additionally
|
||||
`EPOLL_CTL_DEL`s on fire + on early-wake (EPOLLONESHOT only DISABLES, kqueue EV_ONESHOT auto-removes).
|
||||
- The first-entry trampoline was redesigned from a per-OS hand-written global-asm symbol to a **naked
|
||||
sx fn** `fib_tramp` (`mov x0, x19; br x20`) + register-indirect dispatch (spawn presets
|
||||
`regs[1] == x20 == &fib_dispatch`), so no per-OS `.global _fib_tramp`/`fib_tramp` symbol literal is
|
||||
needed. This sidesteps a compiler bug (wrapped top-level `asm` dropped — now **issue 0194**, OPEN).
|
||||
|
||||
**Bug fixed en route (issue 0193 Bug A):** the tramp redesign initially bus-errored on the 1817
|
||||
go/wait/sleep capstone (both OSes) because the WIP had dropped `export "fib_dispatch"`. Without the
|
||||
export `fib_dispatch` uses sx's internal ABI (x0 = implicit `context`, `self` shifted to x1), but the
|
||||
trampoline hands `self` in x0 (C-ABI) → on first entry the body runs (x1 happens to alias `self`) but
|
||||
the closure then loads `regs[1] == &fib_dispatch` as its first capture and recurses forever → stack
|
||||
overflow. **Fix: restore `export "fib_dispatch"`** (pins it to C-ABI, `self` in x0). Root cause found
|
||||
via lldb on an AOT macOS build; confirmed by an adversarial source review (`src/ir/lower/decl.zig`).
|
||||
The 1817 capstone in the suite guards the fix. Suite GREEN **817/0**; 1811/1814/1816/1817 byte-identical
|
||||
macOS host ↔ aarch64-linux container.
|
||||
|
||||
### Earlier — B1 follow-up — `Scheduler.deinit` (close the bounded leaks). Post-B1 non-blocking cleanup: a
|
||||
terminal `deinit` on `library/modules/std/sched.sx`'s `Scheduler` releases the resources B1 documented
|
||||
as leaked. Frees, in order: (1) any fibers still enqueued ready (leak-safety net for `spawn`/`go`
|
||||
without `run()` — `munmap` stack + free struct; a suspended off-queue fiber is unreachable, but a clean
|
||||
`run()` aborts on orphans so none survive it); (2) every heap `*Task` from `go` — newly tracked via a
|
||||
`task_allocs: List(*void)` field appended in `go` (the scheduler otherwise has no handle on its generic
|
||||
`Task($R)`s); (3) the three `List` backings (`task_allocs`/`timers`/`io_waiters`, all grown through
|
||||
`own_allocator`); (4) the lazily-opened kqueue fd (`close`, reset to `-1`). NOT freed (unchanged
|
||||
language limitation): the per-`spawn`/`go` closure env (sx exposes no env-free). Idempotent (rests on
|
||||
`List.deinit` nulling `items` + the `kq`/`ready_head` resets); TERMINAL contract — no scheduler-owned
|
||||
handle (`*Task`, `*Fiber`, the scheduler) is usable after `deinit`.
|
||||
- Added a canonical `close :: (i32) -> i32 extern libc` (matches the dedupe-canonical signature 1816
|
||||
already uses) + the `task_allocs` field.
|
||||
- Locked by `examples/concurrency/1820-concurrency-fiber-scheduler-deinit.sx` (aarch64-macOS `.build
|
||||
{"target":"macos"}`, runs end-to-end): one run touches every freed resource — a SLEEPER (`timers`), a
|
||||
pipe READER `block_on_fd` + WRITER (kqueue fd + `io_waiters`), two `go` tasks (`Task`s + `task_allocs`)
|
||||
— then `deinit`. Verified by a tracking `GPA`: `freed by deinit: 5`, `live after deinit: 5` (the
|
||||
RESIDUAL = the 5 documented closure envs, not a bug), `kq open after run: true` → `kq after deinit:
|
||||
-1` (the genuinely-open kqueue fd is closed), `read: 3 [97 98 99]` (the fd path actually ran). Counts
|
||||
captured into locals BEFORE printing (`print` allocates format temporaries through the same GPA).
|
||||
- **Adversarially reviewed (worker):** no real memory-safety bug in the supported (deinit-after-`run`)
|
||||
path — reap-loop reads `f.next` before freeing `f`, the three freed List backings + Tasks + kq are all
|
||||
disjoint + scheduler-owned, no over-free, idempotent. The one CRITICAL it raised was a DOC contradiction
|
||||
(step-(1) defensive reap vs step-(2) "post-run only"), reconciled by spelling out the terminal contract.
|
||||
Its 0154-over-store concern (`.{}`→`List` writes in `init` could clobber `kq`) was PROBED and cleared:
|
||||
`kq == -1` immediately after `init`, all fields clean. Suite GREEN **759/0**.
|
||||
|
||||
### Earlier — B1.5 — END-TO-END M:1 validation — STREAM B1 COMPLETE
|
||||
A single capstone exercises the whole
|
||||
colorblind pure-sx async runtime together: the M:1 scheduler (B1.5a) + suspending fiber-task async
|
||||
`go`/`wait` (B1.4a) + deterministic virtual-time `sleep`/`now_ms` (B1.4b), over the `abi(.naked)`
|
||||
`swap_context` on guarded `mmap` stacks (B1.0–B1.3). `examples/concurrency/1817-concurrency-fiber-m1-end-to-end.sx`:
|
||||
a coordinator fiber launches three `go` tasks (sleep 30/10/20 → return 100/20/3), awaits all three in
|
||||
SPAWN order, and sums them. The completion log is the deterministic contract — tasks finish in
|
||||
DEADLINE order (`task 2@10, task 3@20, task 1@30`), not spawn/await order; `sum: 123`; final virtual
|
||||
clock 30. Fully reproducible (virtual time, no real clock). Suite GREEN **755/0**.
|
||||
|
||||
**Stream B1 is feature-complete.** The pure-sx async runtime exists end-to-end: fibers behind the
|
||||
`abi(.naked)` context switch (proven on aarch64 + x86_64/Win64), the M:1 cooperative scheduler,
|
||||
suspending `go`/`wait`/`cancel` async, deterministic virtual-time timers, and real fd-readiness via
|
||||
kqueue — all in `library/modules/std/sched.sx`, all adversarially reviewed, locked by `18xx`
|
||||
(1800–1817). Compiler floor delivered: `abi(.naked)` emission (B1.0) + per-fiber `context` (B1.1,
|
||||
zero-change). Five compiler bugs fixed en route (0151/0152/0153 in B1.2; 0154 in B1.5a;
|
||||
0156-Part1 + 0157 in B1.4a). Deferred (documented, non-blocking): issue 0150 (`Future(void)`/`timeout`),
|
||||
0155 (scalar-pointer index), 0156-Part2 (deferred `..` spread); a linux `epoll` twin of `block_on_fd`;
|
||||
routing the suspending async through the erased `context.io` (M:N evolution); the heap-Task / closure-env
|
||||
/ kq-fd leaks (bounded, default-GPA-invisible). Stream B2 (channels/cancel/stdlib) is the next carve.
|
||||
|
||||
### Earlier — B1.4c — REAL fd-readiness blocking via kqueue (macOS)
|
||||
`library/modules/std/sched.sx` now lets a
|
||||
fiber park on a file descriptor and the run loop block on `kevent` until the kernel reports it ready.
|
||||
Reuses the existing verified `library/modules/std/net/kqueue.sx` bindings (`Kevent` (32 bytes),
|
||||
`kqueue`/`kevent`/`kq_apply`/`kq_wait` + the `EVFILT_READ`/`EV_ADD`/`EV_ENABLE`/`EV_ONESHOT`
|
||||
constants) rather than re-deriving the FFI — sched.sx imports it as `kqb`. Added to `Scheduler`:
|
||||
- `kq: i32` (LAZY — `-1` in `init`, opened by the first `block_on_fd`, so a pure-compute /
|
||||
virtual-timer scheduler never opens a kqueue fd; leaks one fd at exit once opened, same class as the
|
||||
documented spawn-env / go-Task leaks — no deinit yet);
|
||||
- `io_waiters: List(IoWaiter)` (`IoWaiter :: struct { fd: i32; fiber: *Fiber; }`, grown through
|
||||
`own_allocator` per the long-lived-container rule);
|
||||
- `block_on_fd(self, fd, want_read)` — lazily opens `kq`, arms a one-shot `EVFILT_READ` registration,
|
||||
records an `IoWaiter{fd, current}`, then `suspend_self()`. Guards a null `current` (loud abort, like
|
||||
`sleep`); `want_read=false` (write-readiness) is not wired yet → loud abort rather than silently
|
||||
arming a read filter.
|
||||
- Run-loop: after the ready queue drains, **Mode 1 (virtual time)** fires the earliest pending timer
|
||||
(takes precedence — a program uses `sleep` OR fds, documented non-unification limitation); **Mode 2
|
||||
(real fd)** — if `io_waiters` is non-empty, BLOCK on `kq_wait(kq, evbuf, MAXEV=16, -1)` (null
|
||||
timeout), then for each fired event match `ev.ident` back to its waiter, evict it, and `wake` the
|
||||
fiber; **else** break. Orphan-deadlock check unchanged in spirit but now correct: an fd waiter is NOT
|
||||
an orphan (while `io_waiters.len > 0` the loop blocks on kqueue rather than reaching the check), and
|
||||
a genuine no-timer/no-fd suspend still aborts loudly (verified with a probe: exit 134).
|
||||
- `wake` now also evicts a pending fd-waiter (`cancel_io_waiter_for`, mirror of `cancel_timer_for`) —
|
||||
same UAF reasoning: a fiber woken by another path must not leave a stale `IoWaiter` pointing at a
|
||||
reaped `*Fiber`. The kqueue registration is `EV_ONESHOT` so we never `EV_DELETE` (a never-fired
|
||||
one-shot lingers harmlessly; the drain ignores an unmatched ident; closing the fd auto-removes it).
|
||||
- DE-RISK probe (run first, no scheduler): confirmed `size_of(Kevent) == 32`, the pipe roundtrip
|
||||
(`kq_wait` returned 1 with `out.ident == read_fd`, `out.filter == -1` (EVFILT_READ), `out.data == 1`
|
||||
byte readable) — the struct layout reads back the fd correctly.
|
||||
- Locked by `examples/concurrency/1816-concurrency-fiber-io-pipe.sx`: a `pipe`; a reader fiber spawned
|
||||
FIRST blocks on the empty read end, then a writer fiber writes `a b c` → the run loop blocks on
|
||||
kqueue, wakes the reader, which reads the 3 bytes. Output `log: wrote read 3 [97 98 99]` /
|
||||
`n_suspended: 0` (the "wrote" before "read" ordering proves the reader actually blocked then woke via
|
||||
kqueue readiness). `.build` `{ "target": "macos" }` (matches host arch → runs end-to-end; ir-only on
|
||||
a mismatch, like 1814/1815 — no `.ir` snapshot needed since it runs here). The example declares its
|
||||
own `read`/`write`/`close` externs with the CANONICAL signatures std already binds
|
||||
(`(i32,[*]u8,usize)->isize` / `(i32)->i32`) — a divergent re-binding is rejected by the extern dedupe.
|
||||
- **Adversarial review (worker) of the run-loop change — found 2 CRITICALs:**
|
||||
- **(1) two fibers on the SAME fd → lost wakeup + permanent hang.** macOS `EV_ADD` for an existing
|
||||
`(ident, filter)` REPLACES the registration (doesn't stack), so two waiters share one registration:
|
||||
the fd fires once, one wakes, the other is stranded in `io_waiters` and the next `kq_wait(-1)` blocks
|
||||
forever. FIXED: `block_on_fd` now enforces one-waiter-per-fd with a loud abort (the model already
|
||||
assumed it). Verified: dup-fd → `sched: block_on_fd: fd N already has a waiter`, not a hang.
|
||||
- **(2) an fd-waiter on a never-ready fd hangs instead of the timer path's loud abort.** Re-examined:
|
||||
this is CORRECT event-loop semantics — blocking on I/O until ready (possibly forever, like a server
|
||||
idling on a socket) is the point; the scheduler cannot know an fd will never become ready, so it must
|
||||
keep waiting. NOT a scheduler deadlock. Fixed the MISLEADING comment that implied the orphan check
|
||||
covers fd-waiters: it does not, by design (it covers only pure `suspend_self` parks). No code change —
|
||||
the "hang" is a caller-side logic issue (waiting on input that never arrives), not a bug to abort on.
|
||||
- Review CLEARED: the IoWaiter UAF parity (early-wake evicts the waiter; a lingering one-shot that later
|
||||
fires hits no match → clean no-op), ident width/sign, `kq_wait` EINTR/error handling, timer-vs-io
|
||||
precedence (timer wins; no hang). All probed safe.
|
||||
- Suite GREEN **754/0** (incl. the dup-fd guard, no new example needed — the abort is host-fragile to
|
||||
pin like 1809's guard-firing). Next: **B1.5** (end-to-end M:1 validation under the deterministic timers
|
||||
/ fd readiness); a linux epoll twin of `block_on_fd` (mirror via `std/net/epoll`, the OS-neutral facade
|
||||
is `std.event`) is future work.
|
||||
|
||||
### Earlier — B1.4b — deterministic VIRTUAL-TIME timer scheduling (the KEYSTONE) — landed + adversarially
|
||||
reviewed (caught a CRITICAL UAF, fixed).
|
||||
`library/modules/std/sched.sx` gained a virtual clock +
|
||||
sleep timers so fibers schedule in reproducible simulated time (no real clock): `clock_ms` (advances
|
||||
ONLY as timers fire), a `timers: List(Timer)` (insertion-order, linear min-scan, FIFO tiebreak),
|
||||
`now_ms()`, `sleep(ms)` (arm `{clock_ms+ms, current}` + `suspend_self`), and a timer-driven `run`
|
||||
(drain ready → fire earliest timer → advance clock → wake → repeat; orphan-deadlock check preserved
|
||||
for a genuine no-timer suspend). Locked by `1814` (5 fibers sleep 30/10/20/15/15 → wake order
|
||||
B@10, D@15, E@15 (FIFO), C@20, A@30 — deadline order, not spawn order; `now_ms()` reads each virtual
|
||||
deadline; final clock 30). §8.1.3 calibration note in the header: the deterministic wake ORDER
|
||||
equals what real `sleep`s produce, reproducing blocking semantics' observable ordering without real
|
||||
time. The deterministic-sim `Io` is realized at the scheduler level (`sleep`/`now_ms`/timer-`run`),
|
||||
not as an erased `Io`-protocol impl (same erasure reason as FiberIo).
|
||||
- **Adversarial review (worker) of the run-loop change: found a CRITICAL use-after-free** — a fiber
|
||||
that armed a `sleep` timer but was woken EARLY by another path (a manual/`Task` `wake`) ran to
|
||||
completion + was reaped (stack `munmap`'d, `Fiber` freed) while its `Timer` still held a dangling
|
||||
`*Fiber`; a later fire would `wake` freed memory (silent-corruption: "passes" only because the
|
||||
freed slot coincidentally read `state != .suspended`). FIXED: `wake` now evicts the woken fiber's
|
||||
pending timer (`cancel_timer_for`) — every re-ready path funnels through `wake` (the timer-fire in
|
||||
`run` already removed the fired timer, so it's a harmless re-scan there), so no stale timer can
|
||||
outlive its fiber. Regression `1815-concurrency-fiber-timer-early-wake.sx` (early wake → `clock: 0`,
|
||||
the stale 100ms timer evicted, not fired). Review CLEARED: `n_suspended` accounting,
|
||||
orphan-deadlock false-positives, timer-list integrity (re-arm during fire), clock monotonicity,
|
||||
termination — all traced/probed safe.
|
||||
- Suite GREEN (count below). Next: **B1.4c** (event-loop `Io` — real fd readiness, kqueue/epoll).
|
||||
|
||||
### Earlier — B1.4a — a truly-SUSPENDING fiber-task async layer (`go`/`wait`/`cancel`)
|
||||
landed + adversarially reviewed; cleared two more compiler blockers en route. `library/modules/std/sched.sx`
|
||||
now carries `Task($R)` + `Scheduler.go(work) -> *Task($R)` + `wait`/`cancel` (a `ufcs` layer over
|
||||
the M:1 scheduler). `s.go(work)` runs the nullary thunk `work` as a REAL fiber; `t.wait()` SUSPENDS
|
||||
the caller until it completes (vs io.sx's blocking `context.io.async`, which runs inline). Locked by
|
||||
`examples/concurrency/1813-concurrency-fiber-async-suspend.sx`: two tasks interleave (A yields
|
||||
mid-body so B runs first → `1 2 3`), awaited values `42`/`100`, and a canceled task's `wait` raises
|
||||
`.Canceled` → `or -99` → `sequence: 1 2 3 42 100 -99`.
|
||||
- **Design: a NULLARY thunk, not `async(worker, ..args)`.** A comptime variadic pack can't cross a
|
||||
deferred (fiber) boundary — `..args` captured into a closure re-expands from the spawner's
|
||||
now-gone locals (issue 0156 Part 2). So `go` takes `work: Closure() -> $R`; the user captures
|
||||
inputs in the lambda at the call site (the `go func(){…}()` idiom). **Self-contained in sched.sx**
|
||||
(NOT io.sx): io.sx importing sched.sx duplicates the `_fib_tramp` global asm when a program also
|
||||
imports sched.sx directly (global asm emits per import-path) — so the Io-protocol
|
||||
`spawn_raw`/`suspend_raw`/`ready` hooks stay reserved for the future M:N model; M:1 uses
|
||||
`go`/`wait` directly. Heap `*Task` (must outlive `go`'s frame; leak documented). `TaskErr` is
|
||||
LOCAL (the `!` failable detection doesn't see through io.sx's `IoErr` re-export alias).
|
||||
- **Two compiler blockers hit + FIXED (user-authorized in-session):**
|
||||
- **issue 0156 Part 1** — a single-type generic `$R` (parsed as `comptime_pack_ref`) used as a
|
||||
type-arg (`Box($R)`, `size_of(Box($R))`) inside a pack-fn body hit a missing arm in
|
||||
`resolveTypeWithBindings` → `.unresolved` → LLVM panic. Fix: mirror `resolveTypeArg`'s
|
||||
`comptime_pack_ref` arm (look up `type_bindings`, else a loud diagnostic). Regression
|
||||
`examples/generics/0216-generics-typearg-in-pack-fn-body.sx`. (Part 2 — deferred `..` spread
|
||||
crashes — reframed OPEN/non-blocking, `issues/0156`.)
|
||||
- **issue 0157** — a user generic `ufcs` method whose name collides with a stdlib re-export
|
||||
(`cancel` on `*Task` vs io.sx's `cancel` on `*Future`) resolved via last-wins `fn_ast_map` with
|
||||
NO receiver filtering → wrong overload → `$R` unbound → LLVM panic. Fix
|
||||
(`src/ir/lower/call.zig` `selectUfcsGenericByReceiver`): every generic-ufcs dispatch enumerates
|
||||
ALL module authors (`module_decls`), keeps receiver-binding ones, picks the most
|
||||
receiver-SPECIFIC (concrete > bare `$T`), dedups re-exports, and flags a genuine 2-specific tie
|
||||
as a deterministic "ambiguous — qualify" diagnostic (never a silent order-dependent pick).
|
||||
Regression `examples/generics/0217-generics-ufcs-method-name-collides-stdlib.sx`.
|
||||
- **Adversarial review (worker) of the 0157 fix + Task layer.** Caught the determinism CRITICAL
|
||||
(fixed: always-run selection + specificity + ambiguity), `wait`-outside-a-fiber null-deref (fixed:
|
||||
loud guard in `suspend_self`/`yield_now`), and cancel-doesn't-skip-work (fixed: worker skips
|
||||
`work()` if already canceled). Lost-wakeup / cancel-after-complete / reap traced safe. Also
|
||||
simplified `1812` (`**Fiber` shared handle → a `Sh.parked` field; output identical).
|
||||
- Suite GREEN 751/0 (749 + 1813 + 0217). Next: **B1.4b** (deterministic-sim `Io`).
|
||||
|
||||
### Earlier — B1.5a — the M:1 cooperative fiber scheduler CORE — landed + adversarially reviewed
|
||||
The hand-bootstrapped ping-pong (1807-1810) is now a reusable scheduler API in pure sx:
|
||||
`library/modules/std/sched.sx` — a generic `Fiber` (`body: Closure() -> void`) + `Scheduler`
|
||||
with `init`/`spawn`/`yield_now`/`suspend_self`/`wake`/`run` over the proven `swap_context` on
|
||||
guarded `mmap` stacks. The ONE generic dispatch (`fib_dispatch`, reached from the `_fib_tramp`
|
||||
trampoline) runs ANY stored closure body on a fresh stack — replacing the fixed `bl _fib_body`.
|
||||
Reaping `munmap`s the stack + frees the heap `Fiber` on completion; an intrusive FIFO gives
|
||||
round-robin order.
|
||||
- **Foundational design de-risked by probe before building:** a fiber can store + call a
|
||||
`Closure() -> void` on its fresh stack via the generic dispatch; outputs flow OUT through
|
||||
pointers captured in the closure (capture-by-value does NOT write back — pushed onto the user).
|
||||
- **Hit + FIXED a blocker compiler bug — issue 0154** (user-authorized in-session fix). `null` /
|
||||
`---` assigned to a struct field picked up a leaked enclosing `target_type` (the function's
|
||||
RETURN type, set for the whole body at decl.zig:2691) and built a WHOLE-STRUCT-typed null →
|
||||
an oversized `zeroinitializer` store through the field's GEP that overran the field's slot and
|
||||
clobbered the saved x29/x30, so the fn `ret`'d to 0x0. This was EXACTLY the `Scheduler.init()`
|
||||
by-value-return shape (`sched_ctx: [13]u64` before `current: *Fiber`). Fix: added
|
||||
`.null_literal, .undef_literal` to the `needs_target` switch in `lowerAssignment`
|
||||
(`src/ir/lower/stmt.zig`) so the field's type is used. Repro → regression test
|
||||
`examples/types/0193-types-sret-array-before-pointer.sx`; `issues/0154-*.md` RESOLVED.
|
||||
- **Adversarial review (worker): asm/bootstrap/lifetime SOUND** (the headline closure-env-lifetime
|
||||
fear was disproven — envs are heap-promoted, survive the spawn frame). Found **1 CRITICAL** +
|
||||
robustness gaps, ALL hardened: (CRITICAL) `wake` re-enqueued an already-queued fiber →
|
||||
FIFO corruption/segfault → now GUARDED on `.suspended` (spurious/double/stale wake = safe
|
||||
no-op); orphan-suspend leak/deadlock → `n_suspended` accounting + a loud `run()`-drain
|
||||
diagnostic+abort; `mmap` `MAP_FAILED` (=-1, not null) / `mprotect` / Fiber-OOM → loud bails
|
||||
(per §8.1.1 the guard is mandatory); the per-fiber closure-env leak (sx exposes no env-free) →
|
||||
documented as a KNOWN LIMITATION (bounded by spawn count; invisible under the default GPA).
|
||||
- **Locked two `18xx` examples** (aarch64-macos `.build`-pinned, ir-only on a mismatch):
|
||||
`1811-concurrency-fiber-scheduler.sx` (3 fibers round-robin via `yield_now` → ordering contract
|
||||
`sequence: 0 1 2 0 1 2 0 1 2`, all `.done`) + `1812-concurrency-fiber-suspend-wake.sx` (park via
|
||||
`suspend_self`, resumed by another fiber's `wake`, + the spurious-wake no-op — the CRITICAL-fix
|
||||
regression → `log: 10 20 21 11` / `suspended-left: 0`).
|
||||
- **Filed issue 0155 (NON-blocking, NOT fixed)** — found incidentally in the review: indexing a
|
||||
scalar pointer (`pc[0]`, `pc: *i64`) panics codegen (`.unresolved` reaching LLVM emission). The
|
||||
scheduler uses array-field indexing + `.*`, never this, so it's filed for its own session.
|
||||
- Suite GREEN **748/0** (746 base + 1811 + 1812 + 0193 regression). Next: **B1.4a** (FiberIo —
|
||||
wire `Io.spawn_raw`/`suspend_raw`/`ready` onto the scheduler so `async`/`await` truly suspend).
|
||||
|
||||
### Earlier — B1.3b-1 — the x86_64 / Win64 `swap_context` sibling — VALIDATED on real hardware
|
||||
The
|
||||
context switch is now proven on a SECOND architecture + ABI. A Win64 `swap_context` saves the
|
||||
COMPLETE Win64 callee-saved set — 8 GP (rbx, rbp, rdi, rsi, r12-r15) + rsp **and xmm6-xmm15**
|
||||
(10 XMM, 128-bit via `movups` — Win64 has callee-saved XMM, unlike SysV/aarch64) — plus a Win64
|
||||
`scribble_verify` (32-byte shadow + 16-align at each `call`, COFF symbols, rsp-carried return
|
||||
addr). Locked by `examples/1810-concurrency-fiber-switch-win64.sx` (pinned `x86_64-windows-gnu`,
|
||||
ir-only here): the 2-fiber mutual scribble printed **`0 0 P`** when built `--target
|
||||
x86_64-windows-gnu --self-contained` and **run on a Windows 7 x64 VM (UTM)** — every GP + XMM
|
||||
callee-saved survived. **Adversarially reviewed before the VM run** (worker emitted the real `.s`
|
||||
and verified every `call` alignment, the 264-byte frame offsets, the rsp/return-addr round-trip,
|
||||
swap ordering, and COFF naming against the Win64 ABI — no critical/minor bugs). The build→VM→run
|
||||
loop was set up this session (cross-build needs `--self-contained`; output via the Win32
|
||||
`WriteFile` boundary, the 1660 pattern). Suite green. Note: this is the GOOD-swap-only mutual
|
||||
scribble (self-validating by construction; the in-process negative control was dropped to avoid an
|
||||
sx fn-ptr-convention rabbit hole — the detection of this exact logic was negative-controlled on
|
||||
aarch64 in 1808). The SysV/Linux x86_64 sibling (different reg set: no callee-saved XMM, args
|
||||
rdi/rsi) remains for a Linux x86_64 host.
|
||||
|
||||
### Earlier — B1.3b-2 — mmap guard-page stacks (commit `dd532ab`)
|
||||
Fiber stacks are `mmap`'d with a `PROT_NONE` GUARD PAGE at the low end (§8.1.1: a
|
||||
fixed stack without a guard silently corrupts neighbors on overflow). `mmap` the `[guard |
|
||||
usable]` region, `mprotect` the low 16KB page `PROT_NONE`; SP descends into the guard and faults
|
||||
loudly at the boundary instead of corrupting a neighbor. Locked by
|
||||
`examples/1809-concurrency-fiber-guard-stack.sx` (aarch64-macos-pinned): `guard armed: 1`
|
||||
(`mprotect`→0) + `sum: 20100` (a fiber runs real recursion on the guarded stack + yields).
|
||||
- **Guard FIRING validated** (manually, not corpus-pinned — a deliberate overflow crash is
|
||||
host-fragile): a fiber recursing past its 128KB stack faults with `Bus error` at the guard page
|
||||
(`region+GUARD`); the sx crash handler turns it into exit 134. Documented in the example header.
|
||||
- **x86_64 sibling:** was deferred here (couldn't run x86_64 on this arm64 host), then DONE as
|
||||
Win64 once a Windows 7 x64 VM became available — see B1.3b-1 above (`examples/1810`, `0 0 P`).
|
||||
|
||||
### Earlier — B1.3a-2 — the context-switch STRESS GATE (design §10.7) — DONE + adversarially reviewed
|
||||
The explicit every-callee-saved-register scribble that B1.3a-1 owed. `swap_context` now saves the
|
||||
COMPLETE AAPCS64 callee-saved set — integer x19-x28 + fp/lr + sp AND FP **d8-d15** (per §6.1.2
|
||||
only the low 64 bits of v8-v15 are callee-saved, so `d8-d15` is exactly sufficient; x18 is Apple's
|
||||
reserved platform reg, untouched). A naked `scribble_verify(self_ctx, peer, base)` loads a unique
|
||||
sentinel into all 18 callee-saved regs, yields, and on resume counts the ones that didn't survive
|
||||
(honoring its own caller ABI via a 176-byte frame that saves+restores the caller's callee-saved;
|
||||
base reloaded from the frame post-swap; the original lr round-trips through the swap). The gate is
|
||||
a **2-fiber MUTUAL scribble** (A and B scribble DISTINCT sentinels into the same physical regs, so
|
||||
each survives only if `swap_context` saved+restored it — a lone fiber yielding to an idle peer
|
||||
would NOT exercise preservation). Locked by `examples/1808-concurrency-fiber-switch-stress.sx`
|
||||
(aarch64-pinned): `A mismatches: 0` / `B mismatches: 0`.
|
||||
- **Validity proven by NEGATIVE controls:** dropping the d8-d15 save/restore → 8/8 mismatches
|
||||
(exactly the FP regs); dropping x27/x28 → 2/2. The gate genuinely catches a broken switch.
|
||||
- **Adversarial review (worker, per the plan): no CRITICAL bugs.** Verified the callee-saved set
|
||||
is complete + correct, all frame offsets/16-alignment, the lr/sp dance, and swap read-ordering
|
||||
against AAPCS64. Applied its one recommendation: `boot` now zeroes the FP ctx slots [13..20] so a
|
||||
first switch-to loads 0 (not garbage) into d8-d15. Residual gaps it flagged (all spec-correct
|
||||
for a call-boundary swap, documented in the example header): NZCV/FPSR not swapped; **FPCR**
|
||||
(rounding mode — thread-global, bleeds across fibers if changed) and **TPIDR_EL0/TLS** (errno,
|
||||
allocator thread-caches — shared by same-thread fibers) not swapped; fp=0 bootstrap blocks
|
||||
unwind/signal walking past a fiber trampoline. These bite at the N×M:1 / signals stages, not the
|
||||
single-thread switch.
|
||||
- Suite green **734/0**, master clean. WIP probes: `.sx-tmp/scribble2.sx` (+ `_broken`/`_gp`).
|
||||
|
||||
### Earlier — B1.3a-1 — the foundational stackful context switch (commit `b234b7d`)
|
||||
Pure sx over `abi(.naked)`: naked `swap_context` (GP-only 13-slot save) + by-hand fiber bootstrap
|
||||
(SP = `alloc_bytes` stack top, LR = global-asm trampoline, x19 = `*Fiber`). Locked by
|
||||
`examples/1807-concurrency-fiber-context-switch.sx`: 2-fiber ping-pong (`rounds: 6` / `canary
|
||||
fails: 0`) + 64-frame deep recursion (`frames verified: 64` / `depth fails: 0`). Indirect
|
||||
register/stack survival; 1808 supersedes its switch with the complete GP+FP save area + the
|
||||
explicit gate.
|
||||
|
||||
### Earlier — B1.2 COMPLETE — the async surface works end-to-end
|
||||
All three surface blockers (0151, 0152, 0153) FIXED + committed; async examples landed + green.
|
||||
- **0151 fixed** (`362674f`): generic `$T` infers through generic-struct / pointer / UFCS-pack
|
||||
params. Regression `0214` + `0215`.
|
||||
- **0152 fixed** (`e5586f6`): `Atomic(bool)` load/store byte-promoted to `i8` in the codegen
|
||||
emitters. Regression `1705`.
|
||||
- **0153 fixed** (`68c1991`): `inferGenericReturnType` now pins return-type resolution to the
|
||||
fn's DEFINING module (mirroring `monomorphizeFunction`), so a re-exported value-failable's
|
||||
`!E` resolves to the real `.error_set` TypeId — the failable channel survives the re-export
|
||||
alias. Regression `1058-errors-reexport-value-failable-channel.sx`.
|
||||
- **Async examples landed:** `examples/1805-concurrency-io-blocking-async.sx`
|
||||
(`context.io.async((a,b)->i64 => a+b, 40, 2).await() or {…}` → `sum: 42` / `double: 42` /
|
||||
`clock ok`) + `examples/1806-concurrency-io-cancel.sx` (`f.cancel()` → `await` raises
|
||||
`.Canceled` → `or` default; `ok: 7` / `canceled: -99`). Both green, snapshots captured.
|
||||
|
||||
### Earlier — the three B1.2 surface fixes (committed)
|
||||
Generic `$T` inference, `Atomic(bool)` byte-promotion, and re-export failable-channel pin —
|
||||
details below.
|
||||
- **0151 fix (committed):** four gaps closed on the inference + UFCS-dispatch path —
|
||||
(1) `extractTypeParam`/`matchTypeParam(Static)` got a `parameterized_type_expr` arm
|
||||
(recover the arg instance's recorded per-param bindings via `struct_instance_bindings` +
|
||||
the template's ordered `type_params`, recurse positionally; this also fixes `*Box($T)` —
|
||||
it recurses into its `Box($T)` pointee); (2) the `pointer_type_expr` arm now falls through
|
||||
to match the pointee against a non-pointer arg (auto-address-of: a `*Box($T)` param accepts
|
||||
a by-value `Box($T)`, e.g. a UFCS receiver `b.m()`); (3) `ExprTyper.inferType` got a
|
||||
`.lambda` arm building the closure type from the lambda's annotations (the UFCS binder types
|
||||
args from the raw AST before they're lowered, so it can now bind `Closure(..) -> $R` from
|
||||
the worker's declared return type); (4) a pack UFCS target routes through the SAME
|
||||
`lowerPackFnCall` the direct call uses, with the receiver spliced in as `args[0]`.
|
||||
- Regression tests: `examples/0214-generics-ufcs-closure-return-pack.sx` (direct + UFCS
|
||||
closure-return pack) + `examples/0215-generics-infer-through-pointer.sx` (by-value /
|
||||
pointer / multi-param / nested / UFCS-auto-ref struct-head inference). Issue 0151 marked
|
||||
RESOLVED; repro moved into the suite.
|
||||
|
||||
### Earlier — B1.2 (Io capability) — LANDED + adversarially reviewed
|
||||
Commits `a1b14f0` (lock) + `45d869d` (Io capability) + `3eeb965` (issue 0151 lock).
|
||||
- **LANDED + review-confirmed correct** (commit 45d869d): `Io :: protocol #inline`
|
||||
(spawn_raw/suspend_raw/ready/poll/now_ms/arm_timer) + `io` field on `Context`
|
||||
(`{allocator; data; io}`, io LAST); BOTH `__sx_default_context` materializers
|
||||
(protocol.zig + comptime_vm.zig) build an identical CBlockingIo→Io vtable (review verified
|
||||
byte-for-byte agreement; `context.io.now_ms()` dispatches at runtime AND comptime); the
|
||||
`push Context.{…}` omitted-field-**inherits-ambient** fix (review: correct, right fix, no
|
||||
bad blast radius); `library/modules/std/io.sx` (`Future($R)`, `CBlockingIo`,
|
||||
`async`/`await`/`cancel`); the `!`-protocol-impl-lint suppression; 37 `.ir` regens
|
||||
(review: pure layout/type-table, no error text, zero .exit/.stdout/.stderr change).
|
||||
- **BLOCKED — async surface non-functional:** `await`/`cancel` take `*Future($R)` and are
|
||||
**uncallable in EVERY form** (not just UFCS) — sx can't infer a generic `$T` from a
|
||||
pointer-wrapped arg (`*Future($R)`). `async(...)` (create) works via explicit call and
|
||||
produces a correct `.ready` Future, but you can't `await` it. Root bug = **issue 0151
|
||||
(WIDENED)**: infer `$T` from `*T`-wrapped params + closure-return-via-pack + UFCS dispatch.
|
||||
Minimal repro: `unbox :: (b: *Box($T)) -> $T` fails to infer `T`.
|
||||
- **No async example in the corpus** (1805 was removed because it needs the blocked surface)
|
||||
→ the green suite does NOT cover async. Restore `1805` (async/await) + add `1806` (cancel)
|
||||
once 0151 is fixed.
|
||||
|
||||
### Earlier — B1.1 (per-fiber `context` root) — DONE. Zero compiler change (confirmed by probe).
|
||||
The fiber-spawn context convention works end-to-end with ordinary language features:
|
||||
- `snap := context` captures the spawner's `Context` as a value;
|
||||
- the snapshot is stored in a struct (the stand-in `Fiber`);
|
||||
- a trampoline running under a *different* ambient context installs the fiber's stored root
|
||||
with `push f.root { … }`, and the body reads the snapshot — not the trampoline's ambient
|
||||
context — because `context` is an implicit slot-0 `*Context` param (call-carried, rides the
|
||||
callee's own stack) and `push` allocates on the caller frame (no global, no TLS).
|
||||
- Locked by `examples/1804-concurrency-context-snapshot.sx`: prints `fiber root: 42` (the
|
||||
installed snapshot wins over ambient 99) + `ambient after: 99` (the `push` scope restores
|
||||
the ambient context on exit). No fiber runtime yet (that's B1.3) — this proves the plumbing
|
||||
it will build on. No `.build` pin (pure sx, host-independent).
|
||||
- **Probe result:** the design doc's "lower as swappable indirection, never raw TLS" guarded
|
||||
a non-problem — context was already param-carried, never TLS. No path re-reads
|
||||
`__sx_default_context` mid-stack, so there is **no compiler obligation** here.
|
||||
- `zig build && zig build test` green: **726 ran, 0 failed**.
|
||||
|
||||
### Earlier — B1.0 (`abi(.naked)` codegen) — complete
|
||||
Replaced the emit bail with real LLVM `naked` emission:
|
||||
- `emit_llvm` declaration pass: for `func.is_naked`, add the LLVM `naked` + `noinline` +
|
||||
`nounwind` attributes and **skip** the `frame-pointer=all` attribute (incompatible with a
|
||||
frameless function). Pass 2 now emits the `.naked` body normally — `naked` makes the
|
||||
backend emit it verbatim (the inline asm + its own `ret`) with no prologue/epilogue.
|
||||
- IR shape (verified): `; Function Attrs: naked noinline nounwind` / `define internal i64
|
||||
@answer() #0 { entry: call void asm sideeffect "…ret…", ""() unreachable }` /
|
||||
`attributes #0 = { naked noinline nounwind }`. The caller invokes it as an ordinary
|
||||
`() -> i64` call (`.naked` is `call_conv == .default`).
|
||||
- `examples/1800-concurrency-naked-asm.sx` — now GREEN, aarch64-pinned (`.build {"target":
|
||||
"macos"}`): runs end-to-end → **exit 42** on this host, ir-only on a mismatch; `.ir`
|
||||
snapshot captured.
|
||||
- `examples/1801-concurrency-naked-generic.sx` (renamed from `-bail`) — the generic `.naked`
|
||||
now emits a correct naked `answer__i64` (exit 42), proving generic.zig produces a naked
|
||||
body, not a framed one. aarch64-pinned.
|
||||
- `examples/1802-concurrency-naked-asm-x86.sx` — x86_64 cross sibling (`.build {"target":
|
||||
"x86_64-linux"}`, ir-only here): `.ir` locks `naked` + `movl $42, %eax` / `ret`.
|
||||
- Unit test `emit: abi(.naked) function gets the naked attribute (no frame-pointer)` in
|
||||
`emit_llvm.test.zig` (asserts `naked` present, `frame-pointer` absent).
|
||||
- **B1.0c (review-hardening):** a param-bearing `.naked` fn emitted invalid LLVM (loud
|
||||
verifier error "cannot use argument of naked function") because the param-alloca loop
|
||||
wasn't gated. Fixed forward (this *enables* the B1.3 context-switch use case rather than
|
||||
rejecting it): gated the param-alloca loop on `fd.abi != .naked` in decl.zig (both paths) +
|
||||
generic.zig; a naked fn's args stay in registers (read by asm), declared-but-unused in
|
||||
LLVM. Locked by `examples/1803-concurrency-naked-asm-param.sx` (`add(a,b)` → x0+x1 → 42).
|
||||
- `zig build && zig build test` green: **725 ran, 0 failed** + unit tests.
|
||||
|
||||
### Earlier — B1.0a (lock + review hardening)
|
||||
Plumbed `Function.is_naked` (set from `fd.abi == .naked` at both decl sites + generic.zig +
|
||||
pack.zig); `funcWantsImplicitCtx` skips `.naked` (no synthetic ctx, like `.c`); all
|
||||
body-lowering paths bypass `lowerValueBody` for `.naked` (asm body + `unreachable` cap — no sx
|
||||
return); `emit_llvm` Pass 2 bailed loudly (since flipped to real emission). Adversarial
|
||||
review caught the generic/pack `is_naked` gap (a generic `.naked` silently shipped a framed
|
||||
body); closed + locked. The review's `.naked`-lambda CRITICAL was a false positive
|
||||
(unparseable — `isLambda` breaks on the `abi` keyword).
|
||||
|
||||
## Current state
|
||||
**STREAM B1 FEATURE-COMPLETE.** `library/modules/std/sched.sx` is the whole pure-sx M:1 async runtime:
|
||||
the scheduler core (B1.5a: `spawn`/`yield_now`/`suspend_self`/`wake`/`run`), suspending fiber-task
|
||||
async (B1.4a: `Task($R)`/`go`/`wait`/`cancel`), deterministic virtual-time timers (B1.4b:
|
||||
`clock_ms`/`now_ms`/`sleep`, timer-driven `run`), AND real fd readiness via kqueue (B1.4c: lazy `kq`,
|
||||
`io_waiters`, `block_on_fd`, run-loop Mode 2) — all over the `abi(.naked)` `swap_context` on guarded
|
||||
`mmap` stacks (B1.0–B1.3), reusing `std/net/kqueue.sx`. Every park path (timer sleep, fd block, raw
|
||||
suspend) is balanced through `wake` (which evicts stale timer + fd waiters — the UAF guards). A terminal
|
||||
`deinit` (B1 follow-up) closes the previously-documented leaks: heap `Task`s (tracked via `task_allocs`),
|
||||
the `timers`/`io_waiters`/`task_allocs` List backings, and the kqueue fd; the per-`spawn`/`go` closure
|
||||
env remains unfreeable (language limitation). Locked by `18xx` 1800–1820 (naked-asm, context-snapshot,
|
||||
blocking async, the switch + §10.7 stress gate + guarded stacks + Win64 sibling, scheduler round-robin,
|
||||
suspend/wake, async go/wait/cancel, sim-timer ordering, timer early-wake eviction, kqueue pipe I/O, the
|
||||
**1817 end-to-end capstone**, sleep-negative/double-wait guards, and **1820 scheduler-deinit**). Suite
|
||||
GREEN **817/0**, committed. **B1.6: now also runs on aarch64-linux** (epoll fd-backend + comptime-branched
|
||||
`MAP_AP` + naked-fn trampoline) — validated byte-identical to macOS in an Apple `container`.
|
||||
|
||||
Future work (none blocking B1): routing the suspending async through
|
||||
the erased `context.io` (forces sched.sx into every std consumer — deferred to the M:N model, where
|
||||
the `Io` protocol's `spawn_raw`/`suspend_raw`/`ready`/
|
||||
`arm_timer`/`poll` hooks take over); `Future(void)`/`timeout` (issue 0150); freeing the heap-Task /
|
||||
closure-env / kq-fd (a Scheduler `deinit` + closure-env-ownership affordance). **Next carve: Stream
|
||||
B2** (channels / structured cancel / async stdlib) — see PLAN-CHANNELS.md when started.
|
||||
|
||||
### Earlier — B1.5a COMPLETE — the M:1 scheduler CORE exists
|
||||
`library/modules/std/sched.sx` drives N fibers
|
||||
(generic `Closure() -> void` bodies) cooperatively over the proven `swap_context`, on guarded
|
||||
`mmap` stacks: `spawn` / `yield_now` (round-robin) / `suspend_self` + `wake` (off-queue park/resume)
|
||||
/ `run` (drives to drain, reaps on `.done`). Adversarially reviewed + hardened (wake guarded, loud
|
||||
mmap/mprotect/OOM/deadlock bails, env-leak documented). Locked by `1811` (round-robin ordering
|
||||
contract) + `1812` (suspend/wake park-resume + spurious-wake guard). Suite GREEN **748/0**.
|
||||
|
||||
The remaining B1.4 work wires this scheduler under the `Io` capability: **B1.4a (FiberIo)** makes
|
||||
`context.io` route `spawn_raw`/`suspend_raw`/`ready` onto the `Scheduler` so `async`/`await` truly
|
||||
SUSPEND (today's `CBlockingIo` runs the worker to completion inline); **B1.4b** the deterministic-sim
|
||||
`Io` (virtual clock + timer queue, calibrated against blocking — the KEYSTONE test harness);
|
||||
**B1.4c** the event-loop `Io` (kqueue/epoll). Then **B1.5** is the end-to-end M:1 validation under
|
||||
the deterministic `Io`.
|
||||
|
||||
### Earlier — B1.2 COMPLETE
|
||||
The full async surface (Io capability on Context + `async`/`await`/`cancel` +
|
||||
blocking `CBlockingIo`) works end-to-end. Master GREEN (732/0), installed `sx` clean. All four
|
||||
B1.2 surface bugs resolved or deferred:
|
||||
- **0151 fixed** (`362674f`): generic `$T` through generic-struct / pointer / UFCS-pack params.
|
||||
Regression `0214` + `0215`.
|
||||
- **0152 fixed** (`e5586f6`): `Atomic(bool)` byte-promoted to `i8` in the load/store emitters.
|
||||
Regression `1705`.
|
||||
- **0153 fixed** (`68c1991`): `inferGenericReturnType` pins return-type resolution to the fn's
|
||||
defining module, so a re-exported value-failable keeps its `!` channel. Regression `1058`.
|
||||
- Issue **0150** (`void` struct field → SIGTRAP) DEFERRED — only `Future(void)` / `timeout`,
|
||||
which are B1.4.
|
||||
|
||||
The async examples are landed + green: `1805` (`async`/`await` + `now_ms` → `sum: 42` /
|
||||
`double: 42` / `clock ok`) + `1806` (`cancel` → `await` raises `.Canceled` → `or` default).
|
||||
The `18xx` concurrency category now covers naked-asm (1800-1803), context-snapshot (1804), and
|
||||
the async surface (1805-1806).
|
||||
|
||||
### B1.2 Io capability — what is LANDED + verified (commit 45d869d)
|
||||
- `Io :: protocol #inline { spawn_raw; suspend_raw -> !; ready; poll; now_ms; arm_timer; }`
|
||||
in `core.sx` next to `Allocator`, with `SpawnOpts{ pin: PinTarget }` + `ParkToken{ handle }`.
|
||||
Six methods, each justified by a downstream consumer (B1.3-B1.5).
|
||||
- `Context :: struct { allocator; data; io: Io; }` — `io` appended LAST so `allocator` stays
|
||||
index 0 (the `call.zig:1229` hardcode) and `data` keeps index 1 (minimal VM-fallback churn).
|
||||
- Both `__sx_default_context` materializers updated in lockstep + verified: `protocol.zig`
|
||||
`emitDefaultContextGlobal` (extended `ctx_fields` 2→3, built the `CBlockingIo→Io` inline
|
||||
7-word vtable `{null-ctx, fn0..fn5}` via `getOrCreateThunks("Io","CBlockingIo")`) and
|
||||
`comptime_vm.zig` `materializeDefaultContext` fallback (wrote the 6 thunk func-refs at
|
||||
`io_base = addr + 4*ps`, offset `+ (i+1)*ps`). The global path auto-followed the 3-field
|
||||
Context type. **`context.io.now_ms()` printed `clock ok` live — the capability threads + the
|
||||
vtable dispatches correctly.**
|
||||
- Stateless `CBlockingIo :: struct {}` + `impl Io for CBlockingIo` (mirror of `CAllocator`):
|
||||
blocking semantics — `spawn_raw`/`ready`/`poll`/`arm_timer` no-op/0, `now_ms` → `time.mono_ms()`.
|
||||
- **push-inherit-omitted fix** (`stmt.zig` `lowerPush`): a `push Context.{...}` now SEEDS the
|
||||
new slot from the ambient context (load+store), then overwrites ONLY the literal's named
|
||||
fields — so omitted fields (now incl. `io`) are INHERITED, never zero-inited to a null
|
||||
vtable. Eliminates the omitted-field footgun globally (zero per-site churn across the 17
|
||||
partial-literal sites). This is the correct capability-bag semantics; it compiled clean.
|
||||
- **`!`-protocol-method warning fix** (`error_analysis.zig` + a new `Lowering.impl_method_names`
|
||||
set populated in `protocols.zig` `registerImplBlock`): a protocol impl method may be declared
|
||||
`!` by contract (e.g. `Io.suspend_raw`) yet never raise; the "declared `!` but never errors —
|
||||
drop the `!`" hint is a false positive for impl methods, now suppressed for them.
|
||||
|
||||
Status of the blockers that originally stopped B1.2:
|
||||
- **issue 0151 — FIXED this session** (generic `$T` through generic-struct / pointer /
|
||||
UFCS-pack params). `async`/`await`/`cancel` are callable. See "Last completed step".
|
||||
- **issue 0152 — NEW, the current blocker** (`Atomic(bool)` → sub-byte i1 atomic; LLVM reject).
|
||||
Blocks the async examples via `Future.canceled: Atomic(bool)`. Filed; codegen-level fix.
|
||||
- **issue 0150** — `void` struct field SIGTRAP; only `Future(void)`/`timeout` (B1.4). DEFERRED.
|
||||
|
||||
Per the IMPASSABLE STOP rule: 0151 fix shipped (suite green 728/0), 0152 filed, STOPPED.
|
||||
Resume B1.2's async examples once 0152 lands.
|
||||
|
||||
### Earlier — B1.0 + B1.1 complete
|
||||
Stream A (atomics) is feature-complete (✅). Stream B1: **B1.0 + B1.1 complete.** The two
|
||||
compiler-floor preconditions for the fiber runtime are in place: (1) `abi(.naked)` emits a
|
||||
real LLVM `naked` function end-to-end (decl, generic, pack paths) — the context-switch
|
||||
substrate; (2) per-fiber `context` root needs **no compiler change** — the spawn convention
|
||||
(snapshot `context`, store, `push` it from the trampoline) is pure library sx. No
|
||||
fibers/Io/scheduler code yet. Grounded floor facts:
|
||||
- `context` is an implicit slot-0 `*Context` param + `push Context` is a stack `alloca` ⇒
|
||||
**fiber-local for free** (confirmed by the B1.1 probe — never TLS, never re-read from the
|
||||
`__sx_default_context` global mid-stack). A spawn passes the snapshot as the fiber-entry
|
||||
fn's slot-0 ctx via `push f.root { entry(args) }`. Locked by `1804-...-context-snapshot`.
|
||||
- Inline asm works end-to-end (lower→emit→JIT, aarch64 + x86_64) — the `.naked` body reuses it.
|
||||
- **`.naked` with PARAMS works** (B1.0c, the B1.3 substrate): the param-alloca loop is gated
|
||||
on `fd.abi != .naked` in decl.zig (both paths) + generic.zig — a naked fn's args stay in
|
||||
ABI registers (read by the asm body), declared-but-unused in LLVM (verifier-legal).
|
||||
Example `1803-concurrency-naked-asm-param.sx` (`add(a,b)` reads x0/x1). **Unsupported (loud,
|
||||
not silent):** a `.naked` *variadic-pack* fn (pack.zig's param loop is intertwined with
|
||||
comptime-param/`#insert` handling, and a naked fn can't read a runtime-sized pack from
|
||||
registers anyway) → loud LLVM-verifier error for that nonsensical construct. Acceptable
|
||||
boundary; a sharper sx diagnostic for it is a candidate polish, not a blocker.
|
||||
|
||||
## Next step
|
||||
**Stream B1 is COMPLETE — no next step in this stream.** The pure-sx M:1 async runtime is feature-
|
||||
complete and committed (1800–1820 green, 759/0), now WITH a `Scheduler.deinit` closing the bounded
|
||||
leaks. Pick up **Stream B2** (channels / structured cancel / async stdlib) as a fresh carve
|
||||
(PLAN-CHANNELS.md), OR one of the remaining non-blocking follow-ups: the linux `epoll` twin of
|
||||
`block_on_fd`, `Future(void)`/`timeout` (needs issue 0150), or routing the suspending async through the
|
||||
erased `context.io` for the M:N model. (`Scheduler.deinit` — DONE, see Last completed step.) None of
|
||||
these block B1. The closure-env leak survives `deinit` (no language affordance to free a closure env);
|
||||
revisit if/when sx grows closure-env ownership.
|
||||
|
||||
**Deferred (future B1.4c sibling): the linux epoll twin of `block_on_fd`.** B1.4c wired the **macOS
|
||||
kqueue** path only (the host is aarch64-macOS). The linux mirror would register interest via
|
||||
`std/net/epoll` and the OS-neutral facade is `std.event` — keep the two as separate run modes inside
|
||||
`run`, branching on the platform, exactly as the timer-vs-fd modes are kept separate now. Documented
|
||||
non-unification: virtual-time timers and real kqueue timeouts are NOT merged — `run` fires a pending
|
||||
timer before ever blocking on kqueue (a program uses `sleep` OR fds); a true "fd-or-real-timeout" wants
|
||||
a kqueue `EVFILT_TIMER`, future work.
|
||||
|
||||
> **▶ LINUX EPOLL — in progress (2026-06-26), via `std.event.Loop` (the OS-neutral facade).**
|
||||
> Chosen over the sched.sx `block_on_fd` twin because the facade is the named home for epoll, is pure
|
||||
> sx + libc (zero compiler change), is consumed by http.sx, and has a runnable darwin sibling. Landed:
|
||||
> (A) **`library/modules/std/net/epoll.sx`** — raw bindings, the linux twin of `std/net/kqueue.sx`.
|
||||
> `epoll_event` is modelled as an **arch-branched struct** (`{events, data_lo, data_hi}` u32 fields →
|
||||
> 12 B x86_64 packed / 16 B aarch64), so layout is byte-exact with NO packed attribute, NO unaligned
|
||||
> access, NO scalar-pointer indexing (issue 0155) — the struct-per-arch approach the user flagged as
|
||||
> better than raw byte poking. Self-contained (libc only — NO build.sx import; the top-level `inline if
|
||||
> ARCH` resolves via the compiler's flatten pre-pass, keeping the IR small). Locked by
|
||||
> `examples/event/1633-event-epoll-bindings-linux.sx` (ir-only x86_64-linux, durable 244-line .ir;
|
||||
> aarch64 16 B layout also probe-verified). (B) **`std.event.Loop` branched on `inline if OS`** into two
|
||||
> top-level OS-selected structs (sx has no conditional struct fields): the kqueue Loop unchanged
|
||||
> (darwin, runs — 1632 green), a new epoll Loop (linux) with the per-fd registration table (combined
|
||||
> EPOLLIN/OUT mask via ADD/MOD/DEL), eventfd wake channel, and EPOLLRDHUP→eof. **RUNTIME-VALIDATED on
|
||||
> real Linux:** a static `aarch64-linux` build of the 1632-equivalent Loop test (+ the eventfd wake path)
|
||||
> ran **6/6 green inside an Apple `container` Linux VM** (kernel 6.18 aarch64) — add_read, idle-timeout,
|
||||
> readable+fd+udata, the MOD-mask add_write path, the eventfd wake channel, and EPOLLRDHUP/HUP eof all
|
||||
> behave identically to kqueue (lone documented difference: `nbytes` is 0 on epoll). Also lowers clean for
|
||||
> both linux arches; the ABI is corpus-locked by 1633. NOT corpus-snapshotted (the corpus runner is
|
||||
> host-based, not container-aware; a Loop example drags the std barrel → ~18k-line brittle IR).
|
||||
> **The epoll deliverable is COMPLETE.** Re-validation recipe in the event.sx VALIDATION note. Optional
|
||||
> follow-on: route sched.sx `block_on_fd` through `std.event` (still needs the linux sched.sx port — mmap
|
||||
> consts, tramp symbol, errno, x86_64 SysV switch).
|
||||
|
||||
> **✅ issue 0192 FIXED (2026-06-26) — epoll work UNBLOCKED.** A qualified-import-member const
|
||||
> (`m.EV_SIZE`) now folds as a compile-time constant in every position the bare/flat form does
|
||||
> (array dim, arithmetic, Vector lane, generic value-param, inline-for) — so the clean
|
||||
> `[MAXEV * ep.EV_SIZE]u8` event buffer the bindings want will work. Fix: a `lookupQualifiedConst`
|
||||
> ctx hook resolving the namespace alias → target module's per-source const, wired into the int/float
|
||||
> const folders (`src/ir/program_index.zig` + `src/ir/lower/comptime.zig`). Regression:
|
||||
> `examples/modules/0842-modules-qualified-import-const-comptime.sx`. The hint stands for the rebuild:
|
||||
> **a struct-per-arch `EpollEvent` (arch-branched u32 fields, 12 B x86_64 / 16 B aarch64) beat raw
|
||||
> byte access** — idiomatic field reads, no issue-0155 scalar-pointer indexing, no unaligned u64.
|
||||
> Resume: rebuild `std/net/epoll.sx`, branch `std.event.Loop` on `inline if OS`, lock with a darwin run
|
||||
> + ir-only linux example.
|
||||
|
||||
> **⛔ (HISTORICAL) BLOCKED on issue 0192 (filed 2026-06-26).** Started the epoll work: chose the `std.event.Loop`
|
||||
> backend (pure sx + libc externs, zero compiler change — per "do this in sx as much as possible") as
|
||||
> the first deliverable, since event.sx already names epoll as its linux backend and it's runnable
|
||||
> (darwin via kqueue) + ir-only-verifiable (linux). De-risked four landmines by probe — arch-dependent
|
||||
> layout const via module-scope `inline if ARCH` (folds + validates in linux IR), slice-based byte access
|
||||
> (sidesteps issue 0155), no unaligned u64 (store the 32-bit fd in epoll `data`), and comptime-dead linux
|
||||
> externs don't break the darwin corpus (just an unreferenced `declare`). Then hit a compiler bug while
|
||||
> sizing the event buffer: a **qualified-import-member const is not a compile-time constant** —
|
||||
> `[m.CAP]u8` / `A :: m.CAP` fail (a *flat*-imported const works). Root cause located:
|
||||
> `evalConstIntExpr` (`src/ir/program_index.zig:325`) has no namespace-member-const arm. Per the STOP
|
||||
> rule the half-built `std/net/epoll.sx` (which used a struct-based layout to route around the bug) was
|
||||
> **removed**, not landed — the unblock session rebuilds it cleanly with the fix in hand. Repro +
|
||||
> investigation prompt: `issues/0192-qualified-import-const-not-comptime.{md,sx}`.
|
||||
|
||||
Design note carried forward: an event-loop `Io` needs a current-`Scheduler` handle. `sched.*` methods
|
||||
thread it via `self`/the `Task`; if B1.4c wants the capability-threaded `context.io` form it'll need
|
||||
an ambient current-scheduler accessor in sched.sx (still deferred — the `sched.*`-method form
|
||||
suffices). The `Io` protocol's `poll`/`arm_timer` map onto this when/if that wiring is built.
|
||||
|
||||
**Side thread (optional, low priority): the SysV/Linux x86_64 sibling.** A THIRD switch variant
|
||||
for `x86_64-linux`: SysV callee-saved = rbx, rbp, r12-r15 + rsp (6 GP + sp; **no** callee-saved
|
||||
XMM, unlike Win64) — a 7-slot ctx, args rdi/rsi/rdx, the rsp-carried return addr. Needs a Linux
|
||||
x86_64 host (or a working cross-run) to RUN + the mutual-scribble gate. Not blocking — the switch
|
||||
is already validated on two arch/ABI pairs.
|
||||
|
||||
**Deferred (do NOT block on these):** issue **0150** (`void` struct field SIGTRAP) — only
|
||||
`Future(void)`/`timeout` (B1.4). The **`::` callable-parameter feature** (named-fn async workers
|
||||
`async(read_a, conn)`) — WIP at `.sx-tmp/wip-callable-params/patch.diff` (parser done, inference
|
||||
incomplete); a dedicated effort; lambda workers are the idiom meanwhile.
|
||||
|
||||
`Context` layout settled: `{ allocator; data; io; }` (allocator index 0 fixed by
|
||||
`call.zig:1229`, io last). Io protocol + materializers + push-inherit are LANDED + reviewed.
|
||||
|
||||
## Known issues / capability gaps
|
||||
- **issue 0157 (OPEN, BLOCKING B1.4a)** — a user-defined generic ufcs method whose NAME collides
|
||||
with a stdlib re-export (`cancel`, re-exported by `std.sx` from `io.sx` as `ufcs (f: *Future($R))`),
|
||||
called via UFCS on a different generic struct (`*Task($R)`), leaves `$R` unresolved → `.unresolved`
|
||||
reaches LLVM emission → panic (`src/backend/llvm/types.zig:196`). Renaming → works; the non-UFCS
|
||||
call form already diagnoses `cannot infer generic type parameter 'R'`, so the UFCS path skips that
|
||||
diagnostic. Surfaced by `cancel :: ufcs (t: *Task($R))` in `std/sched.sx`. Minimal repro (no
|
||||
fibers/closures): `issues/0157-ufcs-generic-method-name-collides-stdlib-unresolved.{md,sx}`.
|
||||
- **✅ issue 0154 — FIXED** (`null`/`---` to a struct field over-stored a whole-struct null when
|
||||
the function's return type leaked as `target_type`, corrupting the frame → `ret` to 0x0;
|
||||
surfaced building `Scheduler.init()`'s by-value return). Fix: `.null_literal`/`.undef_literal`
|
||||
added to `needs_target` in `lowerAssignment` (`src/ir/lower/stmt.zig`). Regression:
|
||||
`examples/types/0193`.
|
||||
- **issue 0155 (OPEN, NON-blocking)** — indexing a scalar pointer (`pc[0]`, `pc: *i64`) panics
|
||||
codegen (`.unresolved` reaching LLVM emission, `src/backend/llvm/types.zig:196`). Found in the
|
||||
B1.5a review; the scheduler doesn't use it (array-field index + `.*` only). Filed for its own
|
||||
session: `issues/0155-scalar-pointer-index-llvm-panic.{md,sx}`.
|
||||
- **✅ issue 0158 — FIXED** — a plain `union` struct-literal (`b : Overlay = .{ f = 3.14 }`) fell
|
||||
through the generic struct-literal path (`getStructFields` empty for a union → malformed
|
||||
`structInit`, overlapping zero-fill clobbered the member → silent `0.0`). Fix: `lowerStructLiteral`
|
||||
detects a plain-union target → new `lowerUnionLiteral` (`src/ir/lower/stmt.zig`) writes each named
|
||||
member into a union-sized slot via the assignment-path lvalue resolver, then loads it back.
|
||||
Single-arm only (one direct member, or same-arm promoted members); overlapping/different-arm/
|
||||
positional literals are diagnosed. specs.md updated. Regressions: `examples/types/0194` +
|
||||
`examples/diagnostics/1191`.
|
||||
- **✅ issue 0157 — FIXED** (B1.4a) — a user generic `ufcs` method whose name collides with a
|
||||
stdlib re-export resolved via last-wins `fn_ast_map` with no receiver filtering → wrong overload →
|
||||
`$R` unbound → LLVM panic. Fix: `selectUfcsGenericByReceiver` (`src/ir/lower/call.zig`) — most
|
||||
receiver-specific binding author across ALL module authors, deterministic, ambiguity-diagnosing.
|
||||
Regression: `examples/generics/0217`.
|
||||
- **✅ issue 0156 Part 1 — FIXED** (B1.4a) — single-type generic `$R` as a type-arg in a pack-fn
|
||||
body (`Box($R)`/`size_of(Box($R))`) → `.unresolved` → panic. Fix: `comptime_pack_ref` arm in
|
||||
`resolveTypeWithBindings`. Regression: `examples/generics/0216`.
|
||||
- **Part 2 (OPEN, NON-blocking)** — a deferred `..` spread (a comptime pack captured into a
|
||||
closure, or a tuple `..t` spread) crashes instead of working/diagnosing. The fiber async layer
|
||||
avoids it by design (nullary thunks), so it's filed for its own session: `issues/0156`.
|
||||
- **Heap leaks in the fiber runtime (documented limitations, NOT bugs):** `spawn`'s closure env +
|
||||
`go`'s heap `Task` are never freed (sx exposes no closure-env free; Task ownership is deferred).
|
||||
Bounded by spawn/go count, invisible under the default GPA. Revisit for a long-running
|
||||
arena-backed scheduler.
|
||||
- **✅ issue 0153 — FIXED** (re-exported generic value-failable `($R, !E)` kept its `!` channel:
|
||||
`inferGenericReturnType` now pins return-type resolution to the fn's defining module).
|
||||
Regression: `examples/1058`. Was the LAST B1.2 surface blocker.
|
||||
- **✅ issue 0152 — FIXED** (`Atomic(bool)` sub-byte i1 atomic → byte-promoted to i8 in the
|
||||
load/store emitters). Regression: `examples/1705`. Unblocked `Future.canceled`.
|
||||
- **✅ issue 0151 — FIXED** (generic `$T` through generic-struct / pointer / UFCS-pack params).
|
||||
Regression: `examples/0214` + `0215`. Was the original B1.2 surface blocker.
|
||||
- **issue 0150** (deferred) — a `void` struct field crashes the compiler (unsized-type SIGTRAP
|
||||
in LLVM `getTypeSizeInBits`). Blocks `Future(void)` → `timeout` (B1.4). Repro: `issues/0150-...`.
|
||||
- (Note: **issue 0149**, filed by another session against an earlier dirty binary, was a
|
||||
manifestation of the pre-fix 0151 — now moot.)
|
||||
- **Orthogonal (not a B1 blocker):** default VALUES for comptime params don't bind on
|
||||
generic-struct methods (free-fn defaults DO work) — inherited from Stream A. Only matters
|
||||
if a B2 lib type wants a defaulted comptime param; atomics/fibers require explicit, so
|
||||
unaffected.
|
||||
- **Issue 0144 (open, independent):** calling an unrecognized bodiless `#builtin` silently
|
||||
returns 0 / exit 0 — a silent-fallback footgun in the generic builtin-call path. Filed;
|
||||
leave for its own fix session unless prioritized. Not a B1 blocker.
|
||||
- **Deferred design gap (documented):** the B1.4 event-loop `Io` does not yet cooperate with
|
||||
a platform UI run loop (CFRunLoop/NSRunLoop/ALooper); pinning gives thread-affinity, not
|
||||
run-loop integration — a §6 app-target concern, out of B1 scope.
|
||||
|
||||
## Decisions (Stream B1 specifics; surface locked in design §4 / §4.6)
|
||||
- **The async runtime is sx LIBRARY code.** The compiler provides only: the general
|
||||
primitives (inline asm ✅, `abi(.naked)` naked [B1.0], atomics ✅) + fiber-safe codegen
|
||||
(`context` already fiber-local — B1.1). Schedulers, fibers, channels, futures, `Io`
|
||||
vtables, `mmap` stacks are all sx.
|
||||
- **`abi(.naked)` is the real spelling of the design's `callconv(.naked)`** — postfix slot,
|
||||
`name :: (sig) -> Ret abi(.naked) { asm { … }; }`. B1.0 = carry it into IR + emit LLVM
|
||||
`naked` + skip prologue/ctx (mirror the existing `.c` skip), NOT extend the enum (it's
|
||||
already there, just inert).
|
||||
- **`.naked` ≠ `.c`:** a `.c` epilogue would restore SP from the wrong stack across a context
|
||||
switch (SP-in ≠ SP-out by design). `.naked` = no prologue/epilogue/frame; the asm emits its
|
||||
own `ret`. This is why the switch must be `.naked`.
|
||||
- **Naming:** sx-facing name is **`naked`** (keyword `abi(.naked)`, field `is_naked`, the
|
||||
diagnostic), matching LLVM's `naked` attribute and the industry term (Zig/Rust/GCC/Clang).
|
||||
The ABI variant was renamed `.pure → .naked` (user direction): "pure" universally means
|
||||
*side-effect-free*, the opposite of a register-clobbering context switch.
|
||||
- **B1.0 snapshot scope:** a `.naked` body is raw per-arch asm; LLVM's `naked` attr text is
|
||||
arch-invariant. **B1.0a** = one host example locked to the emit bail (host-independent —
|
||||
fires before instruction selection; no `.build` pin). **B1.0b** = pin aarch64 + add an
|
||||
x86_64 cross sibling (`.build` target-gated, ir-only on mismatch), like the asm corpus
|
||||
split. The `.ir` proves the `naked` attr + asm emitted, NOT register-save correctness
|
||||
(that's B1.3's stress harness).
|
||||
- **B1.1 — per-fiber context is library-only (CONFIRMED by probe):** push frames are
|
||||
stack-`alloca`'d and the implicit ctx rides slot 0, so the spawn convention — snapshot
|
||||
`context`, store it, `push f.root { entry(args) }` from the trampoline — installs the
|
||||
fiber's root with no compiler change. Verified: the body reads the snapshot over a different
|
||||
ambient context, and `push` restores ambient on exit (`1804-...-context-snapshot`). The
|
||||
design doc's "never raw TLS" guarded a non-problem (context was never TLS).
|
||||
- **Test keystones (design §10):** the **B1.3 switch-stress harness** gates the
|
||||
context-switch (the one piece the deterministic `Io` can't test — §8.1.1, §10.7); the
|
||||
**B1.4 deterministic-sim `Io`** (calibrated against blocking `Io` — §8.1.3) gates all
|
||||
scheduling tests. Both must exist + be calibrated before the async tests they gate are
|
||||
trusted. `18xx` asserts program-emitted ordering contracts, not raw interleaving.
|
||||
|
||||
## Log
|
||||
- **B1.6 — aarch64-linux port of sched.sx.** Comptime-branched the per-OS bits: `MAP_AP` (linux
|
||||
`0x22` / macOS `0x1002`), the fd-readiness backend (epoll on linux, kqueue on darwin — epoll import
|
||||
scoped to the linux branch; `block_on_fd` / run-loop Mode-2 / `cancel_io_waiter_for` each branch,
|
||||
epoll `EPOLL_CTL_DEL`s on fire + early-wake), and the first-entry trampoline (per-OS global-asm
|
||||
symbol → naked sx fn `fib_tramp` + register-indirect `br x20` to `&fib_dispatch` preset in
|
||||
`regs[1]`). **Fixed issue 0193 Bug A:** the tramp redesign bus-errored on 1817 (both OSes) until
|
||||
`export "fib_dispatch"` was restored — without it the fn uses sx's internal ABI (x0 = implicit
|
||||
`context`, `self` → x1) while the trampoline supplies `self` in x0, so the closure loads
|
||||
`regs[1] == &fib_dispatch` as its first capture and recurses forever → stack-overflow bus error.
|
||||
Root cause found via lldb (AOT macOS build) + an adversarial source review. **Bug B** (wrapped
|
||||
top-level `asm` dropped) carved to **issue 0194** (OPEN; no live trigger — the naked-fn tramp
|
||||
sidesteps it). Validated byte-identical on aarch64-macOS host AND aarch64-linux Apple `container`
|
||||
for 1811/1814/1816/1817; full suite GREEN **817/0**.
|
||||
- **B1 follow-up — `Scheduler.deinit`.** Closes the bounded leaks B1 documented. Added a `task_allocs:
|
||||
List(*void)` field (appended in `go` so the scheduler can reach its generic `Task($R)`s) + a canonical
|
||||
`close` extern, then a terminal idempotent `deinit`: reap leftover ready fibers (`munmap` + free) →
|
||||
free tracked Tasks → `List.deinit` the 3 backings → `close` the lazy kqueue fd (reset `-1`). Closure
|
||||
envs stay unfreeable (documented). Probe-observed the accounting under a tracking GPA (deinit drives
|
||||
live allocs 7→3 in a spawn+sleep+2×go run; residual = envs). Locked by
|
||||
`1820-concurrency-fiber-scheduler-deinit.sx` (one run hits timers + kqueue fd + Tasks; `freed by
|
||||
deinit: 5`, `live after deinit: 5` (env residual), `kq open after run: true`→`kq after deinit: -1`,
|
||||
`read: 3 [97 98 99]`), `.build {"target":"macos"}`. Adversarial review: no real UAF/over-free in the
|
||||
supported deinit-after-`run` path; reconciled a doc contradiction (terminal-contract wording); 0154
|
||||
over-store concern probed + cleared (`kq == -1` right after `init`). Suite GREEN **759/0**.
|
||||
- **B1.4c — real fd-readiness blocking via kqueue (macOS).** De-risked first with a no-scheduler probe
|
||||
(confirmed `size_of(Kevent)==32` and the pipe→kevent roundtrip: `kq_wait` returned 1, `out.ident ==
|
||||
read_fd`, `out.filter == -1`, `out.data == 1` — the struct layout reads the fd back correctly). Then
|
||||
added to `library/modules/std/sched.sx` (importing the existing verified `std/net/kqueue.sx` as `kqb`
|
||||
rather than re-deriving the FFI): a lazy `kq: i32` (-1 until first use), `io_waiters: List(IoWaiter)`,
|
||||
`block_on_fd(fd, want_read)` (arm one-shot `EVFILT_READ`, record waiter, `suspend_self`), a run-loop
|
||||
Mode 2 (block on `kq_wait(kq, evbuf, MAXEV=16, -1)` when only fd waiters remain, wake the fiber whose
|
||||
fd fired), and `wake` now also evicts a stale fd-waiter (`cancel_io_waiter_for`, the same UAF guard as
|
||||
`cancel_timer_for`). Timers keep precedence over fds (documented non-unification). Orphan-deadlock
|
||||
check still fires for a genuine no-timer/no-fd suspend (probed: exit 134). Locked by
|
||||
`1816-concurrency-fiber-io-pipe.sx` (reader blocks on empty pipe → writer writes `a b c` → kqueue
|
||||
wakes reader → reads 3 bytes; `log: wrote read 3 [97 98 99]`, `n_suspended: 0`), `.build`
|
||||
`{ "target": "macos" }`, runs end-to-end on host. The example's `read`/`write`/`close` externs use the
|
||||
canonical signatures std already binds (extern-dedupe rejects a divergent re-binding). Suite GREEN
|
||||
**754/0**. Next: B1.5 (end-to-end M:1 validation); linux epoll twin deferred.
|
||||
- **carve** — wrote PLAN-FIBERS.md + CHECKPOINT-FIBERS.md. Grounded the B1 compiler floor:
|
||||
`ABI.naked` inert (type_resolver.zig:237), IR `Function` has no naked flag (inst.zig:605),
|
||||
attribute API pattern (emit_llvm.zig:1339 nounwind), `.c` ctx-skip precedent
|
||||
(decl.zig:515), `push Context` stack-alloca + slot-0 implicit ctx (stmt.zig:1263,
|
||||
lower.zig:259), `__sx_default_context` root (decl.zig:2667/2815), inline-asm corpus
|
||||
(1645/1651). Corrected the design's `callconv(.naked)` → real `abi(.naked)` spelling and
|
||||
the B1.0 snapshot story. B1.1 grounded as likely library-only. Baseline green (721/0).
|
||||
- **B1.0a** — plumbed `Function.is_naked` (set from `fd.abi == .naked` at both decl sites);
|
||||
`funcWantsImplicitCtx` skips `.naked` (no implicit ctx, like `.c`); both body-lowering
|
||||
paths bypass `lowerValueBody` for `.naked` (asm body + `unreachable` cap — no sx return);
|
||||
`emit_llvm` Pass 2 bails loudly on `func.is_naked`. `examples/1800-concurrency-naked-asm.sx`
|
||||
locked to the bail (exit 1 + diagnostic). Suite green (722/0). (ABI variant later renamed
|
||||
`.pure → .naked` — see the Naming decision above — so all `is_*`/`abi(.*)`/example names
|
||||
here read `naked`.)
|
||||
- **B1.0a review-hardening** — adversarial review found generic/pack Function-creation paths
|
||||
left `is_naked` false (silent framed body for a generic `.naked` instance — returned 42 but
|
||||
corrupted the stack). Fixed generic.zig + pack.zig (set `is_naked` + asm-only `unreachable`
|
||||
cap); locked by `examples/1801-concurrency-naked-generic-bail.sx`. The review's `.naked`-
|
||||
lambda CRITICAL was a false positive (unparseable — `isLambda` breaks on `abi`). Suite
|
||||
green (723/0).
|
||||
- **B1.0b** — real `naked` emission: emit_llvm declaration pass adds LLVM `naked`/`noinline`/
|
||||
`nounwind` + skips `frame-pointer` for `func.is_naked`; Pass 2 emits the body verbatim (no
|
||||
prologue). `1800` green aarch64-pinned (exit 42 + `.ir`); renamed `1801` → `-generic`
|
||||
(generic `.naked` emits a naked body, exit 42); added x86_64 sibling `1802` (ir-only, `.ir`
|
||||
locks `naked` + `movl $42, %eax`). Unit test asserts `naked` present + `frame-pointer`
|
||||
absent. Suite green (724/0).
|
||||
- **B1.0c** — review-hardening: param-bearing `.naked` emitted invalid LLVM (loud verifier
|
||||
error). Gated the param-alloca loop on `fd.abi != .naked` (decl.zig both paths + generic.zig)
|
||||
— naked args stay in registers, read by the asm body (the B1.3 context-switch shape).
|
||||
Locked by `examples/1803-concurrency-naked-asm-param.sx`. Pack `.naked` left unsupported
|
||||
(loud, nonsensical). **B1.0 complete.** Suite green (725/0).
|
||||
- **rename** — ABI variant `.pure → .naked` (keyword, `Function.is_naked`, diagnostics,
|
||||
examples 1800-1803 `*-pure-* → *-naked-*`, docs). "pure" universally means side-effect-free
|
||||
— wrong for a register-clobbering switch; "naked" matches LLVM/Zig/Rust/GCC/Clang. Pure
|
||||
cosmetics, no semantic change. Suite green (725/0).
|
||||
- **B1.1** — per-fiber `context` root: **zero compiler change** (probe-confirmed). The spawn
|
||||
convention (snapshot `context` → store in a struct → `push f.root { entry() }` from the
|
||||
trampoline) installs the fiber's root via the implicit slot-0 `*Context` param; the body
|
||||
reads the snapshot, not the trampoline's ambient ctx, and the `push` scope restores ambient
|
||||
on exit. Locked by `examples/1804-concurrency-context-snapshot.sx` (prints `fiber root: 42`
|
||||
/ `ambient after: 99`). Suite green (726/0). **Next: B1.2 (Io interface + context.io).**
|
||||
- **B1.2 (BLOCKED)** — built the full `Io` capability (protocol on `Context`, stateless
|
||||
`CBlockingIo` blocking default, both `__sx_default_context` materializers, push-inherit-omitted
|
||||
fix, `!`-impl-method warning fix) and VERIFIED the core works live (`context.io.now_ms()` →
|
||||
`clock ok`). Two independent compiler bugs blocked the `async`/`await`/`timeout` layer:
|
||||
**0150** (`void` struct field → unsized SIGTRAP, blocks `Future(void)`) and **0151** (type-var
|
||||
from a fn-ptr param's return type not bound in the body, blocks `async`'s `Future(R)`). Both
|
||||
filed with standalone repros + investigation prompts. Per the STOP rule: reverted ALL B1.2
|
||||
working changes (master green again, 726/0; the dirty binary had broken the photo project —
|
||||
see the now-moot 0149), saved WIP to `.sx-tmp/b12-wip/`, STOPPED. Resume after 0150 + 0151.
|
||||
- **0151 FIXED** — generic inference now binds `$T` through a generic-struct param head, a
|
||||
pointer (`*Box($T)`, incl. UFCS auto-ref), and a closure-return-via-pack on the UFCS path.
|
||||
Four gaps closed: `parameterized_type_expr` arm in `extractTypeParam`/`matchTypeParam(Static)`
|
||||
(recovers the arg instance's recorded per-param bindings, recurses positionally); pointer arm
|
||||
falls through to match a value arg (auto-address-of); `ExprTyper.inferType` `.lambda` arm
|
||||
(closure type from annotations — UFCS types args from raw AST pre-lowering); pack UFCS target
|
||||
routes through `lowerPackFnCall` with the receiver spliced in as `args[0]`. Issue 0151 marked
|
||||
RESOLVED; repro → `examples/0214-generics-ufcs-closure-return-pack.sx`; widened cases →
|
||||
`examples/0215-generics-infer-through-pointer.sx`. Suite green 728/0. The now-callable async
|
||||
surface immediately exposed a SEPARATE codegen bug — **issue 0152** (`Atomic(bool)` → sub-byte
|
||||
i1 atomic, LLVM reject; `Future.canceled` hits it). Filed with standalone repro + fix prompt.
|
||||
Per the STOP rule: shipped the 0151 fix, filed 0152, STOPPED. Resume the async examples
|
||||
(1805/1806) after 0152.
|
||||
- **0152 FIXED** — the atomic load/store emitters (`src/backend/llvm/ops.zig`) byte-promote a
|
||||
sub-byte (`bool`→`i1`) access to its `i8` storage type and `trunc`/`zext` the value at the
|
||||
boundary (new `atomicByteType` helper). rmw/cmpxchg left as-is (a `bool` rmw/CAS is rejected
|
||||
at the sx level — integer-only — so a sub-byte element never reaches them; comments record
|
||||
this). Regression `examples/1705-atomics-bool-byte-promoted.sx` (load/store round-trip). Issue
|
||||
0152 marked RESOLVED. Suite green 729/0. With `Atomic(bool)` working, the async surface
|
||||
exposed the TRUE remaining blocker — **issue 0153**: a re-exported generic value-failable
|
||||
`($R, !E)` loses its `!` channel at the call site (the earlier "secondary `or` PHI" symptom
|
||||
was this, NOT an `Atomic` cascade — confirmed it persists after 0152). Narrowed to the
|
||||
generic+re-export co-requirement (non-generic re-export OK; direct generic import OK; only the
|
||||
combination drops `!`). Root cause: the monomorphized return-type's error-set, reached via the
|
||||
re-export alias, resolves to a non-`.error_set` TypeId, so `errorChannelOf`
|
||||
(`lower/error.zig:148`) misses the channel. Filed `issues/0153-...` with a minimal co-located
|
||||
2-file repro + a single-file stdlib-`await` repro + investigation prompt. Per the STOP rule:
|
||||
shipped the 0152 fix, filed 0153, STOPPED. Resume the async examples after 0153.
|
||||
- **0153 FIXED → B1.2 COMPLETE** — `inferGenericReturnType` (`src/ir/generics.zig`) resolved the
|
||||
return-type AST in the CALL-SITE module, so a re-exported error set (`LE :: lib.LE`) resolved
|
||||
to a non-`.error_set` alias and the planned call-result was a plain tuple (channel lost). Fix:
|
||||
pin the source to `fd.body.source_file` around the return-type resolution, exactly as
|
||||
`monomorphizeFunction` does — the `!E` now resolves to the real `.error_set`. One-function
|
||||
change; full suite green (732/0), no regression. Issue 0153 RESOLVED; repro →
|
||||
`examples/1058-errors-reexport-value-failable-channel.sx` (+ companion `lib.sx`). With the
|
||||
channel preserved, landed the async examples: **`1805`** (`async`/`await` + `now_ms` → `sum:
|
||||
42` / `double: 42` / `clock ok`) + **`1806`** (`cancel` → `await` raises `.Canceled` → `or`
|
||||
default; `ok: 7` / `canceled: -99`). **B1.2 (Io capability + M:1 async surface) is COMPLETE.**
|
||||
Next: B1.3 (fiber runtime) on the `.naked` context-switch substrate.
|
||||
- **B1.3a-1 — context switch works.** Implemented the stackful switch in pure sx over
|
||||
`abi(.naked)`: `swap_context(from, to)` (save callee-saved x19-x28 + fp/lr + sp into `*from`,
|
||||
load from `*to`, `ret` onto `to`'s stack) + by-hand fiber bootstrap (SP = top of an
|
||||
`alloc_bytes` stack, LR = a `.global _fib_tramp` global-asm trampoline that does `mov x0, x19;
|
||||
bl _fib_body`, x19 = `*Fiber`). Proven via a probe (main↔fiber), then locked by
|
||||
`examples/1807-concurrency-fiber-context-switch.sx` (aarch64-pinned): a 2-fiber ping-pong
|
||||
(`rounds: 6`, `canary fails: 0` — a per-fiber stack canary survives every switch) + a 64-frame
|
||||
deep recursive chain suspended at the bottom and resumed (`frames verified: 64` / `depth fails:
|
||||
0`). The `bl _fib_body` reaches the sx body via `export "fib_body"` (the 1655 asm→sx pattern);
|
||||
runs under JIT, ir-only on a non-arm host (`.ir` captured — `swap_context` shows `naked noinline
|
||||
nounwind`). Suite green 733/0. **Honest scope:** indirect register/stack survival only; the
|
||||
EXPLICIT every-callee-saved + FP scribble (§10.7) is B1.3a-2, still owed. Next: B1.3a-2.
|
||||
- **B1.3a-2 — the §10.7 stress gate, adversarially reviewed.** Extended `swap_context` to the
|
||||
COMPLETE AAPCS64 callee-saved set (added FP d8-d15 → 21-slot ctx) and wrote a naked
|
||||
`scribble_verify` that loads a unique sentinel into all 18 callee-saved regs, yields, and counts
|
||||
non-survivors on resume (176-byte frame saves/restores the caller's callee-saved + base; lr
|
||||
round-trips the swap). The gate is a 2-fiber MUTUAL scribble (each clobbers the other's regs, so
|
||||
survival ⇒ the switch saved+restored them). Locked by
|
||||
`examples/1808-concurrency-fiber-switch-stress.sx` (`A/B mismatches: 0`). Validity proven by
|
||||
negative controls (drop d8-d15 → 8/8; drop x27/x28 → 2/2). **Spawned an adversarial-review
|
||||
worker (per the plan + user request): NO critical bugs** — callee-saved set complete (x18 rightly
|
||||
excluded; d8-d15 suffices per §6.1.2), offsets/alignment/lr-sp dance all verified. Applied its
|
||||
one rec: `boot` zeroes FP ctx slots so first-entry loads 0, not garbage. Honest residual gaps
|
||||
(spec-correct for a call-boundary swap; in the example header): FPCR/FPSR/NZCV + TPIDR/TLS not
|
||||
swapped, fp=0 blocks unwind — relevant at N×M:1 / signals, not here. Suite green 734/0.
|
||||
Next: B1.3b (x86_64 sibling + mmap guard-page stacks).
|
||||
- **B1.3b — mmap guard-page stacks (x86_64 sibling deferred).** Fiber stacks now `mmap` a
|
||||
`[guard | usable]` region and `mprotect` the low 16KB page `PROT_NONE`, so a stack overflow
|
||||
faults at the guard boundary instead of silently corrupting a neighbor (§8.1.1). Locked by
|
||||
`examples/1809-concurrency-fiber-guard-stack.sx` (aarch64-macos-pinned): `guard armed: 1`
|
||||
(`mprotect`→0) + `sum: 20100` (a fiber runs real recursion on the guarded stack + yields).
|
||||
Guard FIRING validated manually (overflow → `Bus error` at `region+GUARD`, exit 134 via the sx
|
||||
crash handler) — not corpus-pinned because a deliberate-overflow crash is host-fragile (and a
|
||||
mere "child faulted" fork test wouldn't prove the BOUNDARY catch). The x86_64 `swap_context`
|
||||
sibling was DEFERRED: `--target x86_64-macos` mislinks on this arm64 host and `x86_64-linux`
|
||||
can't run here, so it could only ship un-run/un-negative-controlled — which §10.7 forbids for the
|
||||
highest-risk asm. SysV target notes (rbx/rbp/r12-r15/rsp, no callee-saved XMM, rsp-carried return
|
||||
addr) recorded in Next step. Suite green **735/0**. Next: x86_64 sibling (needs an x86_64 host)
|
||||
OR B1.4 (`Io` impls / scheduler) on the proven aarch64 substrate.
|
||||
- **B1.3b-1 — x86_64 / Win64 switch sibling VALIDATED on real hardware.** The user provided a
|
||||
Windows 7 x64 VM (UTM), so the x86_64 switch became RUNNABLE (as Win64). Validated the
|
||||
cross-build→VM→run loop (`--target x86_64-windows-gnu --self-contained` → PE32+; output via the
|
||||
Win32 `WriteFile` boundary, the 1660 pattern). Wrote a Win64 `swap_context` (8 GP rbx/rbp/rdi/
|
||||
rsi/r12-r15 + rsp + **xmm6-xmm15** via `movups` — Win64 has callee-saved XMM) + a Win64
|
||||
`scribble_verify` (264-byte frame, 32-byte shadow + 16-align at each `call`, COFF symbols,
|
||||
rsp-carried return addr) driving the 2-fiber mutual scribble. **Adversarially reviewed (worker
|
||||
emitted the real `.s`, verified every alignment/offset/round-trip against the Win64 ABI — no
|
||||
critical/minor bugs), THEN run on the VM → `0 0 P`** (all 8 GP + 10 XMM callee-saved survived).
|
||||
Locked by `examples/1810-concurrency-fiber-switch-win64.sx` (pinned `x86_64-windows-gnu`,
|
||||
ir-only on this host; the VM run is the runtime-correctness provenance). Good-swap-only (the
|
||||
in-process negative control was dropped to avoid an sx fn-ptr-convention rabbit hole; the
|
||||
detection of this exact logic was negative-controlled on aarch64 in 1808). Suite green **736/0**.
|
||||
The B1.3 context switch is now proven on TWO arch/ABI pairs. Next: **B1.4** (Io impls / M:1
|
||||
scheduler) on the proven substrate. (Side thread: the SysV/Linux x86_64 sibling, when a Linux
|
||||
x86_64 host is available.)
|
||||
- **B1.5a — M:1 scheduler CORE + a fixed blocker bug.** Built `library/modules/std/sched.sx`: a
|
||||
generic `Fiber`/`Scheduler` over `swap_context` on guarded `mmap` stacks. `spawn` heap-allocs a
|
||||
fiber, bootstraps its ctx, enqueues it; the ONE generic dispatch (`fib_dispatch` via `_fib_tramp`)
|
||||
runs ANY stored `Closure() -> void` on a fresh stack (replacing the fixed `bl _fib_body`);
|
||||
`yield_now` round-robins, `suspend_self`/`wake` park/resume off-queue, `run` drives to drain +
|
||||
reaps `.done` fibers (`munmap` + free). **De-risked first by probe** (closure-on-fiber + output
|
||||
via captured pointer). **Hit blocker bug 0154** (user-authorized fix): `null`/`---` to a struct
|
||||
field over-stored a whole-struct null when the fn return type leaked as `target_type`, corrupting
|
||||
the frame (`ret` 0x0) — exactly the `Scheduler.init()` by-value-return shape. Fixed in `stmt.zig`
|
||||
(`needs_target` += `null`/`undef` literals); regression `examples/types/0193`; `0154` RESOLVED.
|
||||
**Adversarial review:** asm/bootstrap/lifetime sound (env-lifetime fear disproven — heap-promoted);
|
||||
1 CRITICAL (`wake` re-enqueue → FIFO segfault) + robustness gaps ALL hardened (wake guarded on
|
||||
`.suspended`, `n_suspended` deadlock diagnostic+abort, loud mmap/mprotect/OOM bails, env-leak
|
||||
documented). Locked `1811` (round-robin `0 1 2 ×3`) + `1812` (suspend/wake + spurious-wake guard,
|
||||
`log: 10 20 21 11`). Filed NON-blocking `0155` (scalar-pointer index panics codegen — review
|
||||
incidental, unused by sched). Suite GREEN **748/0**. Next: **B1.4a** (FiberIo).
|
||||
- **B1.4a (truly-suspending fiber-task async, nullary-thunk design) — BLOCKED on issue 0157.**
|
||||
Implemented the async layer SELF-CONTAINED in `library/modules/std/sched.sx` (kept its lone
|
||||
`#import "modules/std.sx"` to avoid the duplicate-`_fib_tramp` trap): `TaskState`, a LOCAL
|
||||
`TaskErr :: error { Canceled }` (the re-exported `IoErr` alias is NOT seen through by the
|
||||
`raise`/failable-type check — verified), `Task($R)`, and `go`/`wait`/`cancel` ufcs. Design is
|
||||
the validated nullary-thunk (`.sx-tmp/pnullary.sx` → `log: 1 2 3 42 100`): `work` is a
|
||||
`Closure() -> $R`, user captures inputs at the call site, NO `..args` crosses the fiber boundary
|
||||
(deliberately sidesteps 0156). `go`+`wait` run correctly; both wake-orderings traced. Wrote the
|
||||
example `examples/concurrency/1813-concurrency-fiber-async-suspend.sx` (+ `{ "target": "macos" }`
|
||||
`.build`) but its `cancel` ufcs surfaced a NEW compiler bug — issue **0157**: a user generic
|
||||
ufcs whose name collides with a stdlib re-export (`cancel` from io.sx) is mis-resolved on UFCS
|
||||
call over a different generic struct, leaving `$R` unresolved → LLVM panic. Bisected to a minimal
|
||||
no-fiber repro (name is the sole trigger; non-UFCS form diagnoses correctly). Example NOT seeded
|
||||
into the corpus (no `.exit` marker) — do NOT regen its goldens until 0157 lands. Per the STOP
|
||||
rule: filed `issues/0157-*.{md,sx}`, marked state BLOCKED, paused.
|
||||
- **B1.4a COMPLETE (this session) — suspending fiber-task async + two compiler fixes.** Built the
|
||||
`Task($R)` + `go`/`wait`/`cancel` layer in `sched.sx` (nullary-thunk design; self-contained to
|
||||
avoid the `_fib_tramp` duplicate-symbol trap). Locked `1813` (`sequence: 1 2 3 42 100 -99`).
|
||||
FIXED the two blockers the worker had filed: **0156 Part 1** (`comptime_pack_ref` arm in
|
||||
`resolveTypeWithBindings`; regression `0216`) and **0157** (receiver-driven UFCS overload
|
||||
selection `selectUfcsGenericByReceiver`; regression `0217`). Adversarial review of the 0157 fix +
|
||||
Task layer found a determinism CRITICAL (always-run selection + specificity + ambiguity
|
||||
diagnostic), a `wait`-outside-fiber null-deref (loud guard), and cancel-not-skipping-work (skip
|
||||
if pre-canceled) — all fixed. Simplified `1812` (`**Fiber` → `Sh.parked`). 0156 Part 2 reframed
|
||||
OPEN/non-blocking. Suite GREEN **751/0**. Next: B1.4b (deterministic-sim `Io`, the KEYSTONE).
|
||||
- **B1.4b COMPLETE (this session) — deterministic virtual-time timers + a CRITICAL UAF fix.** Added
|
||||
`clock_ms`/`timers`/`now_ms`/`sleep` + a timer-driven `run` to `sched.sx` (worker-built): fibers
|
||||
sleep in reproducible simulated time, waking in deadline order (FIFO tiebreak). Locked `1814`
|
||||
(5 fibers, wake order B@10/D@15/E@15/C@20/A@30). Adversarial review of the run-loop change found a
|
||||
CRITICAL use-after-free — a fiber woken EARLY (manual/Task `wake`) before its `sleep` timer fired
|
||||
was reaped while its `Timer` kept a dangling `*Fiber`; a later fire dereferenced freed memory
|
||||
(silent "pass" only by luck). Fixed: `wake` evicts the fiber's pending timer (`cancel_timer_for`);
|
||||
regression `1815` (early wake → `clock: 0`, stale timer never fires). Review cleared n_suspended
|
||||
accounting, deadlock false-positives, timer-list integrity, clock monotonicity, termination.
|
||||
Suite GREEN **753/0**. Next: B1.4c (event-loop `Io`, kqueue/epoll).
|
||||
- **B1.4c COMPLETE (this session) — real fd readiness via kqueue + 2 CRITICAL review fixes.** Added a
|
||||
lazy `kq` + `io_waiters` + `block_on_fd` + a kqueue-blocking run-loop Mode 2 to `sched.sx`
|
||||
(worker-built, reusing `std/net/kqueue.sx`). Adversarial review found two CRITICALs: same-fd
|
||||
lost-wakeup hang (FIXED — `block_on_fd` enforces one-waiter-per-fd with a loud abort) and a
|
||||
never-ready-fd "hang" (RECLASSIFIED as correct event-loop semantics; misleading orphan-check comment
|
||||
corrected). Locked `1816` (pipe block→kqueue-wake→read). Suite green 754/0.
|
||||
- **B1.5 COMPLETE → STREAM B1 DONE (this session).** Capstone `1817` composes the whole stack
|
||||
(`go`/`wait` + `sleep`/`now_ms` + scheduler) — three tasks complete in DEADLINE order
|
||||
(task 2@10 / 3@20 / 1@30), `sum: 123`, final virtual clock 30. The pure-sx colorblind M:1 async
|
||||
runtime is feature-complete end-to-end (1800–1817), all adversarially reviewed. Suite GREEN
|
||||
**755/0**. Five compiler bugs fixed across the stream (0151/0152/0153/0154/0156-P1/0157 — 0151-3 in
|
||||
B1.2). Next carve: Stream B2 (channels / cancel / async stdlib).
|
||||
- **Post-review hardening (this session) — 6 findings from an adversarial review of the B1 commits.**
|
||||
Fixed: **P1-a** the UFCS generic PLANNER (`calls.zig`) used the last-wins `fn_ast_map` winner while
|
||||
lowering reselected by receiver → plan/lowering could disagree and MISBOX the result; now both share
|
||||
`selectUfcsGenericByReceiver`. **P1-b** the selection scanned `module_decls` globally, flagging a
|
||||
transitively-hidden same-named overload as a FALSE ambiguity; now two-tier — directly-visible authors
|
||||
first (ambiguity only among those), global fallback for receiver-reachable namespaced methods (e.g.
|
||||
`Task.cancel`) that defers to `fd0` on a hidden tie. **P2-b** boolean specificity tied `*$T` with
|
||||
`*Box($T)`; now peels pointer layers so the structurally-narrower receiver wins. **P1-c** a second
|
||||
concurrent `Task.wait` overwrote the single waiter slot → silent deadlock; now one-awaiter-per-task
|
||||
loud abort. **P2-c** `sleep(negative)` rewound the virtual clock; now rejected loudly. (**P2-a**
|
||||
non-generic-winner-hides-generic did not reproduce — the non-generic arm already falls through.)
|
||||
Regressions: `examples/generics/0218` (receiver specificity + plan/lowering agreement),
|
||||
`examples/concurrency/1818` (negative-sleep abort), `1819` (double-wait abort). Suite GREEN **758/0**.
|
||||
@@ -1,62 +0,0 @@
|
||||
# CHECKPOINT-HTTPZ — Stream HTTPZ (production HTTP-server readiness)
|
||||
|
||||
Tracker for the HTTP-server production-readiness stream. Plan:
|
||||
[PLAN-HTTPZ.md](PLAN-HTTPZ.md). Update after every step.
|
||||
|
||||
## Last completed step
|
||||
**Stream established (planning only).** Audited the existing HTTP/socket/thread/event
|
||||
stack against the user's production-readiness checklist and wrote
|
||||
[PLAN-HTTPZ.md](PLAN-HTTPZ.md) + this checkpoint. **No code changed.** Prior HTTP work
|
||||
(socket `S2`, thread `S6`, http `S7a`, pool `S7b`) shipped without a tracked plan; this
|
||||
brings the stream under checkpoint discipline.
|
||||
|
||||
## Current state
|
||||
- **Done & Linux-validated (do NOT rebuild):** `event.sx` (epoll+kqueue, 6/6 green on real
|
||||
aarch64 Linux in Apple `container`), `net/epoll.sx`, `net/kqueue.sx`, `sched.sx` M:1
|
||||
runtime, `json.sx`.
|
||||
- **BROKEN on Linux (Phase C3 keystone):**
|
||||
- `socket.sx` — Darwin-only `SockAddr` (`sin_len`), `O_NONBLOCK=4`, macOS errno values,
|
||||
`__error` binding. Corrupts addresses + breaks WouldBlock detection on Linux.
|
||||
- `thread.sx` — `MutexBuf=64B` (Darwin) vs glibc 40B → 24-byte heap overflow on
|
||||
`pthread_mutex_init`. Pool unsafe on Linux.
|
||||
- **Works, unhardened — `http.sx`:** single-worker loop + inline/pool handlers, keep-alive,
|
||||
delivery timeouts, conn/request caps, 400/413/431/503. Gaps: parser limits, `Server.close()`
|
||||
leaks (`conns`/`PoolState`/`done`), no graceful stop, no handler-exec timeout, zero
|
||||
observability, no streaming.
|
||||
- **Absent entirely:** CI (no Linux CI), fuzz, sanitizers/leak-check (`tests/stress-http.sh`
|
||||
broken — references deleted `32-http-server.sx`), releases/tags, SECURITY.md, deploy docs,
|
||||
routing/form helpers. **TLS:** none yet — to be added natively via mbedTLS FFI (Phase T).
|
||||
|
||||
Full grounded audit (file:line) lives in PLAN-HTTPZ.md "Audit of record".
|
||||
|
||||
## Next step
|
||||
**Phase C3a — `socket.sx` per-OS selection.** Branch `SockAddr`, `O_NONBLOCK`, errno
|
||||
constants, and `errno_slot` on `OS`/`ARCH` (mirror the `inline if OS ==` pattern in
|
||||
`event.sx`/`sched.sx`). Lock a Linux-vs-Darwin layout/const assertion red (cadence rule),
|
||||
then flip green; validate under the Apple `container` Linux VM. No silent fallback defaults.
|
||||
> **NOT STARTED** — user requested plan-only this session. Execution begins next session.
|
||||
|
||||
## Known issues / capability gaps
|
||||
- `socket.sx` / `thread.sx` Linux-broken (above) — blocks all Linux P0 acceptance.
|
||||
- No CI of any kind in the repo → "tested on Linux" cannot be claimed until C4.
|
||||
- Corpus runner (10s/example timeout, no net sandbox) cannot host stress/fuzz/load — those
|
||||
go in separate CI-wired scripts (PLAN decision).
|
||||
- `http.sx` `Server.close()` leaks on shutdown (H2).
|
||||
- No handler-execution timeout in either handler mode (H5).
|
||||
|
||||
## Decisions (HTTPZ specifics — full list in PLAN-HTTPZ.md)
|
||||
- Native TLS via an mbedTLS FFI binding (Phase T) — supersedes the original proxy-only
|
||||
posture; proxy deployment stays supported/documented. No pure-sx TLS stack.
|
||||
- `Transfer-Encoding: chunked` rejected (501) in H1, implemented in S1/S2.
|
||||
- Stress/fuzz/load harnesses live outside the corpus, wired into CI.
|
||||
- C3 branching bails loudly on unhandled OS/arch arms — no Darwin-default fallback.
|
||||
|
||||
## Log
|
||||
- **2026-06-26** — Stream established. Parallel audit of `http.sx`, `socket.sx`,
|
||||
`thread.sx`, `event.sx`, `net/epoll.sx`, `net/kqueue.sx`, `sched.sx`, the test/CI/bench
|
||||
infra, and the docs/release/security posture against the production-readiness checklist.
|
||||
Wrote PLAN-HTTPZ.md (phases C/H/S/D mapping checklist P0/P1/P2) + this checkpoint.
|
||||
No code changes. Next: Phase C3a.
|
||||
- **2026-06-26** — Added **Phase T (native TLS via mbedTLS FFI)** to PLAN-HTTPZ.md, slotted
|
||||
after Phase H; flipped the proxy-only decision to native-TLS-plus-proxy; updated D1.
|
||||
Backend chosen: mbedTLS (static-link-friendly, clean non-blocking API). Still plan-only.
|
||||
276
current/CHECKPOINT-METATYPE.md
Normal file
276
current/CHECKPOINT-METATYPE.md
Normal file
@@ -0,0 +1,276 @@
|
||||
# CHECKPOINT-METATYPE — comptime type metaprogramming (`declare` / `define`)
|
||||
|
||||
Companion to [PLAN-METATYPE.md](PLAN-METATYPE.md). Update after every step (one
|
||||
step at a time, per the cadence rule).
|
||||
|
||||
## Last completed step
|
||||
**`type_info` / `define` widened to TUPLE types — reflect/construct triad
|
||||
complete.** `TypeInfo` gained a `` `tuple(TupleInfo) `` variant (`TupleInfo{
|
||||
elements: []Type }`, positional/unnamed). `reflectTypeInfo` builds `.tuple`
|
||||
(tag 2) as bare `type_tag` elements; `defineTuple` decodes `[]Type` and completes
|
||||
the declare slot as a structural `.tuple` via `replaceKeyedInfo` (tuples are
|
||||
structural, so the declared name is vestigial, but the slot is completed in place
|
||||
so `define` returns the handle like enum/struct). `call.zig`'s `type_info` guard
|
||||
admits `.tuple`. `examples/0623` (programmatic `Pair` + source-tuple round-trip).
|
||||
Suite green (684). All three TypeInfo shapes now reflect + construct + round-trip
|
||||
(`0619` enum, `0622` struct, `0623` tuple).
|
||||
|
||||
## Earlier — struct widening
|
||||
**`type_info` / `define` widened to STRUCT types.** `TypeInfo` gained a
|
||||
`` `struct(StructInfo) `` variant (`StructField{ name, type }`); the metatype
|
||||
system now reflects AND constructs structs, not only enums.
|
||||
- `meta.sx`: `StructField` / `StructInfo` / `` `struct `` TypeInfo variant.
|
||||
- `interp.zig`: `reflectTypeInfo` builds `.struct` (tag 1) for a source
|
||||
`@"struct"`; `define` dispatches on the TypeInfo tag (`defineType` →
|
||||
`defineEnum` (0) / `defineStruct` (1)). `defineStruct` mirrors `defineEnum`
|
||||
(duplicate-field-name check included) but completes the declare slot AS a
|
||||
struct via `replaceKeyedInfo` — a KIND change re-keys the intern map, whereas
|
||||
`updatePreservingKey` (the enum path) asserts the key is unchanged.
|
||||
- `lower/call.zig`: the lower-time `type_info` guard now admits `@"struct"`.
|
||||
- `examples/0622`: programmatic `Vec2` via `.struct(.{ fields = … })` + a
|
||||
source-struct round-trip `define(declare("RowCopy"), type_info(Row))`. Enum
|
||||
path (`0619`) unchanged. Suite green (683). Tuple is the last shape (Next step).
|
||||
|
||||
## Earlier — make_enum
|
||||
**`make_enum(name, variants: []EnumVariant) -> Type`** — the general enum
|
||||
constructor over `declare`/`define`, minting a nominal enum from a variant list
|
||||
passed as a VALUE. Pure sx in `meta.sx`. `examples/0620` assembles the list in a
|
||||
local then mints, exercising `define`'s value-arg SLICE decode.
|
||||
|
||||
## Prior step
|
||||
**`type_info($T)` reflection — enum round-trip.** Reflect a type INTO a `TypeInfo`
|
||||
value (the inverse of `define`'s decode), so `define(declare(n), type_info(T))`
|
||||
mints a byte-identical copy with NO literal variant list.
|
||||
|
||||
- `inst.zig`: new `BuiltinId.type_info` (comptime-only, alongside `declare`/`define`).
|
||||
- `lower/call.zig:tryLowerReflectionCall`: the old "not yet implemented" bail is
|
||||
gone. Resolve `$T` at lower time, reject a non-`enum`/non-`tagged_union` arg
|
||||
loudly (good span: `"type_info: 'X' is not an enum …"`), else emit
|
||||
`callBuiltin(.type_info, [const_type], TypeInfo)`.
|
||||
- `interp.zig:reflectTypeInfo`: builds the exact nested-aggregate Value
|
||||
`defineEnum` decodes — variant `{name, payload}`, slice `{data, len}`, EnumInfo
|
||||
`{variants}`, TypeInfo `{tag0, EnumInfo}`. A `tagged_union` reflects each
|
||||
`field.ty` (tagless variants already carry `void`); a payloadless `` `enum ``
|
||||
reflects `void` per variant. Round-trips both source enums AND constructed
|
||||
(declare/define) enums.
|
||||
- emit unchanged — `type_info` is always comptime-evaluated; the existing
|
||||
comptime-only `else` arm in `emitCallBuiltin` (shared with declare/define)
|
||||
never fires.
|
||||
- Scope: **enum-only** (the symmetric inverse of `define`'s current capability).
|
||||
Struct/tuple `TypeInfo` widening is a separate later step.
|
||||
|
||||
`examples/0619` locks it (source enum `circle:f64 / rect:i64 / empty` reflected →
|
||||
reconstructed → constructs + matches). Full suite green (676 examples + units).
|
||||
|
||||
## Earlier step
|
||||
**Self-reference — recursive enums via `declare("Name")` + `*Name`.** The
|
||||
`declare`/`define` floor now supports self-referential types.
|
||||
|
||||
- `declare(name) -> Type` mints an empty (undefined) nominal slot NAMED `name`;
|
||||
`define(handle, info) -> Type` decodes the `TypeInfo` value (variant names +
|
||||
payload Type-tags), fills the slot byte-identical to a source enum, and returns
|
||||
the handle (one-shot form chains: `T :: define(declare("T"), info)`). Interp
|
||||
executes both against a `mint` TypeTable handle; `defineEnum` +
|
||||
`decodeVariantElements` in `interp.zig`.
|
||||
- **Self-reference:** `evalComptimeType`'s `preregisterForwardTypes` scans the
|
||||
comptime expression (and a called ctor fn's body) for `declare("Name")` calls
|
||||
and, before the body lowers, registers each as an empty forward nominal type AND
|
||||
binds it as a type alias. The alias is essential — a `Name :: ctor()` decl makes
|
||||
`Name` a const_decl author, so a `*Name` self-reference resolves through the
|
||||
forward-ALIAS path (`type_aliases_by_source`), which a bare `findByName`
|
||||
registration alone does NOT satisfy (it returns a pending empty-struct stub). The
|
||||
interp's `declare` returns that same slot; `define` fills it.
|
||||
- A `::` binding or type-fn body calling a `Type`-returning fn is
|
||||
**comptime-evaluated** (`evalComptimeType`) — no constructor-name knowledge.
|
||||
`decl.zig` trigger = `fnReturnsTypeValue`; type-fn trigger = `returnExprMintsType`.
|
||||
- Nominal identity rides the type-fn instantiation cache (`renameNominalType`).
|
||||
- The type NAME is on `declare(name)` (compile-time string), not `EnumInfo`.
|
||||
|
||||
Examples green: `0614` (one-shot), `0615` (type-fn identity), `0617` (channel
|
||||
results), **`0618` (recursive `*List`: construct, match through pointer, recursive
|
||||
traversal)**; `field_type` reflection `0616`. Full suite green (674 examples).
|
||||
|
||||
## Current state
|
||||
- `modules/std/meta.sx`: `EnumVariant` / `EnumInfo{ name, variants }` / `TypeInfo`
|
||||
data types; `declare` / `define` / `type_info` / `field_type` `#builtin`s;
|
||||
`RecvResult($T)` / `TryResult($T)` + the general `make_enum(name, variants)` sx
|
||||
constructors over `define(declare(), …)`.
|
||||
- Compiler primitives only: `declare`/`define` (construction), `field_type`
|
||||
(reflection). No constructor-name knowledge anywhere in the compiler — every
|
||||
named constructor is sx. `declare(name)` carries the type name (compile-time
|
||||
string) for forward-type registration.
|
||||
- `type_info($T)` reflects an `enum`/`tagged_union`/`struct`/`tuple` INTO a
|
||||
`TypeInfo` value (`call.zig` emits `callBuiltin(.type_info)`;
|
||||
`interp.zig:reflectTypeInfo` builds the Value). `define` decodes `.enum` →
|
||||
tagged_union, `.struct` → struct, `.tuple` → tuple (the last via
|
||||
`replaceKeyedInfo`). `examples/0619` (enum) / `0622` (struct) / `0623` (tuple)
|
||||
round-trip. All three TypeInfo shapes ship.
|
||||
|
||||
## Decision (kept)
|
||||
**Meta lives in `modules/std/meta.sx`, not the prelude.** Declaring its data types
|
||||
in the always-loaded prelude interns them into every module's type table and
|
||||
shifts every `.ir` snapshot. On-demand import keeps the prelude clean.
|
||||
|
||||
## Next step
|
||||
The reflect/construct triad is COMPLETE — `` `enum `` (`0619`), `` `struct ``
|
||||
(`0622`), `` `tuple `` (`0623`) all reflect AND construct + round-trip. Remaining
|
||||
METATYPE work is ONE deferred enhancement, a clean diagnostic rather than a crash
|
||||
— filed as **issue 0141** (repro `issues/0141-*.sx` + full two-layer writeup +
|
||||
investigation prompt):
|
||||
- **Comptime `List` growth** — `List(T).append` at comptime bails ("struct_get:
|
||||
base has no fields"). Doesn't block anything: array-literal locals already build
|
||||
variant lists (`examples/0620`/`0624`). Probe `.sx-tmp/probe_makeenum.sx` /
|
||||
`probe_li64.sx`. **Investigated — it's TWO layers** (both reproduce with plain
|
||||
`List(i64)`, not metatype-specific; List works via `#run` because that evaluates
|
||||
at EMIT time, after everything is lowered, while a metatype `::` const evaluates
|
||||
at `scanDecls` time):
|
||||
1. **Null comptime allocator.** `interp.zig:defaultContextValue` builds the
|
||||
comptime `context.allocator` by looking up `__thunk_CAllocator_Allocator_alloc_bytes`
|
||||
by name in the module's functions — but at `scanDecls` time those protocol
|
||||
thunks aren't lowered yet, so `alloc_fn`/`dealloc_fn` are `.null_val` and any
|
||||
comptime allocation fails. FIX (tried, works for this layer): call
|
||||
`self.getOrCreateThunks("Allocator", "CAllocator")` (guarded by the same
|
||||
Context/Allocator/CAllocator-registered check `emitDefaultContextGlobal` uses)
|
||||
before the interp runs in `comptime.zig:runComptimeTypeFunc`.
|
||||
`createProtocolThunk` saves/restores builder state, so calling it mid-lowering
|
||||
is safe. After this, `alloc_fn=func_ref` — but layer 2 still bails.
|
||||
2. **`struct_get` through a `*T` slot_ptr chain.** A `*List` struct receiver
|
||||
(`vs.append(…)` → `append(self: *List, …)`) lands in the interp as a slot
|
||||
whose contents are a slot_ptr to the actual value — `self.field` does
|
||||
`struct_get` on `base=slot_ptr field_index=1` and bails. The auto-deref in
|
||||
`interp.zig:.struct_get` does a single `loadSlot`; a chain-resolve loop did
|
||||
NOT fix it (the final loaded value is a field-pointer aggregate that
|
||||
`resolveFieldLoad` turns back into a slot_ptr — List's comptime representation
|
||||
uses field-pointers + slot_ptrs the struct_get path doesn't fully resolve).
|
||||
This is the deep part: comptime pointer/struct/slot resolution for `*T`
|
||||
receivers, its own focused effort. Both speculative fixes were REVERTED (no
|
||||
end-to-end testable win without layer 2).
|
||||
The metatype surface (declare/define/type_info/field_type + make_enum) is
|
||||
feature-complete for the locked design; generic type-fn body locals now work too.
|
||||
- ~~**Validation + loud diagnostics**~~ — COMPLETE. duplicate variant names
|
||||
(`examples/1180`); `declare()` never `define()`d (`examples/1181`, was a
|
||||
`verifySizes` panic); by-value self-reference for both source (`1178`) and
|
||||
CONSTRUCTED (`1182`) types via `checkInfiniteSize`. **use-before-define needs no
|
||||
new check** — it's subsumed by the existing guards: a by-value cycle →
|
||||
`checkInfiniteSize` ("infinitely sized"); an unfinished slot → declare-never-
|
||||
defined; a bad/non-Type payload → a 0140 clean bail; a forward reference resolves
|
||||
correctly via in-place slot mutation (`updatePreservingKey`); a `*Name` pointer
|
||||
needs no layout. Probes `.sx-tmp/probe_ubd{1..4}.sx` confirmed: no remaining
|
||||
crash or silent-corruption, only clean diagnostics / correct results.
|
||||
|
||||
### make_enum follow-ups (deferred capability gaps — NOT crashes; clean diagnostics)
|
||||
`make_enum` itself is DONE (see Last completed step). Remaining adjacent
|
||||
capabilities would let the variant list be built more freely; both error cleanly
|
||||
(post-0140) rather than crash, so they're enhancements, not blockers:
|
||||
- ~~Comptime slice over a non-string aggregate~~ — DONE. `arr[lo..hi]` over a
|
||||
`[]EnumVariant` array now yields a real slice value at comptime (was: bailed,
|
||||
string-only). Fix threaded `base_ty` onto the `Subslice` op so the interp tells
|
||||
an array from a `{ptr,len}` slice, folded open-ended `hi` to a fixed array's
|
||||
static length at lower time (no runtime/.ir change), and added
|
||||
`interp.zig:subsliceElements`. `examples/0621` locks it.
|
||||
- **Comptime `List` growth** (issue 0141). `List(T).append` at comptime bails
|
||||
("struct_get: base has no fields"). Investigated — two layers (null comptime
|
||||
allocator at scanDecls + `struct_get` through a `*T` slot_ptr chain); see the
|
||||
detailed writeup under "Next step" and `issues/0141-*.md`. Layer 1 has a known
|
||||
fix; layer 2 is deep. Probe `.sx-tmp/probe_makeenum.sx`.
|
||||
- ~~Generic type-fn body locals~~ — DONE. A generic `($T) -> Type` now
|
||||
comptime-evaluates its FULL body (prelude statements + return), so a local
|
||||
before the return resolves. `createComptimeFunctionWithPrelude` +
|
||||
`evalComptimeTypeBody`; no-prelude bodies stay on the old path. `examples/0624`.
|
||||
|
||||
## Known issues
|
||||
- issue 0141 (OPEN, deferred enhancement — not a blocker) — `List(T).append` at
|
||||
comptime bails in a type-construction `::` (two layers: null comptime allocator
|
||||
+ `*T` slot_ptr `struct_get`). Workaround: array-literal locals
|
||||
(`examples/0620`/`0624`). Full writeup + investigation prompt in
|
||||
`issues/0141-*.md`.
|
||||
- issue 0140 — comptime type-construction bail panicked instead of diagnosing —
|
||||
RESOLVED. `evalComptimeType` now clears `last_bail_detail` before the interp
|
||||
call and, on the `catch`, emits a build-gating `.err` at the construction span
|
||||
("comptime type construction failed: {detail}") before returning the
|
||||
`.unresolved` poison — so the reason is shown and no unresolved type reaches
|
||||
emission unannounced. `examples/1179` locks it.
|
||||
- issue 0139 — by-value self-reference segfault — RESOLVED (`checkInfiniteSize`
|
||||
Pass 1g emits a loud "infinitely sized" diagnostic + breaks the cycle;
|
||||
`examples/1178` locks it).
|
||||
|
||||
## Log
|
||||
- **Generic type-fn body locals.** A generic `($T) -> Type` comptime-evaluated
|
||||
only its return EXPRESSION, so a local before the return was unresolved. Now a
|
||||
body with a prelude (statements before the return) has its FULL body evaluated:
|
||||
`createComptimeFunctionWithPrelude` lowers the pre-return statements into the
|
||||
comptime function's scope, then the return expr. No-prelude bodies (RecvResult
|
||||
etc.) stay on the old path → zero regression. `examples/0624`. Suite green (685).
|
||||
- **Tuple widening done — reflect/construct triad complete.** `TypeInfo` gained
|
||||
`` `tuple(TupleInfo) `` (positional `[]Type`); `reflectTypeInfo` reflects a
|
||||
`.tuple` (bare type_tags, tag 2), `defineType` dispatches tag 2 → `defineTuple`
|
||||
(completes the slot as a structural tuple via `replaceKeyedInfo`), and the
|
||||
lower-time `type_info` guard admits `.tuple`. `examples/0623`. Suite green (684).
|
||||
enum/struct/tuple all reflect + construct + round-trip.
|
||||
- **Struct widening done.** `TypeInfo` gained `` `struct(StructInfo) ``; `define`
|
||||
dispatches on the tag (`defineType` → `defineEnum`/`defineStruct`), `reflectTypeInfo`
|
||||
reflects a `@"struct"`, and the lower-time `type_info` guard admits structs.
|
||||
`defineStruct` uses `replaceKeyedInfo` (kind change: tagged_union declare slot →
|
||||
struct). `examples/0622` (programmatic build + source round-trip). Suite green
|
||||
(683). Tuple is the last remaining shape.
|
||||
- **Validation story COMPLETE.** use-before-define needs no new check — subsumed
|
||||
by `checkInfiniteSize` (by-value cycles), declare-never-defined (unfinished
|
||||
slots), 0140 bails (bad payloads), and in-place slot mutation (forward refs);
|
||||
`*Name` pointer use needs no layout. Probed `.sx-tmp/probe_ubd{1..4}.sx`: all
|
||||
clean diagnostics / correct results, no crash. `examples/1182` locks the
|
||||
by-value self-ref rejection for CONSTRUCTED enums (companion to source `1178`).
|
||||
- **declare()-never-defined validation.** A bare `declare("X")` with no `define`
|
||||
left a zero-field nominal slot that panicked at codegen (`verifySizes`).
|
||||
`evalComptimeType` now detects a zero-variant `tagged_union` result and emits a
|
||||
clean diagnostic naming the type. Self-reference (declared slot completed by
|
||||
`define`) is unaffected. `examples/1181` locks it. Suite green (681).
|
||||
- **Duplicate variant-name validation.** Two same-named variants in a constructed
|
||||
enum used to silently succeed (ambiguous construction/match). `defineEnum` now
|
||||
bails naming the duplicate; `evalComptimeType` renders it (post-0140).
|
||||
`examples/1180` locks it. Suite green (680).
|
||||
- **Comptime subslice over non-string aggregates.** `arr[lo..hi]` at comptime
|
||||
used to bail (interp `.subslice` was string-only) and the open-ended `hi` came
|
||||
from a `.length` op that misread a 2-elem array as a `{ptr,len}` fat pointer.
|
||||
Fix (interp-only; runtime already correct via `LLVMTypeOf`): thread `base_ty`
|
||||
onto the `Subslice` op, fold open-ended `hi` to a fixed array's static length at
|
||||
lower time (no IR/.ir change), add `subsliceElements`. `examples/0621` mints an
|
||||
enum from `dirs[0..2]`. Suite green (679).
|
||||
- **`make_enum` done.** General enum constructor `make_enum(name, variants:
|
||||
[]EnumVariant) -> Type` in `meta.sx` (pure sx over declare/define). A non-generic
|
||||
builder assembles the variant list in a local, then mints from it —
|
||||
`examples/0620` exercises `define`'s value-arg SLICE decode. No compiler change.
|
||||
Suite green (678). Deferred free-form gaps (subslice/List at comptime,
|
||||
generic-type-fn locals) noted under Next step — all clean diagnostics now, not
|
||||
crashes (post-0140), so enhancements rather than blockers.
|
||||
- **issue 0140 fixed.** A comptime type-construction bail (`declare`/`define`/
|
||||
reflection) used to panic at LLVM emission ("unresolved type reached LLVM
|
||||
emission") or hide behind a cascade — `evalComptimeType` swallowed the interp's
|
||||
`last_bail_detail`. Now it clears the detail before the call and renders a
|
||||
build-gating `.err` at the construction span on the `catch`. `examples/1179`
|
||||
locks the empty-variants case. Suite green (677). Unblocks make_enum (its
|
||||
computed-slice decode failures now surface cleanly).
|
||||
- **`type_info($T)` reflection done (enum round-trip).** New `BuiltinId.type_info`;
|
||||
`lower/call.zig` resolves `$T`, rejects non-enum loudly, emits the builtin;
|
||||
`interp.zig:reflectTypeInfo` constructs the exact nested-aggregate Value
|
||||
`defineEnum` decodes (variant `{name,payload}` / slice `{data,len}` / EnumInfo /
|
||||
TypeInfo `.enum`). `tagged_union` reflects `field.ty`; payloadless `` `enum ``
|
||||
reflects `void`. Round-trips source AND constructed enums. Enum-only;
|
||||
struct/tuple widening deferred. `examples/0619` locks it. Suite green (676).
|
||||
- **By-value self-reference rejected (issue 0139, F5 partial).** New
|
||||
`checkInfiniteSize` pass (Pass 1g) detects by-VALUE containment cycles (source +
|
||||
comptime types, direct + mutual), emits a loud "infinitely sized" diagnostic,
|
||||
and breaks the cycle (was a `typeSizeBytes` stack-overflow segfault). `*Self`
|
||||
(pointer) stays valid. `examples/1178` locks the message. Suite green (675).
|
||||
- **Self-reference done.** `declare(name)` + `preregisterForwardTypes` (forward
|
||||
type + alias before body lowers) → `*Name` resolves; recursive `*List` enum
|
||||
constructs, matches through the pointer, and traverses recursively. `0618` locks
|
||||
it. `declare` gained its `name` arg; `EnumInfo.name` dropped. Suite green (674).
|
||||
- **declare/define floor established.** The comptime type-construction surface is
|
||||
two primitives (`declare`/`define`); all named constructors are sx. A `::` binding
|
||||
or type-fn body that calls a `Type`-returning fn is comptime-evaluated (the
|
||||
builtins mint the type) — no syntactic constructor recognition in the compiler.
|
||||
Examples 0614 (one-shot) / 0615 (type-fn identity) / 0617 (channel results) on the
|
||||
floor; `field_type` reflection (0616) unchanged.
|
||||
- **Stream carved (earlier).** Selected as the first async-first foundation: gates
|
||||
channel result types (`RecvResult($T)`) and `race`'s synthesized union, fully
|
||||
validated, self-contained, testable in isolation (`06xx` comptime).
|
||||
167
current/PLAN-ASM.md
Normal file
167
current/PLAN-ASM.md
Normal file
@@ -0,0 +1,167 @@
|
||||
# sx Inline Assembly — Implementation Plan (ASM stream)
|
||||
|
||||
**Design source of truth:** [design/inline-asm-design.md](../design/inline-asm-design.md).
|
||||
This plan turns that doc's §II.7 stage-map + §II.8 phasing into ordered,
|
||||
commit-sized, testable steps. Read the design doc first — this file is the
|
||||
*how/when*, not the *what/why*.
|
||||
|
||||
**Surface (decided):**
|
||||
`asm volatile { "template", "=r" -> T, "r" = expr, clobbers(.cc, .memory) }`
|
||||
— brace block; `->` output / `=` input; `clobbers(.…)` dot-name list; N `-> Type`
|
||||
outputs return a tuple; templates are pure AT&T (via LLVM).
|
||||
|
||||
**Feasibility (confirmed):** sx links LLVM@19; `src/llvm_api.zig` `@cImport`s
|
||||
`llvm-c/Core.h`, so `llvm_api.c.*` already exposes `LLVMGetInlineAsm` (9-arg),
|
||||
`LLVMInlineAsmDialectATT`, `LLVMBuildCall2`, `LLVMAppendModuleInlineAsm`. No shim.
|
||||
|
||||
**Relationship to other streams:**
|
||||
- Phases A–E (the inline-asm *expression*) are independent of EXTERN-EXPORT.
|
||||
- Phase F (global asm) consumes `extern`/`export` to import/expose asm symbols —
|
||||
do it **after** `PLAN-EXTERN-EXPORT.md` Phase 2.
|
||||
|
||||
## Cadence (IMPASSIBLE)
|
||||
No commit may both add a test AND make it pass. Each feature step is either a
|
||||
behavior-locking PASSING test, or an xfail test the *next* commit turns green.
|
||||
Arch-pinned tests live in `examples/16xx-platform-asm-*` and declare their target
|
||||
via the `expected/<name>.target` sidecar marker (Phase 0). Never regenerate
|
||||
snapshots while red.
|
||||
|
||||
## Phase 0 — corpus target-gating (test-infra prerequisite; no compiler code)
|
||||
**Why first.** The flagship v1 examples are `x86_64` (syscall-write, divmod,
|
||||
cpuid) but the dev host is `aarch64`-Darwin, and the corpus runner
|
||||
([src/corpus_run.test.zig](../src/corpus_run.test.zig)) currently (a) never threads
|
||||
a per-example `--target` and (b) has no host-arch gate — its only skip is "marker
|
||||
has no `.sx`". So D.0's `…-syscall-write` markers asserting exit/stdout describe
|
||||
output the harness *cannot* produce on this host, which would violate the cadence
|
||||
rule (the "next commit turns it green" can never happen). Phase 0 closes that gap.
|
||||
It touches **only the runner + two fixtures** — zero compiler code, zero risk to
|
||||
A–E, and unblocks every arch-pinned asm example.
|
||||
|
||||
**Marker taxonomy (the cleanup).** The runner currently spreads per-example
|
||||
*directives* across standalone boolean/value sidecars (`.aot` now, `.target`
|
||||
proposed, more later). Replace that sprawl with **one optional config file,
|
||||
`expected/<name>.build`**, holding all build/run directives; the output snapshots
|
||||
(`.exit` / `.stdout` / `.stderr` / `.ir`) stay separate — they are
|
||||
machine-regenerated data, not config. `.exit` remains the **test-discovery key**
|
||||
(every test has one; `.build` is optional).
|
||||
|
||||
**`.build` format** — JSON, parsed with `std.json`:
|
||||
```json
|
||||
{ "aot": true, "target": "x86_64-linux" }
|
||||
```
|
||||
Parse via `std.json.parseFromSlice(BuildConfig, …)` into
|
||||
`struct { aot: bool = false, target: ?[]const u8 = null }`. Field defaults cover
|
||||
omitted keys; `std.json`'s default `ignore_unknown_fields = false` makes an
|
||||
**unknown key a loud `error.UnknownField`** (surfaced as a runner failure, never a
|
||||
silent ignore — CLAUDE.md no-silent-default rule). Extensible: future `"cpu"`,
|
||||
`"link"`, `"cwd"` are just new optional struct fields, no new sidecar file and no
|
||||
custom parser.
|
||||
|
||||
**What the directives do:**
|
||||
|
||||
1. **`target = <triple|shorthand>`** threads `--target <value>` into every `sx`
|
||||
invocation for that example (`run` / `build` / `ir` — `--target` is a global
|
||||
flag, confirmed [main.zig:39](../src/main.zig#L39)), AND **host-match selects
|
||||
the mode.** The runner parses the leading `arch` + `os` tokens of the resolved
|
||||
triple and compares them to `@import("builtin").target` (normalizing
|
||||
`arm64`→`aarch64`):
|
||||
- **match** → *execute* exactly as today (`sx run`, or `aot` build+exec) with
|
||||
the target threaded, plus the `.ir` diff if an `.ir` snapshot exists. ⇒ an
|
||||
x86_64 example gives **real end-to-end coverage on an x86_64 CI runner**.
|
||||
- **mismatch** → **ir-only**: run *only* `sx ir <file> --target <t>`; assert
|
||||
`.exit` (the ir command's exit), `.ir` (normalized stdout), and `.stderr`
|
||||
(diagnostics, normally empty). Do **not** run/build/exec; do **not** assert
|
||||
`.stdout`. An `.ir` snapshot is **required** in ir-only mode — its absence is
|
||||
a loud runner failure ("arch-pinned <name>: ir-only mode requires an .ir
|
||||
snapshot"), never a silent pass. Robust even if `sx ir` treats `--target` as
|
||||
a partial no-op: the `inline_asm` op carries the template + constraint string
|
||||
verbatim, so the IR snapshot still locks the exact thing §II.11 flags as
|
||||
silently-miscompiling (the constraint assembler + template rewrite).
|
||||
2. **`aot`** is the existing JIT-vs-build+exec switch, just relocated from the
|
||||
standalone `.aot` marker into `.build`.
|
||||
|
||||
**Negative compile-error examples need NO `.build`.** `…-missing-volatile`
|
||||
(no-output-without-`volatile`) is a Sema diagnostic raised before codegen/JIT, so
|
||||
plain `sx run` reports it identically on any host — it stays a normal example with
|
||||
no config file.
|
||||
|
||||
**update-goldens interaction:** in ir-only mode, `-Dupdate-goldens` writes `.exit`
|
||||
(ir exit) + `.ir` (+ `.stderr` if non-empty) and skips `.stdout`. Execute mode
|
||||
(incl. `aot`) is unchanged. `.build` is hand-authored — update-goldens never
|
||||
writes it.
|
||||
|
||||
| Step | Commit | What | Files |
|
||||
|---|---|---|---|
|
||||
| 0.0 | lock | Add `BuildConfig` + `std.json` parse of `expected/<name>.build` (unknown-key ⇒ `error.UnknownField`); **migrate** the 2 existing `.aot` markers → `.build` (content `{ "aot": true }`) and delete them; thread `target`'s `--target` into the spawned argv; add `hostMatchesTarget(value) bool` (arch+os token parse, `arm64`→`aarch64`) gating the **execute** path. Lock with `examples/16xx-platform-target-host.sx` (trivial `main`) + a `.build` `{ "target": "<host arch triple>" }` (still runs+passes) and unit `test`s for the JSON parse + `hostMatchesTarget`. | `src/corpus_run.test.zig`, `examples/expected/1226-*.{aot→build}`, `…/1227-*`, + fixture |
|
||||
| 0.1 | lock | Implement the **mismatch ⇒ ir-only** branch (skip run/build/exec; assert `.exit`+`.ir`+`.stderr` from `sx ir --target`; require `.ir`). Lock with `examples/16xx-platform-target-cross.sx` (asm-free `() -> i64 { return 0; }`) + `.build` `{ "target": "x86_64-linux" }` + a checked-in `.ir` snapshot — exercises ir-only on the arm64 host. | `src/corpus_run.test.zig` + fixture |
|
||||
| 0.2 | docs | Update CLAUDE.md §"Test layout"/§"Testing" to document `.build` (format + `aot`/`target` keys) replacing the standalone `.aot` marker prose (lines ~435, ~492). | `CLAUDE.md` |
|
||||
|
||||
Both 0.0 and 0.1 are **lock** commits: the runner change and the fixture that
|
||||
exercises it land together and pass the moment they land (the mechanism works
|
||||
immediately — nothing is left red), which is the cadence rule's "lock in current
|
||||
behavior" flavor, not a feature red→green. No asm lowering is gated on either.
|
||||
|
||||
**Phase 0 verification:** `zig build test` green; deliberately corrupt the
|
||||
cross-target `.ir` fixture and confirm the runner reports an IR mismatch (proves
|
||||
ir-only actually asserts, isn't a no-op); delete it and confirm the
|
||||
"requires an .ir snapshot" failure fires.
|
||||
|
||||
**Estimated runner delta:** ~70–90 lines (sidecar read + `--target` argv threading
|
||||
+ `hostMatchesTarget` + the ir-only branch + update-mode tweak). Within the
|
||||
"no step > ~500 new lines" rule; well under the read budget.
|
||||
|
||||
## Phase A — keyword + AST + parser (parses; no codegen)
|
||||
| Step | Commit | What | Files |
|
||||
|---|---|---|---|
|
||||
| A.0 | lock | add `kw_asm` keyword + map entry; unit lex test `asm → kw_asm` | `src/token.zig`, `src/lexer.zig` + `.test.zig` |
|
||||
| A.1 | xfail | parse `asm { … }` → `AsmExpr`/`AsmOperand` in `parsePrimary`; pin an AST/`sx ir` parse snapshot; lowering still `bailDetail("inline asm codegen unimplemented")` | `src/ast.zig` (:85 union arm, :721 structs), `src/parser.zig` (parsePrimary), `src/ir/interp.zig` |
|
||||
| A.2 | green | parse-shape snapshot lands green; the unimplemented bail is loud + named | — |
|
||||
|
||||
## Phase B — sema / typing
|
||||
| Step | Commit | What | Files |
|
||||
|---|---|---|---|
|
||||
| B.0 | xfail | result-type rule (0→`void` / 1→`T` / N→named-or-positional tuple) + checklist (no-output⇒`volatile`, layout, comptime-string template) — pin error messages | `src/ir/expr_typer.zig` |
|
||||
| B.1 | green | typing + diagnostics implemented; `.unresolved` sentinel on failure (no silent default) | `src/ir/expr_typer.zig`, `src/ir/semantic_diagnostics.zig` |
|
||||
|
||||
## Phase C — IR op + lowering
|
||||
| Step | Commit | What | Files |
|
||||
|---|---|---|---|
|
||||
| C.0 | lock | add `inline_asm: InlineAsm` to `Op` + `AsmOperand` (role/name/constraint/operand) + interp `bailDetail` arm; unit tests for the IR shape | `src/ir/inst.zig` (:80), `src/ir/interp.zig` |
|
||||
| C.1 | xfail→green | `lowerAsmExpr` in `lowerExpr` dispatch — interns template/constraints/clobber-names, lowers input `Ref`s, sets result `TypeId` | `src/ir/lower/expr.zig` |
|
||||
|
||||
## Phase D — LLVM emit (single value-output; the core)
|
||||
| Step | Commit | What | Files |
|
||||
|---|---|---|---|
|
||||
| D.0 | xfail | `examples/16xx-platform-asm-syscall-write.sx` + `…-register-read.sx` + `…-no-output-volatile.sx` + `…-missing-volatile.sx` (expected compile error) — all red | examples + `expected/` markers |
|
||||
| D.1 | green | `emitInlineAsm`: **port `FuncGen.airAssembly`** — constraint-string assembler (outputs `=`/`+`, inputs, `clobbers(.name)`→`~{name}`), `%[name]`→`${N}` / `%%` / `%=` template rewriter, `LLVMGetInlineAsm`+`LLVMBuildCall2`, `sideeffect=volatile`, AT&T dialect | `src/ir/emit_llvm.zig` (emitInst dispatch + handler) |
|
||||
| D.2 | green | lock the template-rewrite + constraint string via an `expected/*.ir` snapshot on `…-template-subst.sx` | examples |
|
||||
|
||||
**Phase D verification:** `zig build test`; the syscall example runs on
|
||||
`x86_64-linux`; IR snapshot matches the design doc's worked `sys_write` lowering.
|
||||
|
||||
## Phase E — multi-return tuples + `clobbers(.…)`
|
||||
| Step | Commit | What | Files |
|
||||
|---|---|---|---|
|
||||
| E.0 | xfail | `…-asm-multi-return.sx` (`divmod`→`(quot,rem)`, `cpuid`→4-tuple) red | examples |
|
||||
| E.1 | green | N `out_value` → LLVM struct return + `extractvalue i` → sx tuple (named when operands named); `clobbers(.name)` dot-name lowering finalized | `src/ir/emit_llvm.zig`, `src/ir/lower/expr.zig` |
|
||||
|
||||
## Phase F — global asm (needs EXTERN-EXPORT Phase 2)
|
||||
| Step | Commit | What | Files |
|
||||
|---|---|---|---|
|
||||
| F.0 | xfail | top-level `asm { … }` decl parsed (reject operands/`volatile`); `…-asm-global.sx` (defines a symbol, imported via `extern`) red | `src/parser.zig`, `src/ast.zig` |
|
||||
| F.1 | green | lower `asm_global` → `c.LLVMAppendModuleInlineAsm`; comptime-call guard (dlsym-miss is loud); blocks concatenate in source order | `src/ir/lower/decl.zig`, `src/ir/emit_llvm.zig`, `src/ir/interp.zig` |
|
||||
|
||||
## Phase G — later (own steps when scheduled)
|
||||
`-> @place` write-through + read-write (`"+r" -> @place`) + indirect-memory
|
||||
(`"=*m"`) outputs · `%=` unique-id · output-to-const rejection · Intel-dialect
|
||||
opt-in · naked functions (`callconv(.naked)`, coordinate with EXTERN-EXPORT).
|
||||
|
||||
## Open decisions (design doc §II.10)
|
||||
Dialect (AT&T-only v1, recommended) · `volatile` contextual-keyword (recommended)
|
||||
· brace separator comma (recommended) · `clobbers(.name)` dot-name sugar now →
|
||||
checked per-arch `Clobber` enum later (Phase 4 of the design doc).
|
||||
|
||||
## End-to-end verification (per phase)
|
||||
`zig build && zig build test`; for arch-pinned examples confirm they run on a
|
||||
matching host or assert on `sx ir`/`.s` snapshots. After intentional output
|
||||
changes only: `zig build test -Dupdate-goldens`, then review the diff.
|
||||
225
current/PLAN-ATOMICS.md
Normal file
225
current/PLAN-ATOMICS.md
Normal file
@@ -0,0 +1,225 @@
|
||||
# PLAN-ATOMICS — Stream A (atomics lowering)
|
||||
|
||||
> **STATUS: ✅ COMPLETE (feature-complete).** All phases A.0 → A.3 landed + green.
|
||||
> Surface shipped: `Atomic($T)` `load`/`store`/`fetch_add`/`sub`/`and`/`or`/`xor`/`min`/`max`/
|
||||
> `swap`/`compare_exchange`/`compare_exchange_weak` (all comptime `$o: Ordering`) + free
|
||||
> `fence(.ordering)`. IR ops `atomic_load`/`store`/`rmw`/`cmpxchg`/`fence`. Both LLVM emit
|
||||
> AND the comptime VM implemented (verified to agree). Enabled by net-new comptime value
|
||||
> params (enum/tagged_union/generic-struct methods — 3 commits). Corpus `17xx` (1700-1704) +
|
||||
> `11xx` diagnostics (1130/1131/1186/1187). Commits: 22af404, 64c7db5, 8144a88, acf3183,
|
||||
> 718f27e, 0531164, 68ed732, dca396e, 79895be, fca4304, b65544a (+ comptime-param 3c4305f,
|
||||
> d7a6857, d95ba0a). **Unblocks Stream B2 channels + Stream C parallel schedulers.**
|
||||
>
|
||||
> Deferred (documented, NOT legacy — intentional scope): RMW/CAS/swap are integer-only
|
||||
> (float fadd / pointer atomics out of scope); fence/orderings explicit (no defaults — the
|
||||
> comptime-default-on-generic-method gap is orthogonal). Asm-level arch divergence +
|
||||
> weak-memory *semantics* remain OUT of corpus scope (Stream-C stress harness).
|
||||
|
||||
Carved from [PLAN-POST-METATYPE.md](PLAN-POST-METATYPE.md) Stream A + the design-of-record
|
||||
[../design/execution-evolution-roadmap.md](../design/execution-evolution-roadmap.md) §3 (N1)
|
||||
+ §4.6 (locked surface). Progress in [CHECKPOINT-ATOMICS.md](CHECKPOINT-ATOMICS.md).
|
||||
|
||||
**Goal:** net-new LLVM atomic codegen. Surface = a pure-sx `Atomic($T)` generic struct +
|
||||
an `Ordering` enum (ordinary sx), with the actual atomic operations recognized as
|
||||
`#builtin` intrinsics at lower-time and emitted as new IR ops. This is **100% net-new** —
|
||||
no atomics scaffolding exists (the only `lower.zig` "ordering" is *comparison* ordering
|
||||
`< <= >=`, unrelated to memory ordering — do not mistake it for groundwork).
|
||||
|
||||
**Cadence (IMPASSIBLE):** no commit both adds a test AND makes it pass (lock-to-bail, then
|
||||
flip to green); `zig build && zig build test` green after every step; never regen snapshots
|
||||
while red; scope regens with `-Dname=examples/NNNN-…sx -Dupdate-goldens` + review the diff.
|
||||
New corpus category: `17xx` atomics.
|
||||
|
||||
---
|
||||
|
||||
## Design (grounded against the tree)
|
||||
|
||||
### Representation — minimal compiler surface
|
||||
- **`Ordering`** is an ordinary sx enum, zero compiler coupling:
|
||||
```sx
|
||||
Ordering :: enum { relaxed; acquire; release; acq_rel; seq_cst; } // tags 0..4
|
||||
```
|
||||
- **`Atomic($T)`** is an ordinary sx **generic struct** (mirrors `List :: struct ($T: Type)`
|
||||
at [list.sx:5](../library/modules/std/list.sx#L5)), a transparent 1-field wrapper —
|
||||
atomicity is a property of the *operation*, not the storage, so `Atomic(i64)` has the
|
||||
exact layout/size/align of `i64`. NO new IR *type*, NO type-system coupling:
|
||||
```sx
|
||||
Atomic :: struct ($T: Type) {
|
||||
value: T;
|
||||
init :: (v: T) -> Atomic(T) { return .{ value = v }; }
|
||||
load :: (self: *Atomic(T), o: Ordering) -> T { return atomic_load(T, @self.value, o); }
|
||||
store :: (self: *Atomic(T), v: T, o: Ordering) { atomic_store(T, @self.value, v, o); }
|
||||
}
|
||||
```
|
||||
- The **operations** are `#builtin` intrinsic free functions, recognized by name at
|
||||
lower-time (the established pattern — `size_of`/`type_info` in
|
||||
[`tryLowerReflectionCall`](../src/ir/lower/call.zig#L1672), recognized BEFORE arg lowering):
|
||||
```sx
|
||||
atomic_load :: ($T: Type, ptr: *T, o: Ordering) -> T #builtin;
|
||||
atomic_store :: ($T: Type, ptr: *T, v: T, o: Ordering) #builtin;
|
||||
```
|
||||
Explicit `$T` first arg follows the `size_of($T)` / `field_name($T, idx)` mixed
|
||||
type+value precedent (lowest-risk; the reflection path already resolves type args).
|
||||
|
||||
### Ordering is compile-time-only by construction — and that forces a capability gap
|
||||
LLVM atomic ordering is an **instruction attribute**, not a runtime operand, so the
|
||||
ordering MUST be known at emit time. The lower-time handler reads the ordering arg's
|
||||
variant name statically (it must be a **constant enum literal** `.seq_cst`) and bakes it
|
||||
into the IR op as a Zig enum field (`AtomicOrdering`). A non-literal ordering is a **loud
|
||||
diagnostic**, never a silent default (REJECTED-PATTERNS).
|
||||
|
||||
**Discovered gap (grounded):** a generic `Atomic(T)` method `load(self, o: Ordering)` would
|
||||
forward `o` — a *runtime parameter* — to the intrinsic, where it is NOT a literal. And
|
||||
**comptime enum value params don't exist** (`$o: Ordering` → `o` is "unresolved" in the
|
||||
body; `resolveValueParamArg` folds integer constraints only). A runtime dispatch hack
|
||||
(`if o == { case .acquire: atomic_load(…, .acquire) … }`) also fails: `load` with a
|
||||
`release`/`acq_rel` ordering is *invalid LLVM*, so the arms can't be uniform. Therefore the
|
||||
**full ordering surface is blocked on a net-new capability** (comptime-constant ordering
|
||||
propagation — either comptime enum value params, or compiler-recognized `Atomic` method
|
||||
calls). That capability is its **own step (A.0.5)**, sequenced before ordering-bearing ops.
|
||||
|
||||
### sx tag → LLVM ordering is EXPLICIT (non-contiguous!)
|
||||
LLVM's `LLVMAtomicOrdering` is **not** 0..4: `Monotonic=2, Acquire=4, Release=5,
|
||||
AcquireRelease=6, SequentiallyConsistent=7` ([Core.h:338-354]). The sx `Ordering` tags
|
||||
(relaxed=0…seq_cst=4) map via an explicit `switch`, never an identity cast:
|
||||
`relaxed→Monotonic, acquire→Acquire, release→Release, acq_rel→AcquireRelease,
|
||||
seq_cst→SequentiallyConsistent`.
|
||||
|
||||
### LLVM-C API (verified present in `llvm-c/Core.h`, no new extern decls needed)
|
||||
- Atomic load = `LLVMBuildLoad2` + `LLVMSetOrdering(v, ord)` + `LLVMSetAlignment(v, size)`
|
||||
(**alignment is mandatory** on atomic load/store — LLVM verifier rejects atomics without
|
||||
it). There is **no** `LLVMBuildAtomicLoad`/`Store` (the Explore agent was wrong).
|
||||
- Atomic store = `LLVMBuildStore` + `LLVMSetOrdering` + `LLVMSetAlignment`.
|
||||
- (Later) `LLVMBuildAtomicRMW(B, op, ptr, val, ord, singleThread)`,
|
||||
`LLVMBuildAtomicCmpXchg(B, ptr, cmp, new, succOrd, failOrd, singleThread)`,
|
||||
`LLVMBuildFence(B, ord, singleThread, name)`, `LLVMSetWeak`.
|
||||
- `singleThread = 0` (multi-thread / cross-thread ordering). Atomic-eligible `T` =
|
||||
integer / pointer / float of size 1·2·4·8(·16). **Reject non-scalar / bad-size `T`
|
||||
loudly** (diagnostic), do not silently emit.
|
||||
|
||||
### Comptime VM treats atomics as ordinary load/store
|
||||
Comptime is single-threaded, so seq_cst is trivially satisfied — the
|
||||
[`comptime_vm`](../src/ir/comptime_vm.zig#L659) arms for `atomic_load`/`atomic_store`
|
||||
reuse the ordinary `load`/`store` paths (correct, NOT a bail). `sx run` JITs via LLVM so
|
||||
runtime atomics execute the real ops; the VM arm only matters for `#run`/const-init.
|
||||
|
||||
### Files the new IR op variants force (exhaustive switches)
|
||||
`atomic_load` / `atomic_store` variants must be handled in every `Op` switch or the Zig
|
||||
build fails (this is the desired tripwire):
|
||||
- [inst.zig:159](../src/ir/inst.zig#L159) — add `atomic_load: AtomicLoad`, `atomic_store: AtomicStore` + the structs (mirror `Store` at [inst.zig:286](../src/ir/inst.zig#L286)).
|
||||
- [lower/call.zig:1672](../src/ir/lower/call.zig#L1672) — recognize the intrinsics, emit the ops (new `tryLowerAtomicIntrinsic`, called alongside `tryLowerReflectionCall` at [call.zig:80](../src/ir/lower/call.zig#L80)).
|
||||
- [print.zig:231](../src/ir/print.zig#L231) — print arms (sx-IR / `ir-dump`).
|
||||
- [emit_llvm.zig:1566](../src/ir/emit_llvm.zig#L1566) — dispatch arms → ops.zig.
|
||||
- [backend/llvm/ops.zig:325](../src/backend/llvm/ops.zig#L325) — `emitAtomicLoad`/`emitAtomicStore` (mirror `emitLoad`/`emitStore`).
|
||||
- [comptime_vm.zig:659](../src/ir/comptime_vm.zig#L659) — arms reusing load/store.
|
||||
- Any other `.op` switch the Zig compiler flags (module.zig / program_index.zig) — let the build tell you.
|
||||
|
||||
### Test snapshots — the arch-`.ir` requirement is a MISCONCEPTION for atomics
|
||||
`sx ir` = [`emitIR`](../src/main.zig#L210), which emits **LLVM IR** (respects `--target`);
|
||||
`sx ir-dump` is the sx-IR printer. At the **LLVM-IR level**, `load atomic i64, ptr %x
|
||||
seq_cst, align 8` is **arch-invariant** — identical text for x86_64 and aarch64. The
|
||||
x86-`lock`/MOV vs aarch64-`ldar`/`stlr` divergence happens only at *instruction selection*
|
||||
(`sx asm`), which the corpus does **not** snapshot. So:
|
||||
- **A single host `.ir` snapshot** proves the achievable gate (the `load atomic <ordering>`
|
||||
keyword + correct ordering + alignment emitted). PLAN-POST §A / design §10.3's
|
||||
"arch-gated x86_64 + aarch64 `.ir`" would capture **byte-identical** files — drop it.
|
||||
- Optionally add ONE cross-arch ir-only example (`.build {"target":"x86_64-linux"}` on an
|
||||
aarch64 host) purely as a **cross-target-emission-doesn't-crash** smoke — note in its
|
||||
header that the IR body is identical to host.
|
||||
- **State loudly (out of snapshot scope, parallel to the ordering-semantics caveat):**
|
||||
asm-level arch lowering AND weak-memory ordering *semantics* are NOT proven by `.ir`;
|
||||
those need the Stream-C stress harness, not the corpus.
|
||||
|
||||
---
|
||||
|
||||
## Phases
|
||||
|
||||
### A.0 — `Atomic($T)` + `Ordering` + **`seq_cst`-only** `load`/`store` ← START HERE
|
||||
**Scope (descoped per the discovered gap above):** ship the net-new atomic load/store
|
||||
codegen with a **`seq_cst` literal baked in the method bodies** — `load(self) -> T` /
|
||||
`store(self, v)` (NO ordering param yet). The intrinsic still carries the full
|
||||
`AtomicOrdering` field (always `.seq_cst` here); the recognizer + emit handle all five
|
||||
orderings already, so A.0.5 only has to plumb the *constant* through. Explicit orderings
|
||||
(`a.load(.acquire)`) land in A.0.5. seq_cst-only is correct (conservative-strongest), not a
|
||||
silent fallback.
|
||||
|
||||
Two-commit cadence (lock-to-bail → green):
|
||||
|
||||
- **A.0a (lock)** — land the lib + IR plumbing with emit deliberately bailing:
|
||||
1. New `library/modules/std/atomic.sx`: `Ordering` enum, `Atomic($T)` struct (value +
|
||||
`init`/`load`/`store`), `atomic_load`/`atomic_store` `#builtin` decls. **Opt-in import
|
||||
(`#import "modules/std/atomic.sx"`), NOT carried by the universal `std.sx` facade** —
|
||||
mirrors `trace`. Rationale (grounded): adding the concrete `Ordering` enum to the
|
||||
universal prelude registers it into EVERY program's global type table, growing
|
||||
`@__sx_type_is_unsigned` (378→380) and shifting all string-global numbering → churned
|
||||
37 unrelated `.ir` snapshots + bloats every binary. Atomics is a deliberate concurrency
|
||||
capability, so consumers import it explicitly.
|
||||
2. Add IR ops `atomic_load`/`atomic_store` + `AtomicOrdering` + the two op structs
|
||||
(inst.zig); print arms; comptime_vm arms (reuse load/store); lower recognition
|
||||
(`tryLowerAtomicIntrinsic`) incl. the const-ordering-literal guard + non-scalar-`T`
|
||||
reject.
|
||||
3. emit_llvm/ops.zig arms **bail loudly** for now: `emitAtomicLoad`/`Store` call the
|
||||
emitter's bail-with-diagnostic path ("atomic load/store LLVM emission not yet
|
||||
implemented") so the Zig build is exhaustive but the example is red-by-diagnostic.
|
||||
4. Add `examples/1700-atomics-load-store.sx` (construct `Atomic(i64).init`, `store`,
|
||||
`load`, `print`). Seed marker; capture snapshot = the emit-bail diagnostic (nonzero
|
||||
exit). `zig build && zig build test` green (matches the locked bail snapshot). Commit.
|
||||
- **A.0b (green)** — replace the emit bail with real emission:
|
||||
`LLVMBuildLoad2`+`LLVMSetOrdering`+`LLVMSetAlignment` / `LLVMBuildStore`+`LLVMSetOrdering`
|
||||
+`LLVMSetAlignment`, ordering via the explicit sx-tag→LLVM `switch`. Regen `1700` to
|
||||
success output + capture its host `.ir` (asserts `load atomic`/`store atomic` + ordering).
|
||||
Add a unit test in `emit_llvm.test.zig` (correct op + ordering + alignment emission).
|
||||
Review the diff (no stray error text). Commit.
|
||||
|
||||
### A.0.5 — comptime-constant ordering propagation (the capability gap)
|
||||
Enable `a.load(.acquire)` etc. — i.e. an `Ordering` that reaches the intrinsic as a
|
||||
compile-time constant through a method. Two candidate designs (pick at pickup):
|
||||
- **(a) comptime enum value params** — make `$o: Ordering` resolve in the body to its
|
||||
variant tag (extend `comptime_value_bindings`/the typer beyond integers). General,
|
||||
reusable; larger typer change.
|
||||
- **(b) compiler-recognized `Atomic` methods** — special-case `Atomic(T).load/store/…`
|
||||
calls (read the literal ordering arg at the method call site), bounded coupling to the
|
||||
std `Atomic` type (cf. how `Vector` is special-cased). Smaller; less general.
|
||||
Also enforce per-op ordering validity (load: relaxed/acquire/seq_cst; store:
|
||||
relaxed/release/seq_cst; CAS's dual orderings) as **compile errors**, which is exactly what
|
||||
the constant-ordering path buys. Retrofit the ordering param onto `load`/`store` here.
|
||||
|
||||
### A.1 — RMW: `fetch_add/sub/and/or/xor` + `fetch_min/max` → `atomicrmw` (no `nand`)
|
||||
One IR op `atomic_rmw` carrying an `RmwKind` (maps to `LLVMAtomicRMWBinOp*`). Signed vs
|
||||
unsigned min/max picks `Max/Min` vs `UMax/UMin` from `T`'s signedness. Same lock→green
|
||||
cadence; `17xx` examples.
|
||||
|
||||
### A.2 — `compare_exchange`/`_weak` → `cmpxchg` (returns **`?T`, null = success**)
|
||||
`atomic_cmpxchg` op (ptr, cmp, new, success_ord, failure_ord, weak). LLVM `cmpxchg`
|
||||
returns `{T, i1}`; lower to `?T` where **null = success** (extract the i1, invert).
|
||||
**Validate the two orderings in the compiler** (design §4.6): failure ordering may not be
|
||||
`release`/`acq_rel` nor stronger than success — loud diagnostic. `_weak` sets `LLVMSetWeak`.
|
||||
|
||||
### A.3 — `swap` + `fence(.ordering)`
|
||||
`swap` = `atomic_rmw` with `Xchg` kind (folds into A.1's op). `fence` = a new `atomic_fence`
|
||||
op (ordering only) → `LLVMBuildFence`. `17xx` examples.
|
||||
|
||||
---
|
||||
|
||||
## Gates (per the corrected snapshot story)
|
||||
- **unit** `emit_llvm.test.zig`: each op emits the right LLVM builder + ordering + alignment.
|
||||
- **corpus** `17xx` single-thread deterministic (`sx run`, JIT executes real atomics).
|
||||
- **host `.ir`** snapshot per op proves the keyword/ordering/alignment lowered.
|
||||
- **OUT of snapshot scope, stated loudly:** asm-level arch divergence (`sx asm`) and
|
||||
weak-memory ordering *semantics* — Stream-C stress harness territory, not the corpus.
|
||||
|
||||
## Kickoff prompt (A.0a — paste into a fresh session)
|
||||
> Implement Stream A step A.0a (atomics lock commit) per `current/PLAN-ATOMICS.md`. Verify
|
||||
> `zig build && zig build test` is green first. Then: (1) create
|
||||
> `library/modules/std/atomic.sx` with the `Ordering` enum, `Atomic($T)` struct, and
|
||||
> `atomic_load`/`atomic_store` `#builtin` decls; wire into `library/modules/std.sx`'s tail.
|
||||
> (2) Add the `atomic_load`/`atomic_store` IR ops + `AtomicOrdering` + op structs in
|
||||
> `src/ir/inst.zig`; handle them in every exhaustive `Op` switch the Zig build flags
|
||||
> (print.zig, comptime_vm.zig reuse load/store, emit_llvm dispatch). (3) Add
|
||||
> `tryLowerAtomicIntrinsic` in `src/ir/lower/call.zig` (recognize the two builtins, bake the
|
||||
> const ordering literal into the op, loud-reject non-literal ordering AND non-scalar/bad-size
|
||||
> `T`). (4) Make `emitAtomicLoad`/`emitAtomicStore` in `src/backend/llvm/ops.zig` BAIL loudly
|
||||
> ("not yet implemented") this commit. (5) Add `examples/1700-atomics-load-store.sx`, seed the
|
||||
> marker, capture the bail diagnostic as the locked snapshot, confirm `zig build test` green,
|
||||
> commit. STOP — A.0b (real emission) is the next step. Do NOT implement emission in the same
|
||||
> commit that adds the example.
|
||||
741
current/PLAN-COMPILER-VM.md
Normal file
741
current/PLAN-COMPILER-VM.md
Normal file
@@ -0,0 +1,741 @@
|
||||
# PLAN — Comptime Bytecode VM + comptime memory (then re-home the compiler-API on it)
|
||||
|
||||
> **Direction change (2026-06-17).** The comptime compiler-API stream pivots off the
|
||||
> **byte-weld**. The weld (sx structs whose layout is validated to mirror the
|
||||
> compiler's Zig types) + the **serialization / marshaling** bridge at the call
|
||||
> boundary is the wrong direction — it bolts a parallel layout regime and hand-built
|
||||
> byte-copies onto a comptime value model that fundamentally isn't bytes. We strip it
|
||||
> and build the right foundation: a **bytecode VM over byte-addressable
|
||||
> memory**, where comptime values ARE native bytes (like runtime). On that base the
|
||||
> compiler-API needs no weld, no validation, no marshaling — the compiler's own types
|
||||
> are read/built directly as memory and its functions take/return real pointers.
|
||||
>
|
||||
> Supersedes the build order in `design/comptime-compiler-api.md` (kept for history).
|
||||
> This is the active plan for the stream. Branch: `reify`.
|
||||
|
||||
## Why
|
||||
|
||||
`src/ir/interp.zig` is a tree-walking interpreter over the SSA IR that represents
|
||||
every value as a tagged `Value` union (`int`, `float`, `aggregate: []const Value`,
|
||||
`type_tag`, `heap_ptr`, …). Two consequences:
|
||||
|
||||
1. **Slow.** Per-value boxing in a tagged union; per-op `switch` over `Inst`; an
|
||||
aggregate is a heap `[]const Value`, walked element-by-element.
|
||||
2. **Not native memory.** A struct value is `[]const Value` (tagged unions), NOT the
|
||||
struct's bytes. So a comptime `@ptrCast(*StructInfo)` reads the `Value` union's
|
||||
memory, not a `StructInfo` — which forced the whole weld+marshal detour.
|
||||
|
||||
Make comptime values **native bytes in byte-addressable memory** and both problems dissolve:
|
||||
structs/arrays/slices are their bytes at natural layout (no weld), the compiler's own
|
||||
records are directly addressable (no marshal), and a bytecode loop over comptime memory is
|
||||
fast.
|
||||
|
||||
## End state
|
||||
|
||||
- Comptime execution = a **bytecode VM** over a **byte-addressable memory** (real
|
||||
host-allocated bytes; layout is **target-aware** via the type table's sizes). Values
|
||||
are bytes at addresses plus a scalar register file. No tagged `Value` union.
|
||||
- The comptime compiler-API: the compiler **exposes its real types + functions** to
|
||||
comptime sx. sx reads/builds them as native memory and calls compiler functions by
|
||||
pointer. No `abi(.zig)` weld, no `validateStructLayout`, no `register_struct`
|
||||
field-by-field marshaling — gone.
|
||||
- `declare`/`define`/`type_info` and `#compiler`/`BuildOptions` ride this one
|
||||
mechanism; the bespoke interp arms are deleted.
|
||||
- **ONE evaluator at the end — non-negotiable.** The legacy tagged-`Value` interpreter
|
||||
(`interp.zig`) is **DELETED**. We do NOT ship both permanently. "Dual-path"
|
||||
(a compiler-API fn with both a legacy `compiler_lib` handler AND a VM-native impl) and
|
||||
the emit-time legacy fallback are **transitional only** — scaffolding while the VM
|
||||
reaches parity at BOTH comptime sites (emit time AND lowering time). The flag
|
||||
`-Dcomptime-flat` is the swap mechanism; once the VM runs everywhere with parity, the
|
||||
flag, the fallback, and `interp.zig` all go. Any "VM-only at emit, legacy at lowering"
|
||||
split is a waypoint, never the destination.
|
||||
|
||||
## Principles (hold at every step)
|
||||
|
||||
- **Green at every step.** `zig build && zig build test` pass after each sub-step. The
|
||||
existing tagged-`Value` interpreter stays the live evaluator until the VM reaches
|
||||
corpus parity; swap behind a build flag, then delete the old path.
|
||||
- **Target-aware, not host-baked.** Flat-memory layout uses the type table's target
|
||||
sizes (`pointer_size`, `typeSizeBytes`/offsets), NEVER host `@sizeOf`. This is what
|
||||
keeps cross-compilation correct (the JIT-comptime alternative could not).
|
||||
- **Sandboxed.** Flat-memory accesses are bounds-checked; step/call-depth budgets
|
||||
remain; an OOB / bad access traps to a build-gating diagnostic with a source span —
|
||||
never a compiler-process crash.
|
||||
- **No silent fallbacks** (per CLAUDE.md): an unhandled op / shape bails loudly with a
|
||||
named reason, never a zero/default that looks like success.
|
||||
|
||||
## Phases
|
||||
|
||||
### Phase 0 — Strip the weld / serialize / marshal machinery
|
||||
Delete the wrong-direction code so the VM builds on a clean base. Pure removal +
|
||||
corpus rebaseline; suite green.
|
||||
|
||||
- `src/ir/compiler_lib.zig`: the reflection (`weldStruct` / `bound_types` /
|
||||
`FieldLayout` / `BoundType`), the layout validation (`validateStructLayout` /
|
||||
`LayoutMismatch` / `SxField`). Decide the fate of the `bound_fns` host-call registry
|
||||
(`intern`/`text_of` handlers) — it is likely subsumed by the VM's compiler-call path
|
||||
in Phase 3, but `intern`/`text_of` may survive as the first such calls.
|
||||
- `src/ir/lower/nominal.zig`: `validateWeldedStruct` + `weldedFieldOrderStr` + the
|
||||
`sd.abi == .zig` validation call in `registerStructDecl`.
|
||||
- `src/ir/interp.zig`: the `compiler_welded` dispatch branch.
|
||||
- `src/backend/llvm/ops.zig`: the `emitCall` comptime-only gate keyed on
|
||||
`compiler_welded` (re-derive the comptime-only guard from a non-weld signal if still
|
||||
needed).
|
||||
- Corpus: retire / convert the weld examples + diagnostics — `0625`, `0627` (welded
|
||||
struct), `1183`, `1186` (weld-layout diagnostics), `1184`/`1185` (welded-fn). Keep
|
||||
`0626` (`intern`/`text_of` round-trip) only if it survives the new call path.
|
||||
- **Keep (re-evaluate in Phase 3), independent of the weld semantics:** the
|
||||
`#library "compiler"` decl, the `abi(.x)` annotation + `extern <lib>` syntax, and the
|
||||
`callconv → abi` unification. These are surface syntax that may still serve the
|
||||
compiler-API; only the *weld semantics* are stripped here.
|
||||
|
||||
**Verification:** `zig build test` green with the weld machinery gone; the surviving
|
||||
syntax still parses (parser unit tests).
|
||||
|
||||
### Phase 1 — Flat-memory value model (still IR-walking, no bytecode yet)
|
||||
Introduce comptime memory and move comptime values onto it, **decoupled from bytecode** so
|
||||
the value-model change is isolated. Each sub-step ports one op group and keeps the
|
||||
corpus green; the OLD tagged path stays behind a build flag (`-Dcomptime-flat`) until
|
||||
all groups land, then the shim is deleted.
|
||||
|
||||
1. **Machine + scalars.** A comptime memory region (host `[]u8`) with a stack (frames) +
|
||||
bump-allocated heap, and a scalar register file. Port `int`/`float`/`bool`/`undef`
|
||||
and arithmetic/compare/branch. Aggregates still go through a compat shim to the old
|
||||
representation.
|
||||
2. **Aggregates.** Structs/arrays/tuples laid out in comptime memory at **target** layout;
|
||||
port `struct_init` / `struct_get` / `array` / `index_gep` to read/write bytes at
|
||||
computed offsets.
|
||||
3. **Slices / strings.** `{ptr, len}` fat pointers in comptime memory.
|
||||
4. **Optionals / enums / tagged unions.** Tag + payload bytes.
|
||||
5. **Pointers.** `alloca` / `store` / `load` / GEP unified onto comptime addresses; retire
|
||||
`slot_ptr` / `heap_ptr` / `byte_ptr` in favor of comptime addresses.
|
||||
6. **Closures.** Fn id + captured env materialized in comptime memory.
|
||||
7. **Extern / host calls.** A struct arg is already bytes → pass its address; this
|
||||
removes most of `marshalExternArg`.
|
||||
8. **Reflection / minting.** `declare` / `define` / `type_info` read comptime
|
||||
values; type-table mutation copies escaping data into compiler-owned memory at the
|
||||
boundary (lifetime), as today.
|
||||
|
||||
**Verification:** with `-Dcomptime-flat` the full corpus (currently 692) is byte-for-
|
||||
byte identical to the tagged path; then make the VM the default and delete the shim.
|
||||
|
||||
### Phase 2 — Bytecode
|
||||
Compile a comptime function's IR → a compact bytecode and execute the bytecode instead
|
||||
of walking `Inst`. Pure encoding/speed; semantics identical to Phase 1. Land at least a
|
||||
minimal register-bytecode loop (the stream's stated goal is a *bytecode* VM); a
|
||||
fragment cache is optional follow-up.
|
||||
|
||||
**Verification:** corpus identical to Phase 1; comptime throughput measurably improved
|
||||
on a heavy-comptime micro-benchmark.
|
||||
|
||||
### Phase 1.final — host wiring (the remaining integration)
|
||||
The wiring ENTRY POINT exists: `comptime_vm.tryEval(gpa, module, func_id) ?Value` runs a
|
||||
comptime function entirely on the VM and returns a legacy `Value`, or `null` to fall
|
||||
back. Unit-tested (pure `6*7` → 42; unsupported → null). Remaining to actually route the
|
||||
host through it:
|
||||
1. **Panic→error hardening (prerequisite).** `Machine.readWord`/`writeWord`/`bytes`
|
||||
currently `assert` (debug panic) on null/OOB. For arbitrary host functions to be
|
||||
safe, make them return `error.OutOfBounds` so a malformed run BAILS (→ null → legacy)
|
||||
instead of crashing the compiler. Ripples through `readField`/`writeField`/slice
|
||||
helpers (add `try`).
|
||||
2. **Implicit context.** Host comptime functions may have `has_implicit_ctx` (param 0 =
|
||||
`*Context`); the legacy `run` materializes a default ctx. The VM `run` does not — so
|
||||
either materialize it too, or only route `tryEval` at funcs without implicit ctx.
|
||||
3. **Wire one site** behind a flag/env (`SX_COMPTIME_FLAT`, → `-Dcomptime-flat` later):
|
||||
the const-init fold in `emit_llvm.zig` `emitGlobals` (`result = tryEval(...) orelse
|
||||
interp.call(...)`). Default off → corpus unaffected.
|
||||
4. **Parity + coverage.** Run the corpus with the flag ON; results must be byte-identical
|
||||
to legacy. Measure how many comptime evals the VM already handles; the bail `detail`s
|
||||
name what to port next (tagged-union payload / any / closures / builtins).
|
||||
5. Grow coverage (port the deferred ops + `call_builtin`/`compiler_call` via the bridge)
|
||||
until the VM is the default and the legacy path is deleted.
|
||||
|
||||
**Status (2026-06-17): steps 1–4 DONE; step 5 = the next session.**
|
||||
- **(1) Hardening — DONE.** `Machine.readWord`/`writeWord`/`bytes` return
|
||||
`error.OutOfBounds` (null / out-of-range / oversized / overflow-safe) instead of
|
||||
asserting. `OutOfBounds` added to `Vm.Error`; `try` threaded through
|
||||
`readField`/`writeField`/`optHas`/`makeSlice`/`sliceLen`/`sliceData`/`elemAddr` and
|
||||
every exec arm + the bridge. New unit tests: hardened-accessor OOB returns, and a
|
||||
null-deref function → `tryEval` returns `null` (legacy fallback), not a panic.
|
||||
- **(2) Implicit context — DONE (materialized, 2026-06-17 step 5).** Initially a
|
||||
conservative skip; now `tryEval` MATERIALIZES the implicit ctx: a comptime entry with
|
||||
`has_implicit_ctx` (whose sole param is the `*Context`) gets a zeroed `Context` of the
|
||||
right size/align allocated in comptime memory, its address passed as arg 0. The common
|
||||
const body never reads the ctx; a body that USES the allocator loads a fn from it and
|
||||
`call_indirect`s (unported) → bails → legacy. No func-ref materialization was needed:
|
||||
handled bodies don't read the ctx contents, and gate-ON corpus parity (688, 0 failed)
|
||||
empirically confirms no divergence. (A body that read+branched on a null allocator fn
|
||||
could in principle diverge; none does — parity is the guard.)
|
||||
- **(3) Wire one site — DONE.** Const-init fold in `emitGlobals` is `(if comptime_flat)
|
||||
tryEval(...) else null) orelse interp.call(...)`. Gated by env `SX_COMPTIME_FLAT`
|
||||
(a `LLVMEmitter.comptime_flat` field read once from `std.c.getenv` in `init`).
|
||||
Default OFF → corpus unaffected (688 green).
|
||||
- **(4) Parity + coverage — DONE.** Gate ON: full corpus byte-identical (688, 0 failed);
|
||||
manual `sx run` of 0605/0606/0607/0608 byte-identical to gate-OFF. Coverage-trace
|
||||
facility in place (`comptime_vm.last_bail_reason` + env `SX_COMPTIME_FLAT_TRACE`,
|
||||
printing HANDLED / fallback+reason per init).
|
||||
- **(5) Implicit-context materialization + memory builtins + f32 — DONE; op-porting CONTINUES.**
|
||||
Coverage climbed **0 → 16 → 27** handled corpus const-inits (fallbacks 22 → 11); parity
|
||||
stays **688/688** (gate ON and OFF) at every step. Landed, in order: implicit ctx
|
||||
materialized (→16); `writeField` null-aggregate fix (storing a `null` non-pointer
|
||||
optional `null_addr` sentinel into an aggregate slot OOB-bailed → now ZEROES the
|
||||
destination = none/empty; unit-test regression); curated libc MEMORY builtins on comptime
|
||||
memory (`Vm.callMemBuiltin`: `malloc`/`calloc` → `allocBytes` 16-aligned & 256-MiB-capped,
|
||||
`free` → no-op, `memcpy`/`memmove`/`memset` on comptime bytes — sandboxed, target-aware,
|
||||
result byte-identical to legacy; unlocked `0604`'s 11 comptime mallocs); and an **f32
|
||||
storage fix** (float registers hold f64 bits, but f32 memory is the 4-byte single —
|
||||
`readField`/`writeField` now `@floatCast` instead of truncating the f64 bits, which had
|
||||
written zeros for `1.0`; a real latent bug `0604` surfaced; unit tests added).
|
||||
- **(6) Real default context + call_indirect + func_ref + global_get — DONE.** Coverage
|
||||
**27 → 31** handled (fallbacks 11 → 7); parity stays **688/688** both gate ON and OFF.
|
||||
Per the user's direction ("the VM can set up a default context"), `runEntry` now
|
||||
materializes the REAL default context (not a zeroed one): the implicit-ctx param is an
|
||||
opaque `*void`, so `materializeDefaultContext` finds the `__sx_default_context` global
|
||||
and lays its initializer constant (`{ {null, alloc_fn, dealloc_fn}, null }`, carrying
|
||||
the CAllocator thunk func-refs) into comptime memory via a new recursive `layoutConst`.
|
||||
With `func_ref` (a function value encoded as `FuncId.index() + 1` so word 0 stays
|
||||
reserved for the NULL function pointer — `funcRefWord`/`funcRefToId`) and `call_indirect`
|
||||
(decode the callee word → `FuncId` → dispatch; 0 → bail) ported, a comptime body
|
||||
that allocates via `context.allocator` now runs ENTIRELY on the VM: `alloc_string` →
|
||||
`context.allocator.alloc_bytes` → `call_indirect` → thunk → `CAllocator.alloc_bytes` →
|
||||
`libc_malloc` → the VM's native comptime `malloc`. Unlocked `0606` (string global via
|
||||
the allocator). Also: `global_get` lazily evaluates a comptime global's `comptime_func`
|
||||
(memoized in `global_cache`) — unlocked `CT_CHAIN`; struct field access (`fieldOffset`/
|
||||
`struct_get`) now handles string/slice `{ptr@0,len@8}` fat pointers (needed by
|
||||
`alloc_string`'s `s.ptr`/`s.len`); and `regToValue` maps a function-typed word back to
|
||||
`.func_ref` so a func-ref result serializes identically to legacy (kept `1128`'s
|
||||
rejection diagnostic byte-identical). Unit tests added (global_get, func_ref +
|
||||
call_indirect). **Note: native `malloc` is still REQUIRED** — the CAllocator thunk
|
||||
bottoms out at libc `malloc`, and the VM can't use a host pointer with comptime
|
||||
load/store, so comptime `malloc` must allocate from comptime memory. The default context
|
||||
lets the allocator PROTOCOL run; native `malloc` is its final step.
|
||||
- **(7) `is_comptime` + failable/error cluster + the signed-load fix — DONE.** Coverage
|
||||
**31 → 36** handled (fallbacks 7 → 2); parity stays **688/688** both gate ON and OFF.
|
||||
- **`is_comptime`** → always 1 on the VM (folds to false in compiled code). Unlocked `1030`.
|
||||
- **Failable / error-channel cluster** (`1037` escape, `1038` handled): `kindOf(error_set)
|
||||
→ word` (a u32 tag id); `regToValue` now bridges TUPLES (the failable `(value…, tag)`
|
||||
shape the host's `checkComptimeFailable` reads); `trace_frame` packs `(func_id<<32 |
|
||||
span.start)` from a new `call_stack` (pushed by `invoke`/`runEntry`); and `sx_trace_push`
|
||||
/ `sx_trace_clear` are serviced NATIVELY (the VM calls the real sx_trace.c functions —
|
||||
linked into the compiler — so the return-trace buffer the host reads is populated
|
||||
identically to the legacy dlsym path). `raise`/`catch`/`or` all run on the VM now.
|
||||
- **Signed sub-64-bit load fix (a real GENERAL bug the failable case surfaced):**
|
||||
`readField` now SIGN-extends `i8`/`i16`/`i32`/`isize` loads (was zero-extending, so a
|
||||
stored `i32 -1` reloaded as `0xFFFFFFFF` = +4.29e9 and `< 0` was false — which silently
|
||||
hid `raise error.Bad`). Affects any negative signed sub-64-bit value stored & reloaded;
|
||||
gate-ON corpus parity confirms it's a strict fix. Unit test added (+ failable tests
|
||||
pass via 1037/1038 in the corpus).
|
||||
- **Remaining fallbacks (2, both principled — the VM correctly stays on legacy):**
|
||||
`intern` (`0626`, the welded compiler-API fn — Phase 3 re-homes it) and the inline-asm
|
||||
global call (`1654`, never comptime-evaluable). Every other measured corpus const-init
|
||||
is handled on the VM.
|
||||
At this point the comptime VM handles essentially the entire real comptime corpus
|
||||
(scalars, control flow, structs/tuples/arrays/slices/strings/optionals/enums, calls +
|
||||
recursion, the implicit context + allocator protocol, globals, failables + return
|
||||
traces). Phase 2 (bytecode) and Phase 3 (compiler-API on comptime memory) are the forward
|
||||
work; flipping the VM to default + deleting the legacy path awaits those.
|
||||
- **(8) Wire the `#run` side-effect path; trace-clear-on-fallback — DONE.** The second
|
||||
comptime call site (`emit_llvm.runComptimeSideEffects`, top-level `#run <expr>;`) now
|
||||
routes through `tryEval` with legacy fallback, like the const-init fold; `tryEval` yields
|
||||
`.void_val` for a void/noreturn entry. Fixed a trace-corruption the new site exposed
|
||||
(`1035`): a side-effect that pushes trace frames then bails (on `print`) had the legacy
|
||||
re-run double-push them — both sites now `sx_trace_clear()` right before the legacy
|
||||
fallback to discard the VM's partial pushes. Parity **688/688** both gate ON and OFF. All
|
||||
comptime evaluation now routes through the VM-with-fallback (uniform).
|
||||
- **(9) `-Dcomptime-flat` build flag — DONE (the "swap behind a build flag" step).** The VM
|
||||
gate is now a build option (`build.zig` → a `build_opts` module on `mod`; `emit_llvm.init`
|
||||
reads `build_opts.comptime_flat or SX_COMPTIME_FLAT env`), default OFF. `zig build test
|
||||
-Dcomptime-flat` runs the FULL corpus on the VM (688/0) — the build-integrated parity
|
||||
gate. Verified the flag toggles the binary (flag-built `sx` uses the VM with no env var;
|
||||
default-built does not). This is the prerequisite to eventually making the VM default +
|
||||
deleting the legacy path (which still awaits Phase 2/3 + broader confidence).
|
||||
- **(10) Compiler-call path on the VM — `intern`/`text_of` native (Phase 3 SEED) — DONE.**
|
||||
`invoke` now services a welded `compiler`-library function (the `compiler_welded` flag is
|
||||
the safety boundary) via `Vm.callCompilerFn` — natively on comptime memory, NO legacy
|
||||
`Interpreter`: `intern(s: string) -> StringId` reads the string bytes from comptime memory and
|
||||
`internString`s into the (const-cast) table (pool-only, never touches type layout, so the
|
||||
VM's cached sizes stay valid); `text_of(id) -> string` materializes the pooled text back
|
||||
into comptime memory as a fat pointer. Unlocked `0626` — the ONLY remaining const-init fallback
|
||||
is now the inline-asm global (`1654`, genuinely not comptime-evaluable). Parity **688/688**
|
||||
both gate ON and OFF; unit test added. This is the mechanism Phase 3 grows: the next
|
||||
compiler functions (`find_type`, `register_struct`, the reflection readers) are added the
|
||||
same way — comptime pointer in, handle/pointer out, no marshaling.
|
||||
|
||||
**Phase 3 progress (2026-06-18):**
|
||||
- **(P3.1) First read-only reflection readers — `find_type` + `type_field_count` (DONE).**
|
||||
Two more `compiler`-library fns bound the same way as the `intern`/`text_of` seed
|
||||
(added to `compiler_lib.bound_fns` AND `Vm.callCompilerFn`, native on comptime memory, no
|
||||
marshaling). A **type handle is a plain `u32` `TypeId`** (exactly like `StringId`), so
|
||||
both calls keep the seed's clean scalar shape — handle in, scalar out:
|
||||
`find_type(name: StringId) -> TypeId` (`TypeTable.findByName`) and
|
||||
`type_field_count(t: TypeId) -> i64` (a new `TypeTable.memberCount` query — struct/union/
|
||||
tagged-union fields, enum variants, array/vector length — that BOTH the legacy handler
|
||||
and the VM call, so the two paths can't drift). Example `0628` chains
|
||||
`intern → find_type → type_field_count` and a not-found lookup, both folded at `#run`,
|
||||
both VM-HANDLED natively (no fallback). Parity **689/689** (gate ON and OFF); VM unit test
|
||||
added.
|
||||
- **Decision (resolves the plan's `find_type → ?Type` sketch):** `find_type` returns a
|
||||
NON-optional `TypeId`, using the codebase's dedicated `unresolved` (0) sentinel for
|
||||
not-found — NOT an `?Type`. Rationale: a `Type` value resolves to `.any`
|
||||
(`type_resolver.zig`), which the comptime VM does not represent; and an optional
|
||||
return can't cross the legacy↔VM eval boundary (`regToValue` bridges only
|
||||
word/string/struct/tuple). `unresolved` is the project-blessed unmistakable "no type"
|
||||
marker (see CLAUDE.md REJECTED PATTERNS — a dedicated sentinel is the required shape),
|
||||
so the caller checks the handle against 0. This keeps the reader a clean scalar mirror
|
||||
of `intern`/`text_of` and defers `.any`/optional plumbing to when it's actually needed.
|
||||
- **(P3.2) Field-level reflection readers — `type_nominal_name` + `type_field_name` +
|
||||
`type_field_type` (DONE).** Three more readers on the same `TypeId`-handle shape (each
|
||||
backed by a new `TypeTable` query that BOTH the legacy handler and the VM call, so no
|
||||
drift): `type_nominal_name(t: TypeId) -> StringId` (`nominalName` — a named type's own
|
||||
name; loud-bail for unnamed types), `type_field_name(t: TypeId, idx: i64) -> StringId`
|
||||
(`memberName` — struct/union/tagged-union field, enum variant, named-tuple element), and
|
||||
`type_field_type(t: TypeId, idx: i64) -> TypeId` (`memberType` — struct/tuple/array/vector
|
||||
member type). All loud-bail on out-of-range idx / no-member (no silent default). These are
|
||||
the first MULTI-ARG compiler fns (the VM's `callCompilerFn` now reads arg 1 = idx); added
|
||||
`Vm.argHandle`/`argTypeId` helpers (range-checked u32/TypeId arg reads). Naming uses the
|
||||
`type_*` family so nothing collides with the std metatype builtins (`field_name`/`type_name`
|
||||
exist in `core.sx`). Example `0629` reflects `Pair { lo: Point; hi: Point }` — reads each
|
||||
field name and the nominal name of a field's type, all folded at `#run`, all VM-HANDLED
|
||||
natively. Parity **690/690** (gate ON and OFF); VM unit test added.
|
||||
- **(P3.2b) Kind + enum-value readers — `type_kind` + `type_field_value` (DONE).** The last
|
||||
two read-only readers the metatype's `type_info(T)` needs, completing the READ side: a
|
||||
comptime sx fn can now fully reflect a struct/enum/tagged-union/tuple into data with no
|
||||
`#builtin`. `type_kind(t: TypeId) -> i64` (`TypeTable.kindCode` — a stable, compiler-owned
|
||||
discriminant: 0 other · 1 struct · 2 enum · 3 tagged_union · 4 tuple · 5 union · 6 array ·
|
||||
7 vector · 8 error_set; TOTAL — never bails, an unnamed/non-aggregate type reads `other`)
|
||||
and `type_field_value(t: TypeId, idx: i64) -> i64` (`TypeTable.memberValue` — an enum
|
||||
variant's explicit value or ordinal; mirrors the `field_value_int` builtin; loud-bail for
|
||||
a non-enum / out-of-range idx). Example `0630` reflects `Color`/`WindowFlags`(flags)/`Point`.
|
||||
Parity **691/691** (gate ON and OFF); VM unit test added.
|
||||
- **READ side now complete:** `find_type` + `type_kind` + `type_field_count` +
|
||||
`type_field_name` + `type_field_type` + `type_nominal_name` + `type_field_value` cover
|
||||
everything `reflectTypeInfo` reads.
|
||||
- **(P3.3) WRITE side — `declare_type` + `pointer_to` + ONE kind-branching `register_type` (DONE).**
|
||||
The mutating side is a SINGLE `register_type(handle, kind, members)` that branches on `kind`
|
||||
IN THE COMPILER (subsuming `define`'s `defineStruct`/`defineEnum`/`defineTuple`), plus
|
||||
`declare_type(name) -> Type` (forward handle) and `pointer_to(t) -> Type` (build `*T`
|
||||
references). They take/return real `Type` values (matching meta.sx's declare/define).
|
||||
- **Timing decision (per the user):** mint LAZILY at LOWERING time (single pass, NOT a
|
||||
pre-emit phase, NOT two-pass) — the existing `runComptimeTypeFunc` path. So the write
|
||||
side is **legacy-only** (`compiler_lib` handlers); the VM isn't wired at lowering time, so
|
||||
no VM mirror is needed (the read-side readers stay dual-path for emit-time reflection). A
|
||||
non-generic `-> Type` builder is now flagged `is_comptime` (`decl.zig`) so its dead body
|
||||
permits the welded calls (the comptime-only gate).
|
||||
- **Graph support:** forward `declare_type` handles + `pointer_to` express a
|
||||
mutually-recursive A↔B graph (`*A`, `*B`, B-by-value) before bodies are filled.
|
||||
`register_type` is **idempotent** — re-filling a nominal slot (same module reached via two
|
||||
import edges) re-mints identically instead of erroring (`nominalIdent` reads identity from
|
||||
any nominal kind). `kind` codes match `type_kind`: 1 struct · 2 enum (actual `.@"enum"`) ·
|
||||
3 tagged_union · 4 tuple.
|
||||
- **Two bugs fixed en route** (issue 0142): (a) a fully payloadless comptime-minted enum
|
||||
was minted as an all-void `tagged_union` → `verifySizes` panic; now mints a real
|
||||
`.@"enum"` (both `register_type` kind 2 AND the metatype `defineEnum`). (b) bare
|
||||
`EnumType.variant` qualified construction of a payloadless variant wasn't supported (failed
|
||||
for hand-written enums too) — added in `lowerFieldAccess` (`isPayloadlessVariant`).
|
||||
- Examples: `0631` (graph + actual-enum + reflection), `0632` (make_enum all-void),
|
||||
`0633`/`0634`/`0635` (namespaced / bare / multi-edge import of a minted type), `0187`
|
||||
(qualified variant construction). Parity 697/697 (gate ON and OFF); unit tests added.
|
||||
- **Next (P3.4):** re-express `declare`/`define`/`type_info` as sx over the read+write
|
||||
compiler-API and DELETE the bespoke interp arms — needs the VM hardened against malformed
|
||||
lowering-time IR first (the metatype runs at lowering time), so either harden + wire the VM
|
||||
there, or migrate the metatype onto the legacy compiler-API calls first. Decide when reached.
|
||||
Phase 2 (bytecode) is the orthogonal speed work.
|
||||
|
||||
### Phase 3 — Compiler-API on comptime memory (resume the stream — no weld)
|
||||
With native-byte comptime values, re-home the compiler-API:
|
||||
|
||||
- **Expose the compiler's real types.** Register the actual `types.zig` records
|
||||
(`StructInfo`, `EnumInfo`, `Field`, …) into the comptime type table under sx-visible
|
||||
names, with their **real (host) layout** — the type IS the compiler's, so there is
|
||||
nothing to validate or keep in sync. (This is the projection that *replaces* the
|
||||
weld's reflection — owned by the compiler, not declared in sx.)
|
||||
- **Expose the compiler's functions.** `register_struct`, `find_type`, `intern`,
|
||||
`text_of`, and the reflection readers operate on comptime pointers / handles
|
||||
directly (no marshaling — the bytes already ARE the record).
|
||||
- **Re-express** `declare` / `define` / `type_info` as sx over these; delete the
|
||||
bespoke interp arms (`defineStruct` / `defineEnum` / `defineTuple` / `reflectTypeInfo`);
|
||||
migrate `examples/0622` (struct), `0619`/`0620`/`0623` (enum/tuple).
|
||||
- **Migrate `BuildOptions`** off `#compiler` onto this mechanism; **delete `#compiler`**.
|
||||
|
||||
**Verification:** the metatype + `#compiler` surfaces are gone, re-expressed as sx over
|
||||
the exposed compiler-API; full corpus green.
|
||||
|
||||
### Phase 4 — Retire the legacy interp (the ONE-evaluator end state)
|
||||
|
||||
The metatype CONSTRUCTION + REFLECTION surface is VM-native (steps 7/8 — `0614`–`0624`,
|
||||
`0632` all HANDLED). This phase moves EVERYTHING ELSE off `interp.zig` and deletes it.
|
||||
|
||||
**What the legacy interp is still used for (audited 2026-06-18) — five roles:**
|
||||
|
||||
| Role | Wired to VM? | Site |
|
||||
|------|--------------|------|
|
||||
| **A. Comptime folds** (type-fn / `::` const-init / `#run`) | ✅ VM + legacy fallback | `comptime.zig:530`, `emit_llvm.zig:871`/`971` |
|
||||
| **B. `#insert` string eval** | ❌ legacy-only (VM wiring reverted — 0737 malformed-IR crash) | `comptime.zig:634` |
|
||||
| **C. Post-link bundler** (`platform.bundle` — Info.plist/codesign/process/fs) | ❌ legacy-only | `core.zig:invokeByFuncId` ← `main.zig:769` |
|
||||
| **D. `#compiler` hooks** (`compiler_call` — BuildOptions/bundling) | ❌ legacy-only; `Value`-based ABI | `compiler_hooks.zig`, `interp.zig:1130` |
|
||||
| **E. Bail diagnostics** (`Interpreter.last_bail_*` statics) | n/a | `main.zig:464` |
|
||||
|
||||
Shared substrate everything traffics in: the **`Value`** tagged union (the
|
||||
`regToValue`/`valueToReg` bridge + the hooks + `core.zig`) and the **host-FFI bridge**
|
||||
(`host_ffi.zig` + `interp.callExtern` — dlsym + cdecl trampolines for real libc).
|
||||
|
||||
**DECISION (2026-06-18, user): UNIFY.** The VM gains a host-FFI escape + real-pointer
|
||||
translation and runs BOTH sandboxed comptime folds AND the unsandboxed post-link bundler.
|
||||
`interp.zig` is fully deleted — true ONE evaluator, two modes (sandboxed / host-effects).
|
||||
|
||||
**Remaining comptime-fold gaps** (full corpus fallback inventory — 15 examples; 1179/1180
|
||||
are legitimate negative-test bails that BECOME VM diagnostics, 1145 is a scan artifact):
|
||||
`box_any`/`unbox_any` (6), `out`/print (2), `global_addr` (1), trace frames (1),
|
||||
`compiler_call` (2 — role D).
|
||||
|
||||
**Sub-phases (dependency order; each its own session, both gates 697/0 after each):**
|
||||
|
||||
- **4A — finish comptime ops (small, parity-guarded).** Drive the fold fallback list to
|
||||
empty except `compiler_call`:
|
||||
- **4A.1** `box_any`/`unbox_any`. Word case = alloc 16B `{tag@0, value@8}`, tag =
|
||||
`source_type.index()` (matches legacy comptime; note runtime `anyTag` normalizes
|
||||
arbitrary-width ints), value via `writeField(source_type)` (so f32 etc. round-trip);
|
||||
unbox = `readField(addr+8, target)`. Aggregate-Any payload needs the runtime
|
||||
pointer-in-value-slot shape (`coerceToI64` alloca+ptrtoint) — implement or bail loudly.
|
||||
- **4A.2** `out`/print → add a VM output buffer; flush through the same path as
|
||||
`core.flushInterpOutput`.
|
||||
- **4A.3** `global_addr` (address-of a global in comptime memory).
|
||||
- **4A.4** trace frames (`sx_trace_*` / `interp_print_frames`).
|
||||
- **4B — VM-native diagnostics (role E). MUST land before deleting legacy.** Today a VM
|
||||
bail silently falls back; with legacy gone the VM bail IS the user-facing build-gating
|
||||
diagnostic. Surface the VM's `detail`/span/file into what `main.zig` renders; turn
|
||||
1179/1180-style bails into proper diagnostics. No diagnostic may regress.
|
||||
- **4C — `#insert` on the VM (role B).** Re-wire `evalComptimeString` through `tryEval`;
|
||||
the lowering-time-IR hardening that forced the 0737 revert is already in place. Verify
|
||||
the `#insert` corpus parity.
|
||||
- **4D — host FFI on the VM (role D substrate). DONE.** Solved by a better allocator, not a
|
||||
pin/tag scheme: the comptime memory is now an **arena** of stable host allocations and `Addr`
|
||||
IS a real host pointer (`4D.0`, `625ba0f`), so a comptime pointer and an FFI-returned host
|
||||
pointer are the same value — no translation, no realloc hazard. `Vm.callHostExtern`
|
||||
(`4D.1`, `e7a8708`) dispatches ANY extern via `host_ffi` dlsym + trampolines (args/returns pass
|
||||
untouched); `4D.2` (`6a7f690`) adds slice/string args (→ NUL-term `char*`) + float guards.
|
||||
Examples 0636/0637. **(Superseded sub-note:** the earlier "pin the buffer / comptime↔host translate"
|
||||
hazard is moot — the arena never moves an allocation.)
|
||||
- **`#compiler` / `compiler_call` — DELETED, replaced by the `abi(.compiler)` ABI (decision 2026-06-18,
|
||||
REVISED from the earlier `abi(.zig) extern compiler` shape).** A function is *compiler-domain* — it runs in
|
||||
the comptime evaluator (VM/interp), NEVER in the shipped binary — because its **ABI says so**: `abi(.compiler)`.
|
||||
No `extern <lib>`, no fake `#library "compiler"`. One annotation covers BOTH roles: (a) the **compiler-API
|
||||
surface** (`intern`/`find_type`/`build_options`/`set_post_link_callback`/… — bodiless decls whose Zig/VM
|
||||
handler is the impl, on `compiler_lib`'s export list, dispatched by `Vm.callCompilerFn`); (b) **user
|
||||
compiler-domain functions** like post-link callbacks (`bundle_main` — BODIED `abi(.compiler)`, lowered for VM
|
||||
eval but emit-skipped). The `#compiler` struct attribute + the `compiler_call` IR op + the `Value`-based hook
|
||||
`Registry` (`compiler_hooks.zig`) all **go away**. **Why this is cleaner than the welded-fn approach:** the
|
||||
former runtime-call enforcement blocker (a `build_options()` call inside an LLVM-emitted callback body) is
|
||||
MOOT — a compiler-domain function is never emitted, so its compiler-API calls never reach `emitCall`.
|
||||
**Staged build (each its own step, both gates green):**
|
||||
- **S1+S2 — DONE (2026-06-18):** introduced `abi(.compiler)`, REMOVED the `.zig` ABI + `abi(.zig) extern
|
||||
compiler` + `#library "compiler"` (clean cutover, no legacy); migrated all compiler-API examples. The
|
||||
binding now keys off `fd.abi == .compiler` (`decl.zig` `weldedCompilerFn`); a bodiless `abi(.compiler)`
|
||||
decl lowers extern-like (declared-not-defined) with no implicit ctx. **700/0 both gates.**
|
||||
- **S3 — DONE (2026-06-18):** emit_llvm skips BODIED `abi(.compiler)` function bodies. Added an
|
||||
`is_compiler_domain` flag to the IR `Function`; a bodied `abi(.compiler)` function LOWERS its body (for VM
|
||||
eval) + is flagged `is_comptime` but is NOT emitted (Pass 2 skip; declared external-linkage so the empty
|
||||
decl verifies). KEY fix: a call to a comptime-only callee (compiler-API `compiler_welded` OR
|
||||
`is_compiler_domain`) inside a dead comptime body now emits `undef` instead of a real `call` (`ops.zig`
|
||||
`emitCall`) — the old `compiler_call` did this; without it an AOT link leaves an undefined `_double`/`_intern`
|
||||
reference (this also fixed a pre-existing untested AOT breakage of the bodiless compiler-API examples).
|
||||
`fnIsBodilessCompiler` distinguishes the API surface (declare-only) from a compiler-domain callback (lowered,
|
||||
emit-skipped). Regression: `examples/0638-comptime-domain-fn-not-emitted` (`double` folds a `#run` const,
|
||||
absent from the binary, JIT+AOT). **701/0 both gates.**
|
||||
- **S4 — callback-param propagation: OPTIONAL / DEFERRED (ergonomics only).** Verified 2026-06-18: an
|
||||
`abi(.compiler)` function is TYPE-compatible with a plain `() -> R` param (the ABI marks the *function* —
|
||||
`is_compiler_domain` — not its *type*, which stays `() -> R` CC-default). So a callback that needs to be
|
||||
compiler-domain just declares itself `abi(.compiler)` (S3) and passes to a plain param fine; auto-propagation
|
||||
from an `abi(.compiler)` PARAM type is a nicety, not a prerequisite for S5. Skipped for now.
|
||||
- **S5a — DONE (2026-06-18):** the corpus-covered slice. `build_options` + `set_post_link_callback` →
|
||||
free `abi(.compiler)` functions (VM `callCompilerFn` arms + legacy `compiler_lib` handlers); **`BuildConfig`
|
||||
threaded into the VM** via a `tryEval` param (the same one `main.zig` forwards — shared with 4E). `build.sx`
|
||||
extracts `set_post_link_callback` from the `struct #compiler` as a free `ufcs` fn; `bundle_main` + the
|
||||
platform registrars (`configure`) are `abi(.compiler)`. 37 examples' `.ir` snapshots regen'd (benign:
|
||||
declaration renumber + `@str` suffix shift — every example imports build.sx via the prelude). Strict
|
||||
`compiler_call` bails 6→2; 0602/0603/1604/1611 HANDLED. **701/0 both gates.**
|
||||
- **S5b/S5c (port the ~37 hooks) — SUPERSEDED 2026-06-18 by the sx-driven build pipeline (below).**
|
||||
Porting each `BuildOptions` accessor to an `abi(.compiler)` function that delegates to a `compiler_hooks`
|
||||
hook just re-encodes sx-level logic (string setters/getters, `is_macos` triple-matching, list appends) as
|
||||
compiler hooks. The hooks need NOTHING from the compiler except the `BuildConfig` state. So instead of 37
|
||||
hooks, **drive the whole build pipeline from sx** (the logical end of "bundling lives in sx"). S5a stays as
|
||||
a green intermediate; the sx-build-pipeline replaces `build_options`/`set_post_link_callback`/the whole
|
||||
`#compiler` surface wholesale.
|
||||
|
||||
### Phase 5 — sx-driven build pipeline (replaces the BuildOptions hooks; decision 2026-06-18, user)
|
||||
|
||||
**The build pipeline becomes an sx program.** `BuildConfig` is plain sx data (an ordinary struct, sx-owned
|
||||
end-to-end — no `#compiler`, no hooks, no shared Zig state, no weld/offset access). The compiler shrinks to
|
||||
a few `abi(.compiler)` PRIMITIVES that take **explicit args** (so nothing is shared by memory), and an sx
|
||||
`build()` driver orchestrates configure → emit → link → bundle. **Chosen boundary: Option B** — the compiler
|
||||
keeps the proven Zig linker as a primitive; sx owns config + orchestration + bundle. (Option A — sx shells
|
||||
`cc`/`ld` itself — is a later refinement once the per-target link-line logic is ported to sx.)
|
||||
|
||||
**File split (user decision 2026-06-19):** the low-level compiler-API PRIMITIVES live in
|
||||
`library/modules/compiler.sx` (the comptime `compiler` library — renamed from the interim `std/build.sx`); the
|
||||
default `build` IMPLEMENTATION (`default_build` + the `on_build` slot + the sx `BuildConfig`) lives in
|
||||
`library/modules/build.sx` alongside the existing `BuildOptions` DSL. So `compiler.sx` = primitives, `build.sx` =
|
||||
orchestration/default impl. **Build-callback fallibility was DROPPED (user 2026-06-19):** the primitives + the
|
||||
build callback are NOT `-> !` — a failed action (e.g. `link`) BAILS on the VM (hard build error). So the shapes
|
||||
below shed their `-> !`.
|
||||
|
||||
Shape (build-callback fallibility dropped 2026-06-19):
|
||||
```sx
|
||||
// library/modules/compiler.sx (the comptime `compiler` library — PRIMITIVES)
|
||||
emit_object :: () -> string abi(.compiler); // emitted .o path (query)
|
||||
link :: (objects: List(string), output: string, libraries: List(string),
|
||||
frameworks: List(string), flags: List(string), target: string) abi(.compiler); // void; bails on failure
|
||||
c_object_paths :: () -> List(string) abi(.compiler); // metadata queries
|
||||
link_libraries :: () -> List(string) abi(.compiler);
|
||||
|
||||
// library/modules/build.sx (the build DSL — DEFAULT IMPLEMENTATION + slot)
|
||||
BuildConfig :: struct { output: string; target: string; flags: List(string);
|
||||
frameworks: List(string); bundle_path: string; bundle_id: string; ... }
|
||||
default_build :: (config: BuildConfig) abi(.compiler) { // the default pipeline (void)
|
||||
obj := emit_object(); objs := c_object_paths(); objs.append(obj);
|
||||
link(objs, config.output, link_libraries(), config.frameworks, config.flags, config.target);
|
||||
if config.bundle_path.len > 0 { bundle_app(config); } } // bundle_app = today's sx bundler
|
||||
on_build : (BuildConfig) abi(.compiler) = default_build; // the override slot
|
||||
// user overrides: build :: (config: BuildConfig) abi(.compiler) { ... } #run on_build = build;
|
||||
```
|
||||
The compiler's whole post-IR role: codegen → build the CLI-derived `BuildConfig` → read `on_build` → invoke
|
||||
`on_build(config)` on the VM; a `raise` fails the build. Plain `sx run` fires none of it.
|
||||
|
||||
**Steps (each its own green step; depends on 4E first):**
|
||||
- **P5.1 — 4E prereq — DONE (2026-06-19).** `core.invokeByFuncId` routes the post-link callback through the
|
||||
**VM** (`comptime_vm.tryEval`), NO fallback (a side-effecting callback can't double-execute): a bail is a hard
|
||||
build error (`comptime_vm.last_bail_reason` surfaced by `main.printInterpBailDiag`). `BuildConfig` +
|
||||
`import_sources` threaded in; `flushInterpOutput` deleted (VM `out` writes direct via host-FFI). Smoke test
|
||||
`examples/1661-platform-post-link-vm-list` (AOT): a post-link callback GROWS a `List` (0141 — works on the VM,
|
||||
bails on legacy with `struct_get`), so the build succeeds (exit 0) only via the VM. Non-empty callback `args`
|
||||
rejected loudly (the `on_build(config)` arg-marshaling entry is P5.3). **702/0 both gates.**
|
||||
- **P5.2 — primitives.** Split: the read-only **metadata queries are DONE (2026-06-19)** — `c_object_paths() ->
|
||||
List(string)` + `link_libraries() -> List(string)` as `abi(.compiler)` fns (stdlib `library/modules/compiler.sx`),
|
||||
serviced by `comptime_vm.callCompilerFn` over `BuildConfig` fields `main.zig` forwards; new VM `makeStringList`
|
||||
builds the `List(string)` in comptime memory from the call's result type (`ins.ty` now threaded through
|
||||
`invoke`/`callCompilerFn`). Smoke test `1662-platform-build-pipeline-queries` (AOT + C companion). 703/0 both
|
||||
gates. **`emit_object() -> string` is also DONE (2026-06-19)** as a QUERY (not an action): the Zig driver emits
|
||||
the object eagerly, so the primitive just returns the path from `BuildConfig.object_path` (no vtable). So all
|
||||
three QUERY primitives are done. **P5.2b — `link(...)` (the one genuine ACTION) — DONE (2026-06-19).** USER
|
||||
DECISION: the build callback is NOT fallible, so `link` is plain VOID (no `-> !`) and a failure BAILS (hard
|
||||
build error) — no failable-tuple construction. It dispatches through a host-installed `compiler_hooks.BuildHooks`
|
||||
vtable (`comptime_vm.zig` can't depend on the driver); `main.LinkHooksCtx.link` adapts to `target.link`. New VM
|
||||
readers `readStringList`/`readStringArg` (inverse of `makeStringList`). Smoke test
|
||||
`1663-platform-build-pipeline-link` (AOT): a post-link callback re-links the build's objects to a temp output —
|
||||
the relinked binary RUNS; negative-probe verified. The Zig driver still auto-links (removed in P5.4). 704/0.
|
||||
- **P5.3 — `on_build` registrar — DONE (2026-06-19).** `on_build(cb)` registers the build callback
|
||||
(`cb: (opt: BuildOptions) -> bool abi(.compiler)`); the compiler force-lowers + auto-invokes the well-known
|
||||
`default_pipeline` when no override. (Implemented as a registrar, not an assignable slot — the opaque
|
||||
`BuildOptions` handle is one word, so arg-passing needs no struct marshaling.)
|
||||
- **P5.4 core — DONE (2026-06-19).** `default_pipeline` in `build.sx` drives the whole build; NO Zig
|
||||
auto-emit/auto-link; `emit_object`/`link` are sx-called actions via the `BuildHooks` vtable;
|
||||
`set_post_link_callback` deleted (all callers on `on_build`). Build-path auto-imports `modules/build.sx`.
|
||||
703/0 both gates.
|
||||
|
||||
### THE FINAL DIRECTION (user, 2026-06-19): FULL MIGRATION — NO LEGACY LEFT.
|
||||
|
||||
**Decision: DROP gate-OFF entirely.** The VM becomes the SOLE comptime evaluator; `-Dcomptime-flat` is made
|
||||
permanent then removed; `interp.zig` (the legacy tagged-`Value` `Interpreter`) is DELETED. There is no
|
||||
dual-path, no legacy `compiler_lib` handler, no `regToValue`/`valueToReg` bridge, no VM→legacy fallback. We
|
||||
migrate the BuildOptions surface DIRECTLY to VM-native `abi(.compiler)` arms (no legacy handler — there is no
|
||||
legacy to handle). **All bundling + code signing for EVERY target lives in the sx `default_pipeline`.**
|
||||
|
||||
- **P5.5 — DONE (2026-06-19).** The 35 `BuildOptions :: struct #compiler` methods migrated to VM-native
|
||||
`abi(.compiler)`: `BuildOptions :: struct { }` (opaque null-sentinel handle) + 35 free
|
||||
`ufcs (self: BuildOptions, …) abi(.compiler)` decls in `build.sx`, serviced by a new
|
||||
`comptime_vm.callBuildOptionFn` arm off `callCompilerFn` — **NO legacy `compiler_lib` handler** (names
|
||||
registered in `bound_fns` with a single bailing stub only so `weldedCompilerFn` accepts them). Setters dupe the
|
||||
arg string into the PERSISTENT `Vm.gpa` (the Compilation allocator — threaded into both `tryEval` and
|
||||
`runBuildCallback` — NOT the per-eval VM arena) and write/append to the threaded `BuildConfig`; string getters
|
||||
return the field (or `""`); bool getters compute from the triple (`predIsMacOS`/…); count/index getters read the
|
||||
`BuildConfig` slices. **Dispatch routing (Option B):** a `#run`/const-init entry that directly calls a
|
||||
compiler-domain/welded fn (`emit_llvm.entryNeedsVm`) runs on the VM with NO legacy fallback regardless of the
|
||||
`-Dcomptime-flat` gate → gate-OFF stays green without a legacy BuildOptions handler. 5 `platform/bundle.sx`
|
||||
getter-calling helpers marked `abi(.compiler)` (comptime-only bundler code). 37 `.ir` regenerated (string-pool
|
||||
churn; behavior-identical, verified `.ir`-only). **703/0 BOTH gates.** BuildOptions `compiler_call` bails GONE
|
||||
(1609/1614/1615 strict-clean); 1616 now bails on `shr` — a SEPARATE unported bitwise/shift VM gap
|
||||
(`shl`/`shr`/`bit_and`/`bit_or`/`bit_xor`/`bit_not`), to port FIRST in P5.6 (1616 is unpinned + can't JIT-run on
|
||||
macOS regardless). Also swept the outdated "flat memory" terminology → "comptime/byte-addressable" (the VM is
|
||||
arena-backed, `Addr` = real host pointer; flag names `-Dcomptime-flat`/`SX_COMPTIME_FLAT` kept).
|
||||
- **P5.6 — ALL bundling + code signing in `default_pipeline` (every target).** `default_pipeline` (or a
|
||||
`bundle()` it calls, in `platform/bundle.sx`) performs, after `link`, the full per-target bundle when
|
||||
`bundle_path()` is set — branching on `is_macos`/`is_ios_device`/`is_ios_simulator`/`is_android`:
|
||||
- **macOS `.app`** — `Contents/{MacOS,Resources,Frameworks}`, `Info.plist`, embed `-framework` dylibs +
|
||||
`install_name_tool` fixups, `codesign` (ad-hoc or with `codesign_identity`).
|
||||
- **iOS device `.app`** — device slice, embedded `.mobileprovision` (`provisioning_profile`), entitlements,
|
||||
`codesign` with the real identity; **iOS simulator `.app`** — sim slice, no provisioning, ad-hoc sign.
|
||||
- **Android `.apk`** — `AndroidManifest.xml` (or `manifest_path` override), asset tree (`add_asset_dir`),
|
||||
`#jni_main` Java → `javac` → `d8` → `classes.dex`, `aapt2` package, `zipalign`, `apksigner` with the
|
||||
debug/`keystore_path` keystore.
|
||||
All of it runs on the VM via the migrated `abi(.compiler)` getters + `fs`/`process` host-FFI (the existing
|
||||
`platform/bundle.sx` logic, now reading the VM-native accessors instead of `#compiler` hooks). The compiler
|
||||
keeps ONLY the linker as a primitive (Option B). Remove the `--bundle`/`post_link_module` Zig shim — bundling
|
||||
is `default_pipeline`'s job; CLI flags feed `BuildConfig` and `default_pipeline` branches on it.
|
||||
- **P5.7 — DELETE all legacy.** Remove the `#compiler` attribute (parse + lower), the `compiler_call` IR op
|
||||
(`inst.zig` + every switch arm + the `interp.zig:1130` dispatch), `compiler_hooks.zig`
|
||||
(`HookFn`/`Registry`/all hooks). Make `-Dcomptime-flat` permanent (VM always) and **delete `interp.zig`**
|
||||
(`Interpreter`/`Value`/`defineEnum`…/`reflectTypeInfo`/`callExtern`/`last_bail_*`); drop the
|
||||
`regToValue`/`valueToReg` bridge and the VM→legacy fallback in `emit_llvm` (`#run`/const-init) and
|
||||
`comptime.zig` (type-fn / `#insert`) — a VM bail is now ALWAYS a build-gating diagnostic (4B wiring), never a
|
||||
fallback. `core.invokeByFuncId` is already VM-only. Re-express `define`/`make_enum` as sx over the
|
||||
compiler-API. Land the 0141 repro as a corpus test. Reconcile 1654 (asm-global at comptime) to the VM wording.
|
||||
- **P5.8 — real-project validation (integration).** Build `~/projects/m3te` and `~/projects/distribution` with
|
||||
the new pipeline end-to-end (their real bundle/codesign/target configs) — these are the acceptance test that
|
||||
`default_pipeline` covers all targets. Fix gaps surfaced there. Add dedicated bundle smoke tests (min `.app` +
|
||||
`.apk`) to the corpus (the bundler still has no `zig build test` coverage — the stream's top risk).
|
||||
|
||||
**End state:** ONE evaluator (the VM); ZERO legacy; the entire build — emit, link, and all bundling + code
|
||||
signing for macOS/iOS-device/iOS-sim/Android — is sx in `default_pipeline`, overridable via `#run on_build(...)`.
|
||||
The compiler is: parse → IR → codegen → invoke `on_build`/`default_pipeline` on the VM (which calls back into
|
||||
the linker primitive). `m3te` + `distribution` build clean.
|
||||
|
||||
**Dependencies:** 4A → (4B, 4C independent) ; `abi(.compiler)` S1+S2(done) → S3 → S4 → S5 (BuildOptions) ;
|
||||
FFI(done)+`BuildConfig`-on-VM → (S5, 4E) → 4F.
|
||||
**Top risks:** (1) the bundler has no corpus guard (4E needs dedicated tests); (2) deleting
|
||||
`#compiler`/`compiler_call` + re-expressing `BuildOptions` over the compiler-API (`abi(.compiler)`) touches the
|
||||
whole build/bundle path — stage it behind real bundle builds; (3) S3's emit-skip relies on DCE dropping the
|
||||
unreferenced compiler-domain declaration — verify no stray runtime reference keeps it alive (link error).
|
||||
|
||||
## Open questions (resolve as reached, record decisions here)
|
||||
|
||||
- **Host-ABI vs target-ABI split.** The compiler runs on the host, so its OWN exposed
|
||||
records are host-laid-out; user comptime types are target-laid-out. The comptime
|
||||
model must carry both regimes (a per-type ABI tag on layout queries). Confirm the
|
||||
boundary where a comptime pointer to a compiler record is handed to host Zig code
|
||||
uses host layout.
|
||||
- **Exposing compiler types to sx.** Mechanism for projecting `types.zig` records into
|
||||
the comptime type table with real offsets (the non-weld replacement) — a registry the
|
||||
compiler owns, keyed by sx-visible name → real Zig type's layout + a host-call ABI.
|
||||
- **Bytecode shape.** IR-derived vs a fresh ISA; register vs stack; fragment caching.
|
||||
- **Pointer escape / lifetime.** Flat-memory pointers stored into the persistent type
|
||||
table must be copied into compiler-owned memory at the boundary (as today).
|
||||
- **Old-path retirement.** Keep the tagged interpreter until Phase 1 parity, then
|
||||
delete — confirm no non-comptime consumer depends on `Value`.
|
||||
|
||||
## File map (current → touched)
|
||||
|
||||
| Area | File | Phase |
|
||||
|------|------|-------|
|
||||
| Comptime evaluator | `src/ir/interp.zig` | 0 (strip weld dispatch), 1–2 (rebuild) |
|
||||
| Weld registry | `src/ir/compiler_lib.zig` | 0 (strip), 3 (replace with type/fn exposure) |
|
||||
| Weld validation | `src/ir/lower/nominal.zig` | 0 (strip `validateWeldedStruct`) |
|
||||
| Comptime-only gate | `src/backend/llvm/ops.zig` | 0 (re-derive without weld signal) |
|
||||
| Host-FFI marshalling | `src/ir/host_ffi.zig` | 1 (struct-by-pointer trims it) |
|
||||
| Metatype arms | `src/ir/interp.zig` (`defineStruct`/…/`reflectTypeInfo`) | 3 (delete, re-express in sx) |
|
||||
| `#compiler` / BuildOptions | `library/modules/build.sx`, `src/ir/compiler_hooks.zig` | 3 (migrate, delete `#compiler`) |
|
||||
| Surface syntax | `src/parser.zig`, `src/ast.zig` (`abi`/`extern`/`#library`) | kept; revisited Phase 3 |
|
||||
|
||||
## Status
|
||||
|
||||
- **Phase 0 — DONE (2026-06-17).** The struct-weld machinery is stripped:
|
||||
`compiler_lib.zig` lost the type registry (`weldStruct`/`bound_types`/`BoundType`/
|
||||
`FieldLayout`/`findType`/`SxField`/`LayoutMismatch`/`validateStructLayout`);
|
||||
`nominal.zig` lost `validateWeldedStruct`/`weldedFieldOrderStr` + the
|
||||
`sd.abi == .zig` call; the struct-weld unit tests + examples `0625`/`0627`/`1183`/
|
||||
`1186` are removed. **Decision (recorded):** the `intern`/`text_of` function
|
||||
host-call bridge is KEPT — it is a clean scalar dispatch (string→handle), not
|
||||
weld/serialize/marshal, and is the seed Phase 3 grows the compiler-call path from.
|
||||
So the `compiler_welded` dispatch (`interp.callExtern` is unchanged at HEAD — the
|
||||
pre-branch in `call()`), `weldedCompilerFn` (decl.zig), the `emitCall` comptime-only
|
||||
gate (ops.zig), and examples `0626`/`1184`/`1185` stay. The `#library`/`abi`/`extern`
|
||||
SYNTAX stays. `zig build test` green (688 corpus, 0 failed; unit tests pass).
|
||||
- **Phase 1 — in progress.**
|
||||
- **Sub-step 1 — DONE.** `src/ir/comptime_vm.zig`: the comptime `Machine`
|
||||
(linear byte memory + bump/stack allocator with `mark`/`reset` reclamation +
|
||||
scalar `readWord`/`writeWord` (1/2/4/8, little-endian) + `bytes` views; addr 0
|
||||
reserved as `null_addr`) and `Frame` (register file indexed by Ref + stack
|
||||
reclamation on `deinit`). A register `Reg` is a raw u64 — immediate scalar OR
|
||||
`Addr`. Standalone + unit-tested (`comptime_vm.test.zig`, in the barrel); does
|
||||
NOT touch the live interpreter, so the corpus stays green (688). No op execution
|
||||
yet.
|
||||
- **Sub-step 2 — DONE.** The executor (`Vm` in `comptime_vm.zig`): walks the SAME
|
||||
IR `Inst` over comptime frames, mirroring the legacy interp's scalar semantics
|
||||
(i64 wrapping/signed + f64 register words, keyed off the result/operand `TypeId`).
|
||||
Ported: constants (`const_int`/`float`/`bool`/`null`/`undef`), arithmetic
|
||||
(`add`/`sub`/`mul`/`div`/`mod`/`neg`), comparison (`cmp_*`), logical
|
||||
(`bool_and`/`or`/`not`), conversions (`widen`/`narrow`/`bitcast` passthrough,
|
||||
`int_to_float`/`float_to_int`), terminators (`br`/`cond_br`/`ret`/`ret_void`) and
|
||||
`block_param` (branch args passed as Refs — the same frame persists, SSA-safe).
|
||||
Any other op bails loudly (`error.Unsupported` + `detail = @tagName(op)`).
|
||||
Unit-tested on hand-built IR (`Fb` builder): integer add, f64 arithmetic, cond_br
|
||||
branch selection, a block-param loop summing i..1, div-by-zero + unsupported-op
|
||||
bails. Corpus untouched (688 green) — the executor is exercised by unit tests only,
|
||||
not yet wired to real comptime eval.
|
||||
- **Sub-step 3 — DONE.** Memory + structs on comptime memory. `Vm` gained an optional
|
||||
`table: *const TypeTable` (target-aware layout). Ported `alloca`/`load`/`store`
|
||||
(over comptime addresses, `Store.val_ty` drives width) and `struct_init`/`struct_get`/
|
||||
`struct_gep` (structs laid out at the table's natural offsets). The value model: a
|
||||
`Kind.word` (scalar/pointer ≤8B) sits in a register; a `Kind.aggregate` (struct)
|
||||
lives in comptime memory and its "value" IS its address (read returns the address,
|
||||
write memcpys), so nested structs compose and `struct_gep` is just base+offset (no
|
||||
field-pointer dance). `kindOf` bails loudly on the not-yet-ported types
|
||||
(slice/string/any/optional/enum/array/tuple/…). The Addr-based value model survives
|
||||
allocator realloc (offsets are stable; slices are only materialized transiently).
|
||||
Unit-tested: struct_init+get round-trip, alloca+gep+store+load, nested-struct
|
||||
aggregate copy + nested read. Corpus untouched (688 green).
|
||||
- **Sub-step 4a — DONE.** Tuples + arrays. `kindOf` widened (`tuple`/`array` →
|
||||
aggregate). Ported `tuple_init`/`tuple_get` (positional, `tupleFieldOffset`),
|
||||
`index_get`/`index_gep` (`elemAddr` = base + idx*elem_size over array/pointer/
|
||||
many_pointer bases; slice/string bases bail), and `length` on an array value
|
||||
(static `ArrayInfo.length`). Unit-tested: mixed tuple round-trip, `[3]i64`
|
||||
gep/store + index_get sum (42), array `length` (3). 688 corpus green.
|
||||
- **Sub-step 4b — DONE.** Slices + strings as `{ptr@0 (pointer_size), len@8 (i64)}`
|
||||
fat pointers (`kindOf`: string/slice → aggregate). Ported `const_string` (materializes
|
||||
text+NUL in comptime memory + a fat pointer), `length`/`data_ptr` (read len/ptr fields),
|
||||
`array_to_slice`, `subslice`, indexing *through* a slice/string (`elemAddr` loads
|
||||
`.ptr` first), and `str_eq`/`str_ne` (len+memcmp). Helpers `makeSlice`/`sliceLen`/
|
||||
`sliceData`. Unit-tested: string length + str_eq/ne, array→slice + slice index +
|
||||
slice length (23), array subslice (43). 688 corpus green.
|
||||
- **Sub-step 4c — DONE (optionals + payloadless enums).** `kindOf`: `enum` → word;
|
||||
`?T` → word if pointer-child (null==0) else `{T@0, i1@sizeof(T)}` aggregate. Ported
|
||||
`optional_wrap`/`unwrap`/`has_value`/`coalesce` (with `optChildIsPtr`/`optHas`
|
||||
helpers; `const_null` → `null_addr` reads as none), `enum_init` (payloadless: tag is
|
||||
the value), `enum_tag` (payloadless/word). Unit-tested: non-pointer `?i64`
|
||||
wrap/unwrap/coalesce (91), pointer `?*i64` null==0 (99), payloadless enum tag (11).
|
||||
688 corpus green.
|
||||
- **Sub-step 4d — partial (`addr_of`/`deref` DONE).** `addr_of` passes through (an
|
||||
aggregate value already IS its address; a pointer is already an address — mirrors
|
||||
the legacy); `deref` = `readField` through the pointer (`ins.ty` is the pointee).
|
||||
Unit-tested (deref a `*i64` → 77; addr_of a struct value + field read → 80).
|
||||
**Deferred to the wiring phase (intentionally, not ported blind):** tagged-union
|
||||
payload (`enum_init` w/ payload, `enum_payload` — the legacy stores *untyped* Values
|
||||
and `field_index` indexes payload sub-fields, not variants, so a byte model's
|
||||
payload type is ambiguous without a real call site), `any` boxing, closures, and the
|
||||
bitwise ops. These have subtleties best resolved against actual corpus cases — the
|
||||
VM's loud `error.Unsupported` + `detail` will name exactly what each real eval needs.
|
||||
|
||||
- **Sub-step 1.5 — direct `call` DONE.** `Vm` gained `module: *const Module`
|
||||
(resolves a callee `FuncId`) + a `depth`/`max_depth` recursion guard. `call`
|
||||
marshals arg Refs → Reg words and recursively `run`s the callee; aggregate args/
|
||||
results pass as their `Addr` over the SHARED comptime memory (no copy). **Stack-lifetime
|
||||
change:** `Frame` no longer reclaims the machine on exit (a returned aggregate's
|
||||
Addr would dangle) — a comptime eval's allocations live to `Vm.deinit`;
|
||||
`Machine.mark`/`reset` stay for explicit use. Extern/builtin callees (no blocks)
|
||||
bail loudly (1.5b). Unit-tested: direct call (`add(20,22)+100` → 142) and recursion
|
||||
(`sum(0..n)` → 15/55). 688 corpus green.
|
||||
- **Sub-step 1.5b — `Reg`↔`Value` boundary bridge DONE.** The builtin/`compiler_call`/
|
||||
extern handlers are all coupled to the legacy `Interpreter` (e.g. `compiler_lib`
|
||||
handlers take `*Interpreter`), so the VM can't call them directly — the wiring uses
|
||||
WHOLE-FUNCTION fallback instead (VM runs pure functions; a bail re-runs the whole
|
||||
eval in the legacy). That needs the boundary bridge: `valueToReg` (host `Value` arg →
|
||||
VM `Reg`, materializing aggregates into comptime memory) + `regToValue` (VM result →
|
||||
`Value`, deep-copied out). Covers scalars + strings + structs (other aggregate shapes
|
||||
bail loudly; added as wiring surfaces them). Transitional — deleted once the VM owns
|
||||
comptime end-to-end. Unit-tested with round-trips. 688 corpus green.
|
||||
- **Then the wiring step** (below) — now unblocked.
|
||||
|
||||
### Decision (2026-06-17): pivot from blind op-porting to CALLS + hybrid wiring
|
||||
The common leaf ops are ported (scalars, control flow, structs, tuples, arrays, slices,
|
||||
strings, optionals, payloadless enums, deref/addr_of) and unit-tested. Continuing to
|
||||
port the rarer ops (tagged-union payload, any, closures) in isolation risks subtle
|
||||
bugs and has low signal. The higher-value path:
|
||||
1. **Calls (sub-step 1.5)** — `call` (direct), then `call_builtin`/`compiler_call`. The
|
||||
shared comptime memory makes aggregate args/results pass naturally (they're Addrs). The
|
||||
one design point: **aggregate-return lifetime** — a callee's stack-reclaim would
|
||||
dangle a returned struct Addr, so for comptime (bounded) the VM should stop
|
||||
reclaiming per-frame and let the whole eval's allocations live until `Vm.deinit`
|
||||
(keep `Machine.mark/reset` for explicit use; drop it from `Frame.deinit`).
|
||||
2. **Hybrid wiring** — `-Dcomptime-flat` routes a comptime eval through the VM, falling
|
||||
back to the legacy interp on `error.Unsupported`. This makes the VM run the REAL
|
||||
corpus, proving parity incrementally and surfacing exactly which ops each real eval
|
||||
needs — far better signal than more isolated unit tests.
|
||||
207
current/PLAN-EXTERN-EXPORT.md
Normal file
207
current/PLAN-EXTERN-EXPORT.md
Normal file
@@ -0,0 +1,207 @@
|
||||
# sx `extern` / `export` + `#foreign` retirement — Plan (FFI-linkage stream)
|
||||
|
||||
**One stream, two parts.** **Part A** adds `extern`/`export` (the linkage surface);
|
||||
**Part B** migrates every `#foreign` onto it and purges `foreign` from the tree.
|
||||
They are *one* plan: Part B can't start until Part A is a behavior-equivalent
|
||||
superset of `#foreign`, and Part A isn't "done" until Part B reaches the invariant.
|
||||
|
||||
**Design rationale:** [design/inline-asm-design.md](../design/inline-asm-design.md) §II.2
|
||||
(Deviation 6) + §II.10 #4 + the syntax evaluation.
|
||||
|
||||
**Decided syntax**
|
||||
```sx
|
||||
name :: (sig) -> Ret [callconv(.x)] [extern | export] [LIB] ["csym"] [;|{…}]; // functions
|
||||
Name :: #objc_class("X") [extern | export] { … }; // aggregates (mirrors `struct #compiler`)
|
||||
g : Type extern [LIB] ["csym"]; // extern global
|
||||
```
|
||||
- `extern` = import (no body, external linkage, C ABI, no sx ctx) — `#foreign`'s role.
|
||||
- `export` = define **and** expose (body + external linkage + C ABI + no ctx) — **new**.
|
||||
- `extern`/`export` imply `callconv(.c)`; write `callconv` only to override.
|
||||
- Optional `LIB` (a `#library` alias) + `"csym"` rename mirror `#foreign LIB "csym"`,
|
||||
so `extern` is a true `#foreign` **superset** (Gate A→B): carried on
|
||||
`extern_lib`/`extern_name`. The `#library` declaration + build-flag linking
|
||||
mechanism stays a separate axis — `extern` *references* a lib, it doesn't fold
|
||||
in `#library` itself. (Revises the original "library fully separate" decision 4.)
|
||||
|
||||
> **END-STATE INVARIANT (hard requirement).** After this stream, `foreign` appears
|
||||
> **nowhere** in the live tree — not the `#foreign` surface, and **not** internal
|
||||
> identifiers. The extern AST is **not** named `foreign_expr`. Enforced by the
|
||||
> Phase 9.4 grep gate. Scope today: 643 `foreign` lines / ~57 identifiers in `src/`
|
||||
> + 28 in live docs — most of it the objc/jni **runtime-class** machinery.
|
||||
|
||||
**Naming constraint (so we can actually reach the invariant):** introduce
|
||||
`extern`-named representations only — do **not** reuse or extend
|
||||
`ForeignExpr`/`foreign_expr`/`VarDecl.is_foreign`. Carry extern/export on a new
|
||||
`FnDecl.extern_export` modifier with a `;`/`{…}` body (so there is **no** `*_expr`
|
||||
node for it) + `FnDecl.extern_lib`/`extern_name`; add `VarDecl.is_extern`/
|
||||
`extern_lib`/`extern_name`. The IR is already extern-named (`Function.is_extern`,
|
||||
`Builder.declareExtern`).
|
||||
|
||||
**Key finding (scopes Part A):** the IR + LLVM emit **already support everything** —
|
||||
`Function.linkage` (`.external/.internal/.private`), `is_extern`, `call_conv`, and a
|
||||
raw un-mangled symbol name are all emitted by `declareFunction`
|
||||
(`emit_llvm.zig:1225-1300`). Part A is a **parser + lowering** job, no codegen change.
|
||||
|
||||
## Cadence (IMPASSIBLE)
|
||||
No commit may both add a test AND make it pass (xfail-then-green, or a behavior-lock).
|
||||
`zig build && zig build test` after every step. Never regenerate snapshots while red.
|
||||
|
||||
---
|
||||
|
||||
# PART A — add `extern` / `export` (alongside `#foreign`)
|
||||
|
||||
## Phase 0 — tokens + parser plumbing
|
||||
| Step | Commit | What | Files |
|
||||
|---|---|---|---|
|
||||
| 0.0 | lock | add `kw_extern`, `kw_export` (Tag enum + `StaticStringMap`, beside `kw_callconv` at `token.zig:45,282`); unit lex test | `src/token.zig` |
|
||||
| 0.1 | lock | `parseOptionalExternExport()` (mirror `parseOptionalCallConv`, `parser.zig:3669`) + `ast.ExternExportModifier` enum + `FnDecl.extern_export` + `VarDecl.is_extern`/`extern_name` fields; **not yet consumed**; unit AST test | `src/parser.zig`, `src/ast.zig` |
|
||||
|
||||
## Phase 1 — `extern` (import; equivalent to lib-less `#foreign`)
|
||||
| Step | Commit | What | Files |
|
||||
|---|---|---|---|
|
||||
| 1.0 | xfail | accept postfix `extern` after the callconv slot (`parser.zig:1950`); `examples/12xx-ffi-extern-fn.sx` extern-binds a libc symbol — red (lowering not wired) | `src/parser.zig` |
|
||||
| 1.1 | green | lowering: `extern` ⇒ `is_extern`, `.external`, `callconv(.c)`, no ctx — route through `declareExtern` like a lib-less `#foreign` (anchors `decl.zig:1123,387,2110,2113`). Example green | `src/ir/lower/decl.zig` |
|
||||
| 1.2 | green | optional `extern "csym"` rename + extern-global form `g : T extern;` (`parser.zig:425` path) | `src/parser.zig`, `src/ir/lower/decl.zig` |
|
||||
|
||||
## Phase 2 — `export` (define + expose; the NEW capability)
|
||||
Fills the four export-gap conditions (all in `src/ir/lower/decl.zig`):
|
||||
| Gap | Anchor | Fix |
|
||||
|---|---|---|
|
||||
| (i) linkage forced `.internal` | `:2382`, `:2514` | also `.external` when `extern_export == .export` |
|
||||
| (ii) C ABI not promoted | `:2110` | also `.c` when `== .export` |
|
||||
| (iii) no symbol-name override | `emit_llvm.zig:1226` raw name | parse optional `export "csym"`; map in the name map |
|
||||
| (iv) ctx param not suppressed | `:387` `funcWantsImplicitCtx` | also suppress when `== .export` |
|
||||
|
||||
| Step | Commit | What | Files |
|
||||
|---|---|---|---|
|
||||
| 2.0 | xfail | multi-file test: an `export fn` called from a companion `.c` caller (same `XXXX-` prefix) — red (still internal) | `examples/12xx-ffi-export-fn.{sx,c}` + `expected/` |
|
||||
| 2.1 | green | gaps (i),(ii),(iv): `export` ⇒ external + C-ABI + no-ctx on a **defined** fn (uses `beginFunction`, not `declareExtern`) | `src/ir/lower/decl.zig` |
|
||||
| 2.2 | green | gap (iii): `export "csym"` symbol-name override | `src/parser.zig`, `src/ir/lower/decl.zig` |
|
||||
|
||||
## Phase 3 — aggregates (objc / jni runtime classes)
|
||||
| Step | Commit | What | Files |
|
||||
|---|---|---|---|
|
||||
| 3.0 | xfail | `#objc_class("X") extern { … }` (import) + `… export { … }` (define) parse alongside legacy `#foreign #objc_class` | `src/parser.zig` (`tryParseForeignClassPrefix` :1305, `parseForeignClassDecl` :1369) |
|
||||
| 3.1 | green | map postfix `extern`→reference, `export`→define+register; per-runtime tests (objc, jni) | `src/parser.zig`, `src/ir/lower/decl.zig`, `src/ir/lower/objc_class.zig` |
|
||||
|
||||
## Phase 4 — interplay, diagnostics, docs
|
||||
`extern`+`callconv` stacking/redundancy; reject `extern`+`export` together;
|
||||
`specs.md` documents `extern`/`export` (the three axes); `#foreign` still documented
|
||||
until Part B cutover.
|
||||
|
||||
> **GATE A→B.** `extern`/`export` are a behavior-equivalent **superset** of
|
||||
> `#foreign`. Lock with a unit test asserting `#foreign` and `extern` lower to
|
||||
> identical IR for a sample fn / global / class. Do not start Part B before this.
|
||||
|
||||
---
|
||||
|
||||
# PART B — migrate `#foreign` → `extern`/`export`, then purge `foreign`
|
||||
|
||||
**Inventory (drives the batches):** `#foreign` = 466 uses. ~391 sx-code (308 fns
|
||||
[207 lib / 196 rename], 75 classes [39 objc / 31 jni], 8 globals) + ~145 example
|
||||
snapshots. 6 libs (`sqlib`98 `libc`61 `objc`22 `tlib`12 `raylib`7 `clib/pcaplib`3).
|
||||
Hotspots: `vendors/sqlite`(98), `platform/{android,uikit,android_jni,sdl3}`,
|
||||
`std/{socket,thread,fs,time}`, `ffi/{objc,raylib}`.
|
||||
|
||||
## Phase 5 — `#foreign` becomes an alias for `extern`
|
||||
| Step | Commit | What | Files |
|
||||
|---|---|---|---|
|
||||
| 5.0 | lock | route the `#foreign` parser paths (`parser.zig:316,425,1305,1970`) to build the *same extern-named* AST as `extern`/`export`. Suite green, snapshots unchanged | `src/parser.zig` |
|
||||
| 5.1 | lock | unit test: `#foreign` and `extern` produce identical IR (fn/global/class) | `src/ir/lower/decl.test.zig` |
|
||||
|
||||
## Phase 6 — migrate stdlib (behavior-preserving; snapshot diff must be EMPTY)
|
||||
One commit per batch; rewrite `#foreign`→`extern` (fns/globals),
|
||||
`#foreign #objc_class`→`#objc_class … extern`, defined classes → `… export`.
|
||||
| Step | Batch | ~sites |
|
||||
|---|---|---|
|
||||
| 6.1 | `library/vendors/sqlite/` | 98 |
|
||||
| 6.2 | `library/modules/platform/` (uikit/android/android_jni/sdl3) | ~95 |
|
||||
| 6.3 | `library/modules/std/` (socket/thread/fs/time/process/…) | ~60 |
|
||||
| 6.4 | `library/modules/ffi/` (objc/raylib/objc_block/…) | ~50 |
|
||||
| 6.5 | remaining `library/` + `vendors/` | remainder |
|
||||
|
||||
## Phase 7 — migrate examples + issues (empty snapshot diff; review every diff)
|
||||
| Step | Batch |
|
||||
|---|---|
|
||||
| 7.1 | `examples/12xx-ffi-*` (plain C) |
|
||||
| 7.2 | `examples/13xx-ffi-objc-*` |
|
||||
| 7.3 | `examples/14xx-ffi-jni-*` |
|
||||
| 7.4 | `issues/*` repros + stragglers |
|
||||
A non-empty diff ⇒ the alias wasn't behavior-equivalent — stop, fix Phase 5.
|
||||
|
||||
## Phase 8 — cutover
|
||||
| Step | Commit | What |
|
||||
|---|---|---|
|
||||
| 8.0 | xfail | `examples/11xx-diagnostics-foreign-removed.sx` expects a "`#foreign` removed; use `extern`/`export`" diagnostic — still accepted (red) |
|
||||
| 8.1 | green | parser hard-rejects `#foreign` (mirrors the variadic `name: ..T` cutover); `specs.md` drops `#foreign`, documents `extern`/`export` |
|
||||
|
||||
## Phase 9 — total `foreign` purge (the invariant)
|
||||
`foreign` must not appear anywhere in the live tree, surface *or* internal. Each step
|
||||
a mechanical, behavior-preserving rename commit (snapshots unchanged), small
|
||||
per-file/subsystem commits — not one sweep.
|
||||
| Step | What | Identifiers (count → new) |
|
||||
|---|---|---|
|
||||
| 9.0 | delete the surface | `hash_foreign`(11) + lexer entry + the 4 parse paths + the alias |
|
||||
| 9.1 | rename **linkage** → `extern*` | `foreign_expr`(25) **eliminated** (folds into modifier) · `is_foreign`(39)→`is_extern` · `foreign_lib`/`foreign_name`→`extern_*` · `foreign_name_map`→`extern_name_map` · `callForeign`(8)→`callExtern` · `marshalForeignArg`→`marshalExternArg` · `is_foreign_c_api`(5)→`is_extern_c_api` · `dedupeForeignSymbol`→`dedupeExternSymbol` |
|
||||
| 9.2 | rename **runtime-class** machinery → `runtime*` (decision 5) | `ForeignClassDecl`(65) · `ForeignMethodDecl`(31) · `ForeignClassMember`(20) · `ForeignFieldDecl`(15) · `foreign_class_map`(44) · `current_foreign_class`(34)/`_method` · `foreign_path`(62) · `ForeignRuntime` · `parse/tryParseForeignClass*` · `lowerForeign{Method,Static}Call` · `findForeign{Method,Property}InChain` · `resolveForeign*` · `register*ForeignClass*` · `foreignClass*Type` · `*ForeignRefs` |
|
||||
| 9.3 | purge **live docs** (28 lines) | `specs.md`/`readme.md`/`CLAUDE.md`: drop `#foreign`, document `extern`/`export`; fix file-roles + FFI/bundling notes |
|
||||
| 9.4 | **acceptance gate** | `grep -rniE 'foreign' src/ library/ examples/ specs.md readme.md CLAUDE.md` → **0** |
|
||||
|
||||
---
|
||||
|
||||
## Open decisions
|
||||
*Part A (ratified — recommendations stand):* 1. bare keywords (not `#extern`).
|
||||
2. aggregate position postfix (`#objc_class(…) extern`, like `struct #compiler`).
|
||||
3. `extern ⇒ callconv(.c)`. 4. **REVISED** (user, 2026-06-14): `extern` carries an
|
||||
optional `LIB`+`"csym"` axis (`extern_lib`/`extern_name`), mirroring `#foreign LIB
|
||||
"csym"`, so it's a true `#foreign` superset (Gate A→B). The `#library` declaration +
|
||||
build-flag linking mechanism stays separate — `extern` references a lib, doesn't
|
||||
fold in `#library`. (Was: "library fully separate / not on `extern`".)
|
||||
*Part B:* 5. runtime-class rename target — **RATIFIED `Runtime*Class*`** (user, 2026-06-14;
|
||||
it's the object-model axis, not linkage). 6. historical carve-out — **STILL OPEN** (user did
|
||||
not confirm at the Part A milestone): keep `issues/*.md` (+ design-doc prose) as provenance &
|
||||
gate only the live tree (recommended) vs purge everything. Confirm 6 before Phase 9.
|
||||
|
||||
## Relationship to ASM
|
||||
`PLAN-ASM.md` Phase F (global asm) consumes `extern` (import the asm symbol) and
|
||||
`export` (let asm call back into sx) — do it after **Part A Phase 2**.
|
||||
|
||||
---
|
||||
|
||||
## Kickoff prompt (paste into a fresh session to start Part A)
|
||||
|
||||
> Work the FFI-linkage stream per `current/PLAN-EXTERN-EXPORT.md` (+ checkpoint
|
||||
> `current/CHECKPOINT-EXTERN-EXPORT.md`). First read the plan's header (Decided
|
||||
> syntax, Naming constraint, Key finding) and Part A; rationale is in
|
||||
> `design/inline-asm-design.md` §II.2 (Deviation 6) + §II.10 #4.
|
||||
>
|
||||
> **This session = Part A, Phases 0 and 1 only** (`extern` works as a bare postfix
|
||||
> keyword equivalent to a lib-less `#foreign` fn/global binding; `#foreign` stays
|
||||
> untouched). Do NOT start Phase 2 (`export`) or Part B (migration).
|
||||
>
|
||||
> **Cadence (IMPASSIBLE):** no commit may both add a test and make it pass — lock
|
||||
> behavior with a passing test, or land an xfail the next commit turns green.
|
||||
> `zig build && zig build test` after every step.
|
||||
>
|
||||
> **Naming constraint (hard):** introduce only `extern`-named AST — do NOT reuse or
|
||||
> extend `ForeignExpr`/`foreign_expr`/`VarDecl.is_foreign`. Use a new
|
||||
> `FnDecl.extern_export` modifier (body `;` or `{…}`) and `VarDecl.is_extern`/
|
||||
> `extern_name`. IR is already extern-named (`Function.is_extern`, `declareExtern`).
|
||||
>
|
||||
> Steps (commit after each; update the checkpoint each time):
|
||||
> - 0.0 lock: `kw_extern`/`kw_export` tokens + map entries beside `kw_callconv`
|
||||
> (`src/token.zig:45,282`) + unit lex test.
|
||||
> - 0.1 lock: `parseOptionalExternExport()` (mirror `parseOptionalCallConv`,
|
||||
> `parser.zig:3669`) + `ast.ExternExportModifier` + `FnDecl.extern_export` +
|
||||
> `VarDecl.is_extern`/`extern_name` (parsed, unconsumed) + unit AST test.
|
||||
> - 1.0 xfail: accept postfix `extern` after the callconv slot (`parser.zig:1950`);
|
||||
> add `examples/12xx-ffi-extern-fn.sx` that extern-binds a libc symbol (red).
|
||||
> - 1.1 green: in `src/ir/lower/decl.zig`, lower `extern` like a lib-less `#foreign`
|
||||
> import — `is_extern`, `.external`, `callconv(.c)`, no ctx, via `declareExtern`
|
||||
> (anchors :1123, :387, :2110, :2113). Example goes green.
|
||||
> - 1.2 green: optional `extern "csym"` rename + extern-global `g : T extern;`
|
||||
> (`parser.zig:425`).
|
||||
>
|
||||
> Stop at end of Phase 1. Verify: suite green; the `extern` libc binding runs;
|
||||
> `#foreign` still works with no snapshot diffs. If you hit an unrelated compiler
|
||||
> bug, follow the CLAUDE.md IMPASSIBLE RULE (file an issue, stop).
|
||||
276
current/PLAN-FIBERS.md
Normal file
276
current/PLAN-FIBERS.md
Normal file
@@ -0,0 +1,276 @@
|
||||
# PLAN-FIBERS — Stream B1 (fibers + Io + M:1 scheduler)
|
||||
|
||||
> **STATUS: ✅ COMPLETE.** The pure-sx M:1 async runtime is feature-complete end-to-end
|
||||
> (`library/modules/std/sched.sx`, examples 1800–1817, suite 755/0): `abi(.naked)` context switch
|
||||
> (aarch64 + x86_64/Win64), M:1 scheduler, suspending `go`/`wait`/`cancel`, deterministic virtual-time
|
||||
> timers (`sleep`/`now_ms`), and real fd readiness via kqueue (`block_on_fd`). Five compiler bugs fixed
|
||||
> en route (0151/0152/0153/0154/0156-P1/0157). Deferred non-blocking follow-ups: linux `epoll` twin,
|
||||
> `Scheduler.deinit`, `Future(void)`/`timeout` (0150), `context.io`-routed async (M:N). Next carve:
|
||||
> Stream B2 (channels / cancel / async stdlib). Historical step-status below.
|
||||
>
|
||||
> B1.0 (`abi(.naked)`) ✅ · B1.1 (per-fiber `context`) ✅ · B1.2
|
||||
> (`Io` interface + `async`/`await`/`cancel` over blocking `CBlockingIo`) ✅ · B1.3 (fiber
|
||||
> runtime: naked `swap_context` + §10.7 stress gate + guarded `mmap` stacks, proven on aarch64
|
||||
> AND x86_64/Win64) ✅ · **B1.5a (M:1 scheduler CORE — `std/sched.sx`: `spawn`/`yield_now`/
|
||||
> `suspend_self`/`wake`/`run`) ✅** (fixed blocker 0154) · **B1.4a (suspending fiber-task async —
|
||||
> `sched.go`/`wait`/`cancel` over `Task($R)`, nullary-thunk) ✅** (adversarially reviewed; fixed
|
||||
> blockers 0156-Part1 + 0157 en route; locked `1813`).
|
||||
> **B1.4b (deterministic virtual-time timers — sched.sleep/now_ms/timer-run) ✅** (reviewed; fixed a CRITICAL timer-vs-early-wake UAF; locked 1814/1815).
|
||||
> **B1.4c (event-loop — real fd readiness via kqueue: `block_on_fd` + run-loop Mode 2) ✅** (reviewed; fixed a CRITICAL same-fd lost-wakeup hang; locked 1816). macOS only — linux epoll twin deferred.
|
||||
> **B1.5 (end-to-end M:1 capstone — `go`/`wait`+`sleep`+scheduler, deterministic ordering) ✅** (locked 1817). **STREAM B1 COMPLETE.** Detailed progress in [CHECKPOINT-FIBERS.md](CHECKPOINT-FIBERS.md). NOTE: suspending async +
|
||||
> deterministic timers live as `sched.*` methods (M:1), NOT routed through the erased `context.io` (avoids forcing sched.sx into every std consumer + the `_fib_tramp` dup-symbol
|
||||
> trap); the `Io` protocol's `spawn_raw`/`suspend_raw`/`ready` stay reserved for M:N. Deferred:
|
||||
> issue 0150 (`Future(void)`/`timeout`); 0156-Part2 (deferred `..` spread); the `::` callable-param
|
||||
> feature.
|
||||
|
||||
Carved from [PLAN-POST-METATYPE.md](PLAN-POST-METATYPE.md) Stream B (§B1) + the
|
||||
design-of-record [../design/execution-evolution-roadmap.md](../design/execution-evolution-roadmap.md)
|
||||
§4 (async), §7 steps 4–9, §8.1 (risks), §10 (testing). Progress in
|
||||
[CHECKPOINT-FIBERS.md](CHECKPOINT-FIBERS.md). Stream B2 (channels/cancel/stdlib) is a
|
||||
separate carve ([PLAN-CHANNELS.md], when reached) and depends on this + atomics (✅).
|
||||
|
||||
**Goal:** the colorblind, stackful, **pure-sx** async runtime — fibers behind an `Io`
|
||||
interface, an M:1 scheduler, blocking + deterministic-sim + event-loop `Io` impls. The
|
||||
**compiler floor is small and net-new**: make `abi(.naked)` actually emit an LLVM `naked`
|
||||
function (B1.0), and confirm/close the per-fiber `context` root (B1.1). **Everything
|
||||
else — the context-switch asm, fiber bootstrap, `mmap` stacks, the scheduler, futures,
|
||||
the `Io` vtables — is ordinary sx library code** (design §4, §4.4). The irreducible FFI
|
||||
floor: the per-arch asm context-switch (in `.sx`), syscall `extern`s, and `mmap`.
|
||||
|
||||
**Cadence (IMPASSIBLE):** no commit both adds a test AND makes it pass (lock-to-bail, then
|
||||
flip to green); `zig build && zig build test` green after every step; never regen snapshots
|
||||
while red; scope regens with `-Dname=examples/NNNN-…sx -Dupdate-goldens` + review the diff.
|
||||
New corpus category: `18xx` concurrency. On an **unrelated** compiler bug → file
|
||||
`issues/NNNN`, mark this checkpoint BLOCKED, STOP (CLAUDE.md). The in-session
|
||||
worker-fix override (delegate a blocker to a worker) applies only with explicit user
|
||||
authorization.
|
||||
|
||||
---
|
||||
|
||||
## Design (grounded against the tree)
|
||||
|
||||
### B1.0 — `abi(.naked)` codegen (the one genuinely net-new compiler piece in B1)
|
||||
|
||||
The design doc spells this `callconv(.naked)`; the **real sx surface is `abi(.naked)`** —
|
||||
written in the postfix slot, `name :: (sig) -> Ret abi(.naked) { asm { … }; }` (cf.
|
||||
`build_options :: () -> BuildOptions abi(.compiler);` in [build.sx:28](../library/modules/build.sx#L28)).
|
||||
The sx-facing name is **`naked`** throughout (keyword, field `is_naked`, diagnostics) —
|
||||
matching LLVM's `naked` attribute (the lowering mechanism) and the industry term
|
||||
(Zig/Rust/GCC/Clang). The ABI variant was renamed `.pure → .naked`: "pure" universally
|
||||
means *side-effect-free*, the opposite of a register-clobbering context switch.
|
||||
|
||||
**Grounding (verified — do not re-derive):**
|
||||
- The `ABI` enum **already carries `.naked`** — `ABI = enum { default, c, compiler, naked }`
|
||||
([ast.zig:142](../src/ast.zig#L142)), documented "naked function (inline asm
|
||||
body), no calling-convention prologue/epilogue." So B1.0 is **NOT** "extend the enum."
|
||||
- `.naked` is **inert today**: [type_resolver.zig:237](../src/ir/type_resolver.zig#L237)
|
||||
maps `.compiler, .naked → .default` CC, and `emit_llvm` emits **no LLVM `naked`
|
||||
attribute**. So the net-new work is exactly: **carry `abi == .naked` into the IR
|
||||
`Function`, emit LLVM's `naked` attr, and skip the implicit-`Context` / prologue
|
||||
lowering** so the body is just the asm block + its own `ret`.
|
||||
- The IR `Function` struct ([inst.zig:605](../src/ir/inst.zig#L605)) carries `call_conv`
|
||||
(default/c) + `is_compiler_domain`, but **no naked flag** — add one (`is_naked: bool`).
|
||||
- Attribute API is in-tree: `nounwind` is set at
|
||||
[emit_llvm.zig:1339](../src/ir/emit_llvm.zig#L1339) via
|
||||
`LLVMGetEnumAttributeKindForName("nounwind", 8)` → `LLVMCreateEnumAttribute(ctx, id, 0)`
|
||||
→ `LLVMAddAttributeAtIndex(func, func_idx_attr /* -1 */, attr)`. The LLVM `naked` attr
|
||||
is the same shape: `LLVMGetEnumAttributeKindForName("naked", 5)`.
|
||||
- The `.c` ABI **already skips the implicit ctx** at lowering — `lam.abi == .c` /
|
||||
`fd.abi == .c` gates (closure.zig:171, [decl.zig:515](../src/ir/lower/decl.zig#L515)).
|
||||
`.naked` must skip it **too** (a `.naked` fn gets no synthetic `__sx_ctx`, no stack frame,
|
||||
no prologue — args arrive in ABI registers and are read directly from asm). The
|
||||
implicit-return machinery (`lowerValueBody`) must also be bypassed: a `.naked` body has no
|
||||
sx return (the asm rets itself), so lower its statements and cap the block with
|
||||
`unreachable`.
|
||||
- **Inline asm already works end-to-end** (lower→emit→JIT): aarch64
|
||||
([examples/1645](../examples/1645-platform-asm-aarch64-add.sx)), x86_64
|
||||
([examples/1651](../examples/1651-platform-asm-x86-syscall-write.sx)), global asm, JIT
|
||||
([1653](../examples/1653-platform-asm-global-jit.sx)). `emitInlineAsm` /
|
||||
`LLVMGetInlineAsm` at [ops.zig:915](../src/backend/llvm/ops.zig#L915). The `.naked` body
|
||||
is a single asm block reusing this path.
|
||||
|
||||
**`.naked` ≠ `.c` (design §4.6 context-switch note):** a `.c` epilogue restores SP from the
|
||||
frame; a context switch deliberately makes SP-in ≠ SP-out, so the `.c` epilogue would
|
||||
restore from the *wrong* stack. `.naked` = no prologue/epilogue/frame — the asm emits its
|
||||
own `ret`. This is *why* the switch must be `.naked`, not `.c`.
|
||||
|
||||
**Snapshot story (per the atomics precedent):** a `.naked` fn's *body is raw per-arch asm*
|
||||
(it can't be portable — that's the point), while LLVM's `naked` attribute text is
|
||||
arch-invariant. **B1.0a** (lock) needs only **one host example** locked to the emit bail —
|
||||
the bail fires at the function level *before* any asm/instruction selection, so it is
|
||||
host-independent (no `.build` target pin). **B1.0b** (green) adds emission, pins that
|
||||
example aarch64 (`.build {"target": "aarch64-macos"}`, end-to-end on a matching host,
|
||||
ir-only on a mismatch), and adds an x86_64 cross sibling — mirroring the existing asm
|
||||
corpus split (1645 aarch64 / 1651 x86). The ir-only `.ir` (only producible once emission
|
||||
lands in B1.0b) asserts the `naked` attribute + the asm body. State loudly: **the `.ir`
|
||||
proves the `naked` keyword + asm emitted, NOT that any hand-written register save/restore
|
||||
is correct** — that is the B1.3 switch-stress harness's job, never the corpus's.
|
||||
|
||||
### B1.1 — per-fiber `context` root (grounding says this is SMALL, likely library-only)
|
||||
|
||||
**Grounding (verified — closes the design doc's open sizing question):**
|
||||
- `context` is an **implicit `*Context` parameter** (`__sx_ctx`, slot 0), threaded through
|
||||
every default-conv sx call ([lower.zig:259](../src/ir/lower.zig#L259)) — **not raw TLS**.
|
||||
Inside a function `current_ctx_ref = Ref.fromIndex(0)` (the param) → it **rides the fiber
|
||||
stack frame for free**.
|
||||
- `push Context.{…}` allocates the new `Context` with a **stack `alloca`** and rebinds
|
||||
`current_ctx_ref` to that slot ([stmt.zig:1263](../src/ir/lower/stmt.zig#L1263)) — "No
|
||||
global, no walk." So **push frames are fiber-local for free**.
|
||||
- The **only shared root** is the `__sx_default_context` **global**, bound at
|
||||
entry-points / `abi(.c)` fns *before any user code runs*
|
||||
([decl.zig:2667](../src/ir/lower/decl.zig#L2667), :2815).
|
||||
|
||||
⇒ The design doc's "lower as swappable indirection, never raw TLS" guards a **non-problem**
|
||||
(confirmed). The **real, now-sized** B1.1 work is purely a **library convention**: a
|
||||
freshly-`spawn`ed fiber must take its root `Context` from the **spawner's snapshot** (passed
|
||||
as the fiber-entry fn's `__sx_ctx` slot-0 arg by the spawn trampoline), **not** the
|
||||
`__sx_default_context` global. That is sx-side (the trampoline already controls slot 0) —
|
||||
**expected to be ZERO compiler change.** B1.1's first action is a probe confirming this; if
|
||||
a fiber genuinely re-reads the global root mid-stack (it should not — entry binds once),
|
||||
*then* and only then is there a compiler obligation. **Ground the probe before sizing any
|
||||
compiler work.** Prerequisite of B1.3 (a fiber needs a valid root before it switches).
|
||||
|
||||
### B1.2–B1.5 — pure sx over the primitives (design §4)
|
||||
- **B1.2 (A1):** `Io` interface + `context.io` + `Future` + `cancel()` — a protocol/vtable
|
||||
threaded exactly like `Allocator` (which already lives at `Context` field 0; see
|
||||
`allocViaContext` [call.zig:1214](../src/ir/lower/call.zig#L1214)). `Io` becomes another
|
||||
`Context` field. No compiler change — protocols + context already carry it.
|
||||
- **B1.3 (A2):** the fiber runtime — naked context-switch asm (per-arch), bootstrap, `mmap`
|
||||
stacks **with mandatory guard pages**. All sx. **Highest corruption risk in the stream**
|
||||
(§8.1.1) and **untestable by the deterministic `Io`** (which tests *scheduling*, not the
|
||||
*switch*). Its **first deliverable, before the scheduler AND the deterministic `Io`**: a
|
||||
standalone **2-fiber ping-pong switch-stress harness** (§10.7) — scribble every
|
||||
callee-saved register + a stack canary before each suspend, deep/recursive chains, verify
|
||||
all survive post-resume. This harness — not B1.4 — is A2's correctness gate.
|
||||
- **B1.4 (A3):** `Io` impls in order **blocking → deterministic-sim (KEYSTONE) → event-loop**
|
||||
(kqueue/epoll/io_uring). Build the deterministic `Io` right after blocking; **calibrate it
|
||||
against blocking `Io`** before trusting it to gate everything async (§8.1.3, §10.7) — a
|
||||
deterministic-but-wrong scheduler snapshots garbage. (Open, deferred: the event loop does
|
||||
**not** yet cooperate with a platform UI run loop — CFRunLoop/ALooper; that's a §6
|
||||
app-target gap, out of B1.)
|
||||
- **B1.5 (A5·M:1):** the single-thread scheduler — validates the whole colorblind stack
|
||||
end-to-end. `18xx` corpus runs under the deterministic `Io`, asserting a **program-emitted
|
||||
ordering contract** (sequence markers), not raw interleaving, so scheduler-policy tweaks
|
||||
don't churn every snapshot.
|
||||
|
||||
### Files the compiler floor touches (B1.0 only; B1.1–B1.5 are library + tests)
|
||||
B1.0 (`.naked`) forces these plumbing sites:
|
||||
- [ast.zig:142](../src/ast.zig#L142) — `ABI.naked` (exists; reference only).
|
||||
- [inst.zig:605](../src/ir/inst.zig#L605) — add `is_naked: bool = false` to `Function`.
|
||||
- [decl.zig](../src/ir/lower/decl.zig) — set `is_naked` from `fd.abi == .naked`; gate the
|
||||
implicit-ctx off for `.naked` in `funcWantsImplicitCtx` (mirror the `.c` skip at
|
||||
decl.zig:515) and bypass `lowerValueBody` for `.naked` bodies (lower statements + cap with
|
||||
`unreachable`, in both body-lowering paths) — a `.naked` fn binds no ctx and has no sx
|
||||
return.
|
||||
- [type_resolver.zig:237](../src/ir/type_resolver.zig#L237) — leave CC `.default` (a `.naked`
|
||||
fn-pointer type has no CC of its own; nakedness is a decl-level emit attribute).
|
||||
- [emit_llvm.zig:402](../src/ir/emit_llvm.zig#L402) Pass 2 — **B1.0a:** bail loudly when
|
||||
`func.is_naked` (build-gating). **B1.0b:** instead emit LLVM's `naked` attr (shape per
|
||||
`nounwind` at emit_llvm.zig:1339) + the asm-only body (no prologue).
|
||||
- Any `.op`/`Function`-field switch the Zig build flags — let the build tell you.
|
||||
|
||||
---
|
||||
|
||||
## Phases (xfail→green steps)
|
||||
|
||||
### B1.0 — `abi(.naked)` codegen — ✅ COMPLETE
|
||||
- **B1.0a (lock) — ✅ DONE.** Carried `abi == .naked` into IR `Function.is_naked`; threaded
|
||||
through `decl.zig` (`funcWantsImplicitCtx` skips `.naked` like `.c`; all body-lowering paths
|
||||
bypass `lowerValueBody` for `.naked`, lowering the asm body + capping with `unreachable`) +
|
||||
generic.zig + pack.zig; `emit_llvm` Pass 2 bailed loudly on `func.is_naked`. Locked by
|
||||
`examples/1800-concurrency-naked-asm.sx` + the generic regression (review-found gap).
|
||||
- **B1.0b (green) — ✅ DONE.** `emit_llvm` declaration pass adds LLVM `naked` + `noinline` +
|
||||
`nounwind` for `func.is_naked` and skips `frame-pointer=all` (incompatible with a frameless
|
||||
function); Pass 2 emits the body normally (`naked` ⇒ verbatim asm + own `ret`, no
|
||||
prologue). `1800` pinned aarch64 → exit 42 + `.ir`; `1801-concurrency-naked-generic.sx`
|
||||
(renamed from `-bail`) proves the generic path emits a naked body (exit 42);
|
||||
`1802-concurrency-naked-asm-x86.sx` x86_64 cross sibling (ir-only here, `.ir` locks `naked`
|
||||
+ `movl $42, %eax`). Unit test `emit: abi(.naked) function gets the naked attribute` asserts
|
||||
`naked` present + `frame-pointer` absent. Suite green (724/0).
|
||||
- **B1.0c (review-hardening) — ✅ DONE.** A param-bearing `.naked` fn emitted invalid LLVM
|
||||
(loud verifier error). Gated the param-alloca loop on `fd.abi != .naked` (decl.zig both
|
||||
paths + generic.zig) so a naked fn's args stay in registers (read by the asm body) — this
|
||||
*enables* B1.3's `swap_context(from, to)`. Locked by `1803-concurrency-naked-asm-param.sx`.
|
||||
Pack `.naked` (variadic + naked, nonsensical) left unsupported → loud verifier error.
|
||||
|
||||
### B1.1 — per-fiber `context` root — ✅ COMPLETE (zero compiler change)
|
||||
Probe confirmed the spawn convention works with ordinary language features: snapshot
|
||||
`context` (`snap := context`), store it in a struct, and `push f.root { entry(args) }` from a
|
||||
trampoline running under a different ambient context — the body reads the snapshot (via the
|
||||
implicit slot-0 `*Context` param), not the ambient ctx, and `push` restores ambient on exit.
|
||||
No path re-reads `__sx_default_context` mid-stack ⇒ **no compiler obligation**; this is a pure
|
||||
library convention. Locked by `examples/1804-concurrency-context-snapshot.sx` (`fiber root:
|
||||
42` / `ambient after: 99`). The design doc's "never raw TLS" guarded a non-problem.
|
||||
|
||||
### B1.2 — A1: `Io` interface + `context.io` + `Future` + `cancel()` API
|
||||
Library-only. `Io` as a protocol added to `Context` (mirror `Allocator`). `Future`/`cancel`
|
||||
API surface. xfail→green via an `18xx` example exercising the blocking `Io` default (real
|
||||
suspend lands in B1.3). No compiler change expected; if a protocol-in-context gap appears,
|
||||
file it.
|
||||
|
||||
### B1.3 — A2: fiber runtime (naked switch + bootstrap + guarded `mmap` stacks) — ✅ COMPLETE
|
||||
- **B1.3a (switch-stress harness FIRST) — ✅** the §10.7 register/canary-survival gate (1807/1808),
|
||||
validity proven by negative controls, adversarially reviewed.
|
||||
- **B1.3b — ✅** fiber bootstrap + guarded `mmap` stacks (1809); the x86_64 sibling landed as Win64
|
||||
on a real VM (1810, `0 0 P`). Switch proven on TWO arch/ABI pairs.
|
||||
|
||||
### B1.5a — M:1 scheduler CORE (`std/sched.sx`) — ✅ COMPLETE
|
||||
The reusable scheduler wrapping `swap_context`: generic `Fiber`/`Scheduler`,
|
||||
`spawn`/`yield_now`/`suspend_self`/`wake`/`run` over guarded `mmap` stacks, one generic
|
||||
`fib_dispatch` running any stored closure body. Adversarially reviewed + hardened; fixed blocker
|
||||
bug 0154 (struct-field `null`/`---` over-store) en route. Locked by `1811` (round-robin) + `1812`
|
||||
(suspend/wake). Built BEFORE the deterministic `Io` because FiberIo (B1.4a) needs it as substrate.
|
||||
|
||||
### B1.4a — suspending fiber-task async (`sched.go`/`wait`/`cancel`) — ✅ COMPLETE
|
||||
`Task($R)` + `Scheduler.go(work) -> *Task($R)` + `wait`/`cancel` in `sched.sx` (nullary-thunk;
|
||||
self-contained). `go` spawns `work` as a fiber, `wait` parks the caller until it completes. Locked
|
||||
by `1813`. Two compiler blockers fixed (0156-Part1, 0157) + adversarially reviewed/hardened.
|
||||
|
||||
### B1.4b/c — A3: `Io` impls (deterministic-sim KEYSTONE → event-loop)
|
||||
Blocking exists (io.sx `CBlockingIo`). Next the deterministic-sim `Io`, **calibrated against
|
||||
blocking** before any `18xx` test trusts it; then the event loop. The deterministic `Io` is the
|
||||
test harness for *all* of B1.5 + Stream B2.
|
||||
|
||||
### B1.5 — A5: M:1 scheduler — ✅ COMPLETE
|
||||
End-to-end validation of the colorblind stack. The `18xx` corpus asserts program-emitted ordering
|
||||
contracts under the scheduler + deterministic timers; the capstone `1817` composes `go`/`wait` +
|
||||
`sleep`/`now_ms` + the scheduler (three tasks complete in deadline order, deterministic sum). Stream
|
||||
B1 is feature-complete.
|
||||
|
||||
---
|
||||
|
||||
## Gates
|
||||
- **B1.0:** unit `emit_llvm.test.zig` (the `naked` attr present on a `.naked` fn); two
|
||||
arch-gated examples (aarch64 + x86_64) run end-to-end on a matching host, ir-only on a
|
||||
mismatch (assert `naked` + asm in `.ir`). **OUT of corpus scope, stated loudly:** the
|
||||
*correctness* of any hand-written register save/restore — that's the B1.3 stress harness.
|
||||
- **B1.1:** an `18xx` example locking context-carried-by-slot-0 behavior + a checkpoint note
|
||||
on the spawn-trampoline convention.
|
||||
- **B1.3:** the **switch-stress harness is A2's gate** (register/canary survival — §10.7),
|
||||
NOT a run/snapshot test; plus arch-gated run tests.
|
||||
- **B1.4:** deterministic `Io` **calibrated** against blocking `Io` (§8.1.3) before trusting
|
||||
it; `18xx` under the deterministic `Io`.
|
||||
- **B1.5:** `18xx` ordering-contract snapshots under the deterministic `Io`.
|
||||
|
||||
## Kickoff prompt (B1.0b — paste into a fresh session)
|
||||
> Implement Stream B1 step **B1.0b** (`abi(.naked)` real emission) per
|
||||
> `current/PLAN-FIBERS.md`. Verify `zig build && zig build test` is green first (B1.0a is
|
||||
> already landed: `Function.is_naked` plumbed, `decl.zig` skips ctx + bypasses implicit-return
|
||||
> for `.naked`, `emit_llvm` Pass 2 bails loudly, `examples/1800-concurrency-naked-asm.sx`
|
||||
> locked to the bail). Then: (1) in `src/ir/emit_llvm.zig` Pass 2 (~line 402), REPLACE the
|
||||
> `func.is_naked` bail with real emission — set LLVM's `naked` attribute on the function
|
||||
> (`LLVMGetEnumAttributeKindForName("naked", 5)` → `LLVMCreateEnumAttribute(ctx, id, 0)` →
|
||||
> `LLVMAddAttributeAtIndex(llvm_func, -1, attr)`; shape per the `nounwind` set at
|
||||
> emit_llvm.zig:1339) and emit the `.naked` body as its asm block only, no prologue/epilogue
|
||||
> (the body already lowers to the inline-asm op + an `unreachable` terminator). (2) Pin
|
||||
> `examples/1800-concurrency-naked-asm.sx` aarch64 with a `.build` sidecar
|
||||
> `{"target":"aarch64-macos"}`; on this aarch64 host it runs end-to-end (exit 42), capture
|
||||
> `.ir` + regen (`-Dname=examples/1800-concurrency-naked-asm.sx -Dupdate-goldens`), review the
|
||||
> diff (assert the `.ir` shows the `naked` attr + `mov x0, #42` / `ret`, NO stray error
|
||||
> text). (3) Add `examples/1802-concurrency-naked-asm-x86.sx` (x86_64 body, `.build
|
||||
> {"target":"x86_64-linux"}`, ir-only on this host — requires its `.ir`, now producible).
|
||||
> (4) Add a unit test in `src/ir/emit_llvm.test.zig` asserting the `naked` attribute is
|
||||
> present on an `abi(.naked)` function. Confirm `zig build test` green, commit. NOTE: the
|
||||
> `.ir` proves the keyword + asm emitted, NOT register-save correctness (that's the B1.3
|
||||
> switch-stress harness). If you hit an UNRELATED compiler bug, file `issues/NNNN`, mark
|
||||
> `CHECKPOINT-FIBERS.md` BLOCKED, and STOP.
|
||||
@@ -1,227 +0,0 @@
|
||||
# PLAN-HTTPZ — Stream HTTPZ (production HTTP-server readiness)
|
||||
|
||||
> **STATUS: 🟡 PLANNED — not started.** This stream is being (re)established as a
|
||||
> *tracked* stream. The HTTP/socket/thread work to date shipped ad-hoc under phase
|
||||
> tags in source comments (`S2` socket nonblocking, `S6` pthreads, `S7a` http server,
|
||||
> `S7b` thread-pool handlers, `C3` per-OS selection) but **never had a PLAN/CHECKPOINT
|
||||
> file** — the comments in [socket.sx:3](../library/modules/std/socket.sx#L3) /
|
||||
> [thread.sx:23](../library/modules/std/thread.sx#L23) reference a "PLAN-HTTPZ" that did
|
||||
> not exist until now. Progress tracked in [CHECKPOINT-HTTPZ.md](CHECKPOINT-HTTPZ.md).
|
||||
|
||||
**Goal:** the low-level guarantees needed to run a long-lived HTTP service on Linux in
|
||||
production — *not* a web framework. Survive malformed clients, slow clients, overload,
|
||||
restarts, memory pressure, and a normal Linux deployment without every app author
|
||||
rediscovering the same failure modes. Driven by the user's production-readiness checklist
|
||||
(P0 blockers, P1 hardening, P2 ergonomics), mapped below to concrete sx work.
|
||||
|
||||
**Cadence (IMPASSIBLE):** no commit both adds a test AND makes it pass (lock-to-bail, then
|
||||
flip to green); `zig build && zig build test` green after every step; never regen snapshots
|
||||
while red; scope regens with `-Dname=examples/<cat>/<file>.sx -Dupdate-goldens` + review the
|
||||
diff. HTTP corpus lives in `examples/http/` (`16xx`/`http` category) + `examples/event/`.
|
||||
Stress/fuzz/load harnesses live OUTSIDE the corpus (the corpus runner has a 10s/example
|
||||
timeout and no network sandbox — see [corpus_run.test.zig](../src/corpus_run.test.zig)).
|
||||
|
||||
---
|
||||
|
||||
## Audit of record (grounded against the tree, 2026-06-26)
|
||||
|
||||
What already exists, so the next session does not redo discovery. **Two layers of P0 #1
|
||||
are already done to a high standard; one piece is a literal blocker.**
|
||||
|
||||
### ✅ Solid / Linux-validated — do NOT rebuild
|
||||
- **[event.sx](../library/modules/std/event.sx)** — `Loop` fully branches `OS == .linux`
|
||||
(epoll, lines ~85–251) vs kqueue (~251–323). Ran **6/6 green on real aarch64 Linux** in
|
||||
an Apple `container` VM (kernel 6.18); ABI corpus-locked by `examples/event/1633`.
|
||||
- **[net/epoll.sx](../library/modules/std/net/epoll.sx)** — arch-aware `EpollEvent` layout
|
||||
(12B packed x86_64 / 16B aligned aarch64 via the u32-split trick), correct flags, EINTR
|
||||
retry, `__errno_location`.
|
||||
- **[net/kqueue.sx](../library/modules/std/net/kqueue.sx)** — macOS-only, correct.
|
||||
- **[sched.sx](../library/modules/std/sched.sx)** — M:1 fiber runtime; epoll/kqueue
|
||||
fd-readiness fully branched incl. `EPOLL_CTL_DEL` after `EPOLLONESHOT`. Linux-tested.
|
||||
- **[json.sx](../library/modules/std/json.sx)** — streaming writer, zero-copy views,
|
||||
explicit allocators, stable key order. (Integers only — no floats.)
|
||||
|
||||
### ❌ BROKEN on Linux — the keystone blocker (Phase C3)
|
||||
- **[socket.sx](../library/modules/std/socket.sx)** — Darwin-only, no `OS` branching:
|
||||
- `SockAddr` ([:32](../library/modules/std/socket.sx#L32)) carries Darwin's `sin_len:u8`
|
||||
at offset 0; Linux `sockaddr_in` has no such field → family/port written to wrong
|
||||
offsets, addresses corrupted.
|
||||
- `O_NONBLOCK = 4` — Linux is `2048`; `set_nonblocking` sets the wrong bit.
|
||||
- errno constants are macOS values (`EAGAIN=35`→Linux 11, `EINPROGRESS=36`→115,
|
||||
`ECONNRESET=54`→104, …) → WouldBlock/reset detection silently breaks.
|
||||
- `errno_slot` binds `__error` ([:52](../library/modules/std/socket.sx#L52)) — Linux is
|
||||
`__errno_location`.
|
||||
- **[thread.sx](../library/modules/std/thread.sx)** — Darwin pthread struct sizes:
|
||||
- `MutexBuf = 64B` ([:44](../library/modules/std/thread.sx#L44)) is Darwin's
|
||||
`pthread_mutex_t`; glibc is **40B** → `pthread_mutex_init` overflows the buffer by
|
||||
24B. **Heap corruption on first mutex init under the thread pool.** (`CondBuf = 48B`
|
||||
happens to match glibc — fragile coincidence.)
|
||||
|
||||
### ⚠️ Works, unhardened — [http.sx](../library/modules/std/http.sx)
|
||||
Single-worker event loop; inline (`thread_pool_count = 0`) + pooled handlers; keep-alive
|
||||
+ pipelining; delivery timeouts (`timeout_request_ms`/`timeout_keepalive_ms`); conn cap
|
||||
(`max_conn`) + per-conn request cap (`request_count`); emits 400/413/431/503. Connection
|
||||
state machine: `CONN_FREE/READING/WRITING/KEEPALIVE/HANDLING` with `gen` counter.
|
||||
**Gaps (this stream's HTTP work):**
|
||||
- **Parser:** `Content-Length` only (no `Transfer-Encoding`); no per-header-line size
|
||||
limit; no header-count limit; no request-line/version syntax validation; no duplicate
|
||||
`Content-Length` rejection; no `Content-Length` overflow guard.
|
||||
- **Memory:** `Server.close()` ([:317](../library/modules/std/http.sx#L317)) frees neither
|
||||
the `conns` array, the `PoolState` struct, nor `ps.done` → shutdown leaks.
|
||||
- **Shutdown:** `run()` is an infinite loop; no `Server.stop()`; `close()` is abrupt (no
|
||||
drain).
|
||||
- **Timeouts:** delivery-only. **No handler-execution timeout** — a hung handler blocks
|
||||
the loop (inline) or pins a pool worker forever; no cancellation.
|
||||
- **Observability:** **none.** All accept/read/write/loop faults close the connection
|
||||
silently — no log hook, no counters/metrics.
|
||||
- **Response:** whole response built in one allocation; no streaming, no body-size
|
||||
backpressure; alloc-failure path unhandled.
|
||||
|
||||
### ❌ Absent entirely
|
||||
- **CI:** no `.github/workflows`, no Linux CI. Local `zig build test` on macOS only.
|
||||
- **Fuzz / sanitizers / leak-check:** none. [tests/stress-http.sh](../tests/stress-http.sh)
|
||||
is broken (references deleted `examples/32-http-server.sx`).
|
||||
- **Releases:** no git tags, no CHANGELOG, no stability tiers.
|
||||
- **Security:** no SECURITY.md, no disclosure process, no posture statement.
|
||||
- **Deploy docs:** cross-compile/static-link documented in [readme.md](../readme.md); no
|
||||
systemd/Docker/reverse-proxy/health-check/graceful-shutdown examples.
|
||||
- **TLS:** none yet — to be added natively via an mbedTLS FFI binding (Phase T). Proxy
|
||||
deployment stays documented as an option (D1).
|
||||
- **Routing / query / form helpers:** manual `if req.path == …` dispatch only.
|
||||
|
||||
---
|
||||
|
||||
## Phases (dependency-ordered; checklist item in parens)
|
||||
|
||||
C3 is the keystone — until socket.sx + thread.sx are correct on Linux, **nothing in P0 is
|
||||
honestly testable on Linux**, so Phase C precedes all else regardless of how the rest is
|
||||
sliced.
|
||||
|
||||
### Phase C — Linux foundation (P0 #1) — unblocks everything
|
||||
- **C3a — `socket.sx` per-OS.** Branch `SockAddr` (drop `sin_len` on Linux), `O_NONBLOCK`,
|
||||
the errno constants, and `errno_slot` (`__error` vs `__errno_location`) on `OS`/`ARCH`,
|
||||
mirroring the `inline if OS ==` pattern already proven in `event.sx`/`sched.sx`. No
|
||||
silent fallback defaults (CLAUDE.md rule). Lock a Linux-vs-Darwin layout/const test red,
|
||||
then green.
|
||||
- **C3b — `thread.sx` per-OS.** Correct `MutexBuf`/`CondBuf` sizes per glibc (40/48) vs
|
||||
Darwin (64/48), branched. Memory-safety fix, not cosmetics. Validate under the Apple
|
||||
`container` Linux VM that the pool no longer corrupts the heap.
|
||||
- **C4 — Linux CI.** A workflow building + running `zig build test` (incl. the HTTP corpus)
|
||||
on Linux. The Apple-`container` path is proven for local validation; CI needs a real
|
||||
Linux runner (GH Actions `ubuntu` and/or self-hosted aarch64). First CI of any kind for
|
||||
the repo.
|
||||
- **C5 — Linux socket I/O corpus.** Examples covering accept/read/write/close/error on
|
||||
Linux (today only the macOS-friendly `1633` covers the happy path). Threaded-handler
|
||||
example included.
|
||||
- *Acceptance:* basic server compiles + runs on Linux; HTTP suite passes on Linux; accept/
|
||||
read/write/close/error paths covered; threaded mode correct.
|
||||
|
||||
### Phase H — HTTP hardening (P0 #2–6)
|
||||
- **H1 — Parser hardening (#2).** Max header-line size, max header count, strict
|
||||
request-line + version validation, CRLF strictness, `Content-Length` overflow guard +
|
||||
duplicate/conflicting rejection, **`Transfer-Encoding: chunked` → 501** (full impl in
|
||||
S1), slowloris coverage (delivery-timeout already mitigates). Outcomes: 400 / 413 / 431 /
|
||||
501 / safe-close. Unit + fuzz-seed corpus.
|
||||
- **H2 — Memory lifecycle (#3).** Fix `Server.close()` to free `conns`, `PoolState`, and
|
||||
`ps.done`. Document allocator ownership (long-lived containers must capture their owner —
|
||||
CLAUDE.md rule; the read buffers are intentionally per-conn). Leak gate: start/stop loop
|
||||
with GPA counters asserting zero + repaired stress script for RSS-over-churn.
|
||||
- **H3 — Graceful shutdown (#4).** `Server.stop()` — stop accepting, drain in-flight within
|
||||
a timeout, close idle keep-alives, return from `run()` cleanly. Tests: start/stop/restart
|
||||
in one process; no FD leak; no mem leak.
|
||||
- **H4 — Explicit errors + observability hooks (#5, #9).** Route accept/read/write/loop
|
||||
faults through a pluggable log/error hook instead of silent close; add counters (active /
|
||||
accepted / closed conns, requests served, parser errors, timeouts, rejected, 4xx/5xx,
|
||||
pool queue depth, optional request duration). Hook-based — no forced logging format.
|
||||
(#5 and #9 interlock; land together.)
|
||||
- **H5 — Handler timeout + cancellation (#6).** Per-request deadline enforced in BOTH
|
||||
inline and pool modes; bound time in `CONN_HANDLING`; timed-out → 504 or safe close. A
|
||||
never-returning handler must not permanently consume capacity.
|
||||
|
||||
### Phase T — Native TLS via mbedTLS (#15) — revises the proxy-only posture
|
||||
Native in-process HTTPS by binding a vetted C library (mbedTLS) over FFI — **not** a
|
||||
pure-sx TLS stack (out of scope: security-critical, multi-year). Slotted after H because
|
||||
TLS folds into the same connection state machine + `read_more`/`write_more` paths, which
|
||||
must be stable first. Backend: **mbedTLS** (small pure-C, clean `WANT_READ`/`WANT_WRITE`
|
||||
non-blocking API, static-links cleanly into `--self-contained` musl ELF; Apache-2.0).
|
||||
- **T1 — mbedTLS FFI binding.** New `library/modules/ffi/mbedtls.sx` (or `std/tls.sx`):
|
||||
`extern "c"` decls for `mbedtls_ssl_{init,setup,handshake,read,write,close_notify}`,
|
||||
`mbedtls_ssl_config`, `mbedtls_x509_crt`, `mbedtls_pk_context`, `mbedtls_ctr_drbg` +
|
||||
`mbedtls_entropy`, `mbedtls_ssl_set_bio`, and the `WANT_READ`/`WANT_WRITE` error
|
||||
constants. Loud failure on any setup error (no silent default — CLAUDE.md rule).
|
||||
- **T2 — Transport abstraction in `http.sx`.** Introduce a transport seam so `read_more`/
|
||||
`write_more` go through plaintext (today's `socket.*_nb`) OR TLS, instead of calling the
|
||||
socket directly. mbedTLS BIO callbacks bridge to the non-blocking fd: map socket
|
||||
`WouldBlock` → `MBEDTLS_ERR_SSL_WANT_READ/WANT_WRITE`.
|
||||
- **T3 — Handshake state + event-loop integration.** New `CONN_TLS_HANDSHAKE` state before
|
||||
`CONN_READING`; drive `mbedtls_ssl_handshake` incrementally, mapping `WANT_READ` →
|
||||
`loop.add_read`, `WANT_WRITE` → `loop.add_write`; handshake deadline (reuse
|
||||
`timeout_request_ms`); graceful `close_notify` on shutdown (ties into H3).
|
||||
- **T4 — TLS config surface.** `Config` gains `tls_enabled`, cert/key/chain paths, min
|
||||
version (default TLS 1.2+, prefer 1.3), optional ALPN, SNI (single default cert first;
|
||||
multi-cert later). Cert/key load failure is a loud `HttpErr`, never a silent fallthrough.
|
||||
- **T5 — Tests + static-link + Linux validation.** TLS corpus example: in-process mbedTLS
|
||||
*client* handshakes against the server over loopback with a self-signed cert fixture
|
||||
(under `examples/http/16xx-…/`); cover bad-cert, handshake-failure, and mid-handshake
|
||||
client-abort paths. Verify a `--self-contained` static build links mbedTLS; run on macOS
|
||||
+ aarch64 Linux (Apple `container`). Document the per-target mbedTLS static-archive
|
||||
requirement for self-contained builds (vendor vs system).
|
||||
|
||||
### Phase S — Streaming, stress, stability (P1)
|
||||
- **S1 — Streaming responses + chunked out (#10).** Explicit `Content-Length`; stream large
|
||||
bodies without buffering the whole response; write-backpressure-aware send; header-set /
|
||||
status / content-type helpers. (Builds on H1's chunked scaffolding.)
|
||||
- **S2 — Request-body streaming (#11).** Incremental body reader, configurable max, early
|
||||
reject, mid-body-disconnect handling, backpressure-aware reads. Enables real inbound
|
||||
chunked bodies.
|
||||
- **S3 — Fuzz harness (#7).** libFuzzer/AFL targets: request-line, header, `Content-Length`,
|
||||
keep-alive + pipeline state machine, partial reads, malformed bodies, random close
|
||||
timing. Runs manually + in CI. Crash/panic/hang = bug.
|
||||
- **S4 — Load/stress suite (#8).** Repair + expand the stress scripts: many short-lived,
|
||||
many keep-alive, slow clients, large bodies at the limit, pool saturation, FD exhaustion
|
||||
→ 503/backpressure (not crash), RSS-over-time. Document expected overload behavior.
|
||||
- **S5 — Concurrency model docs (#12).** Write up the allocator/thread-safety rules already
|
||||
asserted in [thread.sx:11](../library/modules/std/thread.sx#L11) + the http.sx header:
|
||||
handler execution model, per-request lifetime, what may be retained after a handler
|
||||
returns, misuse cases.
|
||||
- **S6 — API stability + security posture (#13, #14).** Tag a milestone; define the stable
|
||||
std subset (`http`/`socket`/`event`/`thread`/`mem`); SECURITY.md + disclosure process +
|
||||
the "reverse-proxy-only, not for direct internet exposure" posture statement + known
|
||||
limitations.
|
||||
|
||||
### Phase D — Deploy & ergonomics (P2)
|
||||
- **D1 — Reverse-proxy + deployment docs (#15, #20).** With native TLS shipping in Phase T,
|
||||
proxy deployment is now *an option, not the only option* — document both. Cover proxy TLS
|
||||
termination, forwarded headers, client-IP, size limits, timeouts, keep-alive, recommended
|
||||
proxy settings; AND native-TLS direct-exposure guidance (cert rotation, cipher/version
|
||||
policy). Plus systemd unit, Docker, health-check endpoint, graceful-shutdown, logging
|
||||
examples; release-binary build + static/dynamic linking notes (incl. the mbedTLS
|
||||
static-archive note from T5; cross-compile already in readme.md).
|
||||
- **D2 — Routing + query/form helpers (#16, #17).** Thin layer over manual dispatch: method
|
||||
+ path routing, path params, query parsing, 404/405, per-route limits/timeouts; form
|
||||
(urlencoded/multipart) + JSON request/response helpers over the existing json.sx.
|
||||
- **D3 — Honest benchmarks (#18).** Revive [bench/run.sh](../bench/run.sh): plain-text,
|
||||
JSON, keep-alive, concurrency, pool, slow-client vs a baseline server; record hardware/
|
||||
OS/flags/command; measure latency, throughput, memory, error rate.
|
||||
- **D4 — Compiler-confidence framing (#19).** Largely already true (corpus + `issues/`
|
||||
regressions, subprocess-isolated runner). Add the "supported vs experimental" labelling
|
||||
for language + std features; ensure production-critical features have corpus coverage.
|
||||
|
||||
---
|
||||
|
||||
## Decisions Log (HTTPZ specifics)
|
||||
- **Native TLS via an mbedTLS FFI binding (Phase T)** — supersedes the original
|
||||
reverse-proxy-only posture (2026-06-26). The server gains in-process HTTPS; reverse-proxy
|
||||
deployment stays supported and documented (D1) as an option. **No pure-sx TLS stack** —
|
||||
TLS is security-critical and is delegated to the vetted C library. mbedTLS chosen over
|
||||
OpenSSL/LibreSSL for its small pure-C footprint, clean non-blocking `WANT_READ`/
|
||||
`WANT_WRITE` API, and clean static-linking into `--self-contained` musl builds
|
||||
(Apache-2.0).
|
||||
- **`Transfer-Encoding: chunked`: reject (501) in H1, implement in S1/S2.** Pragmatic P0
|
||||
minimum is explicit rejection; full chunked support is gated on the streaming work.
|
||||
- **Stress/fuzz/load live outside the corpus.** The corpus runner has a 10s/example
|
||||
timeout and no network sandbox; long-running adversarial harnesses are separate scripts
|
||||
wired into CI, not `examples/`.
|
||||
- **No silent fallback defaults in any C3 branching** (CLAUDE.md REJECTED PATTERNS): a
|
||||
failed/unhandled OS or arch arm bails loudly, never picks a "reasonable-looking" Darwin
|
||||
default.
|
||||
@@ -1,265 +0,0 @@
|
||||
# PLAN-IO-UNIFY — fold the fiber scheduler behind `context.io`, re-home `race`
|
||||
|
||||
## Why
|
||||
Today there are **two parallel async stacks**:
|
||||
|
||||
| stack | behind `context.io`? | real suspension? | cancellation channel |
|
||||
|---|---|---|---|
|
||||
| io.sx `async`/`await`/`cancel`/`Future` | yes (`impl Io for CBlockingIo`) | **no** — runs the worker inline to completion | `suspend_raw -> !` / `IoErr.Canceled` (designed, unused) |
|
||||
| sched.sx `go`/`wait`/`cancel`/`race` (just landed) | **no** | yes (`swap_context` fibers) | none — `suspend_self -> void` |
|
||||
|
||||
`context.io` is structurally Zig's `std.Io` (an `Io` protocol carried *implicitly* in `Context` — better
|
||||
ergonomics than Zig's explicit `io:` param), and the roadmap (§A5, §4.6) already says the fiber
|
||||
scheduler should be **one of its `Io` vtables** and that `race` is **`context.io.race(..)` over Futures**.
|
||||
The just-landed `race` on `sched.Scheduler` over `*Task` is the proven LOGIC at the wrong LAYER.
|
||||
|
||||
**Goal:** make the fiber `Scheduler` an `impl Io`, lift `async`/`await`/`cancel`/`race` onto the `Io`
|
||||
protocol so they run colorblind under either impl, and let cancellation fall out of the existing
|
||||
`suspend_raw -> !` contract (the "true cancellation, model A" the user picked — already the interface's
|
||||
design). One async stack, behind `context.io`.
|
||||
|
||||
## The fiber → `Io` mapping (the crux)
|
||||
`Io :: protocol { spawn_raw, suspend_raw -> !, ready, poll, now_ms, arm_timer }` (core.sx). Map each onto
|
||||
the existing fiber primitives in sched.sx (`spawn`/`suspend_self`/`wake`/`sleep`/`block_on_fd`/`run`):
|
||||
|
||||
| `Io` method | fiber realization |
|
||||
|---|---|
|
||||
| `spawn_raw(entry, arg, opts) -> *void` | `spawn` a fiber whose body invokes `entry(arg)` (raw C-ABI thunk, not a closure — see Bridge below). Returns the `*Fiber` as the opaque handle. |
|
||||
| `suspend_raw(park) -> !` | `suspend_self()`, then on resume CHECK the current task's cancel flag and `raise IoErr.Canceled` if set. `park.handle` = the `*Fiber` to re-ready. **This is the cancellation delivery point.** |
|
||||
| `ready(park)` | `wake(park.handle as *Fiber)` (already guarded on `.suspended`). |
|
||||
| `arm_timer(deadline_ms, park) -> *void` | arm a `Timer{deadline, fiber=park.handle}` (today's `sleep` minus the self-suspend); return the timer handle so a cancel can evict it. |
|
||||
| `poll(deadline_ms) -> i64` | ONE iteration of the `run` loop: drain ready, then fire the earliest timer / block on fds up to `deadline_ms`. Returns the next pending deadline (or sentinel when idle). |
|
||||
| `now_ms() -> i64` | the virtual `clock_ms` (deterministic), NOT a wall clock — keeps 1817/1821-style tests reproducible. |
|
||||
|
||||
`Scheduler.run()` stays as the explicit DRIVER (the top-level loop that calls `poll` to quiescence),
|
||||
installed via `push Context { io = xx scheduler } { … s.run(); }` — exactly the existing sched examples,
|
||||
just with the scheduler now reachable as `context.io`.
|
||||
|
||||
## Status (2026-06-28)
|
||||
- **Follow-up — heap leak reclamation (fiber-env + async). DONE.** Closed the
|
||||
documented per-spawn closure-env leak and most of the async leak, using only the
|
||||
existing `closure.env`/`.fn_ptr` field accessors (now also named by
|
||||
`ClosureRaw`/`SliceRaw` ABI-view structs in core.sx) — NO compiler change.
|
||||
- **Fiber body env:** `Scheduler.reap_fiber` frees `f.body.env` via
|
||||
`f.dctx.allocator` (the spawn-time allocator snapshotted in `dctx`) at all 3
|
||||
reap sites. 1820's `live after deinit` 3 → **0**.
|
||||
- **Async box + closure envs:** `sx_run_boxed_closure` frees the `ThunkBox`, the
|
||||
completion-closure env, and the worker's env (new `ThunkBox.worker_env`) the
|
||||
instant the worker completes.
|
||||
- **Async Future:** two-flag ownership — `Future.worker_done` (set at the end of
|
||||
the completion closure) + `consumed` (set at the end of `await`); `fut_release`
|
||||
frees the heap `Future` (via the stored `Future.alloc`) when BOTH are set, so
|
||||
the LAST of {worker, await} reclaims it. `await` now CONSUMES the future
|
||||
(single-use; documented). Residual for an AWAITED future: **0** (lock:
|
||||
`examples/concurrency/1827-...`). A NEVER-awaited future (fire-and-forget /
|
||||
`race` loser) keeps only its `Future` struct (consumed never set) — the
|
||||
structured-concurrency remainder, deferred.
|
||||
- Self-reviewed across orderings (await-after/before-complete, cancel-then-await,
|
||||
cancel-while-parked, double-free via await+deinit, race residual, blocking
|
||||
impl, cross-allocator reap) — all deterministic, no UAF/double-free. Suite
|
||||
855/0; byte-identical on aarch64-macOS + aarch64-linux; `.ir` churn (core.sx +
|
||||
Future/ThunkBox field additions) regenerated, only 1820 stdout changed
|
||||
otherwise.
|
||||
|
||||
- **Phase 5 — CONVERGE: retire the bespoke fiber async API. DONE. Io unification
|
||||
COMPLETE.** The bespoke `Task` layer (`Task`/`TaskState`/`TaskErr`/`go`/`wait`/
|
||||
`cancel(Task)` + `Scheduler.task_allocs` and its deinit handling, ~130 lines)
|
||||
is removed from sched.sx. There is now ONE async stack: `context.io.async`/
|
||||
`await`/`cancel`/`race`/`sleep` over the `Io` protocol, with the `Scheduler` as
|
||||
the fiber Io's engine + driver (`spawn`/`yield_now`/`suspend_self`/`wake`/`run`/
|
||||
`block_on_fd` stay as the raw primitives). Migrated the four `go`/`wait` users to
|
||||
`context.io`: 1813 (interleave + cancel), 1817 (m1 end-to-end sum=123), 1819
|
||||
(double-AWAIT loud-abort via the Future one-awaiter guard), 1820 (deinit — the
|
||||
`go`/`task_allocs` tasks dropped; it now exercises timers/io_waiters/kq cleanup,
|
||||
`freed=2`/`live=3`). `race` stays in sched.sx (needs meta.sx). Updated readme.md
|
||||
(the user-facing async section now documents `context.io.async`/`await`/`race`/
|
||||
`sleep`) and the stale `sched.go`/`sched.Task` comments in io.sx. Suite 854/0; no
|
||||
`.ir` churn (the Task removal touched no snapshotted IR); migrated examples
|
||||
byte-identical on aarch64-macOS + aarch64-linux. **PLAN-IO-UNIFY Phases 0–5 all
|
||||
complete — the two parallel async stacks are now one, behind `context.io`.**
|
||||
|
||||
- **Phase 4 — `race` over Futures via `context.io.race`. DONE.** Re-homed the
|
||||
proven first-wins race from `sched.race(*Task)` onto `*Future` handles + the
|
||||
`Io` protocol; the old Task-based `race` is REPLACED (ufcs overload-by-receiver
|
||||
is rejected — "duplicate top-level decl" — and only 1821 used it).
|
||||
- **Protocol affordance:** added `Io.current_park() -> ParkToken` (the running
|
||||
fiber as a token, captured WITHOUT parking) so race can register the SAME
|
||||
coordinator across N futures' `park` slots, then park once via `suspend_raw`;
|
||||
any completion `ready`s it. Scheduler returns `{self.current}` (bails outside
|
||||
a fiber); CBlockingIo returns `{null}` (race never parks there — futures born
|
||||
`.ready`). The await comment already anticipated this fan-in.
|
||||
- **race** (`ufcs (io: Io, futures: $T) -> RaceResult(T)`, in sched.sx — it
|
||||
needs meta.sx's `make_enum`/`make_variant`, and pulling that into the io.sx
|
||||
prelude part-file would cycle): winner scan → register+park → deregister →
|
||||
`make_variant` the winner → Phase-3 `cancel` each loser (NO join). `RaceResult`
|
||||
reused unchanged (`*Future(R)` projects field 0 `value` → R).
|
||||
- **Winner-time return:** with true cancellation the parked losers stop at their
|
||||
next suspend (their timers evicted by cancel's wake), so race returns at the
|
||||
winner's virtual time, not the slowest loser's. 1821 re-pointed to
|
||||
`context.io.async` + `context.io.race`: `winner a=111`, losers `.canceled`,
|
||||
completion log ONLY `task 1 @ 10ms`, final clock `10ms` (was 30 under the old
|
||||
cooperative join). Byte-identical on aarch64-macOS + aarch64-linux. Suite
|
||||
853/0; `.ir` churn (current_park vtable method) regenerated, only 1821 stdout
|
||||
changed otherwise.
|
||||
|
||||
- **Phase 3 — TRUE cancellation via `suspend_raw -> !`. DONE.** A cancelled async
|
||||
worker now abandons its body at its next suspend instead of running to
|
||||
completion. Pieces:
|
||||
- **Cancel-flag back-ref (D4 — back-ref pointer, chosen):** `SpawnOpts.cancel_flag:
|
||||
*void` (core.sx) + `Fiber.cancel_flag: *void` (sched.sx), set from
|
||||
`opts.cancel_flag` in `Scheduler.spawn_raw`. `async` passes `xx @f.canceled`
|
||||
(the `Future.canceled` `Atomic(bool)` erased to `*void`).
|
||||
- **Delivery:** `Scheduler.suspend_raw` checks `fiber_canceled(self.current)` (a
|
||||
`*Atomic(bool)` load) PRE-park (raise without parking — no deadlock if cancel
|
||||
landed before the worker ran) and POST-resume (cancel landed while parked),
|
||||
raising `error.Canceled` (a bare `-> !`; set inferred). `cancel(f)` flips the
|
||||
sticky flag, marks `.canceled`, and `ready(.{handle=f.task})`s the worker.
|
||||
- **Worker is failable** `Closure() -> ($R, !)`: the `async` completion closure
|
||||
`f.value = worker() catch { … }` (the captured-failable-closure-call the
|
||||
Phase-3-prereq fix enabled) marks `.canceled`/`.failed` and wakes the awaiter;
|
||||
the worker's post-suspend side effects never run. New failable `io.sleep(ms)`
|
||||
(arm_timer + `try suspend_raw`) is the cancellation point.
|
||||
- **Compiler gap fixed:** a `-> !` fn whose only error source is `try`-ing a
|
||||
protocol method (`io.suspend_raw`) was wrongly flagged "declared `!` but never
|
||||
errors". `collectErrorSites` (error_analysis.zig) now sets a `dyn` flag for a
|
||||
`try` of a non-identifier callee (opaque error channel), suppressing the
|
||||
warning.
|
||||
- **Two UAFs found by adversarial review and FIXED:** (1) cancel-before-park
|
||||
orphaned `io.sleep`'s armed timer → `suspend_raw`'s pre-park raise now evicts
|
||||
the current fiber's timer/waiter first. (2) `cancel(f)` woke a possibly-reaped
|
||||
worker → now only wakes when `was_pending` (`.pending` before the store).
|
||||
- Migrated 1805/1806/1824 to failable workers. Lock:
|
||||
`examples/concurrency/1825-concurrency-fiber-cancel-suspend.sx` (`seq: 1 -99`
|
||||
— post-suspend line never runs). **Validated byte-identical on aarch64-macOS
|
||||
host AND aarch64-linux container** (1824 + 1825). Suite 853/0. Expected `.ir`
|
||||
churn (SpawnOpts layout) regenerated; no non-`.ir` snapshot changed.
|
||||
|
||||
|
||||
- **Phase 3 PREREQUISITE — captured-failable-closure call typing. DONE.** The
|
||||
async completion closure (`b.run = () => { f.value = worker() catch {…} }`)
|
||||
captures a failable `worker` and consumes its error channel; the free-variable
|
||||
capture analysis (`collectCaptures` in `src/ir/lower/closure.zig`) did not
|
||||
descend into the error-handling / context / asm / multi-assign nodes, so
|
||||
`worker` was never captured — inside the lambda it resolved against an empty
|
||||
scope and typed as `.unresolved` (`catch`/`try` then rejected it). Fixed: added
|
||||
`try_expr`, `catch_expr`, `onfail_stmt`, `raise_stmt`, `multi_assign`,
|
||||
`push_stmt`, `comptime_expr`, `insert_expr`, `spread_expr`, `asm_expr` arms to
|
||||
`collectCaptures`. Adversarially reviewed (captures resolve, locals correctly
|
||||
excluded, no false-positive captures, 851/0). Lock: example
|
||||
`examples/closures/0314-closures-capture-failable-call.sx` (catch + try over a
|
||||
captured failable closure; pure language feature, host-only). The `push_stmt`
|
||||
arm also fixes the previously-noted "free-var analysis doesn't descend into a
|
||||
nested `push Context {…}`" gap. **Phase 3 is now unblocked.**
|
||||
- Two PRE-EXISTING, orthogonal bugs surfaced during review (neither blocked
|
||||
Phase 3): (1) calling a closure stored in a **struct data field** typed as
|
||||
`unresolved` (value → garbage; failable → can't `catch`) — **RESOLVED**
|
||||
(`issues/0201`): `CallResolver.plan` gained a closure/fn-pointer field arm and
|
||||
the lowering closure-field arm now also handles bare `.function` fields;
|
||||
regression `examples/closures/0315-closures-struct-field-call.sx`. (2) asm
|
||||
write-through place through a deref (`asm { … "+r" -> @(p.*) }`) fails LLVM
|
||||
verification — repros with NO closure (independent of capture analysis);
|
||||
possibly an unsupported deref-place form rather than a confirmed bug, not
|
||||
filed.
|
||||
|
||||
## Status (2026-06-27)
|
||||
- **Phase 0 — fibers inherit the spawn-time context. DONE** (`2f2d7f1d`). Discovered during Phase 1: a
|
||||
fiber body ran under `__sx_default_context` (the `abi(.c)` `fib_dispatch` dropped the implicit
|
||||
context), so a scheduler installed as `context.io` was invisible inside a worker. Fixed:
|
||||
`Scheduler.spawn` snapshots `context` → `Fiber.dctx`; `fib_dispatch` re-pushes it. Behavior-preserving
|
||||
(suite 828/0), no cross-fiber leak (context is parameter-threaded per stack). Lock: example 1822.
|
||||
- **Phase 1 — `impl Io for Scheduler`. DONE** (`5c30bfe0`, hardened `da7dd1f1`). Six methods over the
|
||||
fiber primitives; `spawn_raw` bridges the erased `(*void)->void` worker thunk via an fn-ptr round-trip.
|
||||
Lock: example 1823 (spawn→arm→suspend→ready→resume entirely through `context.io`, deterministic).
|
||||
Adversarial review fixed: `arm_timer`/`spawn_raw` null guards, `poll` fd-pending abort + `deadline_ms`
|
||||
doc, stale `fib_dispatch` comment.
|
||||
- **Resolved design decisions:** D1 = direct `impl Io for Scheduler` (chosen). D2 = `now_ms` returns the
|
||||
virtual `clock_ms` (deterministic) — a real-clock variant is later. D4 = deferred to Phase 3.
|
||||
- **Phase 2 — `async`/`await` colorblind over the fiber Io. DONE** (`967aed67`, hardened `ada8d162`).
|
||||
`async` heap-allocs a `*Future`, boxes a completion closure in a monomorphic `ThunkBox`, and submits
|
||||
via `io.spawn_raw` (inline under `CBlockingIo`, a fiber under the scheduler); `await` parks via
|
||||
`suspend_raw` until ready. Protocol changed to `suspend_raw(park: *ParkToken)` (write-back of the
|
||||
awaiter). Workers are nullary (call-site capture). Migrated 1805/1806; adopted `push .{ … }`. Lock:
|
||||
example 1824 (deferral visible: `1 2 10 20 123`). Review fixed: one-awaiter `await` guard; documented
|
||||
the Future allocator-lifetime contract + that `cancel` doesn't stop an already-spawned worker (Phase 3).
|
||||
- **Resolved D2 (ParkToken):** `suspend_raw(*ParkToken)` write-back (chosen over a registry). **ready()
|
||||
liveness (CONCERN 6):** safe for single async/await (awaiter is suspended, not reaped, when readied);
|
||||
`race` fan-in must still deregister (Phase 4).
|
||||
- **Carried to convergence:** `async` should capture the scheduler's long-lived allocator (like
|
||||
`sched.go`'s `own_allocator`) instead of the call-site `context.allocator` — needs a protocol
|
||||
affordance; documented as a contract for now.
|
||||
- **Open for later phases:**
|
||||
- **ParkToken↔fiber binding.** `ready(park)` needs `park.handle` = the awaiter `*Fiber`. The scheduler
|
||||
knows `self.current` at suspend; the cleanest is `suspend_raw(park: *ParkToken)` writing
|
||||
`park.handle = self.current` before parking (a small protocol change: the materializer installs
|
||||
thunks by name/order, signature-agnostic — verified low-risk). Decide vs a token→fiber registry.
|
||||
- **`ready()` liveness (review CONCERN 6).** Casting a stale/reaped `*Fiber` handle and `wake`-ing it is
|
||||
a latent UAF once real `await` runs — `wake`'s `.suspended` value-check on freed bytes is luck, not
|
||||
safety. Phase 2 must guarantee single-ready / deregistration (mirror the bespoke-race deregister).
|
||||
- **Out-of-scope compiler bug found by review (not filed yet):** closure free-var analysis does not
|
||||
descend into a nested `push Context {…}` block inside a closure body — a var used only there reports
|
||||
`unresolved`. Phase 0 sidesteps it (capture is at the `Fiber` level, not via closure), so it does NOT
|
||||
block the unification; worth an `issues/` entry in a separate session.
|
||||
|
||||
## Phases (each: implement → lock with an example → `zig build test` green → both platforms)
|
||||
|
||||
1. **`impl Io for Scheduler` (the vehicle).** Implement the six methods over the fiber primitives. Add
|
||||
a `Fiber.canceled`/task back-ref so `suspend_raw` can raise on resume. Keep `CBlockingIo` intact.
|
||||
Lock: install the fiber Io into `context.io`, run a root fiber that `suspend_raw`s and is `ready()`'d —
|
||||
asserts real park/resume through the protocol (not inline). **Bridge** (the one fiddly bit): `async`'s
|
||||
generic `Closure(..$args) -> $R` worker → `spawn_raw`'s raw `entry/arg`. Box the worker thunk on the
|
||||
heap; `entry` is a C-ABI `(env: *void) -> void` invoke-thunk (mirrors `fib_dispatch`), `arg` is the env.
|
||||
|
||||
2. **`async`/`await` over the fiber Io (real interleaving).** Under a suspending Io, `async` calls
|
||||
`spawn_raw` and returns a PENDING `Future($R)` (no longer born `.ready`); the spawned body fills
|
||||
`f.value`/`f.state` and `ready(f.park)`s the awaiter. `await(f)` checks `.ready` else `suspend_raw(f.park)`
|
||||
then returns/raises — the suspending sibling of today's immediate `await`. `CBlockingIo` keeps the
|
||||
run-inline path (degenerate, still correct). Lock: two `context.io.async` tasks interleave under the
|
||||
fiber Io (the io.sx layer, replacing the bespoke `sched.go`).
|
||||
|
||||
3. **True cancellation via `suspend_raw -> !`.** `cancel(f)` flips `f.canceled` AND `ready(f.park)`s /
|
||||
wakes the worker fiber so its NEXT `suspend_raw` raises `IoErr.Canceled`. The worker's suspends
|
||||
(`await`, a future `io.sleep`) propagate via `try`/`!`; the worker body unwinds, the future ends
|
||||
`.canceled`, its post-cancel side-effects DON'T run. This is the model-A "true cancellation" — now
|
||||
delivered through the protocol, not bespoke. Lock: a cancelled task's work stops at its next suspend
|
||||
(assert via a shared log: the post-suspend line never prints).
|
||||
|
||||
4. **`race` over Futures — `context.io.race((a: fa, b: fb))`.** Re-home the proven race logic (winner
|
||||
scan, deregister-all-on-wake, structured cancel+join of losers) from `sched.race(*Task tuple)` onto
|
||||
`*Future` handles + the `Io` protocol. The type-level machinery ports UNCHANGED — `RaceResult($T)`,
|
||||
`make_variant`, the tuple reflection (GAP 1/2, all landed) — only the runtime swaps `*Task`→`*Future`
|
||||
and `suspend_self`→`suspend_raw`/`ready`. Cancellation of losers now uses Phase 3 (their next suspend
|
||||
raises), so `race` returns at WINNER-time, not slowest-loser-time. Lock: re-point 1821 at
|
||||
`context.io.race`; assert winner value + losers' work stopped (not merely flagged).
|
||||
|
||||
5. **Converge — retire the bespoke fiber async API.** Fold `sched.go`/`wait`/`cancel`/`race` into the
|
||||
io.sx layer; `Scheduler` stays as the fiber Io's engine + driver. Migrate 1811–1821 to the
|
||||
`context.io` API. One async stack, all behind the protocol. Update the roadmap/checkpoints.
|
||||
|
||||
## Open decisions (need a call before/within the phase noted)
|
||||
- **D1 (Phase 1) — `impl Io for Scheduler` vs a `FiberIo` wrapper.** Direct impl makes `context.io` BE the
|
||||
scheduler (`xx scheduler` as the Io value, stateful receiver — mirrors the allocator `xx local` rule).
|
||||
A wrapper adds a level but decouples the public Io vtable from the scheduler internals. *Lean: direct
|
||||
impl* (simplest, matches the allocator convention).
|
||||
- **D2 (Phase 1) — virtual vs real clock under the fiber Io.** Tests need the deterministic virtual clock
|
||||
(`clock_ms`); a real deployment wants `time.mono_ms`. Thread it as a Scheduler mode, or two Io impls
|
||||
(`FiberIo` virtual-clock for tests, real-clock for prod). *Lean: a `clock: enum { virtual; real }` field
|
||||
so one impl serves both; tests pin `.virtual`.*
|
||||
- **D3 (Phase 2) — `Future(void)` (issue 0150 SIGTRAP).** A `void`-result task can't build `Future(void)`
|
||||
today. Defer (race/async target non-void), or fix the `void` struct-field path. *Lean: defer, gate with
|
||||
a diagnostic.*
|
||||
- **D4 (Phase 3) — where the cancel flag lives.** The `Future` already has `canceled: Atomic(bool)`; the
|
||||
fiber needs to reach it from `suspend_raw`. Give `Fiber` a `*Atomic(bool)` back-ref to its future's flag
|
||||
(set at `spawn_raw`), so `suspend_raw` consults it with no per-suspend lookup. *Lean: back-ref pointer.*
|
||||
|
||||
## Validation (every phase)
|
||||
- `zig build && zig build test` green (full corpus).
|
||||
- New/changed `18xx` examples byte-identical on aarch64-macOS host AND aarch64-linux container
|
||||
(deterministic virtual clock).
|
||||
- Adversarial review of each phase (worker + read-only reviewer), per the session workflow.
|
||||
|
||||
## What this supersedes
|
||||
- `sched.sx`'s bespoke `go`/`wait`/`cancel`/`race` (Phase 5 retires them; the proven logic moves onto the
|
||||
protocol). The just-landed `race` (commit `9099735e`) is the reference logic for Phase 4, not the final
|
||||
home.
|
||||
- PLAN-RACE.md's "race on `sched.Scheduler`" framing — this plan moves it onto `context.io` per the
|
||||
roadmap's §A5 / §4.6 design-of-record.
|
||||
142
current/PLAN-METATYPE.md
Normal file
142
current/PLAN-METATYPE.md
Normal file
@@ -0,0 +1,142 @@
|
||||
# PLAN-METATYPE — comptime type metaprogramming (`declare` / `define` + reflection)
|
||||
|
||||
## Goal
|
||||
|
||||
Comptime type metaprogramming with the smallest possible compiler surface:
|
||||
|
||||
- **`declare(name) -> Type`** — mint a NEW empty (undefined) nominal type NAMED
|
||||
`name`, returned as a first-class `Type` handle. The compiler registers the
|
||||
forward type at compile time, so the body can reference it (`*Name`).
|
||||
- **`define(handle, info) -> Type`** — fill a declared handle's body from a
|
||||
`TypeInfo` *value*, and return the handle (so the one-shot form chains).
|
||||
- **`type_info($T) -> TypeInfo`** — reflect a type INTO data (the inverse of
|
||||
`define`'s decode). *Done for enums* (`interp.zig:reflectTypeInfo`,
|
||||
`examples/0619`); struct/tuple widening pending.
|
||||
- **`field_type($T, i) -> Type`** — the i-th field / variant-payload / element
|
||||
type of `$T`. *Done.*
|
||||
|
||||
These four `#builtin`s in `library/modules/std/meta.sx` are the **entire**
|
||||
compiler surface. Every higher-level constructor is **plain sx built over
|
||||
`declare`/`define`** — the compiler knows none of them by name:
|
||||
|
||||
```sx
|
||||
// one-shot (non-recursive): declare + define chained, define returns the handle
|
||||
T :: define(declare("T"), .enum(.{ variants = .[ … ] }));
|
||||
|
||||
// recursive: a ctor fn names the forward type via declare, references it as *Name
|
||||
List :: make_list();
|
||||
make_list :: () -> Type {
|
||||
h := declare("List");
|
||||
return define(h, .enum(.{ variants = .[
|
||||
EnumVariant.{ name = "cons", payload = *List }, // self-reference
|
||||
EnumVariant.{ name = "nil", payload = void } ] }));
|
||||
}
|
||||
|
||||
// type-fns are ordinary sx (channel result types, etc.)
|
||||
RecvResult :: ($T: Type) -> Type {
|
||||
return define(declare("RecvResult"), .enum(.{ variants = .[
|
||||
EnumVariant.{ name = "value", payload = T },
|
||||
EnumVariant.{ name = "closed", payload = void } ] }));
|
||||
}
|
||||
```
|
||||
|
||||
This gates channel result types (`RecvResult($T)`) and `race`'s synthesized
|
||||
tagged-union (design [../design/execution-evolution-roadmap.md](../design/execution-evolution-roadmap.md) §7 step 3), and replaces a would-be `enum($T)` language feature.
|
||||
|
||||
## How it works (the locked design)
|
||||
|
||||
1. **Two comptime interp builtins.** `declare` mints an empty `tagged_union` slot
|
||||
in the type table; `define` decodes the `TypeInfo` value (variant-name strings +
|
||||
payload `Type`-tags) and completes the slot byte-identical to a source enum's
|
||||
`buildEnumInfo` output, so it flows through enum codegen unmodified. The interp
|
||||
mutates the type table via a `mint` handle the host sets (`setMintTable`).
|
||||
2. **No syntactic constructor recognition.** A `::` binding or type-fn body that
|
||||
calls a `Type`-returning fn is **comptime-evaluated** (`evalComptimeType`): the
|
||||
expression runs through the interpreter, the `declare`/`define` builtins mint the
|
||||
type, and the result `type_tag` is bound. `decl.zig` triggers on a non-generic
|
||||
`-> Type` fn call; `instantiateTypeFunction` triggers on a type-fn body that
|
||||
returns a `define(…)` call (or a bodied `-> Type` helper) — see
|
||||
`generic.zig:returnExprMintsType`.
|
||||
3. **Name on `declare`.** `declare("Name")` carries the name as a compile-time
|
||||
string so `preregisterForwardTypes` (in `evalComptimeType`) can register the
|
||||
forward type — and bind it as a type alias — BEFORE the body lowers. That's
|
||||
what makes a `*Name` self-reference resolve (a `Name :: ctor()` decl makes
|
||||
`Name` a const_decl author, so `*Name` resolves through the forward-ALIAS path;
|
||||
the alias binding, not just the table registration, is what satisfies it). The
|
||||
interp's `declare` returns the same slot by name; `define` fills it in place.
|
||||
4. **Nominal identity** rides the existing type-fn mangled-name instantiation cache:
|
||||
`RecvResult(i64)` at two sites memoizes to ONE `TypeId` (the body runs once;
|
||||
`renameNominalType` re-keys the minted type to the mangled name).
|
||||
5. **Comptime-only, JIT-free.** `declare`/`define` are interp ops; reaching them at
|
||||
runtime / emit is a hard error.
|
||||
6. **Undefined-until-defined.** `declare()` mints an undefined slot; *using* it
|
||||
(construct / match / size) before its `define` is a loud diagnostic. A *pointer*
|
||||
to an undefined slot (`*Self`) is fine — that's what self-reference needs.
|
||||
|
||||
## Key code anchors
|
||||
|
||||
- Builtins: `BuiltinId.declare` / `.define` (`src/ir/inst.zig`); lowering to
|
||||
`callBuiltin` (`src/ir/lower/call.zig:tryLowerReflectionCall`); interp exec +
|
||||
`defineEnum` + `decodeVariantElements` (`src/ir/interp.zig`); `mint` field +
|
||||
`setMintTable`.
|
||||
- Comptime evaluation: `evalComptimeType` / `renameNominalType`
|
||||
(`src/ir/lower/comptime.zig`); decl trigger `fnReturnsTypeValue`
|
||||
(`src/ir/lower/decl.zig`); type-fn trigger `returnExprMintsType` +
|
||||
`instantiateTypeFunction` (`src/ir/lower/generic.zig`).
|
||||
- Reflection: `field_type` → `fieldTypeOf` (`src/ir/lower/generic.zig`).
|
||||
- Surface: `library/modules/std/meta.sx` (on-demand import — NOT the prelude, to
|
||||
avoid shifting every `.ir` snapshot).
|
||||
|
||||
## Cadence (IMPASSIBLE)
|
||||
|
||||
No commit may both add a test AND make it pass (xfail-then-green, or a behavior
|
||||
lock). `zig build && zig build test` after every step. Never regenerate snapshots
|
||||
while red. Examples: `06xx` (comptime), `11xx` (diagnostics).
|
||||
|
||||
## Status
|
||||
|
||||
- [x] `declare` / `define` comptime builtins + the `mint` plumbing.
|
||||
- [x] Comptime evaluation of a `Type`-returning `::` RHS and type-fn body
|
||||
(the only triggers; no constructor-name knowledge in the compiler).
|
||||
- [x] Name-in-`TypeInfo`; nominal identity via the instantiation cache.
|
||||
- [x] `field_type` reflection (`examples/0616`).
|
||||
- [x] Examples green on the floor: `0614` (one-shot), `0615` (type-fn identity),
|
||||
`0617` (channel result types).
|
||||
- [x] **Self-reference** — recursive enums via `declare("Name")` + `*Name` in a
|
||||
constructor fn (`preregisterForwardTypes` registers the forward type + alias
|
||||
before the body lowers). `examples/0618` (recursive `*List`: construct, match
|
||||
through the pointer, recursive traversal). Mutual recursion / by-value-self-ref
|
||||
rejection fall out of the same mechanism (F5 adds the loud by-value check).
|
||||
- [x] **`make_enum(name, variants: []EnumVariant)`** — the general enum constructor
|
||||
over a COMPUTED (value, non-literal) variant list. Pure sx in `meta.sx`;
|
||||
exercises `define` decoding a value-arg slice. `examples/0620` (array-literal
|
||||
local) / `0624` (generic builder).
|
||||
- [x] **Comptime slice over a non-string aggregate** — `arr[lo..hi]` over an array
|
||||
yields a real slice value at comptime (`base_ty` threaded onto `Subslice`;
|
||||
open-ended `hi` folded to the array's static length; `subsliceElements`).
|
||||
`examples/0621`.
|
||||
- [x] **`type_info($T) -> TypeInfo`** — reflect `enum`/`tagged_union`/`struct`/`tuple`
|
||||
INTO a value (inverse of `define`'s decode); `define` decodes all three back
|
||||
(`defineEnum`/`defineStruct`/`defineTuple`, dispatched on the TypeInfo tag).
|
||||
Round-trips: `examples/0619` (enum) / `0622` (struct) / `0623` (tuple). The
|
||||
reflect/construct triad is complete.
|
||||
- [x] **Generic type-fn body locals** — a generic `($T) -> Type` comptime-evaluates
|
||||
its FULL body (prelude statements + return), so a local before the return
|
||||
resolves (`createComptimeFunctionWithPrelude` / `evalComptimeTypeBody`).
|
||||
`examples/0624`.
|
||||
- [x] **Validation + loud diagnostics** — by-value self-reference (`checkInfiniteSize`,
|
||||
source `1178` + constructed `1182`; issue 0139), duplicate variant/field names
|
||||
(`1180`), `declare()` never `define()`d (`1181`, was a `verifySizes` panic),
|
||||
and the 0140 bail-surfacing (`1179`). use-before-define is subsumed by these
|
||||
(no new check needed). Validation story COMPLETE.
|
||||
- [ ] **Comptime `List` growth** (issue 0141, DEFERRED) — `List(T).append` at
|
||||
comptime bails (two layers: null comptime allocator at scanDecls + `*T`
|
||||
slot_ptr `struct_get`). Non-blocking; array-literal locals cover the use case.
|
||||
|
||||
## Risks / watch
|
||||
|
||||
- **Self-ref timing** — `define` for the two-statement form must complete before any
|
||||
code uses the type's layout; a use-before-define must be a loud diagnostic, not a
|
||||
silent empty enum.
|
||||
- Keep `declare`/`define` **comptime-only**: reaching them at runtime is a hard error
|
||||
(emit should bail loudly if one ever leaks into codegen).
|
||||
@@ -17,7 +17,7 @@ E :: error { Neg }
|
||||
const_one :: () -> i64 { return 1; return 99; }
|
||||
|
||||
// dead `return x;` after an unconditional raise (the failable closure shape)
|
||||
always_raise :: (x: i64) -> (i64, !E) { raise error.Neg; return x; }
|
||||
always_raise :: (x: i64) -> i64 !E { raise error.Neg; return x; }
|
||||
|
||||
// guard: a conditional return must still fall through to the trailing return
|
||||
clamp :: (x: i64) -> i64 { if x > 10 { return 10; } return x; }
|
||||
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
@@ -10,7 +10,7 @@ Ctx :: struct {
|
||||
}
|
||||
|
||||
main :: () -> i32 {
|
||||
c : Ctx = .{ on = (f: Fmt) {
|
||||
c : Ctx = .{ on = (f: Fmt) => {
|
||||
n : i64 = xx f;
|
||||
print("cl f = {}\n", n);
|
||||
}};
|
||||
|
||||
@@ -31,7 +31,7 @@ ticks : i32 = 0;
|
||||
|
||||
main :: () -> i32 {
|
||||
h : Holder = .{};
|
||||
h.set(() { ticks += 1; });
|
||||
h.set(() => { ticks += 1; });
|
||||
|
||||
h.call_direct();
|
||||
h.call_hoisted();
|
||||
|
||||
@@ -11,7 +11,7 @@
|
||||
#import "modules/std.sx";
|
||||
|
||||
main :: () {
|
||||
pick := (p: ?i64) -> i64 {
|
||||
pick := (p: ?i64) -> i64 => {
|
||||
if p == null { return -1; }
|
||||
return p; // narrowed inside the lambda body
|
||||
};
|
||||
|
||||
@@ -1,44 +0,0 @@
|
||||
// A captured FAILABLE closure stays failable when CALLED inside a nested
|
||||
// closure body. The free-variable capture analysis must descend into the
|
||||
// error-handling expressions (`catch`, `try`) that the nested closure uses to
|
||||
// consume the captured worker's error channel — otherwise the worker is never
|
||||
// captured into the env, resolves against an empty scope inside the lambda, and
|
||||
// the call types as `unresolved` (so `catch`/`try` reject it).
|
||||
//
|
||||
// Regression (PLAN-IO-UNIFY Phase 3 blocker): the async completion closure
|
||||
// `() { f.value = worker() catch {…} }` captures a `Closure() -> ($R, !)`
|
||||
// worker and consumes its error channel — exactly this shape.
|
||||
#import "modules/std.sx";
|
||||
|
||||
Box :: struct { run: Closure() -> void; }
|
||||
|
||||
// `catch` path: the nested closure absorbs the worker's error.
|
||||
run_catch :: (worker: Closure() -> (i64, !)) {
|
||||
b : Box = ---;
|
||||
b.run = () {
|
||||
v := worker() catch {
|
||||
print("caught\n");
|
||||
return;
|
||||
};
|
||||
print("ok {}\n", v);
|
||||
};
|
||||
b.run();
|
||||
}
|
||||
|
||||
// `try` path: the nested closure is itself failable and propagates.
|
||||
mk_trier :: (worker: Closure() -> (i64, !)) -> Closure() -> (i64, !) {
|
||||
return () -> (i64, !) {
|
||||
v := try worker();
|
||||
v + 100
|
||||
};
|
||||
}
|
||||
|
||||
main :: () -> i64 {
|
||||
run_catch(() -> (i64, !) { 7 }); // ok 7
|
||||
run_catch(() -> (i64, !) { raise error.Bad; }); // caught
|
||||
|
||||
t := mk_trier(() -> (i64, !) { 5 });
|
||||
r := t() catch { return 1; };
|
||||
print("try {}\n", r); // try 105
|
||||
return 0;
|
||||
}
|
||||
@@ -1,41 +0,0 @@
|
||||
// Calling a closure / function-pointer value stored in a STRUCT FIELD
|
||||
// (`box.run(args)`) resolves the call's return type correctly — value returns
|
||||
// marshal properly, and a failable field (`Closure(..) -> (T, !)`) is `try`/
|
||||
// `catch`-able. The call-type resolver mirrors the lowering dispatch: a
|
||||
// closure/fn-ptr field is called directly (and shadows a same-named method).
|
||||
//
|
||||
// Regression (issue 0201): the field-access call path typed such calls as
|
||||
// `unresolved` — value returns came out as garbage, failable returns rejected
|
||||
// `catch`/`try` ("operand has type 'unresolved'").
|
||||
#import "modules/std.sx";
|
||||
|
||||
CB :: struct {
|
||||
add: Closure(i64, i64) -> i64; // closure field, with args
|
||||
fp: (i64) -> i64; // bare function-pointer field
|
||||
work: Closure(i64) -> (i64, !); // failable closure field
|
||||
}
|
||||
|
||||
triple :: (x: i64) -> i64 { return x * 3; }
|
||||
|
||||
// Field call through a `*CB` receiver inside a method, consuming the failable
|
||||
// field's error channel.
|
||||
run_work :: (self: *CB, n: i64) -> i64 {
|
||||
v := self.work(n) catch { return -1; };
|
||||
return v;
|
||||
}
|
||||
|
||||
main :: () -> i64 {
|
||||
b : CB = ---;
|
||||
b.add = (x: i64, y: i64) => x + y;
|
||||
b.fp = triple;
|
||||
b.work = (n: i64) -> (i64, !) {
|
||||
if n < 0 { raise error.Negative; }
|
||||
n * 10
|
||||
};
|
||||
|
||||
print("{}\n", b.add(3, 4)); // 7
|
||||
print("{}\n", b.fp(5)); // 15
|
||||
print("{}\n", run_work(@b, 6)); // 60
|
||||
print("{}\n", run_work(@b, -1)); // -1 (error path)
|
||||
return 0;
|
||||
}
|
||||
File diff suppressed because one or more lines are too long
@@ -1 +0,0 @@
|
||||
0
|
||||
@@ -1,3 +0,0 @@
|
||||
ok 7
|
||||
caught
|
||||
try 105
|
||||
@@ -1 +0,0 @@
|
||||
0
|
||||
@@ -1,4 +0,0 @@
|
||||
7
|
||||
15
|
||||
60
|
||||
-1
|
||||
@@ -1,47 +0,0 @@
|
||||
// Comptime field reflection (`field_count` / `field_name` / `field_type`) over
|
||||
// ALL aggregate kinds — struct, enum, tuple (positional + named), array, vector.
|
||||
//
|
||||
// Regression (issue 0195): `field_count` / `field_name` were broken on tuples
|
||||
// and arrays/vectors. `field_count` silently returned 0 (a missing `.tuple` arm
|
||||
// in the count switches), and `field_name` SEGFAULTED — the LLVM backend built a
|
||||
// zero-length name array for those kinds while sizing the GEP at the (sometimes
|
||||
// non-zero) count, so `field_name(T, i)` indexed past a `[0 x string]` global.
|
||||
// Fixed by driving BOTH the name-array build and the GEP sizing from the one
|
||||
// source of truth (`memberCount`/`memberName`), so they can never diverge again.
|
||||
// A member with no name (positional-tuple / array / vector element) reflects as
|
||||
// the empty string "" — one slot per member, always in-bounds.
|
||||
#import "modules/std.sx";
|
||||
|
||||
S :: struct { a: i64; b: bool; }
|
||||
E :: enum { X; Y; Z; }
|
||||
|
||||
main :: () -> i32 {
|
||||
// struct: named fields
|
||||
print("struct: fc={} fn=({},{}) ft0={}\n",
|
||||
field_count(S), field_name(S, 0), field_name(S, 1), type_name(field_type(S, 0)));
|
||||
|
||||
// enum: variant names
|
||||
print("enum: fc={} fn=({},{},{})\n",
|
||||
field_count(E), field_name(E, 0), field_name(E, 1), field_name(E, 2));
|
||||
|
||||
// positional tuple: element types, no names → ""
|
||||
print("postuple: fc={} ft=({},{}) fn0=[{}]\n",
|
||||
field_count(Tuple(i64, bool)),
|
||||
type_name(field_type(Tuple(i64, bool), 0)), type_name(field_type(Tuple(i64, bool), 1)),
|
||||
field_name(Tuple(i64, bool), 0));
|
||||
|
||||
// named tuple: element labels recovered
|
||||
print("namtuple: fc={} fn=({},{})\n",
|
||||
field_count(Tuple(a: i64, b: bool)),
|
||||
field_name(Tuple(a: i64, b: bool), 0), field_name(Tuple(a: i64, b: bool), 1));
|
||||
|
||||
// array: length-many elements, type known, no names → ""
|
||||
print("array: fc={} ft0={} fn0=[{}]\n",
|
||||
field_count([4]i64), type_name(field_type([4]i64, 0)), field_name([4]i64, 0));
|
||||
|
||||
// vector: same shape as array
|
||||
print("vector: fc={} ft0={} fn0=[{}]\n",
|
||||
field_count(Vector(4, f32)), type_name(field_type(Vector(4, f32), 0)), field_name(Vector(4, f32), 0));
|
||||
|
||||
return 0;
|
||||
}
|
||||
@@ -1,26 +0,0 @@
|
||||
// `pointee($P: Type) -> Type` — comptime reflection for a pointer's target type
|
||||
// (`pointee(*X)` -> `X`). Folds at lower time like `field_type`, so it composes
|
||||
// inside other type-arg slots — e.g. project a generic handle `*Box(A)` to its
|
||||
// payload type `A` via `field_type(pointee(*Box(A)), 0)`. (Foundation for the
|
||||
// `race` result synthesis, which projects `*Task(A)` -> `A`.)
|
||||
#import "modules/std.sx";
|
||||
|
||||
Box :: struct ($R: Type) { value: R; tag: i64; }
|
||||
|
||||
// Project a pointer-to-Box to the Box's payload type: *Box(A) -> A.
|
||||
Payload :: ($P: Type) -> Type { return field_type(pointee(P), 0); }
|
||||
|
||||
main :: () -> i32 {
|
||||
// plain pointer
|
||||
print("pointee(*i64) = {}\n", type_name(pointee(*i64)));
|
||||
print("pointee(*bool) = {}\n", type_name(pointee(*bool)));
|
||||
|
||||
// pointer to a generic struct → the struct type
|
||||
print("pointee(*Box(f64)) field0 = {}\n", type_name(field_type(pointee(*Box(f64)), 0)));
|
||||
|
||||
// composed projection, used as a real type
|
||||
v : Payload(*Box(i64)) = 42;
|
||||
print("Payload(*Box(i64)) value = {}\n", v);
|
||||
|
||||
return 0;
|
||||
}
|
||||
@@ -1,39 +0,0 @@
|
||||
// Int-returning type-query builtins (`field_count` / `size_of` / `align_of`)
|
||||
// fold as comptime CONSTANTS — usable as an `inline for` bound and an array
|
||||
// dimension, exactly like a plain `K :: 3` const. Previously
|
||||
// they evaluated only as runtime values, so `[field_count(S)]T` and
|
||||
// `inline for 0..field_count(S)` were rejected as "not a compile-time integer".
|
||||
// (This is what lets a `($T) -> Type` builder loop `inline for 0..field_count(T)`
|
||||
// to assemble a variant list from a type's members — the `race` result synthesis.)
|
||||
#import "modules/std.sx";
|
||||
|
||||
S :: struct { a: i64; b: bool; c: f64; }
|
||||
E :: enum { X; Y; Z; W; }
|
||||
|
||||
main :: () -> i32 {
|
||||
// field_count as an inline-for bound
|
||||
s := 0;
|
||||
inline for 0..field_count(S) (i) { s = s + i; } // 0+1+2 = 3
|
||||
print("field_count(S) loop sum = {}\n", s);
|
||||
|
||||
// field_count as an array dimension; fill it in a folded loop
|
||||
xs : [field_count(S)]i64 = ---;
|
||||
inline for 0..field_count(S) (i) { xs[i] = i * 10; }
|
||||
print("array[field_count(S)] len = {} xs[2] = {}\n", xs.len, xs[2]);
|
||||
|
||||
// field_count of an enum (4 variants) driving a loop
|
||||
e := 0;
|
||||
inline for 0..field_count(E) (i) { e = e + 1; }
|
||||
print("field_count(E) = {}\n", e);
|
||||
|
||||
// size_of / align_of fold too
|
||||
bytes : [size_of(i64)]u8 = ---;
|
||||
print("size_of(i64) array len = {}\n", bytes.len);
|
||||
print("align_of(f64) = {}\n", align_of(f64));
|
||||
|
||||
// composed const expression as a dim
|
||||
ys : [field_count(S) + 1]i64 = ---;
|
||||
print("[field_count(S) + 1] len = {}\n", ys.len);
|
||||
|
||||
return 0;
|
||||
}
|
||||
@@ -1,48 +0,0 @@
|
||||
// Comptime type-call COMPOSITION: a `($T) -> Type` builder reflects a named
|
||||
// tuple, projects each element type through `pointee` + `field_type`, and mints a
|
||||
// tagged-union whose variant labels mirror the tuple's labels — the shape the
|
||||
// `race` result synthesis needs (`(a: *Task(A), b: *Task(B))` -> `{ a: A; b: B }`).
|
||||
//
|
||||
// Exercises three things that previously failed when the index was an `inline for`
|
||||
// loop var: a type-call RESULT used as (1) a `Type`-typed struct field value
|
||||
// (`payload = field_type(...)`), (2) a nested type-call arg
|
||||
// (`field_type(pointee(field_type(T, i)), 0)`), and a `field_name(T, i)` folded to
|
||||
// a comptime string for a minted variant NAME. All resolve through the same
|
||||
// type-call fold as a literal index would.
|
||||
#import "modules/std.sx";
|
||||
#import "modules/std/meta.sx";
|
||||
|
||||
// Stand-in for a task handle: a pointer to a generic box carrying the result.
|
||||
Box :: struct ($R: Type) { value: R; }
|
||||
|
||||
// Mint a tagged-union mirroring a named tuple of `*Box(..)` handles:
|
||||
// variant name = tuple label, payload = the box's value type (`*Box(A)` -> `A`).
|
||||
ResultOf :: ($T: Type) -> Type {
|
||||
vs : [field_count(T)]EnumVariant = ---;
|
||||
inline for 0..field_count(T) (i) {
|
||||
vs[i] = EnumVariant.{
|
||||
name = field_name(T, i), // folded to a const string
|
||||
payload = field_type(pointee(field_type(T, i)), 0), // *Box(A) -> Box(A) -> A
|
||||
};
|
||||
}
|
||||
return make_enum("ResultOf", vs[0..field_count(T)]);
|
||||
}
|
||||
|
||||
R :: ResultOf(Tuple(a: *Box(i64), b: *Box(bool), c: *Box(f64)));
|
||||
|
||||
use :: (r: R) {
|
||||
if r == {
|
||||
case .a: (v) { print("a (i64) = {}\n", v); }
|
||||
case .b: (v) { print("b (bool) = {}\n", v); }
|
||||
case .c: (v) { print("c (f64) = {}\n", v); }
|
||||
}
|
||||
}
|
||||
|
||||
main :: () -> i32 {
|
||||
use(.a(42));
|
||||
use(.b(true));
|
||||
use(.c(2.5));
|
||||
print("R: variants={} names=({},{},{})\n",
|
||||
field_count(R), field_name(R, 0), field_name(R, 1), field_name(R, 2));
|
||||
return 0;
|
||||
}
|
||||
@@ -1,63 +0,0 @@
|
||||
// `make_variant($E, idx, payload)` (modules/std/meta.sx) — the WRITE side of the
|
||||
// metatype reflection triad: construct a value of a MINTED tagged-union by
|
||||
// VARIANT INDEX, when the variant is chosen at runtime and the union was
|
||||
// synthesized at comptime (so its labels can't be a literal `.label(…)`). This
|
||||
// is the shape the `race` result uses: an `inline for 0..N (i)` arm builds the
|
||||
// i-th variant of a synthesized result carrying the winner's value.
|
||||
//
|
||||
// Covers heterogeneous + COMPLEX payloads (multi-field struct, string fat
|
||||
// pointer, a larger struct, scalar) — make_variant zeroes the value then writes
|
||||
// the i64 tag @0 and the payload @ size_of(i64), so payloads of any size/shape
|
||||
// round-trip. Built with the natural early-return-per-arm pattern (a `return`
|
||||
// inside an `inline if` inside an `inline for`).
|
||||
#import "modules/std.sx";
|
||||
#import "modules/std/meta.sx";
|
||||
|
||||
Vec3 :: struct { x: f64; y: f64; z: f64; } // 24 bytes
|
||||
Big :: struct { a: i64; b: i64; c: i64; d: i64; tag: bool; } // 40 bytes (largest payload)
|
||||
|
||||
// A synthesized 4-variant tagged-union with complex, differently-sized payloads.
|
||||
R :: make_enum("R", .[
|
||||
EnumVariant.{ name = "v", payload = Vec3 },
|
||||
EnumVariant.{ name = "s", payload = string },
|
||||
EnumVariant.{ name = "big", payload = Big },
|
||||
EnumVariant.{ name = "n", payload = i64 },
|
||||
]);
|
||||
|
||||
// Build the variant chosen by a RUNTIME index, in the matching unrolled arm —
|
||||
// each arm's payload type differs. The `return` inside the `inline if` inside the
|
||||
// `inline for` is the pattern make_variant exists to enable.
|
||||
pick :: (idx: i64, vv: Vec3, sv: string, bv: Big, nv: i64) -> R {
|
||||
inline for 0..field_count(R) (i) {
|
||||
if idx == i {
|
||||
// Comptime match on the loop cursor `i` selects the arm whose payload
|
||||
// type matches variant `i` — cleaner than nested `inline if`/`else`.
|
||||
inline if i == {
|
||||
case 0: { return make_variant(R, i, vv); }
|
||||
case 1: { return make_variant(R, i, sv); }
|
||||
case 2: { return make_variant(R, i, bv); }
|
||||
else: { return make_variant(R, i, nv); }
|
||||
}
|
||||
}
|
||||
}
|
||||
return make_variant(R, 3, -1); // unreachable for a valid idx
|
||||
}
|
||||
|
||||
show :: (r: R) {
|
||||
if r == {
|
||||
case .v: (p) { print("v = {} {} {}\n", p.x, p.y, p.z); }
|
||||
case .s: (p) { print("s = {}\n", p); }
|
||||
case .big: (p) { print("big = {} {} {} {} {}\n", p.a, p.b, p.c, p.d, p.tag); }
|
||||
case .n: (p) { print("n = {}\n", p); }
|
||||
}
|
||||
}
|
||||
|
||||
main :: () -> i32 {
|
||||
v :: Vec3.{ x = 1.5, y = 2.5, z = 3.5 };
|
||||
b :: Big.{ a = 10, b = 20, c = 30, d = 40, tag = true };
|
||||
show(pick(0, v, "hello", b, 99));
|
||||
show(pick(1, v, "hello", b, 99));
|
||||
show(pick(2, v, "hello", b, 99));
|
||||
show(pick(3, v, "hello", b, 99));
|
||||
return 0;
|
||||
}
|
||||
@@ -1,39 +0,0 @@
|
||||
// Regression: a `return` inside an `inline if` (a comptime-folded branch),
|
||||
// itself inside an `inline for`, must NOT make the compiler drop the function's
|
||||
// trailing statements. The `inline if`/`case` branch sets a "block terminated"
|
||||
// flag when its taken arm returns; that flag used to leak past the enclosing
|
||||
// runtime `if`'s merge block, so the trailing `return -1` was skipped and the
|
||||
// function was wrongly rejected as "produces no value". Now the runtime-`if`
|
||||
// merge resets the flag to the merge's actual reachability.
|
||||
#import "modules/std.sx";
|
||||
|
||||
// nested inline-if/else with returns, inside an inline-for, under a runtime if:
|
||||
classify :: (idx: i64) -> i64 {
|
||||
inline for 0..3 (i) {
|
||||
if idx == i {
|
||||
inline if i == 0 { return 100; }
|
||||
else { inline if i == 1 { return 200; } else { return 300; } }
|
||||
}
|
||||
}
|
||||
return -1; // trailing statement — must still be emitted (idx out of range)
|
||||
}
|
||||
|
||||
// the comptime `case` match form, also with per-arm returns:
|
||||
tag :: (idx: i64) -> i64 {
|
||||
inline for 0..3 (i) {
|
||||
if idx == i {
|
||||
inline if i == {
|
||||
case 0: { return 10; }
|
||||
case 1: { return 20; }
|
||||
else: { return 30; }
|
||||
}
|
||||
}
|
||||
}
|
||||
return -1;
|
||||
}
|
||||
|
||||
main :: () -> i32 {
|
||||
print("classify: {} {} {} {}\n", classify(0), classify(1), classify(2), classify(9));
|
||||
print("tag: {} {} {} {}\n", tag(0), tag(1), tag(2), tag(9));
|
||||
return 0;
|
||||
}
|
||||
@@ -1,40 +0,0 @@
|
||||
// Comptime-cursor indexing of a named-tuple VALUE: `tup[i]` where `i` is an
|
||||
// `inline for` cursor (or a literal) reads the i-th tuple field with its
|
||||
// CONCRETE type — not a type-erased `Any`. This is the read side `race` needs:
|
||||
// pull the i-th `*Task(T_i)` handle out of a named-tuple param keeping its real
|
||||
// type, so `field`/method access on it resolves. A tuple's elements are
|
||||
// heterogeneous, so there is no runtime element-indexing op — a comptime index
|
||||
// lowers exactly like the `.N` field-access path (a `structGet`). A runtime
|
||||
// index into a tuple value remains an error (no single element type).
|
||||
//
|
||||
// (GAP 1 of PLAN-RACE.)
|
||||
#import "modules/std.sx";
|
||||
|
||||
Box :: struct ($R: Type) { value: R; }
|
||||
|
||||
// Read each handle with its concrete type via the `inline for` cursor.
|
||||
show_all :: (tup: $T) {
|
||||
inline for 0..field_count(T) (i) {
|
||||
h := tup[i]; // concrete `*Box(T_i)`, not `Any`
|
||||
print("[{}] = {}\n", i, h.value); // field access resolves
|
||||
}
|
||||
}
|
||||
|
||||
// Literal-index form, positional tuple.
|
||||
first_two :: (tup: $T) -> i64 {
|
||||
a := tup[0];
|
||||
b := tup[1];
|
||||
return a.value + b.value;
|
||||
}
|
||||
|
||||
main :: () -> i32 {
|
||||
ba : Box(i64) = .{ value = 7 };
|
||||
bb : Box(bool) = .{ value = true };
|
||||
bc : Box(f64) = .{ value = 2.5 };
|
||||
show_all(.(a = @ba, b = @bb, c = @bc)); // named tuple of *Box(..)
|
||||
|
||||
p0 : Box(i64) = .{ value = 10 };
|
||||
p1 : Box(i64) = .{ value = 32 };
|
||||
print("sum = {}\n", first_two(.(@p0, @p1))); // positional, literal index
|
||||
return 0;
|
||||
}
|
||||
@@ -1,35 +0,0 @@
|
||||
// Comptime-cursor tuple-element L-VALUES: writing a named-tuple element by a
|
||||
// comptime-constant index — the store/address-of siblings of 0652's read path.
|
||||
// A tuple is heterogeneous, so each element L-value is a typed `structGep` of the
|
||||
// i-th field (not a uniform `index_gep`): `tup[i] = v` (direct store), a field
|
||||
// store through an element pointer (`tup[i].f = v`), and `@tup[i]` (address-of).
|
||||
// These are what the `race` runtime needs to register a waiter on the i-th task
|
||||
// handle (`tasks[i].waiter = …`). An out-of-range comptime index is a loud
|
||||
// compile error on every one of these paths (no silent `ptrTo(unresolved)` panic).
|
||||
#import "modules/std.sx";
|
||||
|
||||
Box :: struct ($R: Type) { value: R; }
|
||||
|
||||
main :: () -> i32 {
|
||||
// Direct element store by literal index.
|
||||
t := .(a = 1, b = 2, c = 3);
|
||||
t[0] = 100;
|
||||
t[2] = 300;
|
||||
print("t = ({}, {}, {})\n", t.a, t.b, t.c);
|
||||
|
||||
// Address-of an element, write through the pointer.
|
||||
p := @t[1];
|
||||
p.* = 200;
|
||||
print("t.b via @t[1] = {}\n", t.b);
|
||||
|
||||
// Field store THROUGH an element pointer — `tup[i].field = v` — the exact
|
||||
// L-value shape `race` uses to register a waiter (`tasks[i].waiter = …`): the
|
||||
// i-th element is a `*Box`, and `.value` writes through it to the pointee.
|
||||
ba : Box(i64) = .{ value = 0 };
|
||||
bb : Box(bool) = .{ value = false };
|
||||
handles := .(x = @ba, y = @bb);
|
||||
handles[0].value = 7;
|
||||
handles[1].value = true;
|
||||
print("ba.value = {}, bb.value = {}\n", ba.value, bb.value);
|
||||
return 0;
|
||||
}
|
||||
@@ -1,27 +0,0 @@
|
||||
// Comparing an `Any` against a concrete value (a MIXED `Any == <concrete>`, in
|
||||
// either operand order) compares the boxed value words — the same value-identity
|
||||
// the both-`Any` comparison uses. Boxing the concrete side first keeps the
|
||||
// operands shape-compatible.
|
||||
//
|
||||
// Regression (issue 0199): a mixed `Any == <concrete>` fell through to a plain
|
||||
// `icmp` on a 16-byte `{tag, value}` aggregate vs a scalar, aborting the LLVM
|
||||
// verifier ("Both operands to ICmp are not of the same type"). The both-`Any`
|
||||
// form already worked; this extends it to one-sided `Any` comparisons.
|
||||
|
||||
#import "modules/std.sx";
|
||||
|
||||
main :: () -> i64 {
|
||||
x : Any = 5;
|
||||
print("{}\n", x == 5); // true
|
||||
print("{}\n", x == 6); // false
|
||||
print("{}\n", x != 6); // true
|
||||
print("{}\n", 5 == x); // true (concrete on the left)
|
||||
|
||||
b : Any = true;
|
||||
print("{}\n", b == true); // true
|
||||
print("{}\n", b == false); // false
|
||||
|
||||
y : Any = 5;
|
||||
print("{}\n", x == y); // true (both Any — unchanged)
|
||||
return 0;
|
||||
}
|
||||
@@ -1,6 +0,0 @@
|
||||
struct: fc=2 fn=(a,b) ft0=i64
|
||||
enum: fc=3 fn=(X,Y,Z)
|
||||
postuple: fc=2 ft=(i64,bool) fn0=[]
|
||||
namtuple: fc=2 fn=(a,b)
|
||||
array: fc=4 ft0=i64 fn0=[]
|
||||
vector: fc=4 ft0=f32 fn0=[]
|
||||
@@ -1 +0,0 @@
|
||||
0
|
||||
@@ -1,4 +0,0 @@
|
||||
pointee(*i64) = i64
|
||||
pointee(*bool) = bool
|
||||
pointee(*Box(f64)) field0 = f64
|
||||
Payload(*Box(i64)) value = 42
|
||||
@@ -1 +0,0 @@
|
||||
0
|
||||
@@ -1,6 +0,0 @@
|
||||
field_count(S) loop sum = 3
|
||||
array[field_count(S)] len = 3 xs[2] = 20
|
||||
field_count(E) = 4
|
||||
size_of(i64) array len = 8
|
||||
align_of(f64) = 8
|
||||
[field_count(S) + 1] len = 4
|
||||
@@ -1 +0,0 @@
|
||||
0
|
||||
@@ -1,4 +0,0 @@
|
||||
a (i64) = 42
|
||||
b (bool) = true
|
||||
c (f64) = 2.500000
|
||||
R: variants=3 names=(a,b,c)
|
||||
@@ -1 +0,0 @@
|
||||
0
|
||||
@@ -1,4 +0,0 @@
|
||||
v = 1.500000 2.500000 3.500000
|
||||
s = hello
|
||||
big = 10 20 30 40 true
|
||||
n = 99
|
||||
@@ -1 +0,0 @@
|
||||
0
|
||||
@@ -1,2 +0,0 @@
|
||||
classify: 100 200 300 -1
|
||||
tag: 10 20 30 -1
|
||||
@@ -1 +0,0 @@
|
||||
0
|
||||
@@ -1,4 +0,0 @@
|
||||
[0] = 7
|
||||
[1] = true
|
||||
[2] = 2.500000
|
||||
sum = 42
|
||||
@@ -1 +0,0 @@
|
||||
0
|
||||
@@ -1,3 +0,0 @@
|
||||
t = (100, 2, 300)
|
||||
t.b via @t[1] = 200
|
||||
ba.value = 7, bb.value = true
|
||||
@@ -1 +0,0 @@
|
||||
0
|
||||
@@ -1,7 +0,0 @@
|
||||
true
|
||||
false
|
||||
true
|
||||
true
|
||||
true
|
||||
false
|
||||
true
|
||||
@@ -1,31 +1,28 @@
|
||||
// B1.2 / B2 — the async ergonomic layer over the `Io` capability, blocking
|
||||
// default. `context.io.async(worker)` submits a NULLARY `worker: Closure() -> $R`
|
||||
// and returns a `*Future($R)` handle; under the blocking `CBlockingIo` the worker
|
||||
// runs to completion inline, so the Future is born `.ready`. `f.await()` yields
|
||||
// the result (a value-failable `($R, !IoErr)`, handled with `or`).
|
||||
// `context.io.now_ms()` reads the clock through the same capability.
|
||||
// B1.2 — the async ergonomic layer over the `Io` capability, blocking
|
||||
// default. `context.io.async(worker, ..args)` runs the worker to completion
|
||||
// inline and returns a `.ready` Future($R); `f.await()` yields the result
|
||||
// (a value-failable `($R, !IoErr)`, handled with `or`). `context.io.now_ms()`
|
||||
// reads the monotonic clock through the same capability.
|
||||
//
|
||||
// Worker form: a nullary failable lambda capturing any inputs at the CALL SITE
|
||||
// (`() -> (i64, !) => compute(a, b)`) — the colorblind shape that also works when
|
||||
// the worker is deferred onto a fiber (a captured variadic pack can't cross the
|
||||
// fiber boundary).
|
||||
// Worker form: a lambda whose params are annotated at the call site
|
||||
// (`(a: i64, b: i64) -> i64 => …`); `..args` forwards the call-site
|
||||
// arguments to it.
|
||||
#import "modules/std.sx";
|
||||
|
||||
main :: () {
|
||||
// Inputs captured at the call site. The worker is FAILABLE
|
||||
// (`Closure() -> ($R, !)`) — the unified Phase 3 shape; a body that never
|
||||
// raises is a degenerate failable that always succeeds.
|
||||
s := context.io.async(() -> (i64, !) => 40 + 2);
|
||||
// Homogeneous args.
|
||||
s := context.io.async((a: i64, b: i64) -> i64 => a + b, 40, 2);
|
||||
print("sum: {}\n", s.await() or { -1 });
|
||||
|
||||
d := context.io.async(() -> (i64, !) => 21 * 2);
|
||||
// Single arg.
|
||||
d := context.io.async((x: i64) -> i64 => x * 2, 21);
|
||||
print("double: {}\n", d.await() or { -1 });
|
||||
|
||||
// A worker that closes over a local.
|
||||
base := 42;
|
||||
n := context.io.async(() -> (i64, !) => base);
|
||||
// Nullary worker — the variadic `async` binds an empty pack, so no separate
|
||||
// `async_void` entry is needed.
|
||||
n := context.io.async(() -> i64 => 42);
|
||||
print("nullary: {}\n", n.await() or { -1 });
|
||||
|
||||
// The Io capability also carries a clock.
|
||||
// The Io capability also carries a monotonic clock.
|
||||
if context.io.now_ms() >= 0 { print("clock ok\n"); }
|
||||
}
|
||||
|
||||
@@ -6,13 +6,12 @@
|
||||
#import "modules/std.sx";
|
||||
|
||||
main :: () {
|
||||
// Not canceled → await yields the value. The worker is FAILABLE
|
||||
// (`Closure() -> ($R, !)`) — the unified Phase 3 shape.
|
||||
ok := context.io.async(() -> (i64, !) => 7);
|
||||
// Not canceled → await yields the value.
|
||||
ok := context.io.async((n: i64) -> i64 => n, 7);
|
||||
print("ok: {}\n", ok.await() or { -1 });
|
||||
|
||||
// Canceled → await raises .Canceled → the `or` default is taken.
|
||||
c := context.io.async(() -> (i64, !) => 7);
|
||||
c := context.io.async((n: i64) -> i64 => n, 7);
|
||||
c.cancel();
|
||||
print("canceled: {}\n", c.await() or { -99 });
|
||||
}
|
||||
|
||||
@@ -46,7 +46,7 @@ main :: () -> i64 {
|
||||
// Three DIFFERENT fiber bodies (distinct captured ids), interleaving via
|
||||
// yield_now. Each appends its id once per round for 3 rounds.
|
||||
spawn_worker :: (ps: *sched.Scheduler, psh: *Shared, my_id: i64) {
|
||||
ps.spawn(() {
|
||||
ps.spawn(() => {
|
||||
r := 0;
|
||||
while r < 3 {
|
||||
append(psh, my_id);
|
||||
|
||||
@@ -36,7 +36,7 @@ main :: () -> i64 {
|
||||
// Fiber A: record 10, park, then (after wake) record 11. Store A's handle in
|
||||
// the shared state so B can wake it.
|
||||
mk_a :: (ps: *sched.Scheduler, psh: *Sh) {
|
||||
psh.parked = ps.spawn(() {
|
||||
psh.parked = ps.spawn(() => {
|
||||
rec(psh, 10);
|
||||
ps.suspend_self();
|
||||
rec(psh, 11);
|
||||
@@ -45,7 +45,7 @@ main :: () -> i64 {
|
||||
// Fiber B: record 20, wake A (legit) + a spurious second wake (safe no-op),
|
||||
// record 21.
|
||||
mk_b :: (ps: *sched.Scheduler, psh: *Sh) {
|
||||
ps.spawn(() {
|
||||
ps.spawn(() => {
|
||||
rec(psh, 20);
|
||||
ps.wake(psh.parked); // legitimate: A is parked
|
||||
ps.wake(psh.parked); // spurious: A is now .ready/queued — must no-op
|
||||
|
||||
@@ -1,24 +1,23 @@
|
||||
// Stream B2 — the SUSPENDING `context.io.async` layer over the M:1 fiber
|
||||
// scheduler (PLAN-IO-UNIFY: the unified async stack — the bespoke `go`/`wait` was
|
||||
// retired in Phase 5). In contrast with 1805's `context.io.async` UNDER THE
|
||||
// BLOCKING `Io` (which runs each worker INLINE to completion — no interleaving),
|
||||
// here the scheduler is installed as `context.io`, so `context.io.async(work)`
|
||||
// runs `work` as a REAL fiber and `await()` SUSPENDS the caller until it finishes
|
||||
// — a worker that yields mid-body lets a sibling run first (cooperative
|
||||
// interleaving).
|
||||
// Stream B1 (fibers) B1.4a — a truly-SUSPENDING fiber-task async layer
|
||||
// (`go` / `wait` / `cancel`) over the M:1 scheduler, in pure sx. In contrast
|
||||
// with 1805's `context.io.async` (which runs each worker INLINE to completion
|
||||
// before returning a `.ready` future — no interleaving), here `s.go(work)` runs
|
||||
// `work` as a REAL fiber and `t.wait()` SUSPENDS the caller until that fiber
|
||||
// finishes, so a task that yields mid-body lets a sibling task run before the
|
||||
// first completes — genuine cooperative interleaving.
|
||||
//
|
||||
// `work` is a NULLARY worker: any inputs are captured in the lambda at the call
|
||||
// `work` is a NULLARY thunk: any inputs are captured in the lambda at the call
|
||||
// site (no `..args` pack crosses the fiber boundary — that would hit issue 0156
|
||||
// Part 2). Outputs flow OUT through pointers captured in the worker (the shared
|
||||
// Part 2). Outputs flow OUT through pointers captured in the thunk (the shared
|
||||
// `Log` struct), since closure capture-by-value does not write back.
|
||||
//
|
||||
// What this proves:
|
||||
// - REAL suspend + interleave: worker A records 1, YIELDS; worker B then records
|
||||
// 2 and completes; A resumes, records 3, completes → interleave order 1 2 3.
|
||||
// - awaited VALUES: A returns 42, B returns 100 (recorded after both awaits).
|
||||
// - REAL suspend + interleave: task A records 1, YIELDS; task B then records 2
|
||||
// and completes; A resumes, records 3, completes → interleave order 1 2 3.
|
||||
// - awaited VALUES: A returns 42, B returns 100 (recorded after both waits).
|
||||
// → sequence: 1 2 3 42 100.
|
||||
// - cancel rides the `!` channel (model (a), like 1806): a canceled worker's
|
||||
// `await()` raises `.Canceled`, taken by the `or` default → -99.
|
||||
// - cancel rides the `!` channel (model (a), like 1806): a canceled task's
|
||||
// `wait()` raises `.Canceled`, taken by the `or` default → -99.
|
||||
//
|
||||
// `wait` must run inside a fiber (it parks `self.current`), so the "main task"
|
||||
// is itself a `s.spawn(...)` fiber that drives the two `go` tasks.
|
||||
@@ -39,39 +38,36 @@ main :: () -> i64 {
|
||||
ps := @s;
|
||||
pl := @lg;
|
||||
|
||||
// The coordinator fiber: drives two async workers, awaits both, then exercises
|
||||
// cancel. It runs as a fiber so `await` has a `self.current` to park. The
|
||||
// scheduler is installed as `context.io`, so the unified async layer reaches it.
|
||||
push .{ io = xx s } {
|
||||
ps.spawn(() {
|
||||
// Worker A yields mid-body so B interleaves before A completes.
|
||||
a := context.io.async(() -> (i64, !) {
|
||||
// The "main task" fiber: drives two real tasks, waits both, then exercises
|
||||
// cancel. It runs as a fiber so `wait` has a `self.current` to park.
|
||||
s.spawn(() => {
|
||||
// Task A yields mid-body so B interleaves before A completes.
|
||||
a := ps.go(() -> i64 => {
|
||||
rec(pl, 1);
|
||||
ps.yield_now(); // suspend A; B (already spawned) runs to completion
|
||||
rec(pl, 3);
|
||||
42
|
||||
});
|
||||
// Worker B runs straight through (no yield).
|
||||
b := context.io.async(() -> (i64, !) {
|
||||
// Task B runs straight through (no yield).
|
||||
b := ps.go(() -> i64 => {
|
||||
rec(pl, 2);
|
||||
100
|
||||
});
|
||||
|
||||
// Await both — suspends the coordinator fiber until each completes.
|
||||
va := a.await() or { -1 };
|
||||
vb := b.await() or { -1 };
|
||||
// Wait both — suspends the main-task fiber until each completes.
|
||||
va := a.wait() or { -1 };
|
||||
vb := b.wait() or { -1 };
|
||||
rec(pl, va);
|
||||
rec(pl, vb);
|
||||
|
||||
// Cancel case: cancel before the worker runs; `await` raises .Canceled
|
||||
// off the sticky flag, the `or` default (-99) is taken.
|
||||
c := context.io.async(() -> (i64, !) => 7);
|
||||
// Cancel case: cancel before the worker runs; `wait` raises .Canceled,
|
||||
// the `or` default (-99) is taken.
|
||||
c := ps.go(() -> i64 => 7);
|
||||
c.cancel();
|
||||
rec(pl, c.await() or { -99 });
|
||||
rec(pl, c.wait() or { -99 });
|
||||
});
|
||||
|
||||
ps.run();
|
||||
}
|
||||
s.run();
|
||||
|
||||
// Interleaving + value contract: 1 2 3 42 100, then the cancel default -99.
|
||||
print("sequence:");
|
||||
|
||||
@@ -50,12 +50,12 @@ main :: () -> i64 {
|
||||
pl := @lg;
|
||||
|
||||
// Spawn order A, B, C, D, E — but the WAKE order is set by deadline.
|
||||
ps.spawn(() { ps.sleep(30); rec(pl, 1, ps.now_ms()); }); // A: latest
|
||||
ps.spawn(() { ps.sleep(10); rec(pl, 2, ps.now_ms()); }); // B: earliest
|
||||
ps.spawn(() { ps.sleep(20); rec(pl, 3, ps.now_ms()); }); // C: middle
|
||||
ps.spawn(() => { ps.sleep(30); rec(pl, 1, ps.now_ms()); }); // A: latest
|
||||
ps.spawn(() => { ps.sleep(10); rec(pl, 2, ps.now_ms()); }); // B: earliest
|
||||
ps.spawn(() => { ps.sleep(20); rec(pl, 3, ps.now_ms()); }); // C: middle
|
||||
// Same-deadline FIFO pair: D before E, both at t=15 → wake D then E.
|
||||
ps.spawn(() { ps.sleep(15); rec(pl, 4, ps.now_ms()); }); // D
|
||||
ps.spawn(() { ps.sleep(15); rec(pl, 5, ps.now_ms()); }); // E
|
||||
ps.spawn(() => { ps.sleep(15); rec(pl, 4, ps.now_ms()); }); // D
|
||||
ps.spawn(() => { ps.sleep(15); rec(pl, 5, ps.now_ms()); }); // E
|
||||
|
||||
s.run();
|
||||
|
||||
|
||||
@@ -29,11 +29,11 @@ main :: () -> i64 {
|
||||
|
||||
// Sleeper: arm sleep(100), park; when woken (early), record 1 and finish.
|
||||
mk_sleeper :: (ps: *sched.Scheduler, pst: *S) {
|
||||
pst.sleeper = ps.spawn(() { ps.sleep(100); rec(pst, 1); });
|
||||
pst.sleeper = ps.spawn(() => { ps.sleep(100); rec(pst, 1); });
|
||||
}
|
||||
// Waker: record 2, then wake the sleeper BEFORE its 100ms timer fires.
|
||||
mk_waker :: (ps: *sched.Scheduler, pst: *S) {
|
||||
ps.spawn(() { rec(pst, 2); ps.wake(pst.sleeper); });
|
||||
ps.spawn(() => { rec(pst, 2); ps.wake(pst.sleeper); });
|
||||
}
|
||||
mk_sleeper(ps, pst);
|
||||
mk_waker(ps, pst);
|
||||
|
||||
@@ -56,7 +56,7 @@ main :: () -> i64 {
|
||||
|
||||
// Reader: block on the (empty) pipe until it is readable, then read 3 bytes.
|
||||
mk_reader :: (ps: *sched.Scheduler, pst: *S, rfd: i32) {
|
||||
ps.spawn(() {
|
||||
ps.spawn(() => {
|
||||
ps.block_on_fd(rfd, true); // parks until read_fd is readable
|
||||
n := read(rfd, xx @pst.bytes[0], xx 3);
|
||||
pst.read_n = xx n;
|
||||
@@ -65,7 +65,7 @@ main :: () -> i64 {
|
||||
}
|
||||
// Writer: write 3 bytes ('a','b','c') to the write end.
|
||||
mk_writer :: (ps: *sched.Scheduler, pst: *S, wfd: i32) {
|
||||
ps.spawn(() {
|
||||
ps.spawn(() => {
|
||||
buf : [3]u8 = ---;
|
||||
buf[0] = xx 97; buf[1] = xx 98; buf[2] = xx 99; // 'a' 'b' 'c'
|
||||
write(wfd, xx @buf[0], xx 3);
|
||||
|
||||
@@ -36,26 +36,22 @@ main :: () -> i64 {
|
||||
s := sched.Scheduler.init();
|
||||
ps := @s; pl := @lg;
|
||||
|
||||
// The coordinator runs as a fiber so `await` has a `current` to park. The
|
||||
// scheduler is installed as `context.io`, so the unified async layer
|
||||
// (`context.io.async`/`await`/`sleep`/`now_ms`) reaches it inside the workers.
|
||||
push .{ io = xx s } {
|
||||
ps.spawn(() {
|
||||
// Launch three async workers; each sleeps, logs its completion, returns.
|
||||
a := context.io.async(() -> (i64, !) { try context.io.sleep(30); rec(pl, 1, context.io.now_ms()); 100 });
|
||||
b := context.io.async(() -> (i64, !) { try context.io.sleep(10); rec(pl, 2, context.io.now_ms()); 20 });
|
||||
c := context.io.async(() -> (i64, !) { try context.io.sleep(20); rec(pl, 3, context.io.now_ms()); 3 });
|
||||
// The coordinator runs as a fiber so `wait` has a `current` to park.
|
||||
s.spawn(() => {
|
||||
// Launch three async tasks; each sleeps, logs its completion, returns.
|
||||
a := ps.go(() -> i64 => { ps.sleep(30); rec(pl, 1, ps.now_ms()); 100 });
|
||||
b := ps.go(() -> i64 => { ps.sleep(10); rec(pl, 2, ps.now_ms()); 20 });
|
||||
c := ps.go(() -> i64 => { ps.sleep(20); rec(pl, 3, ps.now_ms()); 3 });
|
||||
|
||||
// Await in SPAWN order; results come back correct regardless.
|
||||
va := a.await() or { -1 };
|
||||
vb := b.await() or { -1 };
|
||||
vc := c.await() or { -1 };
|
||||
va := a.wait() or { -1 };
|
||||
vb := b.wait() or { -1 };
|
||||
vc := c.wait() or { -1 };
|
||||
sum := va + vb + vc;
|
||||
|
||||
rec(pl, 9, sum); // sentinel row: id=9 carries the sum in `at`
|
||||
});
|
||||
ps.run();
|
||||
}
|
||||
s.run();
|
||||
|
||||
print("completion order (id @ virtual-ms):\n");
|
||||
i := 0;
|
||||
|
||||
@@ -8,7 +8,7 @@
|
||||
sched :: #import "modules/std/sched.sx";
|
||||
main :: () -> i64 {
|
||||
s := sched.Scheduler.init(); ps := @s;
|
||||
ps.spawn(() { ps.sleep(10); ps.sleep(-5); }); // -5 → loud abort
|
||||
ps.spawn(() => { ps.sleep(10); ps.sleep(-5); }); // -5 → loud abort
|
||||
s.run();
|
||||
print("unreachable\n");
|
||||
return 0;
|
||||
|
||||
@@ -1,22 +1,18 @@
|
||||
// A `Future` allows ONE awaiter — a second concurrent `await` on the same pending
|
||||
// future would overwrite the single `park` slot, and completion would wake only
|
||||
// A `Task` allows ONE awaiter — a second concurrent `wait` on the same pending
|
||||
// task would overwrite the single `waiter` slot, and completion would wake only
|
||||
// the second, stranding the first forever. Regression (B1.4a review, P1-c): the
|
||||
// guard aborts loudly instead of silently deadlocking. Now over the unified
|
||||
// `context.io` async layer (PLAN-IO-UNIFY Phase 5 — the bespoke `Task`/`wait` is
|
||||
// retired).
|
||||
// guard aborts loudly instead of silently deadlocking.
|
||||
//
|
||||
// aborts (exit 134) after the diagnostic — aarch64-macOS-pinned.
|
||||
#import "modules/std.sx";
|
||||
sched :: #import "modules/std/sched.sx";
|
||||
S :: struct { t: *Future(i64); }
|
||||
S :: struct { t: *sched.Task(i64); }
|
||||
main :: () -> i64 {
|
||||
st : S = ---; st.t = null;
|
||||
s := sched.Scheduler.init(); ps := @s; pst := @st;
|
||||
mkprod :: (ps: *sched.Scheduler, pst: *S) { pst.t = context.io.async(() -> (i64, !) { ps.yield_now(); 42 }); }
|
||||
mkw :: (ps: *sched.Scheduler, pst: *S) { ps.spawn(() { x := pst.t.await() or { -1 }; print("got {}\n", x); }); }
|
||||
push .{ io = xx s } {
|
||||
mkprod :: (ps: *sched.Scheduler, pst: *S) { pst.t = ps.go(() -> i64 => { ps.yield_now(); 42 }); }
|
||||
mkw :: (ps: *sched.Scheduler, pst: *S) { ps.spawn(() => { x := pst.t.wait() or { -1 }; print("got {}\n", x); }); }
|
||||
mkprod(ps, pst); mkw(ps, pst); mkw(ps, pst); // second waiter → loud abort
|
||||
s.run();
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
|
||||
@@ -1,25 +1,25 @@
|
||||
// Stream B1 (fibers) — `Scheduler.deinit` releases the scheduler's owned heap
|
||||
// + fd resources, closing the documented bounded leaks (kq fd / List backings).
|
||||
// Verified by a tracking `GPA`: deinit drives the live allocation count DOWN,
|
||||
// and resets the kqueue fd to -1.
|
||||
// + fd resources, closing the documented bounded leaks (kq fd / heap Tasks /
|
||||
// List backings). Verified by a tracking `GPA`: deinit drives the live
|
||||
// allocation count DOWN, and resets the kqueue fd to -1.
|
||||
//
|
||||
// Scenario (one run that touches every freed resource):
|
||||
// - a SLEEPER fiber `sleep(5)`s → exercises the `timers` List
|
||||
// - a READER fiber `block_on_fd`s a pipe → exercises the kqueue fd + the
|
||||
// `io_waiters` List
|
||||
// - a WRITER fiber writes 3 bytes → makes the pipe readable
|
||||
// After `run()` drains all of it, `deinit()` frees: the `timers` / `io_waiters`
|
||||
// List backings, and CLOSES the kqueue fd (resetting `kq` to -1). The Fibers
|
||||
// were already reaped during `run()`. (The unified `context.io.async` layer's
|
||||
// Futures are NOT scheduler-tracked — they leak with the closure-env residual
|
||||
// below; the bespoke `go`/`Task`/`task_allocs` path was retired in Phase 5.)
|
||||
// - two `go` tasks compute 42 / 7 → exercise the heap `Task`s +
|
||||
// the `task_allocs` List
|
||||
// After `run()` drains all of it, `deinit()` frees: the 2 heap Tasks, the
|
||||
// `timers` / `io_waiters` / `task_allocs` List backings, and CLOSES the kqueue
|
||||
// fd (resetting `kq` to -1). The Fibers were already reaped during `run()`.
|
||||
//
|
||||
// WHAT IT PROVES (the contract; numbers below are the snapshot):
|
||||
// - `freed by deinit: N` — live allocations reclaimed by `deinit` (> 0).
|
||||
// - `live after deinit: 0` — NO residual. Each spawned fiber's body-closure heap
|
||||
// env is reclaimed at reap (`reap_fiber` frees `body.env` via the spawn-time
|
||||
// allocator snapshotted in `dctx`), and `deinit` frees the List backings + kq
|
||||
// fd — so the live count returns to zero.
|
||||
// - `live after deinit` — the RESIDUAL. This is NOT zero and NOT a bug: it is
|
||||
// exactly the documented closure-env leak — one heap env per `spawn`/`go`
|
||||
// that sx cannot free (the runtime has no name for the env pointer). deinit
|
||||
// reclaims everything it CAN; the env residual is a language limitation.
|
||||
// - `kq open after run: 1` then `kq after deinit: -1` — the lazily-opened
|
||||
// kqueue fd was genuinely open after the fd round and is closed by deinit.
|
||||
// - `read: 3 [97 98 99]` — the fd path actually ran (reader blocked, woke via
|
||||
@@ -70,11 +70,11 @@ main :: () -> i64 {
|
||||
ps := @s; pst := @st;
|
||||
|
||||
// SLEEPER — arms a virtual-time timer, then parks.
|
||||
ps.spawn(() { ps.sleep(5); });
|
||||
ps.spawn(() => { ps.sleep(5); });
|
||||
|
||||
// READER — blocks on the empty pipe until kqueue reports it readable.
|
||||
mk_reader :: (ps: *sched.Scheduler, pst: *S, rfd: i32) {
|
||||
ps.spawn(() {
|
||||
ps.spawn(() => {
|
||||
ps.block_on_fd(rfd, true);
|
||||
n := read(rfd, xx @pst.bytes[0], xx 3);
|
||||
pst.read_n = xx n;
|
||||
@@ -83,7 +83,7 @@ main :: () -> i64 {
|
||||
}
|
||||
// WRITER — writes 'a' 'b' 'c', making the pipe readable.
|
||||
mk_writer :: (ps: *sched.Scheduler, wfd: i32) {
|
||||
ps.spawn(() {
|
||||
ps.spawn(() => {
|
||||
buf : [3]u8 = ---;
|
||||
buf[0] = xx 97; buf[1] = xx 98; buf[2] = xx 99;
|
||||
write(wfd, xx @buf[0], xx 3);
|
||||
@@ -92,6 +92,10 @@ main :: () -> i64 {
|
||||
mk_reader(ps, pst, read_fd);
|
||||
mk_writer(ps, write_fd);
|
||||
|
||||
// Two async tasks — heap Tasks tracked for deinit to free.
|
||||
ps.go(() -> i64 => 42);
|
||||
ps.go(() -> i64 => 7);
|
||||
|
||||
ps.run();
|
||||
|
||||
after_run = gpa.alloc_count;
|
||||
|
||||
@@ -1,63 +0,0 @@
|
||||
// Stream B2 — structured first-wins `race` over `context.io` (PLAN-IO-UNIFY
|
||||
// Phase 4). `context.io.race(.(a = fa, b = fb, c = fc))` takes a named tuple of
|
||||
// already-spawned `*Future(..)` handles (from `context.io.async`), SUSPENDS the
|
||||
// calling fiber until the FIRST is `.ready`, and returns a comptime-SYNTHESIZED
|
||||
// tagged-union (`RaceResult`) mirroring the tuple's labels — variant NAME = the
|
||||
// tuple label, payload = that future's result type. Here the three workers return
|
||||
// DIFFERENT types (i64 / bool / f64), so the minted union is
|
||||
// `enum { a: i64; b: bool; c: f64 }` and the winner is matched by label.
|
||||
//
|
||||
// TRUE cancellation (Phase 3): the workers sleep 10/20/30 ms (deterministic
|
||||
// virtual clock), so `a` wins at t=10. The losers `b`/`c` are parked mid-`sleep`
|
||||
// when cancelled; their next `suspend_raw` raises `Canceled` and unwinds the body,
|
||||
// so their POST-SLEEP `rec(...)` NEVER runs and `race` returns at WINNER-time. The
|
||||
// completion log therefore shows ONLY `a @ 10ms`, and the final virtual clock is
|
||||
// 10 — NOT 30 (the old cooperative-join behaviour that let losers run to their
|
||||
// natural end). The losers end `.canceled` with their work stopped.
|
||||
//
|
||||
// aarch64-pinned (the scheduler's per-arch asm + per-OS mmap/event constants):
|
||||
// runs end-to-end on a matching host (macOS + linux), ir-only on a mismatch.
|
||||
#import "modules/std.sx";
|
||||
sched :: #import "modules/std/sched.sx";
|
||||
|
||||
Log :: struct { id: [8]i64; at: [8]i64; n: i64; }
|
||||
rec :: (l: *Log, id: i64, at: i64) { l.id[l.n] = id; l.at[l.n] = at; l.n = l.n + 1; }
|
||||
|
||||
main :: () -> i64 {
|
||||
lg : Log = ---; lg.n = 0;
|
||||
s := sched.Scheduler.init();
|
||||
ps := @s; pl := @lg;
|
||||
|
||||
// The coordinator runs as a fiber so `race` has a `current` to park.
|
||||
push .{ io = xx s } {
|
||||
ps.spawn(() {
|
||||
// Three async workers, DIFFERENT result types and sleep durations.
|
||||
a := context.io.async(() -> (i64, !) { try context.io.sleep(10); rec(pl, 1, context.io.now_ms()); 111 });
|
||||
b := context.io.async(() -> (bool, !) { try context.io.sleep(20); rec(pl, 2, context.io.now_ms()); true });
|
||||
c := context.io.async(() -> (f64, !) { try context.io.sleep(30); rec(pl, 3, context.io.now_ms()); 2.5 });
|
||||
|
||||
// Race them. `a` (sleep 10) wins; `b` and `c` are cancelled — their
|
||||
// post-sleep work never runs (true cancellation).
|
||||
winner := context.io.race(.(a = a, b = b, c = c));
|
||||
if winner == {
|
||||
case .a: (v) { print("winner: a (i64) = {}\n", v); }
|
||||
case .b: (v) { print("winner: b (bool) = {}\n", v); }
|
||||
case .c: (v) { print("winner: c (f64) = {}\n", v); }
|
||||
}
|
||||
|
||||
// The losers were cancelled; their work was stopped at the suspend.
|
||||
print("loser b: canceled={}\n", b.state == .canceled);
|
||||
print("loser c: canceled={}\n", c.state == .canceled);
|
||||
});
|
||||
ps.run();
|
||||
}
|
||||
|
||||
print("completion order (id @ virtual-ms):\n");
|
||||
i := 0;
|
||||
while i < lg.n {
|
||||
print(" task {} @ {}ms\n", lg.id[i], lg.at[i]);
|
||||
i = i + 1;
|
||||
}
|
||||
print("final virtual clock: {}ms\n", s.now_ms());
|
||||
return 0;
|
||||
}
|
||||
@@ -1,35 +0,0 @@
|
||||
// Stream B2/A1 — a fiber INHERITS the dynamic `context` in force when it was
|
||||
// spawned. Previously a fiber body ran under the static `__sx_default_context`
|
||||
// (the `abi(.c)` `fib_dispatch` dropped the implicit context), so a
|
||||
// `push Context { … }` around `spawn` was invisible inside the fiber. Now
|
||||
// `Scheduler.spawn` snapshots `context` into the fiber and `fib_dispatch`
|
||||
// re-pushes it around the body — so a capability installed before `spawn`
|
||||
// (here a marker in `context.data`) is visible to the worker.
|
||||
//
|
||||
// This is the foundation for folding a fiber scheduler behind `context.io`: a
|
||||
// worker's `context.io.*` must resolve to the scheduler that spawned it, not the
|
||||
// blocking default. Behavior-preserving for fibers spawned under the default
|
||||
// context (the snapshot just re-pushes that same default).
|
||||
//
|
||||
// aarch64-pinned (the scheduler's per-arch asm): runs end-to-end on a matching
|
||||
// host (macOS + linux), ir-only on a mismatch.
|
||||
#import "modules/std.sx";
|
||||
sched :: #import "modules/std/sched.sx";
|
||||
|
||||
Marker :: struct { id: i64; }
|
||||
|
||||
main :: () -> i64 {
|
||||
mk := Marker.{ id = 7 };
|
||||
s := sched.Scheduler.init();
|
||||
ps := @s;
|
||||
print("outside: marker id = {}\n", mk.id);
|
||||
push .{ data = xx @mk } {
|
||||
ps.spawn(() {
|
||||
m : *Marker = xx context.data; // inherited from the spawn-time context
|
||||
print("inside fiber: context.data marker id = {}\n", m.id);
|
||||
});
|
||||
ps.run();
|
||||
}
|
||||
print("done\n");
|
||||
return 0;
|
||||
}
|
||||
@@ -1,42 +0,0 @@
|
||||
// Stream B2/A1 — the M:1 fiber scheduler installed AS an `Io` capability vtable,
|
||||
// driven entirely through `context.io`. `impl Io for Scheduler` (sched.sx) folds
|
||||
// the scheduler behind the same `Io` protocol the blocking `CBlockingIo`
|
||||
// implements, so the worker below reaches real suspension/timers through the
|
||||
// PROTOCOL (`context.io.spawn_raw`/`arm_timer`/`suspend_raw`/`now_ms`) rather
|
||||
// than bespoke scheduler methods — the foundation for a colorblind
|
||||
// `async`/`await`/`race` that runs over whichever `Io` is installed.
|
||||
//
|
||||
// Two workers are spawned via `context.io.spawn_raw`; each arms a virtual-time
|
||||
// timer and `suspend_raw`s until it fires. They resume in DEADLINE order (10 then
|
||||
// 20), deterministic on the virtual clock — proving the protocol round-trips
|
||||
// spawn → arm → suspend → ready → resume against the fiber engine. Phase 0 (fibers
|
||||
// inherit the spawn-time context) is what lets the worker's own `context.io`
|
||||
// resolve back to this scheduler.
|
||||
//
|
||||
// aarch64-pinned (the scheduler's per-arch asm): runs end-to-end on a matching
|
||||
// host (macOS + linux), ir-only on a mismatch.
|
||||
#import "modules/std.sx";
|
||||
sched :: #import "modules/std/sched.sx";
|
||||
|
||||
// Worker entry: an sx (*void)->void fn, erased to *void by spawn_raw.
|
||||
sleeper :: (arg: *void) {
|
||||
n : *i64 = xx arg;
|
||||
tok : ParkToken = .{ handle = null };
|
||||
context.io.arm_timer(context.io.now_ms() + n.*, tok);
|
||||
context.io.suspend_raw(@tok) catch {};
|
||||
print("worker(sleep {}) resumed at now_ms = {}\n", n.*, context.io.now_ms());
|
||||
}
|
||||
|
||||
main :: () -> i64 {
|
||||
s := sched.Scheduler.init();
|
||||
ps := @s;
|
||||
d1 : i64 = 20;
|
||||
d2 : i64 = 10;
|
||||
push .{ io = xx s } {
|
||||
context.io.spawn_raw(xx sleeper, xx @d1, .{});
|
||||
context.io.spawn_raw(xx sleeper, xx @d2, .{});
|
||||
ps.run();
|
||||
}
|
||||
print("final clock: {}ms\n", ps.now_ms());
|
||||
return 0;
|
||||
}
|
||||
@@ -1,42 +0,0 @@
|
||||
// Stream B2 — `async`/`await` (the io.sx ergonomic layer) running COLORBLIND over
|
||||
// the fiber `Io` scheduler. The SAME `context.io.async(worker)` that runs inline
|
||||
// under the blocking `CBlockingIo` (1805) here spawns the worker as a real fiber
|
||||
// and returns a PENDING `*Future`; `await` suspends the calling fiber until the
|
||||
// worker completes. No bespoke `go`/`wait` — this is the unified async stack
|
||||
// (io.sx async over the `Io` protocol), reaching the fiber scheduler purely
|
||||
// through `context.io`.
|
||||
//
|
||||
// The completion log makes the deferral visible: the coordinator records 1,2
|
||||
// BEFORE either worker runs (async only SPAWNS them), then `await` parks it while
|
||||
// the workers run (10,20), then it resumes and sums (123). Deterministic.
|
||||
//
|
||||
// aarch64-pinned (the scheduler's per-arch asm): runs end-to-end on a matching
|
||||
// host (macOS + linux), ir-only on a mismatch.
|
||||
#import "modules/std.sx";
|
||||
sched :: #import "modules/std/sched.sx";
|
||||
|
||||
Log :: struct { seq: [8]i64; n: i64; }
|
||||
rec :: (l: *Log, v: i64) { l.seq[l.n] = v; l.n = l.n + 1; }
|
||||
|
||||
main :: () -> i64 {
|
||||
lg : Log = ---; lg.n = 0;
|
||||
s := sched.Scheduler.init();
|
||||
ps := @s; pl := @lg;
|
||||
push .{ io = xx s } {
|
||||
ps.spawn(() {
|
||||
rec(pl, 1); // coordinator starts
|
||||
a := context.io.async(() -> (i64, !) { rec(pl, 10); 100 }); // worker A — deferred
|
||||
b := context.io.async(() -> (i64, !) { rec(pl, 20); 23 }); // worker B — deferred
|
||||
rec(pl, 2); // both spawned, neither has run
|
||||
va := a.await() or { -1 }; // park; A runs, wakes us
|
||||
vb := b.await() or { -1 };
|
||||
rec(pl, va + vb); // 123
|
||||
});
|
||||
ps.run();
|
||||
}
|
||||
print("sequence:");
|
||||
i := 0;
|
||||
while i < lg.n { print(" {}", lg.seq[i]); i = i + 1; }
|
||||
print("\n");
|
||||
return 0;
|
||||
}
|
||||
@@ -1,49 +0,0 @@
|
||||
// Stream B2 — TRUE cancellation (PLAN-IO-UNIFY Phase 3). A `cancel` delivered to
|
||||
// a worker that is PARKED at a suspend point makes the worker ABANDON its body:
|
||||
// the worker's next `suspend_raw` raises `IoErr.Canceled`, which unwinds out
|
||||
// through `try context.io.sleep(..)` and the failable worker, so every line AFTER
|
||||
// the suspend never runs. This is "true cancellation, model (a)" — cancel rides
|
||||
// the `!` channel and stops in-flight work at the next suspend, not merely flags
|
||||
// a result.
|
||||
//
|
||||
// Flow (deterministic, virtual clock): the worker records 1 and parks in
|
||||
// `sleep`; the coordinator (a fiber, so it can `yield`) lets the worker reach its
|
||||
// park, then `cancel`s it. The worker's parked `suspend_raw` is woken and raises
|
||||
// `Canceled` → the post-sleep `rec(pl, 2)` and the `42` return NEVER execute. The
|
||||
// coordinator's `await` raises `Canceled` (sticky flag) → `or` default -99.
|
||||
// Sequence: `1 -99` — the absence of `2` is the proof that the post-suspend work
|
||||
// was truly cancelled.
|
||||
//
|
||||
// aarch64-pinned (the scheduler's per-arch asm): runs end-to-end on a matching
|
||||
// host (macOS + linux, byte-identical under the deterministic virtual clock).
|
||||
#import "modules/std.sx";
|
||||
sched :: #import "modules/std/sched.sx";
|
||||
|
||||
Log :: struct { seq: [8]i64; n: i64; }
|
||||
rec :: (l: *Log, v: i64) { l.seq[l.n] = v; l.n = l.n + 1; }
|
||||
|
||||
main :: () -> i64 {
|
||||
lg : Log = .{ n = 0 };
|
||||
s := sched.Scheduler.init();
|
||||
ps := @s; pl := @lg;
|
||||
push .{ io = xx s } {
|
||||
ps.spawn(() {
|
||||
w := context.io.async(() -> (i64, !) {
|
||||
rec(pl, 1); // worker started
|
||||
try context.io.sleep(10); // park; cancel delivers Canceled HERE
|
||||
rec(pl, 2); // POST-SUSPEND — must NEVER run
|
||||
42
|
||||
});
|
||||
ps.yield_now(); // let the worker run & park in sleep
|
||||
w.cancel(); // cancel while parked → wakes + raises
|
||||
r := w.await() or { -99 }; // await raises Canceled → -99
|
||||
rec(pl, r);
|
||||
});
|
||||
ps.run();
|
||||
}
|
||||
print("seq:");
|
||||
i := 0;
|
||||
while i < lg.n { print(" {}", lg.seq[i]); i = i + 1; }
|
||||
print("\n");
|
||||
return 0;
|
||||
}
|
||||
@@ -1,35 +0,0 @@
|
||||
// Stream B2 — `context.io.race` tolerates a FAILING racer (PLAN-IO-UNIFY Phase 4).
|
||||
// A `race` is first-SUCCESS-wins: a racer that ends `.failed` is simply not a
|
||||
// winner candidate; as long as ANOTHER racer succeeds, `race` returns that winner.
|
||||
// Here `a` raises at t=5 and `b` succeeds (42) at t=10, so `b` wins. The failed
|
||||
// racer keeps its real outcome label (`.failed`) — `race` only cancels still-
|
||||
// in-flight (`.pending`) losers, so it never stomps `a`'s `.failed` to `.canceled`.
|
||||
//
|
||||
// (Regression: an all-FAILING racer set instead bails loudly — "race — all
|
||||
// futures settled without a winner" — rather than dead-locking the scheduler.)
|
||||
//
|
||||
// aarch64-pinned (the scheduler's per-arch asm): runs end-to-end on a matching
|
||||
// host (macOS + linux), ir-only on a mismatch.
|
||||
#import "modules/std.sx";
|
||||
sched :: #import "modules/std/sched.sx";
|
||||
|
||||
main :: () -> i64 {
|
||||
s := sched.Scheduler.init();
|
||||
ps := @s;
|
||||
push .{ io = xx s } {
|
||||
ps.spawn(() {
|
||||
a := context.io.async(() -> (i64, !) { try context.io.sleep(5); raise error.Boom; });
|
||||
b := context.io.async(() -> (i64, !) { try context.io.sleep(10); 42 });
|
||||
winner := context.io.race(.(a = a, b = b));
|
||||
if winner == {
|
||||
case .a: (v) { print("winner: a = {}\n", v); }
|
||||
case .b: (v) { print("winner: b = {}\n", v); }
|
||||
}
|
||||
// The failing loser keeps its real outcome — not stomped to .canceled.
|
||||
print("a: failed={} canceled={}\n", a.state == .failed, a.state == .canceled);
|
||||
});
|
||||
ps.run();
|
||||
}
|
||||
print("final clock: {}ms\n", s.now_ms());
|
||||
return 0;
|
||||
}
|
||||
@@ -1,46 +0,0 @@
|
||||
// The unified `context.io.async` layer reclaims its per-task heap once a future is
|
||||
// AWAITED (PLAN-IO-UNIFY follow-up — closing the documented leaks). Each `async`
|
||||
// allocates: the `Future`, the `ThunkBox`, the completion-closure env, the worker's
|
||||
// env, and the spawn_raw fiber-body env. With ownership wired through, ALL of it is
|
||||
// freed: the box + envs by `sx_run_boxed_closure` the instant the worker completes,
|
||||
// the fiber-body env at fiber reap, and the `Future` by the last of {worker,
|
||||
// `await`} (the two-flag handshake). Verified by a tracking `GPA`: after running +
|
||||
// awaiting three workers and `deinit`, the live-allocation count returns to the
|
||||
// pre-spawn baseline — zero residual.
|
||||
//
|
||||
// (A future that is never awaited — fire-and-forget, or a `race` loser — keeps only
|
||||
// its `Future` struct, since nothing consumes it; that remainder needs a
|
||||
// structured-concurrency scope and is out of scope here.)
|
||||
//
|
||||
// aarch64-pinned (the scheduler's per-arch asm): runs end-to-end on a matching
|
||||
// host (macOS + linux), ir-only on a mismatch.
|
||||
#import "modules/std.sx";
|
||||
sched :: #import "modules/std/sched.sx";
|
||||
|
||||
main :: () -> i64 {
|
||||
sum : i64 = 0; psum := @sum;
|
||||
base : i64 = 0; pbase := @base;
|
||||
after : i64 = 0; pafter := @after;
|
||||
|
||||
gpa := mem.GPA.init();
|
||||
push Context.{ allocator = xx gpa, data = null } {
|
||||
s := sched.Scheduler.init();
|
||||
ps := @s;
|
||||
pbase.* = gpa.alloc_count; // baseline: scheduler is live, no tasks yet
|
||||
push .{ io = xx s, allocator = xx gpa, data = null } {
|
||||
ps.spawn(() {
|
||||
a := context.io.async(() -> (i64, !) { try context.io.sleep(10); 100 });
|
||||
b := context.io.async(() -> (i64, !) { try context.io.sleep(20); 20 });
|
||||
c := context.io.async(() -> (i64, !) { try context.io.sleep(30); 3 });
|
||||
psum.* = (a.await() or 0) + (b.await() or 0) + (c.await() or 0);
|
||||
});
|
||||
ps.run();
|
||||
}
|
||||
s.deinit();
|
||||
pafter.* = gpa.alloc_count; // after run + await-all + deinit
|
||||
}
|
||||
|
||||
print("sum: {}\n", sum);
|
||||
print("residual above baseline: {}\n", after - base); // 0 — every async heap reclaimed
|
||||
return 0;
|
||||
}
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
@@ -1 +1 @@
|
||||
io: await — future already has an awaiter (one awaiter per future in the M:1 model)
|
||||
sched: wait() — task already has a waiter (one awaiter per task in the M:1 model)
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
read: 3 [97 98 99]
|
||||
freed by deinit: 2
|
||||
live after deinit: 0
|
||||
freed by deinit: 5
|
||||
live after deinit: 5
|
||||
kq open after run: true
|
||||
kq after deinit: -1
|
||||
|
||||
@@ -1 +0,0 @@
|
||||
{ "target": "macos" }
|
||||
@@ -1 +0,0 @@
|
||||
0
|
||||
@@ -1,6 +0,0 @@
|
||||
winner: a (i64) = 111
|
||||
loser b: canceled=true
|
||||
loser c: canceled=true
|
||||
completion order (id @ virtual-ms):
|
||||
task 1 @ 10ms
|
||||
final virtual clock: 10ms
|
||||
@@ -1 +0,0 @@
|
||||
{ "target": "macos" }
|
||||
@@ -1 +0,0 @@
|
||||
0
|
||||
@@ -1 +0,0 @@
|
||||
|
||||
@@ -1,3 +0,0 @@
|
||||
outside: marker id = 7
|
||||
inside fiber: context.data marker id = 7
|
||||
done
|
||||
@@ -1 +0,0 @@
|
||||
{ "target": "macos" }
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user