0b4c50b187
Add the COMPILER-API stream to CLAUDE.md's session-start router and a `## ⏯ Resume` block to CHECKPOINT-COMPILER-API.md (next action = sub-step 2.2, read order, build/verify, and the cross-arch snapshot-regen gotcha).
290 lines
19 KiB
Markdown
290 lines
19 KiB
Markdown
# CHECKPOINT-COMPILER-API — comptime `compiler` library (`#library "compiler"` + `abi(.zig) extern`)
|
||
|
||
Companion to the design-of-record
|
||
[../design/comptime-compiler-api.md](../design/comptime-compiler-api.md) (the plan
|
||
+ phased build order live there). This stream supersedes the metatype
|
||
`declare`/`define`/`type_info` `#builtin`s and the `#compiler` struct attribute
|
||
with ONE welded mechanism. Branch: `reify` (off `master`). Update after every step.
|
||
|
||
## ⏯ Resume (fresh session)
|
||
Phase 1 is COMPLETE and committed (`cd5b958`); Phase 2 (full byte-layout weld)
|
||
just started. **Do sub-step 2.2 next** — make `src/backend/llvm/types.zig`'s
|
||
`.@"struct"` case build a welded struct's LLVM type from `compiler_lib.computeWeldPlan`
|
||
(offset-ordered field elements + `[N x i8]` padding) with a build-time
|
||
`LLVMOffsetOfElement == plan offset` + `LLVMABISizeOfType == total_size` assertion;
|
||
cache the plan per TypeId for the GEP sites. The plan math (sub-step 2.1) is done,
|
||
pure, and unit-tested — see `computeWeldPlan` in `src/ir/compiler_lib.zig`. Full
|
||
2.2–2.6 breakdown under **## Next step**. Read order: this file → the design doc →
|
||
`src/ir/compiler_lib.zig`. Build/verify: `zig build && zig build test` (green now).
|
||
|
||
> ⚠ Snapshot gotcha: `zig build test -Dupdate-goldens` on this aarch64 host clobbers
|
||
> cross-arch examples' CI-captured `.stdout` (1228/1231/1639/1651/1657–1660) with
|
||
> host-specific empties. After regenerating, revert those (`git checkout` / `rm`)
|
||
> before committing — they are NOT part of this stream.
|
||
|
||
## Last completed step
|
||
**Phase 2, sub-step 1 — the weld-plan layout math + `StructInfo` registered.**
|
||
The de-risked core of the byte-layout-override ("GEP") engine, pure + unit-tested,
|
||
no emit/interp wiring yet (suite trivially green).
|
||
|
||
Decision (locked 2026-06-17): **full byte-layout weld** — a welded sx struct is
|
||
laid out byte-identically to the bound Zig type (Zig's `@offsetOf`, reordering +
|
||
padding included), so it passes to a Zig handler as raw memory with zero
|
||
marshalling. (The alternative — handlers reading interp `Value` aggregates
|
||
logically, no layout override — was rejected; welded types must also be usable as
|
||
runtime data, and the design wants the literal byte weld.)
|
||
|
||
- Measured: Zig reorders `StructInfo` to `fields`@0, `name`@16, `nominal_id`@20,
|
||
`is_protocol`@24, size 32 — vs sx-natural `name`@0, `fields`@8, … So the override
|
||
is genuinely required (`Field`'s two-u32 natural layout was the easy case).
|
||
- `compiler_lib.zig`: registered `StructInfo` (`weldStruct`, the second
|
||
`bound_types` entry). Added `WeldElement` / `WeldPlan` + `computeWeldPlan(alloc,
|
||
fields, total)` — pure: orders fields by ascending byte offset, inserts padding
|
||
elements for gaps + the alignment tail, and builds the sx-field → LLVM-element
|
||
remap. This is what the LLVM type builder + struct-GEP sites will consume.
|
||
- Unit-tested (`compiler_lib.test.zig`): `Field` → identity plan (2 elems, no pad);
|
||
`StructInfo` → 5 elems `[fields@0, name@16, nominal_id@20, is_protocol@24,
|
||
pad@25..32]`, remap `[1,0,3,2]`.
|
||
- `zig build` + `zig build test` green.
|
||
|
||
### Earlier — Phase 1 polish (comptime-only enforcement)
|
||
**A RUNTIME call to a `fn abi(.zig) extern compiler` is a clean build-gating error
|
||
instead of an undefined-symbol link failure.**
|
||
- `emitCall` (`src/backend/llvm/ops.zig`): when the callee is `compiler_welded`
|
||
AND the ENCLOSING function is not `is_comptime` (i.e. genuine runtime code, not a
|
||
`#run`/`::` initializer wrapper whose LLVM body is dead), print a clear
|
||
"comptime-only … cannot be called at runtime" error and set
|
||
`comptime_failed` (the driver halts before object/JIT emission). The enclosing
|
||
`is_comptime` guard is what keeps the legitimate `#run` use (example 0626) green.
|
||
- Corpus: `examples/1185-diagnostics-weld-fn-runtime-call.sx` (runtime `intern(…)`
|
||
→ clean error, exit 1, no link failure).
|
||
- `zig build` + `zig build test` green (458 unit + 690 corpus).
|
||
|
||
### Earlier — fifth sub-step (host-call bridge)
|
||
**A `fn abi(.zig) extern compiler` dispatches, under the comptime interpreter, to
|
||
its registered Zig handler instead of dlsym.**
|
||
- `compiler_lib.zig`: function registry — `BoundFn { sx_name, handler }`,
|
||
`bound_fns` = `intern(string)->StringId` + `text_of(StringId)->string` (the
|
||
string-pool round-trip), `findFn`, and `FnHandler` (`*Interpreter, []Value ->
|
||
Value`). `intern` mutates via `interp.mint orelse @constCast(&module.types)`
|
||
(the same mutable-table access the metatype mint path uses); `text_of` reads the
|
||
const pool. Imports `interp.zig` (the compiler_hooks↔interp cycle pattern).
|
||
- IR `Function` gained `compiler_welded: bool`. `declareFunction`
|
||
(`src/ir/lower/decl.zig`) sets it via `weldedCompilerFn`, which also VALIDATES:
|
||
the bound lib must be `compiler` and the name must be on the function-export
|
||
list — else a build-gating `.err` (no silent fall-through to dlsym).
|
||
- `interp.call()`: before the dlsym/extern path, a `compiler_welded` function
|
||
routes to `compiler_lib.findFn(name).handler(self, args)` (clean bail off the
|
||
export list).
|
||
- Corpus: `examples/0626-comptime-weld-fn-intern-text-of.sx` (`#run
|
||
text_of(intern("hello, compiler"))` folds to a string constant → prints it);
|
||
`examples/1184-diagnostics-weld-fn-unexported.sx` (unexported welded-fn name →
|
||
build error). `findFn` lookup unit-tested.
|
||
- **Runtime-call rejection is NOT yet clean** — welded fns are comptime-only; a
|
||
RUNTIME call would emit a reference to a non-existent extern symbol → a loud
|
||
LINK error (not silent, but not a tidy diagnostic). The examples call welded fns
|
||
only inside `#run`. A dedicated "comptime-only symbol" emit diagnostic is the
|
||
immediate follow-up.
|
||
- `zig build` + `zig build test` green (458 unit tests + 689 corpus).
|
||
|
||
### Earlier — fourth sub-step (welded-struct layout validation)
|
||
**A `struct abi(.zig) extern compiler { … }` is validated against the binding
|
||
registry as a *header checked against the implementation*.**
|
||
- `compiler_lib.zig`: `validateStructLayout(bt, sx_fields, total)` — pure, returns
|
||
the first `LayoutMismatch` (field count / name / size / total) or null. Plus
|
||
`lib_name = "compiler"` and `SxField`. Unit-tested (faithful `Field` passes;
|
||
each drift flagged as the right variant).
|
||
- `registerStructDecl` (`src/ir/lower/nominal.zig`): for `sd.abi == .zig`,
|
||
`validateWeldedStruct` checks the bound lib is `compiler`, the name is on the
|
||
export list (`findType`), and the sx layout (field names + `typeSizeBytes` +
|
||
total) matches the welded type — emitting a build-gating `.err` (good span into
|
||
the struct body) on any failure. No silent reinterpretation.
|
||
- `#library "compiler"` is the comptime-only internal surface, NOT a dylib —
|
||
`src/main.zig`'s dlopen walker skips it (was emitting a spurious `libcompiler.so`
|
||
load warning).
|
||
- Corpus: `examples/0625-comptime-weld-struct-field.sx` (faithful `Field` welds,
|
||
validates, usable as data → `name=7 ty=3`); `examples/1183-diagnostics-weld-
|
||
struct-field-count.sx` (one-field `Field` → build-gating field-count diagnostic).
|
||
- **Offset-override / GEP emission for non-natural Zig layouts is NOT here** — it
|
||
isn't exercised by `Field` (two u32s = natural layout coincides with the weld).
|
||
It arrives with `StructInfo` in Phase 2 (slices/reordering), where the bound
|
||
offsets actually differ from the sx-natural ones. The validation already checks
|
||
per-field size + total, so a layout drift is caught even before the override
|
||
engine exists.
|
||
- `zig build` + `zig build test` green (456 unit tests + 687 corpus).
|
||
|
||
### Earlier — third sub-step (binding registry)
|
||
**The binding registry (welded-type lookup, layout baked from the real Zig
|
||
type).**
|
||
- New `src/ir/compiler_lib.zig` — the `compiler` library's binding registry, the
|
||
curated safety boundary. `BoundType { sx_name, size, alignment, fields:
|
||
[]FieldLayout{name, offset, size} }`; `weldStruct` bakes the layout from a real
|
||
Zig struct via `@sizeOf`/`@alignOf`/`@offsetOf` at compiler-build time (a
|
||
sx-field-count mismatch is a `@compileError`, never a silent truncation).
|
||
`bound_types` exports `Field` (welded to `types.TypeInfo.StructInfo.Field` —
|
||
two `u32`s); `findType(sx_name) ?*const BoundType` is the lookup the welded-decl
|
||
resolution path will consult (returns null off the export list — clean boundary,
|
||
no silent default).
|
||
- Registered in the barrel (`src/ir/ir.zig`): `compiler_lib` + `compiler_lib_tests`.
|
||
- Tests (`src/ir/compiler_lib.test.zig`): `findType("Field")` equals the real
|
||
`StructInfo.Field` `@sizeOf`/`@alignOf`/`@offsetOf` (8 bytes, two u32s at 0/4);
|
||
an unexported name returns null. Break-verified (a wrong size → suite red,
|
||
named `ir.compiler_lib.test...`).
|
||
- `zig build` + `zig build test` green (454 unit tests).
|
||
|
||
### Earlier — second sub-step (struct-decl parse)
|
||
**`abi(.zig) extern <lib>` PARSES on a STRUCT decl (parse-only, no semantics).**
|
||
- `ast.StructDecl` gained `abi: ABI` + `extern_lib: ?[]const u8` binding fields.
|
||
- `parseStructDecl` (`src/parser.zig`): after `struct` (and the `#compiler`
|
||
check), parse an optional `abi(...)` then optional `extern <lib>` — same slot
|
||
order as fn decls — and thread them onto the node. Ordinary structs are
|
||
unperturbed (`parseOptionalAbi`/`parseOptionalExternExport` no-op when absent).
|
||
- Parser unit tests (`src/parser.test.zig`): `Field :: struct abi(.zig) extern
|
||
compiler { name: StringId; ty: Type; }` parses with `abi == .zig`, `extern_lib
|
||
== "compiler"`, field list intact; a plain struct leaves `abi == .default` /
|
||
`extern_lib == null`. Break-verified (a wrong-sentinel assert turns the suite
|
||
red, confirming the test runs).
|
||
- `zig build` + `zig build test` green.
|
||
|
||
### Earlier — first sub-step (fn decls) + the syntax pivot
|
||
**`abi(.zig) extern <lib>` PARSES on a fn decl (parse-only).** Plus the syntax
|
||
pivot it required.
|
||
|
||
Syntax decision (locked 2026-06-17, supersedes the doc's original
|
||
`extern(.zig) <lib>` single-qualifier form): the ABI/layout selector and the
|
||
linkage keyword are two orthogonal annotations.
|
||
- `abi(.x)` — ABI / calling-convention annotation in the slot **before**
|
||
`extern`/`export`. **Unified replacement for `callconv(...)`, which is removed.**
|
||
`ABI = { default, c, zig, pure }`: `.c` (C ABI), `.zig` (Zig-layout weld → the
|
||
`compiler` library), `.pure` (naked asm), `.default` (unannotated). Can appear
|
||
standalone (no extern) on any fn / fn-type / lambda.
|
||
- `extern <lib>` — linkage keyword + binding source (named library).
|
||
|
||
So a welded binding is `text_of :: (id: StringId) -> string abi(.zig) extern compiler;`.
|
||
|
||
What landed:
|
||
- **AST** (`src/ast.zig`): `CallingConvention` → `ABI { default, c, zig, pure }`;
|
||
the `call_conv` field → `abi: ABI` on `FnDecl` / `Lambda` / `FunctionTypeExpr`.
|
||
- **Lexer/token** (`src/token.zig`, `src/lexer.zig`): `kw_callconv` → `kw_abi`,
|
||
keyword string `"callconv"` → `"abi"`.
|
||
- **Parser** (`src/parser.zig`): `parseOptionalCallConv` → `parseOptionalAbi`
|
||
(parses `abi(.c|.zig|.pure)`); wired in the fn-decl postfix slot (before
|
||
`extern`/`export`), the function-type-expr slot, and the lambda slot;
|
||
`isFunctionDef`/`hasFnBodyAfterArrow` recognise `kw_abi`.
|
||
- **AST→IR map** (`src/ir/type_resolver.zig`, `src/ir/lower/decl.zig`, `sema.zig`,
|
||
`closure.zig`): the AST `.abi == .c` reads kept their C-ABI meaning; the
|
||
function-type resolver maps `.zig`/`.pure` → IR `.default` (no fn-pointer-type
|
||
CC for those decl-level ABIs; neither occurs in a function-TYPE position yet).
|
||
- **CC-mismatch diagnostic** (`src/ir/lower/expr.zig`, `src/sema.zig`): the
|
||
user-facing text `callconv(.c)` → `abi(.c)`.
|
||
- **sx migration**: 52 `.sx` files `callconv(` → `abi(` (all were function-type
|
||
callback annotations — none in the fn-decl postfix slot, so no reordering).
|
||
- **Docs**: `readme.md`, `specs.md`, the design doc, snapshots (0114 / 1104 /
|
||
1200) regenerated for the rename.
|
||
- **Tests**: parser unit tests in `src/parser.test.zig` — `abi(.zig) extern <lib>`
|
||
on a fn decl (asserts `abi == .zig`, `extern_export == .extern_`, `extern_lib ==
|
||
"compiler"`); bare `extern` leaves `abi == .default`; standalone `abi(.c)` /
|
||
`abi(.pure)`. lexer/sema tests updated.
|
||
|
||
`zig build` + `zig build test` green (450/450 unit + 685 corpus).
|
||
|
||
## Current state
|
||
- `compiler :: #library "compiler";` parses + is recognised as the comptime-only
|
||
internal surface (never dlopen'd).
|
||
- `abi(.zig) extern compiler` STRUCTS: layout-validated against the registry
|
||
(faithful → ok; drift → build-gating diagnostic). `Field` welds + usable.
|
||
- `abi(.zig) extern compiler` FUNCTIONS: dispatched under the comptime interp to
|
||
their registered Zig handler (`intern`/`text_of` round-trip works); unexported
|
||
names rejected at declaration. Comptime-only.
|
||
- A RUNTIME call to a welded fn is a clean build-gating error (comptime-only
|
||
enforcement at `emitCall`); the legitimate `#run`/`::` use stays green.
|
||
- The whole Phase 1 foundation (parse → registry → struct-layout validation →
|
||
function host-call bridge → comptime-only enforcement) is in place for the
|
||
two-u32 `Field` case + the two string readers.
|
||
- **Deferred**: offset-override / LLVM byte-offset GEP for non-natural layouts
|
||
(needed by `StructInfo`'s slice field, Phase 2).
|
||
|
||
## Next step — Phase 2 decomposition (byte-layout weld for `StructInfo`)
|
||
|
||
The weld plan (sub-step 1) is the pure layout math. The remaining sub-steps wire
|
||
it through emit + interp so a non-natural welded struct actually works. Each must
|
||
stay green; do ONE per session (the IR-stream split rule).
|
||
|
||
- **2.2 — LLVM type honours the plan.** In `src/backend/llvm/types.zig` `.@"struct"`
|
||
case: if the struct's name is in `compiler_lib.findType`, build the LLVM struct
|
||
from `computeWeldPlan` — elements in offset order (real field types + `[N x i8]`
|
||
padding), and **assert** `LLVMOffsetOfElement(elem) == plan.elements[e].offset`
|
||
for every field element + `LLVMABISizeOfType == total_size` (the build-time
|
||
layout-equality assertion; mismatch = a loud emit failure). Cache the plan per
|
||
TypeId (the GEP sites + interp need the remap). Prove: a welded struct's LLVM
|
||
type has the Zig offsets (an emit-level test or an `.ir`/codegen check).
|
||
- **2.3 — field access honours the remap.** Every `struct_gep` / field load+store
|
||
for a welded struct maps the sx field index → `plan.sx_to_llvm[i]` before
|
||
`LLVMBuildStructGEP2` (`src/backend/llvm/ops.zig` — `emitFieldAccess` /
|
||
struct-literal init / the `field_ptr` paths). Prove with a REORDERED welded
|
||
struct used as runtime data: construct + read each field back correct.
|
||
- **2.4 — interp comptime layout.** The comptime interp represents structs as
|
||
`Value.aggregate` by logical index — fine for field access. The byte layout
|
||
matters at the handler boundary: serialize a welded-struct `Value` into
|
||
Zig-layout memory (via the plan's offsets) so a handler can take `*ZigType`,
|
||
and read a Zig-layout result back into a `Value`. (Or: keep handlers reading
|
||
`Value` aggregates logically — decide when wiring `register_struct`.)
|
||
- **2.5 — `register_struct` / `find_type` handlers.** Bind
|
||
`register_struct(StructInfo) -> Type` (guarded: dup field names, kind) +
|
||
`find_type(StringId) -> ?Type` over the host-call bridge, consuming a welded
|
||
`StructInfo`. Prove: build a struct programmatically + round-trip a source one.
|
||
- **2.6 — re-express `type_info`/`define` (struct) as sx** over `register_struct`/
|
||
`find_type`; migrate `examples/0622`; delete the bespoke struct interp arms
|
||
(`defineStruct`/`reflectTypeInfo` struct path). Design build-order steps 2–3.
|
||
|
||
Then Phase 3+: widen to enum/tuple (`EnumInfo`/`TaggedUnionInfo`/`TupleInfo`,
|
||
optional fields → sentinels), migrate `BuildOptions` to `abi(.zig) extern
|
||
compiler` (the `#compiler` registry re-homes under the `compiler` lib), delete
|
||
`#compiler`.
|
||
|
||
## Known issues
|
||
- None for this stream. (Metatype's deferred enhancement is issue 0141 — comptime
|
||
`List` growth; orthogonal, see `current/CHECKPOINT-METATYPE.md`.)
|
||
|
||
## Log
|
||
- **Phase 2.1 — weld-plan layout math + `StructInfo` registered.** Decision:
|
||
full byte-layout weld (not logical-field marshalling). `computeWeldPlan`
|
||
(offset-order elements + padding + sx→element remap), pure + unit-tested
|
||
against `Field` (identity) and `StructInfo` (reordered, remap `[1,0,3,2]`).
|
||
No emit/interp wiring yet. Build + suite green.
|
||
- **Phase 1 polish — comptime-only enforcement.** A runtime call to a welded fn is
|
||
a clean build-gating error (`emitCall` gate, guarded by enclosing-`is_comptime`
|
||
so `#run`/`::` uses stay green), not a link failure. Example 1185. Build + suite
|
||
green (458 unit, 690 corpus).
|
||
- **Phase 1.1 fifth sub-step — host-call bridge (welded functions).**
|
||
`compiler_lib` function registry (`intern`/`text_of`) + `findFn`; IR `Function`
|
||
`compiler_welded` flag set/validated in `declareFunction` (`weldedCompilerFn`);
|
||
`interp.call()` dispatches welded calls to the Zig handler. Examples 0626 (round-
|
||
trip) + 1184 (unexported-fn diagnostic); `findFn` unit-tested. Runtime-call clean
|
||
rejection deferred (loud link error today). Build + suite green (458 unit, 689
|
||
corpus).
|
||
- **Phase 1.1 fourth sub-step — welded-struct layout validation.**
|
||
`validateStructLayout` (pure, unit-tested) + `validateWeldedStruct` wired into
|
||
`registerStructDecl`: a `struct abi(.zig) extern compiler` is validated against
|
||
the registry (lib == compiler, name exported, layout matches) with build-gating
|
||
diagnostics. `#library "compiler"` no longer dlopen'd. Examples 0625 (faithful
|
||
Field) + 1183 (field-count mismatch diagnostic). Offset-override/GEP deferred to
|
||
Phase 2 (not exercised by Field's natural layout). Build + suite green (456 unit,
|
||
687 corpus).
|
||
- **Phase 1.1 third sub-step — binding registry.** New `src/ir/compiler_lib.zig`:
|
||
the `compiler` lib's welded-type registry; `Field` welded to
|
||
`StructInfo.Field` with layout baked from the real Zig type
|
||
(`@offsetOf`/`@sizeOf`/`@alignOf`); `findType` lookup proven by unit test
|
||
(+ null off the export list). Standalone island — not yet consumed by lowering.
|
||
Build + suite green (454 unit tests). Break-verified.
|
||
- **Phase 1.1 second sub-step — struct-decl binding parses.** `ast.StructDecl`
|
||
gained `abi` + `extern_lib`; `parseStructDecl` parses `abi(.zig) extern <lib>`
|
||
after `struct`. Parser unit tests (welded `Field` + plain struct), break-verified.
|
||
Build + suite green. Parse-only sub-step (fns + structs) of Phase 1.1 complete.
|
||
- **Phase 1.1 first sub-step + `callconv`→`abi` unification.** Parsed `abi(.zig)
|
||
extern <lib>` on fn decls; unified `callconv` into `abi(.c|.zig|.pure)` (removed
|
||
the `callconv` keyword), migrated 52 sx files + compiler diagnostics + docs +
|
||
snapshots. Build + suite green. The original design's `extern(.zig)` single
|
||
qualifier was split into `abi(.zig)` (ABI/layout, before extern) + `extern
|
||
<lib>` (linkage + source) — recorded in the design doc's syntax-decision note.
|