comptime compiler-API: Phase 1 foundation + Phase 2.1 weld plan
Introduce the welded comptime `compiler` library (`#library "compiler"` +
`abi(.zig) extern compiler`), per design/comptime-compiler-api.md, and unify
`callconv(...)` into the new `abi(...)` annotation.
abi(...) replaces callconv(...):
- New ABI enum { default, c, zig, pure }; `abi(.c|.zig|.pure)` parses in the
postfix slot before extern/export (and standalone). `kw_callconv` -> `kw_abi`.
- Migrated 52 sx files, the call-convention-mismatch diagnostic, and docs
(readme/specs) from `callconv(.c)` to `abi(.c)`.
Phase 1 — welded compiler library (parse -> registry -> validation -> bridge):
- `abi(.zig) extern compiler` parses on fn decls (carries abi/extern_lib) and
struct decls (StructDecl.abi/extern_lib).
- `#library "compiler"` is the comptime-only internal surface — never dlopen'd.
- src/ir/compiler_lib.zig: the binding registry (the safety boundary). `Field`
welded to StructInfo.Field with layout baked from the real Zig type
(@offsetOf/@sizeOf); `findType`/`findFn`. Welded structs are layout-validated
at registration (field set + total size) as a header checked against the impl.
- Host-call bridge: a `fn abi(.zig) extern compiler` dispatches under the
comptime interp to its registered Zig handler (intern/text_of round-trip),
never dlsym. IR Function.compiler_welded; validated in declareFunction.
- Comptime-only enforcement: a runtime call to a welded fn is a clean
build-gating error (emitCall), not an undefined-symbol link failure.
Phase 2.1 — byte-layout weld foundation:
- Decision: full byte-layout weld (sx struct laid out byte-identically to the
bound Zig type). Registered StructInfo (first non-natural / Zig-reordered
layout). `computeWeldPlan` — pure offset-ordered element plan + padding +
sx-field->LLVM-element remap; unit-tested. Emit/interp wiring is the next
sub-step (2.2+, see current/CHECKPOINT-COMPILER-API.md).
Examples: 0625/0626 (welded struct + fn round-trip), 1183/1184/1185
(layout-mismatch, unexported-fn, runtime-call diagnostics).
This commit is contained in:
agra
273
current/CHECKPOINT-COMPILER-API.md
Normal file
273
current/CHECKPOINT-COMPILER-API.md
Normal file
@@ -0,0 +1,273 @@
|
||||
# CHECKPOINT-COMPILER-API — comptime `compiler` library (`#library "compiler"` + `abi(.zig) extern`)
|
||||
|
||||
Companion to the design-of-record
|
||||
[../design/comptime-compiler-api.md](../design/comptime-compiler-api.md) (the plan
|
||||
+ phased build order live there). This stream supersedes the metatype
|
||||
`declare`/`define`/`type_info` `#builtin`s and the `#compiler` struct attribute
|
||||
with ONE welded mechanism. Branch: `reify` (off `master`). Update after every step.
|
||||
|
||||
## Last completed step
|
||||
**Phase 2, sub-step 1 — the weld-plan layout math + `StructInfo` registered.**
|
||||
The de-risked core of the byte-layout-override ("GEP") engine, pure + unit-tested,
|
||||
no emit/interp wiring yet (suite trivially green).
|
||||
|
||||
Decision (locked 2026-06-17): **full byte-layout weld** — a welded sx struct is
|
||||
laid out byte-identically to the bound Zig type (Zig's `@offsetOf`, reordering +
|
||||
padding included), so it passes to a Zig handler as raw memory with zero
|
||||
marshalling. (The alternative — handlers reading interp `Value` aggregates
|
||||
logically, no layout override — was rejected; welded types must also be usable as
|
||||
runtime data, and the design wants the literal byte weld.)
|
||||
|
||||
- Measured: Zig reorders `StructInfo` to `fields`@0, `name`@16, `nominal_id`@20,
|
||||
`is_protocol`@24, size 32 — vs sx-natural `name`@0, `fields`@8, … So the override
|
||||
is genuinely required (`Field`'s two-u32 natural layout was the easy case).
|
||||
- `compiler_lib.zig`: registered `StructInfo` (`weldStruct`, the second
|
||||
`bound_types` entry). Added `WeldElement` / `WeldPlan` + `computeWeldPlan(alloc,
|
||||
fields, total)` — pure: orders fields by ascending byte offset, inserts padding
|
||||
elements for gaps + the alignment tail, and builds the sx-field → LLVM-element
|
||||
remap. This is what the LLVM type builder + struct-GEP sites will consume.
|
||||
- Unit-tested (`compiler_lib.test.zig`): `Field` → identity plan (2 elems, no pad);
|
||||
`StructInfo` → 5 elems `[fields@0, name@16, nominal_id@20, is_protocol@24,
|
||||
pad@25..32]`, remap `[1,0,3,2]`.
|
||||
- `zig build` + `zig build test` green.
|
||||
|
||||
### Earlier — Phase 1 polish (comptime-only enforcement)
|
||||
**A RUNTIME call to a `fn abi(.zig) extern compiler` is a clean build-gating error
|
||||
instead of an undefined-symbol link failure.**
|
||||
- `emitCall` (`src/backend/llvm/ops.zig`): when the callee is `compiler_welded`
|
||||
AND the ENCLOSING function is not `is_comptime` (i.e. genuine runtime code, not a
|
||||
`#run`/`::` initializer wrapper whose LLVM body is dead), print a clear
|
||||
"comptime-only … cannot be called at runtime" error and set
|
||||
`comptime_failed` (the driver halts before object/JIT emission). The enclosing
|
||||
`is_comptime` guard is what keeps the legitimate `#run` use (example 0626) green.
|
||||
- Corpus: `examples/1185-diagnostics-weld-fn-runtime-call.sx` (runtime `intern(…)`
|
||||
→ clean error, exit 1, no link failure).
|
||||
- `zig build` + `zig build test` green (458 unit + 690 corpus).
|
||||
|
||||
### Earlier — fifth sub-step (host-call bridge)
|
||||
**A `fn abi(.zig) extern compiler` dispatches, under the comptime interpreter, to
|
||||
its registered Zig handler instead of dlsym.**
|
||||
- `compiler_lib.zig`: function registry — `BoundFn { sx_name, handler }`,
|
||||
`bound_fns` = `intern(string)->StringId` + `text_of(StringId)->string` (the
|
||||
string-pool round-trip), `findFn`, and `FnHandler` (`*Interpreter, []Value ->
|
||||
Value`). `intern` mutates via `interp.mint orelse @constCast(&module.types)`
|
||||
(the same mutable-table access the metatype mint path uses); `text_of` reads the
|
||||
const pool. Imports `interp.zig` (the compiler_hooks↔interp cycle pattern).
|
||||
- IR `Function` gained `compiler_welded: bool`. `declareFunction`
|
||||
(`src/ir/lower/decl.zig`) sets it via `weldedCompilerFn`, which also VALIDATES:
|
||||
the bound lib must be `compiler` and the name must be on the function-export
|
||||
list — else a build-gating `.err` (no silent fall-through to dlsym).
|
||||
- `interp.call()`: before the dlsym/extern path, a `compiler_welded` function
|
||||
routes to `compiler_lib.findFn(name).handler(self, args)` (clean bail off the
|
||||
export list).
|
||||
- Corpus: `examples/0626-comptime-weld-fn-intern-text-of.sx` (`#run
|
||||
text_of(intern("hello, compiler"))` folds to a string constant → prints it);
|
||||
`examples/1184-diagnostics-weld-fn-unexported.sx` (unexported welded-fn name →
|
||||
build error). `findFn` lookup unit-tested.
|
||||
- **Runtime-call rejection is NOT yet clean** — welded fns are comptime-only; a
|
||||
RUNTIME call would emit a reference to a non-existent extern symbol → a loud
|
||||
LINK error (not silent, but not a tidy diagnostic). The examples call welded fns
|
||||
only inside `#run`. A dedicated "comptime-only symbol" emit diagnostic is the
|
||||
immediate follow-up.
|
||||
- `zig build` + `zig build test` green (458 unit tests + 689 corpus).
|
||||
|
||||
### Earlier — fourth sub-step (welded-struct layout validation)
|
||||
**A `struct abi(.zig) extern compiler { … }` is validated against the binding
|
||||
registry as a *header checked against the implementation*.**
|
||||
- `compiler_lib.zig`: `validateStructLayout(bt, sx_fields, total)` — pure, returns
|
||||
the first `LayoutMismatch` (field count / name / size / total) or null. Plus
|
||||
`lib_name = "compiler"` and `SxField`. Unit-tested (faithful `Field` passes;
|
||||
each drift flagged as the right variant).
|
||||
- `registerStructDecl` (`src/ir/lower/nominal.zig`): for `sd.abi == .zig`,
|
||||
`validateWeldedStruct` checks the bound lib is `compiler`, the name is on the
|
||||
export list (`findType`), and the sx layout (field names + `typeSizeBytes` +
|
||||
total) matches the welded type — emitting a build-gating `.err` (good span into
|
||||
the struct body) on any failure. No silent reinterpretation.
|
||||
- `#library "compiler"` is the comptime-only internal surface, NOT a dylib —
|
||||
`src/main.zig`'s dlopen walker skips it (was emitting a spurious `libcompiler.so`
|
||||
load warning).
|
||||
- Corpus: `examples/0625-comptime-weld-struct-field.sx` (faithful `Field` welds,
|
||||
validates, usable as data → `name=7 ty=3`); `examples/1183-diagnostics-weld-
|
||||
struct-field-count.sx` (one-field `Field` → build-gating field-count diagnostic).
|
||||
- **Offset-override / GEP emission for non-natural Zig layouts is NOT here** — it
|
||||
isn't exercised by `Field` (two u32s = natural layout coincides with the weld).
|
||||
It arrives with `StructInfo` in Phase 2 (slices/reordering), where the bound
|
||||
offsets actually differ from the sx-natural ones. The validation already checks
|
||||
per-field size + total, so a layout drift is caught even before the override
|
||||
engine exists.
|
||||
- `zig build` + `zig build test` green (456 unit tests + 687 corpus).
|
||||
|
||||
### Earlier — third sub-step (binding registry)
|
||||
**The binding registry (welded-type lookup, layout baked from the real Zig
|
||||
type).**
|
||||
- New `src/ir/compiler_lib.zig` — the `compiler` library's binding registry, the
|
||||
curated safety boundary. `BoundType { sx_name, size, alignment, fields:
|
||||
[]FieldLayout{name, offset, size} }`; `weldStruct` bakes the layout from a real
|
||||
Zig struct via `@sizeOf`/`@alignOf`/`@offsetOf` at compiler-build time (a
|
||||
sx-field-count mismatch is a `@compileError`, never a silent truncation).
|
||||
`bound_types` exports `Field` (welded to `types.TypeInfo.StructInfo.Field` —
|
||||
two `u32`s); `findType(sx_name) ?*const BoundType` is the lookup the welded-decl
|
||||
resolution path will consult (returns null off the export list — clean boundary,
|
||||
no silent default).
|
||||
- Registered in the barrel (`src/ir/ir.zig`): `compiler_lib` + `compiler_lib_tests`.
|
||||
- Tests (`src/ir/compiler_lib.test.zig`): `findType("Field")` equals the real
|
||||
`StructInfo.Field` `@sizeOf`/`@alignOf`/`@offsetOf` (8 bytes, two u32s at 0/4);
|
||||
an unexported name returns null. Break-verified (a wrong size → suite red,
|
||||
named `ir.compiler_lib.test...`).
|
||||
- `zig build` + `zig build test` green (454 unit tests).
|
||||
|
||||
### Earlier — second sub-step (struct-decl parse)
|
||||
**`abi(.zig) extern <lib>` PARSES on a STRUCT decl (parse-only, no semantics).**
|
||||
- `ast.StructDecl` gained `abi: ABI` + `extern_lib: ?[]const u8` binding fields.
|
||||
- `parseStructDecl` (`src/parser.zig`): after `struct` (and the `#compiler`
|
||||
check), parse an optional `abi(...)` then optional `extern <lib>` — same slot
|
||||
order as fn decls — and thread them onto the node. Ordinary structs are
|
||||
unperturbed (`parseOptionalAbi`/`parseOptionalExternExport` no-op when absent).
|
||||
- Parser unit tests (`src/parser.test.zig`): `Field :: struct abi(.zig) extern
|
||||
compiler { name: StringId; ty: Type; }` parses with `abi == .zig`, `extern_lib
|
||||
== "compiler"`, field list intact; a plain struct leaves `abi == .default` /
|
||||
`extern_lib == null`. Break-verified (a wrong-sentinel assert turns the suite
|
||||
red, confirming the test runs).
|
||||
- `zig build` + `zig build test` green.
|
||||
|
||||
### Earlier — first sub-step (fn decls) + the syntax pivot
|
||||
**`abi(.zig) extern <lib>` PARSES on a fn decl (parse-only).** Plus the syntax
|
||||
pivot it required.
|
||||
|
||||
Syntax decision (locked 2026-06-17, supersedes the doc's original
|
||||
`extern(.zig) <lib>` single-qualifier form): the ABI/layout selector and the
|
||||
linkage keyword are two orthogonal annotations.
|
||||
- `abi(.x)` — ABI / calling-convention annotation in the slot **before**
|
||||
`extern`/`export`. **Unified replacement for `callconv(...)`, which is removed.**
|
||||
`ABI = { default, c, zig, pure }`: `.c` (C ABI), `.zig` (Zig-layout weld → the
|
||||
`compiler` library), `.pure` (naked asm), `.default` (unannotated). Can appear
|
||||
standalone (no extern) on any fn / fn-type / lambda.
|
||||
- `extern <lib>` — linkage keyword + binding source (named library).
|
||||
|
||||
So a welded binding is `text_of :: (id: StringId) -> string abi(.zig) extern compiler;`.
|
||||
|
||||
What landed:
|
||||
- **AST** (`src/ast.zig`): `CallingConvention` → `ABI { default, c, zig, pure }`;
|
||||
the `call_conv` field → `abi: ABI` on `FnDecl` / `Lambda` / `FunctionTypeExpr`.
|
||||
- **Lexer/token** (`src/token.zig`, `src/lexer.zig`): `kw_callconv` → `kw_abi`,
|
||||
keyword string `"callconv"` → `"abi"`.
|
||||
- **Parser** (`src/parser.zig`): `parseOptionalCallConv` → `parseOptionalAbi`
|
||||
(parses `abi(.c|.zig|.pure)`); wired in the fn-decl postfix slot (before
|
||||
`extern`/`export`), the function-type-expr slot, and the lambda slot;
|
||||
`isFunctionDef`/`hasFnBodyAfterArrow` recognise `kw_abi`.
|
||||
- **AST→IR map** (`src/ir/type_resolver.zig`, `src/ir/lower/decl.zig`, `sema.zig`,
|
||||
`closure.zig`): the AST `.abi == .c` reads kept their C-ABI meaning; the
|
||||
function-type resolver maps `.zig`/`.pure` → IR `.default` (no fn-pointer-type
|
||||
CC for those decl-level ABIs; neither occurs in a function-TYPE position yet).
|
||||
- **CC-mismatch diagnostic** (`src/ir/lower/expr.zig`, `src/sema.zig`): the
|
||||
user-facing text `callconv(.c)` → `abi(.c)`.
|
||||
- **sx migration**: 52 `.sx` files `callconv(` → `abi(` (all were function-type
|
||||
callback annotations — none in the fn-decl postfix slot, so no reordering).
|
||||
- **Docs**: `readme.md`, `specs.md`, the design doc, snapshots (0114 / 1104 /
|
||||
1200) regenerated for the rename.
|
||||
- **Tests**: parser unit tests in `src/parser.test.zig` — `abi(.zig) extern <lib>`
|
||||
on a fn decl (asserts `abi == .zig`, `extern_export == .extern_`, `extern_lib ==
|
||||
"compiler"`); bare `extern` leaves `abi == .default`; standalone `abi(.c)` /
|
||||
`abi(.pure)`. lexer/sema tests updated.
|
||||
|
||||
`zig build` + `zig build test` green (450/450 unit + 685 corpus).
|
||||
|
||||
## Current state
|
||||
- `compiler :: #library "compiler";` parses + is recognised as the comptime-only
|
||||
internal surface (never dlopen'd).
|
||||
- `abi(.zig) extern compiler` STRUCTS: layout-validated against the registry
|
||||
(faithful → ok; drift → build-gating diagnostic). `Field` welds + usable.
|
||||
- `abi(.zig) extern compiler` FUNCTIONS: dispatched under the comptime interp to
|
||||
their registered Zig handler (`intern`/`text_of` round-trip works); unexported
|
||||
names rejected at declaration. Comptime-only.
|
||||
- A RUNTIME call to a welded fn is a clean build-gating error (comptime-only
|
||||
enforcement at `emitCall`); the legitimate `#run`/`::` use stays green.
|
||||
- The whole Phase 1 foundation (parse → registry → struct-layout validation →
|
||||
function host-call bridge → comptime-only enforcement) is in place for the
|
||||
two-u32 `Field` case + the two string readers.
|
||||
- **Deferred**: offset-override / LLVM byte-offset GEP for non-natural layouts
|
||||
(needed by `StructInfo`'s slice field, Phase 2).
|
||||
|
||||
## Next step — Phase 2 decomposition (byte-layout weld for `StructInfo`)
|
||||
|
||||
The weld plan (sub-step 1) is the pure layout math. The remaining sub-steps wire
|
||||
it through emit + interp so a non-natural welded struct actually works. Each must
|
||||
stay green; do ONE per session (the IR-stream split rule).
|
||||
|
||||
- **2.2 — LLVM type honours the plan.** In `src/backend/llvm/types.zig` `.@"struct"`
|
||||
case: if the struct's name is in `compiler_lib.findType`, build the LLVM struct
|
||||
from `computeWeldPlan` — elements in offset order (real field types + `[N x i8]`
|
||||
padding), and **assert** `LLVMOffsetOfElement(elem) == plan.elements[e].offset`
|
||||
for every field element + `LLVMABISizeOfType == total_size` (the build-time
|
||||
layout-equality assertion; mismatch = a loud emit failure). Cache the plan per
|
||||
TypeId (the GEP sites + interp need the remap). Prove: a welded struct's LLVM
|
||||
type has the Zig offsets (an emit-level test or an `.ir`/codegen check).
|
||||
- **2.3 — field access honours the remap.** Every `struct_gep` / field load+store
|
||||
for a welded struct maps the sx field index → `plan.sx_to_llvm[i]` before
|
||||
`LLVMBuildStructGEP2` (`src/backend/llvm/ops.zig` — `emitFieldAccess` /
|
||||
struct-literal init / the `field_ptr` paths). Prove with a REORDERED welded
|
||||
struct used as runtime data: construct + read each field back correct.
|
||||
- **2.4 — interp comptime layout.** The comptime interp represents structs as
|
||||
`Value.aggregate` by logical index — fine for field access. The byte layout
|
||||
matters at the handler boundary: serialize a welded-struct `Value` into
|
||||
Zig-layout memory (via the plan's offsets) so a handler can take `*ZigType`,
|
||||
and read a Zig-layout result back into a `Value`. (Or: keep handlers reading
|
||||
`Value` aggregates logically — decide when wiring `register_struct`.)
|
||||
- **2.5 — `register_struct` / `find_type` handlers.** Bind
|
||||
`register_struct(StructInfo) -> Type` (guarded: dup field names, kind) +
|
||||
`find_type(StringId) -> ?Type` over the host-call bridge, consuming a welded
|
||||
`StructInfo`. Prove: build a struct programmatically + round-trip a source one.
|
||||
- **2.6 — re-express `type_info`/`define` (struct) as sx** over `register_struct`/
|
||||
`find_type`; migrate `examples/0622`; delete the bespoke struct interp arms
|
||||
(`defineStruct`/`reflectTypeInfo` struct path). Design build-order steps 2–3.
|
||||
|
||||
Then Phase 3+: widen to enum/tuple (`EnumInfo`/`TaggedUnionInfo`/`TupleInfo`,
|
||||
optional fields → sentinels), migrate `BuildOptions` to `abi(.zig) extern
|
||||
compiler` (the `#compiler` registry re-homes under the `compiler` lib), delete
|
||||
`#compiler`.
|
||||
|
||||
## Known issues
|
||||
- None for this stream. (Metatype's deferred enhancement is issue 0141 — comptime
|
||||
`List` growth; orthogonal, see `current/CHECKPOINT-METATYPE.md`.)
|
||||
|
||||
## Log
|
||||
- **Phase 2.1 — weld-plan layout math + `StructInfo` registered.** Decision:
|
||||
full byte-layout weld (not logical-field marshalling). `computeWeldPlan`
|
||||
(offset-order elements + padding + sx→element remap), pure + unit-tested
|
||||
against `Field` (identity) and `StructInfo` (reordered, remap `[1,0,3,2]`).
|
||||
No emit/interp wiring yet. Build + suite green.
|
||||
- **Phase 1 polish — comptime-only enforcement.** A runtime call to a welded fn is
|
||||
a clean build-gating error (`emitCall` gate, guarded by enclosing-`is_comptime`
|
||||
so `#run`/`::` uses stay green), not a link failure. Example 1185. Build + suite
|
||||
green (458 unit, 690 corpus).
|
||||
- **Phase 1.1 fifth sub-step — host-call bridge (welded functions).**
|
||||
`compiler_lib` function registry (`intern`/`text_of`) + `findFn`; IR `Function`
|
||||
`compiler_welded` flag set/validated in `declareFunction` (`weldedCompilerFn`);
|
||||
`interp.call()` dispatches welded calls to the Zig handler. Examples 0626 (round-
|
||||
trip) + 1184 (unexported-fn diagnostic); `findFn` unit-tested. Runtime-call clean
|
||||
rejection deferred (loud link error today). Build + suite green (458 unit, 689
|
||||
corpus).
|
||||
- **Phase 1.1 fourth sub-step — welded-struct layout validation.**
|
||||
`validateStructLayout` (pure, unit-tested) + `validateWeldedStruct` wired into
|
||||
`registerStructDecl`: a `struct abi(.zig) extern compiler` is validated against
|
||||
the registry (lib == compiler, name exported, layout matches) with build-gating
|
||||
diagnostics. `#library "compiler"` no longer dlopen'd. Examples 0625 (faithful
|
||||
Field) + 1183 (field-count mismatch diagnostic). Offset-override/GEP deferred to
|
||||
Phase 2 (not exercised by Field's natural layout). Build + suite green (456 unit,
|
||||
687 corpus).
|
||||
- **Phase 1.1 third sub-step — binding registry.** New `src/ir/compiler_lib.zig`:
|
||||
the `compiler` lib's welded-type registry; `Field` welded to
|
||||
`StructInfo.Field` with layout baked from the real Zig type
|
||||
(`@offsetOf`/`@sizeOf`/`@alignOf`); `findType` lookup proven by unit test
|
||||
(+ null off the export list). Standalone island — not yet consumed by lowering.
|
||||
Build + suite green (454 unit tests). Break-verified.
|
||||
- **Phase 1.1 second sub-step — struct-decl binding parses.** `ast.StructDecl`
|
||||
gained `abi` + `extern_lib`; `parseStructDecl` parses `abi(.zig) extern <lib>`
|
||||
after `struct`. Parser unit tests (welded `Field` + plain struct), break-verified.
|
||||
Build + suite green. Parse-only sub-step (fns + structs) of Phase 1.1 complete.
|
||||
- **Phase 1.1 first sub-step + `callconv`→`abi` unification.** Parsed `abi(.zig)
|
||||
extern <lib>` on fn decls; unified `callconv` into `abi(.c|.zig|.pure)` (removed
|
||||
the `callconv` keyword), migrated 52 sx files + compiler diagnostics + docs +
|
||||
snapshots. Build + suite green. The original design's `extern(.zig)` single
|
||||
qualifier was split into `abi(.zig)` (ABI/layout, before extern) + `extern
|
||||
<lib>` (linkage + source) — recorded in the design doc's syntax-decision note.
|
||||
Reference in New Issue
Block a user