sx/current/CHECKPOINT-COMPILER-API.md

CHECKPOINT-COMPILER-API — comptime compiler library (#library "compiler" + abi(.zig) extern)

Companion to the design-of-record ../design/comptime-compiler-api.md (the plan

• phased build order live there). This stream supersedes the metatype declare/define/type_info #builtins and the #compiler struct attribute with ONE welded mechanism. Branch: reify (off master). Update after every step.

⏯ Resume (fresh session)

Phase 1 is COMPLETE and committed (cd5b958); Phase 2 (full byte-layout weld) just started. Do sub-step 2.2 next — make src/backend/llvm/types.zig's .@"struct" case build a welded struct's LLVM type from compiler_lib.computeWeldPlan (offset-ordered field elements + [N x i8] padding) with a build-time LLVMOffsetOfElement == plan offset + LLVMABISizeOfType == total_size assertion; cache the plan per TypeId for the GEP sites. The plan math (sub-step 2.1) is done, pure, and unit-tested — see computeWeldPlan in src/ir/compiler_lib.zig. Full 2.2–2.6 breakdown under ## Next step. Read order: this file → the design doc → src/ir/compiler_lib.zig. Build/verify: zig build && zig build test (green now).

⚠ Snapshot gotcha: zig build test -Dupdate-goldens on this aarch64 host clobbers cross-arch examples' CI-captured .stdout (1228/1231/1639/1651/1657–1660) with host-specific empties. After regenerating, revert those (git checkout / rm) before committing — they are NOT part of this stream.

Last completed step

Phase 2, sub-step 1 — the weld-plan layout math + StructInfo registered. The de-risked core of the byte-layout-override ("GEP") engine, pure + unit-tested, no emit/interp wiring yet (suite trivially green).

Decision (locked 2026-06-17): full byte-layout weld — a welded sx struct is laid out byte-identically to the bound Zig type (Zig's @offsetOf, reordering + padding included), so it passes to a Zig handler as raw memory with zero marshalling. (The alternative — handlers reading interp Value aggregates logically, no layout override — was rejected; welded types must also be usable as runtime data, and the design wants the literal byte weld.)

• Measured: Zig reorders StructInfo to fields@0, name@16, nominal_id@20, is_protocol@24, size 32 — vs sx-natural name@0, fields@8, … So the override is genuinely required (Field's two-u32 natural layout was the easy case).
• compiler_lib.zig: registered StructInfo (weldStruct, the second bound_types entry). Added WeldElement / WeldPlan + computeWeldPlan(alloc, fields, total) — pure: orders fields by ascending byte offset, inserts padding elements for gaps + the alignment tail, and builds the sx-field → LLVM-element remap. This is what the LLVM type builder + struct-GEP sites will consume.
• Unit-tested (compiler_lib.test.zig): Field → identity plan (2 elems, no pad); StructInfo → 5 elems [fields@0, name@16, nominal_id@20, is_protocol@24, pad@25..32], remap [1,0,3,2].
• zig build + zig build test green.

Earlier — Phase 1 polish (comptime-only enforcement)

A RUNTIME call to a fn abi(.zig) extern compiler is a clean build-gating error instead of an undefined-symbol link failure.

• emitCall (src/backend/llvm/ops.zig): when the callee is compiler_welded AND the ENCLOSING function is not is_comptime (i.e. genuine runtime code, not a #run/:: initializer wrapper whose LLVM body is dead), print a clear "comptime-only … cannot be called at runtime" error and set comptime_failed (the driver halts before object/JIT emission). The enclosing is_comptime guard is what keeps the legitimate #run use (example 0626) green.
• Corpus: examples/1185-diagnostics-weld-fn-runtime-call.sx (runtime intern(…) → clean error, exit 1, no link failure).
• zig build + zig build test green (458 unit + 690 corpus).

Earlier — fifth sub-step (host-call bridge)

A fn abi(.zig) extern compiler dispatches, under the comptime interpreter, to its registered Zig handler instead of dlsym.

• compiler_lib.zig: function registry — BoundFn { sx_name, handler }, bound_fns = intern(string)->StringId + text_of(StringId)->string (the string-pool round-trip), findFn, and FnHandler (*Interpreter, []Value -> Value). intern mutates via interp.mint orelse @constCast(&module.types) (the same mutable-table access the metatype mint path uses); text_of reads the const pool. Imports interp.zig (the compiler_hooks↔interp cycle pattern).
• IR Function gained compiler_welded: bool. declareFunction (src/ir/lower/decl.zig) sets it via weldedCompilerFn, which also VALIDATES: the bound lib must be compiler and the name must be on the function-export list — else a build-gating .err (no silent fall-through to dlsym).
• interp.call(): before the dlsym/extern path, a compiler_welded function routes to compiler_lib.findFn(name).handler(self, args) (clean bail off the export list).
• Corpus: examples/0626-comptime-weld-fn-intern-text-of.sx (#run text_of(intern("hello, compiler")) folds to a string constant → prints it); examples/1184-diagnostics-weld-fn-unexported.sx (unexported welded-fn name → build error). findFn lookup unit-tested.
• Runtime-call rejection is NOT yet clean — welded fns are comptime-only; a RUNTIME call would emit a reference to a non-existent extern symbol → a loud LINK error (not silent, but not a tidy diagnostic). The examples call welded fns only inside #run. A dedicated "comptime-only symbol" emit diagnostic is the immediate follow-up.
• zig build + zig build test green (458 unit tests + 689 corpus).

Earlier — fourth sub-step (welded-struct layout validation)

A struct abi(.zig) extern compiler { … } is validated against the binding registry as a header checked against the implementation.

• compiler_lib.zig: validateStructLayout(bt, sx_fields, total) — pure, returns the first LayoutMismatch (field count / name / size / total) or null. Plus lib_name = "compiler" and SxField. Unit-tested (faithful Field passes; each drift flagged as the right variant).
• registerStructDecl (src/ir/lower/nominal.zig): for sd.abi == .zig, validateWeldedStruct checks the bound lib is compiler, the name is on the export list (findType), and the sx layout (field names + typeSizeBytes + total) matches the welded type — emitting a build-gating .err (good span into the struct body) on any failure. No silent reinterpretation.
• #library "compiler" is the comptime-only internal surface, NOT a dylib — src/main.zig's dlopen walker skips it (was emitting a spurious libcompiler.so load warning).
• Corpus: examples/0625-comptime-weld-struct-field.sx (faithful Field welds, validates, usable as data → name=7 ty=3); examples/1183-diagnostics-weld- struct-field-count.sx (one-field Field → build-gating field-count diagnostic).
• Offset-override / GEP emission for non-natural Zig layouts is NOT here — it isn't exercised by Field (two u32s = natural layout coincides with the weld). It arrives with StructInfo in Phase 2 (slices/reordering), where the bound offsets actually differ from the sx-natural ones. The validation already checks per-field size + total, so a layout drift is caught even before the override engine exists.
• zig build + zig build test green (456 unit tests + 687 corpus).

Earlier — third sub-step (binding registry)

The binding registry (welded-type lookup, layout baked from the real Zig type).

• New src/ir/compiler_lib.zig — the compiler library's binding registry, the curated safety boundary. BoundType { sx_name, size, alignment, fields: []FieldLayout{name, offset, size} }; weldStruct bakes the layout from a real Zig struct via @sizeOf/@alignOf/@offsetOf at compiler-build time (a sx-field-count mismatch is a @compileError, never a silent truncation). bound_types exports Field (welded to types.TypeInfo.StructInfo.Field — two u32s); findType(sx_name) ?*const BoundType is the lookup the welded-decl resolution path will consult (returns null off the export list — clean boundary, no silent default).
• Registered in the barrel (src/ir/ir.zig): compiler_lib + compiler_lib_tests.
• Tests (src/ir/compiler_lib.test.zig): findType("Field") equals the real StructInfo.Field @sizeOf/@alignOf/@offsetOf (8 bytes, two u32s at 0/4); an unexported name returns null. Break-verified (a wrong size → suite red, named ir.compiler_lib.test...).
• zig build + zig build test green (454 unit tests).

Earlier — second sub-step (struct-decl parse)

abi(.zig) extern <lib> PARSES on a STRUCT decl (parse-only, no semantics).

• ast.StructDecl gained abi: ABI + extern_lib: ?[]const u8 binding fields.
• parseStructDecl (src/parser.zig): after struct (and the #compiler check), parse an optional abi(...) then optional extern <lib> — same slot order as fn decls — and thread them onto the node. Ordinary structs are unperturbed (parseOptionalAbi/parseOptionalExternExport no-op when absent).
• Parser unit tests (src/parser.test.zig): Field :: struct abi(.zig) extern compiler { name: StringId; ty: Type; } parses with abi == .zig, extern_lib == "compiler", field list intact; a plain struct leaves abi == .default / extern_lib == null. Break-verified (a wrong-sentinel assert turns the suite red, confirming the test runs).
• zig build + zig build test green.

Earlier — first sub-step (fn decls) + the syntax pivot

abi(.zig) extern <lib> PARSES on a fn decl (parse-only). Plus the syntax pivot it required.

Syntax decision (locked 2026-06-17, supersedes the doc's original extern(.zig) <lib> single-qualifier form): the ABI/layout selector and the linkage keyword are two orthogonal annotations.

• abi(.x) — ABI / calling-convention annotation in the slot before extern/export. Unified replacement for callconv(...), which is removed. ABI = { default, c, zig, pure }: .c (C ABI), .zig (Zig-layout weld → the compiler library), .pure (naked asm), .default (unannotated). Can appear standalone (no extern) on any fn / fn-type / lambda.
• extern <lib> — linkage keyword + binding source (named library).

So a welded binding is text_of :: (id: StringId) -> string abi(.zig) extern compiler;.

What landed:

• AST (src/ast.zig): CallingConvention → ABI { default, c, zig, pure }; the call_conv field → abi: ABI on FnDecl / Lambda / FunctionTypeExpr.
• Lexer/token (src/token.zig, src/lexer.zig): kw_callconv → kw_abi, keyword string "callconv" → "abi".
• Parser (src/parser.zig): parseOptionalCallConv → parseOptionalAbi (parses abi(.c|.zig|.pure)); wired in the fn-decl postfix slot (before extern/export), the function-type-expr slot, and the lambda slot; isFunctionDef/hasFnBodyAfterArrow recognise kw_abi.
• AST→IR map (src/ir/type_resolver.zig, src/ir/lower/decl.zig, sema.zig, closure.zig): the AST .abi == .c reads kept their C-ABI meaning; the function-type resolver maps .zig/.pure → IR .default (no fn-pointer-type CC for those decl-level ABIs; neither occurs in a function-TYPE position yet).
• CC-mismatch diagnostic (src/ir/lower/expr.zig, src/sema.zig): the user-facing text callconv(.c) → abi(.c).
• sx migration: 52 .sx files callconv( → abi( (all were function-type callback annotations — none in the fn-decl postfix slot, so no reordering).
• Docs: readme.md, specs.md, the design doc, snapshots (0114 / 1104 / 1200) regenerated for the rename.
• Tests: parser unit tests in src/parser.test.zig — abi(.zig) extern <lib> on a fn decl (asserts abi == .zig, extern_export == .extern_, extern_lib == "compiler"); bare extern leaves abi == .default; standalone abi(.c) / abi(.pure). lexer/sema tests updated.

zig build + zig build test green (450/450 unit + 685 corpus).

Current state

• compiler :: #library "compiler"; parses + is recognised as the comptime-only internal surface (never dlopen'd).
• abi(.zig) extern compiler STRUCTS: layout-validated against the registry (faithful → ok; drift → build-gating diagnostic). Field welds + usable.
• abi(.zig) extern compiler FUNCTIONS: dispatched under the comptime interp to their registered Zig handler (intern/text_of round-trip works); unexported names rejected at declaration. Comptime-only.
• A RUNTIME call to a welded fn is a clean build-gating error (comptime-only enforcement at emitCall); the legitimate #run/:: use stays green.
• The whole Phase 1 foundation (parse → registry → struct-layout validation → function host-call bridge → comptime-only enforcement) is in place for the two-u32 Field case + the two string readers.
• Deferred: offset-override / LLVM byte-offset GEP for non-natural layouts (needed by StructInfo's slice field, Phase 2).

Next step — Phase 2 decomposition (byte-layout weld for StructInfo)

The weld plan (sub-step 1) is the pure layout math. The remaining sub-steps wire it through emit + interp so a non-natural welded struct actually works. Each must stay green; do ONE per session (the IR-stream split rule).

• 2.2 — LLVM type honours the plan. In src/backend/llvm/types.zig .@"struct" case: if the struct's name is in compiler_lib.findType, build the LLVM struct from computeWeldPlan — elements in offset order (real field types + [N x i8] padding), and assert LLVMOffsetOfElement(elem) == plan.elements[e].offset for every field element + LLVMABISizeOfType == total_size (the build-time layout-equality assertion; mismatch = a loud emit failure). Cache the plan per TypeId (the GEP sites + interp need the remap). Prove: a welded struct's LLVM type has the Zig offsets (an emit-level test or an .ir/codegen check).
• 2.3 — field access honours the remap. Every struct_gep / field load+store for a welded struct maps the sx field index → plan.sx_to_llvm[i] before LLVMBuildStructGEP2 (src/backend/llvm/ops.zig — emitFieldAccess / struct-literal init / the field_ptr paths). Prove with a REORDERED welded struct used as runtime data: construct + read each field back correct.
• 2.4 — interp comptime layout. The comptime interp represents structs as Value.aggregate by logical index — fine for field access. The byte layout matters at the handler boundary: serialize a welded-struct Value into Zig-layout memory (via the plan's offsets) so a handler can take *ZigType, and read a Zig-layout result back into a Value. (Or: keep handlers reading Value aggregates logically — decide when wiring register_struct.)
• 2.5 — register_struct / find_type handlers. Bind register_struct(StructInfo) -> Type (guarded: dup field names, kind) + find_type(StringId) -> ?Type over the host-call bridge, consuming a welded StructInfo. Prove: build a struct programmatically + round-trip a source one.
• 2.6 — re-express type_info/define (struct) as sx over register_struct/ find_type; migrate examples/0622; delete the bespoke struct interp arms (defineStruct/reflectTypeInfo struct path). Design build-order steps 2–3.

Then Phase 3+: widen to enum/tuple (EnumInfo/TaggedUnionInfo/TupleInfo, optional fields → sentinels), migrate BuildOptions to abi(.zig) extern compiler (the #compiler registry re-homes under the compiler lib), delete #compiler.

Known issues

• None for this stream. (Metatype's deferred enhancement is issue 0141 — comptime List growth; orthogonal, see current/CHECKPOINT-METATYPE.md.)

Log

• Phase 2.1 — weld-plan layout math + StructInfo registered. Decision: full byte-layout weld (not logical-field marshalling). computeWeldPlan (offset-order elements + padding + sx→element remap), pure + unit-tested against Field (identity) and StructInfo (reordered, remap [1,0,3,2]). No emit/interp wiring yet. Build + suite green.
• Phase 1 polish — comptime-only enforcement. A runtime call to a welded fn is a clean build-gating error (emitCall gate, guarded by enclosing-is_comptime so #run/:: uses stay green), not a link failure. Example 1185. Build + suite green (458 unit, 690 corpus).
• Phase 1.1 fifth sub-step — host-call bridge (welded functions). compiler_lib function registry (intern/text_of) + findFn; IR Function compiler_welded flag set/validated in declareFunction (weldedCompilerFn); interp.call() dispatches welded calls to the Zig handler. Examples 0626 (round- trip) + 1184 (unexported-fn diagnostic); findFn unit-tested. Runtime-call clean rejection deferred (loud link error today). Build + suite green (458 unit, 689 corpus).
• Phase 1.1 fourth sub-step — welded-struct layout validation. validateStructLayout (pure, unit-tested) + validateWeldedStruct wired into registerStructDecl: a struct abi(.zig) extern compiler is validated against the registry (lib == compiler, name exported, layout matches) with build-gating diagnostics. #library "compiler" no longer dlopen'd. Examples 0625 (faithful Field) + 1183 (field-count mismatch diagnostic). Offset-override/GEP deferred to Phase 2 (not exercised by Field's natural layout). Build + suite green (456 unit, 687 corpus).
• Phase 1.1 third sub-step — binding registry. New src/ir/compiler_lib.zig: the compiler lib's welded-type registry; Field welded to StructInfo.Field with layout baked from the real Zig type (@offsetOf/@sizeOf/@alignOf); findType lookup proven by unit test (+ null off the export list). Standalone island — not yet consumed by lowering. Build + suite green (454 unit tests). Break-verified.
• Phase 1.1 second sub-step — struct-decl binding parses. ast.StructDecl gained abi + extern_lib; parseStructDecl parses abi(.zig) extern <lib> after struct. Parser unit tests (welded Field + plain struct), break-verified. Build + suite green. Parse-only sub-step (fns + structs) of Phase 1.1 complete.
• Phase 1.1 first sub-step + callconv→abi unification. Parsed abi(.zig) extern <lib> on fn decls; unified callconv into abi(.c|.zig|.pure) (removed the callconv keyword), migrated 52 sx files + compiler diagnostics + docs + snapshots. Build + suite green. The original design's extern(.zig) single qualifier was split into abi(.zig) (ABI/layout, before extern) + extern <lib> (linkage + source) — recorded in the design doc's syntax-decision note.