sx/current/CHECKPOINT-COMPILER-API.md

CHECKPOINT-COMPILER-API — comptime compiler library (#library "compiler" + abi(.zig) extern)

Companion to the design-of-record ../design/comptime-compiler-api.md (the plan

• phased build order live there). This stream supersedes the metatype declare/define/type_info #builtins and the #compiler struct attribute with ONE welded mechanism. Branch: reify (off master). Update after every step.

⏯ Resume (fresh session)

⚠ DIRECTION CHANGED (2026-06-17). The active plan is now PLAN-COMPILER-VM.md, NOT the weld. The byte-weld + serialization/marshaling approach is the wrong direction and is being stripped. New foundation: a bytecode VM over flat, byte-addressable memory so comptime values are native bytes; then the compiler-API rides on it with direct memory access (no weld, no validation, no marshaling). Everything below this banner describes the now-superseded weld state (committed on reify through 40d075c) and is kept only to scope the Phase 0 strip. Read PLAN-COMPILER-VM.md first.

Why the pivot: the comptime evaluator (src/ir/interp.zig) represents values as tagged Value unions, NOT native bytes — so a comptime @ptrCast(*StructInfo) reads the Value union's memory, not a struct. The weld tried to bridge that with hand-marshaling — exactly what the design set out to kill. Flat memory makes comptime values real bytes, so the bridge disappears. (JIT-native comptime was rejected: it breaks cross-compilation — host vs target layout — and loses the sandbox. A flat-memory VM keeps both while getting native bytes + speed.)

Next action (2026-06-18): Phase 3 is UNDER WAY. The VM now hosts the first read-only reflection readers — find_type(name: StringId) -> TypeId and type_field_count(t: TypeId) -> i64 — bound exactly like the intern/text_of seed (a type handle is a plain u32 TypeId, so the calls stay clean scalar host-calls). Example 0628 chains intern → find_type → type_field_count, VM-HANDLED natively. Parity 689/689 (gate ON and OFF), VM unit test added. Phase 1.final op-porting was already complete (the VM covers scalars/control-flow/aggregates/strings/optionals/enums, calls+recursion, the implicit context + full allocator protocol, globals, failables + return traces); both comptime call sites route through the VM with legacy fallback. Forward (P3.2): more read-only readers on the same TypeId-handle shape (type_name, field_name, field_type, kind queries), then register_struct (the first MUTATING fn — mints a TypeId; resolve the mutable-table / host-ABI-vs-target-ABI boundary deliberately). Re-expressing declare/define/type_info as sx (the metatype, which runs at LOWERING time) needs the VM hardened against malformed lowering-time IR first — keep it on the legacy path until then. Phase 2 (bytecode) is the orthogonal speed work. Decision recorded: find_type returns a non-optional TypeId using the unresolved (0) sentinel, NOT ?Type (a Type value is .any-typed, which the VM doesn't represent, and an optional can't cross the eval bridge) — see PLAN-COMPILER-VM.md Phase 3 progress note. Build/verify: zig build && zig build test (689, gate OFF). Run the corpus ON the VM: zig build test -Dcomptime-flat (the build flag) OR env SX_COMPTIME_FLAT=1. Coverage trace: SX_COMPTIME_FLAT_TRACE=1.

(superseded) prior weld resume

Phase 1 done; Phase 2 welded structs were working via reflection + memory-order validation (the computeWeldPlan/byte-blob "GEP engine" was explored + DROPPED even earlier). A welded Name :: struct abi(.zig) extern compiler { … } declared fields in the compiler type's MEMORY order; the compiler reflected the bound Zig type and VALIDATED the header. This whole mechanism is now being stripped — see the banner.

⚠ Snapshot workflow: use -Dname=examples/NNNN-foo.sx[,…] -Dupdate-goldens to regenerate ONLY the named example(s) — a full -Dupdate-goldens re-runs all ~690 and a flaky/host-divergent example (AOT/cross-arch) can clobber good snapshots. See CLAUDE.md → Snapshot integrity.

Last completed step

Phase 2 — welded structs by reflection + memory-order validation (byte-identical, no GEP engine). A welded struct abi(.zig) extern compiler { … } now works end-to-end as a byte-identical mirror of the bound Zig type.

Design (locked, supersedes the byte-layout-override plan):

• The sx header declares fields in the compiler type's MEMORY order. The compiler REFLECTS the bound Zig type — field names from @typeInfo, offsets from @offsetOf, size from @sizeOf — and validates the header matches. Nothing is maintained by hand; a types.zig change re-reflects on the next compiler build.
• On pass it's an ORDINARY struct whose natural layout already equals the Zig layout → @ptrCast to the compiler type + deref is byte-identical. No byte-blob, no index/remap tables, no reorder, no special LLVM path.
• Loud, precise diagnostics on any drift: field not found (+ memory order), wrong field order at position N (+ expected memory order), type layout mismatch (field size), layout mismatch (total size / count).

What changed from the dropped plan:

• compiler_lib.zig: weldStruct now REFLECTS field names (@typeInfo) and bakes bound_types fields in ascending-OFFSET (memory) order — no hand-listed names. Deleted computeWeldPlan/WeldPlan/WeldElement. validateStructLayout checks the sx header against the memory-ordered registry.
• nominal.zig validateWeldedStruct: renders the precise diagnostics (+ weldedFieldOrderStr).
• Examples: 0627 (StructInfo in memory order, byte-identical, usable); 1186 (source-order StructInfo → wrong-field-order diagnostic). 1183 message refreshed.
• zig build + zig build test green (692 corpus, unit tests pass).

Earlier — Phase 2.1 (weld-plan layout math, now removed)

The weld-plan offset math + StructInfo registered. Was the core of the byte-layout-override engine; superseded by the reflection+validation design above.

Decision (locked 2026-06-17): full byte-layout weld — a welded sx struct is laid out byte-identically to the bound Zig type (Zig's @offsetOf, reordering + padding included), so it passes to a Zig handler as raw memory with zero marshalling. (The alternative — handlers reading interp Value aggregates logically, no layout override — was rejected; welded types must also be usable as runtime data, and the design wants the literal byte weld.)

• Measured: Zig reorders StructInfo to fields@0, name@16, nominal_id@20, is_protocol@24, size 32 — vs sx-natural name@0, fields@8, … So the override is genuinely required (Field's two-u32 natural layout was the easy case).
• compiler_lib.zig: registered StructInfo (weldStruct, the second bound_types entry). Added WeldElement / WeldPlan + computeWeldPlan(alloc, fields, total) — pure: orders fields by ascending byte offset, inserts padding elements for gaps + the alignment tail, and builds the sx-field → LLVM-element remap. This is what the LLVM type builder + struct-GEP sites will consume.
• Unit-tested (compiler_lib.test.zig): Field → identity plan (2 elems, no pad); StructInfo → 5 elems [fields@0, name@16, nominal_id@20, is_protocol@24, pad@25..32], remap [1,0,3,2].
• zig build + zig build test green.

Earlier — Phase 1 polish (comptime-only enforcement)

A RUNTIME call to a fn abi(.zig) extern compiler is a clean build-gating error instead of an undefined-symbol link failure.

• emitCall (src/backend/llvm/ops.zig): when the callee is compiler_welded AND the ENCLOSING function is not is_comptime (i.e. genuine runtime code, not a #run/:: initializer wrapper whose LLVM body is dead), print a clear "comptime-only … cannot be called at runtime" error and set comptime_failed (the driver halts before object/JIT emission). The enclosing is_comptime guard is what keeps the legitimate #run use (example 0626) green.
• Corpus: examples/1185-diagnostics-weld-fn-runtime-call.sx (runtime intern(…) → clean error, exit 1, no link failure).
• zig build + zig build test green (458 unit + 690 corpus).

Earlier — fifth sub-step (host-call bridge)

A fn abi(.zig) extern compiler dispatches, under the comptime interpreter, to its registered Zig handler instead of dlsym.

• compiler_lib.zig: function registry — BoundFn { sx_name, handler }, bound_fns = intern(string)->StringId + text_of(StringId)->string (the string-pool round-trip), findFn, and FnHandler (*Interpreter, []Value -> Value). intern mutates via interp.mint orelse @constCast(&module.types) (the same mutable-table access the metatype mint path uses); text_of reads the const pool. Imports interp.zig (the compiler_hooks↔interp cycle pattern).
• IR Function gained compiler_welded: bool. declareFunction (src/ir/lower/decl.zig) sets it via weldedCompilerFn, which also VALIDATES: the bound lib must be compiler and the name must be on the function-export list — else a build-gating .err (no silent fall-through to dlsym).
• interp.call(): before the dlsym/extern path, a compiler_welded function routes to compiler_lib.findFn(name).handler(self, args) (clean bail off the export list).
• Corpus: examples/0626-comptime-weld-fn-intern-text-of.sx (#run text_of(intern("hello, compiler")) folds to a string constant → prints it); examples/1184-diagnostics-weld-fn-unexported.sx (unexported welded-fn name → build error). findFn lookup unit-tested.
• Runtime-call rejection is NOT yet clean — welded fns are comptime-only; a RUNTIME call would emit a reference to a non-existent extern symbol → a loud LINK error (not silent, but not a tidy diagnostic). The examples call welded fns only inside #run. A dedicated "comptime-only symbol" emit diagnostic is the immediate follow-up.
• zig build + zig build test green (458 unit tests + 689 corpus).

Earlier — fourth sub-step (welded-struct layout validation)

A struct abi(.zig) extern compiler { … } is validated against the binding registry as a header checked against the implementation.

• compiler_lib.zig: validateStructLayout(bt, sx_fields, total) — pure, returns the first LayoutMismatch (field count / name / size / total) or null. Plus lib_name = "compiler" and SxField. Unit-tested (faithful Field passes; each drift flagged as the right variant).
• registerStructDecl (src/ir/lower/nominal.zig): for sd.abi == .zig, validateWeldedStruct checks the bound lib is compiler, the name is on the export list (findType), and the sx layout (field names + typeSizeBytes + total) matches the welded type — emitting a build-gating .err (good span into the struct body) on any failure. No silent reinterpretation.
• #library "compiler" is the comptime-only internal surface, NOT a dylib — src/main.zig's dlopen walker skips it (was emitting a spurious libcompiler.so load warning).
• Corpus: examples/0625-comptime-weld-struct-field.sx (faithful Field welds, validates, usable as data → name=7 ty=3); examples/1183-diagnostics-weld- struct-field-count.sx (one-field Field → build-gating field-count diagnostic).
• Offset-override / GEP emission for non-natural Zig layouts is NOT here — it isn't exercised by Field (two u32s = natural layout coincides with the weld). It arrives with StructInfo in Phase 2 (slices/reordering), where the bound offsets actually differ from the sx-natural ones. The validation already checks per-field size + total, so a layout drift is caught even before the override engine exists.
• zig build + zig build test green (456 unit tests + 687 corpus).

Earlier — third sub-step (binding registry)

The binding registry (welded-type lookup, layout baked from the real Zig type).

• New src/ir/compiler_lib.zig — the compiler library's binding registry, the curated safety boundary. BoundType { sx_name, size, alignment, fields: []FieldLayout{name, offset, size} }; weldStruct bakes the layout from a real Zig struct via @sizeOf/@alignOf/@offsetOf at compiler-build time (a sx-field-count mismatch is a @compileError, never a silent truncation). bound_types exports Field (welded to types.TypeInfo.StructInfo.Field — two u32s); findType(sx_name) ?*const BoundType is the lookup the welded-decl resolution path will consult (returns null off the export list — clean boundary, no silent default).
• Registered in the barrel (src/ir/ir.zig): compiler_lib + compiler_lib_tests.
• Tests (src/ir/compiler_lib.test.zig): findType("Field") equals the real StructInfo.Field @sizeOf/@alignOf/@offsetOf (8 bytes, two u32s at 0/4); an unexported name returns null. Break-verified (a wrong size → suite red, named ir.compiler_lib.test...).
• zig build + zig build test green (454 unit tests).

Earlier — second sub-step (struct-decl parse)

abi(.zig) extern <lib> PARSES on a STRUCT decl (parse-only, no semantics).

• ast.StructDecl gained abi: ABI + extern_lib: ?[]const u8 binding fields.
• parseStructDecl (src/parser.zig): after struct (and the #compiler check), parse an optional abi(...) then optional extern <lib> — same slot order as fn decls — and thread them onto the node. Ordinary structs are unperturbed (parseOptionalAbi/parseOptionalExternExport no-op when absent).
• Parser unit tests (src/parser.test.zig): Field :: struct abi(.zig) extern compiler { name: StringId; ty: Type; } parses with abi == .zig, extern_lib == "compiler", field list intact; a plain struct leaves abi == .default / extern_lib == null. Break-verified (a wrong-sentinel assert turns the suite red, confirming the test runs).
• zig build + zig build test green.

Earlier — first sub-step (fn decls) + the syntax pivot

abi(.zig) extern <lib> PARSES on a fn decl (parse-only). Plus the syntax pivot it required.

Syntax decision (locked 2026-06-17, supersedes the doc's original extern(.zig) <lib> single-qualifier form): the ABI/layout selector and the linkage keyword are two orthogonal annotations.

• abi(.x) — ABI / calling-convention annotation in the slot before extern/export. Unified replacement for callconv(...), which is removed. ABI = { default, c, zig, pure }: .c (C ABI), .zig (Zig-layout weld → the compiler library), .pure (naked asm), .default (unannotated). Can appear standalone (no extern) on any fn / fn-type / lambda.
• extern <lib> — linkage keyword + binding source (named library).

So a welded binding is text_of :: (id: StringId) -> string abi(.zig) extern compiler;.

What landed:

• AST (src/ast.zig): CallingConvention → ABI { default, c, zig, pure }; the call_conv field → abi: ABI on FnDecl / Lambda / FunctionTypeExpr.
• Lexer/token (src/token.zig, src/lexer.zig): kw_callconv → kw_abi, keyword string "callconv" → "abi".
• Parser (src/parser.zig): parseOptionalCallConv → parseOptionalAbi (parses abi(.c|.zig|.pure)); wired in the fn-decl postfix slot (before extern/export), the function-type-expr slot, and the lambda slot; isFunctionDef/hasFnBodyAfterArrow recognise kw_abi.
• AST→IR map (src/ir/type_resolver.zig, src/ir/lower/decl.zig, sema.zig, closure.zig): the AST .abi == .c reads kept their C-ABI meaning; the function-type resolver maps .zig/.pure → IR .default (no fn-pointer-type CC for those decl-level ABIs; neither occurs in a function-TYPE position yet).
• CC-mismatch diagnostic (src/ir/lower/expr.zig, src/sema.zig): the user-facing text callconv(.c) → abi(.c).
• sx migration: 52 .sx files callconv( → abi( (all were function-type callback annotations — none in the fn-decl postfix slot, so no reordering).
• Docs: readme.md, specs.md, the design doc, snapshots (0114 / 1104 / 1200) regenerated for the rename.
• Tests: parser unit tests in src/parser.test.zig — abi(.zig) extern <lib> on a fn decl (asserts abi == .zig, extern_export == .extern_, extern_lib == "compiler"); bare extern leaves abi == .default; standalone abi(.c) / abi(.pure). lexer/sema tests updated.

zig build + zig build test green (450/450 unit + 685 corpus).

Current state

Pivoted — see the banner + PLAN-COMPILER-VM.md. The items below are the weld machinery as it stands on reify HEAD (40d075c); they are the strip list for Phase 0, not the forward direction. The #library/abi/extern syntax stays; the weld semantics (layout reflection/validation, marshaling dispatch) go.

• compiler :: #library "compiler"; parses + is recognised as the comptime-only internal surface (never dlopen'd).
• abi(.zig) extern compiler STRUCTS: layout-validated against the registry (faithful → ok; drift → build-gating diagnostic). Field welds + usable.
• abi(.zig) extern compiler FUNCTIONS: dispatched under the comptime interp to their registered Zig handler (intern/text_of round-trip works); unexported names rejected at declaration. Comptime-only.
• A RUNTIME call to a welded fn is a clean build-gating error (comptime-only enforcement at emitCall); the legitimate #run/:: use stays green.
• The whole Phase 1 foundation (parse → registry → struct-layout validation → function host-call bridge → comptime-only enforcement) is in place for the two-u32 Field case + the two string readers.
• Deferred: offset-override / LLVM byte-offset GEP for non-natural layouts (needed by StructInfo's slice field, Phase 2).

Next step — execute PLAN-COMPILER-VM.md

The weld is being stripped. The next step is Phase 0 of PLAN-COMPILER-VM.md — remove the weld / serialize / marshal machinery (compiler_lib.zig reflection+validation, nominal.zig validateWeldedStruct, the compiler_welded dispatch, the weld examples/diagnostics 0625/0627/1183/1184/1185/1186), keeping the #library/abi/extern syntax. Then Phase 1 (flat-memory value model). The weld-era "next step" below is obsolete — kept only as a record of what the weld surface was about to do.

(obsolete) weld-era next step

Welded structs were byte-identical mirrors, so the API surface was set to grow:

• Bind register_struct / find_type over the host-call bridge (compiler_lib.zig bound_fns, like intern/text_of). register_struct takes a welded StructInfo and mints a real TypeId (guarded: dup field names, kind well-formedness — the checks define does today). Because the welded StructInfo is byte-identical, the handler can read it as the real Zig *StructInfo (cast + deref) rather than marshalling a Value field-by-field — the payoff of the byte-weld. find_type(StringId) -> ?Type reads the table. Prove: build a struct programmatically + round-trip a source one.
• Re-express type_info/define (struct) as sx over register_struct/ find_type; migrate examples/0622; delete the bespoke struct interp arms (defineStruct / the reflectTypeInfo struct path).

Then Phase 3+: widen the welded types to EnumInfo/TaggedUnionInfo/TupleInfo (optional fields → sentinels) — each just needs an sx header in the compiler type's memory order + the matching register_* fn. Finally migrate BuildOptions to abi(.zig) extern compiler (re-home the #compiler registry) and delete #compiler.

Note: a welded struct with an ?T / union(enum) field (e.g. EnumInfo's backing_type: ?TypeId, explicit_values: ?[]const i64) is the next layout wrinkle — the sx header must mirror Zig's optional/union representation. Handle when reached (sentinels or accessor fns; see the design doc Risks).

Known issues

• None for this stream. (Metatype's deferred enhancement is issue 0141 — comptime List growth; orthogonal, see current/CHECKPOINT-METATYPE.md.)

Log

• Phase 3 P3.1 (VM plan) — first read-only reflection readers: find_type + type_field_count (2026-06-18). Two more compiler-library fns, bound the same way as the intern/text_of seed (added to compiler_lib.bound_fns for the legacy handler + the welded-decl export check, AND to Vm.callCompilerFn for the native flat-memory path — NO marshaling). A type handle is a plain u32 TypeId (like StringId), so both keep the seed's clean scalar shape: find_type(name: StringId) -> TypeId (TypeTable.findByName, unresolved/0 if absent) and type_field_count(t: TypeId) -> i64 (a NEW TypeTable.memberCount query — struct/union/ tagged-union fields, enum variants, array/vector length — called by BOTH paths so they can't drift; bails loudly, never a silent 0). New example 0628-comptime-compiler-find-type chains intern → find_type → type_field_count (and a not-found lookup → 0), both folded at #run, both VM-HANDLED natively (trace confirms no fallback). VM unit test added (find_type + type_field_count, struct found → 3 fields, missing → unresolved). Parity 689/689 (gate ON and OFF). Decision (resolves the plan's find_type → ?Type sketch): return a NON-optional TypeId with the unresolved (0) sentinel for not-found, NOT ?Type — a Type value resolves to .any (which the flat-memory VM doesn't represent) and an optional can't cross the legacy↔VM eval boundary; unresolved is the project-blessed unmistakable "no type" marker. Forward (P3.2): more readers on the same handle shape (type_name/field_name/field_type/kind), then register_struct (first mutating fn).
• VM robustness — Frame bounds-check; lowering-time #insert wiring explored + reverted (2026-06-18). Explored wiring the VM at the LOWERING-time comptime site (evalComptimeString, the #insert string fold). 12/13 #insert examples ran on the VM with parity, but 0737 (an #insert of an unresolved secret()) CRASHED the VM (SIGABRT): lowering-time IR can be malformed (a ret Ref.none from the unresolved name) and Frame.get panicked on the out-of-range index. Decision: reverted the lowering-time wiring — unlike the emit-time folds (fully lowered IR), lowering-time IR can be erroneous, and hardening the VM against ALL malformed IR (every ref_types[...] / aggType access, not just Frame) is out of scope here. The emit-time sites already give full corpus coverage. KEPT the defensive fix regardless (CLAUDE.md "never crash"): Frame.get/set now bounds-check and flip a bad_ref flag; the run loop bails (badRef) instead of panicking. Unit test added (malformed ret Ref.none → bail, not crash). Parity 688/688 both ways.
• Phase 3 SEED (VM plan) — compiler-call path: intern/text_of native on the VM (2026-06-18). invoke now dispatches a welded compiler-library fn (gated on compiler_welded) to Vm.callCompilerFn, serviced NATIVELY on flat memory (no legacy Interpreter): intern(string)->StringId reads the flat-memory string bytes and internStrings into the const-cast table (pool-only — doesn't touch type layout, so cached sizes stay valid); text_of(StringId)->string materializes the pooled text back into flat memory. Unlocked 0626; the ONLY remaining const-init fallback is now the inline-asm global (1654). Parity 688/688 (gate ON and OFF); unit test added. This is the mechanism Phase 3 grows — the next compiler fns (find_type, register_struct, reflection readers) bind the same way (flat-memory pointer in, handle/pointer out, no marshaling).
• Phase 1.final step 9 (VM plan) — -Dcomptime-flat build flag (the "swap behind a build flag" step) (2026-06-18). Added the -Dcomptime-flat build option (build.zig → a build_opts options module on mod; emit_llvm.init reads build_opts.comptime_flat or SX_COMPTIME_FLAT env). This is the plan's "reach parity → swap behind a build flag → delete the old path" mechanism. zig build test -Dcomptime-flat runs the FULL corpus on the VM (688/0). Verified the flag toggles the binary: flag-built sx reports VM HANDLED with no env var; default-built does not. Default OFF — zig build test unchanged (688/0). Env var still works for ad-hoc runs. Next (forward): Phase 2 (bytecode) / Phase 3 (compiler-API on flat memory); eventual default-flip + legacy deletion.
• Phase 1.final step 8 (VM plan) — wire the #run side-effect path + trace-clear-on-fallback (2026-06-18). Wired the SECOND comptime call site (runComptimeSideEffects, top-level #run <expr>;) through tryEval with legacy fallback, mirroring the const-init fold. tryEval now handles void/noreturn entries (→ .void_val) so a void side-effect doesn't bail at the result conversion. Fixed a trace-corruption the new site exposed (1035): a side-effect that pushes return-trace frames and then bails (e.g. on print) had the legacy re-run DOUBLE-push them (sx_trace_push is a side effect on the shared buffer). Both wiring sites now sx_trace_clear() right before the legacy fallback, discarding the VM's partial pushes. Parity 688/688 (gate ON and OFF). Most side-effects still bail (print/global_addr/call_builtin) → legacy, but the path is now uniform. All comptime evaluation routes through the VM-with-fallback.
• Phase 1.final step 7 (VM plan) — is_comptime + failable/error cluster + signed-load fix; coverage 31→36 (2026-06-18). is_comptime → 1 (unlocked 1030). Ported the failable/error-channel cluster (1037 escape, 1038 handled): kindOf(error_set)→word, regToValue bridges TUPLES (the failable (value…,tag) shape checkComptimeFailable reads), trace_frame packs (func_id<<32|span.start) from a new call_stack (pushed by invoke/runEntry), and sx_trace_push/sx_trace_clear serviced NATIVELY (the VM calls the real sx_trace.c functions linked into the compiler, so the return-trace buffer is populated identically to legacy). raise/catch/or now run on the VM. Surfaced + fixed a real GENERAL bug: readField was ZERO-extending signed sub-64-bit loads, so a stored i32 -1 reloaded as 0xFFFFFFFF (+4.29e9) and < 0 was false — silently hiding raise error.Bad; now SIGN-extends i8/i16/i32/isize (gate-ON parity confirms it's a strict fix; unit test added). VM HANDLES 36 corpus const-inits (was 31); parity 688/688 (gate ON and OFF). Only 2 fallbacks remain, both principled: intern (0626, welded compiler-API fn — Phase 3) + inline-asm global (1654). Forward work: Phase 2 (bytecode), Phase 3 (compiler-API on flat memory).
• Phase 1.final step 6 (VM plan) — real default context + call_indirect + func_ref + global_get; coverage 27→31 (2026-06-17). Per the user's direction ("the VM can set up a default context"), runEntry now materializes the REAL default context instead of a zeroed one. The implicit-ctx param is an opaque *void, so materializeDefaultContext finds the __sx_default_context global and lays its initializer ({ {null, alloc_fn, dealloc_fn}, null }, the CAllocator thunk func-refs) into flat memory via a new recursive layoutConst. With func_ref (function value encoded FuncId.index()+1, reserving word 0 for the null fn-ptr) and call_indirect (decode word → FuncId → dispatch; 0 → bail) ported, the whole allocator protocol runs on the VM: context.allocator.alloc_bytes → call_indirect → thunk → CAllocator.alloc_bytes → libc_malloc → native flat malloc. Unlocked 0606 (string global). Also: global_get lazily evaluates a comptime global's comptime_func (memoized) — unlocked CT_CHAIN; field access (fieldOffset/struct_get) handles string/slice {ptr@0,len@8} fat pointers (needed by alloc_string); regToValue maps function-typed words → .func_ref (kept 1128's rejection byte-identical). Native malloc is still required (the thunk bottoms out at it; a host pointer can't be used with flat-memory load/store). VM HANDLES 31 corpus const-inits (was 27); parity 688/688 (gate ON and OFF). Unit tests: global_get, func_ref+call_indirect. Remaining fallbacks (7): .unsupported aggregates (3× — 1037/1038), extern/builtin intern+asm (2×), trace_frame, is_comptime.
• Phase 1.final step 5 cont. (VM plan) — libc memory builtins + f32 fix; coverage 16→27 (2026-06-17). Identified the dominant fallback (call to extern/builtin) as 11× malloc (0604) + 1× intern. Modeled a curated set of libc MEMORY builtins natively on flat memory (Vm.callMemBuiltin): malloc/calloc → allocBytes (16-aligned, 256-MiB cap → bail), free → no-op, memcpy/memmove/memset on flat bytes — sandboxed (no host heap/dlsym), target-aware; the computed result is byte-identical to legacy (which calls real libc). This surfaced a real latent f32 bug: float registers hold f64 bits, but f32 MEMORY is the 4-byte single — readField/writeField were truncating the f64 bits (writing zeros for 1.0); now they @floatCast on f32 load/store (mirrors legacy storeAtRawPtr). Result: VM HANDLES 27 corpus const-inits (was 16); parity 688/688 (gate ON and OFF). Unit tests added (f32 round-trip; malloc → usable flat memory). Next: the kindOf .unsupported aggregates (3×), global_get (2×), the rest.
• Phase 1.final step 5 (VM plan) — implicit-context materialization; coverage 0→16 (2026-06-17). tryEval now MATERIALIZES the implicit ctx instead of skipping it: a has_implicit_ctx comptime entry (sole param *Context) gets a zeroed Context of the right size/align in flat memory, its address passed as arg 0. Const bodies that ignore the ctx run; a body that uses the allocator hits unported call_indirect → bails → legacy. No func-ref materialization needed (handled bodies don't read ctx contents; parity is the guard). Fixed a real bug surfaced by the coverage pass: storing a null non-pointer optional (the null_addr sentinel) into an aggregate slot OOB-bailed — writeField now ZEROES the destination for a null_addr aggregate source (= none/empty); unit-test regression added. Result: VM HANDLES 16 corpus const-inits (was 0); parity 688/688 both gate ON and OFF. Next: port the ops the trace names — call_builtin/compiler_call/ extern (13×, via the bridge), kindOf .unsupported aggregates (3×), global_get (2×), func_ref / call_indirect / trace_frame / is_comptime.
• Phase 1.final steps 1–4 (VM plan) — host wiring landed; coverage measured (2026-06-17). (1) Hardening: Machine.readWord/writeWord/bytes now return error.OutOfBounds (null / out-of-range / oversized / overflow-safe) instead of assert-panicking; OutOfBounds added to Vm.Error; try threaded through every helper + exec arm + the bridge. New unit tests (accessor OOB returns; null-deref → tryEval null, not a crash). (2) Implicit context: tryEval returns null for has_implicit_ctx funcs (legacy fallback) — conservative; full ctx materialization deferred to step 5. (3) Wiring: const-init fold in emit_llvm.zig emitGlobals is (if comptime_flat) tryEval else null) orelse interp.call(...), gated by env SX_COMPTIME_FLAT (read once into LLVMEmitter.comptime_flat). Default OFF. (4) Parity + coverage: gate ON → full corpus byte-identical (688, 0 failed) + manual 0605/0606/0607 byte-identical. Finding: 0 of 37 measured corpus const-inits are VM-handled — ALL are has_implicit_ctx-gated. Added a coverage-trace facility (comptime_vm.last_bail_reason
  • env SX_COMPTIME_FLAT_TRACE). Next: step 5 = implicit-context materialization (the unblocker), then port the deferred ops. 688 corpus green (gate OFF).
• Phase 1.final start (VM plan) — wiring entry point tryEval (2026-06-17). comptime_vm.tryEval(gpa, module, func_id) ?Value runs a comptime function entirely on the VM, returns a legacy Value (deep-copied to gpa) or null to fall back. Unit-tested (pure 6*7 → 42; unbox_any → null). NOT yet routed into the host: needs (1) panic→error hardening of Machine accessors so arbitrary funcs bail instead of crashing, (2) implicit-ctx handling, (3) wiring at emit_llvm const-init behind SX_COMPTIME_FLAT, (4) corpus parity run. See PLAN-COMPILER-VM.md Phase 1.final. 688 corpus green.
• Phase 1 sub-step 1.5b (VM plan) — Reg↔Value boundary bridge (2026-06-17). Builtin/compiler_call/extern handlers are coupled to the legacy Interpreter, so the wiring will use WHOLE-FUNCTION fallback (VM runs pure functions; bail → legacy re-runs the whole eval). Built the boundary bridge that enables it: valueToReg (Value arg → Reg, aggregates into flat memory) + regToValue (VM result → Value, deep-copied). Covers scalars/strings/structs; other shapes bail. Transitional. Round-trip unit-tested. 688 corpus green. Next: the wiring (flag + route a comptime entry through the VM with legacy fallback).
• Phase 1 sub-step 1.5 (VM plan) — direct call + stack-lifetime change (2026-06-17). Vm gained module (callee resolution) + depth/max_depth guard. call marshals arg Refs → Reg and recursively runs the callee; aggregates pass as Addrs over shared flat memory. Frame no longer reclaims the machine on exit (else a returned aggregate Addr dangles) — allocations live to Vm.deinit. Extern/builtin callees bail (1.5b). Unit-tested: direct call (142), recursion sum(0..n) (15/55). 688 corpus green. Next: 1.5b (call_builtin/compiler_call/extern), then hybrid wiring.
• Phase 1 sub-step 4d (VM plan) — deref/addr_of; pivot decision (2026-06-17). Ported addr_of (pass-through) + deref (readField through pointer), unit-tested (deref *i64 → 77, addr_of struct + field → 80). DECIDED to stop porting rarer ops (tagged-union payload/any/closures) blind — their byte semantics are ambiguous without real call sites — and pivot to CALLS (sub-step 1.5: call, then builtin/compiler) + HYBRID WIRING (-Dcomptime-flat → VM with legacy fallback on error.Unsupported), so the VM runs the real corpus and surfaces exactly what's needed. Key design point for calls: aggregate-return lifetime → drop per-frame stack reclaim (let a comptime eval's allocations live to Vm.deinit). 688 corpus green. See PLAN-COMPILER-VM.md decision block.
• Phase 1 sub-step 4c (VM plan) — optionals + payloadless enums (2026-06-17). kindOf: enum → word; ?T → word (pointer-child, null==0) or {T@0,i1@sizeof(T)} aggregate. Ported optional_wrap/unwrap/has_value/coalesce (optChildIsPtr/optHas; const_null reads as none) + payloadless enum_init/enum_tag. Unit-tested (?i64 → 91, ?*i64 null==0 → 99, enum tag → 11). 688 corpus green. Next: 4d (tagged unions, any, closures).
• Phase 1 sub-step 4b (VM plan) — slices + strings on flat memory (2026-06-17). {ptr@0(pointer_size), len@8(i64)} fat pointers (kindOf: string/slice → aggregate). Ported const_string (text+NUL + fat pointer in flat memory), length/data_ptr, array_to_slice, subslice, index-through-slice (elemAddr loads .ptr), and str_eq/str_ne (memcmp). Unit-tested (str length+eq/ne, array→slice index sum=23, subslice sum=43). 688 corpus green. Next: 4c (optionals/enums/any/closures).
• Phase 1 sub-step 4a (VM plan) — tuples + arrays on flat memory (2026-06-17). kindOf widened (tuple/array → aggregate). Ported tuple_init/tuple_get (tupleFieldOffset), index_get/index_gep (elemAddr = base + idx*elem_size over array/pointer/many_pointer; slice/string bases bail), length on array values. Unit-tested (mixed tuple, [3]i64 index sum=42, length=3). 688 corpus green. Next: sub-step 4b (slices/strings, then optionals/enums/any/closures).
• Phase 1 sub-step 3 (VM plan) — memory + structs on flat memory (2026-06-17). Vm gained optional table: *const TypeTable (target-aware layout). Ported alloca/load/store + struct_init/struct_get/struct_gep, laying structs out at the table's natural offsets. Value model: scalar/pointer → register word; struct → lives in flat memory, its value IS its address (read→addr, write→memcpy), so nested structs compose and struct_gep = base+offset. kindOf bails loudly on not-yet-ported types. Addr-based values survive allocator realloc. Unit-tested (struct round-trip, alloca+gep+store+load, nested struct). 688 corpus green. Next: sub-step 4 (arrays/slices/strings/optionals/enums/tuples/any/closures, then calls).
• Phase 1 sub-step 2 (VM plan) — flat-memory executor: scalars + control flow (2026-06-17). Added Vm to comptime_vm.zig: walks the same IR Inst over flat-memory frames (register Reg = scalar bits or Addr), mirroring the legacy interp's scalar semantics (i64 wrapping/signed, f64). Ported constants, arithmetic, comparison, logical, conversions, terminators (br/cond_br/ret/ret_void) and block_param; every other op bails loudly (error.Unsupported + op name in detail). Unit-tested on hand-built tiny IR (Fb builder): int add, f64 arithmetic, cond_br selection, a block-param loop, div-by-zero + unsupported-op bails. Corpus untouched (688 green). Next: sub-step 3 (memory + aggregates on flat memory, where target-aware layout enters).
• Phase 1 sub-step 1 (VM plan) — flat-memory machine substrate (2026-06-17). New src/ir/comptime_vm.zig: Machine (linear byte memory + bump/stack allocator with mark/reset, scalar readWord/writeWord 1/2/4/8 LE, bytes views, addr 0 reserved as null_addr) + Frame (Ref-indexed register file, stack reclamation on deinit). Reg = raw u64 (immediate scalar OR Addr). Unit-tested (comptime_vm.test.zig), registered in the barrel; standalone — the legacy interpreter stays live, corpus untouched (688 green). Next: sub-step 2 (executor + scalar/branch ops over the same IR). Also removed the "~500 lines / split step" rule from CLAUDE.md per request.
• Phase 0 (VM plan) — struct-weld stripped; intern/text_of bridge kept (2026-06-17). Removed the struct-weld registry from compiler_lib.zig (weldStruct/bound_types/BoundType/FieldLayout/findType/SxField/ LayoutMismatch/validateStructLayout), validateWeldedStruct/weldedFieldOrderStr
  • the sd.abi == .zig call from nominal.zig, the struct-weld unit tests, and examples 0625/0627/1183/1186. KEPT (decision) the intern/text_of function host-call bridge — a clean scalar dispatch, not weld/serialize/marshal, the Phase-3 compiler-call seed — so weldedCompilerFn, the compiler_welded dispatch, the emitCall comptime-only gate, the #library/abi/extern syntax, and examples 0626/1184/1185 remain. zig build test green (688 corpus, 0 failed). Next: Phase 1 (flat-memory value model) per PLAN-COMPILER-VM.md.
• DIRECTION CHANGE — pivot off the byte-weld to a flat-memory bytecode VM (2026-06-17). Decided the weld + serialization/marshaling bridge is the wrong direction (it hand-marshals onto a comptime value model that isn't bytes — exactly what the design set out to kill). New foundation: a bytecode VM over flat memory so comptime values are native bytes; the compiler-API then rides on it via direct memory (no weld/validation/marshaling). JIT-native comptime was weighed and rejected (breaks cross-compilation, loses the sandbox). Wrote current/PLAN-COMPILER-VM.md (Phase 0 strip → Phase 1 flat-memory value model → Phase 2 bytecode → Phase 3 compiler-API on flat memory). Banner added to design/comptime-compiler-api.md (superseded). Reverted the session's uncommitted register_struct/find_type marshaling experiment back to reify HEAD (40d075c). No code stripped yet — Phase 0 is the next action.
• Phase 2 — welded structs by reflection + memory-order validation. Dropped the byte-layout-override engine (computeWeldPlan / offset-ordered LLVM struct / byte-blob — all explored, all unnecessary). Instead: the sx header declares fields in the compiler type's memory order; the compiler reflects the bound Zig type (@typeInfo/@offsetOf/@sizeOf) and validates the header matches with loud diagnostics (field-not-found, wrong-order+expected-order, size mismatch). On pass it's an ordinary byte-identical struct — cast + deref just works. Examples 0627 (usable) / 1186 (wrong-order diagnostic). Suite green (692).
• Phase 2.1 — weld-plan layout math (REMOVED). The byte-layout-override math; superseded by the reflection+validation design and deleted.
• Phase 1 polish — comptime-only enforcement. A runtime call to a welded fn is a clean build-gating error (emitCall gate, guarded by enclosing-is_comptime so #run/:: uses stay green), not a link failure. Example 1185. Build + suite green (458 unit, 690 corpus).
• Phase 1.1 fifth sub-step — host-call bridge (welded functions). compiler_lib function registry (intern/text_of) + findFn; IR Function compiler_welded flag set/validated in declareFunction (weldedCompilerFn); interp.call() dispatches welded calls to the Zig handler. Examples 0626 (round- trip) + 1184 (unexported-fn diagnostic); findFn unit-tested. Runtime-call clean rejection deferred (loud link error today). Build + suite green (458 unit, 689 corpus).
• Phase 1.1 fourth sub-step — welded-struct layout validation. validateStructLayout (pure, unit-tested) + validateWeldedStruct wired into registerStructDecl: a struct abi(.zig) extern compiler is validated against the registry (lib == compiler, name exported, layout matches) with build-gating diagnostics. #library "compiler" no longer dlopen'd. Examples 0625 (faithful Field) + 1183 (field-count mismatch diagnostic). Offset-override/GEP deferred to Phase 2 (not exercised by Field's natural layout). Build + suite green (456 unit, 687 corpus).
• Phase 1.1 third sub-step — binding registry. New src/ir/compiler_lib.zig: the compiler lib's welded-type registry; Field welded to StructInfo.Field with layout baked from the real Zig type (@offsetOf/@sizeOf/@alignOf); findType lookup proven by unit test (+ null off the export list). Standalone island — not yet consumed by lowering. Build + suite green (454 unit tests). Break-verified.
• Phase 1.1 second sub-step — struct-decl binding parses. ast.StructDecl gained abi + extern_lib; parseStructDecl parses abi(.zig) extern <lib> after struct. Parser unit tests (welded Field + plain struct), break-verified. Build + suite green. Parse-only sub-step (fns + structs) of Phase 1.1 complete.
• Phase 1.1 first sub-step + callconv→abi unification. Parsed abi(.zig) extern <lib> on fn decls; unified callconv into abi(.c|.zig|.pure) (removed the callconv keyword), migrated 52 sx files + compiler diagnostics + docs + snapshots. Build + suite green. The original design's extern(.zig) single qualifier was split into abi(.zig) (ABI/layout, before extern) + extern <lib> (linkage + source) — recorded in the design doc's syntax-decision note.