sx/current/PLAN-COMPILER-VM.md

PLAN — Comptime Bytecode VM + flat memory (then re-home the compiler-API on it)

Direction change (2026-06-17). The comptime compiler-API stream pivots off the byte-weld. The weld (sx structs whose layout is validated to mirror the compiler's Zig types) + the serialization / marshaling bridge at the call boundary is the wrong direction — it bolts a parallel layout regime and hand-built byte-copies onto a comptime value model that fundamentally isn't bytes. We strip it and build the right foundation: a bytecode VM over flat, byte-addressable memory, where comptime values ARE native bytes (like runtime). On that base the compiler-API needs no weld, no validation, no marshaling — the compiler's own types are read/built directly as memory and its functions take/return real pointers.

Supersedes the build order in design/comptime-compiler-api.md (kept for history). This is the active plan for the stream. Branch: reify.

Why

src/ir/interp.zig is a tree-walking interpreter over the SSA IR that represents every value as a tagged Value union (int, float, aggregate: []const Value, type_tag, heap_ptr, …). Two consequences:

1. Slow. Per-value boxing in a tagged union; per-op switch over Inst; an aggregate is a heap []const Value, walked element-by-element.
2. Not native memory. A struct value is []const Value (tagged unions), NOT the struct's bytes. So a comptime @ptrCast(*StructInfo) reads the Value union's memory, not a StructInfo — which forced the whole weld+marshal detour.

Make comptime values native bytes in a flat memory and both problems dissolve: structs/arrays/slices are their bytes at natural layout (no weld), the compiler's own records are directly addressable (no marshal), and a bytecode loop over flat memory is fast.

End state

• Comptime execution = a bytecode VM over a flat linear memory (real host-allocated bytes; layout is target-aware via the type table's sizes). Values are bytes at addresses plus a scalar register file. No tagged Value union.
• The comptime compiler-API: the compiler exposes its real types + functions to comptime sx. sx reads/builds them as native memory and calls compiler functions by pointer. No abi(.zig) weld, no validateStructLayout, no register_struct field-by-field marshaling — gone.
• declare/define/type_info and #compiler/BuildOptions ride this one mechanism; the bespoke interp arms are deleted.
• ONE evaluator at the end — non-negotiable. The legacy tagged-Value interpreter (interp.zig) is DELETED. We do NOT ship both permanently. "Dual-path" (a compiler-API fn with both a legacy compiler_lib handler AND a VM-native impl) and the emit-time legacy fallback are transitional only — scaffolding while the VM reaches parity at BOTH comptime sites (emit time AND lowering time). The flag -Dcomptime-flat is the swap mechanism; once the VM runs everywhere with parity, the flag, the fallback, and interp.zig all go. Any "VM-only at emit, legacy at lowering" split is a waypoint, never the destination.

Principles (hold at every step)

• Green at every step. zig build && zig build test pass after each sub-step. The existing tagged-Value interpreter stays the live evaluator until the VM reaches corpus parity; swap behind a build flag, then delete the old path.
• Target-aware, not host-baked. Flat-memory layout uses the type table's target sizes (pointer_size, typeSizeBytes/offsets), NEVER host @sizeOf. This is what keeps cross-compilation correct (the JIT-comptime alternative could not).
• Sandboxed. Flat-memory accesses are bounds-checked; step/call-depth budgets remain; an OOB / bad access traps to a build-gating diagnostic with a source span — never a compiler-process crash.
• No silent fallbacks (per CLAUDE.md): an unhandled op / shape bails loudly with a named reason, never a zero/default that looks like success.

Phases

Phase 0 — Strip the weld / serialize / marshal machinery

Delete the wrong-direction code so the VM builds on a clean base. Pure removal + corpus rebaseline; suite green.

• src/ir/compiler_lib.zig: the reflection (weldStruct / bound_types / FieldLayout / BoundType), the layout validation (validateStructLayout / LayoutMismatch / SxField). Decide the fate of the bound_fns host-call registry (intern/text_of handlers) — it is likely subsumed by the VM's compiler-call path in Phase 3, but intern/text_of may survive as the first such calls.
• src/ir/lower/nominal.zig: validateWeldedStruct + weldedFieldOrderStr + the sd.abi == .zig validation call in registerStructDecl.
• src/ir/interp.zig: the compiler_welded dispatch branch.
• src/backend/llvm/ops.zig: the emitCall comptime-only gate keyed on compiler_welded (re-derive the comptime-only guard from a non-weld signal if still needed).
• Corpus: retire / convert the weld examples + diagnostics — 0625, 0627 (welded struct), 1183, 1186 (weld-layout diagnostics), 1184/1185 (welded-fn). Keep 0626 (intern/text_of round-trip) only if it survives the new call path.
• Keep (re-evaluate in Phase 3), independent of the weld semantics: the #library "compiler" decl, the abi(.x) annotation + extern <lib> syntax, and the callconv → abi unification. These are surface syntax that may still serve the compiler-API; only the weld semantics are stripped here.

Verification: zig build test green with the weld machinery gone; the surviving syntax still parses (parser unit tests).

Phase 1 — Flat-memory value model (still IR-walking, no bytecode yet)

Introduce flat memory and move comptime values onto it, decoupled from bytecode so the value-model change is isolated. Each sub-step ports one op group and keeps the corpus green; the OLD tagged path stays behind a build flag (-Dcomptime-flat) until all groups land, then the shim is deleted.

1. Machine + scalars. A flat memory region (host []u8) with a stack (frames) + bump-allocated heap, and a scalar register file. Port int/float/bool/undef and arithmetic/compare/branch. Aggregates still go through a compat shim to the old representation.
2. Aggregates. Structs/arrays/tuples laid out in flat memory at target layout; port struct_init / struct_get / array / index_gep to read/write bytes at computed offsets.
3. Slices / strings. {ptr, len} fat pointers in flat memory.
4. Optionals / enums / tagged unions. Tag + payload bytes.
5. Pointers. alloca / store / load / GEP unified onto flat addresses; retire slot_ptr / heap_ptr / byte_ptr in favor of flat-memory addresses.
6. Closures. Fn id + captured env materialized in flat memory.
7. Extern / host calls. A struct arg is already bytes → pass its address; this removes most of marshalExternArg.
8. Reflection / minting. declare / define / type_info read flat-memory values; type-table mutation copies escaping data into compiler-owned memory at the boundary (lifetime), as today.

Verification: with -Dcomptime-flat the full corpus (currently 692) is byte-for- byte identical to the tagged path; then make flat the default and delete the shim.

Phase 2 — Bytecode

Compile a comptime function's IR → a compact bytecode and execute the bytecode instead of walking Inst. Pure encoding/speed; semantics identical to Phase 1. Land at least a minimal register-bytecode loop (the stream's stated goal is a bytecode VM); a fragment cache is optional follow-up.

Verification: corpus identical to Phase 1; comptime throughput measurably improved on a heavy-comptime micro-benchmark.

Phase 1.final — host wiring (the remaining integration)

The wiring ENTRY POINT exists: comptime_vm.tryEval(gpa, module, func_id) ?Value runs a comptime function entirely on the VM and returns a legacy Value, or null to fall back. Unit-tested (pure 6*7 → 42; unsupported → null). Remaining to actually route the host through it:

1. Panic→error hardening (prerequisite). Machine.readWord/writeWord/bytes currently assert (debug panic) on null/OOB. For arbitrary host functions to be safe, make them return error.OutOfBounds so a malformed run BAILS (→ null → legacy) instead of crashing the compiler. Ripples through readField/writeField/slice helpers (add try).
2. Implicit context. Host comptime functions may have has_implicit_ctx (param 0 = *Context); the legacy run materializes a default ctx. The VM run does not — so either materialize it too, or only route tryEval at funcs without implicit ctx.
3. Wire one site behind a flag/env (SX_COMPTIME_FLAT, → -Dcomptime-flat later): the const-init fold in emit_llvm.zig emitGlobals (result = tryEval(...) orelse interp.call(...)). Default off → corpus unaffected.
4. Parity + coverage. Run the corpus with the flag ON; results must be byte-identical to legacy. Measure how many comptime evals the VM already handles; the bail details name what to port next (tagged-union payload / any / closures / builtins).
5. Grow coverage (port the deferred ops + call_builtin/compiler_call via the bridge) until the VM is the default and the legacy path is deleted.

Status (2026-06-17): steps 1–4 DONE; step 5 = the next session.

• (1) Hardening — DONE. Machine.readWord/writeWord/bytes return error.OutOfBounds (null / out-of-range / oversized / overflow-safe) instead of asserting. OutOfBounds added to Vm.Error; try threaded through readField/writeField/optHas/makeSlice/sliceLen/sliceData/elemAddr and every exec arm + the bridge. New unit tests: hardened-accessor OOB returns, and a null-deref function → tryEval returns null (legacy fallback), not a panic.
• (2) Implicit context — DONE (materialized, 2026-06-17 step 5). Initially a conservative skip; now tryEval MATERIALIZES the implicit ctx: a comptime entry with has_implicit_ctx (whose sole param is the *Context) gets a zeroed Context of the right size/align allocated in flat memory, its address passed as arg 0. The common const body never reads the ctx; a body that USES the allocator loads a fn from it and call_indirects (unported) → bails → legacy. No func-ref materialization was needed: handled bodies don't read the ctx contents, and gate-ON corpus parity (688, 0 failed) empirically confirms no divergence. (A body that read+branched on a null allocator fn could in principle diverge; none does — parity is the guard.)
• (3) Wire one site — DONE. Const-init fold in emitGlobals is (if comptime_flat) tryEval(...) else null) orelse interp.call(...). Gated by env SX_COMPTIME_FLAT (a LLVMEmitter.comptime_flat field read once from std.c.getenv in init). Default OFF → corpus unaffected (688 green).
• (4) Parity + coverage — DONE. Gate ON: full corpus byte-identical (688, 0 failed); manual sx run of 0605/0606/0607/0608 byte-identical to gate-OFF. Coverage-trace facility in place (comptime_vm.last_bail_reason + env SX_COMPTIME_FLAT_TRACE, printing HANDLED / fallback+reason per init).
• (5) Implicit-context materialization + memory builtins + f32 — DONE; op-porting CONTINUES. Coverage climbed 0 → 16 → 27 handled corpus const-inits (fallbacks 22 → 11); parity stays 688/688 (gate ON and OFF) at every step. Landed, in order: implicit ctx materialized (→16); writeField null-aggregate fix (storing a null non-pointer optional null_addr sentinel into an aggregate slot OOB-bailed → now ZEROES the destination = none/empty; unit-test regression); curated libc MEMORY builtins on flat memory (Vm.callMemBuiltin: malloc/calloc → allocBytes 16-aligned & 256-MiB-capped, free → no-op, memcpy/memmove/memset on flat bytes — sandboxed, target-aware, result byte-identical to legacy; unlocked 0604's 11 comptime mallocs); and an f32 storage fix (float registers hold f64 bits, but f32 memory is the 4-byte single — readField/writeField now @floatCast instead of truncating the f64 bits, which had written zeros for 1.0; a real latent bug 0604 surfaced; unit tests added).
• (6) Real default context + call_indirect + func_ref + global_get — DONE. Coverage 27 → 31 handled (fallbacks 11 → 7); parity stays 688/688 both gate ON and OFF. Per the user's direction ("the VM can set up a default context"), runEntry now materializes the REAL default context (not a zeroed one): the implicit-ctx param is an opaque *void, so materializeDefaultContext finds the __sx_default_context global and lays its initializer constant ({ {null, alloc_fn, dealloc_fn}, null }, carrying the CAllocator thunk func-refs) into flat memory via a new recursive layoutConst. With func_ref (a function value encoded as FuncId.index() + 1 so word 0 stays reserved for the NULL function pointer — funcRefWord/funcRefToId) and call_indirect (decode the callee word → FuncId → dispatch; 0 → bail) ported, a comptime body that allocates via context.allocator now runs ENTIRELY on the VM: alloc_string → context.allocator.alloc_bytes → call_indirect → thunk → CAllocator.alloc_bytes → libc_malloc → the VM's native flat-memory malloc. Unlocked 0606 (string global via the allocator). Also: global_get lazily evaluates a comptime global's comptime_func (memoized in global_cache) — unlocked CT_CHAIN; struct field access (fieldOffset/ struct_get) now handles string/slice {ptr@0,len@8} fat pointers (needed by alloc_string's s.ptr/s.len); and regToValue maps a function-typed word back to .func_ref so a func-ref result serializes identically to legacy (kept 1128's rejection diagnostic byte-identical). Unit tests added (global_get, func_ref + call_indirect). Note: native malloc is still REQUIRED — the CAllocator thunk bottoms out at libc malloc, and the VM can't use a host pointer with flat-memory load/store, so comptime malloc must allocate from flat memory. The default context lets the allocator PROTOCOL run; native malloc is its final step.
• (7) is_comptime + failable/error cluster + the signed-load fix — DONE. Coverage 31 → 36 handled (fallbacks 7 → 2); parity stays 688/688 both gate ON and OFF.
  • is_comptime → always 1 on the VM (folds to false in compiled code). Unlocked 1030.
  • Failable / error-channel cluster (1037 escape, 1038 handled): kindOf(error_set) → word (a u32 tag id); regToValue now bridges TUPLES (the failable (value…, tag) shape the host's checkComptimeFailable reads); trace_frame packs (func_id<<32 | span.start) from a new call_stack (pushed by invoke/runEntry); and sx_trace_push / sx_trace_clear are serviced NATIVELY (the VM calls the real sx_trace.c functions — linked into the compiler — so the return-trace buffer the host reads is populated identically to the legacy dlsym path). raise/catch/or all run on the VM now.
  • Signed sub-64-bit load fix (a real GENERAL bug the failable case surfaced): readField now SIGN-extends i8/i16/i32/isize loads (was zero-extending, so a stored i32 -1 reloaded as 0xFFFFFFFF = +4.29e9 and < 0 was false — which silently hid raise error.Bad). Affects any negative signed sub-64-bit value stored & reloaded; gate-ON corpus parity confirms it's a strict fix. Unit test added (+ failable tests pass via 1037/1038 in the corpus).
  • Remaining fallbacks (2, both principled — the VM correctly stays on legacy): intern (0626, the welded compiler-API fn — Phase 3 re-homes it) and the inline-asm global call (1654, never comptime-evaluable). Every other measured corpus const-init is handled on the VM. At this point the flat-memory VM handles essentially the entire real comptime corpus (scalars, control flow, structs/tuples/arrays/slices/strings/optionals/enums, calls + recursion, the implicit context + allocator protocol, globals, failables + return traces). Phase 2 (bytecode) and Phase 3 (compiler-API on flat memory) are the forward work; flipping the VM to default + deleting the legacy path awaits those.
• (8) Wire the #run side-effect path; trace-clear-on-fallback — DONE. The second comptime call site (emit_llvm.runComptimeSideEffects, top-level #run <expr>;) now routes through tryEval with legacy fallback, like the const-init fold; tryEval yields .void_val for a void/noreturn entry. Fixed a trace-corruption the new site exposed (1035): a side-effect that pushes trace frames then bails (on print) had the legacy re-run double-push them — both sites now sx_trace_clear() right before the legacy fallback to discard the VM's partial pushes. Parity 688/688 both gate ON and OFF. All comptime evaluation now routes through the VM-with-fallback (uniform).
• (9) -Dcomptime-flat build flag — DONE (the "swap behind a build flag" step). The VM gate is now a build option (build.zig → a build_opts module on mod; emit_llvm.init reads build_opts.comptime_flat or SX_COMPTIME_FLAT env), default OFF. zig build test -Dcomptime-flat runs the FULL corpus on the VM (688/0) — the build-integrated parity gate. Verified the flag toggles the binary (flag-built sx uses the VM with no env var; default-built does not). This is the prerequisite to eventually making the VM default + deleting the legacy path (which still awaits Phase 2/3 + broader confidence).
• (10) Compiler-call path on the VM — intern/text_of native (Phase 3 SEED) — DONE. invoke now services a welded compiler-library function (the compiler_welded flag is the safety boundary) via Vm.callCompilerFn — natively on flat memory, NO legacy Interpreter: intern(s: string) -> StringId reads the string bytes from flat memory and internStrings into the (const-cast) table (pool-only, never touches type layout, so the VM's cached sizes stay valid); text_of(id) -> string materializes the pooled text back into flat memory as a fat pointer. Unlocked 0626 — the ONLY remaining const-init fallback is now the inline-asm global (1654, genuinely not comptime-evaluable). Parity 688/688 both gate ON and OFF; unit test added. This is the mechanism Phase 3 grows: the next compiler functions (find_type, register_struct, the reflection readers) are added the same way — flat-memory pointer in, handle/pointer out, no marshaling.

Phase 3 progress (2026-06-18):

• (P3.1) First read-only reflection readers — find_type + type_field_count (DONE). Two more compiler-library fns bound the same way as the intern/text_of seed (added to compiler_lib.bound_fns AND Vm.callCompilerFn, native on flat memory, no marshaling). A type handle is a plain u32 TypeId (exactly like StringId), so both calls keep the seed's clean scalar shape — handle in, scalar out: find_type(name: StringId) -> TypeId (TypeTable.findByName) and type_field_count(t: TypeId) -> i64 (a new TypeTable.memberCount query — struct/union/ tagged-union fields, enum variants, array/vector length — that BOTH the legacy handler and the VM call, so the two paths can't drift). Example 0628 chains intern → find_type → type_field_count and a not-found lookup, both folded at #run, both VM-HANDLED natively (no fallback). Parity 689/689 (gate ON and OFF); VM unit test added.
  • Decision (resolves the plan's find_type → ?Type sketch): find_type returns a NON-optional TypeId, using the codebase's dedicated unresolved (0) sentinel for not-found — NOT an ?Type. Rationale: a Type value resolves to .any (type_resolver.zig), which the flat-memory VM does not represent; and an optional return can't cross the legacy↔VM eval boundary (regToValue bridges only word/string/struct/tuple). unresolved is the project-blessed unmistakable "no type" marker (see CLAUDE.md REJECTED PATTERNS — a dedicated sentinel is the required shape), so the caller checks the handle against 0. This keeps the reader a clean scalar mirror of intern/text_of and defers .any/optional plumbing to when it's actually needed.
• (P3.2) Field-level reflection readers — type_nominal_name + type_field_name + type_field_type (DONE). Three more readers on the same TypeId-handle shape (each backed by a new TypeTable query that BOTH the legacy handler and the VM call, so no drift): type_nominal_name(t: TypeId) -> StringId (nominalName — a named type's own name; loud-bail for unnamed types), type_field_name(t: TypeId, idx: i64) -> StringId (memberName — struct/union/tagged-union field, enum variant, named-tuple element), and type_field_type(t: TypeId, idx: i64) -> TypeId (memberType — struct/tuple/array/vector member type). All loud-bail on out-of-range idx / no-member (no silent default). These are the first MULTI-ARG compiler fns (the VM's callCompilerFn now reads arg 1 = idx); added Vm.argHandle/argTypeId helpers (range-checked u32/TypeId arg reads). Naming uses the type_* family so nothing collides with the std metatype builtins (field_name/type_name exist in core.sx). Example 0629 reflects Pair { lo: Point; hi: Point } — reads each field name and the nominal name of a field's type, all folded at #run, all VM-HANDLED natively. Parity 690/690 (gate ON and OFF); VM unit test added.
• (P3.2b) Kind + enum-value readers — type_kind + type_field_value (DONE). The last two read-only readers the metatype's type_info(T) needs, completing the READ side: a comptime sx fn can now fully reflect a struct/enum/tagged-union/tuple into data with no #builtin. type_kind(t: TypeId) -> i64 (TypeTable.kindCode — a stable, compiler-owned discriminant: 0 other · 1 struct · 2 enum · 3 tagged_union · 4 tuple · 5 union · 6 array · 7 vector · 8 error_set; TOTAL — never bails, an unnamed/non-aggregate type reads other) and type_field_value(t: TypeId, idx: i64) -> i64 (TypeTable.memberValue — an enum variant's explicit value or ordinal; mirrors the field_value_int builtin; loud-bail for a non-enum / out-of-range idx). Example 0630 reflects Color/WindowFlags(flags)/Point. Parity 691/691 (gate ON and OFF); VM unit test added.
  • READ side now complete: find_type + type_kind + type_field_count + type_field_name + type_field_type + type_nominal_name + type_field_value cover everything reflectTypeInfo reads.
• (P3.3) WRITE side — declare_type + pointer_to + ONE kind-branching register_type (DONE). The mutating side is a SINGLE register_type(handle, kind, members) that branches on kind IN THE COMPILER (subsuming define's defineStruct/defineEnum/defineTuple), plus declare_type(name) -> Type (forward handle) and pointer_to(t) -> Type (build *T references). They take/return real Type values (matching meta.sx's declare/define).
  • Timing decision (per the user): mint LAZILY at LOWERING time (single pass, NOT a pre-emit phase, NOT two-pass) — the existing runComptimeTypeFunc path. So the write side is legacy-only (compiler_lib handlers); the VM isn't wired at lowering time, so no VM mirror is needed (the read-side readers stay dual-path for emit-time reflection). A non-generic -> Type builder is now flagged is_comptime (decl.zig) so its dead body permits the welded calls (the comptime-only gate).
  • Graph support: forward declare_type handles + pointer_to express a mutually-recursive A↔B graph (*A, *B, B-by-value) before bodies are filled. register_type is idempotent — re-filling a nominal slot (same module reached via two import edges) re-mints identically instead of erroring (nominalIdent reads identity from any nominal kind). kind codes match type_kind: 1 struct · 2 enum (actual .@"enum") · 3 tagged_union · 4 tuple.
  • Two bugs fixed en route (issue 0142): (a) a fully payloadless comptime-minted enum was minted as an all-void tagged_union → verifySizes panic; now mints a real .@"enum" (both register_type kind 2 AND the metatype defineEnum). (b) bare EnumType.variant qualified construction of a payloadless variant wasn't supported (failed for hand-written enums too) — added in lowerFieldAccess (isPayloadlessVariant).
  • Examples: 0631 (graph + actual-enum + reflection), 0632 (make_enum all-void), 0633/0634/0635 (namespaced / bare / multi-edge import of a minted type), 0187 (qualified variant construction). Parity 697/697 (gate ON and OFF); unit tests added.
• Next (P3.4): re-express declare/define/type_info as sx over the read+write compiler-API and DELETE the bespoke interp arms — needs the VM hardened against malformed lowering-time IR first (the metatype runs at lowering time), so either harden + wire the VM there, or migrate the metatype onto the legacy compiler-API calls first. Decide when reached. Phase 2 (bytecode) is the orthogonal speed work.

Phase 3 — Compiler-API on flat memory (resume the stream — no weld)

With native-byte comptime values, re-home the compiler-API:

• Expose the compiler's real types. Register the actual types.zig records (StructInfo, EnumInfo, Field, …) into the comptime type table under sx-visible names, with their real (host) layout — the type IS the compiler's, so there is nothing to validate or keep in sync. (This is the projection that replaces the weld's reflection — owned by the compiler, not declared in sx.)
• Expose the compiler's functions. register_struct, find_type, intern, text_of, and the reflection readers operate on flat-memory pointers / handles directly (no marshaling — the bytes already ARE the record).
• Re-express declare / define / type_info as sx over these; delete the bespoke interp arms (defineStruct / defineEnum / defineTuple / reflectTypeInfo); migrate examples/0622 (struct), 0619/0620/0623 (enum/tuple).
• Migrate BuildOptions off #compiler onto this mechanism; delete #compiler.

Verification: the metatype + #compiler surfaces are gone, re-expressed as sx over the exposed compiler-API; full corpus green.

Phase 4 — Retire the legacy interp (the ONE-evaluator end state)

The metatype CONSTRUCTION + REFLECTION surface is VM-native (steps 7/8 — 0614–0624, 0632 all HANDLED). This phase moves EVERYTHING ELSE off interp.zig and deletes it.

What the legacy interp is still used for (audited 2026-06-18) — five roles:

Role	Wired to VM?	Site
A. Comptime folds (type-fn / :: const-init / #run)	✅ VM + legacy fallback	comptime.zig:530, emit_llvm.zig:871/971
B. #insert string eval	❌ legacy-only (VM wiring reverted — 0737 malformed-IR crash)	comptime.zig:634
C. Post-link bundler (platform.bundle — Info.plist/codesign/process/fs)	❌ legacy-only	core.zig:invokeByFuncId ← main.zig:769
D. #compiler hooks (compiler_call — BuildOptions/bundling)	❌ legacy-only; Value-based ABI	compiler_hooks.zig, interp.zig:1130
E. Bail diagnostics (Interpreter.last_bail_* statics)	n/a	main.zig:464

Shared substrate everything traffics in: the Value tagged union (the regToValue/valueToReg bridge + the hooks + core.zig) and the host-FFI bridge (host_ffi.zig + interp.callExtern — dlsym + cdecl trampolines for real libc).

DECISION (2026-06-18, user): UNIFY. The VM gains a host-FFI escape + real-pointer translation and runs BOTH sandboxed comptime folds AND the unsandboxed post-link bundler. interp.zig is fully deleted — true ONE evaluator, two modes (sandboxed / host-effects).

Remaining comptime-fold gaps (full corpus fallback inventory — 15 examples; 1179/1180 are legitimate negative-test bails that BECOME VM diagnostics, 1145 is a scan artifact): box_any/unbox_any (6), out/print (2), global_addr (1), trace frames (1), compiler_call (2 — role D).

Sub-phases (dependency order; each its own session, both gates 697/0 after each):

• 4A — finish comptime ops (small, parity-guarded). Drive the fold fallback list to empty except compiler_call:
  • 4A.1 box_any/unbox_any. Word case = alloc 16B {tag@0, value@8}, tag = source_type.index() (matches legacy comptime; note runtime anyTag normalizes arbitrary-width ints), value via writeField(source_type) (so f32 etc. round-trip); unbox = readField(addr+8, target). Aggregate-Any payload needs the runtime pointer-in-value-slot shape (coerceToI64 alloca+ptrtoint) — implement or bail loudly.
  • 4A.2 out/print → add a VM output buffer; flush through the same path as core.flushInterpOutput.
  • 4A.3 global_addr (address-of a global in flat memory).
  • 4A.4 trace frames (sx_trace_* / interp_print_frames).
• 4B — VM-native diagnostics (role E). MUST land before deleting legacy. Today a VM bail silently falls back; with legacy gone the VM bail IS the user-facing build-gating diagnostic. Surface the VM's detail/span/file into what main.zig renders; turn 1179/1180-style bails into proper diagnostics. No diagnostic may regress.
• 4C — #insert on the VM (role B). Re-wire evalComptimeString through tryEval; the lowering-time-IR hardening that forced the 0737 revert is already in place. Verify the #insert corpus parity.
• 4D — host FFI on the VM (role D substrate). DONE. Solved by a better allocator, not a pin/tag scheme: the comptime memory is now an arena of stable host allocations and Addr IS a real host pointer (4D.0, 625ba0f), so a flat-memory pointer and an FFI-returned host pointer are the same value — no translation, no realloc hazard. Vm.callHostExtern (4D.1, e7a8708) dispatches ANY extern via host_ffi dlsym + trampolines (args/returns pass untouched); 4D.2 (6a7f690) adds slice/string args (→ NUL-term char*) + float guards. Examples 0636/0637. (Superseded sub-note: the earlier "pin the buffer / flat↔host translate" hazard is moot — the arena never moves an allocation.)
• #compiler / compiler_call — DELETED, replaced by the abi(.compiler) ABI (decision 2026-06-18, REVISED from the earlier abi(.zig) extern compiler shape). A function is compiler-domain — it runs in the comptime evaluator (VM/interp), NEVER in the shipped binary — because its ABI says so: abi(.compiler). No extern <lib>, no fake #library "compiler". One annotation covers BOTH roles: (a) the compiler-API surface (intern/find_type/build_options/set_post_link_callback/… — bodiless decls whose Zig/VM handler is the impl, on compiler_lib's export list, dispatched by Vm.callCompilerFn); (b) user compiler-domain functions like post-link callbacks (bundle_main — BODIED abi(.compiler), lowered for VM eval but emit-skipped). The #compiler struct attribute + the compiler_call IR op + the Value-based hook Registry (compiler_hooks.zig) all go away. Why this is cleaner than the welded-fn approach: the former runtime-call enforcement blocker (a build_options() call inside an LLVM-emitted callback body) is MOOT — a compiler-domain function is never emitted, so its compiler-API calls never reach emitCall. Staged build (each its own step, both gates green):
  • S1+S2 — DONE (2026-06-18): introduced abi(.compiler), REMOVED the .zig ABI + abi(.zig) extern compiler + #library "compiler" (clean cutover, no legacy); migrated all compiler-API examples. The binding now keys off fd.abi == .compiler (decl.zig weldedCompilerFn); a bodiless abi(.compiler) decl lowers extern-like (declared-not-defined) with no implicit ctx. 700/0 both gates.
  • S3 — DONE (2026-06-18): emit_llvm skips BODIED abi(.compiler) function bodies. Added an is_compiler_domain flag to the IR Function; a bodied abi(.compiler) function LOWERS its body (for VM eval) + is flagged is_comptime but is NOT emitted (Pass 2 skip; declared external-linkage so the empty decl verifies). KEY fix: a call to a comptime-only callee (compiler-API compiler_welded OR is_compiler_domain) inside a dead comptime body now emits undef instead of a real call (ops.zig emitCall) — the old compiler_call did this; without it an AOT link leaves an undefined _double/_intern reference (this also fixed a pre-existing untested AOT breakage of the bodiless compiler-API examples). fnIsBodilessCompiler distinguishes the API surface (declare-only) from a compiler-domain callback (lowered, emit-skipped). Regression: examples/0638-comptime-domain-fn-not-emitted (double folds a #run const, absent from the binary, JIT+AOT). 701/0 both gates.
  • S4 — callback-param propagation: OPTIONAL / DEFERRED (ergonomics only). Verified 2026-06-18: an abi(.compiler) function is TYPE-compatible with a plain () -> R param (the ABI marks the function — is_compiler_domain — not its type, which stays () -> R CC-default). So a callback that needs to be compiler-domain just declares itself abi(.compiler) (S3) and passes to a plain param fine; auto-propagation from an abi(.compiler) PARAM type is a nicety, not a prerequisite for S5. Skipped for now.
  • S5a — DONE (2026-06-18): the corpus-covered slice. build_options + set_post_link_callback → free abi(.compiler) functions (VM callCompilerFn arms + legacy compiler_lib handlers); BuildConfig threaded into the VM via a tryEval param (the same one main.zig forwards — shared with 4E). build.sx extracts set_post_link_callback from the struct #compiler as a free ufcs fn; bundle_main + the platform registrars (configure) are abi(.compiler). 37 examples' .ir snapshots regen'd (benign: declaration renumber + @str suffix shift — every example imports build.sx via the prelude). Strict compiler_call bails 6→2; 0602/0603/1604/1611 HANDLED. 701/0 both gates.
  • S5b/S5c (port the ~37 hooks) — SUPERSEDED 2026-06-18 by the sx-driven build pipeline (below). Porting each BuildOptions accessor to an abi(.compiler) function that delegates to a compiler_hooks hook just re-encodes sx-level logic (string setters/getters, is_macos triple-matching, list appends) as compiler hooks. The hooks need NOTHING from the compiler except the BuildConfig state. So instead of 37 hooks, drive the whole build pipeline from sx (the logical end of "bundling lives in sx"). S5a stays as a green intermediate; the sx-build-pipeline replaces build_options/set_post_link_callback/the whole #compiler surface wholesale.

Phase 5 — sx-driven build pipeline (replaces the BuildOptions hooks; decision 2026-06-18, user)

The build pipeline becomes an sx program. BuildConfig is plain sx data (an ordinary struct, sx-owned end-to-end — no #compiler, no hooks, no shared Zig state, no weld/offset access). The compiler shrinks to a few abi(.compiler) PRIMITIVES that take explicit args (so nothing is shared by memory), and an sx build() driver orchestrates configure → emit → link → bundle. Chosen boundary: Option B — the compiler keeps the proven Zig linker as a primitive; sx owns config + orchestration + bundle. (Option A — sx shells cc/ld itself — is a later refinement once the per-target link-line logic is ported to sx.)

File split (user decision 2026-06-19): the low-level compiler-API PRIMITIVES live in library/modules/compiler.sx (the comptime compiler library — renamed from the interim std/build.sx); the default build IMPLEMENTATION (default_build + the on_build slot + the sx BuildConfig) lives in library/modules/build.sx alongside the existing BuildOptions DSL. So compiler.sx = primitives, build.sx = orchestration/default impl. Build-callback fallibility was DROPPED (user 2026-06-19): the primitives + the build callback are NOT -> ! — a failed action (e.g. link) BAILS on the VM (hard build error). So the shapes below shed their -> !.

Shape (build-callback fallibility dropped 2026-06-19):

// library/modules/compiler.sx (the comptime `compiler` library — PRIMITIVES)
emit_object    :: () -> string abi(.compiler);                        // emitted .o path (query)
link           :: (objects: List(string), output: string, libraries: List(string),
                   frameworks: List(string), flags: List(string), target: string) abi(.compiler);  // void; bails on failure
c_object_paths :: () -> List(string) abi(.compiler);                  // metadata queries
link_libraries :: () -> List(string) abi(.compiler);

// library/modules/build.sx (the build DSL — DEFAULT IMPLEMENTATION + slot)
BuildConfig :: struct { output: string; target: string; flags: List(string);
                        frameworks: List(string); bundle_path: string; bundle_id: string; ... }
default_build  :: (config: BuildConfig) abi(.compiler) {              // the default pipeline (void)
    obj := emit_object(); objs := c_object_paths(); objs.append(obj);
    link(objs, config.output, link_libraries(), config.frameworks, config.flags, config.target);
    if config.bundle_path.len > 0 { bundle_app(config); } }           // bundle_app = today's sx bundler
on_build : (BuildConfig) abi(.compiler) = default_build;              // the override slot
// user overrides:  build :: (config: BuildConfig) abi(.compiler) { ... }   #run on_build = build;

The compiler's whole post-IR role: codegen → build the CLI-derived BuildConfig → read on_build → invoke on_build(config) on the VM; a raise fails the build. Plain sx run fires none of it.

Steps (each its own green step; depends on 4E first):

• P5.1 — 4E prereq — DONE (2026-06-19). core.invokeByFuncId routes the post-link callback through the VM (comptime_vm.tryEval), NO fallback (a side-effecting callback can't double-execute): a bail is a hard build error (comptime_vm.last_bail_reason surfaced by main.printInterpBailDiag). BuildConfig + import_sources threaded in; flushInterpOutput deleted (VM out writes direct via host-FFI). Smoke test examples/1661-platform-post-link-vm-list (AOT): a post-link callback GROWS a List (0141 — works on the VM, bails on legacy with struct_get), so the build succeeds (exit 0) only via the VM. Non-empty callback args rejected loudly (the on_build(config) arg-marshaling entry is P5.3). 702/0 both gates.
• P5.2 — primitives. Split: the read-only metadata queries are DONE (2026-06-19) — c_object_paths() -> List(string) + link_libraries() -> List(string) as abi(.compiler) fns (stdlib library/modules/compiler.sx), serviced by comptime_vm.callCompilerFn over BuildConfig fields main.zig forwards; new VM makeStringList builds the List(string) in flat memory from the call's result type (ins.ty now threaded through invoke/callCompilerFn). Smoke test 1662-platform-build-pipeline-queries (AOT + C companion). 703/0 both gates. emit_object() -> string is also DONE (2026-06-19) as a QUERY (not an action): the Zig driver emits the object eagerly, so the primitive just returns the path from BuildConfig.object_path (no vtable). So all three QUERY primitives are done. P5.2b — link(...) (the one genuine ACTION) — DONE (2026-06-19). USER DECISION: the build callback is NOT fallible, so link is plain VOID (no -> !) and a failure BAILS (hard build error) — no failable-tuple construction. It dispatches through a host-installed compiler_hooks.BuildHooks vtable (comptime_vm.zig can't depend on the driver); main.LinkHooksCtx.link adapts to target.link. New VM readers readStringList/readStringArg (inverse of makeStringList). Smoke test 1663-platform-build-pipeline-link (AOT): a post-link callback re-links the build's objects to a temp output — the relinked binary RUNS; negative-probe verified. The Zig driver still auto-links (removed in P5.4). 704/0.
• P5.3 — on_build slot: a comptime-assignable compiler slot (GENERALIZES today's post_link_callback_fn: an assignable typed global with a stdlib default, vs a setter). #run on_build = build; captures the FuncId; the compiler invokes it post-codegen with the CLI-derived BuildConfig.
• P5.4 — sx default_build + BuildConfig: write the stdlib pipeline; move config/orchestration into sx. Delete #compiler / compiler_call / compiler_hooks (HookFn/Registry) + the S5a build_options/set_post_link_callback (config is now sx data passed as primitive args).
• 4E — post-link bundler on the VM (role C). Depends on the FFI escape (done) + the BuildConfig-on-the-VM threading above. Route core.invokeByFuncId / main.zig's post-link call through the VM. No corpus coverage (only runs on sx build --bundle/--apk) — add dedicated bundle smoke tests (min .app + .apk); gate on real bundle builds, not just zig build test. Riskiest phase.
• 4F — flip + delete (the end state). Flip the VM to default (retire -Dcomptime-flat); delete interp.zig (Interpreter/Value/defineEnum…/reflectTypeInfo/callExtern); drop the regToValue/valueToReg bridge; simplify core.zig (invokeByFuncId → VM) + main.zig (last_bail_* → VM diagnostics); remove the dual-path compiler_lib handlers (keep only VM-native callCompilerFn); re-express define/make_enum as sx over the compiler-API (allocation works on the sole evaluator) and land the original 0141 repro as a corpus test.

Dependencies: 4A → (4B, 4C independent) ; abi(.compiler) S1+S2(done) → S3 → S4 → S5 (BuildOptions) ; FFI(done)+BuildConfig-on-VM → (S5, 4E) → 4F. Top risks: (1) the bundler has no corpus guard (4E needs dedicated tests); (2) deleting #compiler/compiler_call + re-expressing BuildOptions over the compiler-API (abi(.compiler)) touches the whole build/bundle path — stage it behind real bundle builds; (3) S3's emit-skip relies on DCE dropping the unreferenced compiler-domain declaration — verify no stray runtime reference keeps it alive (link error).

Open questions (resolve as reached, record decisions here)

• Host-ABI vs target-ABI split. The compiler runs on the host, so its OWN exposed records are host-laid-out; user comptime types are target-laid-out. The flat-memory model must carry both regimes (a per-type ABI tag on layout queries). Confirm the boundary where a flat-memory pointer to a compiler record is handed to host Zig code uses host layout.
• Exposing compiler types to sx. Mechanism for projecting types.zig records into the comptime type table with real offsets (the non-weld replacement) — a registry the compiler owns, keyed by sx-visible name → real Zig type's layout + a host-call ABI.
• Bytecode shape. IR-derived vs a fresh ISA; register vs stack; fragment caching.
• Pointer escape / lifetime. Flat-memory pointers stored into the persistent type table must be copied into compiler-owned memory at the boundary (as today).
• Old-path retirement. Keep the tagged interpreter until Phase 1 parity, then delete — confirm no non-comptime consumer depends on Value.

File map (current → touched)

Area	File	Phase
Comptime evaluator	src/ir/interp.zig	0 (strip weld dispatch), 1–2 (rebuild)
Weld registry	src/ir/compiler_lib.zig	0 (strip), 3 (replace with type/fn exposure)
Weld validation	src/ir/lower/nominal.zig	0 (strip validateWeldedStruct)
Comptime-only gate	src/backend/llvm/ops.zig	0 (re-derive without weld signal)
Host-FFI marshalling	src/ir/host_ffi.zig	1 (struct-by-pointer trims it)
Metatype arms	src/ir/interp.zig (defineStruct/…/reflectTypeInfo)	3 (delete, re-express in sx)
#compiler / BuildOptions	library/modules/build.sx, src/ir/compiler_hooks.zig	3 (migrate, delete #compiler)
Surface syntax	src/parser.zig, src/ast.zig (abi/extern/#library)	kept; revisited Phase 3

Status

• Phase 0 — DONE (2026-06-17). The struct-weld machinery is stripped: compiler_lib.zig lost the type registry (weldStruct/bound_types/BoundType/ FieldLayout/findType/SxField/LayoutMismatch/validateStructLayout); nominal.zig lost validateWeldedStruct/weldedFieldOrderStr + the sd.abi == .zig call; the struct-weld unit tests + examples 0625/0627/1183/ 1186 are removed. Decision (recorded): the intern/text_of function host-call bridge is KEPT — it is a clean scalar dispatch (string→handle), not weld/serialize/marshal, and is the seed Phase 3 grows the compiler-call path from. So the compiler_welded dispatch (interp.callExtern is unchanged at HEAD — the pre-branch in call()), weldedCompilerFn (decl.zig), the emitCall comptime-only gate (ops.zig), and examples 0626/1184/1185 stay. The #library/abi/extern SYNTAX stays. zig build test green (688 corpus, 0 failed; unit tests pass).
• Phase 1 — in progress.

  • Sub-step 1 — DONE. src/ir/comptime_vm.zig: the flat-memory Machine (linear byte memory + bump/stack allocator with mark/reset reclamation + scalar readWord/writeWord (1/2/4/8, little-endian) + bytes views; addr 0 reserved as null_addr) and Frame (register file indexed by Ref + stack reclamation on deinit). A register Reg is a raw u64 — immediate scalar OR Addr. Standalone + unit-tested (comptime_vm.test.zig, in the barrel); does NOT touch the live interpreter, so the corpus stays green (688). No op execution yet.

  • Sub-step 2 — DONE. The executor (Vm in comptime_vm.zig): walks the SAME IR Inst over flat-memory frames, mirroring the legacy interp's scalar semantics (i64 wrapping/signed + f64 register words, keyed off the result/operand TypeId). Ported: constants (const_int/float/bool/null/undef), arithmetic (add/sub/mul/div/mod/neg), comparison (cmp_*), logical (bool_and/or/not), conversions (widen/narrow/bitcast passthrough, int_to_float/float_to_int), terminators (br/cond_br/ret/ret_void) and block_param (branch args passed as Refs — the same frame persists, SSA-safe). Any other op bails loudly (error.Unsupported + detail = @tagName(op)). Unit-tested on hand-built IR (Fb builder): integer add, f64 arithmetic, cond_br branch selection, a block-param loop summing i..1, div-by-zero + unsupported-op bails. Corpus untouched (688 green) — the executor is exercised by unit tests only, not yet wired to real comptime eval.

  • Sub-step 3 — DONE. Memory + structs on flat memory. Vm gained an optional table: *const TypeTable (target-aware layout). Ported alloca/load/store (over flat addresses, Store.val_ty drives width) and struct_init/struct_get/ struct_gep (structs laid out at the table's natural offsets). The value model: a Kind.word (scalar/pointer ≤8B) sits in a register; a Kind.aggregate (struct) lives in flat memory and its "value" IS its address (read returns the address, write memcpys), so nested structs compose and struct_gep is just base+offset (no field-pointer dance). kindOf bails loudly on the not-yet-ported types (slice/string/any/optional/enum/array/tuple/…). The Addr-based value model survives allocator realloc (offsets are stable; slices are only materialized transiently). Unit-tested: struct_init+get round-trip, alloca+gep+store+load, nested-struct aggregate copy + nested read. Corpus untouched (688 green).

  • Sub-step 4a — DONE. Tuples + arrays. kindOf widened (tuple/array → aggregate). Ported tuple_init/tuple_get (positional, tupleFieldOffset), index_get/index_gep (elemAddr = base + idx*elem_size over array/pointer/ many_pointer bases; slice/string bases bail), and length on an array value (static ArrayInfo.length). Unit-tested: mixed tuple round-trip, [3]i64 gep/store + index_get sum (42), array length (3). 688 corpus green.

  • Sub-step 4b — DONE. Slices + strings as {ptr@0 (pointer_size), len@8 (i64)} fat pointers (kindOf: string/slice → aggregate). Ported const_string (materializes text+NUL in flat memory + a fat pointer), length/data_ptr (read len/ptr fields), array_to_slice, subslice, indexing through a slice/string (elemAddr loads .ptr first), and str_eq/str_ne (len+memcmp). Helpers makeSlice/sliceLen/ sliceData. Unit-tested: string length + str_eq/ne, array→slice + slice index + slice length (23), array subslice (43). 688 corpus green.

  • Sub-step 4c — DONE (optionals + payloadless enums). kindOf: enum → word; ?T → word if pointer-child (null==0) else {T@0, i1@sizeof(T)} aggregate. Ported optional_wrap/unwrap/has_value/coalesce (with optChildIsPtr/optHas helpers; const_null → null_addr reads as none), enum_init (payloadless: tag is the value), enum_tag (payloadless/word). Unit-tested: non-pointer ?i64 wrap/unwrap/coalesce (91), pointer ?*i64 null==0 (99), payloadless enum tag (11). 688 corpus green.

  • Sub-step 4d — partial (addr_of/deref DONE). addr_of passes through (an aggregate value already IS its address; a pointer is already an address — mirrors the legacy); deref = readField through the pointer (ins.ty is the pointee). Unit-tested (deref a *i64 → 77; addr_of a struct value + field read → 80). Deferred to the wiring phase (intentionally, not ported blind): tagged-union payload (enum_init w/ payload, enum_payload — the legacy stores untyped Values and field_index indexes payload sub-fields, not variants, so a byte model's payload type is ambiguous without a real call site), any boxing, closures, and the bitwise ops. These have subtleties best resolved against actual corpus cases — the VM's loud error.Unsupported + detail will name exactly what each real eval needs.

  • Sub-step 1.5 — direct call DONE. Vm gained module: *const Module (resolves a callee FuncId) + a depth/max_depth recursion guard. call marshals arg Refs → Reg words and recursively runs the callee; aggregate args/ results pass as their Addr over the SHARED flat memory (no copy). Stack-lifetime change: Frame no longer reclaims the machine on exit (a returned aggregate's Addr would dangle) — a comptime eval's allocations live to Vm.deinit; Machine.mark/reset stay for explicit use. Extern/builtin callees (no blocks) bail loudly (1.5b). Unit-tested: direct call (add(20,22)+100 → 142) and recursion (sum(0..n) → 15/55). 688 corpus green.

  • Sub-step 1.5b — Reg↔Value boundary bridge DONE. The builtin/compiler_call/ extern handlers are all coupled to the legacy Interpreter (e.g. compiler_lib handlers take *Interpreter), so the VM can't call them directly — the wiring uses WHOLE-FUNCTION fallback instead (VM runs pure functions; a bail re-runs the whole eval in the legacy). That needs the boundary bridge: valueToReg (host Value arg → VM Reg, materializing aggregates into flat memory) + regToValue (VM result → Value, deep-copied out). Covers scalars + strings + structs (other aggregate shapes bail loudly; added as wiring surfaces them). Transitional — deleted once the VM owns comptime end-to-end. Unit-tested with round-trips. 688 corpus green.

  • Then the wiring step (below) — now unblocked.

Decision (2026-06-17): pivot from blind op-porting to CALLS + hybrid wiring

The common leaf ops are ported (scalars, control flow, structs, tuples, arrays, slices, strings, optionals, payloadless enums, deref/addr_of) and unit-tested. Continuing to port the rarer ops (tagged-union payload, any, closures) in isolation risks subtle bugs and has low signal. The higher-value path:

1. Calls (sub-step 1.5) — call (direct), then call_builtin/compiler_call. The shared flat memory makes aggregate args/results pass naturally (they're Addrs). The one design point: aggregate-return lifetime — a callee's stack-reclaim would dangle a returned struct Addr, so for comptime (bounded) the VM should stop reclaiming per-frame and let the whole eval's allocations live until Vm.deinit (keep Machine.mark/reset for explicit use; drop it from Frame.deinit).
2. Hybrid wiring — -Dcomptime-flat routes a comptime eval through the VM, falling back to the legacy interp on error.Unsupported. This makes the VM run the REAL corpus, proving parity incrementally and surfacing exactly which ops each real eval needs — far better signal than more isolated unit tests.