a446550013
0143: a ..$args pack forwarded as a []Type argument across a call is backed by a [N x Any] (16B) array but viewed as []type_value (8B) -> half-stride reads. A lowering bug the legacy Value model masks; the byte-accurate VM exposes it. Blocks examples/0114 on the VM. Filed per CLAUDE.md (not worked around; the type_name .unresolved guard only makes the VM decline rather than emit garbage). Checkpoint also records the sequencing insight: comptime `out` (print) can only land once the fallback is removed (a print-then-bail double-prints under the legacy re-run), so side-effecting ops + fallback-removal are the FINAL step; pure ops + migrations land first.
1020 lines
82 KiB
Markdown
1020 lines
82 KiB
Markdown
# CHECKPOINT-COMPILER-API — comptime `compiler` library (`#library "compiler"` + `abi(.zig) extern`)
|
||
|
||
Companion to the design-of-record
|
||
[../design/comptime-compiler-api.md](../design/comptime-compiler-api.md) (the plan
|
||
+ phased build order live there). This stream supersedes the metatype
|
||
`declare`/`define`/`type_info` `#builtin`s and the `#compiler` struct attribute
|
||
with ONE welded mechanism. Branch: `reify` (off `master`). Update after every step.
|
||
|
||
## ⏯ Resume (fresh session)
|
||
|
||
> **⚠ DIRECTION CHANGED (2026-06-17). The active plan is now
|
||
> [`PLAN-COMPILER-VM.md`](PLAN-COMPILER-VM.md), NOT the weld.**
|
||
> The **byte-weld + serialization/marshaling** approach is the wrong direction and is
|
||
> being **stripped**. New foundation: a **bytecode VM over flat, byte-addressable
|
||
> memory** so comptime values are native bytes; then the compiler-API rides on it with
|
||
> direct memory access (no weld, no validation, no marshaling). Everything below this
|
||
> banner describes the now-superseded weld state (committed on `reify` through
|
||
> `40d075c`) and is kept only to scope the Phase 0 strip. Read
|
||
> `PLAN-COMPILER-VM.md` first.
|
||
>
|
||
> **Why the pivot:** the comptime evaluator (`src/ir/interp.zig`) represents values as
|
||
> tagged `Value` unions, NOT native bytes — so a comptime `@ptrCast(*StructInfo)`
|
||
> reads the `Value` union's memory, not a struct. The weld tried to bridge that with
|
||
> hand-marshaling — exactly what the design set out to kill. Flat memory makes comptime
|
||
> values real bytes, so the bridge disappears. (JIT-native comptime was rejected: it
|
||
> breaks cross-compilation — host vs target layout — and loses the sandbox. A
|
||
> flat-memory VM keeps both while getting native bytes + speed.)
|
||
>
|
||
> **Next action (2026-06-18) — the WHOLE metatype surface is VM-native (steps 7+8, committed through
|
||
> `d0ebc55`; step 8 uncommitted).** `declare`/`define`/`type_info` + tagged-union `enum_init` all run
|
||
> NATIVELY on the VM (`.call_builtin` exec arm → `callBuiltinVm`; `defineFromInfo` decodes a
|
||
> `TypeInfo` from flat memory, `buildTypeInfo` reflects one INTO flat memory — faithful ports of
|
||
> legacy `defineEnum`/`Struct`/`Tuple`/`reflectTypeInfo`). The ENTIRE metatype range `0614`–`0624` +
|
||
> `0632` runs **HANDLED with ZERO fallback** (incl. the `define(declare, type_info(T))` round-trips
|
||
> `0619`/`0622`/`0623`); VM output byte-matches legacy. `enum_init`/`define`/`type_info` bail loudly
|
||
> on a `backing_type` tagged union rather than silent-clobber. **697/0 BOTH gates + all unit tests.**
|
||
> **THE NEXT STEP — Phase 4D.3 (`compiler_call` / #compiler hooks on the VM).** Phase 4 (legacy-interp
|
||
> retirement) is PLANNED in `PLAN-COMPILER-VM.md`; **user decision: UNIFY** (the VM runs the post-link
|
||
> bundler too, `interp.zig` fully deleted). DONE this arc (all green): **4A.1** box_any/unbox_any +
|
||
> `.any` as a 16-byte aggregate (`1526d19`); **4D.0** comptime memory → an ARENA, `Addr` = real host
|
||
> pointer, no buffer/cap/move (`625ba0f`); **4D.1** general host-FFI escape — `Vm.callHostExtern`
|
||
> dlsym + host_ffi, any extern, args/returns pass untouched since Addr IS a host pointer (`e7a8708`,
|
||
> example 0636 `toupper`); **4D.2** slice/string args → NUL-term `char*` + float-arg/return guards
|
||
> (`6a7f690`, example 0637 `strlen([:0]u8)`). **699/0 BOTH gates.**
|
||
>
|
||
> **DIRECTION CORRECTION (2026-06-18, user): `#compiler` / `compiler_call` is DELETED, not bridged
|
||
> on the VM.** `BuildOptions` is RE-EXPRESSED as **`abi(.zig) extern compiler`** functions (the
|
||
> compiler-API surface the VM already dispatches via `callCompilerFn`); the `#compiler` attribute, the
|
||
> `compiler_call` IR op, and the `Value`-based hook `Registry` (`compiler_hooks.zig`) all go away.
|
||
> So there is **NO transitional `compiler_call`→hooks shim** on the VM (I started one — threading the
|
||
> legacy interp into `tryEval` for the hook registry — and reverted it; tree clean at `b05c74f`).
|
||
> `0602`/`0603` stay on legacy fallback until the BuildOptions migration lands. **Migration shape**
|
||
> (end-state, shares the `BuildConfig`-on-the-VM prerequisite with the bundler 4E): (1) each
|
||
> `BuildOptions` setter/getter becomes a `compiler` fn in `compiler_lib.bound_fns` + `Vm.callCompilerFn`,
|
||
> reading flat-memory args + a `*BuildConfig` threaded into the `Vm` (the same `BuildConfig`
|
||
> `main.zig` forwards); (2) `library/modules/build.sx` declares them `abi(.zig) extern compiler`
|
||
> instead of `struct #compiler`; (3) delete the `compiler_call` op + `compiler_hooks.zig`
|
||
> `HookFn`/`Registry` + the `#compiler` parse/lower path. See `PLAN-COMPILER-VM.md` Phase 4.
|
||
>
|
||
> **Corpus-driven remainder (independent of the BuildOptions migration):** **4A leftovers** —
|
||
> out/print (double-output-on-fallback caveat: write directly only once the whole eval is VM-handled),
|
||
> global_addr, trace, and `switch_br` for the Any-tag type-switch (0114/0520–0524/1035, the box_any
|
||
> examples that now bail further at `switch_br`/`type_name`) · **4B** VM diagnostics · **4C** `#insert`.
|
||
> Then the BuildOptions migration + **4E** bundler (+ dedicated bundle tests) + **4F** flip default +
|
||
> delete `interp.zig`/`Value` + re-express `define`/`make_enum`.
|
||
> Starting at **4A.1 (box_any/unbox_any)**. See `PLAN-COMPILER-VM.md` → Phase 4 for the full plan +
|
||
> top risks (flat-pointer escape on buffer realloc; bundler test coverage).
|
||
> Earlier landed: dedicated `Type` builtin TypeId (`6844fb9`/`94f60c5`/`554871b`); WRITE side
|
||
> declare_type/register_type/pointer_to VM-native (`66005af`); real lowering-time Context (`eb68d9e`);
|
||
> metatype construction declare/define/enum_init (`d0ebc55`).
|
||
>
|
||
> Done so far in Phase 3:
|
||
> - **READ side (7 readers, dual-path):** `find_type`/`type_kind`/`type_field_count`/
|
||
> `type_nominal_name`/`type_field_name`/`type_field_type`/`type_field_value`, each backed by a
|
||
> `TypeTable` query both the legacy handler and the VM call (no drift). Examples 0628–0630.
|
||
> - **WRITE side (P3.3, legacy-only at lowering time):** `declare_type` + `pointer_to` + ONE
|
||
> kind-branching `register_type` (subsumes `define`'s per-kind dispatch; codes match
|
||
> `type_kind`: 1 struct · 2 actual `.@"enum"` · 3 tagged_union · 4 tuple). Idempotent re-fill
|
||
> (two-edge import). Plus two fixes (issue 0142): all-void enum → real `.@"enum"` (was a
|
||
> verifySizes panic); bare `EnumType.variant` qualified construction. Examples 0631–0635, 0187.
|
||
> - **Lowering-time VM (P3.4):** hardened the VM against malformed lowering-time IR (`refTy`,
|
||
> bailing `aggType`, bounds-checked branch targets — bails, never panics); wired `tryEval`
|
||
> into `runComptimeTypeFunc` behind the flag with legacy fallback; materialized a zeroed
|
||
> lowering-time `Context` (the global isn't built yet at lowering). All measured green.
|
||
>
|
||
> **THE WALL (next step):** a `Type` *value* is an 8-byte tid, but `.any` (the boxed-any) is a
|
||
> 16-byte `{tag,value}` — and they share one TypeId (`.any`). So a `Type` in an aggregate
|
||
> (`Member.ty`/`EnumVariant.payload`) is sized 16B while the value is 8B → every lowering-time
|
||
> type-fn bails at `const_type` / the Member-array build. Can't make `kindOf(.any)` a word:
|
||
> at EMIT time `.any` really is a 16B box (variadic any, 0603), so that would silently corrupt
|
||
> it. The correct fix is a **dedicated `Type` builtin TypeId (8B), distinct from `.any`** —
|
||
> measured at **~123 `.any` references across ~25 files** (pack.zig has 30), a ~100-touch-point
|
||
> cross-cutting change → its own focused session (USER CHOSE to pause rather than rush it).
|
||
> Rejected alternatives: a scoped "lowering-mode treats `.any` as a word" flag (silent-wrong on
|
||
> a real Any box in a reflection type-fn); scalar-only Type-fns (safe but no real corpus type-fn
|
||
> is scalar-only — they all build a Member/variant aggregate).
|
||
>
|
||
> **Decisions recorded:** `find_type` returns a non-optional `TypeId` using `unresolved`(0), NOT
|
||
> `?Type`; reader names use the `type_*` family (avoid colliding with std `field_name`/`type_name`);
|
||
> the write side is a single kind-branching `register_type`; the write side stays LEGACY-only
|
||
> until the VM runs at lowering time (needs the `Type` TypeId). End-state guarantee: ONE
|
||
> evaluator — `interp.zig` deleted; dual-path + fallback are transitional (see PLAN end state).
|
||
> Build/verify: `zig build && zig build test` (**697**, gate OFF). Run the corpus ON the VM:
|
||
> `zig build test -Dcomptime-flat` OR env `SX_COMPTIME_FLAT=1`. Coverage trace:
|
||
> `SX_COMPTIME_FLAT_TRACE=1` (now also prints lowering-time `type-fn` HANDLED/fallback lines).
|
||
|
||
### (superseded) prior weld resume
|
||
Phase 1 done; Phase 2 welded structs were working via reflection + memory-order
|
||
validation (the `computeWeldPlan`/byte-blob "GEP engine" was explored + DROPPED even
|
||
earlier). A welded `Name :: struct abi(.zig) extern compiler { … }` declared fields in
|
||
the compiler type's MEMORY order; the compiler reflected the bound Zig type and
|
||
VALIDATED the header. **This whole mechanism is now being stripped — see the banner.**
|
||
|
||
> ⚠ Snapshot workflow: use `-Dname=examples/NNNN-foo.sx[,…] -Dupdate-goldens` to
|
||
> regenerate ONLY the named example(s) — a full `-Dupdate-goldens` re-runs all ~690
|
||
> and a flaky/host-divergent example (AOT/cross-arch) can clobber good snapshots.
|
||
> See CLAUDE.md → Snapshot integrity.
|
||
|
||
## Last completed step
|
||
**Phase 2 — welded structs by reflection + memory-order validation (byte-identical,
|
||
no GEP engine).** A welded `struct abi(.zig) extern compiler { … }` now works
|
||
end-to-end as a byte-identical mirror of the bound Zig type.
|
||
|
||
Design (locked, supersedes the byte-layout-override plan):
|
||
- The sx header declares fields in the compiler type's MEMORY order. The compiler
|
||
REFLECTS the bound Zig type — field names from `@typeInfo`, offsets from
|
||
`@offsetOf`, size from `@sizeOf` — and validates the header matches. Nothing is
|
||
maintained by hand; a `types.zig` change re-reflects on the next compiler build.
|
||
- On pass it's an ORDINARY struct whose natural layout already equals the Zig
|
||
layout → `@ptrCast` to the compiler type + deref is byte-identical. No
|
||
byte-blob, no index/remap tables, no reorder, no special LLVM path.
|
||
- Loud, precise diagnostics on any drift: *field not found* (+ memory order),
|
||
*wrong field order at position N* (+ expected memory order), *type layout
|
||
mismatch* (field size), *layout mismatch* (total size / count).
|
||
|
||
What changed from the dropped plan:
|
||
- `compiler_lib.zig`: `weldStruct` now REFLECTS field names (`@typeInfo`) and bakes
|
||
`bound_types` fields in ascending-OFFSET (memory) order — no hand-listed names.
|
||
Deleted `computeWeldPlan`/`WeldPlan`/`WeldElement`. `validateStructLayout` checks
|
||
the sx header against the memory-ordered registry.
|
||
- `nominal.zig` `validateWeldedStruct`: renders the precise diagnostics
|
||
(+ `weldedFieldOrderStr`).
|
||
- Examples: `0627` (StructInfo in memory order, byte-identical, usable);
|
||
`1186` (source-order StructInfo → wrong-field-order diagnostic). `1183` message
|
||
refreshed.
|
||
- `zig build` + `zig build test` green (692 corpus, unit tests pass).
|
||
|
||
### Earlier — Phase 2.1 (weld-plan layout math, now removed)
|
||
**The weld-plan offset math + `StructInfo` registered.** Was the core of the
|
||
byte-layout-override engine; superseded by the reflection+validation design above.
|
||
|
||
Decision (locked 2026-06-17): **full byte-layout weld** — a welded sx struct is
|
||
laid out byte-identically to the bound Zig type (Zig's `@offsetOf`, reordering +
|
||
padding included), so it passes to a Zig handler as raw memory with zero
|
||
marshalling. (The alternative — handlers reading interp `Value` aggregates
|
||
logically, no layout override — was rejected; welded types must also be usable as
|
||
runtime data, and the design wants the literal byte weld.)
|
||
|
||
- Measured: Zig reorders `StructInfo` to `fields`@0, `name`@16, `nominal_id`@20,
|
||
`is_protocol`@24, size 32 — vs sx-natural `name`@0, `fields`@8, … So the override
|
||
is genuinely required (`Field`'s two-u32 natural layout was the easy case).
|
||
- `compiler_lib.zig`: registered `StructInfo` (`weldStruct`, the second
|
||
`bound_types` entry). Added `WeldElement` / `WeldPlan` + `computeWeldPlan(alloc,
|
||
fields, total)` — pure: orders fields by ascending byte offset, inserts padding
|
||
elements for gaps + the alignment tail, and builds the sx-field → LLVM-element
|
||
remap. This is what the LLVM type builder + struct-GEP sites will consume.
|
||
- Unit-tested (`compiler_lib.test.zig`): `Field` → identity plan (2 elems, no pad);
|
||
`StructInfo` → 5 elems `[fields@0, name@16, nominal_id@20, is_protocol@24,
|
||
pad@25..32]`, remap `[1,0,3,2]`.
|
||
- `zig build` + `zig build test` green.
|
||
|
||
### Earlier — Phase 1 polish (comptime-only enforcement)
|
||
**A RUNTIME call to a `fn abi(.zig) extern compiler` is a clean build-gating error
|
||
instead of an undefined-symbol link failure.**
|
||
- `emitCall` (`src/backend/llvm/ops.zig`): when the callee is `compiler_welded`
|
||
AND the ENCLOSING function is not `is_comptime` (i.e. genuine runtime code, not a
|
||
`#run`/`::` initializer wrapper whose LLVM body is dead), print a clear
|
||
"comptime-only … cannot be called at runtime" error and set
|
||
`comptime_failed` (the driver halts before object/JIT emission). The enclosing
|
||
`is_comptime` guard is what keeps the legitimate `#run` use (example 0626) green.
|
||
- Corpus: `examples/1185-diagnostics-weld-fn-runtime-call.sx` (runtime `intern(…)`
|
||
→ clean error, exit 1, no link failure).
|
||
- `zig build` + `zig build test` green (458 unit + 690 corpus).
|
||
|
||
### Earlier — fifth sub-step (host-call bridge)
|
||
**A `fn abi(.zig) extern compiler` dispatches, under the comptime interpreter, to
|
||
its registered Zig handler instead of dlsym.**
|
||
- `compiler_lib.zig`: function registry — `BoundFn { sx_name, handler }`,
|
||
`bound_fns` = `intern(string)->StringId` + `text_of(StringId)->string` (the
|
||
string-pool round-trip), `findFn`, and `FnHandler` (`*Interpreter, []Value ->
|
||
Value`). `intern` mutates via `interp.mint orelse @constCast(&module.types)`
|
||
(the same mutable-table access the metatype mint path uses); `text_of` reads the
|
||
const pool. Imports `interp.zig` (the compiler_hooks↔interp cycle pattern).
|
||
- IR `Function` gained `compiler_welded: bool`. `declareFunction`
|
||
(`src/ir/lower/decl.zig`) sets it via `weldedCompilerFn`, which also VALIDATES:
|
||
the bound lib must be `compiler` and the name must be on the function-export
|
||
list — else a build-gating `.err` (no silent fall-through to dlsym).
|
||
- `interp.call()`: before the dlsym/extern path, a `compiler_welded` function
|
||
routes to `compiler_lib.findFn(name).handler(self, args)` (clean bail off the
|
||
export list).
|
||
- Corpus: `examples/0626-comptime-weld-fn-intern-text-of.sx` (`#run
|
||
text_of(intern("hello, compiler"))` folds to a string constant → prints it);
|
||
`examples/1184-diagnostics-weld-fn-unexported.sx` (unexported welded-fn name →
|
||
build error). `findFn` lookup unit-tested.
|
||
- **Runtime-call rejection is NOT yet clean** — welded fns are comptime-only; a
|
||
RUNTIME call would emit a reference to a non-existent extern symbol → a loud
|
||
LINK error (not silent, but not a tidy diagnostic). The examples call welded fns
|
||
only inside `#run`. A dedicated "comptime-only symbol" emit diagnostic is the
|
||
immediate follow-up.
|
||
- `zig build` + `zig build test` green (458 unit tests + 689 corpus).
|
||
|
||
### Earlier — fourth sub-step (welded-struct layout validation)
|
||
**A `struct abi(.zig) extern compiler { … }` is validated against the binding
|
||
registry as a *header checked against the implementation*.**
|
||
- `compiler_lib.zig`: `validateStructLayout(bt, sx_fields, total)` — pure, returns
|
||
the first `LayoutMismatch` (field count / name / size / total) or null. Plus
|
||
`lib_name = "compiler"` and `SxField`. Unit-tested (faithful `Field` passes;
|
||
each drift flagged as the right variant).
|
||
- `registerStructDecl` (`src/ir/lower/nominal.zig`): for `sd.abi == .zig`,
|
||
`validateWeldedStruct` checks the bound lib is `compiler`, the name is on the
|
||
export list (`findType`), and the sx layout (field names + `typeSizeBytes` +
|
||
total) matches the welded type — emitting a build-gating `.err` (good span into
|
||
the struct body) on any failure. No silent reinterpretation.
|
||
- `#library "compiler"` is the comptime-only internal surface, NOT a dylib —
|
||
`src/main.zig`'s dlopen walker skips it (was emitting a spurious `libcompiler.so`
|
||
load warning).
|
||
- Corpus: `examples/0625-comptime-weld-struct-field.sx` (faithful `Field` welds,
|
||
validates, usable as data → `name=7 ty=3`); `examples/1183-diagnostics-weld-
|
||
struct-field-count.sx` (one-field `Field` → build-gating field-count diagnostic).
|
||
- **Offset-override / GEP emission for non-natural Zig layouts is NOT here** — it
|
||
isn't exercised by `Field` (two u32s = natural layout coincides with the weld).
|
||
It arrives with `StructInfo` in Phase 2 (slices/reordering), where the bound
|
||
offsets actually differ from the sx-natural ones. The validation already checks
|
||
per-field size + total, so a layout drift is caught even before the override
|
||
engine exists.
|
||
- `zig build` + `zig build test` green (456 unit tests + 687 corpus).
|
||
|
||
### Earlier — third sub-step (binding registry)
|
||
**The binding registry (welded-type lookup, layout baked from the real Zig
|
||
type).**
|
||
- New `src/ir/compiler_lib.zig` — the `compiler` library's binding registry, the
|
||
curated safety boundary. `BoundType { sx_name, size, alignment, fields:
|
||
[]FieldLayout{name, offset, size} }`; `weldStruct` bakes the layout from a real
|
||
Zig struct via `@sizeOf`/`@alignOf`/`@offsetOf` at compiler-build time (a
|
||
sx-field-count mismatch is a `@compileError`, never a silent truncation).
|
||
`bound_types` exports `Field` (welded to `types.TypeInfo.StructInfo.Field` —
|
||
two `u32`s); `findType(sx_name) ?*const BoundType` is the lookup the welded-decl
|
||
resolution path will consult (returns null off the export list — clean boundary,
|
||
no silent default).
|
||
- Registered in the barrel (`src/ir/ir.zig`): `compiler_lib` + `compiler_lib_tests`.
|
||
- Tests (`src/ir/compiler_lib.test.zig`): `findType("Field")` equals the real
|
||
`StructInfo.Field` `@sizeOf`/`@alignOf`/`@offsetOf` (8 bytes, two u32s at 0/4);
|
||
an unexported name returns null. Break-verified (a wrong size → suite red,
|
||
named `ir.compiler_lib.test...`).
|
||
- `zig build` + `zig build test` green (454 unit tests).
|
||
|
||
### Earlier — second sub-step (struct-decl parse)
|
||
**`abi(.zig) extern <lib>` PARSES on a STRUCT decl (parse-only, no semantics).**
|
||
- `ast.StructDecl` gained `abi: ABI` + `extern_lib: ?[]const u8` binding fields.
|
||
- `parseStructDecl` (`src/parser.zig`): after `struct` (and the `#compiler`
|
||
check), parse an optional `abi(...)` then optional `extern <lib>` — same slot
|
||
order as fn decls — and thread them onto the node. Ordinary structs are
|
||
unperturbed (`parseOptionalAbi`/`parseOptionalExternExport` no-op when absent).
|
||
- Parser unit tests (`src/parser.test.zig`): `Field :: struct abi(.zig) extern
|
||
compiler { name: StringId; ty: Type; }` parses with `abi == .zig`, `extern_lib
|
||
== "compiler"`, field list intact; a plain struct leaves `abi == .default` /
|
||
`extern_lib == null`. Break-verified (a wrong-sentinel assert turns the suite
|
||
red, confirming the test runs).
|
||
- `zig build` + `zig build test` green.
|
||
|
||
### Earlier — first sub-step (fn decls) + the syntax pivot
|
||
**`abi(.zig) extern <lib>` PARSES on a fn decl (parse-only).** Plus the syntax
|
||
pivot it required.
|
||
|
||
Syntax decision (locked 2026-06-17, supersedes the doc's original
|
||
`extern(.zig) <lib>` single-qualifier form): the ABI/layout selector and the
|
||
linkage keyword are two orthogonal annotations.
|
||
- `abi(.x)` — ABI / calling-convention annotation in the slot **before**
|
||
`extern`/`export`. **Unified replacement for `callconv(...)`, which is removed.**
|
||
`ABI = { default, c, zig, pure }`: `.c` (C ABI), `.zig` (Zig-layout weld → the
|
||
`compiler` library), `.pure` (naked asm), `.default` (unannotated). Can appear
|
||
standalone (no extern) on any fn / fn-type / lambda.
|
||
- `extern <lib>` — linkage keyword + binding source (named library).
|
||
|
||
So a welded binding is `text_of :: (id: StringId) -> string abi(.zig) extern compiler;`.
|
||
|
||
What landed:
|
||
- **AST** (`src/ast.zig`): `CallingConvention` → `ABI { default, c, zig, pure }`;
|
||
the `call_conv` field → `abi: ABI` on `FnDecl` / `Lambda` / `FunctionTypeExpr`.
|
||
- **Lexer/token** (`src/token.zig`, `src/lexer.zig`): `kw_callconv` → `kw_abi`,
|
||
keyword string `"callconv"` → `"abi"`.
|
||
- **Parser** (`src/parser.zig`): `parseOptionalCallConv` → `parseOptionalAbi`
|
||
(parses `abi(.c|.zig|.pure)`); wired in the fn-decl postfix slot (before
|
||
`extern`/`export`), the function-type-expr slot, and the lambda slot;
|
||
`isFunctionDef`/`hasFnBodyAfterArrow` recognise `kw_abi`.
|
||
- **AST→IR map** (`src/ir/type_resolver.zig`, `src/ir/lower/decl.zig`, `sema.zig`,
|
||
`closure.zig`): the AST `.abi == .c` reads kept their C-ABI meaning; the
|
||
function-type resolver maps `.zig`/`.pure` → IR `.default` (no fn-pointer-type
|
||
CC for those decl-level ABIs; neither occurs in a function-TYPE position yet).
|
||
- **CC-mismatch diagnostic** (`src/ir/lower/expr.zig`, `src/sema.zig`): the
|
||
user-facing text `callconv(.c)` → `abi(.c)`.
|
||
- **sx migration**: 52 `.sx` files `callconv(` → `abi(` (all were function-type
|
||
callback annotations — none in the fn-decl postfix slot, so no reordering).
|
||
- **Docs**: `readme.md`, `specs.md`, the design doc, snapshots (0114 / 1104 /
|
||
1200) regenerated for the rename.
|
||
- **Tests**: parser unit tests in `src/parser.test.zig` — `abi(.zig) extern <lib>`
|
||
on a fn decl (asserts `abi == .zig`, `extern_export == .extern_`, `extern_lib ==
|
||
"compiler"`); bare `extern` leaves `abi == .default`; standalone `abi(.c)` /
|
||
`abi(.pure)`. lexer/sema tests updated.
|
||
|
||
`zig build` + `zig build test` green (450/450 unit + 685 corpus).
|
||
|
||
## Current state
|
||
|
||
> **Pivoted — see the banner + `PLAN-COMPILER-VM.md`.** The items below are the weld
|
||
> machinery as it stands on `reify` HEAD (`40d075c`); they are the **strip list** for
|
||
> Phase 0, not the forward direction. The `#library`/`abi`/`extern` *syntax* stays; the
|
||
> weld *semantics* (layout reflection/validation, marshaling dispatch) go.
|
||
|
||
- `compiler :: #library "compiler";` parses + is recognised as the comptime-only
|
||
internal surface (never dlopen'd).
|
||
- `abi(.zig) extern compiler` STRUCTS: layout-validated against the registry
|
||
(faithful → ok; drift → build-gating diagnostic). `Field` welds + usable.
|
||
- `abi(.zig) extern compiler` FUNCTIONS: dispatched under the comptime interp to
|
||
their registered Zig handler (`intern`/`text_of` round-trip works); unexported
|
||
names rejected at declaration. Comptime-only.
|
||
- A RUNTIME call to a welded fn is a clean build-gating error (comptime-only
|
||
enforcement at `emitCall`); the legitimate `#run`/`::` use stays green.
|
||
- The whole Phase 1 foundation (parse → registry → struct-layout validation →
|
||
function host-call bridge → comptime-only enforcement) is in place for the
|
||
two-u32 `Field` case + the two string readers.
|
||
- **Deferred**: offset-override / LLVM byte-offset GEP for non-natural layouts
|
||
(needed by `StructInfo`'s slice field, Phase 2).
|
||
|
||
## Next step — execute `PLAN-COMPILER-VM.md`
|
||
|
||
> The weld is being stripped. The next step is **Phase 0 of
|
||
> [`PLAN-COMPILER-VM.md`](PLAN-COMPILER-VM.md)** — remove the weld / serialize /
|
||
> marshal machinery (`compiler_lib.zig` reflection+validation, `nominal.zig`
|
||
> `validateWeldedStruct`, the `compiler_welded` dispatch, the weld examples/diagnostics
|
||
> 0625/0627/1183/1184/1185/1186), keeping the `#library`/`abi`/`extern` *syntax*. Then
|
||
> Phase 1 (flat-memory value model). The weld-era "next step" below is **obsolete** —
|
||
> kept only as a record of what the weld surface was about to do.
|
||
|
||
### (obsolete) weld-era next step
|
||
Welded structs were byte-identical mirrors, so the API surface was set to grow:
|
||
|
||
- **Bind `register_struct` / `find_type`** over the host-call bridge
|
||
(`compiler_lib.zig` `bound_fns`, like `intern`/`text_of`). `register_struct`
|
||
takes a welded `StructInfo` and mints a real `TypeId` (guarded: dup field names,
|
||
kind well-formedness — the checks `define` does today). Because the welded
|
||
`StructInfo` is byte-identical, the handler can read it as the real Zig
|
||
`*StructInfo` (cast + deref) rather than marshalling a `Value` field-by-field —
|
||
the payoff of the byte-weld. `find_type(StringId) -> ?Type` reads the table.
|
||
Prove: build a struct programmatically + round-trip a source one.
|
||
- **Re-express `type_info`/`define` (struct) as sx** over `register_struct`/
|
||
`find_type`; migrate `examples/0622`; delete the bespoke struct interp arms
|
||
(`defineStruct` / the `reflectTypeInfo` struct path).
|
||
|
||
Then Phase 3+: widen the welded types to `EnumInfo`/`TaggedUnionInfo`/`TupleInfo`
|
||
(optional fields → sentinels) — each just needs an sx header in the compiler
|
||
type's memory order + the matching `register_*` fn. Finally migrate `BuildOptions`
|
||
to `abi(.zig) extern compiler` (re-home the `#compiler` registry) and delete
|
||
`#compiler`.
|
||
|
||
Note: a welded struct with an `?T` / `union(enum)` field (e.g. `EnumInfo`'s
|
||
`backing_type: ?TypeId`, `explicit_values: ?[]const i64`) is the next layout
|
||
wrinkle — the sx header must mirror Zig's optional/union representation. Handle
|
||
when reached (sentinels or accessor fns; see the design doc Risks).
|
||
|
||
## Known issues
|
||
- None for this stream. (Metatype's deferred enhancement is issue 0141 — comptime
|
||
`List` growth; orthogonal, see `current/CHECKPOINT-METATYPE.md`.)
|
||
|
||
## Log
|
||
- **Phase 4 burndown — switch_br + type_name ported; issue 0143 filed; KEY sequencing insight: `out` is end-state-only (2026-06-18).**
|
||
Ported two PURE comptime ops (`379ed05`): `switch_br` (i64-discriminant multi-way branch — enum/error
|
||
tag or `.type_value` index) and `type_name` (Type value / Any box → `table.typeName`, with an
|
||
`.unresolved`-bail guard). Correct in isolation; 0520–0524 run GREEN under strict. **Two blockers found:**
|
||
1. **issue 0143 (FILED, OPEN) — pack-as-`[]Type` stride mismatch.** A `..$args` pack forwarded as a
|
||
`[]Type` ARGUMENT across a call is backed by a `[N x Any]` (16B) array but viewed as `[]type_value`
|
||
(8B) → half-stride reads (`[i64 <unresolved> string]` vs legacy `[i64 string bool]`). A LOWERING bug
|
||
the legacy's Value model masks; the byte-accurate VM exposes it. Blocks `examples/0114` from running
|
||
HANDLED. **Per CLAUDE.md: filed, NOT worked around** (the `type_name` `.unresolved` guard just makes
|
||
the VM decline rather than emit garbage). Repro + fix-prompt in `issues/0143-…md`.
|
||
2. **`out` (comptime print) is an END-STATE op — it cannot land while the fallback exists.** Under the
|
||
legacy fallback, an eval that prints via `out` then BAILS double-prints (the VM wrote to fd 1, then
|
||
legacy re-runs the whole eval — no rewind). 0114 demonstrated it. So a direct-write `out` is only
|
||
safe once the fallback is GONE (strict-by-default). **Revised ordering:** land the PURE ops
|
||
(switch_br/type_name/type_is_unsigned/error_tag_name_get/global_addr/interp_print_frames) + the
|
||
BuildOptions migration + #insert + bundler FIRST; then in the FINAL step flip strict-to-default
|
||
(removing the fallback) AND add `out` together — at which point every `out`-using example flips
|
||
atomically with deletion. (Most of the gap-list examples print, so they stay on fallback until that
|
||
final flip — that's expected, not a regression.) 699/0 both default gates.
|
||
- **Phase 4 — STRICT no-fallback mode (the interp-retirement enumeration gate) + full gap list (2026-06-18).**
|
||
Added `-Dcomptime-flat-strict` / env `SX_COMPTIME_FLAT_STRICT` (implies `comptime_flat`): at all
|
||
THREE comptime sites (type-fn in `lower/comptime.zig`, const-init + `#run` in `emit_llvm.zig`) a VM
|
||
bail becomes a build-gating error naming the reason INSTEAD of falling back to legacy. This forces
|
||
every comptime eval onto the VM so the complete gap set is enumerable in one sweep; when the corpus
|
||
is green under strict mode AND every example MATCHES legacy, the VM handles everything and
|
||
`interp.zig` can be deleted (4F). Default behaviour unchanged — **699/0 both default gates**.
|
||
(Fixed a wiring bug: the type-fn site's local `comptime_flat` didn't include the strict flag, so
|
||
every type-fn falsely reported `<unknown>`; now strict implies flat there too.)
|
||
**THE DELETION CHECKLIST (19 strict bails, swept via `SX_COMPTIME_FLAT_STRICT=1` over examples+issues;
|
||
0103/0800 "WRONG" were false positives — raw heap-pointer addresses the corpus normalizes):**
|
||
- `switch_br` (5): 0114, 0521, 0522, 0524, 1035 — port the type-category multi-way branch (trivial
|
||
jump). **CAUTION:** porting it (+`type_name`) UNMASKS a silent-wrong in 0114 — a `[]Type` slice
|
||
materialized when a pack (`$args`) is passed ACROSS A CALL reads its `string` element as
|
||
`<unresolved>`. Must fix that VM pack-Type-materialization bug, not just add the op.
|
||
- `compiler_call` (6): 0602, 0603, 1604, 1609, 1611, 1615 — the **BuildOptions → `abi(.zig) extern
|
||
compiler`** migration (delete `#compiler`/`compiler_call`; thread `BuildConfig` into the VM). Big.
|
||
- `out` (2): 0613, 1038 — comptime print. Direct write to fd 1, BUT only safe when the WHOLE eval is
|
||
VM-handled (a print-then-bail double-prints under the legacy re-run — 0613). Flip atomically.
|
||
- `type_name` (1): 0520 — reflection reader (`.type_value` word / Any-box tag → `table.typeName`).
|
||
- `global_addr` (1): 0600 — only `&__sx_default_context` is materialised (mirror legacy).
|
||
- `interp_print_frames` (1): 1034 — return-trace frame printing.
|
||
- VM-native diagnostics (4B) (2): 1179, 1180 — NEGATIVE tests; the VM bail (`define: enum has no
|
||
variants` / `duplicate variant name`) IS the expected outcome → must surface as the proper
|
||
build-gating diagnostic, not the generic strict error.
|
||
- dlsym not found (1): 1654 — a target-specific `extern` (asm global) called at comptime; likely a
|
||
legitimately-unresolvable case → confirm it stays a clean diagnostic.
|
||
**Sweep command:** `SX_COMPTIME_FLAT_STRICT=1 ./zig-out/bin/sx run <ex>` per example, diff vs legacy;
|
||
a strict bail prints `... bailed on the VM (strict, no fallback): <reason>`.
|
||
- **Phase 4D.2 (VM plan) — extern SLICE/string args (→ NUL-terminated `char*`) + float guards (2026-06-18).**
|
||
Extracted `marshalExternArg`: a scalar/pointer WORD passes verbatim (a `cstring` arg already works
|
||
as a pointer word via 4D.1); a `string`/slice `{ptr,len}` fat pointer is copied into a
|
||
NUL-terminated arena buffer and its `char*` passed (mirrors legacy `marshalExternArg` — what the
|
||
bundler's `popen(cmd: [:0]u8, …)` needs). Added FLOAT guards on args AND returns: floats are
|
||
`kindOf == .word` but the host_ffi trampolines have no float variant, so they bail loudly rather
|
||
than miscall through an integer register (the legacy interp doesn't support float FFI either, so
|
||
parity holds — no corpus float-FFI example exists). New example `0637-comptime-extern-slice-arg`
|
||
(`#run strlen("hello, world")` with a `[:0]u8` param → 12) runs **HANDLED on the VM**, byte-matching
|
||
legacy. **699/0 BOTH gates.** On `reify`. The FFI escape is now complete for scalar/pointer/cstring/
|
||
slice args + scalar/pointer returns — enough for the bundler's libc surface. **Next (4D.3):**
|
||
`compiler_call` (#compiler hooks — 0602/0603), the last legacy-only role besides #insert/bundler.
|
||
- **Phase 4D.1 (VM plan) — general host-FFI escape: the VM calls any extern libc fn via dlsym + host_ffi (2026-06-18).**
|
||
Replaced the "extern not ported → bail" stub in `Vm.invoke` with `callHostExtern`: resolve the
|
||
symbol via `host_ffi.lookupSymbol` (dlsym RTLD_DEFAULT) and dispatch through the `host_ffi`
|
||
trampolines, exactly like the legacy `interp.callExtern`. **Marshalling is now trivial because
|
||
`Addr` is a real host pointer (4D.0):** every WORD-kind arg passes as `usize` verbatim — a
|
||
scalar's bits OR a pointer, no translation — and a pointer return is a valid `Addr`. Picks
|
||
`callPtrRet` (void*-ABI) for pointer-ish returns, `callIntRet` (i64-ABI) otherwise; honors
|
||
variadic (`is_variadic and args > fixed`). Non-word (aggregate/string/float) args+returns bail
|
||
loudly (no silent miscall — 4D.2 adds NUL-term cstring marshalling + float). NOT per-builtin: ONE
|
||
general mechanism for all externs. New example `0636-comptime-extern-libc` (`#run toupper(97)`/
|
||
`tolower(90)` fold to 65/122) runs **HANDLED on the VM**, output byte-matching legacy. (`abs`
|
||
doesn't dlsym-resolve on macOS — a compiler builtin — and the VM fails identically to legacy,
|
||
confirming parity.) **698/0 BOTH gates** (one new example). On `reify`. **Next (4D.2):**
|
||
string/aggregate extern args (string→NUL-term cstring) + float args/returns, then `compiler_call`
|
||
(#compiler hooks, 4D.3).
|
||
- **Phase 4D.0 (VM plan) — comptime VM memory = an ARENA of stable host allocations; `Addr` = real host pointer (2026-06-18).**
|
||
Replaced the growable `ArrayList(u8)` flat buffer (which reallocs/MOVES on growth) with a
|
||
`std.heap.ArenaAllocator`: each `allocBytes` is a separate arena allocation that never moves and
|
||
is freed wholesale on `deinit` (no per-object free, no cap, no fixed buffer). **`Addr` is now the
|
||
allocation's absolute host pointer** (`@intFromPtr`), not an offset — so a flat-memory pointer and
|
||
an FFI-returned host pointer are the SAME kind of value, and the FFI bridge (4D.1) can pass them
|
||
to/from libc with ZERO translation and no per-call pinning (the original moving-buffer hazard is
|
||
gone by construction). `Machine.readWord/writeWord/bytes` deref the absolute pointer directly,
|
||
keeping the null-check bail (the malformed-IR / null-deref safety contract). Dropped the
|
||
offset-based upper-bounds check (can't bound an absolute pointer; the `Frame.bad_ref` guard still
|
||
catches the dominant malformed-IR vector) and the test-only `mark`/`reset` (the arena has no
|
||
cheap reset-to-mark; the VM never used them outside tests). Decision rationale (user): use a
|
||
GPA-like allocator, no artificial buffer limits. **697/0 BOTH gates + all unit tests** (rewrote
|
||
the two Machine tests: null-deref bail + arena-stability-across-grows). Pure refactor, no
|
||
comptime behavior change. **Next (4D.1):** extern-call dispatch in `Vm.invoke` — marshal args
|
||
(scalars by value, pointers as the host pointer they already are), call via `host_ffi`
|
||
trampolines, return scalars/pointers; a new `#run` libc example as the corpus guard.
|
||
- **Phase 4A.1 (VM plan) — `box_any`/`unbox_any` on the VM + `.any` as a 16-byte aggregate (2026-06-18).**
|
||
Ported the Any-boxing conversion pair: `box_any` allocates the 16-byte `{ type_tag@0, value@8 }`
|
||
box (tag = source TypeId index, matching the legacy comptime interp), writing a word source's
|
||
scalar via `writeField(source_type)` (so f32 round-trips) or an aggregate source's flat-memory
|
||
ADDR (the runtime pointer-in-value-slot shape); `unbox_any` reads the value slot back (word →
|
||
`readField`, aggregate → the stored ADDR). **Required making `.any` a first-class flat-memory
|
||
aggregate** (it was `kindOf → .unsupported`): `kindOf(.any) = .aggregate` (16B, by-address) +
|
||
`fieldOffset` special-cases `.any` to the `{@0, @8}` layout (shared with string/slice) — without
|
||
the latter, a `struct_get` on an Any panicked (`union field 'struct' while 'any' is active`),
|
||
caught + fixed (no crash; "never crash" upheld). Updated two unit tests that used `unbox_any` as
|
||
the "unported op" example → now `compiler_call`; added a box→unbox round-trip test. **697/0 BOTH
|
||
gates + all unit tests.** On `reify`. The 6 box_any examples (0114/0520–0524/1035) no longer bail
|
||
at box_any and produce VM output byte-matching legacy, but are not YET fully HANDLED — they now
|
||
fall back further at `switch_br` (comptime Any-tag type-switch), `type_name`, and `out`/print
|
||
(4A.2+/later steps). **Next (4A.2):** comptime `out`/print (VM output buffer + flush).
|
||
- **Phase 3 P3.4 step 8 (VM plan) — VM-native `type_info` REFLECTION → the whole metatype surface is HANDLED (2026-06-18).**
|
||
Ported `type_info($T)` into the VM (`callBuiltinVm` `.type_info` arm → new `buildTypeInfo`), the
|
||
inverse of step 7's `define`: reflect a type INTO a `TypeInfo` VALUE built in FLAT MEMORY (the
|
||
VM-native mirror of legacy `reflectTypeInfo`). Decodes the source type into a tag + members
|
||
(tagged-union/struct field & enum variant → `{ name, ty }`, a payloadless variant → `void`;
|
||
tuple → bare positional `Type`s), then lays out the nested value bottom-up using layouts derived
|
||
from the `TypeInfo` RESULT type (`ins.ty`, now threaded into `callBuiltinVm`): element array →
|
||
`{ptr,len}` slice → info struct (`EnumInfo`/`StructInfo`/`TupleInfo`) → `TypeInfo { tag, payload }`
|
||
tagged union (reusing step 7's tagged-union write). Variant/field names materialize via a
|
||
`makeStringValue` helper extracted from `text_of`. Same `backing_type` guard as step 7. **Result:
|
||
the ENTIRE metatype surface runs HANDLED on the VM with ZERO fallback** — `0614`–`0624` + `0632`
|
||
(0616 `field_type` folds at lower time, no comptime eval); the `define(declare, type_info(T))`
|
||
round-trips (`0619`/`0622`/`0623`) mint byte-identical copies on the VM. VM output byte-matches
|
||
legacy for all. **697/0 BOTH gates + all unit tests.** On `reify`. **Remaining VM fallbacks in the
|
||
comptime corpus are now genuinely-non-metatype** emit-time side effects: `print`/`out` (0613),
|
||
`global_addr` (0600), `compiler_call` #compiler hooks (0602/0603), and the inline-asm global
|
||
(1654). **Next:** port those (or confirm each is a legitimately-non-comptime case) to drive the
|
||
fallback list to empty, then — with user go-ahead — flip the VM to default + delete `interp.zig`.
|
||
- **Phase 3 P3.4 step 7 (VM plan) — VM-native metatype CONSTRUCTION: `declare`/`define` + tagged-union `enum_init` (2026-06-18).**
|
||
Ported the metatype type-CONSTRUCTION builtins into the VM so the construction examples run
|
||
HANDLED end-to-end (no `call_builtin` fallback). Three pieces: (1) **tagged-union `enum_init`
|
||
with payload** — the arm previously bailed; now allocates the value (zeroed), writes the tag at
|
||
offset 0 (`{ header(tag)@0, [N x i8] payload@tag_size }`, the LLVM `backend/llvm/types.zig`
|
||
layout) and copies the payload at `tag_size`. (2) A **`.call_builtin` exec arm** → new
|
||
`callBuiltinVm`, the VM-native mirror of the legacy `execBuiltinInner`: `declare(name)` mints an
|
||
empty forward nominal slot (shared `declareNominal` helper, also used by `declare_type`);
|
||
`define(handle, info)` reads the `TypeInfo` tagged-union VALUE from FLAT MEMORY (tag@0, active
|
||
payload `EnumInfo`/`StructInfo`/`TupleInfo` struct at `tag_size`, its single slice field) and
|
||
mints via `defineFromInfo`, a faithful port of legacy `defineEnum`/`defineStruct`/`defineTuple`
|
||
(all-void enum → real `.@"enum"` per issue 0142, dup-name rejection, `updatePreservingKey` vs
|
||
`replaceKeyedInfo`). (3) Refactored the `[]{name,ty}` decode out of `registerTypeVm` into a
|
||
shared `decodeMemberSlice` (+ `decodeTypeSlice` for bare-`Type` tuple elements), keyed to the
|
||
module-level `NamedMember`. Unmodeled builtins (`type_info`/`type_name`/…) return null → bail
|
||
with the builtin name → legacy fallback (dual-path parity). **Correctness guard (caught via
|
||
review):** `enum_init`/`define` assume a tag-headed layout, which is WRONG for a `backing_type`
|
||
tagged union (laid out as the backing struct) — both now bail loudly on `backing_type != null`
|
||
rather than silent-clobber. **Result:** examples `0614`/`0620`/`0621`/`0624`/`0632` run **fully
|
||
HANDLED** on the VM (define is the whole eval); `0622`/`0623` run define HANDLED then fall back
|
||
cleanly at the still-unported `type_info` reflection. VM output byte-matches legacy for all 7.
|
||
**697/0 BOTH gates + all unit tests (added: tagged-union `enum_init` payload layout).** On
|
||
`reify`. **Next:** port `type_info` (REFLECT a type → build a `TypeInfo` value in flat memory,
|
||
the inverse — reuses the tagged-union `enum_init` write) so `0619`/`0622`/`0623` go fully HANDLED;
|
||
then the rest of the comptime corpus (drive the SX_COMPTIME_FLAT_TRACE fallback list toward the
|
||
genuinely-non-comptime cases) before the VM-default flip + legacy deletion.
|
||
- **Phase 3 P3.4 step 6 (VM plan) — REAL lowering-time Context: allocating + List-building type-fns now run HANDLED on the VM (2026-06-18).**
|
||
The VM can now evaluate a comptime type-fn that ALLOCATES at lowering time (the 0141 family) —
|
||
the legacy interp cannot. Four changes: (1) `runComptimeTypeFunc` (lower/comptime.zig) FORCES the
|
||
CAllocator→Allocator thunks to exist (`getOrCreateThunks`, idempotent, guarded by Allocator/
|
||
CAllocator registered) BEFORE eval — a type-fn const runs at scanDecls (Pass 1), before Pass 1c
|
||
builds the default-context global + thunks, so the comptime allocator was otherwise null;
|
||
(2) `materializeDefaultContext` builds a REAL context at lowering time when the global is absent —
|
||
finds the two thunks by name (`findFuncByName`) and lays their func-refs into the inline
|
||
`Allocator` value `{ctx=null, alloc_fn@+ptr, dealloc_fn@+2*ptr}` at the head of `Context`, so
|
||
`context.allocator.alloc_bytes` dispatches `call_indirect` → thunk → native VM `malloc`;
|
||
(3) `aggType` now DEREFS a pointer `base_type` (the List write path emits `struct_gep` with
|
||
`base_type = *Struct` — `fieldOffset` panicked on the pointer; now derefs to the pointee, no
|
||
panic); (4) `subslice` handles a `[*]T` many-pointer / `*T` base (a List's `items` field — the
|
||
base IS the data pointer). **Verified end-to-end (manual probe):** a compiler-API type-fn that
|
||
builds its `[]Member` in a `List(Member)` (`.append` ×3, then `register_type(handle, kind,
|
||
vs.items[0..vs.len])`) runs **HANDLED on the VM** and mints correctly (`green=7`) — the exact
|
||
0141 List-growth pattern, on the VM. **Can't be a corpus test yet** (gate-OFF/legacy still can't
|
||
allocate at lowering time — the dual-path bind), so locked in via VM unit tests instead
|
||
(many-pointer subslice; `struct_gep` with a pointer `base_type`). **697/0 BOTH gates + all unit
|
||
tests, EXIT=0.** On `reify`. **Remaining for the original 0141 repro (uses metatype `define`/
|
||
`make_enum` → `call_builtin` → legacy fallback → legacy fails):** re-express the metatype over the
|
||
compiler-API so the whole type-fn runs on the VM (no `call_builtin`). THEN the repro works on the
|
||
VM — and the dual-path bind resolves only at the VM-default-flip + legacy-deletion end-state.
|
||
- **Phase 3 P3.4 — investigation: the "real lowering-time Context" is BLOCKED by issue 0141 (2026-06-18).**
|
||
Probed whether the VM needs a REAL lowering-time `Context` (CAllocator thunk func-refs) for
|
||
allocating type-fns. **Finding: lowering-time comptime ALLOCATION fails in the LEGACY interp
|
||
too** — a type-fn that calls `context.allocator.alloc_bytes` at lowering time bails in legacy
|
||
with `comptime call_indirect: callee is not a func_ref Value (raw fn-pointers from extern calls
|
||
aren't dispatchable in interp)`, and the VM bails at parity (`call_indirect through a null
|
||
function pointer`). This is exactly issue **0141**'s root cause (its analysis already notes "the
|
||
null allocator is the same story for the CAllocator thunks") — an OPEN deferred issue. So:
|
||
(1) the VM is CORRECT (parity — both bail; no regression); (2) the real-context work is
|
||
PREMATURE — its only consumer (allocating lowering-time type-fns) can't pass gate-OFF, so no
|
||
corpus test can validate it, and even a more-capable VM can't ship a divergence during the
|
||
dual-path phase. **Consequence for the metatype re-expression:** re-expressing `define`/`make_enum`
|
||
over the compiler-API needs to BUILD `[]Member` slices dynamically (allocation) — which is
|
||
blocked by 0141 at lowering time. The viable paths are: (a) avoid allocation by passing the
|
||
caller's existing slice through (needs `EnumVariant`/`StructField` to be usable AS `Member` —
|
||
they're layout-identical `{string, Type}`, but distinct nominal types — a metatype-API decision),
|
||
or (b) wait for 0141. **No code change this step** (the VM already bails correctly). Recorded so
|
||
the next session doesn't re-derive it. 697/0 both gates unchanged.
|
||
- **Phase 3 P3.4 step 5 (VM plan) — WRITE side ported to the VM → FIRST HANDLED lowering-time type-fns (2026-06-18).**
|
||
Ported `declare_type` / `pointer_to` / `register_type` into `Vm.callCompilerFn`, mirroring the
|
||
legacy `compiler_lib` handlers (mint via `@constCast(table)` — the same mutable access the
|
||
read-side `intern` uses; the lowering-time mint target IS `&module.types`). `register_type`
|
||
reads the `[]Member` slice from FLAT MEMORY: threaded `ref_types` through `invoke` →
|
||
`callCompilerFn` so the slice's element type (`Member = {name: string, ty: Type}`) gives the
|
||
field offsets + stride; decodes each `{name, ty}` and branches on `kind` (1 struct · 2 enum ·
|
||
3 tagged_union · 4 tuple) exactly as legacy (dup-name / payload-on-enum rejections, idempotent
|
||
re-fill via `nominalIdentOf`). **Key unblock:** the synthesized comptime type-fn wrapper
|
||
(`createComptimeFunction`/`…WithPrelude`) was built with return type `.any` → `regToValue`
|
||
bailed at the VM↔legacy boundary; changed to `.type_value` (the legacy path reads via `asTypeId`
|
||
regardless, so no legacy change). **Result: the compiler-API write type-fns now run HANDLED
|
||
end-to-end on the VM at LOWERING time** — `0631` (register-graph: 2 HANDLED, A↔B cycle via
|
||
forward handles + `pointer_to`) and `0635` (multi-edge import: 2 HANDLED), parity-correct. They
|
||
run on the ZEROED lowering-time context (fixed `.[…]` member arrays, no allocation). The
|
||
metatype `make_enum`/`define` examples (`0632`) still fall back CLEANLY through
|
||
`call_builtin(define)` (the separate metatype path — re-expressing it onto the compiler-API is
|
||
the other half of P3.4). **697/0 BOTH gates + EXIT=0.** On `reify`. **Next:** (optional, deferred)
|
||
a REAL lowering-time Context (CAllocator thunk func-refs) for List-growing type-fns; and
|
||
re-express the metatype `define`/`make_enum` over the compiler-API to delete the bespoke interp
|
||
arms (the end-state: ONE evaluator).
|
||
- **Phase 3 P3.4 step 4 (VM plan) — model `.type_value` natively in the comptime VM (2026-06-18).**
|
||
The VM now HANDLES Type values instead of bailing: `kindOf(.type_value)` → `.word`; a new
|
||
`const_type` exec arm → the word `TypeId.index()`; `regToValue` maps a `.type_value` word back
|
||
to a `.type_tag` Value at the legacy boundary (`valueToReg` already mapped `.type_tag` →
|
||
index). Surfaced + fixed a VM PANIC (forbidden): `struct_init` assumed a `.@"struct"` result
|
||
type and union-access-panicked on an ARRAY literal (`EnumVariant.[ … ]`, reached now that Type
|
||
args no longer bail early) — it's the generic aggregate-literal op, so it now dispatches on the
|
||
result kind (struct / array / tuple) and BAILS loudly on anything else, never panics. **697/0
|
||
both gates** (the make_enum type-fns now run further on the VM, then bail cleanly at the
|
||
`define`/`make_enum` `call_builtin` → legacy mints — no mutation before the bail, parity holds).
|
||
VM unit test added (const_type → word → regToValue → `.type_tag`). On `reify`. **Next (the
|
||
payoff):** port the WRITE side (declare_type / register_type / pointer_to) into
|
||
`Vm.callCompilerFn` + give the lowering-time path a REAL Context (CAllocator thunk func-refs,
|
||
not zeroed) → the first HANDLED lowering-time type-fn end-to-end on the VM.
|
||
- **Phase 3 P3.4 step 3 (VM plan) — dedicated `Type` builtin TypeId: RESOLVER FLIPPED + `.any` migration (2026-06-18).**
|
||
Flipped `type_resolver:64` (`"Type"` → `.type_value`), `module.zig` `constType` (result type
|
||
→ `.type_value`), and `emitConstType` (a bare i64 carrying `tid.index()`, NOT a 16-byte Any
|
||
box). Then migrated every `.any` reference that means "a Type value", classified per CLAUDE.md
|
||
(leave the real boxed-Any refs): (a) the "Any holds a Type" **meta-marker tag** moved `.any` →
|
||
`.type_value` at all 4 consumers — `reflectArgTypeId` (LLVM), `reflectTypeId` + the
|
||
`.type_tag`-as-struct-field comptime path (interp), and `resolveTypeCategoryTags("type")`
|
||
(generic.zig); (b) reflection-builtin RETURN types `.any` → `.type_value` (`type_of`/`declare`/
|
||
`define`); the runtime `type_of(any)` now reads the tag AS a `.type_value` (no re-box); (c)
|
||
expr_typer infers a bare type-name expr as `.type_value` (with a `is_raw` backtick exemption —
|
||
`` `string `` is a value, never the reserved type); (d) `reflectionArgIsType` accepts
|
||
`.type_value` OR `.any` (a reflection arg can be a bare Type OR a boxed Any — the over-narrow
|
||
`==.type_value` was the catastrophic-regression cause, caught + fixed); (e) the comptime
|
||
`switch_br` accepts a `.type_tag` discriminant (type-category match); (f) a bare function name
|
||
in a `Type` slot now lowers to `const_type(its real function type)` instead of a func-ref
|
||
(fixed a JIT crash — was a func-ref word read as a TypeId), keeping the old string-box path only
|
||
for genuine `Any` params; (g) the field-not-found diagnostic + `formatTypeName` render
|
||
`.type_value` as "Type". Fixed 3 unit tests asserting the old `.any` Type behavior.
|
||
**697/0 BOTH gates** + all 494 unit tests (EXIT=0). Gate ON stays green because the VM's
|
||
`kindOf(.type_value)` → `.unsupported` → bails CLEANLY to legacy (no silent-wrong) — the VM
|
||
doesn't model `Type` values YET (next step), but parity holds. Regenerated 24 snapshots (22
|
||
`.ir` const_type-shape; 2 `.stderr` Any→Type — diff reviewed, only the intended changes). On
|
||
`reify`. **Next:** model `.type_value` natively in the VM (`kindOf` → word, `const_type` → word
|
||
= `TypeId.index()`, `regToValue` word → `.type_tag`) for COVERAGE, then port the WRITE side into
|
||
`callCompilerFn` + a real lowering-time Context → the first HANDLED lowering-time type-fn.
|
||
- **Phase 3 P3.4 step 2 (VM plan) — dedicated `Type` builtin TypeId: FOUNDATION landed (dead/additive) (2026-06-18).**
|
||
Added `TypeId.type_value` (slot 19) + a matching `TypeInfo.type_value` variant + the builtins
|
||
init entry — an **8-byte type handle distinct from the 16-byte boxed `.any`** (THE WALL). All
|
||
`types.zig` layout handlers wired: `sizeOf`/`typeSizeBytes` → 8, `typeAlignBytes` → 8,
|
||
`typeName` → "Type", `hashTypeInfo`/`typeInfoEql` no-payload arms. Only ONE exhaustive switch
|
||
needed a new arm (`backend/llvm/types.zig` `toLLVMTypeInfo` → `cached_i64`); every other
|
||
`switch(TypeInfo)` site has an `else` (audited when the resolver flips). **`first_user` 19 → 100**
|
||
(per the user): slots 20–99 are RESERVED builtin headroom (infos padded with the `unresolved`
|
||
tripwire), so future builtins don't renumber user TypeIds / churn `sx ir` snapshots. Cost:
|
||
~80 default entries in each binary's per-type reflection arrays (user opted in). **Still dead:**
|
||
`type_resolver.zig:64` STILL returns `.any` for "Type" — nothing produces `.type_value` yet, so
|
||
NO behavior change. Regenerated 22 IR snapshots (pure TypeId renumber to 100-base; `git diff
|
||
--name-only` confirmed ONLY `.ir` files + the 2 source files changed — no stdout/stderr/exit).
|
||
**697/0 both gates** (OFF and `-Dcomptime-flat`). **Next:** flip `type_resolver:64` →
|
||
`.type_value`, then migrate the `.any` refs that mean "a Type value" (const_type result /
|
||
reflection returns / metatype `Type` params / `.type_tag` checks) — leave the real boxed-Any
|
||
refs — file-by-file with a build after each.
|
||
- **Phase 3 P3.4 step 1 (VM plan) — lowering-time default context; first blocker cleared (2026-06-18).**
|
||
`materializeDefaultContext` now falls back to a ZEROED `Context` (found by name) when the
|
||
`__sx_default_context` global is absent — i.e. at LOWERING time, where the global isn't
|
||
emitted yet. A type-fn that never touches the allocator now runs past context setup; one
|
||
that allocates reads a null `alloc_fn` (zeroed) → `call_indirect` on the null func-ref
|
||
bails → legacy fallback (a REAL lowering-time context with the CAllocator thunk func-refs,
|
||
so allocating type-fns also run on the VM, is a follow-up). **Measurement: the bail moved
|
||
deeper** — metatype `make_enum` now bails at `const_type` (the `Type`-literal op, unported);
|
||
`register_type` type-fns bail at the welded write call (declare_type/register_type aren't in
|
||
`callCompilerFn`). No table mutation happens before either bail (the write fns bail before
|
||
minting), so parity holds: both gates **697/0**, no crashes. **Next blockers (the "model
|
||
Type" chunk):** (a) the `const_type` op → a word = `TypeId.index()`; (b) the Type-return
|
||
bridge (`regToValue` for a `Type`/`.any` word → `.type_tag`); (c) the VM-native write side
|
||
(declare_type/register_type/pointer_to in `callCompilerFn`) + a real lowering-time context.
|
||
Only once those land does a type-fn actually run end-to-end on the VM (a HANDLED case).
|
||
- **Phase 3 P3.4 (VM plan) — wire the VM at the LOWERING-time site + measure (2026-06-18).**
|
||
Routed `runComptimeTypeFunc` (the type-fn fold — the THIRD comptime call site) through
|
||
`comptime_vm.tryEval` behind `-Dcomptime-flat`/`SX_COMPTIME_FLAT` with legacy fallback,
|
||
mirroring the two emit-time folds. Extracted the shared post-check (`checkComptimeTypeResult`
|
||
— the declared-but-never-defined zero-field guard) so both paths use it. **Measurement
|
||
(SX_COMPTIME_FLAT_TRACE):** every metatype/compiler-API type-fn currently bails CLEANLY
|
||
with `no __sx_default_context global to materialize the implicit context` — at lowering
|
||
time the default-context global doesn't exist yet (it's built at emit time), so the VM bails
|
||
at context materialization, BEFORE running the body (no partial mint, no crash → legacy
|
||
mints). The hardening holds: **no crashes** across the corpus on the VM lowering-time path.
|
||
Both gates **697/0**. **So the FIRST lowering-time blocker is the implicit context, not
|
||
`Type` modeling** — the VM needs a way to materialize/skip the default context at lowering
|
||
time (most type-fns get an implicit ctx for potential `List`-growth alloc; many don't use
|
||
it). Next: materialize a lowering-time default context for the VM (or pass a null ctx +
|
||
bail only if the allocator is actually used), THEN model `Type` values + the VM-native write
|
||
side. This is near-pure fallback today — permanent scaffolding that lights up as those land.
|
||
- **Phase 3 P3.4-prep (VM plan) — harden the VM against malformed lowering-time IR (2026-06-18).**
|
||
Prerequisite for wiring the VM at the LOWERING-time comptime site (`runComptimeTypeFunc`),
|
||
where IR can be malformed (an unresolved name lowers to a dangling / `Ref.none` operand —
|
||
the 0737 crash). Closed the remaining panic vectors so the VM BAILS (→ legacy fallback)
|
||
instead of aborting: (1) a checked `Vm.refTy(ref_types, r)` replaces every raw
|
||
`ref_types[ref.index()]` in `exec` (the type-side companion to `Frame.get`'s `bad_ref`
|
||
value-side guard); (2) `aggType` is now a bailing method (`Error!TypeId`) using `refTy`;
|
||
(3) the block-dispatch loop bounds-checks the branch target before indexing
|
||
`func.blocks.items`. `global_get` was already guarded. No behavior change — gate OFF and
|
||
ON both **697/0**; unit test added (a `cmp_lt` with a `Ref.none` operand bails, not
|
||
panics). **Next:** wire `tryEval` into `runComptimeTypeFunc` behind the flag with legacy
|
||
fallback and measure (most minting type-fns will still bail at the welded-write call /
|
||
`Type`-result conversion until the VM models `Type` values + the VM-native write side land
|
||
— those are the steps that actually move lowering-time comptime onto the VM, toward
|
||
deleting legacy).
|
||
- **Phase 3 P3.3 (VM plan) — WRITE side: declare_type + pointer_to + ONE kind-branching register_type (2026-06-18).**
|
||
The mutating compiler-API: `declare_type(name) -> Type` (forward handle), `pointer_to(t) -> Type`
|
||
(build `*T`), and `register_type(handle, kind, members: []Member) -> Type` which branches on
|
||
`kind` IN THE COMPILER (subsuming define's per-kind dispatch). Take/return real `Type` values
|
||
(matching meta.sx declare/define). **Timing (per user): mint LAZILY at lowering time, single
|
||
pass** (the existing `runComptimeTypeFunc`), so the write side is **legacy-only** (`compiler_lib`
|
||
handlers) — the VM isn't wired at lowering time, no VM mirror needed; readers stay dual-path.
|
||
A non-generic `-> Type` builder is now flagged `is_comptime` (decl.zig) so its dead body permits
|
||
the welded calls. **Graph:** forward handles + `pointer_to` express mutually-recursive A↔B (`*A`,
|
||
`*B`, B-by-value); `register_type` is **idempotent** (re-fill a nominal slot reached via two
|
||
import edges — `nominalIdent`). `kind` codes match `type_kind` (1 struct · 2 actual `.@"enum"` ·
|
||
3 tagged_union · 4 tuple). **Fixed two bugs (issue 0142):** (a) a fully payloadless minted enum
|
||
was an all-void tagged_union → verifySizes panic; now a real `.@"enum"` (register_type kind 2 AND
|
||
metatype `defineEnum`); (b) bare `EnumType.variant` payloadless qualified construction wasn't
|
||
supported (failed for hand-written enums too) — added in `lowerFieldAccess` (`isPayloadlessVariant`).
|
||
Examples 0631 (graph + actual enum + reflection), 0632 (make_enum all-void), 0633/0634/0635
|
||
(namespaced / bare / multi-edge import of a minted type), 0187 (qualified variant construction).
|
||
**Parity 697/697** (gate ON and OFF); unit tests added. **Next (P3.4):** re-express
|
||
declare/define/type_info as sx over the compiler-API + delete the bespoke interp arms (needs the
|
||
VM hardened for lowering-time IR, or the metatype migrated onto the legacy compiler-API calls).
|
||
- **Phase 3 P3.2b (VM plan) — kind + enum-value readers: `type_kind` + `type_field_value`; READ side complete (2026-06-18).**
|
||
The last two read-only readers the metatype's `type_info(T)` needs (added to
|
||
`compiler_lib.bound_fns` AND `Vm.callCompilerFn`, each backed by a `TypeTable` query both
|
||
call): `type_kind(t) -> i64` (`kindCode` — a stable, compiler-owned discriminant: 0 other ·
|
||
1 struct · 2 enum · 3 tagged_union · 4 tuple · 5 union · 6 array · 7 vector · 8 error_set;
|
||
TOTAL, never bails) and `type_field_value(t, idx) -> i64` (`memberValue` — an enum variant's
|
||
explicit value or ordinal; mirrors the `field_value_int` builtin; loud-bail for non-enum /
|
||
out-of-range). Example `0630-comptime-compiler-type-kind` reflects `Color` / `WindowFlags`
|
||
(flags) / `Point`. **The READ side is now COMPLETE** — `find_type` + `type_kind` +
|
||
`type_field_count` + `type_field_name`/`type_field_type`/`type_nominal_name` +
|
||
`type_field_value` cover everything `reflectTypeInfo` reads. VM unit test added. **Parity
|
||
691/691** (gate ON and OFF). **Revised forward direction (per the user):** the WRITE side is
|
||
ONE `register_type(info)` fn that branches on the kind IN THE COMPILER (subsuming `define`'s
|
||
per-kind dispatch), not a per-kind `register_struct`.
|
||
- **Phase 3 P3.2 (VM plan) — field-level reflection readers: `type_nominal_name` + `type_field_name` + `type_field_type` (2026-06-18).**
|
||
Three more `compiler`-library readers on the same `TypeId`-handle shape (added to
|
||
`compiler_lib.bound_fns` AND `Vm.callCompilerFn`), each backed by a new `TypeTable` query
|
||
BOTH paths call (no drift): `nominalName` (a named type's own name handle; loud-bail for
|
||
unnamed types like `i64`/pointers), `memberName` (struct/union/tagged-union field, enum
|
||
variant, named-tuple element), `memberType` (struct/tuple/array/vector member type). All
|
||
loud-bail on out-of-range idx / no-member (no silent default). First MULTI-ARG compiler
|
||
fns — `callCompilerFn` reads arg 1 = idx; added `Vm.argHandle`/`argTypeId` (range-checked
|
||
u32/TypeId arg reads) and refactored `find_type`/`type_field_count` onto them. Named
|
||
`type_*` to avoid clashing with the std metatype builtins (`field_name`/`type_name` exist
|
||
in core.sx); `nominalName` (the TypeTable method) is distinct from the existing
|
||
`typeName(id) []const u8` display-string renderer. Example `0629-comptime-compiler-field-reflect`
|
||
reflects `Pair { lo: Point; hi: Point }` — each field name + the nominal name of a field's
|
||
type, all `#run`-folded, all VM-HANDLED natively. VM unit test added (type_field_name → "hi";
|
||
type_nominal_name(type_field_type(Pair,0)) → "Point"). **Parity 690/690** (gate ON and OFF).
|
||
- **Phase 3 P3.1 (VM plan) — first read-only reflection readers: `find_type` + `type_field_count` (2026-06-18).**
|
||
Two more `compiler`-library fns, bound the same way as the `intern`/`text_of` seed (added
|
||
to `compiler_lib.bound_fns` for the legacy handler + the welded-decl export check, AND to
|
||
`Vm.callCompilerFn` for the native flat-memory path — NO marshaling). A **type handle is a
|
||
plain `u32` `TypeId`** (like `StringId`), so both keep the seed's clean scalar shape:
|
||
`find_type(name: StringId) -> TypeId` (`TypeTable.findByName`, `unresolved`/0 if absent) and
|
||
`type_field_count(t: TypeId) -> i64` (a NEW `TypeTable.memberCount` query — struct/union/
|
||
tagged-union fields, enum variants, array/vector length — called by BOTH paths so they
|
||
can't drift; bails loudly, never a silent 0). New example `0628-comptime-compiler-find-type`
|
||
chains `intern → find_type → type_field_count` (and a not-found lookup → 0), both folded at
|
||
`#run`, both VM-HANDLED natively (trace confirms no fallback). VM unit test added
|
||
(`find_type` + `type_field_count`, struct found → 3 fields, missing → `unresolved`).
|
||
**Parity 689/689** (gate ON and OFF). **Decision (resolves the plan's `find_type → ?Type`
|
||
sketch):** return a NON-optional `TypeId` with the `unresolved` (0) sentinel for not-found,
|
||
NOT `?Type` — a `Type` value resolves to `.any` (which the flat-memory VM doesn't represent)
|
||
and an optional can't cross the legacy↔VM eval boundary; `unresolved` is the project-blessed
|
||
unmistakable "no type" marker. Forward (P3.2): more readers on the same handle shape
|
||
(`type_name`/`field_name`/`field_type`/kind), then `register_struct` (first mutating fn).
|
||
- **VM robustness — `Frame` bounds-check; lowering-time `#insert` wiring explored + reverted (2026-06-18).**
|
||
Explored wiring the VM at the LOWERING-time comptime site (`evalComptimeString`, the
|
||
`#insert` string fold). 12/13 `#insert` examples ran on the VM with parity, but `0737`
|
||
(an `#insert` of an unresolved `secret()`) CRASHED the VM (SIGABRT): lowering-time IR can
|
||
be malformed (a `ret Ref.none` from the unresolved name) and `Frame.get` panicked on the
|
||
out-of-range index. **Decision: reverted the lowering-time wiring** — unlike the emit-time
|
||
folds (fully lowered IR), lowering-time IR can be erroneous, and hardening the VM against
|
||
ALL malformed IR (every `ref_types[...]` / `aggType` access, not just `Frame`) is out of
|
||
scope here. The emit-time sites already give full corpus coverage. **KEPT** the defensive
|
||
fix regardless (CLAUDE.md "never crash"): `Frame.get`/`set` now bounds-check and flip a
|
||
`bad_ref` flag; the `run` loop bails (`badRef`) instead of panicking. Unit test added
|
||
(malformed `ret Ref.none` → bail, not crash). Parity **688/688** both ways.
|
||
- **Phase 3 SEED (VM plan) — compiler-call path: `intern`/`text_of` native on the VM (2026-06-18).**
|
||
`invoke` now dispatches a welded `compiler`-library fn (gated on `compiler_welded`) to
|
||
`Vm.callCompilerFn`, serviced NATIVELY on flat memory (no legacy `Interpreter`):
|
||
`intern(string)->StringId` reads the flat-memory string bytes and `internString`s into the
|
||
const-cast table (pool-only — doesn't touch type layout, so cached sizes stay valid);
|
||
`text_of(StringId)->string` materializes the pooled text back into flat memory. Unlocked
|
||
`0626`; the ONLY remaining const-init fallback is now the inline-asm global (`1654`).
|
||
Parity **688/688** (gate ON and OFF); unit test added. This is the mechanism Phase 3 grows
|
||
— the next compiler fns (`find_type`, `register_struct`, reflection readers) bind the same
|
||
way (flat-memory pointer in, handle/pointer out, no marshaling).
|
||
- **Phase 1.final step 9 (VM plan) — `-Dcomptime-flat` build flag (the "swap behind a build flag" step) (2026-06-18).**
|
||
Added the `-Dcomptime-flat` build option (build.zig → a `build_opts` options module on
|
||
`mod`; `emit_llvm.init` reads `build_opts.comptime_flat or SX_COMPTIME_FLAT env`). This is
|
||
the plan's "reach parity → swap behind a build flag → delete the old path" mechanism.
|
||
`zig build test -Dcomptime-flat` runs the FULL corpus on the VM (688/0). Verified the flag
|
||
toggles the binary: flag-built `sx` reports VM HANDLED with no env var; default-built does
|
||
not. Default OFF — `zig build test` unchanged (688/0). Env var still works for ad-hoc runs.
|
||
Next (forward): Phase 2 (bytecode) / Phase 3 (compiler-API on flat memory); eventual
|
||
default-flip + legacy deletion.
|
||
- **Phase 1.final step 8 (VM plan) — wire the `#run` side-effect path + trace-clear-on-fallback (2026-06-18).**
|
||
Wired the SECOND comptime call site (`runComptimeSideEffects`, top-level `#run <expr>;`)
|
||
through `tryEval` with legacy fallback, mirroring the const-init fold. `tryEval` now
|
||
handles void/noreturn entries (→ `.void_val`) so a void side-effect doesn't bail at the
|
||
result conversion. **Fixed a trace-corruption** the new site exposed (`1035`): a
|
||
side-effect that pushes return-trace frames and then bails (e.g. on `print`) had the
|
||
legacy re-run DOUBLE-push them (`sx_trace_push` is a side effect on the shared buffer).
|
||
Both wiring sites now `sx_trace_clear()` right before the legacy fallback, discarding the
|
||
VM's partial pushes. **Parity 688/688** (gate ON and OFF). Most side-effects still bail
|
||
(print/global_addr/call_builtin) → legacy, but the path is now uniform. All comptime
|
||
evaluation routes through the VM-with-fallback.
|
||
- **Phase 1.final step 7 (VM plan) — is_comptime + failable/error cluster + signed-load fix; coverage 31→36 (2026-06-18).**
|
||
`is_comptime` → 1 (unlocked `1030`). Ported the failable/error-channel cluster (`1037`
|
||
escape, `1038` handled): `kindOf(error_set)→word`, `regToValue` bridges TUPLES (the
|
||
failable `(value…,tag)` shape `checkComptimeFailable` reads), `trace_frame` packs
|
||
`(func_id<<32|span.start)` from a new `call_stack` (pushed by invoke/runEntry), and
|
||
`sx_trace_push`/`sx_trace_clear` serviced NATIVELY (the VM calls the real sx_trace.c
|
||
functions linked into the compiler, so the return-trace buffer is populated identically
|
||
to legacy). raise/catch/or now run on the VM. **Surfaced + fixed a real GENERAL bug:**
|
||
`readField` was ZERO-extending signed sub-64-bit loads, so a stored `i32 -1` reloaded as
|
||
`0xFFFFFFFF` (+4.29e9) and `< 0` was false — silently hiding `raise error.Bad`; now
|
||
SIGN-extends `i8`/`i16`/`i32`/`isize` (gate-ON parity confirms it's a strict fix; unit
|
||
test added). VM HANDLES **36** corpus const-inits (was 31); **parity 688/688** (gate ON
|
||
and OFF). Only **2 fallbacks** remain, both principled: `intern` (`0626`, welded
|
||
compiler-API fn — Phase 3) + inline-asm global (`1654`). Forward work: Phase 2 (bytecode),
|
||
Phase 3 (compiler-API on flat memory).
|
||
- **Phase 1.final step 6 (VM plan) — real default context + call_indirect + func_ref + global_get; coverage 27→31 (2026-06-17).**
|
||
Per the user's direction ("the VM can set up a default context"), `runEntry` now
|
||
materializes the REAL default context instead of a zeroed one. The implicit-ctx param is
|
||
an opaque `*void`, so `materializeDefaultContext` finds the `__sx_default_context` global
|
||
and lays its initializer (`{ {null, alloc_fn, dealloc_fn}, null }`, the CAllocator thunk
|
||
func-refs) into flat memory via a new recursive `layoutConst`. With `func_ref` (function
|
||
value encoded `FuncId.index()+1`, reserving word 0 for the null fn-ptr) and
|
||
`call_indirect` (decode word → FuncId → dispatch; 0 → bail) ported, the whole allocator
|
||
protocol runs on the VM:
|
||
`context.allocator.alloc_bytes` → call_indirect → thunk → `CAllocator.alloc_bytes` →
|
||
`libc_malloc` → native flat malloc. Unlocked `0606` (string global). Also: `global_get`
|
||
lazily evaluates a comptime global's `comptime_func` (memoized) — unlocked `CT_CHAIN`;
|
||
field access (`fieldOffset`/`struct_get`) handles string/slice `{ptr@0,len@8}` fat
|
||
pointers (needed by `alloc_string`); `regToValue` maps function-typed words → `.func_ref`
|
||
(kept `1128`'s rejection byte-identical). Native `malloc` is still required (the thunk
|
||
bottoms out at it; a host pointer can't be used with flat-memory load/store). VM HANDLES
|
||
**31** corpus const-inits (was 27); **parity 688/688** (gate ON and OFF). Unit tests:
|
||
global_get, func_ref+call_indirect. Remaining fallbacks (7): `.unsupported` aggregates
|
||
(3× — `1037`/`1038`), extern/builtin `intern`+asm (2×), `trace_frame`, `is_comptime`.
|
||
- **Phase 1.final step 5 cont. (VM plan) — libc memory builtins + f32 fix; coverage 16→27 (2026-06-17).**
|
||
Identified the dominant fallback (`call to extern/builtin`) as **11× `malloc`** (0604) +
|
||
1× `intern`. Modeled a curated set of libc MEMORY builtins natively on flat memory
|
||
(`Vm.callMemBuiltin`): `malloc`/`calloc` → `allocBytes` (16-aligned, 256-MiB cap → bail),
|
||
`free` → no-op, `memcpy`/`memmove`/`memset` on flat bytes — sandboxed (no host heap/dlsym),
|
||
target-aware; the computed result is byte-identical to legacy (which calls real libc).
|
||
This surfaced a **real latent f32 bug**: float registers hold f64 bits, but f32 MEMORY is
|
||
the 4-byte single — `readField`/`writeField` were truncating the f64 bits (writing zeros
|
||
for `1.0`); now they `@floatCast` on f32 load/store (mirrors legacy `storeAtRawPtr`).
|
||
Result: VM HANDLES **27** corpus const-inits (was 16); **parity 688/688** (gate ON and
|
||
OFF). Unit tests added (f32 round-trip; malloc → usable flat memory). Next: the `kindOf`
|
||
`.unsupported` aggregates (3×), `global_get` (2×), the rest.
|
||
- **Phase 1.final step 5 (VM plan) — implicit-context materialization; coverage 0→16 (2026-06-17).**
|
||
`tryEval` now MATERIALIZES the implicit ctx instead of skipping it: a `has_implicit_ctx`
|
||
comptime entry (sole param `*Context`) gets a zeroed `Context` of the right size/align
|
||
in flat memory, its address passed as arg 0. Const bodies that ignore the ctx run; a
|
||
body that uses the allocator hits unported `call_indirect` → bails → legacy. No func-ref
|
||
materialization needed (handled bodies don't read ctx contents; parity is the guard).
|
||
Fixed a real bug surfaced by the coverage pass: storing a `null` non-pointer optional
|
||
(the `null_addr` sentinel) into an aggregate slot OOB-bailed — `writeField` now ZEROES
|
||
the destination for a `null_addr` aggregate source (= none/empty); unit-test regression
|
||
added. Result: VM HANDLES **16** corpus const-inits (was 0); **parity 688/688 both
|
||
gate ON and OFF**. Next: port the ops the trace names — `call_builtin`/`compiler_call`/
|
||
extern (13×, via the bridge), `kindOf` `.unsupported` aggregates (3×), `global_get` (2×),
|
||
func_ref / call_indirect / trace_frame / is_comptime.
|
||
- **Phase 1.final steps 1–4 (VM plan) — host wiring landed; coverage measured (2026-06-17).**
|
||
(1) **Hardening:** `Machine.readWord`/`writeWord`/`bytes` now return `error.OutOfBounds`
|
||
(null / out-of-range / oversized / overflow-safe) instead of `assert`-panicking;
|
||
`OutOfBounds` added to `Vm.Error`; `try` threaded through every helper + exec arm + the
|
||
bridge. New unit tests (accessor OOB returns; null-deref → `tryEval` null, not a crash).
|
||
(2) **Implicit context:** `tryEval` returns null for `has_implicit_ctx` funcs (legacy
|
||
fallback) — conservative; full ctx materialization deferred to step 5. (3) **Wiring:**
|
||
const-init fold in `emit_llvm.zig` `emitGlobals` is `(if comptime_flat) tryEval else
|
||
null) orelse interp.call(...)`, gated by env `SX_COMPTIME_FLAT` (read once into
|
||
`LLVMEmitter.comptime_flat`). Default OFF. (4) **Parity + coverage:** gate ON → full
|
||
corpus byte-identical (688, 0 failed) + manual 0605/0606/0607 byte-identical.
|
||
**Finding: 0 of 37 measured corpus const-inits are VM-handled — ALL are
|
||
`has_implicit_ctx`-gated.** Added a coverage-trace facility (`comptime_vm.last_bail_reason`
|
||
+ env `SX_COMPTIME_FLAT_TRACE`). **Next: step 5 = implicit-context materialization** (the
|
||
unblocker), then port the deferred ops. 688 corpus green (gate OFF).
|
||
- **Phase 1.final start (VM plan) — wiring entry point `tryEval` (2026-06-17).**
|
||
`comptime_vm.tryEval(gpa, module, func_id) ?Value` runs a comptime function entirely on
|
||
the VM, returns a legacy `Value` (deep-copied to `gpa`) or `null` to fall back.
|
||
Unit-tested (pure 6*7 → 42; unbox_any → null). NOT yet routed into the host: needs
|
||
(1) panic→error hardening of `Machine` accessors so arbitrary funcs bail instead of
|
||
crashing, (2) implicit-ctx handling, (3) wiring at `emit_llvm` const-init behind
|
||
`SX_COMPTIME_FLAT`, (4) corpus parity run. See `PLAN-COMPILER-VM.md` Phase 1.final.
|
||
688 corpus green.
|
||
- **Phase 1 sub-step 1.5b (VM plan) — Reg↔Value boundary bridge (2026-06-17).**
|
||
Builtin/compiler_call/extern handlers are coupled to the legacy `Interpreter`, so the
|
||
wiring will use WHOLE-FUNCTION fallback (VM runs pure functions; bail → legacy re-runs
|
||
the whole eval). Built the boundary bridge that enables it: `valueToReg` (Value arg →
|
||
Reg, aggregates into flat memory) + `regToValue` (VM result → Value, deep-copied).
|
||
Covers scalars/strings/structs; other shapes bail. Transitional. Round-trip
|
||
unit-tested. 688 corpus green. Next: the wiring (flag + route a comptime entry through
|
||
the VM with legacy fallback).
|
||
- **Phase 1 sub-step 1.5 (VM plan) — direct `call` + stack-lifetime change (2026-06-17).**
|
||
`Vm` gained `module` (callee resolution) + `depth`/`max_depth` guard. `call` marshals
|
||
arg Refs → Reg and recursively runs the callee; aggregates pass as Addrs over shared
|
||
flat memory. `Frame` no longer reclaims the machine on exit (else a returned aggregate
|
||
Addr dangles) — allocations live to `Vm.deinit`. Extern/builtin callees bail (1.5b).
|
||
Unit-tested: direct call (142), recursion sum(0..n) (15/55). 688 corpus green. Next:
|
||
1.5b (call_builtin/compiler_call/extern), then hybrid wiring.
|
||
- **Phase 1 sub-step 4d (VM plan) — deref/addr_of; pivot decision (2026-06-17).**
|
||
Ported `addr_of` (pass-through) + `deref` (readField through pointer), unit-tested
|
||
(deref *i64 → 77, addr_of struct + field → 80). DECIDED to stop porting rarer ops
|
||
(tagged-union payload/any/closures) blind — their byte semantics are ambiguous without
|
||
real call sites — and pivot to CALLS (sub-step 1.5: `call`, then builtin/compiler) +
|
||
HYBRID WIRING (`-Dcomptime-flat` → VM with legacy fallback on `error.Unsupported`), so
|
||
the VM runs the real corpus and surfaces exactly what's needed. Key design point for
|
||
calls: aggregate-return lifetime → drop per-frame stack reclaim (let a comptime eval's
|
||
allocations live to `Vm.deinit`). 688 corpus green. See `PLAN-COMPILER-VM.md` decision
|
||
block.
|
||
- **Phase 1 sub-step 4c (VM plan) — optionals + payloadless enums (2026-06-17).**
|
||
`kindOf`: enum → word; `?T` → word (pointer-child, null==0) or `{T@0,i1@sizeof(T)}`
|
||
aggregate. Ported optional_wrap/unwrap/has_value/coalesce (`optChildIsPtr`/`optHas`;
|
||
const_null reads as none) + payloadless enum_init/enum_tag. Unit-tested (?i64 → 91,
|
||
?*i64 null==0 → 99, enum tag → 11). 688 corpus green. Next: 4d (tagged unions, any,
|
||
closures).
|
||
- **Phase 1 sub-step 4b (VM plan) — slices + strings on flat memory (2026-06-17).**
|
||
`{ptr@0(pointer_size), len@8(i64)}` fat pointers (kindOf: string/slice → aggregate).
|
||
Ported `const_string` (text+NUL + fat pointer in flat memory), `length`/`data_ptr`,
|
||
`array_to_slice`, `subslice`, index-through-slice (`elemAddr` loads `.ptr`), and
|
||
`str_eq`/`str_ne` (memcmp). Unit-tested (str length+eq/ne, array→slice index sum=23,
|
||
subslice sum=43). 688 corpus green. Next: 4c (optionals/enums/any/closures).
|
||
- **Phase 1 sub-step 4a (VM plan) — tuples + arrays on flat memory (2026-06-17).**
|
||
`kindOf` widened (tuple/array → aggregate). Ported `tuple_init`/`tuple_get`
|
||
(`tupleFieldOffset`), `index_get`/`index_gep` (`elemAddr` = base + idx*elem_size over
|
||
array/pointer/many_pointer; slice/string bases bail), `length` on array values.
|
||
Unit-tested (mixed tuple, [3]i64 index sum=42, length=3). 688 corpus green. Next:
|
||
sub-step 4b (slices/strings, then optionals/enums/any/closures).
|
||
- **Phase 1 sub-step 3 (VM plan) — memory + structs on flat memory (2026-06-17).**
|
||
`Vm` gained optional `table: *const TypeTable` (target-aware layout). Ported
|
||
`alloca`/`load`/`store` + `struct_init`/`struct_get`/`struct_gep`, laying structs out
|
||
at the table's natural offsets. Value model: scalar/pointer → register word;
|
||
struct → lives in flat memory, its value IS its address (read→addr, write→memcpy), so
|
||
nested structs compose and `struct_gep` = base+offset. `kindOf` bails loudly on
|
||
not-yet-ported types. Addr-based values survive allocator realloc. Unit-tested
|
||
(struct round-trip, alloca+gep+store+load, nested struct). 688 corpus green. Next:
|
||
sub-step 4 (arrays/slices/strings/optionals/enums/tuples/any/closures, then calls).
|
||
- **Phase 1 sub-step 2 (VM plan) — flat-memory executor: scalars + control flow
|
||
(2026-06-17).** Added `Vm` to `comptime_vm.zig`: walks the same IR `Inst` over
|
||
flat-memory frames (register `Reg` = scalar bits or `Addr`), mirroring the legacy
|
||
interp's scalar semantics (i64 wrapping/signed, f64). Ported constants, arithmetic,
|
||
comparison, logical, conversions, terminators (`br`/`cond_br`/`ret`/`ret_void`) and
|
||
`block_param`; every other op bails loudly (`error.Unsupported` + op name in
|
||
`detail`). Unit-tested on hand-built tiny IR (`Fb` builder): int add, f64 arithmetic,
|
||
cond_br selection, a block-param loop, div-by-zero + unsupported-op bails. Corpus
|
||
untouched (688 green). Next: sub-step 3 (memory + aggregates on flat memory, where
|
||
target-aware layout enters).
|
||
- **Phase 1 sub-step 1 (VM plan) — flat-memory machine substrate (2026-06-17).**
|
||
New `src/ir/comptime_vm.zig`: `Machine` (linear byte memory + bump/stack allocator
|
||
with `mark`/`reset`, scalar `readWord`/`writeWord` 1/2/4/8 LE, `bytes` views, addr 0
|
||
reserved as `null_addr`) + `Frame` (Ref-indexed register file, stack reclamation on
|
||
deinit). `Reg` = raw u64 (immediate scalar OR `Addr`). Unit-tested
|
||
(`comptime_vm.test.zig`), registered in the barrel; standalone — the legacy
|
||
interpreter stays live, corpus untouched (688 green). Next: sub-step 2 (executor +
|
||
scalar/branch ops over the same IR). Also removed the "~500 lines / split step" rule
|
||
from CLAUDE.md per request.
|
||
- **Phase 0 (VM plan) — struct-weld stripped; `intern`/`text_of` bridge kept
|
||
(2026-06-17).** Removed the struct-weld registry from `compiler_lib.zig`
|
||
(`weldStruct`/`bound_types`/`BoundType`/`FieldLayout`/`findType`/`SxField`/
|
||
`LayoutMismatch`/`validateStructLayout`), `validateWeldedStruct`/`weldedFieldOrderStr`
|
||
+ the `sd.abi == .zig` call from `nominal.zig`, the struct-weld unit tests, and
|
||
examples `0625`/`0627`/`1183`/`1186`. KEPT (decision) the `intern`/`text_of` function
|
||
host-call bridge — a clean scalar dispatch, not weld/serialize/marshal, the Phase-3
|
||
compiler-call seed — so `weldedCompilerFn`, the `compiler_welded` dispatch, the
|
||
`emitCall` comptime-only gate, the `#library`/`abi`/`extern` syntax, and examples
|
||
`0626`/`1184`/`1185` remain. `zig build test` green (688 corpus, 0 failed). Next:
|
||
Phase 1 (flat-memory value model) per `PLAN-COMPILER-VM.md`.
|
||
- **DIRECTION CHANGE — pivot off the byte-weld to a flat-memory bytecode VM
|
||
(2026-06-17).** Decided the weld + serialization/marshaling bridge is the wrong
|
||
direction (it hand-marshals onto a comptime value model that isn't bytes — exactly
|
||
what the design set out to kill). New foundation: a bytecode VM over flat memory so
|
||
comptime values are native bytes; the compiler-API then rides on it via direct memory
|
||
(no weld/validation/marshaling). JIT-native comptime was weighed and rejected (breaks
|
||
cross-compilation, loses the sandbox). Wrote `current/PLAN-COMPILER-VM.md` (Phase 0
|
||
strip → Phase 1 flat-memory value model → Phase 2 bytecode → Phase 3 compiler-API on
|
||
flat memory). Banner added to `design/comptime-compiler-api.md` (superseded). Reverted
|
||
the session's uncommitted `register_struct`/`find_type` marshaling experiment back to
|
||
`reify` HEAD (40d075c). No code stripped yet — Phase 0 is the next action.
|
||
- **Phase 2 — welded structs by reflection + memory-order validation.** Dropped
|
||
the byte-layout-override engine (computeWeldPlan / offset-ordered LLVM struct /
|
||
byte-blob — all explored, all unnecessary). Instead: the sx header declares
|
||
fields in the compiler type's memory order; the compiler reflects the bound Zig
|
||
type (`@typeInfo`/`@offsetOf`/`@sizeOf`) and validates the header matches with
|
||
loud diagnostics (field-not-found, wrong-order+expected-order, size mismatch).
|
||
On pass it's an ordinary byte-identical struct — cast + deref just works.
|
||
Examples 0627 (usable) / 1186 (wrong-order diagnostic). Suite green (692).
|
||
- **Phase 2.1 — weld-plan layout math (REMOVED).** The byte-layout-override math;
|
||
superseded by the reflection+validation design and deleted.
|
||
- **Phase 1 polish — comptime-only enforcement.** A runtime call to a welded fn is
|
||
a clean build-gating error (`emitCall` gate, guarded by enclosing-`is_comptime`
|
||
so `#run`/`::` uses stay green), not a link failure. Example 1185. Build + suite
|
||
green (458 unit, 690 corpus).
|
||
- **Phase 1.1 fifth sub-step — host-call bridge (welded functions).**
|
||
`compiler_lib` function registry (`intern`/`text_of`) + `findFn`; IR `Function`
|
||
`compiler_welded` flag set/validated in `declareFunction` (`weldedCompilerFn`);
|
||
`interp.call()` dispatches welded calls to the Zig handler. Examples 0626 (round-
|
||
trip) + 1184 (unexported-fn diagnostic); `findFn` unit-tested. Runtime-call clean
|
||
rejection deferred (loud link error today). Build + suite green (458 unit, 689
|
||
corpus).
|
||
- **Phase 1.1 fourth sub-step — welded-struct layout validation.**
|
||
`validateStructLayout` (pure, unit-tested) + `validateWeldedStruct` wired into
|
||
`registerStructDecl`: a `struct abi(.zig) extern compiler` is validated against
|
||
the registry (lib == compiler, name exported, layout matches) with build-gating
|
||
diagnostics. `#library "compiler"` no longer dlopen'd. Examples 0625 (faithful
|
||
Field) + 1183 (field-count mismatch diagnostic). Offset-override/GEP deferred to
|
||
Phase 2 (not exercised by Field's natural layout). Build + suite green (456 unit,
|
||
687 corpus).
|
||
- **Phase 1.1 third sub-step — binding registry.** New `src/ir/compiler_lib.zig`:
|
||
the `compiler` lib's welded-type registry; `Field` welded to
|
||
`StructInfo.Field` with layout baked from the real Zig type
|
||
(`@offsetOf`/`@sizeOf`/`@alignOf`); `findType` lookup proven by unit test
|
||
(+ null off the export list). Standalone island — not yet consumed by lowering.
|
||
Build + suite green (454 unit tests). Break-verified.
|
||
- **Phase 1.1 second sub-step — struct-decl binding parses.** `ast.StructDecl`
|
||
gained `abi` + `extern_lib`; `parseStructDecl` parses `abi(.zig) extern <lib>`
|
||
after `struct`. Parser unit tests (welded `Field` + plain struct), break-verified.
|
||
Build + suite green. Parse-only sub-step (fns + structs) of Phase 1.1 complete.
|
||
- **Phase 1.1 first sub-step + `callconv`→`abi` unification.** Parsed `abi(.zig)
|
||
extern <lib>` on fn decls; unified `callconv` into `abi(.c|.zig|.pure)` (removed
|
||
the `callconv` keyword), migrated 52 sx files + compiler diagnostics + docs +
|
||
snapshots. Build + suite green. The original design's `extern(.zig)` single
|
||
qualifier was split into `abi(.zig)` (ABI/layout, before extern) + `extern
|
||
<lib>` (linkage + source) — recorded in the design doc's syntax-decision note.
|