comptime VM: host wiring, full corpus parity, build flag, Phase 3 seed

Phase 1.final of the flat-memory comptime VM — wire the host through it,
reach corpus parity, and gate it behind a build flag — plus the first
Phase 3 (compiler-API) step. Default OFF; legacy interpreter unchanged.

Host wiring + hardening:
- Machine accessors return error.OutOfBounds (no debug panic) on bad
  addresses; Frame.get/set bounds-check and bail (no panic) on a malformed
  operand ref (e.g. a ret Ref.none from an unresolved name).
- tryEval routed at both comptime call sites in emit_llvm — the const-init
  fold and the #run side-effect path — with per-eval legacy fallback;
  yields .void_val for void/noreturn entries. Both sites sx_trace_clear()
  before the legacy fallback so a partial VM run that pushed trace frames
  doesn't double-push on re-run.

VM coverage (all corpus const-inits except the inline-asm global):
- Implicit context materialized from the __sx_default_context global; the
  full allocator protocol runs on the VM (context.allocator.alloc ->
  call_indirect -> CAllocator thunk -> libc_malloc -> native flat malloc).
- Native libc memory builtins (malloc/calloc/free/memcpy/memmove/memset)
  on flat memory; f32 stored/loaded as the 4-byte single; signed sub-64-bit
  loads sign-extended; global_get (lazy + memoized); func_ref/call_indirect
  (func-ref encoded fid+1, 0 reserved for null); string/slice fat-pointer
  field access; is_comptime; the failable/error cluster (error_set tuples,
  trace_frame + native sx_trace_push/clear -> raise/catch/or + return traces).

Build flag + Phase 3 seed:
- -Dcomptime-flat (build_opts module) OR SX_COMPTIME_FLAT env enables the VM;
  zig build test -Dcomptime-flat runs the full corpus on the VM (688/0).
- intern/text_of serviced natively on flat memory via Vm.callCompilerFn
  (compiler_welded boundary) — the seed the rest of the compiler-API grows on.

Parity 688/688 gate ON and OFF. Unit tests added throughout. The
lowering-time #insert wiring was explored and reverted (lowering-time IR can
be malformed; full malformed-IR hardening is a prerequisite, deferred).
This commit is contained in:
agra
2026-06-18 08:27:58 +03:00
parent b8f3d6fd78
commit 0367d96d9b
7 changed files with 1142 additions and 108 deletions

View File

@@ -142,6 +142,117 @@ host through it:
5. Grow coverage (port the deferred ops + `call_builtin`/`compiler_call` via the bridge)
until the VM is the default and the legacy path is deleted.
**Status (2026-06-17): steps 14 DONE; step 5 = the next session.**
- **(1) Hardening — DONE.** `Machine.readWord`/`writeWord`/`bytes` return
`error.OutOfBounds` (null / out-of-range / oversized / overflow-safe) instead of
asserting. `OutOfBounds` added to `Vm.Error`; `try` threaded through
`readField`/`writeField`/`optHas`/`makeSlice`/`sliceLen`/`sliceData`/`elemAddr` and
every exec arm + the bridge. New unit tests: hardened-accessor OOB returns, and a
null-deref function → `tryEval` returns `null` (legacy fallback), not a panic.
- **(2) Implicit context — DONE (materialized, 2026-06-17 step 5).** Initially a
conservative skip; now `tryEval` MATERIALIZES the implicit ctx: a comptime entry with
`has_implicit_ctx` (whose sole param is the `*Context`) gets a zeroed `Context` of the
right size/align allocated in flat memory, its address passed as arg 0. The common
const body never reads the ctx; a body that USES the allocator loads a fn from it and
`call_indirect`s (unported) → bails → legacy. No func-ref materialization was needed:
handled bodies don't read the ctx contents, and gate-ON corpus parity (688, 0 failed)
empirically confirms no divergence. (A body that read+branched on a null allocator fn
could in principle diverge; none does — parity is the guard.)
- **(3) Wire one site — DONE.** Const-init fold in `emitGlobals` is `(if comptime_flat)
tryEval(...) else null) orelse interp.call(...)`. Gated by env `SX_COMPTIME_FLAT`
(a `LLVMEmitter.comptime_flat` field read once from `std.c.getenv` in `init`).
Default OFF → corpus unaffected (688 green).
- **(4) Parity + coverage — DONE.** Gate ON: full corpus byte-identical (688, 0 failed);
manual `sx run` of 0605/0606/0607/0608 byte-identical to gate-OFF. Coverage-trace
facility in place (`comptime_vm.last_bail_reason` + env `SX_COMPTIME_FLAT_TRACE`,
printing HANDLED / fallback+reason per init).
- **(5) Implicit-context materialization + memory builtins + f32 — DONE; op-porting CONTINUES.**
Coverage climbed **0 → 16 → 27** handled corpus const-inits (fallbacks 22 → 11); parity
stays **688/688** (gate ON and OFF) at every step. Landed, in order: implicit ctx
materialized (→16); `writeField` null-aggregate fix (storing a `null` non-pointer
optional `null_addr` sentinel into an aggregate slot OOB-bailed → now ZEROES the
destination = none/empty; unit-test regression); curated libc MEMORY builtins on flat
memory (`Vm.callMemBuiltin`: `malloc`/`calloc` → `allocBytes` 16-aligned & 256-MiB-capped,
`free` → no-op, `memcpy`/`memmove`/`memset` on flat bytes — sandboxed, target-aware,
result byte-identical to legacy; unlocked `0604`'s 11 comptime mallocs); and an **f32
storage fix** (float registers hold f64 bits, but f32 memory is the 4-byte single —
`readField`/`writeField` now `@floatCast` instead of truncating the f64 bits, which had
written zeros for `1.0`; a real latent bug `0604` surfaced; unit tests added).
- **(6) Real default context + call_indirect + func_ref + global_get — DONE.** Coverage
**27 → 31** handled (fallbacks 11 → 7); parity stays **688/688** both gate ON and OFF.
Per the user's direction ("the VM can set up a default context"), `runEntry` now
materializes the REAL default context (not a zeroed one): the implicit-ctx param is an
opaque `*void`, so `materializeDefaultContext` finds the `__sx_default_context` global
and lays its initializer constant (`{ {null, alloc_fn, dealloc_fn}, null }`, carrying
the CAllocator thunk func-refs) into flat memory via a new recursive `layoutConst`.
With `func_ref` (a function value encoded as `FuncId.index() + 1` so word 0 stays
reserved for the NULL function pointer — `funcRefWord`/`funcRefToId`) and `call_indirect`
(decode the callee word → `FuncId` → dispatch; 0 → bail) ported, a comptime body
that allocates via `context.allocator` now runs ENTIRELY on the VM: `alloc_string` →
`context.allocator.alloc_bytes` → `call_indirect` → thunk → `CAllocator.alloc_bytes` →
`libc_malloc` → the VM's native flat-memory `malloc`. Unlocked `0606` (string global via
the allocator). Also: `global_get` lazily evaluates a comptime global's `comptime_func`
(memoized in `global_cache`) — unlocked `CT_CHAIN`; struct field access (`fieldOffset`/
`struct_get`) now handles string/slice `{ptr@0,len@8}` fat pointers (needed by
`alloc_string`'s `s.ptr`/`s.len`); and `regToValue` maps a function-typed word back to
`.func_ref` so a func-ref result serializes identically to legacy (kept `1128`'s
rejection diagnostic byte-identical). Unit tests added (global_get, func_ref +
call_indirect). **Note: native `malloc` is still REQUIRED** — the CAllocator thunk
bottoms out at libc `malloc`, and the VM can't use a host pointer with flat-memory
load/store, so comptime `malloc` must allocate from flat memory. The default context
lets the allocator PROTOCOL run; native `malloc` is its final step.
- **(7) `is_comptime` + failable/error cluster + the signed-load fix — DONE.** Coverage
**31 → 36** handled (fallbacks 7 → 2); parity stays **688/688** both gate ON and OFF.
- **`is_comptime`** → always 1 on the VM (folds to false in compiled code). Unlocked `1030`.
- **Failable / error-channel cluster** (`1037` escape, `1038` handled): `kindOf(error_set)
→ word` (a u32 tag id); `regToValue` now bridges TUPLES (the failable `(value…, tag)`
shape the host's `checkComptimeFailable` reads); `trace_frame` packs `(func_id<<32 |
span.start)` from a new `call_stack` (pushed by `invoke`/`runEntry`); and `sx_trace_push`
/ `sx_trace_clear` are serviced NATIVELY (the VM calls the real sx_trace.c functions —
linked into the compiler — so the return-trace buffer the host reads is populated
identically to the legacy dlsym path). `raise`/`catch`/`or` all run on the VM now.
- **Signed sub-64-bit load fix (a real GENERAL bug the failable case surfaced):**
`readField` now SIGN-extends `i8`/`i16`/`i32`/`isize` loads (was zero-extending, so a
stored `i32 -1` reloaded as `0xFFFFFFFF` = +4.29e9 and `< 0` was false — which silently
hid `raise error.Bad`). Affects any negative signed sub-64-bit value stored & reloaded;
gate-ON corpus parity confirms it's a strict fix. Unit test added (+ failable tests
pass via 1037/1038 in the corpus).
- **Remaining fallbacks (2, both principled — the VM correctly stays on legacy):**
`intern` (`0626`, the welded compiler-API fn — Phase 3 re-homes it) and the inline-asm
global call (`1654`, never comptime-evaluable). Every other measured corpus const-init
is handled on the VM.
At this point the flat-memory VM handles essentially the entire real comptime corpus
(scalars, control flow, structs/tuples/arrays/slices/strings/optionals/enums, calls +
recursion, the implicit context + allocator protocol, globals, failables + return
traces). Phase 2 (bytecode) and Phase 3 (compiler-API on flat memory) are the forward
work; flipping the VM to default + deleting the legacy path awaits those.
- **(8) Wire the `#run` side-effect path; trace-clear-on-fallback — DONE.** The second
comptime call site (`emit_llvm.runComptimeSideEffects`, top-level `#run <expr>;`) now
routes through `tryEval` with legacy fallback, like the const-init fold; `tryEval` yields
`.void_val` for a void/noreturn entry. Fixed a trace-corruption the new site exposed
(`1035`): a side-effect that pushes trace frames then bails (on `print`) had the legacy
re-run double-push them — both sites now `sx_trace_clear()` right before the legacy
fallback to discard the VM's partial pushes. Parity **688/688** both gate ON and OFF. All
comptime evaluation now routes through the VM-with-fallback (uniform).
- **(9) `-Dcomptime-flat` build flag — DONE (the "swap behind a build flag" step).** The VM
gate is now a build option (`build.zig` → a `build_opts` module on `mod`; `emit_llvm.init`
reads `build_opts.comptime_flat or SX_COMPTIME_FLAT env`), default OFF. `zig build test
-Dcomptime-flat` runs the FULL corpus on the VM (688/0) — the build-integrated parity
gate. Verified the flag toggles the binary (flag-built `sx` uses the VM with no env var;
default-built does not). This is the prerequisite to eventually making the VM default +
deleting the legacy path (which still awaits Phase 2/3 + broader confidence).
- **(10) Compiler-call path on the VM — `intern`/`text_of` native (Phase 3 SEED) — DONE.**
`invoke` now services a welded `compiler`-library function (the `compiler_welded` flag is
the safety boundary) via `Vm.callCompilerFn` — natively on flat memory, NO legacy
`Interpreter`: `intern(s: string) -> StringId` reads the string bytes from flat memory and
`internString`s into the (const-cast) table (pool-only, never touches type layout, so the
VM's cached sizes stay valid); `text_of(id) -> string` materializes the pooled text back
into flat memory as a fat pointer. Unlocked `0626` — the ONLY remaining const-init fallback
is now the inline-asm global (`1654`, genuinely not comptime-evaluable). Parity **688/688**
both gate ON and OFF; unit test added. This is the mechanism Phase 3 grows: the next
compiler functions (`find_type`, `register_struct`, the reflection readers) are added the
same way — flat-memory pointer in, handle/pointer out, no marshaling.
### Phase 3 — Compiler-API on flat memory (resume the stream — no weld)
With native-byte comptime values, re-home the compiler-API: