From da6a8423c79840ee51811f22a81859e0a22e5c20 Mon Sep 17 00:00:00 2001 From: agra Date: Thu, 18 Jun 2026 18:19:44 +0300 Subject: [PATCH] =?UTF-8?q?plan/checkpoint:=20#compiler/compiler=5Fcall=20?= =?UTF-8?q?is=20DELETED=20not=20bridged=20=E2=80=94=20BuildOptions=20?= =?UTF-8?q?=E2=86=92=20abi(.zig)=20extern=20compiler?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Direction correction (user): the VM does NOT get a transitional compiler_call → hook-registry shim. BuildOptions is re-expressed as abi(.zig) extern compiler functions (the compiler-API the VM already dispatches via callCompilerFn), and the #compiler attribute + compiler_call op + Value-based hook Registry are deleted. Also record that 4D (host FFI) is DONE via the arena/absolute-pointer allocator (the earlier pin/tag hazard is moot — the arena never moves an allocation), and that 0602/0603 stay on legacy fallback until the BuildOptions migration lands. No code change (reverted the speculative compiler_call bridge). --- current/CHECKPOINT-COMPILER-API.md | 39 +++++++++---------- current/PLAN-COMPILER-VM.md | 61 +++++++++++++++++------------- 2 files changed, 55 insertions(+), 45 deletions(-) diff --git a/current/CHECKPOINT-COMPILER-API.md b/current/CHECKPOINT-COMPILER-API.md index bb02c4dd..1545fef2 100644 --- a/current/CHECKPOINT-COMPILER-API.md +++ b/current/CHECKPOINT-COMPILER-API.md @@ -43,25 +43,26 @@ with ONE welded mechanism. Branch: `reify` (off `master`). Update after every st > example 0636 `toupper`); **4D.2** slice/string args → NUL-term `char*` + float-arg/return guards > (`6a7f690`, example 0637 `strlen([:0]u8)`). **699/0 BOTH gates.** > -> **4D.3 findings (for a fast restart):** the corpus drivers are `0602`/`0603` — their `#run -> configure()` calls BuildOptions hooks (`build_options`, `BuildOptions.set_post_link_callback`) → -> the `.compiler_call` op → currently bails on the VM. The hook ABI is `Value`-based: -> `HookFn = fn(interp: *const Interpreter, args: []const Value, bc: *BuildConfig, alloc) HookError!Value` -> (`compiler_hooks.zig:103`); the registry is `Registry` (built by `registerDefaults`), reached on the -> interp as `self.hooks` + `self.build_config` (`interp.zig:204`/`:201`, set by emit_llvm/core before -> the eval). Plan for 4D.3 (transitional Value shim — rewrite the ABI to flat memory only in 4F): -> (1) thread `*BuildConfig` (+ the hook `Registry`) into the `Vm` and through `tryEval` from the -> emit-time sites (mirror how `interp_inst.build_config` is set at `emit_llvm.zig:860`/`:106x`); -> (2) add a `.compiler_call` exec arm: resolve the hook by name, `regToValue` each flat arg → `Value`, -> call the hook, `valueToReg` the result. **Open question:** the hooks take `*const Interpreter` — -> audit whether any DEREFERENCE it for the arg kinds in play (`.string`/`.func_ref`/`.int` decode via -> `asString`/`asFuncRef`, which for a `regToValue`-built Value don't need the interp). If some hook -> needs a real interp, the pragmatic transitional route is to pass the `interp_inst` already created -> at the emit-time site into `tryEval`. Then `0602`/`0603` should run HANDLED. Dependency-ordered -> remainder: **4A leftovers** (out/print [double-output-on-fallback caveat], global_addr, trace, -> switch_br for the Any-tag type-switch in 0114/0520–0524/1035) · **4B** VM diagnostics · **4C** -> `#insert` · **4E** bundler (+ dedicated bundle tests) · **4F** flip default + delete -> `interp.zig`/`Value` + flat-memory `HookFn` + re-express `define`/`make_enum`. +> **DIRECTION CORRECTION (2026-06-18, user): `#compiler` / `compiler_call` is DELETED, not bridged +> on the VM.** `BuildOptions` is RE-EXPRESSED as **`abi(.zig) extern compiler`** functions (the +> compiler-API surface the VM already dispatches via `callCompilerFn`); the `#compiler` attribute, the +> `compiler_call` IR op, and the `Value`-based hook `Registry` (`compiler_hooks.zig`) all go away. +> So there is **NO transitional `compiler_call`→hooks shim** on the VM (I started one — threading the +> legacy interp into `tryEval` for the hook registry — and reverted it; tree clean at `b05c74f`). +> `0602`/`0603` stay on legacy fallback until the BuildOptions migration lands. **Migration shape** +> (end-state, shares the `BuildConfig`-on-the-VM prerequisite with the bundler 4E): (1) each +> `BuildOptions` setter/getter becomes a `compiler` fn in `compiler_lib.bound_fns` + `Vm.callCompilerFn`, +> reading flat-memory args + a `*BuildConfig` threaded into the `Vm` (the same `BuildConfig` +> `main.zig` forwards); (2) `library/modules/build.sx` declares them `abi(.zig) extern compiler` +> instead of `struct #compiler`; (3) delete the `compiler_call` op + `compiler_hooks.zig` +> `HookFn`/`Registry` + the `#compiler` parse/lower path. See `PLAN-COMPILER-VM.md` Phase 4. +> +> **Corpus-driven remainder (independent of the BuildOptions migration):** **4A leftovers** — +> out/print (double-output-on-fallback caveat: write directly only once the whole eval is VM-handled), +> global_addr, trace, and `switch_br` for the Any-tag type-switch (0114/0520–0524/1035, the box_any +> examples that now bail further at `switch_br`/`type_name`) · **4B** VM diagnostics · **4C** `#insert`. +> Then the BuildOptions migration + **4E** bundler (+ dedicated bundle tests) + **4F** flip default + +> delete `interp.zig`/`Value` + re-express `define`/`make_enum`. > Starting at **4A.1 (box_any/unbox_any)**. See `PLAN-COMPILER-VM.md` → Phase 4 for the full plan + > top risks (flat-pointer escape on buffer realloc; bundler test coverage). > Earlier landed: dedicated `Type` builtin TypeId (`6844fb9`/`94f60c5`/`554871b`); WRITE side diff --git a/current/PLAN-COMPILER-VM.md b/current/PLAN-COMPILER-VM.md index 22106f04..f26b1cfa 100644 --- a/current/PLAN-COMPILER-VM.md +++ b/current/PLAN-COMPILER-VM.md @@ -408,34 +408,43 @@ are legitimate negative-test bails that BECOME VM diagnostics, 1145 is a scan ar - **4C — `#insert` on the VM (role B).** Re-wire `evalComptimeString` through `tryEval`; the lowering-time-IR hardening that forced the 0737 revert is already in place. Verify the `#insert` corpus parity. -- **4D — host FFI + `#compiler` hooks on the VM (role D + substrate). The big one.** - - **4D.1** Host-FFI escape in `Vm.invoke`: for an extern/bodyless callee not modeled by - `callMemBuiltin`, route to `host_ffi` trampolines — marshal each arg (scalar word, or - flat `Addr` → real host pointer `machine.mem.items.ptr + addr`), call, map the return. - **Central hazard:** flat-memory buffer realloc-on-growth invalidates a live host - pointer — PIN the buffer (or copy) while a host pointer is outstanding. (Note: `#run` - already calls host extern via the interp today, so host effects at comptime are not a - new policy.) - - **4D.2** `compiler_call`: route to the hook registry. Transitional shim first — convert - flat↔`Value` at the boundary (`regToValue`/`valueToReg`) so hooks work unchanged; then - (in 4F) rewrite `HookFn` to a flat-memory ABI to drop `Value`. -- **4E — post-link bundler on the VM (role C).** Depends on 4D. Route - `core.invokeByFuncId` / `main.zig`'s post-link call through the VM. **No corpus - coverage** (only runs on `sx build --bundle/--apk`) — add dedicated bundle smoke tests - (min `.app` + `.apk`); gate on real bundle builds, not just `zig build test`. Riskiest - phase. +- **4D — host FFI on the VM (role D substrate). DONE.** Solved by a better allocator, not a + pin/tag scheme: the comptime memory is now an **arena** of stable host allocations and `Addr` + IS a real host pointer (`4D.0`, `625ba0f`), so a flat-memory pointer and an FFI-returned host + pointer are the same value — no translation, no realloc hazard. `Vm.callHostExtern` + (`4D.1`, `e7a8708`) dispatches ANY extern via `host_ffi` dlsym + trampolines (args/returns pass + untouched); `4D.2` (`6a7f690`) adds slice/string args (→ NUL-term `char*`) + float guards. + Examples 0636/0637. **(Superseded sub-note:** the earlier "pin the buffer / flat↔host translate" + hazard is moot — the arena never moves an allocation.) +- **`#compiler` / `compiler_call` — NOT bridged on the VM. DELETED, not ported (decision + 2026-06-18).** `BuildOptions` is RE-EXPRESSED as **`abi(.zig) extern compiler`** functions (the + compiler-API surface the VM already dispatches via `callCompilerFn`), and the `#compiler` struct + attribute + the `compiler_call` IR op + the `Value`-based hook `Registry` (`compiler_hooks.zig`) + all **go away**. So there is NO transitional `compiler_call`→hooks shim on the VM — `0602`/`0603` + stay on legacy fallback until this migration lands. Migration shape (part of the end state, with + the bundler): (1) each `BuildOptions` setter/getter becomes a `compiler` function in + `compiler_lib.bound_fns` + `Vm.callCompilerFn`, reading flat-memory args + a **`BuildConfig` + threaded into the VM** (the same `BuildConfig` `main.zig` forwards); (2) `library/modules/build.sx` + declares them `abi(.zig) extern compiler` instead of `struct #compiler`; (3) delete the + `compiler_call` op, `compiler_hooks.zig`'s `HookFn`/`Registry`, and the `#compiler` parse/lower + path. The `BuildConfig` threading is the shared prerequisite with the bundler (4E). +- **4E — post-link bundler on the VM (role C).** Depends on the FFI escape (done) + the + `BuildConfig`-on-the-VM threading above. Route `core.invokeByFuncId` / `main.zig`'s post-link + call through the VM. **No corpus coverage** (only runs on `sx build --bundle/--apk`) — add + dedicated bundle smoke tests (min `.app` + `.apk`); gate on real bundle builds, not just + `zig build test`. Riskiest phase. - **4F — flip + delete (the end state).** Flip the VM to default (retire `-Dcomptime-flat`); - delete `interp.zig` (`Interpreter`/`Value`/`defineEnum`…/`reflectTypeInfo`/`callExtern`); - drop the `regToValue`/`valueToReg` bridge; rewrite `HookFn` to the flat-memory ABI - (drops `Value`); simplify `core.zig` (`invokeByFuncId` → VM) + `main.zig` (`last_bail_*` - → VM diagnostics); remove the dual-path `compiler_lib` handlers (keep only VM-native - `callCompilerFn`); re-express `define`/`make_enum` as sx over the compiler-API (allocation - works on the sole evaluator) and land the original 0141 repro as a corpus test. + delete `interp.zig` (`Interpreter`/`Value`/`defineEnum`…/`reflectTypeInfo`/`callExtern`); drop + the `regToValue`/`valueToReg` bridge; simplify `core.zig` (`invokeByFuncId` → VM) + `main.zig` + (`last_bail_*` → VM diagnostics); remove the dual-path `compiler_lib` handlers (keep only + VM-native `callCompilerFn`); re-express `define`/`make_enum` as sx over the compiler-API + (allocation works on the sole evaluator) and land the original 0141 repro as a corpus test. -**Dependencies:** 4A → (4B, 4C independent) ; 4D → 4E ; all → 4F. -**Top risks:** (1) flat-pointer escape to host FFI on buffer realloc (4D.1); (2) the bundler -has no corpus guard (4E needs dedicated tests); (3) `Value` deletion is gated on the -flat-memory `HookFn` rewrite (4D.2→4F). +**Dependencies:** 4A → (4B, 4C independent) ; FFI(done)+`BuildConfig`-on-VM → (BuildOptions +migration, 4E) → 4F. +**Top risks:** (1) the bundler has no corpus guard (4E needs dedicated tests); (2) deleting +`#compiler`/`compiler_call` + re-expressing `BuildOptions` over the compiler-API touches the whole +build/bundle path — stage it behind real bundle builds. ## Open questions (resolve as reached, record decisions here)