diff --git a/current/CHECKPOINT-COMPILER-API.md b/current/CHECKPOINT-COMPILER-API.md index c62cf1d3..5d04abbc 100644 --- a/current/CHECKPOINT-COMPILER-API.md +++ b/current/CHECKPOINT-COMPILER-API.md @@ -102,15 +102,16 @@ with ONE welded mechanism. Branch: `reify` (off `master`). Update after every st > a failure BAILS (hard build error), no `-> !`/failable-tuple needed. New VM readers `readStringList`/ > `readStringArg`. Smoke test `1663-platform-build-pipeline-link` (AOT): a post-link callback re-links the build's > objects to a temp output via sx `link` — the relinked binary RUNS; negative-probe verified (bad path → bail → -> build exit 1). The Zig driver STILL auto-links (1663 links a separate temp output); removing the auto-link is -> P5.4. **704/0 both gates.** **ALL build primitives now exist** (queries `emit_object`/`c_object_paths`/ -> `link_libraries` + action `link`). **NEXT — P5.3: the `on_build` slot.** A comptime-assignable compiler global -> (generalizes `post_link_callback_fn`: an assignable typed global w/ a stdlib default, vs a setter). `#run -> on_build = build;` captures the FuncId; the compiler invokes `on_build(config)` post-codegen WITH the -> CLI-derived `BuildConfig` as an ARG — which needs a VM entry that MARSHALS args (the gap `core.invokeByFuncId` -> rejects today with `ComptimeVmArgsUnsupported`). Then P5.4 (sx `default_build` that calls the primitives + -> RESTRUCTURE the driver to stop auto-emit/auto-link + delete `#compiler`/`compiler_call`/`compiler_hooks` + the -> S5a `build_options`/`set_post_link_callback`) — P5.4 kills the 4 strict `compiler_call` bails (1609/1614/1615/1616). +> build exit 1). **P5.3 (`on_build` registrar) + P5.4 CORE DONE (2026-06-19, newest Log entries):** the whole +> build is sx-driven via `default_pipeline` (force-lowered + auto-invoked; NO Zig auto-emit/auto-link); +> `on_build(cb)` is the sole callback mechanism; `set_post_link_callback` deleted. **703/0 both gates.** +> **NEXT — the FULL MIGRATION (no legacy left), spec'd as Phase 5 steps P5.5–P5.8 in `PLAN-COMPILER-VM.md`:** +> P5.5 migrate the 36 `BuildOptions` `#compiler` methods → VM-native `abi(.compiler)` arms (NO legacy handler — +> direct migration; thread a persistent allocator for setter strings; kills the 4 strict `compiler_call` bails +> 1609/1614/1615/1616) · P5.6 ALL bundling + code signing for EVERY target (macOS/iOS-device/iOS-sim/Android) in +> the sx `default_pipeline` · P5.7 DELETE `#compiler`/`compiler_call`/`compiler_hooks`/`interp.zig` + the +> `regToValue` bridge + VM→legacy fallback (drop gate-OFF; VM is the SOLE evaluator) · P5.8 build +> `~/projects/m3te` + `~/projects/distribution` end-to-end as the acceptance test + add `.app`/`.apk` smoke tests. > **FINAL atomic step (4F):** (`out` already done — VM-native via libc `write`) handle `interp_print_frames` + > flip strict-to-default (remove the fallback) + delete `interp.zig`/`Value` + re-express `define`/`make_enum`. > See `PLAN-COMPILER-VM.md` → Phase 4 for the full plan + top risks (bundler test coverage). @@ -440,33 +441,23 @@ when reached (sentinels or accessor fns; see the design doc Risks). deleted 1661/1663 (primitives now exercised by EVERY AOT build). `sx run` (JIT) is UNTOUCHED (emits in-process, never invokes `default_pipeline`). Benign `.ir` churn each step; **703/0 both gates.** > **REMAINING P5.4 (the BuildOptions-surface migration — large, mechanical, dual-path, string-lifetime-sensitive; - > NOT YET DONE — paused to avoid rushing core-IR deletion on a depleted context budget):** - > - **Migrate the 36 `BuildOptions :: struct #compiler` methods → free `ufcs … abi(.compiler)` fns** (the - > pattern S5a used for `set_post_link_callback`). Families: string SETTERS (`set_bundle_path`/`set_bundle_id`/ - > `set_codesign_identity`/`set_provisioning_profile`/`set_manifest_path`/`set_keystore_path`/`add_framework`/ - > `add_link_flag`/`set_output_path`/`set_wasm_shell`/`set_post_link_module`/`add_asset_dir`) — write/append to - > `BuildConfig`; string GETTERS (`binary_path`/`bundle_path`/`bundle_id`/`codesign_identity`/ - > `provisioning_profile`/`target_triple`/`manifest_path`/`keystore_path`) — read `BuildConfig`; BOOL getters - > (`is_macos`/`is_ios`/`is_ios_device`/`is_ios_simulator`/`is_android`) — compute from the triple; LIST/index - > getters (`framework_count`/`framework_at`/`framework_path_*`/`asset_dir_*`/`jni_main_*`). - > - **Each method = dual-path:** a VM `callCompilerFn` arm (flat memory) + a legacy `compiler_lib` handler. The - > legacy handlers can REUSE the existing `compiler_hooks` hook bodies (make them `pub`; the hook already dupes - > strings into a long-lived allocator + reads/writes `BuildConfig`). **String lifetime:** a setter called at - > `#run` (emit-time VM eval) must dupe the flat-memory string into a PERSISTENT allocator (NOT the per-eval VM - > arena which frees at `Vm.deinit`) — thread `emit_llvm.alloc` in (e.g. a `BuildConfig.string_alloc` field) so - > real bundle builds keep the strings; for the strict-bail JIT case the config is set-but-never-read so any - > alloc works, but do it right for real `sx build`. - > - **This kills the 4 strict `compiler_call` bails** (1609 `add_framework`; 1614/1615/1616 `set_bundle_*`/ - > `set_codesign_*`/`set_provisioning_*` — all SETTERS at `#run configure()`). Migrating the SETTERS alone - > suffices for strict-green (the GETTERS run only at post-link, which JIT/strict-run never fires). - > - **Then DELETE** `#compiler` (parse/lower attribute), the `compiler_call` IR op (`inst.zig` + every - > switch arm + `interp.zig:1130` dispatch), and `compiler_hooks.zig` `HookFn`/`Registry` (keep the pub hook - > bodies only if the legacy handlers call them — else inline + delete). The `BuildOptions` struct becomes a - > plain (zero-field) handle type, or is dropped if `opt` is unused. **Migrate the bundler** (`platform/bundle.sx` - > `bundle_main`) so `default_pipeline` calls it when `bundle_path` is set (the `#compiler` accessors it reads - > become the migrated `abi(.compiler)` getters) + the `--bundle` shim / `post_link_module` path is reworked or - > removed (default_pipeline owns bundling). **No corpus AOT bundle test exists** → add dedicated bundle smoke - > tests (the stream's top risk). After strict goes green: flip VM-to-default + delete `interp.zig` (4F). + > NOT YET DONE):** **FINAL DIRECTION (user 2026-06-19): FULL MIGRATION — NO LEGACY. Drop gate-OFF entirely; + > the VM is the SOLE evaluator; delete `interp.zig`. Migrate DIRECTLY to VM-native `abi(.compiler)` arms — NO + > legacy `compiler_lib` handlers, NO dual-path.** See `PLAN-COMPILER-VM.md` → Phase 5 steps **P5.5–P5.8** for + > the full spec. In brief: + > - **P5.5** — migrate all 36 `BuildOptions :: struct #compiler` methods → free `ufcs … abi(.compiler)` decls + + > `comptime_vm.callCompilerFn` arms (NO legacy handler). Setters dupe strings into a PERSISTENT allocator + > (thread `emit_llvm.alloc` via e.g. `BuildConfig.string_alloc`). Kills the 4 strict `compiler_call` bails. + > - **P5.6** — ALL bundling + code signing for EVERY target (macOS `.app`, iOS device/sim, Android `.apk`: + > Info.plist/codesign/provisioning/entitlements/framework-embed/AndroidManifest/javac/d8/aapt2/zipalign/ + > apksigner) runs in the sx `default_pipeline` (via the migrated getters + `fs`/`process` host-FFI). Remove the + > `--bundle`/`post_link_module` Zig shim. Compiler keeps ONLY the linker primitive (Option B). + > - **P5.7** — DELETE `#compiler` + `compiler_call` op + `compiler_hooks` (Registry/HookFn) + `interp.zig` + > (Interpreter/Value/reflectTypeInfo/callExtern) + `regToValue`/`valueToReg` + the VM→legacy fallback; make + > `-Dcomptime-flat` permanent. A VM bail is ALWAYS a build diagnostic now. Re-express `define`/`make_enum` as + > sx. Land the 0141 repro; reconcile 1654. + > - **P5.8** — build `~/projects/m3te` + `~/projects/distribution` end-to-end as the acceptance test that + > `default_pipeline` covers all targets; add `.app` + `.apk` bundle smoke tests (no corpus coverage today). - **P5.3 (`on_build` registrar) — the build-callback registration mechanism; callback takes `BuildOptions` (2026-06-19).** Per the user's design: `on_build(cb)` is the build-callback registrar (a FREE fn), generalizing `set_post_link_callback` — the callback is `(opt: BuildOptions) -> bool abi(.compiler)` and the compiler invokes diff --git a/current/PLAN-COMPILER-VM.md b/current/PLAN-COMPILER-VM.md index 82a8341a..e690417f 100644 --- a/current/PLAN-COMPILER-VM.md +++ b/current/PLAN-COMPILER-VM.md @@ -523,23 +523,67 @@ The compiler's whole post-IR role: codegen → build the CLI-derived `BuildConfi readers `readStringList`/`readStringArg` (inverse of `makeStringList`). Smoke test `1663-platform-build-pipeline-link` (AOT): a post-link callback re-links the build's objects to a temp output — the relinked binary RUNS; negative-probe verified. The Zig driver still auto-links (removed in P5.4). 704/0. -- **P5.3 — `on_build` slot:** a comptime-assignable compiler slot (GENERALIZES today's `post_link_callback_fn`: - an assignable typed global with a stdlib default, vs a setter). `#run on_build = build;` captures the - `FuncId`; the compiler invokes it post-codegen with the CLI-derived `BuildConfig`. -- **P5.4 — sx `default_build` + `BuildConfig`:** write the stdlib pipeline; move config/orchestration into sx. - **Delete** `#compiler` / `compiler_call` / `compiler_hooks` (`HookFn`/`Registry`) + the S5a - `build_options`/`set_post_link_callback` (config is now sx data passed as primitive args). -- **4E — post-link bundler on the VM (role C).** Depends on the FFI escape (done) + the - `BuildConfig`-on-the-VM threading above. Route `core.invokeByFuncId` / `main.zig`'s post-link - call through the VM. **No corpus coverage** (only runs on `sx build --bundle/--apk`) — add - dedicated bundle smoke tests (min `.app` + `.apk`); gate on real bundle builds, not just - `zig build test`. Riskiest phase. -- **4F — flip + delete (the end state).** Flip the VM to default (retire `-Dcomptime-flat`); - delete `interp.zig` (`Interpreter`/`Value`/`defineEnum`…/`reflectTypeInfo`/`callExtern`); drop - the `regToValue`/`valueToReg` bridge; simplify `core.zig` (`invokeByFuncId` → VM) + `main.zig` - (`last_bail_*` → VM diagnostics); remove the dual-path `compiler_lib` handlers (keep only - VM-native `callCompilerFn`); re-express `define`/`make_enum` as sx over the compiler-API - (allocation works on the sole evaluator) and land the original 0141 repro as a corpus test. +- **P5.3 — `on_build` registrar — DONE (2026-06-19).** `on_build(cb)` registers the build callback + (`cb: (opt: BuildOptions) -> bool abi(.compiler)`); the compiler force-lowers + auto-invokes the well-known + `default_pipeline` when no override. (Implemented as a registrar, not an assignable slot — the opaque + `BuildOptions` handle is one word, so arg-passing needs no struct marshaling.) +- **P5.4 core — DONE (2026-06-19).** `default_pipeline` in `build.sx` drives the whole build; NO Zig + auto-emit/auto-link; `emit_object`/`link` are sx-called actions via the `BuildHooks` vtable; + `set_post_link_callback` deleted (all callers on `on_build`). Build-path auto-imports `modules/build.sx`. + 703/0 both gates. + +### THE FINAL DIRECTION (user, 2026-06-19): FULL MIGRATION — NO LEGACY LEFT. + +**Decision: DROP gate-OFF entirely.** The VM becomes the SOLE comptime evaluator; `-Dcomptime-flat` is made +permanent then removed; `interp.zig` (the legacy tagged-`Value` `Interpreter`) is DELETED. There is no +dual-path, no legacy `compiler_lib` handler, no `regToValue`/`valueToReg` bridge, no VM→legacy fallback. We +migrate the BuildOptions surface DIRECTLY to VM-native `abi(.compiler)` arms (no legacy handler — there is no +legacy to handle). **All bundling + code signing for EVERY target lives in the sx `default_pipeline`.** + +- **P5.5 — migrate the 36 `BuildOptions :: struct #compiler` methods → VM-native `abi(.compiler)`.** Each + becomes a free `ufcs (self: BuildOptions, …) abi(.compiler)` decl (so `opt.method(...)` still resolves via + UFCS) with a `comptime_vm.callCompilerFn` arm — and **NO legacy `compiler_lib` handler** (the user's directive; + the legacy interp is going away). Families: string SETTERS (`set_bundle_path`/`set_bundle_id`/ + `set_codesign_identity`/`set_provisioning_profile`/`set_manifest_path`/`set_keystore_path`/`add_framework`/ + `add_link_flag`/`set_output_path`/`set_wasm_shell`/`set_post_link_module`/`add_asset_dir`) — write/append to the + threaded `BuildConfig`; string GETTERS (`binary_path`/`bundle_path`/`bundle_id`/`codesign_identity`/ + `provisioning_profile`/`target_triple`/`manifest_path`/`keystore_path`); BOOL getters (`is_macos`/`is_ios`/ + `is_ios_device`/`is_ios_simulator`/`is_android` — compute from the triple); LIST/index getters + (`framework_count`/`framework_at`/`framework_path_*`/`asset_dir_*`/`jni_main_*`, built via `makeStringList`). + **String lifetime:** a setter at `#run` must dupe the flat-memory string into a PERSISTENT allocator (NOT the + per-eval VM arena) — thread `emit_llvm.alloc` into the VM (e.g. `BuildConfig.string_alloc`) so the strings + survive to post-link. This kills the 4 strict `compiler_call` bails (1609/1614/1615/1616). +- **P5.6 — ALL bundling + code signing in `default_pipeline` (every target).** `default_pipeline` (or a + `bundle()` it calls, in `platform/bundle.sx`) performs, after `link`, the full per-target bundle when + `bundle_path()` is set — branching on `is_macos`/`is_ios_device`/`is_ios_simulator`/`is_android`: + - **macOS `.app`** — `Contents/{MacOS,Resources,Frameworks}`, `Info.plist`, embed `-framework` dylibs + + `install_name_tool` fixups, `codesign` (ad-hoc or with `codesign_identity`). + - **iOS device `.app`** — device slice, embedded `.mobileprovision` (`provisioning_profile`), entitlements, + `codesign` with the real identity; **iOS simulator `.app`** — sim slice, no provisioning, ad-hoc sign. + - **Android `.apk`** — `AndroidManifest.xml` (or `manifest_path` override), asset tree (`add_asset_dir`), + `#jni_main` Java → `javac` → `d8` → `classes.dex`, `aapt2` package, `zipalign`, `apksigner` with the + debug/`keystore_path` keystore. + All of it runs on the VM via the migrated `abi(.compiler)` getters + `fs`/`process` host-FFI (the existing + `platform/bundle.sx` logic, now reading the VM-native accessors instead of `#compiler` hooks). The compiler + keeps ONLY the linker as a primitive (Option B). Remove the `--bundle`/`post_link_module` Zig shim — bundling + is `default_pipeline`'s job; CLI flags feed `BuildConfig` and `default_pipeline` branches on it. +- **P5.7 — DELETE all legacy.** Remove the `#compiler` attribute (parse + lower), the `compiler_call` IR op + (`inst.zig` + every switch arm + the `interp.zig:1130` dispatch), `compiler_hooks.zig` + (`HookFn`/`Registry`/all hooks). Make `-Dcomptime-flat` permanent (VM always) and **delete `interp.zig`** + (`Interpreter`/`Value`/`defineEnum`…/`reflectTypeInfo`/`callExtern`/`last_bail_*`); drop the + `regToValue`/`valueToReg` bridge and the VM→legacy fallback in `emit_llvm` (`#run`/const-init) and + `comptime.zig` (type-fn / `#insert`) — a VM bail is now ALWAYS a build-gating diagnostic (4B wiring), never a + fallback. `core.invokeByFuncId` is already VM-only. Re-express `define`/`make_enum` as sx over the + compiler-API. Land the 0141 repro as a corpus test. Reconcile 1654 (asm-global at comptime) to the VM wording. +- **P5.8 — real-project validation (integration).** Build `~/projects/m3te` and `~/projects/distribution` with + the new pipeline end-to-end (their real bundle/codesign/target configs) — these are the acceptance test that + `default_pipeline` covers all targets. Fix gaps surfaced there. Add dedicated bundle smoke tests (min `.app` + + `.apk`) to the corpus (the bundler still has no `zig build test` coverage — the stream's top risk). + +**End state:** ONE evaluator (the VM); ZERO legacy; the entire build — emit, link, and all bundling + code +signing for macOS/iOS-device/iOS-sim/Android — is sx in `default_pipeline`, overridable via `#run on_build(...)`. +The compiler is: parse → IR → codegen → invoke `on_build`/`default_pipeline` on the VM (which calls back into +the linker primitive). `m3te` + `distribution` build clean. **Dependencies:** 4A → (4B, 4C independent) ; `abi(.compiler)` S1+S2(done) → S3 → S4 → S5 (BuildOptions) ; FFI(done)+`BuildConfig`-on-VM → (S5, 4E) → 4F.