docs(plans): inline-asm design + ASM and FFI-linkage plans/checkpoints

Two new workstreams:
- ASM: inline assembly — asm { "tmpl", "=r" -> T, "r" = expr, clobbers(.…) },
  multi-return tuples; lowers via the existing llvm_api.c (no shim).
- FFI-linkage: add extern/export postfix keywords, migrate every #foreign onto
  them, then purge 'foreign' from the tree (end-state invariant).

Drop current/ from .gitignore so plans + checkpoints are tracked normally
(the dir was ignored; only checkpoints had been force-added). Includes
docs/inline-asm-design.md. specs.md change left uncommitted.
This commit is contained in:
agra
2026-06-14 12:14:16 +03:00
parent e386a0d0b4
commit c562fe236d
6 changed files with 1345 additions and 2 deletions

3
.gitignore vendored
View File

@@ -3,5 +3,4 @@ zig-out
.DS_Store
.vscode/
.sx-cache
.sx-tmp
current/
.sx-tmp

24
current/CHECKPOINT-ASM.md Normal file
View File

@@ -0,0 +1,24 @@
# sx Inline Assembly — Checkpoint (ASM stream)
Companion to `current/PLAN-ASM.md`; design in
[docs/inline-asm-design.md](../docs/inline-asm-design.md). Update after every
commit, one step at a time per the cadence rule (no commit may both add a test
and make it pass).
## Last completed step
None — plan authored, not yet started.
## Current state
Design fully converged (`docs/inline-asm-design.md`). Feasibility confirmed:
`llvm_api.c.*` exposes `LLVMGetInlineAsm` / `LLVMBuildCall2` /
`LLVMAppendModuleInlineAsm` (LLVM@19). No code written.
## Next step
**A.0** — add the `kw_asm` keyword (`src/token.zig` Tag + `StaticStringMap`) and a
unit lex test. Then A.1 (parse `asm { … }``AsmExpr`, lowering bails loudly).
## Log
- (init) Plan + design doc written; ASM stream opened.
## Known issues
None yet.

View File

@@ -0,0 +1,35 @@
# sx `extern`/`export` + `#foreign` retirement — Checkpoint (FFI-linkage stream)
Companion to `current/PLAN-EXTERN-EXPORT.md` — one merged plan: **Part A** adds
`extern`/`export`, **Part B** migrates `#foreign` and purges `foreign`. Update after
every commit, one step at a time per the cadence rule.
## Last completed step
None — plan authored, not yet started. (Merged the former EXTERN-EXPORT +
FOREIGN-MIGRATION plans into this single plan.)
## Current state
Syntax decided + ratified: bare `extern`/`export`, postfix in the `callconv(.c)`
slot, `extern ⇒ callconv(.c)`, library separate. Touch-points mapped — token
`token.zig:45,282`; parser `1950,3669,316,425,1305`; lowering
`decl.zig:1123,387,2110,2382,2514`; IR/emit already capable (no codegen change).
Export gap = 4 lowering conditions. Part B `foreign` footprint to purge: 643 lines /
~57 identifiers in `src/` + 28 doc lines. End-state invariant: **zero `foreign`** in
the live tree (Phase 9.4 gate). No code written.
## Next step
**Phase 0.0** — add `kw_extern`/`kw_export` tokens + keyword-map entries (beside
`kw_callconv`) + unit lex test. Then 0.1 (parser plumbing, unconsumed), then Phase 1
(`extern` working). See the plan's **"Kickoff prompt"** section.
## Open decisions
Part A ratified (bare / postfix / `⇒ callconv(.c)` / lib-separate). Part B (confirm
before Phase 9): runtime-class rename target — `Runtime*Class*` (recommended);
historical carve-out — keep `issues/*.md` provenance, gate the live tree only.
## Log
- (init) Plan written; FFI-linkage stream opened.
- (merge) Folded FOREIGN-MIGRATION in as Part B; deleted the split plan + checkpoint.
## Known issues
None yet.

82
current/PLAN-ASM.md Normal file
View File

@@ -0,0 +1,82 @@
# sx Inline Assembly — Implementation Plan (ASM stream)
**Design source of truth:** [docs/inline-asm-design.md](../docs/inline-asm-design.md).
This plan turns that doc's §II.7 stage-map + §II.8 phasing into ordered,
commit-sized, testable steps. Read the design doc first — this file is the
*how/when*, not the *what/why*.
**Surface (decided):**
`asm volatile { "template", "=r" -> T, "r" = expr, clobbers(.cc, .memory) }`
— brace block; `->` output / `=` input; `clobbers(.…)` dot-name list; N `-> Type`
outputs return a tuple; templates are pure AT&T (via LLVM).
**Feasibility (confirmed):** sx links LLVM@19; `src/llvm_api.zig` `@cImport`s
`llvm-c/Core.h`, so `llvm_api.c.*` already exposes `LLVMGetInlineAsm` (9-arg),
`LLVMInlineAsmDialectATT`, `LLVMBuildCall2`, `LLVMAppendModuleInlineAsm`. No shim.
**Relationship to other streams:**
- Phases AE (the inline-asm *expression*) are independent of EXTERN-EXPORT.
- Phase F (global asm) consumes `extern`/`export` to import/expose asm symbols —
do it **after** `PLAN-EXTERN-EXPORT.md` Phase 2.
## Cadence (IMPASSIBLE)
No commit may both add a test AND make it pass. Each feature step is either a
behavior-locking PASSING test, or an xfail test the *next* commit turns green.
Arch-pinned tests live in `examples/16xx-platform-asm-*` (must declare `target=`).
Never regenerate snapshots while red.
## Phase A — keyword + AST + parser (parses; no codegen)
| Step | Commit | What | Files |
|---|---|---|---|
| A.0 | lock | add `kw_asm` keyword + map entry; unit lex test `asm → kw_asm` | `src/token.zig`, `src/lexer.zig` + `.test.zig` |
| A.1 | xfail | parse `asm { … }``AsmExpr`/`AsmOperand` in `parsePrimary`; pin an AST/`sx ir` parse snapshot; lowering still `bailDetail("inline asm codegen unimplemented")` | `src/ast.zig` (:85 union arm, :721 structs), `src/parser.zig` (parsePrimary), `src/ir/interp.zig` |
| A.2 | green | parse-shape snapshot lands green; the unimplemented bail is loud + named | — |
## Phase B — sema / typing
| Step | Commit | What | Files |
|---|---|---|---|
| B.0 | xfail | result-type rule (0→`void` / 1→`T` / N→named-or-positional tuple) + checklist (no-output⇒`volatile`, layout, comptime-string template) — pin error messages | `src/ir/expr_typer.zig` |
| B.1 | green | typing + diagnostics implemented; `.unresolved` sentinel on failure (no silent default) | `src/ir/expr_typer.zig`, `src/ir/semantic_diagnostics.zig` |
## Phase C — IR op + lowering
| Step | Commit | What | Files |
|---|---|---|---|
| C.0 | lock | add `inline_asm: InlineAsm` to `Op` + `AsmOperand` (role/name/constraint/operand) + interp `bailDetail` arm; unit tests for the IR shape | `src/ir/inst.zig` (:80), `src/ir/interp.zig` |
| C.1 | xfail→green | `lowerAsmExpr` in `lowerExpr` dispatch — interns template/constraints/clobber-names, lowers input `Ref`s, sets result `TypeId` | `src/ir/lower/expr.zig` |
## Phase D — LLVM emit (single value-output; the core)
| Step | Commit | What | Files |
|---|---|---|---|
| D.0 | xfail | `examples/16xx-platform-asm-syscall-write.sx` + `…-register-read.sx` + `…-no-output-volatile.sx` + `…-missing-volatile.sx` (expected compile error) — all red | examples + `expected/` markers |
| D.1 | green | `emitInlineAsm`: **port `FuncGen.airAssembly`** — constraint-string assembler (outputs `=`/`+`, inputs, `clobbers(.name)``~{name}`), `%[name]``${N}` / `%%` / `%=` template rewriter, `LLVMGetInlineAsm`+`LLVMBuildCall2`, `sideeffect=volatile`, AT&T dialect | `src/ir/emit_llvm.zig` (emitInst dispatch + handler) |
| D.2 | green | lock the template-rewrite + constraint string via an `expected/*.ir` snapshot on `…-template-subst.sx` | examples |
**Phase D verification:** `zig build test`; the syscall example runs on
`x86_64-linux`; IR snapshot matches the design doc's worked `sys_write` lowering.
## Phase E — multi-return tuples + `clobbers(.…)`
| Step | Commit | What | Files |
|---|---|---|---|
| E.0 | xfail | `…-asm-multi-return.sx` (`divmod``(quot,rem)`, `cpuid`→4-tuple) red | examples |
| E.1 | green | N `out_value` → LLVM struct return + `extractvalue i` → sx tuple (named when operands named); `clobbers(.name)` dot-name lowering finalized | `src/ir/emit_llvm.zig`, `src/ir/lower/expr.zig` |
## Phase F — global asm (needs EXTERN-EXPORT Phase 2)
| Step | Commit | What | Files |
|---|---|---|---|
| F.0 | xfail | top-level `asm { … }` decl parsed (reject operands/`volatile`); `…-asm-global.sx` (defines a symbol, imported via `extern`) red | `src/parser.zig`, `src/ast.zig` |
| F.1 | green | lower `asm_global``c.LLVMAppendModuleInlineAsm`; comptime-call guard (dlsym-miss is loud); blocks concatenate in source order | `src/ir/lower/decl.zig`, `src/ir/emit_llvm.zig`, `src/ir/interp.zig` |
## Phase G — later (own steps when scheduled)
`-> @place` write-through + read-write (`"+r" -> @place`) + indirect-memory
(`"=*m"`) outputs · `%=` unique-id · output-to-const rejection · Intel-dialect
opt-in · naked functions (`callconv(.naked)`, coordinate with EXTERN-EXPORT).
## Open decisions (design doc §II.10)
Dialect (AT&T-only v1, recommended) · `volatile` contextual-keyword (recommended)
· brace separator comma (recommended) · `clobbers(.name)` dot-name sugar now →
checked per-arch `Clobber` enum later (Phase 4 of the design doc).
## End-to-end verification (per phase)
`zig build && zig build test`; for arch-pinned examples confirm they run on a
matching host or assert on `sx ir`/`.s` snapshots. After intentional output
changes only: `zig build test -Dupdate-goldens`, then review the diff.

View File

@@ -0,0 +1,198 @@
# sx `extern` / `export` + `#foreign` retirement — Plan (FFI-linkage stream)
**One stream, two parts.** **Part A** adds `extern`/`export` (the linkage surface);
**Part B** migrates every `#foreign` onto it and purges `foreign` from the tree.
They are *one* plan: Part B can't start until Part A is a behavior-equivalent
superset of `#foreign`, and Part A isn't "done" until Part B reaches the invariant.
**Design rationale:** [docs/inline-asm-design.md](../docs/inline-asm-design.md) §II.2
(Deviation 6) + §II.10 #4 + the syntax evaluation.
**Decided syntax**
```sx
name :: (sig) -> Ret [callconv(.x)] [extern | export] [;|{…}]; // functions
Name :: #objc_class("X") [extern | export] { … }; // aggregates (mirrors `struct #compiler`)
g : Type extern ["csym"]; // extern global
```
- `extern` = import (no body, external linkage, C ABI, no sx ctx) — `#foreign`'s role.
- `export` = define **and** expose (body + external linkage + C ABI + no ctx) — **new**.
- `extern`/`export` imply `callconv(.c)`; write `callconv` only to override.
- Library stays a separate axis (`#library`/build flags), not folded into `extern`.
> **END-STATE INVARIANT (hard requirement).** After this stream, `foreign` appears
> **nowhere** in the live tree — not the `#foreign` surface, and **not** internal
> identifiers. The extern AST is **not** named `foreign_expr`. Enforced by the
> Phase 9.4 grep gate. Scope today: 643 `foreign` lines / ~57 identifiers in `src/`
> + 28 in live docs — most of it the objc/jni **runtime-class** machinery.
**Naming constraint (so we can actually reach the invariant):** introduce
`extern`-named representations only — do **not** reuse or extend
`ForeignExpr`/`foreign_expr`/`VarDecl.is_foreign`. Carry extern/export on a new
`FnDecl.extern_export` modifier with a `;`/`{…}` body (so there is **no** `*_expr`
node for it); add `VarDecl.is_extern`/`extern_name`. The IR is already extern-named
(`Function.is_extern`, `Builder.declareExtern`).
**Key finding (scopes Part A):** the IR + LLVM emit **already support everything**
`Function.linkage` (`.external/.internal/.private`), `is_extern`, `call_conv`, and a
raw un-mangled symbol name are all emitted by `declareFunction`
(`emit_llvm.zig:1225-1300`). Part A is a **parser + lowering** job, no codegen change.
## Cadence (IMPASSIBLE)
No commit may both add a test AND make it pass (xfail-then-green, or a behavior-lock).
`zig build && zig build test` after every step. Never regenerate snapshots while red.
---
# PART A — add `extern` / `export` (alongside `#foreign`)
## Phase 0 — tokens + parser plumbing
| Step | Commit | What | Files |
|---|---|---|---|
| 0.0 | lock | add `kw_extern`, `kw_export` (Tag enum + `StaticStringMap`, beside `kw_callconv` at `token.zig:45,282`); unit lex test | `src/token.zig` |
| 0.1 | lock | `parseOptionalExternExport()` (mirror `parseOptionalCallConv`, `parser.zig:3669`) + `ast.ExternExportModifier` enum + `FnDecl.extern_export` + `VarDecl.is_extern`/`extern_name` fields; **not yet consumed**; unit AST test | `src/parser.zig`, `src/ast.zig` |
## Phase 1 — `extern` (import; equivalent to lib-less `#foreign`)
| Step | Commit | What | Files |
|---|---|---|---|
| 1.0 | xfail | accept postfix `extern` after the callconv slot (`parser.zig:1950`); `examples/12xx-ffi-extern-fn.sx` extern-binds a libc symbol — red (lowering not wired) | `src/parser.zig` |
| 1.1 | green | lowering: `extern``is_extern`, `.external`, `callconv(.c)`, no ctx — route through `declareExtern` like a lib-less `#foreign` (anchors `decl.zig:1123,387,2110,2113`). Example green | `src/ir/lower/decl.zig` |
| 1.2 | green | optional `extern "csym"` rename + extern-global form `g : T extern;` (`parser.zig:425` path) | `src/parser.zig`, `src/ir/lower/decl.zig` |
## Phase 2 — `export` (define + expose; the NEW capability)
Fills the four export-gap conditions (all in `src/ir/lower/decl.zig`):
| Gap | Anchor | Fix |
|---|---|---|
| (i) linkage forced `.internal` | `:2382`, `:2514` | also `.external` when `extern_export == .export` |
| (ii) C ABI not promoted | `:2110` | also `.c` when `== .export` |
| (iii) no symbol-name override | `emit_llvm.zig:1226` raw name | parse optional `export "csym"`; map in the name map |
| (iv) ctx param not suppressed | `:387` `funcWantsImplicitCtx` | also suppress when `== .export` |
| Step | Commit | What | Files |
|---|---|---|---|
| 2.0 | xfail | multi-file test: an `export fn` called from a companion `.c` caller (same `XXXX-` prefix) — red (still internal) | `examples/12xx-ffi-export-fn.{sx,c}` + `expected/` |
| 2.1 | green | gaps (i),(ii),(iv): `export` ⇒ external + C-ABI + no-ctx on a **defined** fn (uses `beginFunction`, not `declareExtern`) | `src/ir/lower/decl.zig` |
| 2.2 | green | gap (iii): `export "csym"` symbol-name override | `src/parser.zig`, `src/ir/lower/decl.zig` |
## Phase 3 — aggregates (objc / jni runtime classes)
| Step | Commit | What | Files |
|---|---|---|---|
| 3.0 | xfail | `#objc_class("X") extern { … }` (import) + `… export { … }` (define) parse alongside legacy `#foreign #objc_class` | `src/parser.zig` (`tryParseForeignClassPrefix` :1305, `parseForeignClassDecl` :1369) |
| 3.1 | green | map postfix `extern`→reference, `export`→define+register; per-runtime tests (objc, jni) | `src/parser.zig`, `src/ir/lower/decl.zig`, `src/ir/lower/objc_class.zig` |
## Phase 4 — interplay, diagnostics, docs
`extern`+`callconv` stacking/redundancy; reject `extern`+`export` together;
`specs.md` documents `extern`/`export` (the three axes); `#foreign` still documented
until Part B cutover.
> **GATE A→B.** `extern`/`export` are a behavior-equivalent **superset** of
> `#foreign`. Lock with a unit test asserting `#foreign` and `extern` lower to
> identical IR for a sample fn / global / class. Do not start Part B before this.
---
# PART B — migrate `#foreign` → `extern`/`export`, then purge `foreign`
**Inventory (drives the batches):** `#foreign` = 466 uses. ~391 sx-code (308 fns
[207 lib / 196 rename], 75 classes [39 objc / 31 jni], 8 globals) + ~145 example
snapshots. 6 libs (`sqlib`98 `libc`61 `objc`22 `tlib`12 `raylib`7 `clib/pcaplib`3).
Hotspots: `vendors/sqlite`(98), `platform/{android,uikit,android_jni,sdl3}`,
`std/{socket,thread,fs,time}`, `ffi/{objc,raylib}`.
## Phase 5 — `#foreign` becomes an alias for `extern`
| Step | Commit | What | Files |
|---|---|---|---|
| 5.0 | lock | route the `#foreign` parser paths (`parser.zig:316,425,1305,1970`) to build the *same extern-named* AST as `extern`/`export`. Suite green, snapshots unchanged | `src/parser.zig` |
| 5.1 | lock | unit test: `#foreign` and `extern` produce identical IR (fn/global/class) | `src/ir/lower/decl.test.zig` |
## Phase 6 — migrate stdlib (behavior-preserving; snapshot diff must be EMPTY)
One commit per batch; rewrite `#foreign``extern` (fns/globals),
`#foreign #objc_class``#objc_class … extern`, defined classes → `… export`.
| Step | Batch | ~sites |
|---|---|---|
| 6.1 | `library/vendors/sqlite/` | 98 |
| 6.2 | `library/modules/platform/` (uikit/android/android_jni/sdl3) | ~95 |
| 6.3 | `library/modules/std/` (socket/thread/fs/time/process/…) | ~60 |
| 6.4 | `library/modules/ffi/` (objc/raylib/objc_block/…) | ~50 |
| 6.5 | remaining `library/` + `vendors/` | remainder |
## Phase 7 — migrate examples + issues (empty snapshot diff; review every diff)
| Step | Batch |
|---|---|
| 7.1 | `examples/12xx-ffi-*` (plain C) |
| 7.2 | `examples/13xx-ffi-objc-*` |
| 7.3 | `examples/14xx-ffi-jni-*` |
| 7.4 | `issues/*` repros + stragglers |
A non-empty diff ⇒ the alias wasn't behavior-equivalent — stop, fix Phase 5.
## Phase 8 — cutover
| Step | Commit | What |
|---|---|---|
| 8.0 | xfail | `examples/11xx-diagnostics-foreign-removed.sx` expects a "`#foreign` removed; use `extern`/`export`" diagnostic — still accepted (red) |
| 8.1 | green | parser hard-rejects `#foreign` (mirrors the variadic `name: ..T` cutover); `specs.md` drops `#foreign`, documents `extern`/`export` |
## Phase 9 — total `foreign` purge (the invariant)
`foreign` must not appear anywhere in the live tree, surface *or* internal. Each step
a mechanical, behavior-preserving rename commit (snapshots unchanged), small
per-file/subsystem commits — not one sweep.
| Step | What | Identifiers (count → new) |
|---|---|---|
| 9.0 | delete the surface | `hash_foreign`(11) + lexer entry + the 4 parse paths + the alias |
| 9.1 | rename **linkage**`extern*` | `foreign_expr`(25) **eliminated** (folds into modifier) · `is_foreign`(39)→`is_extern` · `foreign_lib`/`foreign_name``extern_*` · `foreign_name_map``extern_name_map` · `callForeign`(8)→`callExtern` · `marshalForeignArg``marshalExternArg` · `is_foreign_c_api`(5)→`is_extern_c_api` · `dedupeForeignSymbol``dedupeExternSymbol` |
| 9.2 | rename **runtime-class** machinery → `runtime*` (decision 5) | `ForeignClassDecl`(65) · `ForeignMethodDecl`(31) · `ForeignClassMember`(20) · `ForeignFieldDecl`(15) · `foreign_class_map`(44) · `current_foreign_class`(34)/`_method` · `foreign_path`(62) · `ForeignRuntime` · `parse/tryParseForeignClass*` · `lowerForeign{Method,Static}Call` · `findForeign{Method,Property}InChain` · `resolveForeign*` · `register*ForeignClass*` · `foreignClass*Type` · `*ForeignRefs` |
| 9.3 | purge **live docs** (28 lines) | `specs.md`/`readme.md`/`CLAUDE.md`: drop `#foreign`, document `extern`/`export`; fix file-roles + FFI/bundling notes |
| 9.4 | **acceptance gate** | `grep -rniE 'foreign' src/ library/ examples/ specs.md readme.md CLAUDE.md`**0** |
---
## Open decisions
*Part A (ratified — recommendations stand):* 1. bare keywords (not `#extern`).
2. aggregate position postfix (`#objc_class(…) extern`, like `struct #compiler`).
3. `extern ⇒ callconv(.c)`. 4. library separate.
*Part B (confirm before Phase 9):* 5. runtime-class rename target — **`Runtime*Class*`**
(recommended; it's the object-model axis, not linkage) vs `Extern*Class*`.
6. historical carve-out — keep `issues/*.md` (+ design-doc prose) as provenance,
gate only the live tree (recommended) vs purge everything.
## Relationship to ASM
`PLAN-ASM.md` Phase F (global asm) consumes `extern` (import the asm symbol) and
`export` (let asm call back into sx) — do it after **Part A Phase 2**.
---
## Kickoff prompt (paste into a fresh session to start Part A)
> Work the FFI-linkage stream per `current/PLAN-EXTERN-EXPORT.md` (+ checkpoint
> `current/CHECKPOINT-EXTERN-EXPORT.md`). First read the plan's header (Decided
> syntax, Naming constraint, Key finding) and Part A; rationale is in
> `docs/inline-asm-design.md` §II.2 (Deviation 6) + §II.10 #4.
>
> **This session = Part A, Phases 0 and 1 only** (`extern` works as a bare postfix
> keyword equivalent to a lib-less `#foreign` fn/global binding; `#foreign` stays
> untouched). Do NOT start Phase 2 (`export`) or Part B (migration).
>
> **Cadence (IMPASSIBLE):** no commit may both add a test and make it pass — lock
> behavior with a passing test, or land an xfail the next commit turns green.
> `zig build && zig build test` after every step.
>
> **Naming constraint (hard):** introduce only `extern`-named AST — do NOT reuse or
> extend `ForeignExpr`/`foreign_expr`/`VarDecl.is_foreign`. Use a new
> `FnDecl.extern_export` modifier (body `;` or `{…}`) and `VarDecl.is_extern`/
> `extern_name`. IR is already extern-named (`Function.is_extern`, `declareExtern`).
>
> Steps (commit after each; update the checkpoint each time):
> - 0.0 lock: `kw_extern`/`kw_export` tokens + map entries beside `kw_callconv`
> (`src/token.zig:45,282`) + unit lex test.
> - 0.1 lock: `parseOptionalExternExport()` (mirror `parseOptionalCallConv`,
> `parser.zig:3669`) + `ast.ExternExportModifier` + `FnDecl.extern_export` +
> `VarDecl.is_extern`/`extern_name` (parsed, unconsumed) + unit AST test.
> - 1.0 xfail: accept postfix `extern` after the callconv slot (`parser.zig:1950`);
> add `examples/12xx-ffi-extern-fn.sx` that extern-binds a libc symbol (red).
> - 1.1 green: in `src/ir/lower/decl.zig`, lower `extern` like a lib-less `#foreign`
> import — `is_extern`, `.external`, `callconv(.c)`, no ctx, via `declareExtern`
> (anchors :1123, :387, :2110, :2113). Example goes green.
> - 1.2 green: optional `extern "csym"` rename + extern-global `g : T extern;`
> (`parser.zig:425`).
>
> Stop at end of Phase 1. Verify: suite green; the `extern` libc binding runs;
> `#foreign` still works with no snapshot diffs. If you hit an unrelated compiler
> bug, follow the CLAUDE.md IMPASSIBLE RULE (file an issue, stop).

1005
docs/inline-asm-design.md Normal file

File diff suppressed because it is too large Load Diff