lab/sx
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
fde767913b59b205a7cfff9ee77af540a594f2c8
sx/current/CHECKPOINT-EXTERN-EXPORT.md

16 KiB
Raw Blame History

sx extern/export + #foreign retirement — Checkpoint (FFI-linkage stream)

Companion to current/PLAN-EXTERN-EXPORT.md — one merged plan: Part A adds extern/export, Part B migrates #foreign and purges foreign. Update after every commit, one step at a time per the cadence rule.

Last completed step

Phase 4 (green) — PHASE 4 COMPLETE → PART A DONE; GATE A→B LOCKED. Four pieces: (1) GATE A→B unit test (lower.test.zig, lowerSrcToIr helper + "GATE A→B" test) — asserts #foreign and extern/export lower to byte-identical printed IR for a sample fn, data global, and Obj-C runtime class. This is the hard gate: Part B may not start migrating #foreign until it's green. Verified live (negative-probe: mutating one side fails the assertion). (2) Diagnostic — #foreign + postfix conflict (1174): prefix #foreign combined with postfix extern/export on an aggregate is now a clean parse error (was a confusing internal "compiler bug" during class synthesis). (3) Diagnostic — extern+export mutual exclusion (1175): both keywords on one fn decl is a clean error (was bare "expected ';'"). (4) Docs: specs.md + readme.md document the three extern/export axes (fns, globals, aggregates) alongside #foreign (which stays documented until the Part B cutover). Suite green (643 corpus / 444 unit, 0 fail). NOTE: extern+callconv redundancy needs no diagnostic — callconv(.c) extern is a harmless dup (both .c), and any non-.c callconv already errors on its own.

Prior: Phase 3.1 (green) — PHASE 3 COMPLETE. Postfix extern/export on #objc_class/

#jni_class aggregates fully works. parseForeignClassDecl now parses an optional extern/export modifier in the slot between the ("X") directive args and the { body (parser.zig:~1409): extern→is_foreign_eff = true (reference an existing runtime class, == legacy #foreign); export→is_foreign_eff = false (define + register a new sx class, == bare #objc_class with no #foreign). The modifier maps straight onto the same is_foreign decision the prefix #foreign already fed the node, so no objc_class.zig/ lowering change was needed — the new surface reuses the existing reference-vs-define path. Examples: 1348 (objc extern import, dispatches NSObject.alloc().init() → green via JIT), 1349 (objc export defined class, SxBar.alloc()/bump/get → counter: 2), 1426 (jni extern import, parse-only parse-only ok). Suite green (641 corpus / 443 unit, 0 fail).

Prior: Phase 2.2 (green) — PHASE 2 COMPLETE. export (define + expose) fully works:

external linkage + C ABI + no sx ctx + force-lowered root + optional "csym" rename. All four export-gap conditions filled in decl.zig: (i) .external linkage for extern_export == .export_ on both define paths (lowerFunctionBodyInto, lowerFunction); (ii) C-ABI promotion on the define paths + declareFunction stub cc; (iv) funcWantsImplicitCtx returns false for any non-.none modifier; force-lower: export fns are lowering roots in lowerMainAndComptime (else an uncalled export fn stays a bodiless declare); (iii) export … "csym" declares the stub under the C name

  • lazyLowerFunction promotes the body into it via foreign_name_map. Examples 1226 (bare export, C calls sx_square → 37/82) + 1227 (export "triple_c", C calls triple_c → 22) green via the new AOT corpus mode. Suite green (638 corpus / 443 unit, 0 fail).

AOT corpus mode + run_examples.sh retired. C→sx-by-name can't link under the corpus's sx run JIT mode (a JIT-resident symbol is invisible to a dlopen'd C dylib's flat-namespace lookup), so an expected/<name>.aot marker switches an example to a sx build + execute flow. The standalone tests/run_examples.sh was deleted — zig build test is now the sole corpus runner (verify-step.sh + CLAUDE.md updated).

Current state

Syntax: bare extern/export, postfix after callconv(.c), extern ⇒ callconv(.c). Decision 4 revised (user 2026-06-14): extern carries an optional LIB+"csym" axis (extern_lib/extern_name) like #foreign; the #library decl + build-flag linking stays separate. extern (PHASE 1) + export (PHASE 2) FULLY WORKING. extern: functions — bare (f :: (…) -> R extern;) AND renamed (extern [LIB] "csym"); data globals — bare + renamed. export: functions — bare (f :: (…) -> R export {…}) AND renamed (export "csym"); external linkage, C ABI, no ctx, force-lowered as a root. All behavior-equivalent to the matching #foreign form. extern_lib is parsed + stored but is a reference only — actual linking stays the #library/build-flag axis. Aggregates DONE (Phase 3): postfix extern/export on #objc_class/#jni_class (reference vs define+register). Interplay/diagnostics/docs DONE (Phase 4) + the A→B GATE IS LOCKED (#foreign ≡ extern/export IR for fn/global/class). PART A COMPLETE. Part B foreign footprint to purge: 643 lines / ~57 identifiers in src/ + ~28 doc lines. End-state invariant: zero foreign (Phase 9.4 gate). Examples: 1223 (extern bare fn), 1224 (extern fn rename), 1225 (extern bare global), 1226 (export bare fn, AOT), 1227 (export fn rename, AOT), 1348 (objc extern class), 1349 (objc export class), 1426 (jni extern class), 1174/1175 (interplay diagnostics).

Next step

PART B — Phase 5 (#foreign becomes an alias for extern). Part A is complete and the A→B gate is locked, so migration can begin. Phase 5.0 (lock): route the four #foreign parser paths (parser.zig:316,425,1305,1970) to build the same extern-named AST as extern/export — suite green, all snapshots unchanged. Phase 5.1 (lock): unit test that #foreign and extern produce identical IR (the gate already covers this — extend or reuse lowerSrcToIr). Then Phases 6–7 migrate stdlib + examples (empty snapshot diff per batch), Phase 8 cutover (hard-reject #foreign), Phase 9 total foreign purge.

⚠ CONFIRM BEFORE PART B (Open decisions 5 & 6): runtime-class rename target (Runtime*Class* recommended vs Extern*Class*) and the historical carve-out (keep issues/*.md provenance, gate live tree only — recommended). These decide Phase 9 renames; the plan says confirm before Phase 9, but worth raising with the user before sinking Part B effort. Also pick up the two Deferred items below at the start of Part B (the visibility-gate equivalence in particular needs a cross-module example).

FUTURE MILESTONE — C→sx-by-name in JIT (sx run). Investigated this session (user-requested spike, RESOLVED feasible-but-blocked). Adding the C #source objects directly into the ORC JITDylib (LLVMOrcLLJITAddObjectFile) instead of dlopen'ing a dylib makes C↔sx cross-references resolve both ways in one link domain — proven: a ~20-line spike ran 1226 via sx run (37/82) and all 13 existing #source FFI examples still passed. BLOCKER: C objects using _Thread_local (the return-trace runtime sx_trace.c) SIGABRT under JITLink — MachO thread-local-variable handling needs the ORC MachOPlatform set up (the bare LLVMOrcCreateLLJIT default doesn't), and C constructors/__mod_init_func won't run without ORC initializer support. 42 errors-* examples crashed in the spike. A real impl needs a C++ shim in llvm_shim.c (LLJITBuilder().setObjectLinkingLayerCreator(...) + MachOPlatform::Create) — its own milestone, NOT Phase 2/3 scope. The AOT .aot-marker corpus mode is the pragmatic test path and works today. Spike fully reverted (target.zig/main.zig at HEAD).

Deferred (carry into Part B): (a) docs — DONE in Phase 4 (specs.md/readme.md document extern/export; #foreign stays until the Part B cutover); (b) visibility-gate equivalence — bare extern (no extern_lib) is currently unconditionally visible via the c_import_bare gate (decl.zig:~2241, fd.body.data != .foreign_expr → return true), whereas a lib-less #foreign is policed by visibleOverEdges. Single-file examples don't exercise this; verify/align it at the A→B gate (a bare-extern lib-less fn should be policed like its #foreign twin). Adding it now would be untested — needs a cross-module example.

Open decisions

Part A ratified (bare / postfix / ⇒ callconv(.c) / lib-separate). Part B:

  • Decision 5 RATIFIED (user, 2026-06-14): runtime-class rename target = Runtime*Class* (object-model axis, not linkage). Drives the Phase 9.2 identifier renames.
  • Decision 6 STILL OPEN: historical carve-out — keep issues/*.md (+ design-doc prose) as provenance & gate only the live tree (recommended) vs purge everything. The user did NOT confirm this at the Part A milestone; confirm before Phase 9.

Log

  • (init) Plan written; FFI-linkage stream opened.
  • (merge) Folded FOREIGN-MIGRATION in as Part B; deleted the split plan + checkpoint.
  • (0.0) Added kw_extern/kw_export tokens + keyword-map entries + LSP keyword classification + lex linkage keywords test. Suite green; no identifier collisions in the corpus. lock commit.
  • (0.1) Added ast.ExternExportModifier + FnDecl.extern_export + VarDecl.is_extern/extern_name + parseOptionalExternExport() (unconsumed) + 2 parser unit tests. Suite green (443/633). lock commit.
  • (1.0a) Wired fn-path extern parsing (parseFnDecl + both lookahead predicates) + added FnDecl.extern_lib/extern_name + VarDecl.extern_lib per user feedback (decision 4 revised: extern carries an optional lib axis). Unconsumed by lowering. Suite green (443/633). lock commit.
  • (1.0b) Added examples/1223-ffi-extern-fn.sx + hand-authored success snapshots. RED (634 ran, 1 failed — sema body produces no value). xfail commit; 1.1 greens it.
  • (1.1) Wired extern fn lowering (6 edits in decl.zig, all declare-only routing mirroring foreign_expr): funcWantsImplicitCtx + declareFunction cc + lazyLowerFunction/lowerFunction/lowerFunctionBodyInto guards. 1223 green; declare i32 @abs(i32) (C ABI, no ctx). Suite green (634/443). green commit.
  • (1.2a) Added examples/1224-ffi-extern-fn-rename.sx (c_abs :: … extern "abs";) + hand-authored success snapshot (c_abs(-42) = 42). RED (635 ran, 1 failed — parse error: "abs" after extern not yet accepted). xfail; 1.2b greens it. (Also recovered a formatter-clobbered parser.zig — see Known issues.)
  • (1.2b) parseFnDecl parses the optional [LIB] ["csym"] tail into extern_lib/extern_name; declareFunction unifies the rename (foreign c_name OR extern_name → declare under C name, map sx→C) and extends the dedupe guard to extern. 1224 green (c_abs→abs); 1223 unregressed. Suite green (635/443). green commit. extern_lib parsed+stored (lib linking stays the #library axis).
  • (1.2c) Added examples/1225-ffi-extern-global.sx (__stdinp : *void extern;, mirrors #foreign global 1205) + success snapshot. RED (636 ran, 1 failed — parse error: var-decl extern not accepted). xfail; 1.2d greens it.
  • (1.2d) Parser kw_extern branch in the var-decl path ([LIB] ["csym"] → is_extern/extern_lib/extern_name) + registerTopLevelGlobal/globalInitValue consume is_extern. 1225 green (@__stdinp = external global ptr). Suite green (636/443). green commit. PHASE 1 COMPLETE — extern fns + globals fully work.
  • (JIT spike) User-requested feasibility investigation of C→sx-by-name in sx run (JIT). Verdict: feasible via LLVMOrcLLJITAddObjectFile (C objects into the ORC JITDylib) — proven by a throwaway spike — but blocked by JITLink MachO TLV handling (sx_trace.c's _Thread_local SIGABRTs without the ORC MachOPlatform). Own future milestone (see Next step). Spike reverted; no commit.
  • (2.0) Added the AOT corpus mode (expected/<name>.aot → sx build + execute) to corpus_run.test.zig + retired tests/run_examples.sh (verify-step.sh/CLAUDE.md updated) + examples/1226-ffi-export-fn.{sx,c,h} (C calls sx_square back). RED (AOT link fails: _sx_square undefined — export not lowered). xfail; 2.1 greens it.
  • (2.1) Filled export gaps i/ii/iv in decl.zig (.external linkage + .c cc on both define paths; funcWantsImplicitCtx false for any non-.none modifier) + force-lower export fns as roots in lowerMainAndComptime. 1226 green via AOT (37/82). Suite green (637/443). green commit.
  • (2.2a) Added examples/1227-ffi-export-fn-rename.sx (export "triple_c", C calls triple_c). RED (define path emits @sx_triple, ignores extern_name → C ref undefined). xfail; 2.2b greens it.
  • (2.2b) declareFunction rename branch fires for export (stub under C name + sx→C in foreign_name_map); lazyLowerFunction resolves the stub by that C name so the body promotes into the C-named function (define @triple_c). sx-side call sites resolve via the same map (probe: 5*5→25). 1227 green (22); 1226 unregressed. Suite green (638/443). green commit. PHASE 2 COMPLETE — export fully works.
  • (3.0) Added examples/1348-ffi-objc-extern-class.sx (postfix extern on #objc_class, new spelling of #foreign #objc_class). RED (parser: expected '{' after the directive). Hand-authored green snapshots. xfail commit; 3.1 greens it.
  • (3.1a) Wired the postfix extern/export aggregate slot in parseForeignClassDecl (optional modifier between ("X") and {; var is_foreign_eff overrides the passed is_foreign, threaded into the foreign_class_decl node). No lowering change — reuses the existing is_foreign reference-vs-define path. 1348 green. Suite green (639/443). green commit. PHASE 3 COMPLETE.
  • (3.1b) Behavior-lock: added examples/1426-ffi-jni-extern-class.sx (jni extern, parse-only) + examples/1349-ffi-objc-export-class.sx (objc export defined class, counter: 2). Both pass against the 3.1a parser change (locked in their own commit per the cadence rule). Suite green (641/443). lock commit. (Note: -Dupdate-goldens newline-normalizes empty stderr → reverted unrelated 1226/1227 churn, kept new stderr 0-byte per repo convention; runner normalizes both.)
  • (4.gate) GATE A→B — added lowerSrcToIr helper + "GATE A→B" test to lower.test.zig: #foreign ≡ extern/export byte-identical printed IR for fn / global / Obj-C class. Verified live via negative-probe (mutate one side → assertion fails). Behavior-lock; the equivalence was prototyped first with sx ir (LLVM IR byte-identical for all three). Suite green (641/444). test commit.
  • (4.diag1) Added examples/1174-diagnostics-foreign-postfix-conflict.sx — prefix #foreign
    • postfix export on an aggregate previously surfaced a confusing internal "emitObjcDefinedClassAllocImp … compiler bug". xfail (golden = clean message) → green: parseForeignClassDecl rejects the combo at the postfix keyword (failFmt). Suite green.
  • (4.docs) specs.md (new "extern/export linkage keywords" subsection after the #foreign FFI docs) + readme.md (C Interop section) document the three axes. docs commit.
  • (4.diag2) Added examples/1175-diagnostics-extern-export-conflict.sx — extern export on one fn decl previously gave bare "expected ';'". xfail (golden = clean message) → green: parseFnDecl rejects a second linkage keyword after parseOptionalExternExport. Suite green (643/444). PHASE 4 COMPLETE → PART A DONE. (Workflow note: -Dupdate-goldens keeps newline-normalizing the 5 empty-stderr files [1226/1227/1348/1349/1426] — revert that churn after each regen; a flaky 0712-sha256-streaming timeout appears only under concurrent zig build load, not a real failure.)

Known issues

  • Workflow hazard (1.2): an editor format-on-save (or zig fmt) clobbered the working-tree src/parser.zig between commits — it reformatted one-liners AND silently dropped my hasFnBodyAfterArrow extern edit, reverting 1223 to a parse error. Recovered with git checkout src/parser.zig (HEAD had the correct, committed version). After any Edit-tool change to a file the IDE may have open, rebuild + run the affected example before trusting the edit.
Reference in New Issue View Git Blame Copy Permalink