Compare commits

1 Commits

Author SHA1 Message Date
agra
86615a2289 ... 2026-06-13 06:57:38 +03:00
3 changed files with 407 additions and 0 deletions

View File

@@ -0,0 +1,77 @@
# Checkpoint — distribution
- Status: **RUNNING**
- Plan branch: `distribution-plan` Base: `master`
- Current: A2 (pooled distd, PLAN-HTTPZ S7b) done 2026-06-12 → `74b41f5` + `c734f47`: 4 handler workers, writes serialized behind one mutex, /api/apps 5.7k → 12.0k req/s; en route the vendored sqlite flipped THREADSAFE=0→1 in sx (`3948813`) after the pool exposed heap corruption (pinned by server_http's concurrency case + sqlite_api). A1 → `48a13c4` (std.http readiness loop + keep-alive; 6 idle preconnects 3.7ms ttfb, was 3.9s behind two). sx grew time/socket-nb/kqueue/event/thread/http (PLAN-HTTPZ; issue 0131 found+fixed). Benchmark vs httpz: sx/current/BENCH-HTTPZ.md. Next: P6.2 admin UI actions; packaging (07) still blocked on sx Linux targets
- Updated: 2026-06-12 HEAD: `48a13c4`
## Progress
| step | status | attempts | merge |
|------|--------|----------|-------|
| P1.1 | merged | 1 | 331a3e06 |
| P2.1 | merged | 3 | b5529583 |
| P2.2 | merged | 2 | a2f7ad2a |
| P2.3 | merged | 3 | 882dce4f |
| P3.1 | merged | 1 | 94e24118 |
| P3.2 | merged | 1 | 1c4bba05 |
| P3.3 | merged | 1 | 20d520d0 |
| P3.4a | merged | 1 | e56a921 |
| P3.4b | merged | 1 | e56a921 |
| P3.5 | merged | 1 | e56a921 |
| P4.1 | merged | 1 | 734c00f |
| P4.2 | done | 1 | cf39589 |
| P4.3 | done | 1 | d8b7a7b |
| P4.4 | done | 1 | e2a5150 |
| P4.5 | done | 1 | 0a6fa65 |
| P4.6 | done | 1 | aea3d62 |
| P5.1 | done | 1 | afec94a |
| P5.2 | done | 1 | a1f13c4 |
| P5.3 | done | 1 | dc6908d |
| P6.1 | done | 1 | eef3d5c |
P4.2 onward is implemented directly in-session on `distribution-plan` (flow
harness retired). The sN→iN sx lang migration landed as `6c19f10`. P4.3
(token security, subplan 02 Slice 5) → `d8b7a7b`; P4.4 (bearer auth + write
endpoints, 04 Slice 2 + write half of 3/4) → `e2a5150`; P4.5 (remote
publish, 03 Slice 3) → `0a6fa65`; P4.6 (install pages + iOS modes, 04
Slice 5) → `aea3d62`. `make test` is 19/19 green at HEAD. The PLAN.md CI
contract works end-to-end against a running distd, and install pages
honor the iOS Install Policy (artifact_only / testflight / enterprise OTA
manifest). sx issue 0098 (enum literal into optional target lowers to
variant 0) bit `ua_platform`; worked around via typed locals — the sx-side
fix is still tracked as a separate followup. P5.1 → `afec94a`: SQLite
3.53.2 is VENDORED (vendor/sqlite/, API renamed dist_sqlite3_* so the
JIT's dlsym(RTLD_DEFAULT) can never bind the OS copy) with sx bindings in
src/db/sqlite.sx — SQLite is no longer blocked. The bindings then adopted
honest C integer widths (`3747c40`, 21/21 green). P5.2 → `a1f13c4`: the
store persists to `<store>/dist.db` (schema + whole-model save/load in
src/repo/db.sx, BEGIN IMMEDIATE transactions, busy_timeout for CLI/distd
interleaving); a pre-SQLite db.json imports once on first load (renamed
db.json.imported); consumers gate on `db.store_exists`; every test that
parsed db.json now queries dist.db through the bindings, plus new pinned
tests db_import.sx and the SQLite repo_roundtrip.sx. `make test` is 22/22
green at HEAD. En route, sx issue 0130 (`#library` behind two aliased
imports dropped from link/dlopen) was filed AND fixed in the sx repo
(`d739c5b`, examples/1617 regression). P5.3 → `dc6908d` (02 Slice 4,
closing subplan 02): `Channel.retention_keep` (0 = keep everything, N =
keep the newest N published lineage releases) set via `dist channel set`;
`dist store cleanup` prunes lineage-expired releases (never one ANY
channel points at — cross-promoted releases survive), GCs unreferenced
objects/ blobs, sweeps stale staging/ files, one audit event per deletion,
model saved before any unlink. En route it fixed repo.publish silently
resetting an existing channel's policy/rollout/retention on every publish
(pinned in tests/retention_cleanup.sx, which fails on the pre-fix code).
cleanup.sx carries a local opendir/readdir/closedir FFI shim — std.fs
still has no directory listing (sx wishlist). P6.1 → `eef3d5c` (subplan
06, first slice): the read-only admin console at /admin, server-rendered
from src/server/admin.sx exactly like the index/install pages (sx
strings, no client framework): apps overview, app detail (channels with
policy/rollout/retention, releases newest-first), release detail
(validation chips, serving channels, audit timeline), tokens (lifecycle
status, never secret material), audit log. Reads public per v0; writes
stay on the token-gated POSTs. The root-level index.html/styles.css/
app.js mock (rejected design, flow-harness era) is SUPERSEDED by the
real console — delete when convenient. `make test` is 24/24 green at
HEAD. Remaining for Milestone 1: admin UI actions/iteration (06),
Docker/UGREEN packaging (07, still blocked on sx Linux targets).

100
current/PLAN.md Normal file
View File

@@ -0,0 +1,100 @@
<!-- formerly generated by the retired flow harness; hand-maintained since the 2026-06-11 sx re-evaluation -->
# Distribution Platform — Milestone 1, Slice A: local content-addressed `dist ci publish`
First independently-shippable slice of PLAN.md's Milestone 1, scoped to what is implementable AND verifiable entirely inside /Users/agra/projects/distribution. The deliverable is the local CI publish path: `dist ci publish --manifest dist.json --local-store .dist --json` parses a manifest, creates/finds an app, drafts a release, stores each artifact content-addressed by SHA-256 in a local object store, runs common validation, publishes, promotes the requested channel, persists state to a human-readable db.json, and prints stable JSON (release id, artifact ids, digests, local download URLs). This is subplan 03 Slice 2 (local publish) standing on subplan 02 Slice 1+3 (in-memory domain + content-addressed storage) and the common-validation subset of subplan 05 — deliberately the no-server, no-SQLite, no-auth core so it dogfoods without a running distd.
PHASE-0 / CROSS-REPO DECISION (hard constraint 2). Subplan 01 (sx language + stdlib) lives in the SEPARATE sx repo (/Users/agra/projects/sx) and is OUT OF SCOPE — no step here edits the sx compiler or stdlib. RE-EVALUATED 2026-06-11 against current sx: the stdlib restructure moved everything under `modules/std/` behind the `modules/std.sx` facade — the flat legacy `modules/fs.sx`-style files are deleted. There is no `pub` keyword; aliases are the re-export mechanism, and the facade's namespace tail (`mem`, `fs`, `process`, `socket`, `json`, `xml`, `cli`, `hash`, `log`, `test`) carries one level into flat importers. sx primitives this milestone USES: `List(T)`, `print`/`format`, string helpers (`concat`, `substr`, `int_to_string`, `int_to_hex_string`, `path_join`), `std.fs` (read_file/write_file/exists/create_dir_all/copy_file/rename/basename/dirname), `std.process` (run/env/find_executable/exit/assert), `std.json` (Value/Object/Sink reader+writer), `std.cli` (argv, `<group> <command>` dispatch, flag specs, exit codes), `std.hash` (streaming SHA-256), `?T` optionals and the `!` error channel (`catch`/`onfail` bindings take parens), `Allocator.alloc_bytes`/`dealloc_bytes` plus the typed `std.mem` helpers, and reflection builtins. The originally planned TEMPORARY IN-REPO shims under `src/infra/` (hash/json/cli) were SUPERSEDED before being written — sx shipped `std.hash`, `std.json`, and `std.cli` natively, the product code uses them directly, and `src/infra/` has been removed. Pieces still missing from sx and intentionally avoided for this slice: HashMap (use linear scan over `List` pairs for tiny N), StringBuilder (use `concat`/`format`), std.time (publish.sx shims `time(2)` locally via FFI), real SQLite (replaced by a human-readable `db.json` in the store), HTTP/TLS (no server), archive/zip inspection (deep IPA/APK validation deferred), token-at-rest (no network auth in local mode).
DEFERRED to later slices/runs (named per constraint 3): subplan 01 sx-language work (`pub` exports, alias imports, namespace re-exports) and the bulk of the Phase 1 stdlib upstreaming; real SQLite persistence (subplan 02 Slice 2); retention/cleanup (02 Slice 4); token security at rest + scoped auth (02 Slice 5); the HTTP API and install pages (subplan 04 / Phases 5,7); remote publish + streaming upload + retries (03 Slice 3); deep platform-specific IPA/APK/macOS/Linux/Windows archive validation (subplan 05); admin UI (subplan 06 / Phase 8); Docker + UGREEN NAS packaging (subplan 07 / Phase 9); CI templates and observability polish (03 Slices 4-5).
## Acceptance criteria
- A build/test gate exists in the distribution repo (a `Makefile` + `tests/run.sh` that locate the sx binary via `SX ?= /Users/agra/projects/sx/zig-out/bin/sx`): `make test` runs every `tests/**/*.sx` test through the sx compiler, prints per-test ok/FAIL, exits 0 when all pass and non-zero on any failure; an intentionally-failing probe test makes it exit non-zero (gate proven, not vacuous).
- (SUPERSEDED 2026-06-11) The three planned `src/infra/` pure-sx shims were replaced by the sx stdlib before being written: SHA-256 comes from `std.hash`, JSON from `std.json`, and subcommand/flag parsing from `std.cli`, used directly from `modules/std/` with no edits to `/Users/agra/projects/sx`. Their behavior is exercised in-repo by the store, manifest, and CLI tests under `tests/`.
- `./dist ci publish --manifest examples/dist.json --local-store <dir> --json` (invoked via the gate, e.g. `make publish-example`) runs end-to-end against a fixture app whose manifest lists at least one Android APK and one iOS IPA artifact, exits 0, and prints a single pure-JSON object on stdout containing the release id, one artifact id + sha256 digest per artifact, and a local download URL/path per artifact (no human-only text on stdout under `--json`).
- Each published artifact's bytes are physically present in the local store under its SHA-256 key (e.g. `<dir>/objects/<sha256>`), the recorded digest equals an independent `shasum -a 256` of the source file, and re-running publish with byte-identical artifacts does not create a second copy of those bytes (content-addressed dedup).
- Platform state persists across separate CLI invocations via a human-readable `db.json` (or per-entity sidecars) under the store recording apps, releases, artifacts, channels, and audit events; a follow-up `./dist release rollback` (and `release promote`) invocation reads that state, moves the channel pointer, appends an audit event, and the change is verifiable by inspecting the store between invocations.
- Failure paths are loud and machine-readable: a malformed manifest, a missing artifact file, an unknown platform id, or a SHA-256/size mismatch each yields a non-zero exit and a JSON error object under `--json` (error code + message), with no partially-published release left pointed at by a channel.
- Every step is implemented and verified inside /Users/agra/projects/distribution using the sx compiler; the git working tree builds and `make test` is green at each step boundary; no change is made to the sx compiler or stdlib.
## Phases
### P1 — Build/test gate + in-repo sx infra shims (the greenfield foundation)
- **P1.1** Establish repo skeleton + sx build/test gate
- intent: Stand up the greenfield gate: a product source layout (`src/`, `src/infra/`, `tests/`, `examples/`) and a reproducible way to compile+run sx in this repo. Add a `Makefile` with `build`/`test`/`publish-example` targets and a `tests/run.sh` runner (mirroring sx's own tests/run_examples.sh convention) that locates the compiler via `SX ?= /Users/agra/projects/sx/zig-out/bin/sx`, runs each `tests/**/*.sx` via `$SX run`, and reports per-test pass/fail with a non-zero aggregate exit on any failure. Include one trivial passing sx program (imports `modules/std.sx`, prints, exits 0) to prove toolchain wiring.
- acceptance: `make test` exits 0 and prints `ok` for the smoke test; temporarily adding a deliberately-failing test makes `make test` exit non-zero (then remove it). `make build` compiles the smoke program without error. No edits outside the distribution repo.
- **P1.2** infra/hash.sx — SHA-256 (pure sx, known-answer tested)
- SUPERSEDED 2026-06-11: sx shipped `std.hash` (streaming SHA-256, lowercase hex) in `modules/std/hash.sx`; the store and validation code use it directly (see P2.2/P3.3). No in-repo shim exists.
- **P1.3** infra/json.sx — minimal JSON reader + writer (round-trip tested)
- SUPERSEDED 2026-06-11: sx shipped `std.json` (reader + writer, `Value`/`Object`/`Sink`) in `modules/std/json.sx`; manifest parsing, `db.json` persistence, and `--json` output use it directly (see P3.2/P2.3/P3.1). No in-repo shim exists.
- **P1.4** infra/cli.sx — minimal subcommand/flag parser (tested)
- SUPERSEDED 2026-06-11: sx shipped `std.cli` (argv access, `<group> <command>` dispatch, flag specs, exit-code contract) in `modules/std/cli.sx`; the `dist` entrypoint uses it directly (see P3.1). No in-repo shim exists.
### P2 — Product domain model + content-addressed store (in-memory, file-backed)
- **P2.1** Domain structs + boundary validation
- intent: Define the core domain in sx under `src/domain/` per subplan 02: App (id, slug, display_name, bundle ids by platform, owner, visibility, timestamps), Platform enum (ios, android_apk, macos, linux, windows), Release (id, app_id, version, build, channel, notes, created_by, timestamps), Artifact (id, app_id, release_id, platform, filename, content_type, size_bytes, sha256, storage_key, validation_status), Channel (app_id, name, current_release_id, policy, rollout_percent), AuditEvent (id, actor, action, target_type, target_id, metadata, created_at). Add domain validation for slugs, semantic-ish versions, channel names, and platform ids (subplan 02 Slice 1). Lookups use linear scan over `List` (HashMap deferred).
- acceptance: A test constructs valid and invalid Apps/Releases/Artifacts and asserts the validator accepts the valid ones and rejects each invalid case (bad slug, empty version, unknown platform, bad channel name) with a typed error. `make test` green.
- **P2.2** Content-addressed artifact store (staging → atomic move, dedup)
- intent: Implement a local artifact store under `src/store/` (subplan 02 Slice 3): write incoming bytes to a staging path, compute SHA-256 (P1.2), place the object at `<root>/objects/<sha256>` via atomic rename, and skip the copy when the object already exists. Storage key = the digest. Interrupted/failed writes must not leave a published object.
- acceptance: A test stores a fixture file, asserts the object exists at `objects/<sha256>` with matching bytes, asserts the returned storage_key equals the digest, and asserts storing identical bytes twice yields one object (dedup) while a staging failure leaves no object. `make test` green.
- **P2.3** In-memory repository + db.json persistence (SQLite stand-in)
- intent: Implement an in-memory repository over the domain (`create`/`get`/`list`/`update` for apps, releases, artifacts, channels, audit events) with create/find-by-slug, find-artifact-by-digest, and an atomic-ish publish transaction wrapper, plus serialize/reload of the whole model to a human-readable `<root>/db.json` via infra/json (subplan 02 Slice 1; the explicit temporary boundary replacing subplan 02 Slice 2's SQLite). Persistence is required so separate CLI invocations share state.
- acceptance: A test creates an app+release+artifacts, writes db.json, reloads into a fresh repository, and asserts the reloaded model equals the original; a test asserts a failed publish transaction leaves no dangling release referenced by a channel. `make test` green and db.json is valid JSON re-parseable by infra/json.
### P3 — `dist` CLI — local content-addressed publish path (the shippable product)
- **P3.1** `dist` entrypoint: parser wiring, help, exit-code + --json contract
- intent: Create the `dist` program under `src/` wiring infra/cli to subcommand handlers (`ci publish`, `release promote`, `release rollback` as stubs for now), with help text and the exit-code contract. Enforce `--json` purity: in JSON mode only machine-readable JSON goes to stdout; human text goes to stderr (subplan 03 Slice 1).
- acceptance: `make build` builds `dist`; running it with no args / an unknown command prints readable help/error and a non-zero exit; running a stub command with `--json` emits only JSON on stdout (verified by piping stdout through infra/json's parser in a test). `make test` green.
- **P3.2** Manifest model + parse + validation
- intent: Define the v0 manifest shape (app slug, version, channel, artifacts[] each with platform, path, optional filename/content-type override, metadata) and parse it via infra/json into a typed struct, validating required fields and that each artifact path exists and platform is known (subplan 03 Manifest Shape). Add `examples/dist.json` plus small fixture APK/IPA byte files under `examples/`.
- acceptance: A test parses `examples/dist.json` into the manifest struct and asserts fields; tests assert that a missing required field, an unknown platform, and a non-existent artifact path each fail with a machine-readable error. `make test` green.
- **P3.3** Common artifact validation subset
- intent: Implement the platform-agnostic validation pass (subplan 05 Common Validation subset): file exists, size matches, SHA-256 matches the stored digest, content type allowed, extension matches platform policy (.apk→android_apk, .ipa→ios, etc.). Result is a validation_status (pending/passed/failed) with reason. Deep IPA/APK zip inspection is explicitly deferred.
- acceptance: A test runs validation on a good fixture (passed) and on tampered cases — wrong size, digest mismatch, .apk declared as ios — each yielding failed with a specific reason. `make test` green.
- **P3.4** `dist ci publish --local-store --json` end-to-end
- intent: Implement the local publish path (subplan 03 Slice 2 + the PLAN CI contract, local mode): validate manifest → create/find app → create draft release → store each artifact content-addressed (P2.2) → run common validation (P3.3) → publish the release → promote the requested channel if specified → write an audit event per upload/publish/promotion → persist db.json (P2.3) → print stable JSON (release id, artifact ids, digests, local download URLs). On any validation failure, abort without leaving a channel pointing at the release.
- acceptance: `make publish-example` runs `dist ci publish --manifest examples/dist.json --local-store <tmp> --json` on a 2-artifact (APK+IPA) fixture, exits 0, and emits JSON whose release id, per-artifact ids, sha256 digests, and local URLs match the store contents; objects exist under `<tmp>/objects/<sha256>`; db.json records the release, artifacts, channel pointer, and audit events; a digest/size-mismatch fixture run exits non-zero with a JSON error and leaves no promoted channel. Captured as a pinned test.
- **P3.5** `dist release promote` / `dist release rollback` over the local store
- intent: Implement standalone channel promotion and rollback against the persisted store (subplan 03; subplan 02 Slice 2 rollback acceptance): `promote` points a channel at a given release (policy gate check stub-allowed for v0), `rollback` moves the channel pointer to the previous valid release for that channel; both append audit events and re-persist db.json. Rollback never points a channel at a non-published/invalid release.
- acceptance: A scripted test: publish release A (channel beta) then release B (channel beta), assert beta→B; run `dist release rollback --app <slug> --channel beta`, assert beta→A and an audit event recorded; assert `dist release promote` of an unknown release id exits non-zero with a JSON error. `make test` green; full suite (`make test`) passes.
### P4 — `distd` over the local store (subplan 04, read-only start; added 2026-06-12)
- **P4.1** `dist server run` — read-only HTTP API + downloads
- intent: Serve the store over HTTP on 0.0.0.0:<port> (default 8787): `/healthz`, `/api/apps`, `/api/apps/<slug>` (app + releases + channels), `/download/<sha256>` (object bytes, X-Checksum-SHA256). Errors reuse the CLI's JSON error shape with matching HTTP statuses. db.json reloads per request so CLI writes are visible immediately. HTTP/1.1 is a TEMPORARY IN-REPO shim over `std.socket` (`src/server/http.sx`, liftable into the sx stdlib later): GET-only, Connection: close, no TLS (reverse proxy terminates), sequential accept loop. KNOWN sx BOUNDARY: stack arrays of 64K+ in one frame crash the sx LLVM backend (DAGCombiner segfault) — response buffers are heap slices from the per-request arena.
- acceptance: `tests/server_http.sx` publishes a fixture, starts `build/dist server run` on a test port, and asserts every route via curl (healthz, apps index, app detail, unknown-slug/route/digest JSON errors, byte-identical download) plus per-request freshness (a publish while the server runs appears on the next request). `make test` green.
- DEFERRED within subplan 04: token auth + write endpoints (Slice 2), streaming uploads (Slice 4 write half), install pages (Slice 5).
- **P4.2** HTML index at `/` — the install-page seed (DONE 2026-06-12, cf39589)
- intent: `GET /` serves a dense server-rendered console page: each app with its channels' current releases and every release's artifacts as direct `/download/<sha256>` links. Read-only off the per-request db.json reload; all dynamic text HTML-escaped. The full per-app install pages (platform detection, iOS install modes) stay in subplan 04 Slice 5; the admin UI stays subplan 06.
- acceptance: `tests/server_http.sx` asserts `/` returns 200 text/html containing the published app's slug, channel→release row, and a working artifact download link. `make test` green (14/14 after the sN→iN lang migration, 6c19f10).
- **P4.3** Token security at rest + `dist token` CLI (subplan 02 Slice 5)
- intent: The auth groundwork the P4.4 write endpoints will consume. Token domain entity (id, name, token_hash, scopes, app_slug + channel scoping, created_at, expires_at, last_used_at, revoked_at) with boundary validation; secret generation (`dist_` + 64 hex chars from 32 `arc4random_buf` bytes — libc FFI, the same thin-platform-backend boundary as publish's `time(2)`); ONLY the SHA-256 of the secret is persisted (`std.hash`); `check_token` enforces revocation, expiry, scope membership, and app/channel scope match; `mark_token_used` records last-use. CLI: `dist token create` (prints the raw secret exactly once; absent db.json starts an empty store, so CI tokens can be minted before the first publish), `dist token list` (status: active/revoked/expired; never the secret), `dist token revoke` (strict: unknown id and double-revoke are distinct errors; aborts before db.save). Every mutation appends an audit event and re-persists db.json; `tokens` joins the persisted arrays (an absent member loads as empty, so older db.json files stay readable).
- acceptance: `tests/token_ops.sx` drives `build/dist`: create → exit 0, JSON carries the secret once, db.json stores sha256(secret) and never the raw secret; list shows both tokens active; revoke flips status + audit event; unknown-id and already-revoked revokes exit 1 with distinct JSON codes and leave db.json unchanged; a publish into the same store still works. `tests/token_check.sx` unit-covers check_token (ok / revoked / expired / missing scope / app mismatch / channel mismatch / unscoped-matches-all) and that a db.json without `tokens` loads as zero tokens. `make test` green.
- DEFERRED: bearer parsing + write endpoints over HTTP (P4.4); remote `dist ci publish --server --token` (subplan 03 Slice 3, needs an HTTP client shim).
- **P4.4** Bearer auth + write endpoints on distd (subplan 04 Slice 2 + the write half of 3/4)
- intent: distd stops being read-only. `src/server/http.sx` learns the request surface writes need: header capture (case-insensitive lookup), a Content-Length-bounded body read loop (8K header cap, hard body cap with 413; 411 when POST omits the length), and the 401/403/411/413 status texts. Auth (`src/server/auth.sx`): `Authorization: Bearer <secret>` is re-hashed (`std.hash`) and resolved via `find_token_by_hash`, then gated through the domain's `check_token` — missing/malformed/unknown credentials are 401, a live token refused for revocation/expiry/scope/app/channel is 403, each with a refusal-specific dotted code; successful auth stamps `mark_token_used` and persists. Reads stay public. Write routes (all POST, all token-gated with scope `publish`): `/api/upload` (raw bytes → content-addressed store, returns the sha256 key, dedup is free), `/api/apps/<slug>/releases` (JSON body naming version/channel + artifacts by sha256 of ALREADY-UPLOADED objects → the same find/create-app → transaction → audit → save pipeline as the CLI, via a `commit_publish` core extracted from publish.sx; unknown object/app/platform aborts before any channel moves), `/api/apps/<slug>/channels/<name>/promote` + `/rollback` (delegate to the P3.5 `run_promote`/`run_rollback`, so CLI and HTTP semantics cannot drift). Server-side artifact checks reuse the P3.3 policy tables (content-type allow-list, filename-extension-vs-platform).
- acceptance: `tests/server_write.sx` publishes+mints tokens via the CLI, starts the server, and asserts over curl: 401 for no/garbage/unknown bearer; 403 with distinct codes for a read-scoped and a wrong-app-scoped token; upload returns the fixture's sha256 and the object lands in the store (re-upload dedups); a release POST referencing the uploaded digest publishes, moves the channel, and is visible on the next GET; promote/rollback move the pointer like their CLI twins; a release POST naming an unknown digest exits 4xx with a JSON error and leaves no channel moved; db.json records `last_used_at` > 0 for the used token. `make test` green.
- DEFERRED: remote `dist ci publish --server --token` (the CLI's HTTP client half, subplan 03 Slice 3); streaming/resumable uploads (bodies are read whole into the per-request arena, capped); install pages (Slice 5).
- **P4.5** Remote `dist ci publish --server --token` (subplan 03 Slice 3 — the PLAN.md CI contract, remote mode)
- intent: The CLI learns to publish against a running distd instead of a local store directory. `src/client/http_client.sx` is the TEMPORARY IN-REPO client shim over `std.socket` (mirroring the server shim): `connect(2)` via libc FFI, `--server` URLs of the form `http://<ipv4-or-localhost>:<port>` (DNS/getaddrinfo and TLS are explicit v0 boundaries — refuse `https://` loudly; the deployment story terminates TLS at a reverse proxy), one POST per call with Bearer/Content-Type/Content-Length, response read to EOF (the server speaks Connection: close) and split into status + body. `src/publish/remote.sx` drives the same manifest front half as local publish (parse + validate + on-disk existence), then per artifact: read bytes → POST `/api/upload` → trust-but-verify the returned digest against a manifest-declared sha256 when present; then POST `/api/apps/<slug>/releases` naming the uploaded digests with the manifest's filename/content-type/metadata. The server's response IS the publish outcome — it is parsed back into a `PublishOutcome` so `--json` and human rendering reuse the exact P3.4a paths, and every non-2xx is surfaced by passing the server's JSON error code/message through `report_failure` (exit 1). `dist ci publish` now takes EITHER `--local-store <dir>` OR `--server <url> --token <secret>`; supplying both or neither is a usage error.
- acceptance: `tests/remote_publish.sx` mints a token in a fresh store, starts `build/dist server run`, and drives `build/dist ci publish --manifest <m> --server http://127.0.0.1:<port> --token <secret> --json`: exit 0 with `status:"published"` JSON naming the release and per-artifact sha256; the SERVER's store contains `objects/<sha>` and a db.json release + channel pointer + `token:<name>` audit actors; re-running the same manifest exits 1 with the server's `transaction.integrity` code; a wrong secret exits 1 with `auth.unknown_token`; an `https://` or malformed `--server` is a loud usage failure; `--local-store` together with `--server` is a usage error. `make test` green.
- DEFERRED: DNS hostnames + TLS in the client shim; retries/resume on flaky links (subplan 03 Slice 3 hardening); install pages (Slice 5).
- **P4.6** Install pages with accurate iOS modes (subplan 04 Slice 5; PLAN.md "iOS Install Policy")
- intent: The human install/download surface, honest about what each platform can do. App gains the iOS policy fields: `ios_mode` (`artifact_only` default / `testflight` / `enterprise`) and `testflight_url`; absent db.json members load as defaults (the token-compat pattern). `dist app set --app <slug> --local-store <dir> [--display-name] [--ios-mode] [--testflight-url] [--ios-bundle-id]` is the admin mutator (apps are created by publish; set requires the app to exist), enforcing the policy preconditions as loud CLI errors — testflight requires a URL, enterprise requires an iOS bundle id — and appending an `app.update` audit event. Server: `GET /install/<slug>/<channel>` renders the channel's current release as platform sections — the requester's platform (User-Agent sniff: iPhone/iPad, Android, Macintosh, Windows, Linux) is ordered first and marked — with per-mode iOS actions: TestFlight link, `itms-services://?action=download-manifest&url=https://<host>/install/<slug>/<channel>/manifest.plist` (https hardcoded — itms-services requires it; the proxy terminates TLS), or an IPA download explicitly labeled as NOT installable from the page. Every artifact row shows size + sha256 (the Android acceptance). `GET /install/<slug>/<channel>/manifest.plist` serves the enterprise OTA manifest XML (software-package URL from the Host header, bundle-identifier from the app's iOS bundle id, bundle-version from the release) and 404s for any other mode or a missing iOS artifact/bundle id. The HTML index links every channel row to its install page.
- acceptance: `tests/server_install.sx` publishes the APK+IPA example, then walks the modes via `dist app set`: default page labels the IPA artifact-only (no install claim); testflight without URL exits 1 (`app.testflight_url_required`), with URL the page links TestFlight; enterprise without bundle id exits 1 (`app.bundle_id_required`), with id the page carries the itms-services link and manifest.plist serves XML naming the bundle id, version, and an https `/download/<sha>` URL; flipping back to testflight 404s the plist; an Android User-Agent gets the android section marked detected, an iPhone UA the ios section; unknown slug/channel are JSON 404s; the index links `/install/acme-app/stable`. `make test` green.
- DEFERRED: QR rendering; access-gated downloads for private apps (visibility exists in the model; pages and downloads stay public in v0); remote-mode `dist app set` (admin writes stay local-store).
### P5 — SQLite persistence (subplan 02 Slice 2; unblocks retiring db.json)
- **P5.1** Vendor SQLite + sx bindings (the foundation)
- intent: Vendor the SQLite amalgamation (3.53.2, public domain) under `vendor/sqlite/` with provenance in its README. `make build` compiles it twice from one source: `build/vendor/libsqlite3.a` (statically linked into `dist` via `-L build/vendor`) and `build/vendor/jit/libsqlite3.dylib` (dlopen'd by `sx run`, which is how the test runner executes tests) — two DIRECTORIES because the macOS linker prefers a dylib over an archive in one search dir. KNOWN sx BOUNDARY: the JIT resolves `#foreign` symbols via `dlsym(RTLD_DEFAULT)`, which searches all images already loaded into the compiler process — the OS libsqlite3 is among them and wins by load order. So the vendored build renames its API surface to `dist_sqlite3_*` (`vendor/sqlite/rename.h`, injected with `-include`), making resolution unambiguous in BOTH modes: those symbols exist only in the vendored products. `src/db/sqlite.sx` binds the renamed surface (open/exec/prepare/bind/step/column/finalize, errmsg, last_insert_rowid, changes, libversion) behind `Sqlite`/`SqliteStmt` wrappers; handles cross the FFI as `usize`, strings read from SQLite are copied into `context.allocator` before sqlite's buffers die.
- acceptance: `tests/sqlite_smoke.sx` asserts `sqlite3_libversion()` equals the VENDORED version (3.53.2 — the OS copy is 3.51.0, so a fallback fails loudly), a typed create/insert/select round trip with text/int64/null bindings, persistence across close+reopen, BEGIN/ROLLBACK atomicity, and errmsg detail on bad SQL. Verified under `sx run` (vendored dylib) AND as an AOT binary (static archive, no libsqlite3 in `otool -L`). `make test` green.
- EXTENDED 2026-06-12 (full mapping): `src/db/sqlite.sx` now binds the complete practical C API (~100 functions, one variant per duplicate family): open_v2 + flags, extended errcodes + error_offset, txn_state/autocommit, changes64, limits, full bind/column families (double/blob/zeroblob), parameter + column introspection (built with `SQLITE_ENABLE_COLUMN_METADATA`), `table_column_metadata`, statement introspection (sql/expanded_sql/readonly/status), incremental blob I/O (`SqliteBlob`), online backup (`SqliteBackup` + `sqlite_backup_run`), serialize/deserialize, and library utilities (complete, strglob/strlike, randomness, memory). NOT bound by design: callback-taking APIs (hooks/UDFs/collations need C→sx callbacks), `sqlite3_value_*` (UDF-coupled), varargs config, UTF-16, and omitted subsystems. `build/vendor/rename.h` is now GENERATED by make from the bindings' `#foreign` names — single source of truth, no drift. `tests/sqlite_api.sx` pins all wrapper families (15 cases). KNOWN sx BOUNDARY (filed): `if !e` on an error binding evaluates wrong (true on a set error) — tests use plain bools for negated error logic.
- **P5.2** Store persistence on SQLite — db.json retired (subplan 02 Slice 2)
- intent: `src/repo/db.sx` persists the whole Repo to `<store>/dist.db` through the P5.1 bindings, keeping the load-whole/save-whole call shape so publish/ops/server code stays storage-agnostic. Schema: one table per entity (apps + app_bundle_ids, releases, artifacts, channels, tokens, audit_events), enums stored as lowercase variant names, list order round-tripped via rowid (save inserts in list order into emptied tables, load reads ORDER BY rowid). Uniqueness the domain guarantees is enforced — apps.slug, channels(app_id, name), tokens.token_hash — plus lookup indexes releases(app_id) and artifacts(sha256) (NON-unique: identical bytes may ship in several releases; audit event ids repeat by design, so audit_events carries no constraints). `save` is DELETE-all + INSERT-all inside BEGIN IMMEDIATE…COMMIT, rolled back on any failure; every connection sets busy_timeout(5000), so the CLI and a running distd interleave safely on one store (db.json never guaranteed that). A store holding only a pre-SQLite db.json imports ONCE on first load, then the file is renamed db.json.imported; a store with neither starts empty (token create still works on a fresh store). Consumers gate on `db.store_exists` instead of probing for db.json; the JSON read-back stays for the import path and the entity→json writers for distd's /api responses. SX BOUNDARY FIXED IN-FLIGHT: the first product chain nesting the bindings two aliased imports deep (dist → ops → db → sqlite) exposed sx issue 0130 — `#library` declared behind ≥2 aliased imports was dropped from the AOT link line and the JIT dlopen list; fixed in sx (extractLibraries/extractFrameworks now recurse over nested namespaces) before this step landed.
- acceptance: `tests/repo_roundtrip.sx` (field-for-field SQLite round-trip + an independent SQL read of dist.db; no db.json written), `tests/db_import.sx` (one-time import: every entity survives, db.json → db.json.imported, writes work after, no re-import, a store with no database refuses with store.load), and the migrated pinned suite — token_ops / release_ops / publish_happy / publish_persist / publish_fail / remote_publish / server_http / server_write assert store state by QUERYING dist.db through the SQLite bindings (no test reads db.json). `make test` green (22/22).

230
current/state.json Normal file
View File

@@ -0,0 +1,230 @@
{
"alias": "distribution",
"plan_branch": "distribution-plan",
"base": "master",
"status": "RUNNING",
"last_head": "20d520d08cb1cba0e8d1968c571179da1bc2feb8",
"current": {
"step": "P3.4a",
"phase": "stepwork",
"attempt": 1
},
"blockers": [],
"acceptance_criteria": [
"A build/test gate exists in the distribution repo (a `Makefile` + `tests/run.sh` that locate the sx binary via `SX ?= /Users/agra/projects/sx/zig-out/bin/sx`): `make test` runs every `tests/**/*.sx` test through the sx compiler, prints per-test ok/FAIL, exits 0 when all pass and non-zero on any failure; an intentionally-failing probe test makes it exit non-zero (gate proven, not vacuous).",
"The product consumes the sx stdlib and makes NO changes under /Users/agra/projects/sx: SHA-256 comes from `std.hash`, JSON from `std.json`, and CLI/flag parsing from `std.cli` (imported via `#import \"modules/std/<name>.sx\"`). There is NO `src/infra/` shim directory; `git -C /Users/agra/projects/sx status` is unaffected by this work.",
"`./dist ci publish --manifest examples/dist.json --local-store <dir> --json` (invoked via the gate, e.g. `make publish-example`) runs end-to-end against a fixture app whose manifest lists at least one Android APK and one iOS IPA artifact, exits 0, and prints a single pure-JSON object on stdout (parseable by `std.json`) containing the release id, one artifact id + sha256 digest per artifact, and a local download URL/path per artifact (no human-only text on stdout under `--json`).",
"Each published artifact's bytes are physically present in the local store under its SHA-256 key (e.g. `<dir>/objects/<sha256>`), the recorded digest equals an independent `shasum -a 256` of the source file AND equals the `std.hash` digest, and re-running publish with byte-identical artifacts does not create a second copy of those bytes (content-addressed dedup).",
"Platform state persists across separate CLI invocations via a human-readable `db.json` (written/read with `std.json`) under the store, recording apps, releases, artifacts, channels, and audit events; a follow-up `./dist release rollback` (and `release promote`) invocation reads that state, moves the channel pointer, appends an audit event, and the change is verifiable by inspecting the store between invocations.",
"Failure paths are loud and machine-readable: a malformed manifest, a missing artifact file, an unknown platform id, or a SHA-256/size mismatch each yields a non-zero exit and a JSON error object under `--json` (error code + message), with no partially-published release left pointed at by a channel.",
"Every step is implemented and verified inside /Users/agra/projects/distribution using the sx compiler; the git working tree builds and `make test` is green at each step boundary; no change is made to the sx compiler or stdlib."
],
"phases": [
{
"phase": "P1",
"title": "Gate + skeleton consuming the sx stdlib (no shims)",
"steps": [
{
"id": "P1.1",
"title": "Skeleton + gate that resolves the std modules",
"status": "merged",
"step_branch": "flow/distribution/P1.1",
"merge_commit": "331a3e06dc91654507739444ce2f8d32c7a22048",
"attempts": 1,
"max_attempts": 4,
"seen_findings": [],
"contested": [],
"note": "prior skeleton/gate merged (331a3e0) is reusable; revised step adds std-submodule import + re-verify once foundation lands"
}
]
},
{
"phase": "P2",
"title": "Domain model + content-addressed store",
"steps": [
{
"id": "P2.1",
"title": "Domain structs + boundary validation",
"status": "merged",
"step_branch": "flow/distribution/P2.1",
"merge_commit": "b552958378b7faaa34af45ec100b43c320281745",
"attempts": 3,
"max_attempts": 4,
"seen_findings": [],
"contested": [],
"worker_session": {
"id": "47cc652a-1cd6-471e-9386-56bd1c5005c2",
"ctx_pct": 51
},
"review_round1": "REQUEST_CHANGES: domain incomplete vs P2 contract (missing Token model, Release published_at vs updated_at, artifact metadata, in-memory repo slice; validator accepts invalid artifact)",
"review_round1_findings": [
"1 validator accepts invalid artifact (in-scope bug)",
"2 Token missing (scope?)",
"3 published_at vs updated_at (scope?)",
"4 artifact metadata missing (scope?)",
"5 in-memory repository (planspec puts in P2.3)"
],
"scope_ruling": "PO RULED: fix validator(1); add published_at(3)+metadata(4); DROP Token(2); DEFER repository to P2.3(5)",
"round2": "reviewer conceded 1/3/4; held 2(Token)+5(repo) procedurally; attempt 3 records slice->step mapping (Token=Slice5, repo=P2.3)"
},
{
"id": "P2.2",
"title": "Content-addressed store (atomic move, dedup) using std.hash",
"status": "merged",
"step_branch": "flow/distribution/P2.2",
"merge_commit": "a2f7ad2a793f507a01000d7bc05a6fb93e16d6d5",
"attempts": 2,
"max_attempts": 4,
"seen_findings": [],
"contested": [],
"worker_session": {
"id": "28344e4a-7f96-48eb-b2ff-d7f4d7b05830",
"ctx_pct": 69
},
"round1": "F1 major: put_file double-reads source (key may not match published bytes) -> fix: hash the staged copy"
},
{
"id": "P2.3",
"title": "In-memory repo + db.json persistence using std.json",
"status": "merged",
"step_branch": "flow/distribution/P2.3",
"merge_commit": "882dce4f6d48cbe29cce30ee9f5f21be22703afd",
"attempts": 3,
"max_attempts": 4,
"seen_findings": [],
"contested": [],
"worker_session": {
"id": "23a1860f-436e-49f2-9b54-6e222df92b1e",
"ctx_pct": 85
},
"round1": "F1 major: publish lacks cross-identity checks (channel/artifact app_id vs release.app_id)",
"round2": "F1 cont: channel-name mismatch (chan.name vs release.channel) - close all aggregate-consistency edges"
}
]
},
{
"phase": "P3",
"title": "dist CLI \u2014 local content-addressed publish path",
"steps": [
{
"id": "P3.1",
"title": "dist entrypoint: std.cli wiring, help, exit-code + --json",
"status": "merged",
"step_branch": "flow/distribution/P3.1",
"merge_commit": "94e241180f91adb92a6aa354b4b58748424744ce",
"attempts": 1,
"max_attempts": 4,
"seen_findings": [],
"contested": [],
"worker_session": {
"id": "7481e5d2-0080-4100-8b89-2899c0f3313c",
"ctx_pct": 47
}
},
{
"id": "P3.2",
"title": "Manifest model + parse + validation using std.json",
"status": "merged",
"step_branch": "flow/distribution/P3.2",
"merge_commit": "1c4bba05f797306c140fcdfd3ad60018f83abb40",
"attempts": 1,
"max_attempts": 4,
"seen_findings": [],
"contested": [],
"worker_session": {
"id": "a2c7afb4-c342-458e-bbd5-7d48296f3fe1",
"ctx_pct": 59
}
},
{
"id": "P3.3",
"title": "Common artifact validation subset using std.hash",
"status": "merged",
"step_branch": "flow/distribution/P3.3",
"merge_commit": "20d520d08cb1cba0e8d1968c571179da1bc2feb8",
"attempts": 1,
"max_attempts": 4,
"seen_findings": [],
"contested": [],
"worker_session": {
"id": "dd86cafa-443d-4d35-a58e-d4d5d3f821c7",
"ctx_pct": 42
}
},
{
"id": "P3.4a",
"title": "publish happy path + persistence (the success pipeline)",
"status": "in_progress",
"attempts": 0,
"max_attempts": 4,
"seen_findings": [],
"contested": [],
"step_branch": "flow/distribution/P3.4a",
"merge_commit": null,
"intent": "Real local publish SUCCESS pipeline replacing the dist.sx ci-publish stub: validate manifest (P3.2) -> find/create app -> draft release -> per artifact [store.put_file (P2.2) + validate_artifact_file (P3.3) + build Artifact] -> repo.publish incl. channel promotion + audit (P2.3) -> db.json (P2.3) -> stable JSON with --json purity (P3.1). Owns the size/sha256 sourcing ruling: add optional declared size/sha256 to the manifest; derive from store/on-disk when absent so examples/dist.json passes unchanged. Wire `make publish-example`.",
"acceptance": "`make publish-example` on the APK+IPA examples/dist.json fixture exits 0; JSON release id / artifact ids / sha256 / local URLs match the store; <dir>/objects/<sha256> exist and re-hash to their keys; db.json records release, artifacts, channel pointer, audit events. Pinned happy-path test that FAILS against the current stub and PASSES after. `make test` green. No sx-repo changes."
},
{
"id": "P3.4b",
"title": "failure-path / abort semantics",
"status": "pending",
"attempts": 0,
"max_attempts": 4,
"seen_findings": [],
"contested": [],
"step_branch": "flow/distribution/P3.4b",
"merge_commit": null,
"intent": "Through the same CLI: a manifest-declared digest/size mismatch (or unknown platform / missing artifact) ABORTS the whole publish \u2014 non-zero exit, machine-readable JSON error on stdout (--json), no promoted channel, no dangling release (the abort-before-commit invariant; repo.publish's internal rollback stays unit-covered by tests/repo_transaction.sx). Add the JSON-error formatter.",
"acceptance": "A digest/size-mismatch fixture run exits non-zero with a JSON error on stdout; reloading the store/db.json shows no channel points at the aborted release and no half-written release graph. Pinned failure-path test that FAILS before this step and PASSES after. `make test` green. No sx-repo changes."
},
{
"id": "P3.5",
"title": "dist release promote / rollback over the local store",
"status": "pending",
"step_branch": "flow/distribution/P3.5",
"merge_commit": null,
"attempts": 0,
"max_attempts": 4,
"seen_findings": [],
"contested": []
}
]
}
],
"updated": "2026-06-06T02:02:52.215395Z",
"notes": "FOUNDATION DONE (sx master e466bd5: std.hash/json/cli landed). P1.2 in-repo shim retired (superseded by std.hash). Revised planspec adopted; driving from P2.1. Gate: make build && make test, SX=/Users/agra/projects/sx/zig-out/bin/sx.",
"blocker_fix": {
"step": "fix-0100",
"repo": "/Users/agra/projects/sx",
"branch": "flow/distribution/fix-0100",
"base": "master",
"status": "RESOLVED (merged sx master 52310b6, gate 456); binary rebuilt; blocker cleared"
},
"scheduled_followups": [
{
"id": "0098",
"repo": "sx",
"title": "enum literal in non-enum target silently lowers to 0 (resolveVariantValue silent-zero)",
"disposition": "orthogonal+pre-existing (manager-adjudicated, ground-truth bisected); NOT a distribution blocker; drive as separate sx fix",
"draft": "runs/distribution/issue-0098-enum-literal-non-enum-target-silent-zero.md"
}
],
"side_fixes": [
{
"id": "0099",
"title": "LSP analyzer panic on identifier array dim + LSP test-suite coverage",
"repo": "sx",
"branch": "flow/distribution/fix-0099-lsp",
"status": "RESOLVED (merged bef2c66); sweep 0/470 on master",
"requested_by": "Agra (live LSP crash + \"LSP must be in the test suite\")"
}
],
"lsp_plan": {
"doc": "runs/distribution/LSP-PLAN.md",
"mode": "A then B, systematic; B-discovered crashes appended + fixed one at a time",
"A": "MERGED to master (bef2c66); 0099 RESOLVED; sweep 0/470",
"B": "worker done (commit 503dfd8); zig build test sweeps 470 examples + 1 issue, 397/397 tests; reviewer verifying",
"discovery": "18/470 examples crash LSP, ALL at sema.zig:367 (one root). A fixes it; sweep verifies 0 crashes.",
"status": "COMPLETE \u2014 A (0099 fix) + B (corpus sweep) merged to master (fb3fdaf); zig build test sweeps 470 examples + issues, 0 crashes"
}
}