test: group examples into per-category folders
Move examples/*.sx and their expected/ snapshots into per-category subfolders (examples/<category>/...). Folder = leading filename token, with ffi-objc/ffi-jni kept whole; filenames are unchanged. The corpus runner and LSP sweep now discover each category's expected/ dir, while issues/ stays flat. Example 1058's repo-root-relative companion import is made file-relative. Path strings embedded in 164 snapshots were regenerated (path-only changes). Test-layout docs in CLAUDE.md updated.
This commit is contained in:
71
CLAUDE.md
71
CLAUDE.md
@@ -452,24 +452,33 @@ Examples and pinned issue repros use the `XXXX-category-test-name` scheme — a
|
||||
4-digit number in per-category 100-blocks: `basic` 00xx, `types` 01xx, `generics`
|
||||
02xx, `closures` 03xx, `protocols` 04xx, `packs` 05xx, `comptime` 06xx, `modules`
|
||||
07xx, `memory` 08xx, `optionals` 09xx, `errors` 10xx, `diagnostics` 11xx, `ffi`
|
||||
12xx, `ffi-objc` 13xx, `ffi-jni` 14xx, `vectors` 15xx, `platform` 16xx.
|
||||
12xx, `ffi-objc` 13xx, `ffi-jni` 14xx, `vectors` 15xx, `platform` 16xx. (Newer
|
||||
categories have grown past 16xx — `atomics` 17xx, `concurrency` 18xx — and some
|
||||
share 16xx; the **category is the leading name token**, not the number block.)
|
||||
|
||||
Expected output lives in an `expected/` directory **next to the test file**,
|
||||
split into three streams (no more merged `2>&1`) plus an optional IR snapshot:
|
||||
`examples/` is organized into **per-category subfolders** — the folder name is
|
||||
the leading token of the filename (`ffi-objc`/`ffi-jni` kept whole). The full
|
||||
`XXXX-category-...` filename is unchanged; the folder just groups it. Each
|
||||
category folder has its own `expected/` directory holding the snapshots, split
|
||||
into three streams (no more merged `2>&1`) plus an optional IR snapshot:
|
||||
|
||||
```
|
||||
<root>/XXXX-category-name.sx
|
||||
<root>/expected/XXXX-category-name.exit # process exit code
|
||||
<root>/expected/XXXX-category-name.stdout # normalized stdout
|
||||
<root>/expected/XXXX-category-name.stderr # normalized stderr
|
||||
<root>/expected/XXXX-category-name.ir # optional `sx ir` snapshot
|
||||
<root>/expected/XXXX-category-name.build # optional JSON build/run directives
|
||||
examples/<category>/XXXX-category-name.sx
|
||||
examples/<category>/expected/XXXX-category-name.exit # process exit code
|
||||
examples/<category>/expected/XXXX-category-name.stdout # normalized stdout
|
||||
examples/<category>/expected/XXXX-category-name.stderr # normalized stderr
|
||||
examples/<category>/expected/XXXX-category-name.ir # optional `sx ir` snapshot
|
||||
examples/<category>/expected/XXXX-category-name.build # optional JSON build/run directives
|
||||
```
|
||||
|
||||
A test is any `<name>.sx` with an `expected/<name>.exit` marker. The runner
|
||||
scans two roots: `examples/` (the feature suite) and `issues/` (pinned bug
|
||||
repros). Multi-file tests keep companions (`.c`/`.h`, imported `.sx`, fixture
|
||||
dirs) under the same `XXXX-` prefix.
|
||||
`issues/` stays **flat** (`issues/<name>.sx` + `issues/expected/<name>.exit`).
|
||||
A test is any `<name>.sx` with a sibling `expected/<name>.exit` marker. The
|
||||
runner scans two roots — `examples/` (the feature suite, recursing one level
|
||||
into category folders) and `issues/` (pinned bug repros) — discovering every
|
||||
`expected/` directory under each. Multi-file tests keep companions (`.c`/`.h`,
|
||||
imported `.sx`, fixture dirs) under the same `XXXX-` prefix **in the same
|
||||
category folder**, and reference them with file-relative imports (e.g.
|
||||
`#import "XXXX-foo/lib.sx"`), never a repo-root-relative `examples/...` path.
|
||||
|
||||
The optional `<name>.build` JSON sidecar carries per-example directives
|
||||
(unknown keys are a hard error — never silently ignored):
|
||||
@@ -498,18 +507,18 @@ The optional `<name>.build` JSON sidecar carries per-example directives
|
||||
Safe workflow:
|
||||
1. Fix the code until `zig build test` passes against the **existing** snapshots.
|
||||
2. Only run `zig build test -Dupdate-goldens` when you've intentionally changed output (new feature, new test, changed formatting).
|
||||
3. After regenerating, review the diff (`git diff examples/expected/ issues/expected/`) to confirm no error messages or empty output were captured.
|
||||
3. After regenerating, review the diff (`git diff examples/ issues/expected/`) to confirm no error messages or empty output were captured.
|
||||
|
||||
**Scope a regen to specific examples with `-Dname`.** A *full* `-Dupdate-goldens`
|
||||
re-runs and rewrites all ~690 snapshots, so a single flaky/host-divergent example
|
||||
(AOT links, cross-arch `target` examples, anything that intermittently fails) can
|
||||
silently clobber a good snapshot. To capture just the example(s) you added, pass
|
||||
their full repo-relative `.sx` path(s), comma-separated — this rewrites ONLY those
|
||||
and touches nothing else:
|
||||
their full repo-relative `.sx` path(s) — now including the category folder —
|
||||
comma-separated; this rewrites ONLY those and touches nothing else:
|
||||
|
||||
```sh
|
||||
zig build test -Dname=examples/0625-comptime-weld-struct-field.sx -Dupdate-goldens
|
||||
zig build test -Dname=examples/0625-foo.sx,examples/0626-bar.sx # verify just these
|
||||
zig build test -Dname=examples/comptime/0625-comptime-weld-struct-field.sx -Dupdate-goldens
|
||||
zig build test -Dname=examples/comptime/0625-foo.sx,examples/types/0126-bar.sx # verify just these
|
||||
```
|
||||
|
||||
`-Dname` also busts the test-run cache (the corpus enumerates `.sx`/`expected/`
|
||||
@@ -521,11 +530,11 @@ compiler source — forces a fresh run.
|
||||
|
||||
There is no monolithic smoke file — each feature is its own focused example.
|
||||
|
||||
1. Create `examples/XXXX-<category>-<name>.sx` (next free number in the matching
|
||||
category block).
|
||||
2. Run it: `./zig-out/bin/sx run examples/XXXX-<category>-<name>.sx`
|
||||
1. Create `examples/<category>/XXXX-<category>-<name>.sx` (next free number in
|
||||
the matching category block, in that category's folder).
|
||||
2. Run it: `./zig-out/bin/sx run examples/<category>/XXXX-<category>-<name>.sx`
|
||||
3. Seed the marker and capture expected output:
|
||||
`: > examples/expected/XXXX-<category>-<name>.exit` then
|
||||
`: > examples/<category>/expected/XXXX-<category>-<name>.exit` then
|
||||
`zig build test -Dupdate-goldens`
|
||||
4. Verify all tests still pass: `zig build test`
|
||||
|
||||
@@ -533,9 +542,9 @@ There is no monolithic smoke file — each feature is its own focused example.
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `examples/XXXX-category-name.sx` | Focused feature example — one feature per file. |
|
||||
| `examples/expected/XXXX-category-name.{exit,stdout,stderr}` | Expected exit code + the two output streams. Regenerate with `zig build test -Dupdate-goldens`. |
|
||||
| `examples/expected/XXXX-category-name.ir` | Optional `sx ir` snapshot — present only where lowering shape is locked. |
|
||||
| `examples/<category>/XXXX-category-name.sx` | Focused feature example — one feature per file, in its category folder. |
|
||||
| `examples/<category>/expected/XXXX-category-name.{exit,stdout,stderr}` | Expected exit code + the two output streams. Regenerate with `zig build test -Dupdate-goldens`. |
|
||||
| `examples/<category>/expected/XXXX-category-name.ir` | Optional `sx ir` snapshot — present only where lowering shape is locked. |
|
||||
| `issues/NNNN-slug.md` | Open-issue / bug-report writeup (mark RESOLVED in a banner when fixed; the `.md` stays). |
|
||||
| `issues/NNNN-slug.sx` (+ `issues/NNNN-slug/`) | The issue's minimal repro, co-located with the `.md`. A repro with an `issues/expected/NNNN-slug.exit` marker runs in the suite; unpinned ones don't. |
|
||||
| `src/corpus_run.test.zig` | The corpus runner inside `zig build test` — spawns `sx` per example, diffs stdout/stderr/exit (+ optional IR); regenerates snapshots under `-Dupdate-goldens`. |
|
||||
@@ -557,10 +566,12 @@ All Zig unit tests live in separate `*.test.zig` files alongside the source they
|
||||
|
||||
### Creating a new standalone test
|
||||
|
||||
1. Create `examples/XXXX-<category>-<name>.sx` (focused example) **or**, for an
|
||||
open bug, `issues/NNNN-slug.{md,sx}` (repro co-located with the writeup).
|
||||
1. Create `examples/<category>/XXXX-<category>-<name>.sx` (focused example)
|
||||
**or**, for an open bug, `issues/NNNN-slug.{md,sx}` (repro co-located with
|
||||
the writeup).
|
||||
2. Run it: `./zig-out/bin/sx run <path>.sx`
|
||||
3. Seed the marker (`: > <root>/expected/<name>.exit`) and capture expected:
|
||||
3. Seed the marker (`: > <dir>/expected/<name>.exit`, where `<dir>` is the
|
||||
example's category folder or `issues/`) and capture expected:
|
||||
`zig build test -Dupdate-goldens`
|
||||
4. Verify: `zig build test`
|
||||
|
||||
@@ -569,8 +580,8 @@ All Zig unit tests live in separate `*.test.zig` files alongside the source they
|
||||
When a bug filed under `issues/NNNN-slug.{md,sx}` is fixed:
|
||||
|
||||
1. Move the repro into the feature suite as a regression test:
|
||||
`git mv issues/NNNN-slug.sx examples/XXXX-<category>-<name>.sx`.
|
||||
2. Seed `examples/expected/XXXX-<category>-<name>.exit`, capture with
|
||||
`git mv issues/NNNN-slug.sx examples/<category>/XXXX-<category>-<name>.sx`.
|
||||
2. Seed `examples/<category>/expected/XXXX-<category>-<name>.exit`, capture with
|
||||
`zig build test -Dupdate-goldens`, and review the diff.
|
||||
3. Tighten the example's comment header to describe the feature (keep a one-line
|
||||
`Regression (issue NNNN)` note for provenance).
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user