docs: update CLAUDE.md testing workflow to the new test layout

Reflect the migrated layout: XXXX-category-test-name naming with per-category
100-blocks; expected output in an expected/ dir next to each test, split into
.exit/.stdout/.stderr (+ optional .ir); runner scans examples/ and issues/.
Replace the old 50-smoke / tests/expected / examples/issue-* workflow with:
add a feature as a focused example, file open bugs as issues/NNNN-slug.{md,sx}
co-located, and resolve an issue by moving its repro into examples/ as a
regression test + marking the .md RESOLVED. Update stale test count (29 -> 324).
This commit is contained in:
agra
2026-06-01 19:40:39 +03:00
parent 34819f05de
commit 9e660f30c2

View File

@@ -416,9 +416,33 @@ After any compiler change:
1. **Build**: `zig build && zig build test`
2. **Run regression tests**: `bash tests/run_examples.sh`
- All 29 tests must show `ok`
- Every test must show `ok` (currently 324)
- Zero failures, zero timeouts
### Test layout
Examples and pinned issue repros use the `XXXX-category-test-name` scheme — a
4-digit number in per-category 100-blocks: `basic` 00xx, `types` 01xx, `generics`
02xx, `closures` 03xx, `protocols` 04xx, `packs` 05xx, `comptime` 06xx, `modules`
07xx, `memory` 08xx, `optionals` 09xx, `errors` 10xx, `diagnostics` 11xx, `ffi`
12xx, `ffi-objc` 13xx, `ffi-jni` 14xx, `vectors` 15xx, `platform` 16xx.
Expected output lives in an `expected/` directory **next to the test file**,
split into three streams (no more merged `2>&1`) plus an optional IR snapshot:
```
<root>/XXXX-category-name.sx
<root>/expected/XXXX-category-name.exit # process exit code
<root>/expected/XXXX-category-name.stdout # normalized stdout
<root>/expected/XXXX-category-name.stderr # normalized stderr
<root>/expected/XXXX-category-name.ir # optional `sx ir` snapshot
```
A test is any `<name>.sx` with an `expected/<name>.exit` marker. The runner
scans two roots: `examples/` (the feature suite) and `issues/` (pinned bug
repros). Multi-file tests keep companions (`.c`/`.h`, imported `.sx`, fixture
dirs) under the same `XXXX-` prefix.
### Snapshot integrity
**Never run `--update` while tests are failing.** The `--update` flag blindly overwrites expected output with whatever the compiler produces — including error messages. If you update snapshots during a broken state, the test suite will "pass" against garbage output and real regressions become invisible.
@@ -426,27 +450,30 @@ After any compiler change:
Safe workflow:
1. Fix the code until `bash tests/run_examples.sh` passes against the **existing** snapshots.
2. Only run `--update` when you've intentionally changed output (new feature, new test, changed formatting).
3. After `--update`, review the diff (`git diff tests/expected/`) to confirm no error messages or empty output were captured.
3. After `--update`, review the diff (`git diff examples/expected/ issues/expected/`) to confirm no error messages or empty output were captured.
### Adding a new language feature
When implementing a new feature:
There is no monolithic smoke file — each feature is its own focused example.
1. Add test case(s) to `examples/50-smoke.sx` in the appropriate section
2. Run `./zig-out/bin/sx run examples/50-smoke.sx` to verify it works
3. Regenerate expected output: `bash tests/run_examples.sh --update`
1. Create `examples/XXXX-<category>-<name>.sx` (next free number in the matching
category block).
2. Run it: `./zig-out/bin/sx run examples/XXXX-<category>-<name>.sx`
3. Seed the marker and capture expected output:
`: > examples/expected/XXXX-<category>-<name>.exit` then
`bash tests/run_examples.sh --update`
4. Verify all tests still pass: `bash tests/run_examples.sh`
### Test file roles
| File | Purpose |
|------|---------|
| `examples/50-smoke.sx` | Comprehensive feature coverage (~200 tests). Add new features here. |
| `examples/NN-name.sx` | Focused feature examples (e.g. `01-basic.sx`, `52-frameworks.sx`). Created either fresh or by renaming a resolved `issue-*.sx` once its bug is fixed. |
| `examples/issue-NNNN.sx` | **Open bug repros.** Each file is a literal item on the open-issue list, named after the issue number. When the bug is fixed, rename the file (and its `tests/expected/issue-NNNN.{txt,exit}`) to a focused feature example so the `issue-*` namespace shrinks to exactly the unresolved set. A file without a matching `tests/expected/issue-NNNN.txt` is an open issue that hasn't been pinned in the test suite yet. |
| `tests/expected/*.txt` | Expected output per example. Regenerate with `--update`. |
| `tests/expected/*.exit` | Expected exit codes. Auto-generated with `--update`. |
| `tests/run_examples.sh` | Test runner. Compares actual vs expected output. |
| `examples/XXXX-category-name.sx` | Focused feature example — one feature per file. |
| `examples/expected/XXXX-category-name.{exit,stdout,stderr}` | Expected exit code + the two output streams. Regenerate with `--update`. |
| `examples/expected/XXXX-category-name.ir` | Optional `sx ir` snapshot — present only where lowering shape is locked. |
| `issues/NNNN-slug.md` | Open-issue / bug-report writeup (mark RESOLVED in a banner when fixed; the `.md` stays). |
| `issues/NNNN-slug.sx` (+ `issues/NNNN-slug/`) | The issue's minimal repro, co-located with the `.md`. A repro with an `issues/expected/NNNN-slug.exit` marker runs in the suite; unpinned ones don't. |
| `tests/run_examples.sh` | Test runner. Scans `examples/` and `issues/`; compares stdout/stderr/exit (+ optional IR) per test. |
### Unit test file convention
@@ -465,22 +492,28 @@ All Zig unit tests live in separate `*.test.zig` files alongside the source they
### Creating a new standalone test
1. Create `examples/NN-name.sx` (focused feature example) **or** `examples/issue-NNNN.sx` (open-bug repro).
2. Run it: `./zig-out/bin/sx run examples/<file>.sx`
3. Create expected output: `bash tests/run_examples.sh --update`
1. Create `examples/XXXX-<category>-<name>.sx` (focused example) **or**, for an
open bug, `issues/NNNN-slug.{md,sx}` (repro co-located with the writeup).
2. Run it: `./zig-out/bin/sx run <path>.sx`
3. Seed the marker (`: > <root>/expected/<name>.exit`) and capture expected:
`bash tests/run_examples.sh --update`
4. Verify: `bash tests/run_examples.sh`
### Resolving an open issue
When a bug filed as `examples/issue-NNNN.sx` is fixed, the file should leave the issue-namespace:
When a bug filed under `issues/NNNN-slug.{md,sx}` is fixed:
1. Pick a focused feature name with the next free number, e.g. `examples/52-frameworks.sx`.
2. `git mv examples/issue-NNNN.sx examples/NN-name.sx`.
3. `git mv tests/expected/issue-NNNN.txt tests/expected/NN-name.txt` (and the `.exit` file).
4. Tighten the comment header to describe the feature (drop the issue-NNNN provenance — that lives in git history now).
1. Move the repro into the feature suite as a regression test:
`git mv issues/NNNN-slug.sx examples/XXXX-<category>-<name>.sx`.
2. Seed `examples/expected/XXXX-<category>-<name>.exit`, capture with `--update`,
and review the diff.
3. Tighten the example's comment header to describe the feature (keep a one-line
`Regression (issue NNNN)` note for provenance).
4. Mark `issues/NNNN-slug.md` RESOLVED with a short banner (root cause + fix +
regression-test path). The `.md` stays as the writeup.
5. Run the suite to confirm nothing else broke.
The `examples/issue-*` glob after step 5 is the literal list of what's still open.
The set of `issues/*.md` without a RESOLVED banner is the open-issue list.
## Known bugs