docs: update CLAUDE.md testing workflow to the new test layout
Reflect the migrated layout: XXXX-category-test-name naming with per-category
100-blocks; expected output in an expected/ dir next to each test, split into
.exit/.stdout/.stderr (+ optional .ir); runner scans examples/ and issues/.
Replace the old 50-smoke / tests/expected / examples/issue-* workflow with:
add a feature as a focused example, file open bugs as issues/NNNN-slug.{md,sx}
co-located, and resolve an issue by moving its repro into examples/ as a
regression test + marking the .md RESOLVED. Update stale test count (29 -> 324).
This commit is contained in:
75
CLAUDE.md
75
CLAUDE.md
@@ -416,9 +416,33 @@ After any compiler change:
|
||||
|
||||
1. **Build**: `zig build && zig build test`
|
||||
2. **Run regression tests**: `bash tests/run_examples.sh`
|
||||
- All 29 tests must show `ok`
|
||||
- Every test must show `ok` (currently 324)
|
||||
- Zero failures, zero timeouts
|
||||
|
||||
### Test layout
|
||||
|
||||
Examples and pinned issue repros use the `XXXX-category-test-name` scheme — a
|
||||
4-digit number in per-category 100-blocks: `basic` 00xx, `types` 01xx, `generics`
|
||||
02xx, `closures` 03xx, `protocols` 04xx, `packs` 05xx, `comptime` 06xx, `modules`
|
||||
07xx, `memory` 08xx, `optionals` 09xx, `errors` 10xx, `diagnostics` 11xx, `ffi`
|
||||
12xx, `ffi-objc` 13xx, `ffi-jni` 14xx, `vectors` 15xx, `platform` 16xx.
|
||||
|
||||
Expected output lives in an `expected/` directory **next to the test file**,
|
||||
split into three streams (no more merged `2>&1`) plus an optional IR snapshot:
|
||||
|
||||
```
|
||||
<root>/XXXX-category-name.sx
|
||||
<root>/expected/XXXX-category-name.exit # process exit code
|
||||
<root>/expected/XXXX-category-name.stdout # normalized stdout
|
||||
<root>/expected/XXXX-category-name.stderr # normalized stderr
|
||||
<root>/expected/XXXX-category-name.ir # optional `sx ir` snapshot
|
||||
```
|
||||
|
||||
A test is any `<name>.sx` with an `expected/<name>.exit` marker. The runner
|
||||
scans two roots: `examples/` (the feature suite) and `issues/` (pinned bug
|
||||
repros). Multi-file tests keep companions (`.c`/`.h`, imported `.sx`, fixture
|
||||
dirs) under the same `XXXX-` prefix.
|
||||
|
||||
### Snapshot integrity
|
||||
|
||||
**Never run `--update` while tests are failing.** The `--update` flag blindly overwrites expected output with whatever the compiler produces — including error messages. If you update snapshots during a broken state, the test suite will "pass" against garbage output and real regressions become invisible.
|
||||
@@ -426,27 +450,30 @@ After any compiler change:
|
||||
Safe workflow:
|
||||
1. Fix the code until `bash tests/run_examples.sh` passes against the **existing** snapshots.
|
||||
2. Only run `--update` when you've intentionally changed output (new feature, new test, changed formatting).
|
||||
3. After `--update`, review the diff (`git diff tests/expected/`) to confirm no error messages or empty output were captured.
|
||||
3. After `--update`, review the diff (`git diff examples/expected/ issues/expected/`) to confirm no error messages or empty output were captured.
|
||||
|
||||
### Adding a new language feature
|
||||
|
||||
When implementing a new feature:
|
||||
There is no monolithic smoke file — each feature is its own focused example.
|
||||
|
||||
1. Add test case(s) to `examples/50-smoke.sx` in the appropriate section
|
||||
2. Run `./zig-out/bin/sx run examples/50-smoke.sx` to verify it works
|
||||
3. Regenerate expected output: `bash tests/run_examples.sh --update`
|
||||
1. Create `examples/XXXX-<category>-<name>.sx` (next free number in the matching
|
||||
category block).
|
||||
2. Run it: `./zig-out/bin/sx run examples/XXXX-<category>-<name>.sx`
|
||||
3. Seed the marker and capture expected output:
|
||||
`: > examples/expected/XXXX-<category>-<name>.exit` then
|
||||
`bash tests/run_examples.sh --update`
|
||||
4. Verify all tests still pass: `bash tests/run_examples.sh`
|
||||
|
||||
### Test file roles
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `examples/50-smoke.sx` | Comprehensive feature coverage (~200 tests). Add new features here. |
|
||||
| `examples/NN-name.sx` | Focused feature examples (e.g. `01-basic.sx`, `52-frameworks.sx`). Created either fresh or by renaming a resolved `issue-*.sx` once its bug is fixed. |
|
||||
| `examples/issue-NNNN.sx` | **Open bug repros.** Each file is a literal item on the open-issue list, named after the issue number. When the bug is fixed, rename the file (and its `tests/expected/issue-NNNN.{txt,exit}`) to a focused feature example so the `issue-*` namespace shrinks to exactly the unresolved set. A file without a matching `tests/expected/issue-NNNN.txt` is an open issue that hasn't been pinned in the test suite yet. |
|
||||
| `tests/expected/*.txt` | Expected output per example. Regenerate with `--update`. |
|
||||
| `tests/expected/*.exit` | Expected exit codes. Auto-generated with `--update`. |
|
||||
| `tests/run_examples.sh` | Test runner. Compares actual vs expected output. |
|
||||
| `examples/XXXX-category-name.sx` | Focused feature example — one feature per file. |
|
||||
| `examples/expected/XXXX-category-name.{exit,stdout,stderr}` | Expected exit code + the two output streams. Regenerate with `--update`. |
|
||||
| `examples/expected/XXXX-category-name.ir` | Optional `sx ir` snapshot — present only where lowering shape is locked. |
|
||||
| `issues/NNNN-slug.md` | Open-issue / bug-report writeup (mark RESOLVED in a banner when fixed; the `.md` stays). |
|
||||
| `issues/NNNN-slug.sx` (+ `issues/NNNN-slug/`) | The issue's minimal repro, co-located with the `.md`. A repro with an `issues/expected/NNNN-slug.exit` marker runs in the suite; unpinned ones don't. |
|
||||
| `tests/run_examples.sh` | Test runner. Scans `examples/` and `issues/`; compares stdout/stderr/exit (+ optional IR) per test. |
|
||||
|
||||
### Unit test file convention
|
||||
|
||||
@@ -465,22 +492,28 @@ All Zig unit tests live in separate `*.test.zig` files alongside the source they
|
||||
|
||||
### Creating a new standalone test
|
||||
|
||||
1. Create `examples/NN-name.sx` (focused feature example) **or** `examples/issue-NNNN.sx` (open-bug repro).
|
||||
2. Run it: `./zig-out/bin/sx run examples/<file>.sx`
|
||||
3. Create expected output: `bash tests/run_examples.sh --update`
|
||||
1. Create `examples/XXXX-<category>-<name>.sx` (focused example) **or**, for an
|
||||
open bug, `issues/NNNN-slug.{md,sx}` (repro co-located with the writeup).
|
||||
2. Run it: `./zig-out/bin/sx run <path>.sx`
|
||||
3. Seed the marker (`: > <root>/expected/<name>.exit`) and capture expected:
|
||||
`bash tests/run_examples.sh --update`
|
||||
4. Verify: `bash tests/run_examples.sh`
|
||||
|
||||
### Resolving an open issue
|
||||
|
||||
When a bug filed as `examples/issue-NNNN.sx` is fixed, the file should leave the issue-namespace:
|
||||
When a bug filed under `issues/NNNN-slug.{md,sx}` is fixed:
|
||||
|
||||
1. Pick a focused feature name with the next free number, e.g. `examples/52-frameworks.sx`.
|
||||
2. `git mv examples/issue-NNNN.sx examples/NN-name.sx`.
|
||||
3. `git mv tests/expected/issue-NNNN.txt tests/expected/NN-name.txt` (and the `.exit` file).
|
||||
4. Tighten the comment header to describe the feature (drop the issue-NNNN provenance — that lives in git history now).
|
||||
1. Move the repro into the feature suite as a regression test:
|
||||
`git mv issues/NNNN-slug.sx examples/XXXX-<category>-<name>.sx`.
|
||||
2. Seed `examples/expected/XXXX-<category>-<name>.exit`, capture with `--update`,
|
||||
and review the diff.
|
||||
3. Tighten the example's comment header to describe the feature (keep a one-line
|
||||
`Regression (issue NNNN)` note for provenance).
|
||||
4. Mark `issues/NNNN-slug.md` RESOLVED with a short banner (root cause + fix +
|
||||
regression-test path). The `.md` stays as the writeup.
|
||||
5. Run the suite to confirm nothing else broke.
|
||||
|
||||
The `examples/issue-*` glob after step 5 is the literal list of what's still open.
|
||||
The set of `issues/*.md` without a RESOLVED banner is the open-issue list.
|
||||
|
||||
## Known bugs
|
||||
|
||||
|
||||
Reference in New Issue
Block a user