From 9e660f30c2a9b40a4d20271afc74b0e32c61ef1e Mon Sep 17 00:00:00 2001 From: agra Date: Mon, 1 Jun 2026 19:40:39 +0300 Subject: [PATCH] docs: update CLAUDE.md testing workflow to the new test layout Reflect the migrated layout: XXXX-category-test-name naming with per-category 100-blocks; expected output in an expected/ dir next to each test, split into .exit/.stdout/.stderr (+ optional .ir); runner scans examples/ and issues/. Replace the old 50-smoke / tests/expected / examples/issue-* workflow with: add a feature as a focused example, file open bugs as issues/NNNN-slug.{md,sx} co-located, and resolve an issue by moving its repro into examples/ as a regression test + marking the .md RESOLVED. Update stale test count (29 -> 324). --- CLAUDE.md | 75 +++++++++++++++++++++++++++++++++++++++---------------- 1 file changed, 54 insertions(+), 21 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 781a297..fc56b82 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -416,9 +416,33 @@ After any compiler change: 1. **Build**: `zig build && zig build test` 2. **Run regression tests**: `bash tests/run_examples.sh` - - All 29 tests must show `ok` + - Every test must show `ok` (currently 324) - Zero failures, zero timeouts +### Test layout + +Examples and pinned issue repros use the `XXXX-category-test-name` scheme — a +4-digit number in per-category 100-blocks: `basic` 00xx, `types` 01xx, `generics` +02xx, `closures` 03xx, `protocols` 04xx, `packs` 05xx, `comptime` 06xx, `modules` +07xx, `memory` 08xx, `optionals` 09xx, `errors` 10xx, `diagnostics` 11xx, `ffi` +12xx, `ffi-objc` 13xx, `ffi-jni` 14xx, `vectors` 15xx, `platform` 16xx. + +Expected output lives in an `expected/` directory **next to the test file**, +split into three streams (no more merged `2>&1`) plus an optional IR snapshot: + +``` +/XXXX-category-name.sx +/expected/XXXX-category-name.exit # process exit code +/expected/XXXX-category-name.stdout # normalized stdout +/expected/XXXX-category-name.stderr # normalized stderr +/expected/XXXX-category-name.ir # optional `sx ir` snapshot +``` + +A test is any `.sx` with an `expected/.exit` marker. The runner +scans two roots: `examples/` (the feature suite) and `issues/` (pinned bug +repros). Multi-file tests keep companions (`.c`/`.h`, imported `.sx`, fixture +dirs) under the same `XXXX-` prefix. + ### Snapshot integrity **Never run `--update` while tests are failing.** The `--update` flag blindly overwrites expected output with whatever the compiler produces — including error messages. If you update snapshots during a broken state, the test suite will "pass" against garbage output and real regressions become invisible. @@ -426,27 +450,30 @@ After any compiler change: Safe workflow: 1. Fix the code until `bash tests/run_examples.sh` passes against the **existing** snapshots. 2. Only run `--update` when you've intentionally changed output (new feature, new test, changed formatting). -3. After `--update`, review the diff (`git diff tests/expected/`) to confirm no error messages or empty output were captured. +3. After `--update`, review the diff (`git diff examples/expected/ issues/expected/`) to confirm no error messages or empty output were captured. ### Adding a new language feature -When implementing a new feature: +There is no monolithic smoke file — each feature is its own focused example. -1. Add test case(s) to `examples/50-smoke.sx` in the appropriate section -2. Run `./zig-out/bin/sx run examples/50-smoke.sx` to verify it works -3. Regenerate expected output: `bash tests/run_examples.sh --update` +1. Create `examples/XXXX--.sx` (next free number in the matching + category block). +2. Run it: `./zig-out/bin/sx run examples/XXXX--.sx` +3. Seed the marker and capture expected output: + `: > examples/expected/XXXX--.exit` then + `bash tests/run_examples.sh --update` 4. Verify all tests still pass: `bash tests/run_examples.sh` ### Test file roles | File | Purpose | |------|---------| -| `examples/50-smoke.sx` | Comprehensive feature coverage (~200 tests). Add new features here. | -| `examples/NN-name.sx` | Focused feature examples (e.g. `01-basic.sx`, `52-frameworks.sx`). Created either fresh or by renaming a resolved `issue-*.sx` once its bug is fixed. | -| `examples/issue-NNNN.sx` | **Open bug repros.** Each file is a literal item on the open-issue list, named after the issue number. When the bug is fixed, rename the file (and its `tests/expected/issue-NNNN.{txt,exit}`) to a focused feature example so the `issue-*` namespace shrinks to exactly the unresolved set. A file without a matching `tests/expected/issue-NNNN.txt` is an open issue that hasn't been pinned in the test suite yet. | -| `tests/expected/*.txt` | Expected output per example. Regenerate with `--update`. | -| `tests/expected/*.exit` | Expected exit codes. Auto-generated with `--update`. | -| `tests/run_examples.sh` | Test runner. Compares actual vs expected output. | +| `examples/XXXX-category-name.sx` | Focused feature example — one feature per file. | +| `examples/expected/XXXX-category-name.{exit,stdout,stderr}` | Expected exit code + the two output streams. Regenerate with `--update`. | +| `examples/expected/XXXX-category-name.ir` | Optional `sx ir` snapshot — present only where lowering shape is locked. | +| `issues/NNNN-slug.md` | Open-issue / bug-report writeup (mark RESOLVED in a banner when fixed; the `.md` stays). | +| `issues/NNNN-slug.sx` (+ `issues/NNNN-slug/`) | The issue's minimal repro, co-located with the `.md`. A repro with an `issues/expected/NNNN-slug.exit` marker runs in the suite; unpinned ones don't. | +| `tests/run_examples.sh` | Test runner. Scans `examples/` and `issues/`; compares stdout/stderr/exit (+ optional IR) per test. | ### Unit test file convention @@ -465,22 +492,28 @@ All Zig unit tests live in separate `*.test.zig` files alongside the source they ### Creating a new standalone test -1. Create `examples/NN-name.sx` (focused feature example) **or** `examples/issue-NNNN.sx` (open-bug repro). -2. Run it: `./zig-out/bin/sx run examples/.sx` -3. Create expected output: `bash tests/run_examples.sh --update` +1. Create `examples/XXXX--.sx` (focused example) **or**, for an + open bug, `issues/NNNN-slug.{md,sx}` (repro co-located with the writeup). +2. Run it: `./zig-out/bin/sx run .sx` +3. Seed the marker (`: > /expected/.exit`) and capture expected: + `bash tests/run_examples.sh --update` 4. Verify: `bash tests/run_examples.sh` ### Resolving an open issue -When a bug filed as `examples/issue-NNNN.sx` is fixed, the file should leave the issue-namespace: +When a bug filed under `issues/NNNN-slug.{md,sx}` is fixed: -1. Pick a focused feature name with the next free number, e.g. `examples/52-frameworks.sx`. -2. `git mv examples/issue-NNNN.sx examples/NN-name.sx`. -3. `git mv tests/expected/issue-NNNN.txt tests/expected/NN-name.txt` (and the `.exit` file). -4. Tighten the comment header to describe the feature (drop the issue-NNNN provenance — that lives in git history now). +1. Move the repro into the feature suite as a regression test: + `git mv issues/NNNN-slug.sx examples/XXXX--.sx`. +2. Seed `examples/expected/XXXX--.exit`, capture with `--update`, + and review the diff. +3. Tighten the example's comment header to describe the feature (keep a one-line + `Regression (issue NNNN)` note for provenance). +4. Mark `issues/NNNN-slug.md` RESOLVED with a short banner (root cause + fix + + regression-test path). The `.md` stays as the writeup. 5. Run the suite to confirm nothing else broke. -The `examples/issue-*` glob after step 5 is the literal list of what's still open. +The set of `issues/*.md` without a RESOLVED banner is the open-issue list. ## Known bugs