ducoterra
4febdfe780
**Final verification pass — phase 03 `simulation_movement_and_interaction` (all 5 task files already in `complete/`)** - Verified all phase-03 deliverables exist and work: `sim/` kernel (`step`, `state_build`, `rng`, `movement`, `collision`, `interact`, `carry`, `replay`), replay fixtures + 3 committed goldens, `docs/SIMULATION.md`. - `bash .agents/validate.sh` → **passed**: ruff `All checks passed!`, pyright `0 errors`, `undercooked-levels validate` green, full `pytest` **1033 passed** (coverage **99.71%** ≥ 90% floor), slow sim stage green, repo-hygiene clean. - Perf smoke (4 players, 2 000 ticks): mean **0.049 ms/tick, 20 423 ticks/s** — ≥ 10 000 ticks/s criterion met (test lives in `test_perf_smoke.py`, which the gate's slow stage runs; the overview names it `test_replay.py`). - Completion criteria: import `undercooked.sim.step.step` ✅ · `tests/sim` green incl. determinism + goldens ✅ · perf ✅ · constant-change-fails-by-design ✅ (committed-hash equality + mutation test naming first bad tick/field; goldens never rewritten by tests) · `docs/SIMULATION.md` ✅ (constants, 8-step order, purity/determinism, re-bless) · full suite + coverage ✅ · app launches: `/healthz` 200, `/` 200, `/api/levels` serves content ✅ · replay CLI `check walk_solo.json` exit 0 ✅ · no regression in phases 01/02 (their suites green) ✅. - Documentation: README.md — fixed one stale inconsistency (Status opening listed only 01/02 complete with 03 pending; now lists 01–03 complete, next pending `04_kitchen_pipeline_and_scoring`); all other README claims (CLI flags, config table, shell text, replay CLI exit codes) verified against code. AGENTS.md — verified consistent with the project as it stands, no changes needed. - No deviations. Next pending phase: **`04_kitchen_pipeline_and_scoring`**.
91 lines
7.0 KiB
Markdown
91 lines
7.0 KiB
Markdown
# TESTING.md — how this project is verified
|
|
|
|
**Status:** current as of phase 02 · **Filled in by phase 14**, which re-runs every layer
|
|
below against the finished game and records the release-time suite inventory.
|
|
|
|
## Purpose
|
|
|
|
This document is the single reference for *how we know the game is right*: the test tree,
|
|
what belongs in each directory, how to run each layer without waiting for the others, the
|
|
marker set, the coverage floor, and the artifact policy. The layered verification workflow
|
|
behind it is `.agents/PLAN.md` §9; the gate that enforces all of it before any phase
|
|
completes is `.agents/validate.sh`.
|
|
|
|
## Layout
|
|
|
|
`tests/` is deliberately **not** a package (no `__init__.py`); fixtures shared by every
|
|
layer live in `tests/conftest.py`, and layer-specific fixtures live in that layer's own
|
|
`conftest.py`.
|
|
|
|
| Directory | Contains | I/O allowed |
|
|
|---|---|---|
|
|
| `tests/unit/` | pure logic with no I/O: config parsing, header values, CLI precedence, protocol/validator helpers, and the doc/content guards that keep README, `.env.example`, `PLAN.md` and the code agreeing | none (in-process `TestClient` requests are fine) |
|
|
| `tests/sim/` | tests of the pure kernel that need no server: grid geometry, the entity model, the A5 import-purity guard, and the level validator (phase 02), plus the tick itself (phase 03) — the seeded RNG's published golden vectors (`test_rng.py`), the kernel's clamping/accumulation/purity/determinism suite (`test_step_kernel.py`), and the movement stage: hand-computed geometry per rule (`test_movement.py`), a scripted 2000-substep four-cook walk on the sim's own RNG that re-checks containment, non-overlap, bounded displacement and hash equality after every substep (`test_movement_stress.py`), and the same walk through the whole `step()` on the shipped levels (`test_movement_kernel.py`), and the action button (phase 03 task 03): the reach-zone geometry and one-press-one-event press edge (`test_targeting.py`), and the table-driven verb ladder that covers every `Verb` and every `DeniedReason` plus the gate's contested arbitration (`test_verbs.py`); the golden-state-hash replay suites — `test_replay_carry.py` (task 04's two-player pass and contest), `test_replay.py` (the committed goldens against `sim/replay.py`'s input-script format, the `record`/`check` CLI and the perf seam) — and the `slow`-marked perf smoke (`test_perf_smoke.py`) | none — the sim must stay callable without a server |
|
|
| `tests/integration/` | multi-module, in-process tests: app boot through `create_app()`, HTTP routes, static serving, room/protocol behaviour with real WebSocket clients (room + protocol integration is PLAN §9 layer 4, and lives here) | in-process app, loopback sockets |
|
|
| `tests/e2e/` | one Playwright suite per user story, named `test_usNN_*.py`; multiplayer stories drive **two browser contexts** against one room | real browser, real socket |
|
|
| `tests/fixtures/` | fixture data only. `tests/fixtures/levels/` is the content dir every test sees — an autouse session fixture points `UNDERCOOKED_LEVELS_DIR` at it so no test can silently read the shipped `levels/` (`_bad/` holds the deliberately-broken campaign the `_` discovery rule ignores; `_replay/` holds the `pick_drop_pass` fixture kitchen, ignored the same way). `tests/fixtures/replays/` holds the committed input scripts and `tests/goldens/` the committed golden hashes they are gated on (`sim/replay.py`) | none |
|
|
|
|
There is **no `tests/server/`**: PLAN §9 counts server/route layers of coverage, not
|
|
directories. A route asserted as a pure function (header values, status codes, CLI
|
|
precedence) is a `tests/unit/` test; a route exercised against a process that actually
|
|
binds a socket is `tests/integration/`. Adding a fifth directory would only hide which of
|
|
those two a test is.
|
|
|
|
## Markers
|
|
|
|
Registered in `pyproject.toml` and enforced with `--strict-markers`:
|
|
|
|
| Marker | Meaning | Default run |
|
|
|---|---|---|
|
|
| `unit` | pure, no I/O, fast | yes |
|
|
| `integration` | multi-module, in-process server/room tests | yes |
|
|
| `e2e` | Playwright browser suites (needs `uv run playwright install chromium`) | yes |
|
|
| `slow` | replay/stress suites that exceed the quick loop (the 200-room stress run lives here from phase 12) | yes, excluded explicitly |
|
|
|
|
The everyday loop is `uv run pytest -m "not e2e"` (no browser launches). Before a phase
|
|
completes, and whenever `.agents/validate.sh` runs, the **full** suite runs — coverage gate
|
|
included — because the gate is "the entire project is green", not "my files are green".
|
|
|
|
## Running everything
|
|
|
|
```bash
|
|
uv sync --extra dev # dependencies + test toolchain
|
|
uv run playwright install chromium # once, before the first browser run
|
|
uv run pytest -q -m "not e2e" # quick loop
|
|
uv run pytest -q # everything, with the coverage floor
|
|
uv run pytest tests/unit -q -p no:cacheprovider # one file, no nested coverage writes
|
|
uv run ruff check . # lint (E,F,W,I,N,UP,B,C4,SIM,ASYNC,S,RUF,ARG,RET,PL,PERF)
|
|
uv run pyright # types
|
|
bash .agents/validate.sh # the pipeline gate, in that order + hygiene
|
|
```
|
|
|
|
The gate runs one check that is not a pytest command: `uv run undercooked-levels
|
|
validate` between pyright and pytest (docs/LEVELS.md **Authoring workflow**). It is
|
|
unconditional — a campaign that breaks a PLAN §8 rule stops the pipeline even when no
|
|
test file changed — and it is the same `load_content` + `validate_bundle` pair
|
|
`create_app()` runs at startup, so it can disagree with the server about nothing.
|
|
Content tests stay hermetic the same way: the autouse session fixture points
|
|
`UNDERCOOKED_LEVELS_DIR` at `tests/fixtures/levels/` (a recipe book and zero levels),
|
|
and a test that wants real levels builds one under `tmp_path` or names `levels/`
|
|
explicitly. `tests/unit/test_content_not_hardcoded.py` is the A8 lock on top of that.
|
|
|
|
Coverage is configured in `pyproject.toml`: `source = ["src/undercooked"]`, branch
|
|
coverage on, `--cov-fail-under=90` (AGENTS.md A20 asks for > 90 % on new/changed code; the
|
|
floor is the mechanical half of that).
|
|
|
|
Test processes never listen on a routable interface. The only fixture that binds a socket
|
|
is `live_server` (session-scoped, `tests/conftest.py`), which asks uvicorn for an
|
|
ephemeral `127.0.0.1` port and shuts the lifespan down cleanly.
|
|
|
|
## Artifact policy
|
|
|
|
- Test output is disposable and **never committed**: `test-results/`, `playwright-report/`,
|
|
`blob-report/`, `.playwright/`, `.pytest_cache/`, `.coverage`, `htmlcov/`,
|
|
`coverage.xml`, `coverage.json` are all gitignored.
|
|
- No test writes into `app/static/`, `levels/`, `src/`, or `docs/`; anything a test needs
|
|
is created under `tmp_path`.
|
|
- Nothing is downloaded during a test run: browsers are installed explicitly by a human
|
|
(or CI step), and no test reaches the network except against `127.0.0.1`.
|
|
- A test that cannot pass deterministically is a bug in the test — sleeps are bounded by
|
|
conditions, and the sim-facing suites are seeded through the `random_seed` fixture.
|