ducoterra
3426264ffb
Phase `02_level_content_and_validation` — final verification **green**. - **Gate:** `bash .agents/validate.sh` → ruff ✅ · pyright 0 errors ✅ · `undercooked-levels validate` OK (2 levels) ✅ · **668 passed** (incl. 10 Playwright e2e) ✅ · coverage **100%** ✅ · repo hygiene ✅ - **Criteria:** `validate` exit 0, one line per level; `list` = `01_first_kitchen`, `02_double_shift` in campaign order; live `:8000` `/healthz` 200, `/api/levels` (+detail) carries both levels with numeric `target_score`/`time_limit` and no grid/station leakage, `/api/config` correct, unknown id → 404, startup log `content loaded: 2 level(s), 4 recipe(s)`; `validate --dir tests/fixtures/levels/_bad/` → exit 1 with named `RECIPE_STATION_MISSING` / `SERVE_REACHABLE` / `TARGET_IMPLAUSIBLE`; A8 lock test proves a YAML-only level reaches loader+API+CLI. - **Defect fixed (pre-existing path, phase-02-triggered):** `undercooked` with unusable content printed the report then a raw traceback and exited `1`. `src/undercooked/__main__.py` now catches `ContentError` → one closing line, exit `2` (like a bad setting); 2 new tests in `tests/unit/test_main.py` (incl. non-content failures still propagate). - **Docs pass (full, project-wide):** README (content gate + exit `2`, config section), AGENTS.md §3, `docs/LEVELS.md` (startup refusal + exact CLI exit codes), `docs/DEPLOYMENT.md` (crash-loop diagnosis bullet); DESIGN/TESTING re-read, consistent. - **Container re-verified (no phase-01 regression):** `podman build -f Containerfile` succeeds; the image ships `levels/` and serves `/healthz` 200, `/api/levels` (2 levels), `/` 200 as non-root. - **Deviation to note:** the phase overview names `tests/fixtures/levels/bad/`; the actual broken-fixture dir is `tests/fixtures/levels/_bad/` (the `_` prefix is what keeps it out of normal discovery). Overview files are read-only to me — README/LEVELS.md document the real path. - No git index/history commands run; all changes left unstaged for the harness. ntfy notification sent.
91 lines
5.7 KiB
Markdown
91 lines
5.7 KiB
Markdown
# TESTING.md — how this project is verified
|
|
|
|
**Status:** current as of phase 02 · **Filled in by phase 14**, which re-runs every layer
|
|
below against the finished game and records the release-time suite inventory.
|
|
|
|
## Purpose
|
|
|
|
This document is the single reference for *how we know the game is right*: the test tree,
|
|
what belongs in each directory, how to run each layer without waiting for the others, the
|
|
marker set, the coverage floor, and the artifact policy. The layered verification workflow
|
|
behind it is `.agents/PLAN.md` §9; the gate that enforces all of it before any phase
|
|
completes is `.agents/validate.sh`.
|
|
|
|
## Layout
|
|
|
|
`tests/` is deliberately **not** a package (no `__init__.py`); fixtures shared by every
|
|
layer live in `tests/conftest.py`, and layer-specific fixtures live in that layer's own
|
|
`conftest.py`.
|
|
|
|
| Directory | Contains | I/O allowed |
|
|
|---|---|---|
|
|
| `tests/unit/` | pure logic with no I/O: config parsing, header values, CLI precedence, protocol/validator helpers, and the doc/content guards that keep README, `.env.example`, `PLAN.md` and the code agreeing | none (in-process `TestClient` requests are fine) |
|
|
| `tests/sim/` | tests of the pure kernel that need no server: grid geometry, the entity model, the A5 import-purity guard, and the level validator (phase 02) — plus the headless determinism/replay suites with golden state hashes (`tests/sim/test_replay_*.py`) that arrive with the tick in phase 03 | none — the sim must stay callable without a server |
|
|
| `tests/integration/` | multi-module, in-process tests: app boot through `create_app()`, HTTP routes, static serving, room/protocol behaviour with real WebSocket clients (room + protocol integration is PLAN §9 layer 4, and lives here) | in-process app, loopback sockets |
|
|
| `tests/e2e/` | one Playwright suite per user story, named `test_usNN_*.py`; multiplayer stories drive **two browser contexts** against one room | real browser, real socket |
|
|
| `tests/fixtures/` | fixture data only. `tests/fixtures/levels/` is the content dir every test sees — an autouse session fixture points `UNDERCOOKED_LEVELS_DIR` at it so no test can silently read the shipped `levels/` | none |
|
|
|
|
There is **no `tests/server/`**: PLAN §9 counts server/route layers of coverage, not
|
|
directories. A route asserted as a pure function (header values, status codes, CLI
|
|
precedence) is a `tests/unit/` test; a route exercised against a process that actually
|
|
binds a socket is `tests/integration/`. Adding a fifth directory would only hide which of
|
|
those two a test is.
|
|
|
|
## Markers
|
|
|
|
Registered in `pyproject.toml` and enforced with `--strict-markers`:
|
|
|
|
| Marker | Meaning | Default run |
|
|
|---|---|---|
|
|
| `unit` | pure, no I/O, fast | yes |
|
|
| `integration` | multi-module, in-process server/room tests | yes |
|
|
| `e2e` | Playwright browser suites (needs `uv run playwright install chromium`) | yes |
|
|
| `slow` | replay/stress suites that exceed the quick loop (the 200-room stress run lives here from phase 12) | yes, excluded explicitly |
|
|
|
|
The everyday loop is `uv run pytest -m "not e2e"` (no browser launches). Before a phase
|
|
completes, and whenever `.agents/validate.sh` runs, the **full** suite runs — coverage gate
|
|
included — because the gate is "the entire project is green", not "my files are green".
|
|
|
|
## Running everything
|
|
|
|
```bash
|
|
uv sync --extra dev # dependencies + test toolchain
|
|
uv run playwright install chromium # once, before the first browser run
|
|
uv run pytest -q -m "not e2e" # quick loop
|
|
uv run pytest -q # everything, with the coverage floor
|
|
uv run pytest tests/unit -q -p no:cacheprovider # one file, no nested coverage writes
|
|
uv run ruff check . # lint (E,F,W,I,N,UP,B,C4,SIM,ASYNC,S,RUF,ARG,RET,PL,PERF)
|
|
uv run pyright # types
|
|
bash .agents/validate.sh # the pipeline gate, in that order + hygiene
|
|
```
|
|
|
|
The gate runs one check that is not a pytest command: `uv run undercooked-levels
|
|
validate` between pyright and pytest (docs/LEVELS.md **Authoring workflow**). It is
|
|
unconditional — a campaign that breaks a PLAN §8 rule stops the pipeline even when no
|
|
test file changed — and it is the same `load_content` + `validate_bundle` pair
|
|
`create_app()` runs at startup, so it can disagree with the server about nothing.
|
|
Content tests stay hermetic the same way: the autouse session fixture points
|
|
`UNDERCOOKED_LEVELS_DIR` at `tests/fixtures/levels/` (a recipe book and zero levels),
|
|
and a test that wants real levels builds one under `tmp_path` or names `levels/`
|
|
explicitly. `tests/unit/test_content_not_hardcoded.py` is the A8 lock on top of that.
|
|
|
|
Coverage is configured in `pyproject.toml`: `source = ["src/undercooked"]`, branch
|
|
coverage on, `--cov-fail-under=90` (AGENTS.md A20 asks for > 90 % on new/changed code; the
|
|
floor is the mechanical half of that).
|
|
|
|
Test processes never listen on a routable interface. The only fixture that binds a socket
|
|
is `live_server` (session-scoped, `tests/conftest.py`), which asks uvicorn for an
|
|
ephemeral `127.0.0.1` port and shuts the lifespan down cleanly.
|
|
|
|
## Artifact policy
|
|
|
|
- Test output is disposable and **never committed**: `test-results/`, `playwright-report/`,
|
|
`blob-report/`, `.playwright/`, `.pytest_cache/`, `.coverage`, `htmlcov/`,
|
|
`coverage.xml`, `coverage.json` are all gitignored.
|
|
- No test writes into `app/static/`, `levels/`, `src/`, or `docs/`; anything a test needs
|
|
is created under `tmp_path`.
|
|
- Nothing is downloaded during a test run: browsers are installed explicitly by a human
|
|
(or CI step), and no test reaches the network except against `127.0.0.1`.
|
|
- A test that cannot pass deterministically is a bug in the test — sleeps are bounded by
|
|
conditions, and the sim-facing suites are seeded through the `random_seed` fixture.
|