undercooked/docs/TESTING.md

TESTING.md — how this project is verified

Status: current as of phase 01 · Filled in by phase 14, which re-runs every layer below against the finished game and records the release-time suite inventory.

Purpose

This document is the single reference for how we know the game is right: the test tree, what belongs in each directory, how to run each layer without waiting for the others, the marker set, the coverage floor, and the artifact policy. The layered verification workflow behind it is .agents/PLAN.md §9; the gate that enforces all of it before any phase completes is .agents/validate.sh.

Layout

tests/ is deliberately not a package (no __init__.py); fixtures shared by every layer live in tests/conftest.py, and layer-specific fixtures live in that layer's own conftest.py.

Directory	Contains	I/O allowed
tests/unit/	pure logic with no I/O: config parsing, header values, CLI precedence, protocol/validator helpers, and the doc/content guards that keep README, .env.example, PLAN.md and the code agreeing	none (in-process TestClient requests are fine)
tests/sim/	headless determinism and replay tests against the pure kernel (tests/sim/test_replay_*.py, golden state hashes)	none — the sim must stay callable without a server
tests/integration/	multi-module, in-process tests: app boot through create_app(), HTTP routes, static serving, room/protocol behaviour with real WebSocket clients (room + protocol integration is PLAN §9 layer 4, and lives here)	in-process app, loopback sockets
tests/e2e/	one Playwright suite per user story, named test_usNN_*.py; multiplayer stories drive two browser contexts against one room	real browser, real socket
tests/fixtures/	fixture data only. tests/fixtures/levels/ is the content dir every test sees — an autouse session fixture points UNDERCOOKED_LEVELS_DIR at it so no test can silently read the shipped levels/	none

There is no tests/server/: PLAN §9 counts server/route layers of coverage, not directories. A route asserted as a pure function (header values, status codes, CLI precedence) is a tests/unit/ test; a route exercised against a process that actually binds a socket is tests/integration/. Adding a fifth directory would only hide which of those two a test is.

Markers

Registered in pyproject.toml and enforced with --strict-markers:

Marker	Meaning	Default run
unit	pure, no I/O, fast	yes
integration	multi-module, in-process server/room tests	yes
e2e	Playwright browser suites (needs uv run playwright install chromium)	yes
slow	replay/stress suites that exceed the quick loop (the 200-room stress run lives here from phase 12)	yes, excluded explicitly

The everyday loop is uv run pytest -m "not e2e" (no browser launches). Before a phase completes, and whenever .agents/validate.sh runs, the full suite runs — coverage gate included — because the gate is "the entire project is green", not "my files are green".

Running everything

uv sync --extra dev                            # dependencies + test toolchain
uv run playwright install chromium             # once, before the first browser run
uv run pytest -q -m "not e2e"                  # quick loop
uv run pytest -q                               # everything, with the coverage floor
uv run pytest tests/unit -q -p no:cacheprovider # one file, no nested coverage writes
uv run ruff check .                            # lint (E,F,W,I,N,UP,B,C4,SIM,ASYNC,S,RUF,ARG,RET,PL,PERF)
uv run pyright                                 # types
bash .agents/validate.sh                       # the pipeline gate, in that order + hygiene

Coverage is configured in pyproject.toml: source = ["src/undercooked"], branch coverage on, --cov-fail-under=90 (AGENTS.md A20 asks for > 90 % on new/changed code; the floor is the mechanical half of that).

Test processes never listen on a routable interface. The only fixture that binds a socket is live_server (session-scoped, tests/conftest.py), which asks uvicorn for an ephemeral 127.0.0.1 port and shuts the lifespan down cleanly.

Artifact policy

• Test output is disposable and never committed: test-results/, playwright-report/, blob-report/, .playwright/, .pytest_cache/, .coverage, htmlcov/, coverage.xml, coverage.json are all gitignored.
• No test writes into app/static/, levels/, src/, or docs/; anything a test needs is created under tmp_path.
• Nothing is downloaded during a test run: browsers are installed explicitly by a human (or CI step), and no test reaches the network except against 127.0.0.1.
• A test that cannot pass deterministically is a bug in the test — sleeps are bounded by conditions, and the sim-facing suites are seeded through the random_seed fixture.