Scaffold project and author 14-phase roadmap

Browser co-op kitchen game: FastAPI + native WebSockets, server-authoritative
20 Hz simulation, in-memory anonymous lobbies, data-driven YAML levels, Canvas 2D
vanilla ES-module client with locally authored SVG art and no external CDNs.

.agents/PLAN.md records the LOCKED decisions; .agents/user_stories/ holds the 12
flow contracts; .agents/phases/todo/ breaks them into 14 sequential, test-gated
phases (71 task files), each leaving the app launchable and the suite green.

LLM-Generated: true

.agents/phases/todo/12_reconnect_resilience_and_scale/00_phase.md

@@ -0,0 +1,48 @@
+# Phase 12 — Resilience, garbage collection, and scale
+
+**Story:** `US-09_disconnect_and_rejoin`, `US-01_create_anonymous_lobby`
+**Context:** `.agents/PLAN.md` §10 (performance budgets), §8 (lifecycle), A6
+
+## Objective
+Make the service boringly reliable: rooms that always empty, memory that always comes back,
+rejoins that work under churn, and measured capacity for 200 concurrent rooms with hard
+limits that fail safe instead of degrading everyone.
+
+## Dependencies
+- `11_campaign_levels_and_progression` (complete) — realistic room load (10 levels, hazards) to
+  scale against
+- `06_realtime_websocket_protocol` (complete) — heartbeats, seat grace, socket-kill seam
+
+## Tasks
+1. `01_room_garbage_collection.md` — every room-releasing path, idle GC, leak test that memory returns to baseline
+2. `02_rejoin_chaos_matrix.md` — disconnect/rejoin/restart churn under load with invariant checks
+3. `03_tick_scheduler_under_load.md` — one scheduler for all rooms, adaptive coalescing, per-room fairness
+4. `04_capacity_and_degradation.md` — 200-room capacity test, admission control, graceful "full" behavior
+5. `05_ops_observability_and_runbook.md` — `/api/health` deep metrics, structured logs, `docs/DEPLOYMENT.md` runbook
+
+## Testing & Quality
+- Integration + perf (`slow`): leak test (200 create/play/close cycles → RSS within 5 % of
+  baseline), churn matrix, 200-room capacity with p95 tick budget respected, admission control
+  returning the documented `503`/`room_full` instead of queuing.
+- Invariants under every stress: no orphaned tasks (`asyncio.all_tasks` bounded), no room whose
+  `last_activity` exceeds the GC window, no session record for a closed room beyond the TTL, zero
+  unhandled exceptions in the server log.
+- Coverage: **>90%** on new/modified code.
+- The phase is "Complete" only when the **entire** project test suite passes 100%.
+
+## Documentation & Consistency
+- The final pass reviews `README.md` and `AGENTS.md` in full against the project as it
+  stands after this phase — fixing every inconsistency found, **including pre-existing
+  ones not caused by this phase**.
+- Tasks finish `docs/DEPLOYMENT.md` (capacity numbers, limits, runbook: "rooms not emptying",
+  "tick lag", "memory climb"), and README's "what this demo is / is not" section.
+
+## Completion Criteria
+- [ ] `uv run pytest -m slow tests/integration/test_capacity.py -q` → 200 rooms, p95 tick cost ≤ 2 ms, memory within budget
+- [ ] `uv run pytest tests/integration/test_room_gc.py -q` → green: 200 create/close cycles return RSS to within 5 % of baseline
+- [ ] `uv run pytest -m slow tests/integration/test_rejoin_chaos.py -q` → green, zero invariant violations
+- [ ] admission control returns the documented response at the cap (asserted), never a silent queue
+- [ ] `docs/DEPLOYMENT.md` runbook covers the three named failure modes with the actual commands
+- [ ] full test suite **100% passing**, coverage >90%
+- [ ] README.md and AGENTS.md are accurate and consistent with the whole project (verified in the final pass)
+- [ ] no behavior change in completed phases

.agents/phases/todo/12_reconnect_resilience_and_scale/01_room_garbage_collection.md

@@ -0,0 +1,47 @@
+# Task 01 — Room garbage collection and leak proof
+
+**Phase:** `12_reconnect_resilience_and_scale` · **Story:** `US-09_disconnect_and_rejoin`
+
+## Objective
+Every code path that could keep a room alive must be enumerated, tested, and provably
+non-leaking.
+
+## Work
+1. `src/undercooked/server/gc.py`:
+   - single owner of reclamation: `RoomReaper` with rules — empty room past
+     `empty_grace_s`, all guests disconnected past `abandon_s`, round finished past
+     `result_hold_s` with no activity, lobby idle past `lobby_idle_s`, session records past
+     `session_ttl_s` with no connected seat;
+   - every release path funnels through `release(room_code, reason)` with a `reason` enum
+     (`host_leave`, `all_empty`, `idle`, `finished`, `evicted`, `shutdown`) — no ad-hoc
+     `del`/dict pops anywhere else (enforced by a grep test);
+   - cancellation is total: room task, tick task, per-connection send tasks, heartbeat timers;
+     `release` awaits them with a timeout and logs (counted in `/api/health`) any that had to be
+     abandoned;
+   - `sweep_once(now)` returns the swept codes + reasons; runs on a timer **and** is callable
+     from tests with a `FakeClock`.
+2. `tests/integration/test_room_gc.py`:
+   - one test per release reason, each asserting: room gone from the registry, no tasks left,
+     connections closed with the right code, session record TTL'd, and `sweep_once` is idempotent
+     (a second sweep sweeps nothing);
+   - the leak cycle: 200 create → start → play 5 s → close cycles; assert RSS within 5 % of
+     baseline after forced `gc.collect()` (measured with `tracemalloc` top-3 report printed on
+     failure), and `asyncio.all_tasks()` count back to the baseline;
+   - a room with one connected spectator is never swept while the spectator watches.
+3. `tests/unit/test_release_paths_exclusive.py` — static guard: no `rooms._rooms.pop` outside
+   `gc.py` (AST scan of `src/undercooked/server/`), so future code cannot leak by side door.
+4. `/api/health` gains `rooms`, `seats`, `spectators`, `tasks`, `rss_mb`, `sweeps_total` broken
+   out by reason.
+
+## Testing & Quality
+- Integration + unit (AST guard); tracemalloc-backed leak assertions.
+- Coverage: **>90%** on `gc.py` (target 95%).
+
+## Completion Criteria
+- [ ] `uv run pytest tests/integration/test_room_gc.py tests/unit/test_release_paths_exclusive.py -q` → green
+- [ ] 200 create/close cycles leave RSS within 5 % of baseline (asserted, with a tracemalloc report on failure)
+- [ ] planting a `rooms._rooms.pop` outside `gc.py` fails the AST guard (demonstrated, then reverted)
+- [ ] every sweep reason is covered by exactly one named test
+- [ ] full test suite **100% passing**
+- [ ] README.md/AGENTS.md updated for anything user-visible this task changes (the phase's final pass re-checks both project-wide)
+- [ ] no behavior change in completed work

.agents/phases/todo/12_reconnect_resilience_and_scale/02_rejoin_chaos_matrix.md

@@ -0,0 +1,44 @@
+# Task 02 — Rejoin chaos matrix
+
+**Phase:** `12_reconnect_resilience_and_scale` · **Story:** `US-09_disconnect_and_rejoin`
+
+## Objective
+Throw connection churn at real rooms and prove nothing corrupts: no double seats, no lost
+items, no score jumps, no stuck countdowns.
+
+## Work
+1. `tests/integration/test_rejoin_chaos.py` (`slow`):
+   - matrix over `{players: 2, 4} × {level: 1, 7, 10} × {churn: abrupt_close, offline_event,
+     server_side_kill, token_reuse_storm, join_during_countdown, rejoin_after_grace}`;
+   - each scenario: 30 random churn events during a live round (seeded), then assert the
+     invariants below;
+   - invariants after every scenario: exactly one seat per token, no seat with two connections,
+     item conservation (`counted + wasted + lost == produced`), score monotonic non-decreasing,
+     `round` payload produced exactly once per round, no room stuck in `COUNTDOWN` past
+     `countdown_s + 2`, ghost states converge (`GHOST_ON` ⇒ eventually `GHOST_OFF` or seat
+     released), zero unhandled exceptions in the captured server log;
+   - a `token_reuse_storm` (same token connecting 10× concurrently) must end with exactly one
+     live connection for that seat and the others closed with `4404` or `1000` (documented).
+2. Fix the failures it finds in `server/room.py` / `server/ws.py`; each fix gets a focused
+   regression test in `tests/integration/test_ws_disconnect.py` (extend) with a name that states
+   the race.
+3. `docs/SIMULATION.md` / `docs/PROTOCOL.md` — document the concurrency rules that make this
+   work (single-writer room task, inputs applied only inside the tick, no awaits inside the sim
+   step) so future code does not break them.
+4. `/api/health` gains `rejoins_total`, `ghost_seats`, `seat_conflicts_total` (the last must stay
+   0 — asserted in the chaos suite).
+
+## Testing & Quality
+- Integration stress (`slow`), seeded and reproducible (failure prints the seed and a
+  reproduction command).
+- Coverage: **>90%** on new/modified code.
+- Full suite green.
+
+## Completion Criteria
+- [ ] `uv run pytest -m slow tests/integration/test_rejoin_chaos.py -q` → green for all 12 scenarios
+- [ ] `seat_conflicts_total == 0` across the matrix (asserted)
+- [ ] item conservation and score monotonicity hold in every scenario (asserted)
+- [ ] every fix has a named regression test; the report lists the races found
+- [ ] full test suite **100% passing**, coverage >90%
+- [ ] README.md and AGENTS.md are accurate and consistent with the whole project (verified in the final pass)
+- [ ] no behavior change in completed phases

.agents/phases/todo/12_reconnect_resilience_and_scale/03_tick_scheduler_under_load.md

@@ -0,0 +1,45 @@
+# Task 03 — One scheduler for all rooms
+
+**Phase:** `12_reconnect_resilience_and_scale` · **Story:** `US-01_create_anonymous_lobby`
+
+## Objective
+Replace per-room tick loops with a single fair scheduler so 200 rooms cost ~200× the sim and
+not 200× the event-loop overhead.
+
+## Work
+1. `src/undercooked/server/scheduler.py`:
+   - one `RoomScheduler` task owning all room tick timers; rooms register/unregister;
+   - fixed 20 Hz budget loop: each cycle computes the due rooms, runs their `tick_once` in
+     registration order (deterministic, no starvation), and yields between rooms every
+     `YIELD_EVERY_ROOMS = 16` to keep the event loop responsive;
+   - adaptive degradation ladder (documented + metrics): if the cycle overruns
+     `budget_ms`, degrade in order — drop snapshot rate for distant spectators → halve tick rate
+     for rooms with all-seats-idle → refuse new rooms (task 04's admission control) — never
+     drop a room's inputs, and always emit a `DEGRADED` metric with the ladder step;
+   - per-room fairness counter (`starved_cycles`) exposed in `/api/health`; must stay 0 in the
+     capacity test.
+2. Remove per-room tick tasks from `server/room.py` (keep the room's single-writer queue
+   semantics: inputs still applied only inside `tick_once`).
+3. `tests/integration/test_scheduler.py`:
+   - 50 rooms tick at the correct rate (measured ticks/second per room within 2 %),
+   - a deliberately slow room (test-only injected sleep) does not starve others
+     (`starved_cycles == 0` for the other 49),
+   - degradation ladder engages and reports the right step, then recovers when the load drops,
+   - scheduler shutdown cancels cleanly (no pending tasks), and a room unregistering mid-cycle
+     is safe (no "task was destroyed", no tick on a released room).
+4. `docs/DEPLOYMENT.md` — the ladder, what each step means for players, and the metric to watch.
+
+## Testing & Quality
+- Integration with a controllable clock and injected slow rooms.
+- Coverage: **>90%** on `scheduler.py` (target 95%).
+- Full suite green (the phase-06 two-client round suite must pass unchanged — it is the real
+  behavior guard).
+
+## Completion Criteria
+- [ ] `uv run pytest tests/integration/test_scheduler.py -q` → green
+- [ ] 50 rooms hold 20 Hz within 2 % (asserted)
+- [ ] one slow room does not starve the others and the ladder reports the correct step (asserted)
+- [ ] `uv run pytest tests/integration/test_ws_round_two_clients.py -q` → still green
+- [ ] full test suite **100% passing**
+- [ ] README.md/AGENTS.md updated for anything user-visible this task changes (the phase's final pass re-checks both project-wide)
+- [ ] no behavior change in completed work

.agents/phases/todo/12_reconnect_resilience_and_scale/04_capacity_and_degradation.md

@@ -0,0 +1,50 @@
+# Task 04 — Capacity, admission control, and fail-safe limits
+
+**Phase:** `12_reconnect_resilience_and_scale` · **Story:** `US-01_create_anonymous_lobby`
+
+## Objective
+Know the ceiling, enforce it politely, and prove the service degrades for nobody rather than
+for everybody.
+
+## Work
+1. `src/undercooked/server/admission.py`:
+   - limits from config: `max_rooms`, `max_connections_total`, `max_connections_per_ip`,
+     `max_rooms_per_ip`, `max_starting_per_second`;
+   - `POST /api/rooms` returns `503` + `Retry-After` + JSON `{code: "server_full"}` when at
+     capacity (documented in `docs/PROTOCOL.md`), `/ws` closes with `4403` (new close code) for
+     the same condition, and the client shows "the kitchen is full, try again in a moment"
+     (retry affordance, code preserved) instead of spinning;
+   - never queue: refuse immediately (a queue would create the timeout UX we are avoiding);
+   - `/api/health` reports `capacity: {rooms, connections, limit, degraded}`;
+     `ErrorCode` gains `server_full` and close code `4403` is added, so bump
+     `PROTOCOL_VERSION`, update `docs/PROTOCOL.md`, and let
+     `tests/unit/test_client_enum_parity.py` force the JS side to follow (AGENTS §5).
+2. `tests/integration/test_capacity.py` (`slow`):
+   - 200 rooms × 4 seats created and started (bots idle at the counter), measured over 60 s:
+     p95 `tick_once` cost ≤ 2 ms, mean ≤ 1 ms, RSS ≤ `max_memory_mb`, snapshot downlink per
+     connection ≤ 60 KB/s, `starved_cycles == 0`, zero errors in the log;
+   - 201st room → `503`/`server_full` and the 200 existing rooms keep their budgets (measured
+     before/after, no regression > 5 %);
+   - per-IP limits enforced with a rotated-source simulation (fixture patches the ASGI
+     `client` scope), and `max_rooms_per_ip` prevents one browser from hoarding codes;
+   - memory after the test returns to baseline (uses task 01's reaper).
+3. Client: `screens/error.js` handles `server_full` and close `4403` with a countdown retry and
+   the join code preserved; `docs/CLIENT.md` documents the codes.
+4. `docs/DEPLOYMENT.md` — measured capacity table (rooms, seats, RSS, CPU), the limits and how
+   to raise them, and the "one process, no horizontal scaling in v1" statement with the reason
+   (in-memory rooms, A6) and what v2 would need (consistent hashing / sticky routing — explicitly
+   out of scope).
+
+## Testing & Quality
+- Integration perf (`slow`) with hard numeric gates; the suite is the release capacity proof.
+- Coverage: **>90%** on `admission.py`.
+- Full suite green.
+
+## Completion Criteria
+- [ ] `uv run pytest -m slow tests/integration/test_capacity.py -q` → green with all budgets met
+- [ ] the 201st create request returns `503 server_full` and existing rooms are unaffected (asserted)
+- [ ] `/ws` at capacity closes `4403` and the client shows the retry copy (e2e-asserted)
+- [ ] `docs/DEPLOYMENT.md` records the measured numbers with the date and machine profile
+- [ ] full test suite **100% passing**, coverage >90%
+- [ ] README.md and AGENTS.md are accurate and consistent with the whole project (verified in the final pass)
+- [ ] no behavior change in completed phases

.agents/phases/todo/12_reconnect_resilience_and_scale/05_ops_observability_and_runbook.md

@@ -0,0 +1,50 @@
+# Task 05 — Observability and the operations runbook
+
+**Phase:** `12_reconnect_resilience_and_scale` · **Story:** `US-09_disconnect_and_rejoin`
+
+## Objective
+Make the running service diagnosable from the outside, with logs that never leak anything
+personal, and write the runbook someone else would follow at 2 a.m.
+
+## Work
+1. `src/undercooked/server/observability.py`:
+   - structured JSON logs (stdlib `logging` + a JSON formatter): `ts`, `level`, `event`,
+     `room` (short code), `seat`, `tick`, `dur_ms`, `code` — no names, no tokens, no IPs in
+     info logs (IP only at `debug`, and `debug` is off by default);
+   - counters/histograms in-process (`metrics.py`): rooms live, seats, rejoins, ghosts, sweeps
+     by reason, denies by reason, tick cost histogram, snapshot bytes, ws closes by code,
+     admission rejections, degradation steps;
+   - `GET /api/health` (public, counts only, no room codes) and `GET /api/metrics`
+     (text/plain, Prometheus-style exposition) gated by `metrics_enabled` **and**
+     `metrics_bind_private` (default: only served when the request comes from a private/
+     loopback client — the reverse proxy is the intended consumer), documented in
+     `docs/PROTOCOL.md`.
+2. Log-leak guard `tests/unit/test_no_pii_in_logs.py`: run a scripted round with capability-tagged
+   fake secrets (`SECRET_TOKEN_marker`, `SECRET_NAME_marker`) as tokens/names, capture logs, and
+   fail if any `SECRET_*_marker` appears in INFO/WARNING output; also assert room codes *are*
+   present (they are the support handle) so the guard is not trivially satisfied.
+3. `tests/server/test_metrics.py` — exposition format parses, the documented metric names exist,
+   private-network gate allows loopback and refuses a public client IP, `metrics_enabled=false`
+   yields `404`.
+4. `docs/DEPLOYMENT.md` — full runbook: build, run, TLS via the deployment repo's certbot/nginx
+   pattern (WebSocket `proxy_http_version 1.1` + `Upgrade` headers spelled out), systemd unit,
+   log rotation, the three failure modes ("rooms not emptying", "tick lag", "memory climb") with
+   the exact `/api/health` + `/api/metrics` fields to read, and the restart-is-wipes-everything
+   caveat (A6) stated plainly.
+5. `README.md` — "Running this for real" section pointing at `docs/DEPLOYMENT.md`, plus an honest
+   "what this is / is not" paragraph (single process, in-memory, demo-grade persistence,
+   production-grade transport).
+
+## Testing & Quality
+- Unit (log leak, formatter) + server (metrics, gating).
+- Coverage: **>90%** on `observability.py`/`metrics.py`.
+- Full suite green.
+
+## Completion Criteria
+- [ ] `uv run pytest tests/unit/test_no_pii_in_logs.py tests/server/test_metrics.py -q` → green
+- [ ] planting a token in an INFO log makes the guard fail (demonstrated, then reverted)
+- [ ] `/api/metrics` is 404 for a public client IP and 200 for loopback (asserted)
+- [ ] `docs/DEPLOYMENT.md` nginx block actually proxies a WebSocket (verified once against a local nginx config or stated as unverified in the report)
+- [ ] full test suite **100% passing**, coverage >90%
+- [ ] README.md and AGENTS.md are accurate and consistent with the whole project (verified in the final pass)
+- [ ] no behavior change in completed phases