phase: 14_store_performance

**Phase 14 `store_performance` — final verification pass (all 4 tasks already in `complete/`)**

- Re-verified every task's work in the tree: read caches (`series_names`/`stats`), capped compound `pick_res_many`, `history_many` + `api._history_batch`, windowed `maintain()` + per-resolution prune + `PRAGMA optimize`/`automatic_analyze=0`/WAL pragmas, `store_perf.BUDGETS` (14 ceilings), probe, live watch.
- **Completion criteria:** baseline+after side by side in `data/logs/NOTES-store-perf.md` ✅ · budget tests green (`uv run pytest -m store_perf --store-perf` → **4 passed**, 14/14 inside) ✅ · 25-min dev-server watch re-run **independently** (74 passes, `scripts/watch-store-perf.py`): **0 `store-error`** (total still 190, newest 2026‑10‑18 06:33:30), WAL 7.77 MB ≤ 67.1 MB, medians 1.7–3.2 ms, p95 720/613 ms = the documented cold-cache window; a 3rd 27-pass watch on a freshly restarted server reproduced it (3.1/708, 1.8/572) ✅ · `maintain()` 5,368 ms → 2.7–6.8 ms and WAL bounded across two cycles (test + notes) ✅ · no behaviour change (all four history routes pinned against `query()`) ✅
- **Full gates:** `uv run pytest` → **1267 passed, 15 skipped**; `bash .agents/validate.sh` → **exit 0**, coverage **98.75%** (floor 94), `ruff check src tests agent` clean, `pyright` 0 errors, `node --check` ok, repo hygiene ok; `uv run pytest -m live --live` → **11 passed**; dev server up, 9/9 sources `ok`, pages 200 in 1.5–3.3 ms.
- **Docs fixed:** README/AGENTS still claimed a flat "20 s per pass" while `host_agent.py` (in this same uncommitted change) scales the budget to 5 s × disk count — corrected in both plus the two illustrative "20 s of wedged disk" sentences; broken bold (`` `driveripper** ``) repaired; stale gap-4 title + "tuning is its own phase" wording updated to say phase 14 ran; README's absolute "nothing … is answered by a whole-table scan" qualified to *per call* with the cold-window cost named; "third opt-in marker" → second; conftest `--store-perf` help named the real marker; **`data/logs/watch-store-perf.py` moved to `scripts/watch-store-perf.py`** (both docs told a clone to run a gitignored path) with root-anchored paths, `DASH_WATCH_OUT`, and dead code removed.
- **New finding recorded:** gap 9's "~10 s process-wide stall is still gap 4's / `14_store_performance`'s problem" is now closed — coincident >8 s `llm` + >12 s `ups` raw-stream gaps went 393 (00:00–12:00) and 198 (12:00–21:36) to **0** across 110+ min of the phase-14 store; written into AGENTS.md gaps 4 and 9 and the notes table.
- **Deviations:** did *not* pull either deferred cost knob (`STATS_CACHE_TTL_SECONDS`, warming caches from the maintenance tick) — AGENTS.md records that decision and its reasoning; `uv pip list --outdated` still reports certifi/fastapi/starlette/websockets (pre-existing advisory, no dependency touched this phase, gate treats it as advisory). Overwrote nothing: task 04's watch preserved as `store-perf-watch.task04.*`, my two runs as `.verify25.*` / `.current-code.*`.
- **Next pending phase:** `15_agent_mtls`, then `16_agent_smart_pass_budget` (owns the scaled-budget tests + its doc criterion, now partly pre-satisfied by this pass), `17_installer_device_filter`.

README.md

@@ -91,7 +91,8 @@ a labelled window, so the agent never has to remember who polled it last.
 
 Per-disk `smart` health is a best-effort add-on on the whole-disk `/sys/block` list (no
 partitions): the agent runs `smartctl -i -A -H` per physical disk — 5 s bound per device,
-20 s per pass, results cached 4 minutes — and parses health, temperature, power-on hours,
+a pass budget of 20 s as a floor that scales to 5 s × the disk count, results cached 4 minutes —
+and parses health, temperature, power-on hours,
 the sector alarms and SSD wear into `disk_io.devices[].smart`. Refreshing an expired cache
 happens on a background thread while the request is served the previous result, so a wedged
 disk can never cost a poll more than its own timeout (only the very first pass after the
@@ -334,7 +335,7 @@ documentation and is never loaded, and `scripts/dev-server.sh` exports `.env.dev
 | `HOST_INTERVAL_SECONDS` / `LLM_INTERVAL_SECONDS` / `NUT_INTERVAL_SECONDS` | how often each source is polled (10 / 2 / 5 s) |
 | `DISKS_INTERVAL_SECONDS` | how often the disk-health source re-reads each agent for SMART (default 300 s — SMART is not a per-second signal). No targets of its own: the `disks:<host>` jobs exist exactly when `HOST_TARGETS` does |
 | `HOST_TIMEOUT_SECONDS` / `LLM_TIMEOUT_SECONDS` / `NUT_TIMEOUT_SECONDS` | per-source fetch timeout — one slow source never holds up the others |
-| `DISKS_TIMEOUT_SECONDS` | the disks fetch's own budget (default 30 s), because the agent runs `smartctl` inside its snapshot handler and 20 s of wedged disk must not read as a dead host |
+| `DISKS_TIMEOUT_SECONDS` | the disks fetch's own budget (default 30 s), because the agent runs `smartctl` inside its snapshot handler and a wedged SMART pass must not read as a dead host |
 | `GUFO_API_KEY` / `VLLM_API_KEY`       | only if an engine is behind a bearer token; they never leave the server |
 | `NUT_HOST` / `NUT_PORT`               | upsd address — **whatever port it really listens on**, not always 3493. Reads are unauthenticated in stock NUT |
 | `NUT_USER` / `NUT_PASSWORD`           | only needed for instant commands (battery tests)                |
@@ -383,9 +384,27 @@ auth.py ── verifier + session key, loaded once at startup (no row and no see
   poller, so a SQLite write that gets refused is reported (`ok: false`, the real reason, the
   numbers aging, one line in the event log) instead of freezing the source under a green dot.
 * `store.py` — raw samples roll up to 1-minute and 1-hour averages and the raw rows are
-  pruned, so a year of history is a few hundred MB rather than a disk emergency. Its `meta`
-  table is the one place state that must outlive a restart goes (`get_meta`/`set_meta`/
-  `delete_meta`), and its first user is the admin credential below.
+  pruned, so a year of history is a few hundred MB rather than a disk emergency. Nothing a
+  per-request path asks is answered by a whole-table scan **on every call** (phase 14): the two
+  answers that used to be scanned on every call — `series_names()` behind `/api/metrics` and the
+  row counts behind `/api/health` — come from a 60 s / 30 s cache, so the scan is paid once per
+  window (its cost is the honest remainder in the known-gaps list) instead of once per request. `series_names`' is ended only by a write that can
+  change it: a `(source, host, metric)` the store has never written, or any `DELETE`. A poll of an
+  existing series leaves it alone, which is what makes the cache serve at all on a store nine pollers
+  write to (task 04's live watch is the proof: 535 ms per call before, 1.8 ms median after), and the
+  counts age on the clock. A chart's resolution comes from one capped compound statement instead of
+  three full `COUNT`s per series. `last_maintain` and `size_bytes` are deliberately never cached —
+  the first is how an operator tells a stalled rollup loop from a healthy one.
+  The write side is windowed too: `maintain()` rolls up only the buckets that became due since its
+  last run (read from `last_maintain`, overlapped by one bucket and snapped down to a bucket
+  boundary, so a gap in polling is a catch-up scan rather than missing charts) and prunes on its
+  full retention bound, one statement per resolution. It ends with `PRAGMA optimize` — the store's
+  only statistics refresh, deliberately not inside a poller's commit — and the WAL is kept honest by
+  `wal_autocheckpoint` plus a 64 MB `journal_size_limit` cap. On a 6.4 M-row store that took the
+  once-a-minute write transaction from 5.4 s to a few milliseconds, which is what stopped a poller's
+  insert losing its 5 s `busy_timeout` (known gap 4).
+  Its `meta` table is the one place state that must outlive a restart goes (`get_meta`/
+  `set_meta`/`delete_meta`), and its first user is the admin credential below.
 * `auth.py` — the admin login's decisions, in stdlib and without HTTP: a self-describing
   PBKDF2-HMAC-SHA256 verifier, an HMAC-signed stateless session cookie (no session table, and
   rotating the stored key is what evicts every other browser), a session-bound CSRF token, and
@@ -423,7 +442,12 @@ auth.py ── verifier + session key, loaded once at startup (no row and no see
   no env config answers `404 {"configured": false}` **without dialling** (so a page says
   "not configured" instead of going red), and a history query for a series that does not
   exist answers an empty series instead of an error (so a card asks for every plot it could
-  draw and reads `n/a` for the ones the drive never reported). `/api/disks/snapshot` is the
+  draw and reads `n/a` for the ones the drive never reported). The four `history` routes take
+  one batched door into the store (`api._history_batch` → `store.history_many`), so a page's
+  chart request is one grouped resolution query plus one index seek per series rather than four
+  queries per series — and the JSON it answers with is the same document the per-series loop
+  produced, series for series and in the same order (`tests/test_api.py` pins that against
+  points read straight out of the store with `query()`). `/api/disks/snapshot` is the
   drive-health surface: one entry per host with its physical disk rows, the agent's `smart`
   block verbatim, `smartctl_available` (`true` / `false` = install smartmontools / `null` =
   the agent never said), a `severity` beside each row's rated numbers, and a `fleet` block
@@ -557,6 +581,11 @@ uv run pytest                                  # unit + API tests, fully mocked,
 uv run pytest --cov                            # coverage gate
 set -a; . ./.env.dev; set +a                 # the live suite reads the same env as the app
 uv run pytest -m live --live                   # optional: real fleet / engines / upsd
+uv run pytest -m store_perf --store-perf -s    # optional: the store's hot-path timings (phase 14)
+uv run python scripts/store_perf_probe.py      # the same measurements as a report in data/logs/
+DASH_WATCH_SECONDS=1500 uv run python scripts/watch-store-perf.py
+                                               # ... and the live watch: 25 signed-in minutes against
+                                               # the running dev server, counting store-error and WAL
 uv run ruff check src tests agent && uv run pyright            # lint + types
 ./scripts/capture-fixtures.sh                  # re-record ground truth: host agents, UPS replies, gufo /slots
 ./scripts/capture-fixtures.sh --with-llm        # ... and the engines' /metrics while they are up
@@ -611,6 +640,38 @@ local-testing word stays legal in prose, which is the only place it can be usefu
 the four public paths, the reset command, the four variables and the `DASH_COOKIE_SECURE=1`
 deployment note. Nothing starts a server there; a sentence that quietly disappears is the bug.
 
+`tests/store_perf.py` is the second opt-in marker's machinery (`live` is the other). It seeds a
+**production-shaped
+SQLite file** — `SampleStore`'s own retention (`raw 48 h / 1m 30 d / 1h 400 d`), the fleet's real
+metric names, the real 10 s cadence: ~6.4 M rows and ~720 MB, the dev DB's order of magnitude — and
+times every hot path of `store.py`/`api.py` against it, with `EXPLAIN QUERY PLAN` for the queries
+phase 14 changes and a two-connection race that reproduces the `database is locked` line in the
+known-gaps list — beside the **pre-fix control** of the same race, because a clean race that never
+met a lock would prove nothing. Every pass that *writes* (a rollup catch-up, a race) runs on a
+scratch copy of the seeded file, so the baseline the next measurement compares against comes back
+exactly as it went in. The default suite builds that store and asserts its **shape and determinism**
+(row counts, retention windows, rollups that are the mean of the raw rows they came from, a second
+`maintain()` that changes nothing) but never its wall time — except behind the marker, where
+`store_perf.BUDGETS` is the phase's pinned regression table (14 ceilings, 2-3x the measured "after",
+`maintain()` < 1 s, a racing insert at zero `database is locked`) and a stale budget pointer fails a
+default-run test. The cached reads are reported twice —
+**cold** (the cache invalidated before every run) and **cached** — because comparing a cache hit
+against the baseline's full scan would book a 1 000x speed-up that is really just a cache. Only
+`-m store_perf --store-perf` runs the timing passes, and `scripts/store_perf_probe.py` is the CLI
+form that writes `data/logs/store-perf-baseline.{md,json}` (task 03's write-side run is
+`store-perf-task03.*`, task 04's budget run `store-perf-task04.*`) for `data/logs/NOTES-store-perf.md`. Two rules keep it
+honest: **no test ever opens `data/dashboards.sqlite3`** (the `--dev-db` flag exists for a human
+running the probe read-only), and the seeded file is a cache under `data/perf/`
+(`DASH_PERF_STORE_DIR` relocates it) because a before/after has to measure the same file — pytest's
+`tmp_path` on this box is `tmpfs`, i.e. RAM, which times a store nobody deploys.
+
+The non-timing half of that lives in `tests/test_store.py` and runs every time: a cache hit
+issues **no** SQL at all (counted through `sqlite3`'s trace callback, never through a wall clock),
+the series list is invalidated by the write that introduces a name and not by the poll that repeats
+one (and the tracking that decides has a cap, below which it falls back to invalidating),
+`last_maintain` is never served from a cache, and `pick_res_many()`/`history_many()` answer exactly
+what the per-series calls answer — on a small store and on the seeded one, at every window.
+
 One more file is worth knowing about: `tests/card_contract.py` is the table of
 `/api/system/snapshot` paths the system page's per-host cards are allowed to read. It is
 checked against all three captured host shapes (discrete GPU, APU, no GPU), over HTTP, and