phase: 14_store_performance

**Phase 14 `store_performance` — final verification pass (all 4 tasks already in `complete/`)** - Re-verified every task's work in the tree: read caches (`series_names`/`stats`), capped compound `pick_res_many`, `history_many` + `api._history_batch`, windowed `maintain()` + per-resolution prune + `PRAGMA optimize`/`automatic_analyze=0`/WAL pragmas, `store_perf.BUDGETS` (14 ceilings), probe, live watch. - **Completion criteria:** baseline+after side by side in `data/logs/NOTES-store-perf.md` ✅ · budget tests green (`uv run pytest -m store_perf --store-perf` → **4 passed**, 14/14 inside) ✅ · 25-min dev-server watch re-run **independently** (74 passes, `scripts/watch-store-perf.py`): **0 `store-error`** (total still 190, newest 2026‑10‑18 06:33:30), WAL 7.77 MB ≤ 67.1 MB, medians 1.7–3.2 ms, p95 720/613 ms = the documented cold-cache window; a 3rd 27-pass watch on a freshly restarted server reproduced it (3.1/708, 1.8/572) ✅ · `maintain()` 5,368 ms → 2.7–6.8 ms and WAL bounded across two cycles (test + notes) ✅ · no behaviour change (all four history routes pinned against `query()`) ✅ - **Full gates:** `uv run pytest` → **1267 passed, 15 skipped**; `bash .agents/validate.sh` → **exit 0**, coverage **98.75%** (floor 94), `ruff check src tests agent` clean, `pyright` 0 errors, `node --check` ok, repo hygiene ok; `uv run pytest -m live --live` → **11 passed**; dev server up, 9/9 sources `ok`, pages 200 in 1.5–3.3 ms. - **Docs fixed:** README/AGENTS still claimed a flat "20 s per pass" while `host_agent.py` (in this same uncommitted change) scales the budget to 5 s × disk count — corrected in both plus the two illustrative "20 s of wedged disk" sentences; broken bold (`` `driveripper** ``) repaired; stale gap-4 title + "tuning is its own phase" wording updated to say phase 14 ran; README's absolute "nothing … is answered by a whole-table scan" qualified to *per call* with the cold-window cost named; "third opt-in marker" → second; conftest `--store-perf` help named the real marker; **`data/logs/watch-store-perf.py` moved to `scripts/watch-store-perf.py`** (both docs told a clone to run a gitignored path) with root-anchored paths, `DASH_WATCH_OUT`, and dead code removed. - **New finding recorded:** gap 9's "~10 s process-wide stall is still gap 4's / `14_store_performance`'s problem" is now closed — coincident >8 s `llm` + >12 s `ups` raw-stream gaps went 393 (00:00–12:00) and 198 (12:00–21:36) to **0** across 110+ min of the phase-14 store; written into AGENTS.md gaps 4 and 9 and the notes table. - **Deviations:** did *not* pull either deferred cost knob (`STATS_CACHE_TTL_SECONDS`, warming caches from the maintenance tick) — AGENTS.md records that decision and its reasoning; `uv pip list --outdated` still reports certifi/fastapi/starlette/websockets (pre-existing advisory, no dependency touched this phase, gate treats it as advisory). Overwrote nothing: task 04's watch preserved as `store-perf-watch.task04.*`, my two runs as `.verify25.*` / `.current-code.*`. - **Next pending phase:** `15_agent_mtls`, then `16_agent_smart_pass_budget` (owns the scaled-budget tests + its doc criterion, now partly pre-satisfied by this pass), `17_installer_device_filter`.
2026-10-19 23:44:35 -04:00
parent 1bbd868401
commit 569ca25fdb
49 changed files with 5072 additions and 124 deletions
@@ -0,0 +1,42 @@
 # Phase 16 — Pin the agent's scaled smart-pass budget
 **Source:** chat 2026-10-19 — the `driveripper` 16-drive host: the flat 20 s smart-pass
 budget truncates a 16-device pass (a real read takes ~30–80 s), and because the deadline
 resets each pass, the tail drives would have stayed `smart: null` every pass. The fix —
 budget = 20 s floor scaled to 5 s × disk count, an explicit override still winning — was
 applied to the working tree on 2026-10-19 (operator-confirmed, verified on 16/16 live
 drives). This phase pins that already-applied change with tests and documentation.
 **Story:** n/a
 **Context:** `agent/host_agent.py` (`_smart_pass` — the scaled `budget = max(SMARTCTL_BUDGET_SECONDS, self.smart_timeout_seconds * len(device_names)) when the default is in force, else the explicit override; the `pass budget of {budget:.0f}s exhausted` error line; the module docstring "5 s per device and a pass budget of 20 s scaled to the disk count"), `tests/test_agent.py::test_the_pass_budget_skips_the_rest_with_one_error` (sets `collector.smart_budget_seconds = 0.0` — the override-wins branch), `AGENTS.md` line 185 ("5 s per device / 20 s per pass / 240 s cache") and the `deploy/install-agent.sh` header comment (line 25 area, "bounded 5 s per device and 20 s per pass" if still present there), `README.md` line 94 ("20 s per pass, results cached 4 minutes")
 ## Objective
 The smart pass budget scale-up that is already in the working tree is pinned: a test proves the *default* scales (16 devices → 80 s deadline, one device → 20 s floor, the exhaustion line names the scaled figure), the override-wins branch stays covered, and every doc that still says "20 s per pass" says the floor-scaled truth. No new behaviour — the code change is already deployed on `driveripper` and verified live.
 ## Dependencies
 - `09_disks_smartctl` (complete) — the smart pass, its bounds, and `tests/test_agent.py`'s in-process agent harness
 ## Tasks
 1. `01_pin_the_scaled_budget.md` — the tests that the default budget scales with the disk count and that the error line names the scaled figure
 2. `02_docs_and_gate.md` — every "20 s per pass" mention in `AGENTS.md`/`README.md`/`agent/host_agent.py` says the floor-scaled truth; the gate is green
 ## Testing & Quality
 - Unit: `tests/test_agent.py` — the scaled default (a default-attribute collector over 16 device names exhausts only when the 80 s budget runs out; over one device the floor is 20 s; the exhaustion error line carries the scaled figure, not the floor)
 - Coverage: **>90%** on the modified code (the `_smart_pass` branch)
 - The phase is "Complete" only when the **entire** project test suite passes 100% (every test in the project, not just this phase's)
 ## Documentation & Consistency
 - The final pass reviews `README.md` and `AGENTS.md` in full against the project as it stands after this phase — fixing every inconsistency found, **including pre-existing ones not caused by this phase**.
 - Tasks update `README.md`/`AGENTS.md` immediately when their own changes are user-visible (the smart-pass bounds are a documented contract of `host_agent.py`).
 ## Completion Criteria
 - [ ] `uv run pytest tests/test_agent.py` green, including the new scaled-budget tests and the untouched override test
 - [ ] `grep -rn "20 s per pass" AGENTS.md README.md agent/ deploy/` returns nothing (the wording is floor-scaled everywhere)
 - [ ] `bash .agents/validate.sh` green (pytest --cov --cov-fail-under=94, ruff, pyright, node --check)
 - [ ] README.md and AGENTS.md are accurate and consistent with the whole project (verified in the final pass)
 - [ ] no behavior change in completed work: the deployed agent on `driveripper` is byte-identical to the repo's `agent/host_agent.py` (md5), the live suite is green
@@ -0,0 +1,26 @@
 # Task 01 — Pin the scaled smart-pass budget
 **Phase:** `16_agent_smart_pass_budget` · **Story:** n/a
 ## Objective
 The default smart-pass budget (20 s floor, scaled to 5 s × disk count) is proven by tests: the deadline actually moves with the disk count, the exhaustion error line names the *scaled* figure, and the existing override-wins branch (a test sets `smart_budget_seconds = 0.0`) keeps passing untouched. The code change is already in `agent/host_agent.py::_smart_pass` — this task adds the tests, it does not rewrite the code.
 ## Work
 1. `tests/test_agent.py` — add a helper that builds the in-process collector with **default** smart attributes (do not touch `smart_budget_seconds`/`smart_timeout_seconds`) and a `smartctl` fake that records each device's deadline window, in the same style as `test_the_pass_budget_skips_the_rest_with_one_error` (which stays untouched):
   - `test_the_pass_budget_scales_with_the_disk_count`: 16 device names → the last device's `smartctl` run happens with `left` (its deadline margin) consistent with a ~80 s budget (assert the deadline the code computes: `max(20, 5 × 16)`; the fake binary is instant, so assert via the recorded `min(left, 5.0)` timeout being 5.0 for a late device and the pass *not* truncating — every device got a `smartctl` run); 1 device name → budget 20 s (assert the recorded timeout is 5.0 and, with the fake made slow enough to exhaust a 20 s but not an 80 s budget in a *separate* case if feasible — if the harness cannot fake wall time, assert the computed `budget` expression via the error-line route instead: force exhaustion with a `time.monotonic` monkeypatch offset).
   - `test_the_pass_budget_error_line_names_the_scaled_figure`: force exhaustion on 16 devices (monkeypatch the clock forward past the 80 s deadline between two device runs, as the existing test's `0.0` override does for its 0 s budget) → the error line reads `smart: pass budget of 80s exhausted, N disk(s) skipped` — the scaled figure, not the 20 s floor.
 2. `agent/host_agent.py` — **no code change expected.** If a test exposes a real bug in the already-applied change (e.g. the `== SMARTCTL_BUDGET_SECONDS` default check), fix it minimally and record the diff in the task's completion note; do not redesign.
 ## Testing & Quality
 - Unit: the two new tests above + the untouched `test_the_pass_budget_skips_the_rest_with_one_error` (override wins: `0.0` still exhausts immediately on 3 devices with one error line)
 - Coverage: **>90%** on `agent/host_agent.py::_smart_pass`'s budget branch (the default-scaled path must now be exercised, not just the override)
 ## Completion Criteria
 - [ ] `uv run pytest tests/test_agent.py -q` green: the two new tests, the untouched override test, and every pre-existing agent test
 - [ ] the 80 s case asserts the *scaled* figure in the error line (a test that passed with a 20 s budget would fail on the scaled code — no tautological assertions)
 - [ ] `uv run ruff check agent tests` and `uv run pyright` clean
 - [ ] no behavior change in completed work: `agent/host_agent.py` unchanged unless a real bug was found (then the fix is minimal and recorded)
@@ -0,0 +1,25 @@
 # Task 02 — Docs and gate for the scaled budget
 **Phase:** `16_agent_smart_pass_budget` · **Story:** n/a
 ## Objective
 Every place that still says the smart pass is "20 s per pass" says the floor-scaled truth (20 s floor, 5 s × disk count), and the phase gate is green. The agent's module docstring was already updated on 2026-10-19 to the scaled wording — this task verifies it and fixes the two remaining doc lines.
 ## Work
 1. `AGENTS.md` line ~185 (the `host_agent.py` bullet): `5 s per device / 20 s per pass / 240 s cache` → `5 s per device / a pass budget of 20 s scaled to 5 s × disk count / 240 s cache` (keep the bullet's voice; one line).
 2. `README.md` line ~94: `20 s per pass, results cached 4 minutes` → `a pass budget of 20 s scaled to the disk count (5 s each), results cached 4 minutes` (one line).
 3. `agent/host_agent.py` module docstring — verify it reads `bounded 5 s per device and a pass budget of 20 s scaled to the disk count, cached 240 s`; fix only if it drifted.
 4. `bash .agents/validate.sh` — the four checks (pytest --cov --cov-fail-under=94, ruff, pyright, node --check) + repo hygiene, all `uv run`.
 ## Testing & Quality
 - No new logic in this task; the gate run is the check.
 - Coverage: **>90%** gate holds (task 01's tests carry the new branch).
 ## Completion Criteria
 - [ ] `grep -rn "20 s per pass" AGENTS.md README.md agent/ deploy/` returns nothing
 - [ ] `bash .agents/validate.sh` green
 - [ ] the two edited doc lines read the floor-scaled truth and nothing else changed in either file
@@ -0,0 +1,59 @@
 # Phase 17 — The installer's broken device filter and the kernel-gate variant
 **Source:** chat 2026-10-19 — the `driveripper` install: the template's
 `DevicePolicy=strict` + `DeviceAllow` lines produced a **deny-all device filter** on
 kernel 7.2 (systemd resolves `DeviceAllow` globs by name against `/proc/devices`, where
 this kernel lists NVMe as `blkext` (major 259), so `nvme*` matched nothing and the whole
 device namespace was locked). The operator removed the lines by hand; the unit now also
 runs `User=root` + `CapabilityBoundingSet=CAP_SYS_RAWIO CAP_SYS_ADMIN` (kernel 7.2's
 unprivileged-passthrough gate — `data/logs/NOTES-driveripper-agent.md`). The template
 still ships the broken lines, `User=$AGENT_USER`, and an empty bounding set, and re-running
 the installer on `driveripper` would **clobber the working unit** (unconditional
 `cat > "$UNIT"`). This phase makes the template the single source of the working unit.
 **Story:** n/a
 **Context:** `deploy/install-agent.sh` (`do_install` — the unconditional unit heredoc with `DevicePolicy=strict` + nine `DeviceAllow` lines, the `PrivateDevices=false` rationale comment block, `User=$AGENT_USER`, `CapabilityBoundingSet=`, the `--bind/--port/--token/--rotate-token/--status/--uninstall` flag set), `tests/test_deploy_scripts.py` (the fake-root install sandbox `test_install_path_installs_into_its_own_paths_and_prints_the_token_once`, the token-file permission test, the `bash -n` + function-naming pins), `AGENTS.md` (the fleet bullet "The deployed unit has since diverged from the template in the three ways the SMART block below records", gap 10's "the template change is its own phase"), `README.md` (the install section, line ~46 `sudo deploy/install-agent.sh --bind <addr> --port 9120`)
 ## Objective
 Re-running `deploy/install-agent.sh` on any host — including `driveripper`, with
 `--as-root --caps CAP_SYS_RAWIO,CAP_SYS_ADMIN --force` — produces the working unit: no
 `DevicePolicy`/`DeviceAllow` lines (device access falls back to DAC via the `disk` group
 + SELinux, the standard SMART-monitoring arrangement), the kernel-gate variant expressible
 by flags, and a re-run that would change an existing unit prints the diff and refuses
 without `--force`. The default install (dedicated user, empty bounding set) is otherwise
 byte-identical to what the template wrote before this phase.
 ## Dependencies
 - `09_disks_smartctl` (complete) — the smart pass the unit exists to serve
 - `15_agent_mtls` (todo) — **shared-file, run-order note**: 15's task 06 adds `--cert-bundle`/`RuntimeDirectory` to the same `do_install`; apply on top of the file state at the time it runs, never restore a line the other phase owns. If 15 lands first, its new lines survive this phase's filter-line removal untouched.
 ## Tasks
 1. `01_drop_the_broken_device_filter.md` — remove `DevicePolicy=strict` + all nine `DeviceAllow` lines and rewrite the rationale comment block
 2. `02_root_and_caps_flags.md` — `--as-root` and `--caps CAP1,CAP2` write the kernel-gate unit lines
 3. `03_no_silent_clobber.md` — a re-run that would change an existing unit prints the diff and refuses without `--force`
 4. `04_tests_docs_gate.md` — the unit pins in `tests/test_deploy_scripts.py`, `AGENTS.md`/`README.md` updated, gate green
 ## Testing & Quality
 - Integration: `tests/test_deploy_scripts.py` — the template unit carries **no** `DevicePolicy`/`DeviceAllow` line; a default-install sandbox unit is byte-identical to the pre-phase template unit minus the filter lines; `--as-root --caps CAP_SYS_RAWIO,CAP_SYS_ADMIN` writes `User=root` and `CapabilityBoundingSet=CAP_SYS_RAWIO CAP_SYS_ADMIN` exactly; the re-run guard refuses (exit 1, diff on stdout) and overwrites with `--force`; `bash -n` and the existing function-naming pin still pass
 - Live (task 04, operator-verified): `driveripper`'s deployed unit is reproduced by `sudo deploy/install-agent.sh --bind 10.1.0.126 --port 9120 --as-root --caps CAP_SYS_RAWIO,CAP_SYS_ADMIN --force` — `systemctl diff` empty, `CapEff` still `0000000000220000`, 16/16 drives still live; `deskwork`/`gateway` untouched (their units are not re-run)
 - Coverage: the script itself is pinned by text assertions + `bash -n` (house pattern; >90% applies to the new Python test code)
 - The phase is "Complete" only when the **entire** project test suite passes 100% (every test in the project, not just this phase's)
 ## Documentation & Consistency
 - The final pass reviews `README.md` and `AGENTS.md` in full against the project as it stands after this phase — fixing every inconsistency found, **including pre-existing ones not caused by this phase**.
 - Tasks update `README.md`/`AGENTS.md` immediately when their own changes are user-visible (the new flags are operator-facing; gap 10's "the template change is its own phase" becomes "done in phase 17" with the `driveripper` flag line).
 ## Completion Criteria
 - [ ] `grep -n "DevicePolicy\|DeviceAllow" deploy/install-agent.sh` returns nothing (the `PrivateDevices=false` line stays)
 - [ ] `sudo deploy/install-agent.sh --help` names `--as-root` and `--caps` with the kernel-gate sentence
 - [ ] a fresh sandbox install (fake root, no flags) produces a unit byte-identical to the pre-phase template unit minus the filter lines (pinned in `tests/test_deploy_scripts.py`)
 - [ ] a re-run that would change an existing unit exits 1 with the diff on stdout; with `--force` it overwrites
 - [ ] operator-verified on `driveripper`: the flags reproduce the working unit (`systemctl diff` empty, 16/16 drives live after restart)
 - [ ] `uv run pytest` green, `bash .agents/validate.sh` green
 - [ ] README.md and AGENTS.md are accurate and consistent with the whole project (verified in the final pass)
 - [ ] no behavior change in completed work: the default install's unit (minus the filter lines) and every existing flag behave as before; the live suite is green
@@ -0,0 +1,25 @@
 # Task 01 — Drop the broken device filter
 **Phase:** `17_installer_device_filter` · **Story:** n/a
 ## Objective
 The template unit no longer carries `DevicePolicy=strict` or any `DeviceAllow` line, and the comment block that justified them is rewritten to say why the *absence* is the fix. The `PrivateDevices=false` line stays (explicit, default behaviour, and the rest of the lockdown is unchanged).
 ## Work
 1. `deploy/install-agent.sh` — in the `do_install` unit heredoc: delete the `DevicePolicy=strict` line and all nine `DeviceAllow=…` lines (`/dev/null`, `/dev/zero`, `/dev/full`, `/dev/random`, `/dev/urandom`, `/dev/nvme*`, `/dev/sd*`, `/dev/vd*`, `/dev/sg*`).
 2. `deploy/install-agent.sh` — rewrite the comment block above `PrivateDevices=false` (the "The pseudo nodes are not decoration…" paragraph) to the new truth: **no device filter on purpose** — systemd resolves `DeviceAllow` globs by name against `/proc/devices` (fnmatch in `bpf_devices_allow_list_major`), where kernel 7.2 lists NVMe as `blkext` (major 259), so `nvme*` matched nothing and `DevicePolicy=strict` denied the *whole* device namespace (the `driveripper` install went deny-all, root included; `data/logs/NOTES-driveripper-agent.md`). Without the policy, device access falls back to DAC (the `disk` group the script already grants) + SELinux — the standard SMART-monitoring arrangement. Keep the one operational fact that survives: `PrivateDevices=false` is the default (a `true` sandbox with no `DeviceAllow` list is what dies at spawn with `status=208/STDIN` — keep that warning sentence, it is still why the line is written explicitly).
 3. `deploy/install-agent.sh` — the script's top-of-file header comment (line ~25: "the unit device-allows the disk nodes") → "the unit applies no device filter (kernel 7.2's `/proc/devices` name matching made the old `nvme*` glob deny-all); device access is DAC via the `disk` group + SELinux".
 ## Testing & Quality
 - Integration: `bash -n deploy/install-agent.sh` clean (the existing pin covers it); a quick sandbox run (the `test_install_path_installs_into_its_own_paths_and_prints_the_token_once` pattern, run manually) shows no `DevicePolicy`/`DeviceAllow` in the written unit — the formal pins are task 04, this task keeps the change small and reviewable.
 - Coverage: **>90%** n/a (shell script; the pins are text assertions per the house pattern).
 ## Completion Criteria
 - [ ] `grep -n "DevicePolicy\|DeviceAllow" deploy/install-agent.sh` returns nothing
 - [ ] `bash -n deploy/install-agent.sh` clean
 - [ ] a manual sandbox install (fake root, `--bind 127.0.0.1 --port 9199`) writes a unit with the filter lines gone, `PrivateDevices=false` present, and every other line byte-identical to the pre-phase template unit
 - [ ] the rewritten comment names the `/proc/devices`/`blkext` mechanism in one sentence (an executor reading only the script can understand *why* there is no filter)
@@ -0,0 +1,27 @@
 # Task 02 — `--as-root` and `--caps` flags
 **Phase:** `17_installer_device_filter` · **Story:** n/a
 ## Objective
 The kernel-gate variant of the unit (what `driveripper` runs) is expressible from the template: `--as-root` writes `User=root` instead of `User=$AGENT_USER`, and `--caps CAP1,CAP2` (comma-separated, e.g. `--caps CAP_SYS_RAWIO,CAP_SYS_ADMIN`) writes `CapabilityBoundingSet=CAP1 CAP2` instead of the empty `CapabilityBoundingSet=` line. Both default off; a flag without its friend is refused with a message.
 ## Work
 1. `deploy/install-agent.sh` — flag parsing: add `--as-root` (sets `AS_ROOT=1`) and `--caps <list>` (sets `CAPS="<list>"`, validates non-empty, rejects an empty list with exit 1) in the existing `case`/flag block, and `--help` text:
   - `--as-root      run the service as root (for a kernel-gated host: kernel 7.2 gates unprivileged smartctl passthrough behind capabilities, not uid — a root process under the default empty bounding set is capability-less, so this flag needs --caps to mean anything)`
   - `--caps LIST    CapabilityBoundingSet line for the unit (comma-separated, e.g. CAP_SYS_RAWIO,CAP_SYS_ADMIN); default: empty (capability-less service)`
 2. `deploy/install-agent.sh` — validation: `--as-root` without `--caps` → print "note: --as-root with the default empty CapabilityBoundingSet leaves the process capability-less (the kernel gates check capabilities, not uid); pair it with --caps for a kernel-gated host" and **still proceed** (a capability-less root is a legitimate, if unusual, choice — do not hard-refuse; the note is the operator's reminder). `--caps` without `--as-root` → refuse, exit 1 ("--caps needs --as-root: a capability grant on the service user is not the kernel-gate fix").
 3. `deploy/install-agent.sh` — the unit heredoc: `User=$AGENT_USER` → `User=$( [ "$AS_ROOT" -eq 1 ] && echo root || echo "$AGENT_USER" )`; the empty `CapabilityBoundingSet=` line → the `CAPS` list converted comma→space when set. Add one comment line above: `# --caps: kernel 7.2 gates unprivileged smartctl passthrough (NVMe admin log: CAP_SYS_ADMIN; SG_IO allow-list: CAP_SYS_RAWIO); a root process needs them named, because an empty bounding set bounds even root to CapEff 0`.
 4. `deploy/install-agent.sh` — the `--status` probe is unchanged (it probes the agent over HTTP; the user the unit runs as is invisible to it). The token file stays `0640 root:$AGENT_USER` — with `--as-root` the group read is irrelevant (root reads it), so do **not** change the token-file logic.
 ## Testing & Quality
 - Integration: manual sandbox runs (fake root): default → `User=dashboards-agent` + empty `CapabilityBoundingSet=`; `--as-root --caps CAP_SYS_RAWIO,CAP_SYS_ADMIN` → `User=root` + `CapabilityBoundingSet=CAP_SYS_RAWIO CAP_SYS_ADMIN`; `--caps` alone → exit 1 with the message; `--as-root` alone → proceeds with the note on stdout. (Formal pins: task 04.)
 - Coverage: **>90%** n/a (shell script; text pins per the house pattern).
 ## Completion Criteria
 - [ ] `sudo deploy/install-agent.sh --help` names both flags with the kernel-gate sentence (a fresh clone's executor can learn the `driveripper` recipe from `--help` alone)
 - [ ] the two sandbox units match the exact lines above (byte-for-byte on the `User=` and `CapabilityBoundingSet=` lines)
 - [ ] `bash -n deploy/install-agent.sh` clean; every pre-existing flag (`--bind/--port/--token/--rotate-token/--status/--uninstall`) behaves as before in a sandbox run
@@ -0,0 +1,27 @@
 # Task 03 — No silent clobber on re-run
 **Phase:** `17_installer_device_filter` · **Story:** n/a
 ## Objective
 A re-run that would change an already-installed unit stops: it prints the diff between the installed unit and the unit the script would write, and exits 1 without touching the file. `--force` proceeds with the overwrite. A re-run whose generated unit is identical to the installed one proceeds silently (idempotent re-runs stay the zero-noise path the script has always had).
 ## Work
 1. `deploy/install-agent.sh` — add a `--force` flag to the flag block (`FORCE=1`) and `--help` text: `--force         overwrite an existing unit even if the generated unit differs (prints the diff first)`.
 2. `deploy/install-agent.sh` — in `do_install`, immediately before the `cat > "$UNIT"` heredoc write: build the would-be unit into a temp file (the heredoc already exists — render it once to `$(mktemp)`), and if `"$UNIT"` exists and `! cmp -s "$UNIT" "$tmp_unit"`:
   - if `FORCE=1`: print the diff (`diff -u "$UNIT" "$tmp_unit"` on stdout) and proceed with the overwrite;
   - else: print `refusing to overwrite $UNIT (generated unit differs); run with --force to apply:` + the same diff, and `exit 1` **before** the token-file write, the `systemctl daemon-reload`, and the `systemctl enable --now` — a refused run must leave the host exactly as it was (token file, unit, and running service all untouched; note the token-reuse logic runs earlier in `do_install` and only *reads* the existing token, so ordering the guard before the unit write is enough).
   - identical unit → no output, normal proceed.
 3. `deploy/install-agent.sh` — `--uninstall` is unchanged (it already removes the unit deliberately; the guard applies to `do_install` only).
 ## Testing & Quality
 - Integration: manual sandbox sequence (fake root): install → modify the installed unit by one line → re-run with the same flags → exit 1, diff on stdout, installed unit byte-identical to the modified version, service still the old one; re-run with `--force` → diff printed, unit overwritten; re-run again unchanged → silent, exit 0. (Formal pins: task 04.)
 - Coverage: **>90%** n/a (shell script; text/behaviour pins per the house pattern).
 ## Completion Criteria
 - [ ] the three-way sequence above (refuse → force → silent) holds in the sandbox, with the token file and the running service untouched by a refused run
 - [ ] `--force` prints the diff *before* overwriting (an operator sees exactly what changed)
 - [ ] `bash -n deploy/install-agent.sh` clean; a fresh install (no existing unit) is unaffected by the guard
@@ -0,0 +1,34 @@
 # Task 04 — Pins, docs, gate
 **Phase:** `17_installer_device_filter` · **Story:** n/a
 ## Objective
 The template's new shape is pinned in `tests/test_deploy_scripts.py` (no filter lines, the flags' exact lines, the clobber guard's refuse/force/silent sequence), `AGENTS.md` and `README.md` name the `driveripper` recipe, and the gate is green. The operator-verified `driveripper` reproduction (the completion criterion in `00_phase.md`) is the last check.
 ## Work
 1. `tests/test_deploy_scripts.py` — extend the existing sandbox pattern:
   - `test_the_template_unit_has_no_device_filter_lines` — read `deploy/install-agent.sh`'s unit heredoc region and assert no line starts with `DevicePolicy=` or `DeviceAllow=`, and `PrivateDevices=false` is still present (the filter's absence is the fix; a regression that re-adds `DeviceAllow=/dev/nvme*` is the deny-all returning).
   - `test_a_default_install_unit_is_the_pre_phase_unit_minus_the_filter_lines` — the sandbox-installed unit equals the known pre-phase template unit text minus the ten filter lines (pin the whole unit, so a future template line drift fails here).
   - `test_as_root_and_caps_write_the_kernel_gate_unit_lines` — `--as-root --caps CAP_SYS_RAWIO,CAP_SYS_ADMIN` → `User=root` and `CapabilityBoundingSet=CAP_SYS_RAWIO CAP_SYS_ADMIN` exactly; `--caps` alone → exit 1 with the `--as-root` message; `--as-root` alone → proceeds with the capability-less note on stdout.
   - `test_a_re_run_that_differs_refuses_without_force_and_applies_with_it` — the refuse/force/silent sequence (exit 1 + diff on stdout + installed unit untouched; `--force` overwrites with the diff printed; an unchanged re-run is silent).
 2. `AGENTS.md` — the fleet bullet's "re-running `deploy/install-agent.sh` on driveripper would clobber the working unit, because it rewrites the unit unconditionally (gap 10)" → the post-phase truth: the installer no longer writes the device-filter lines, and the `driveripper` unit is reproducible with `sudo deploy/install-agent.sh --bind 10.1.0.126 --port 9120 --as-root --caps CAP_SYS_RAWIO,CAP_SYS_ADMIN --force` (a re-run that would change an existing unit refuses without `--force`). Update gap 10's "the template change is its own phase" → "done in phase 17" with the flag line.
 3. `README.md` — the install section: one line naming the flags' purpose (the kernel-gated-host recipe), keeping the how-to voice.
 4. `bash .agents/validate.sh` — all four checks green.
 5. **Operator-verified (live, read-only + one restart):** on `driveripper`, `sudo deploy/install-agent.sh --bind 10.1.0.126 --port 9120 --as-root --caps CAP_SYS_RAWIO,CAP_SYS_ADMIN --force` → `systemctl diff dashboards-agent` empty, `systemctl is-active` `active`, `CapEff` still `0000000000220000`, and 16/16 drives still live in the next `/snapshot` (the restart is the only fleet action; `deskwork`/`gateway` are not re-run). Screenshot `data/logs/disks-17-installer-repro.png` per invariant 10.
 ## Testing & Quality
 - Integration: the four new pins above + the pre-existing pins (sudoers, `bash -n`, function naming, token-file permissions) all green.
 - Coverage: **>90%** on the new test code; the 94% gate holds.
 - Live: the operator-verified reproduction (item 5) — the phase's `driveripper` criterion.
 ## Completion Criteria
 - [ ] the four new pins pass and every pre-existing `test_deploy_scripts.py` pin passes
 - [ ] `grep -n "DevicePolicy\|DeviceAllow" deploy/install-agent.sh` returns nothing
 - [ ] `AGENTS.md` fleet bullet + gap 10 and `README.md` name the `driveripper` flag recipe
 - [ ] the operator-verified reproduction: `systemctl diff` empty, 16/16 drives live, screenshot in `data/logs/`
 - [ ] `uv run pytest` green, `bash .agents/validate.sh` green
 - [ ] the live suite (`set -a; . ./.env.dev; set +a; uv run pytest -m live --live`) is green
@@ -0,0 +1,9 @@
 **Phase 14 `store_performance` — final verification pass (all 4 tasks already in `complete/`)**
 - Re-verified every task's work in the tree: read caches (`series_names`/`stats`), capped compound `pick_res_many`, `history_many` + `api._history_batch`, windowed `maintain()` + per-resolution prune + `PRAGMA optimize`/`automatic_analyze=0`/WAL pragmas, `store_perf.BUDGETS` (14 ceilings), probe, live watch.
 - **Completion criteria:** baseline+after side by side in `data/logs/NOTES-store-perf.md` ✅ · budget tests green (`uv run pytest -m store_perf --store-perf` → **4 passed**, 14/14 inside) ✅ · 25-min dev-server watch re-run **independently** (74 passes, `scripts/watch-store-perf.py`): **0 `store-error`** (total still 190, newest 2026‑10‑18 06:33:30), WAL 7.77 MB ≤ 67.1 MB, medians 1.7–3.2 ms, p95 720/613 ms = the documented cold-cache window; a 3rd 27-pass watch on a freshly restarted server reproduced it (3.1/708, 1.8/572) ✅ · `maintain()` 5,368 ms → 2.7–6.8 ms and WAL bounded across two cycles (test + notes) ✅ · no behaviour change (all four history routes pinned against `query()`) ✅
 - **Full gates:** `uv run pytest` → **1267 passed, 15 skipped**; `bash .agents/validate.sh` → **exit 0**, coverage **98.75%** (floor 94), `ruff check src tests agent` clean, `pyright` 0 errors, `node --check` ok, repo hygiene ok; `uv run pytest -m live --live` → **11 passed**; dev server up, 9/9 sources `ok`, pages 200 in 1.5–3.3 ms.
 - **Docs fixed:** README/AGENTS still claimed a flat "20 s per pass" while `host_agent.py` (in this same uncommitted change) scales the budget to 5 s × disk count — corrected in both plus the two illustrative "20 s of wedged disk" sentences; broken bold (`` `driveripper** ``) repaired; stale gap-4 title + "tuning is its own phase" wording updated to say phase 14 ran; README's absolute "nothing … is answered by a whole-table scan" qualified to *per call* with the cold-window cost named; "third opt-in marker" → second; conftest `--store-perf` help named the real marker; **`data/logs/watch-store-perf.py` moved to `scripts/watch-store-perf.py`** (both docs told a clone to run a gitignored path) with root-anchored paths, `DASH_WATCH_OUT`, and dead code removed.
 - **New finding recorded:** gap 9's "~10 s process-wide stall is still gap 4's / `14_store_performance`'s problem" is now closed — coincident >8 s `llm` + >12 s `ups` raw-stream gaps went 393 (00:00–12:00) and 198 (12:00–21:36) to **0** across 110+ min of the phase-14 store; written into AGENTS.md gaps 4 and 9 and the notes table.
 - **Deviations:** did *not* pull either deferred cost knob (`STATS_CACHE_TTL_SECONDS`, warming caches from the maintenance tick) — AGENTS.md records that decision and its reasoning; `uv pip list --outdated` still reports certifi/fastapi/starlette/websockets (pre-existing advisory, no dependency touched this phase, gate treats it as advisory). Overwrote nothing: task 04's watch preserved as `store-perf-watch.task04.*`, my two runs as `.verify25.*` / `.current-code.*`.
 - **Next pending phase:** `15_agent_mtls`, then `16_agent_smart_pass_budget` (owns the scaled-budget tests + its doc criterion, now partly pre-satisfied by this pass), `17_installer_device_filter`.
@@ -0,0 +1 @@
 ok
@@ -0,0 +1,64 @@
 == uv run pytest --cov (hermetic: no network, live tests skip) ==
 ........................................................................ [  5%]
 ........................................................................ [ 11%]
 ........................................................................ [ 16%]
 ........................................................................ [ 22%]
 .............................................s.......................... [ 28%]
 ........................................................................ [ 33%]
 ........................................................................ [ 39%]
 ........................................................................ [ 44%]
 ........................................................................ [ 50%]
 ........................................................................ [ 56%]
 ........................................................................ [ 61%]
 ........................sssssssssss..................................... [ 67%]
 ........................................................................ [ 73%]
 ........................................................................ [ 78%]
 ........................................................................ [ 84%]
 ........................................................................ [ 89%]
 ........................................................................ [ 95%]
 sss.......................................................               [100%]
 =============================== warnings summary ===============================
 .venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
  /var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
    from starlette.testclient import TestClient as TestClient  # noqa
 -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
 ================================ tests coverage ================================
 _______________ coverage: platform linux, python 3.14.4-final-0 ________________
 Name                                 Stmts   Miss  Cover   Missing
 ------------------------------------------------------------------
 src/dashboards/__init__.py               3      0   100%
 src/dashboards/__main__.py              14      1    93%   32
 src/dashboards/alerts.py               176      0   100%
 src/dashboards/api.py                  356      0   100%
 src/dashboards/app.py                  178      0   100%
 src/dashboards/auth.py                 265      0   100%
 src/dashboards/collectors.py           155      5    97%   141-142, 146-147, 164
 src/dashboards/config.py               120      0   100%
 src/dashboards/debug.py                 14      0   100%
 src/dashboards/poller.py               165      3    98%   192-194, 279
 src/dashboards/prom.py                 112      5    96%   126, 163, 177, 180, 183
 src/dashboards/rates.py                 73      0   100%
 src/dashboards/severity.py              49      0   100%
 src/dashboards/sources/__init__.py       0      0   100%
 src/dashboards/sources/disks.py        130      0   100%
 src/dashboards/sources/host.py         199      1    99%   147
 src/dashboards/sources/llm.py          178      1    99%   285
 src/dashboards/sources/nut.py          413     20    95%   153, 172, 186, 198, 204, 217, 225, 241, 319-320, 322, 579, 591, 626, 657, 661, 718-719, 724, 736
 src/dashboards/sources/nut_sim.py      134      0   100%
 src/dashboards/store.py                313      2    99%   553-554
 ------------------------------------------------------------------
 TOTAL                                 3047     38    99%
 Required test coverage of 94% reached. Total coverage: 98.75%
 1267 passed, 15 skipped, 1 warning in 110.62s (0:01:50)
 == uv run ruff check src tests agent ==
 All checks passed!
 == uv run pyright ==
 0 errors, 0 warnings, 0 informations
 == node --check src/dashboards/web/static/*.js ==
 advisory: dependencies are out of date (AGENTS.md rule 8 — update on purpose):
  certifi    2026.7.22 2026.10.19 wheel
 fastapi    0.143.5   0.150.0    wheel
 starlette  1.8.0     1.9.0      wheel
 websockets 17.2      17.3       wheel
@@ -0,0 +1 @@
 Now let me write the baseline NOTES document.
@@ -0,0 +1 @@
 ok
@@ -0,0 +1,63 @@
 == uv run pytest --cov (hermetic: no network, live tests skip) ==
 ........................................................................ [  5%]
 ........................................................................ [ 11%]
 ........................................................................ [ 17%]
 ........................................................................ [ 23%]
 ........................................................................ [ 29%]
 ........................................................................ [ 35%]
 ........................................................................ [ 41%]
 ........................................................................ [ 47%]
 ........................................................................ [ 53%]
 ........................................................................ [ 59%]
 ........................................................................ [ 65%]
 ....................sssssssssss......................................... [ 71%]
 ........................................................................ [ 77%]
 ........................................................................ [ 83%]
 ........................................................................ [ 89%]
 ........................................................................ [ 95%]
 ...s.......................................................              [100%]
 =============================== warnings summary ===============================
 .venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
  /var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
    from starlette.testclient import TestClient as TestClient  # noqa
 -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
 ================================ tests coverage ================================
 _______________ coverage: platform linux, python 3.14.4-final-0 ________________
 Name                                 Stmts   Miss  Cover   Missing
 ------------------------------------------------------------------
 src/dashboards/__init__.py               3      0   100%
 src/dashboards/__main__.py              14      1    93%   32
 src/dashboards/alerts.py               176      0   100%
 src/dashboards/api.py                  348      0   100%
 src/dashboards/app.py                  178      0   100%
 src/dashboards/auth.py                 265      0   100%
 src/dashboards/collectors.py           155      5    97%   141-142, 146-147, 164
 src/dashboards/config.py               120      0   100%
 src/dashboards/debug.py                 14      0   100%
 src/dashboards/poller.py               165      3    98%   192-194, 279
 src/dashboards/prom.py                 112      5    96%   126, 163, 177, 180, 183
 src/dashboards/rates.py                 73      0   100%
 src/dashboards/severity.py              49      0   100%
 src/dashboards/sources/__init__.py       0      0   100%
 src/dashboards/sources/disks.py        130      0   100%
 src/dashboards/sources/host.py         199      1    99%   147
 src/dashboards/sources/llm.py          178      1    99%   285
 src/dashboards/sources/nut.py          413     20    95%   153, 172, 186, 198, 204, 217, 225, 241, 319-320, 322, 579, 591, 626, 657, 661, 718-719, 724, 736
 src/dashboards/sources/nut_sim.py      134      0   100%
 src/dashboards/store.py                188      2    99%   255-256
 ------------------------------------------------------------------
 TOTAL                                 2914     38    99%
 Required test coverage of 94% reached. Total coverage: 98.70%
 1199 passed, 12 skipped, 1 warning in 117.25s (0:01:57)
 == uv run ruff check src tests agent ==
 All checks passed!
 == uv run pyright ==
 0 errors, 0 warnings, 0 informations
 == node --check src/dashboards/web/static/*.js ==
 advisory: dependencies are out of date (AGENTS.md rule 8 — update on purpose):
  certifi    2026.7.22 2026.10.18 wheel
 fastapi    0.143.5   0.149.0    wheel
 starlette  1.8.0     1.9.0      wheel
 websockets 17.2      17.3       wheel
@@ -0,0 +1,17 @@
 **Task 02 — no full table scan on a per-request path: done, all green.**
 **Implemented**
 - `store.py`: `series_names()` cached 60 s (invalidated by `add`/`add_many`/`delete_series`/`maintain` via a generation counter; bounded to 256 `(source,host)` keys since `/api/metrics` takes both as free text); `stats()` counts cached 30 s with `last_maintain`/`size_bytes` deliberately never cached; `invalidate_read_caches()` for cold probes.
 - `pick_res()` now one compound statement of per-`res` `COUNT`s capped by `LIMIT max_points+1` (exact: the decision only asks `0` / `≤max_points` / neither), plus new `pick_res_many()` and `history_many()` (chunked at `COMPOUND_CHUNK`, index-range seek per branch).
 - `api.py`: all four history routes go through `_history_batch()` → `store.history_many()`; response dicts unchanged.
 - `tests/store_perf.py`: stats/`series_names` measured **cold and cached**, batch-vs-loop measured at 27 series × 24 h/90 d, EXPLAIN of the statement that actually runs.
 - Fixed a task-01 defect found in the full suite: the seeded determinism test pinned `SUM(value)`, whose float accumulation order changes when `INSERT OR REPLACE` moves rowids — now counts/windows exact, sum `rel=1e-9`, plus an exact PK-ordered row slice.
 - Docs: `AGENTS.md` (store.py/api.py bullets, gap 4 split into read-half-closed, commands), `README.md` (store.py, api.py, store_perf, new test pins), `data/logs/NOTES-store-perf.md` task-02 section (before/after, rejected shapes, EXPLAIN, live curl proof).
 **Measured** (`uv run pytest -m store_perf --store-perf`): HTTP `/api/health` median 1 272 ms → **1.2 ms**; `/api/metrics?source=host` 1 415 → **1.6 ms**; `/api/system/history` 3×9 at 90 d 100.3 → **49.8 ms**; `pick_res()`×3 3.0 → **0.3 ms**. Live dev server on the real 636 MB store: `/api/health` 0.700 s → **1.5 ms**, all nine sources `ok`.
 **Tests/lint** (exact): `bash .agents/validate.sh` → pytest `--cov --cov-fail-under=94` **1214 passed, 12 skipped, coverage 98.73%**; `ruff` All checks passed; `pyright` 0 errors; `node --check` ok. Final confirmation run `uv run pytest -q --cov --cov-fail-under=94` → **EXIT=0**, 0 FAILED (log `data/logs/pytest-final2.log`). No dependencies touched (validate's outdated-dep line is pre-existing advisory).
 **Deviations**: batched the resolution *decision*, not the row fetch — `ROW_NUMBER()` window (73 ms) and a compound of the 27 tail selects (38 ms) both measured slower than 27 index seeks (27 ms) on the seeded store, and task 01 requires the numbers not to regress; recorded with its table in the notes. The literal `SELECT res, COUNT(*) … GROUP BY res` shape was also measured slower (4.3 vs 2.7 ms) than the capped compound that shipped. No live-infra changes beyond `./scripts/dev-server.sh restart` (local dev, invariant 10).
 **Next pending task**: `.agents/phases/todo/14_store_performance/03_windowed_rollups_and_wal_tuning.md`.
@@ -0,0 +1 @@
 ok
@@ -0,0 +1,64 @@
 == uv run pytest --cov (hermetic: no network, live tests skip) ==
 ........................................................................ [  5%]
 ........................................................................ [ 11%]
 ........................................................................ [ 17%]
 ........................................................................ [ 23%]
 ........................................................................ [ 29%]
 ........................................................................ [ 35%]
 ........................................................................ [ 41%]
 ........................................................................ [ 46%]
 ........................................................................ [ 52%]
 ........................................................................ [ 58%]
 ........................................................................ [ 64%]
 .....................sssssssssss........................................ [ 70%]
 ........................................................................ [ 76%]
 ........................................................................ [ 82%]
 ........................................................................ [ 88%]
 ........................................................................ [ 93%]
 ..................s..................................................... [ 99%]
 ..                                                                       [100%]
 =============================== warnings summary ===============================
 .venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
  /var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
    from starlette.testclient import TestClient as TestClient  # noqa
 -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
 ================================ tests coverage ================================
 _______________ coverage: platform linux, python 3.14.4-final-0 ________________
 Name                                 Stmts   Miss  Cover   Missing
 ------------------------------------------------------------------
 src/dashboards/__init__.py               3      0   100%
 src/dashboards/__main__.py              14      1    93%   32
 src/dashboards/alerts.py               176      0   100%
 src/dashboards/api.py                  356      0   100%
 src/dashboards/app.py                  178      0   100%
 src/dashboards/auth.py                 265      0   100%
 src/dashboards/collectors.py           155      5    97%   141-142, 146-147, 164
 src/dashboards/config.py               120      0   100%
 src/dashboards/debug.py                 14      0   100%
 src/dashboards/poller.py               165      3    98%   192-194, 279
 src/dashboards/prom.py                 112      5    96%   126, 163, 177, 180, 183
 src/dashboards/rates.py                 73      0   100%
 src/dashboards/severity.py              49      0   100%
 src/dashboards/sources/__init__.py       0      0   100%
 src/dashboards/sources/disks.py        130      0   100%
 src/dashboards/sources/host.py         199      1    99%   147
 src/dashboards/sources/llm.py          178      1    99%   285
 src/dashboards/sources/nut.py          413     20    95%   153, 172, 186, 198, 204, 217, 225, 241, 319-320, 322, 579, 591, 626, 657, 661, 718-719, 724, 736
 src/dashboards/sources/nut_sim.py      134      0   100%
 src/dashboards/store.py                266      2    99%   445-446
 ------------------------------------------------------------------
 TOTAL                                 3000     38    99%
 Required test coverage of 94% reached. Total coverage: 98.73%
 1214 passed, 12 skipped, 1 warning in 122.10s (0:02:02)
 == uv run ruff check src tests agent ==
 All checks passed!
 == uv run pyright ==
 0 errors, 0 warnings, 0 informations
 == node --check src/dashboards/web/static/*.js ==
 advisory: dependencies are out of date (AGENTS.md rule 8 — update on purpose):
  certifi    2026.7.22 2026.10.18 wheel
 fastapi    0.143.5   0.149.0    wheel
 starlette  1.8.0     1.9.0      wheel
 websockets 17.2      17.3       wheel
@@ -0,0 +1,22 @@
 **Phase 14 / Task 03 — windowed rollups + WAL tuning: done.**
 Implementation (`src/dashboards/store.py`)
 - `maintain()` now windows both rollup steps off `last_maintain` (read via `get_meta()` *outside* the lock), overlapped by one bucket + `MAINTAIN_SLACK_SECONDS=60` and **snapped down to a bucket boundary** — a floor that cut into a bucket re-rolled a partial average from its tail rows (2.5 → 4.5); missing/non-numeric row falls back to full retention with a fixed-line warning.
 - Prune keeps its **full retention bound** but is now one `DELETE` per resolution: the old single `OR` of the three was unplannable (`SCAN sample`, 554 ms of a 562 ms pass while deleting nothing).
 - Statistics are the other half of the cost bound: `maintain()` ends with `PRAGMA optimize`, `__init__` sets `automatic_analyze=0` (SQLite's own refresh lands inside a poller's commit). WAL: `wal_autocheckpoint=1000`, `journal_size_limit=64 MB`; `busy_timeout=5000` deliberately unchanged.
 - Measured (seeded 6.4 M-row store): `maintain()` **5,368 ms → 2.7 ms** steady, 48 ms ten-minute catch-up; race at the app's cadence **100/100 inserts, 0 `database is locked`, max wait 21 ms** where the pre-fix control still waits 4.76 s; WAL 38 MB-forever → 0–4 MB. Live store: WAL 52.3 → 5.9 MB, `sqlite_stat1` created, real-store passes ~40 ms, `/api/health` 3.5 ms.
 Tests / gate
 - `tests/test_store.py`: +9 tests (window bounds from a trace callback, catch-up-not-a-gap, windowed-vs-full-window row-for-row equivalence, re-roll only gains rows, corrupt-row fallback, `maintain` leaves statistics, optimize failure is a log line not a failed tick, WAL pragmas, two-cycle WAL cap).
 - `tests/store_perf.py`: write passes and races moved to a `scratch_copy()` (baseline never moves), added `RACE_MAINTAIN_GAP_SECONDS` cadence races + the whole-window **control**, per-window plans derived from `ROLLUP_INSERT`.
 - `bash .agents/validate.sh`: **1240 passed, 12 skipped**, coverage **98.75%** (floor 94), ruff + pyright + `node --check` clean; `-m store_perf --store-perf` 1 passed; dev server restarted, all 9 sources `ok`.
 Docs
 - `AGENTS.md` (store.py bullet incl. the bucket-alignment rule, known gap 4 → write half closed, Commands, new testing-policy rule "a cost claim is pinned as a shape and a control"), `README.md` (store.py + `store_perf` paragraphs), `data/logs/NOTES-store-perf.md` (full task-03 tables, race-shape key, no-read-regression table, live smoke check).
 Deviations
 - The task's "windowing + WAL tuning" needed two extra fixes to actually reach the budget (prune `OR`, missing statistics); both are measured and recorded rather than assumed, and the prune's retention bound is unchanged.
 - The saturated task-01 race is kept for comparability but the pass line moved to the cadence-shaped race (a maintainer looping with no gap starves a writer at *any* pass length); the control beside it proves the race can still see the failure.
 - Left alone: `uv pip list --outdated` advisory (certifi/fastapi/starlette/websockets) — pre-existing, no dependencies touched.
 Next pending task: `.agents/phases/todo/14_store_performance/04_budget_gates_and_live_proof.md`.
@@ -0,0 +1 @@
 ok
@@ -0,0 +1,111 @@
 == uv run pytest --cov (hermetic: no network, live tests skip) ==
 ........................................................................ [  5%]
 ........................................................................ [ 11%]
 ........................................................................ [ 17%]
 ........................................................................ [ 23%]
 ........................................................................ [ 28%]
 ........................................................................ [ 34%]
 ........................................................................ [ 40%]
 ........................................................................ [ 46%]
 ........................................................................ [ 51%]
 ........................................................................ [ 57%]
 ........................................................................ [ 63%]
 .....................sssssssssss........................................ [ 69%]
 ........................................................................ [ 74%]
 ........................................................................ [ 80%]
 ........................................................................ [ 86%]
 ........................................................................ [ 92%]
 ..............F.............................s........................... [ 97%]
 ............................                                             [100%]
 =================================== FAILURES ===================================
 _______ test_a_second_maintain_reads_only_the_window_since_the_last_run ________
 db = <dashboards.store.SampleStore object at 0x7fe8d2126c50>
    def test_a_second_maintain_reads_only_the_window_since_the_last_run(db) -> None:
        """The cost change, pinned as a bound and as behaviour.
        A bucket that filled after the previous pass is inside the next window and gets rolled. A raw
        row that arrives for a bucket ten minutes behind it is not, and that bucket keeps the average
        the earlier pass computed — the documented cost of a window, stated here as a passing test
        rather than as a surprise. Reaching it in production costs a poller stall measured in minutes
        (the poller stamps samples with its own wall clock); the alternative it replaced was
        re-averaging 3.97 M rows every minute.
        """
        now = time.time()
        bucket_a = bucket_start(now - 10 * MINUTE)
        add_series(db, "host", "h", "m", [10, 20], start=bucket_a, step=1)
        db.maintain(now=now)
        assert db.query("host", "h", "m", res=RES_MIN, start=bucket_a - 1,
                        end=bucket_a + MINUTE) == [(bucket_a, 15.0)]
        second = now + 60
        bucket_b = bucket_start(second - 2 * MINUTE)
        add_series(db, "host", "h", "m", [40, 50], start=bucket_b, step=1)
        add_series(db, "host", "h", "m", [1000], start=bucket_a + 30, step=1)
        with query_log(db) as seen:
            counts = db.maintain(now=second)
        windows = rollup_windows(seen)
        # the bound: one minute (one hour) of rows plus the overlap, not 48 h of raw (not 30 d of 1m)
        assert windows[RES_RAW] == pytest.approx(
            (second - MINUTE - MAINTAIN_SLACK_SECONDS, bucket_start(now - MINUTE - MAINTAIN_SLACK_SECONDS)))
 >       assert windows[RES_MIN] == pytest.approx(
            (second - HOUR - MAINTAIN_SLACK_SECONDS, bucket_start(now - HOUR - MAINTAIN_SLACK_SECONDS)))
 E       assert (1792449272.5... 1792447200.0) == approx((17924....0 ± 1.8e+03))
 E         
 E         comparison failed. Mismatched elements: 1 / 2:
 E         Max absolute difference: 1980.0
 E         Max relative difference: 1.104635048664195e-06
 E         Index | Obtained     | Expected              
 E         1     | 1792447200.0 | 1792449180.0 ± 1.8e+03
 tests/test_store.py:802: AssertionError
 =============================== warnings summary ===============================
 .venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
  /var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
    from starlette.testclient import TestClient as TestClient  # noqa
 -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
 ================================ tests coverage ================================
 _______________ coverage: platform linux, python 3.14.4-final-0 ________________
 Name                                 Stmts   Miss  Cover   Missing
 ------------------------------------------------------------------
 src/dashboards/__init__.py               3      0   100%
 src/dashboards/__main__.py              14      1    93%   32
 src/dashboards/alerts.py               176      0   100%
 src/dashboards/api.py                  356      0   100%
 src/dashboards/app.py                  178      0   100%
 src/dashboards/auth.py                 265      0   100%
 src/dashboards/collectors.py           155      5    97%   141-142, 146-147, 164
 src/dashboards/config.py               120      0   100%
 src/dashboards/debug.py                 14      0   100%
 src/dashboards/poller.py               165      3    98%   192-194, 279
 src/dashboards/prom.py                 112      5    96%   126, 163, 177, 180, 183
 src/dashboards/rates.py                 73      0   100%
 src/dashboards/severity.py              49      0   100%
 src/dashboards/sources/__init__.py       0      0   100%
 src/dashboards/sources/disks.py        130      0   100%
 src/dashboards/sources/host.py         199      1    99%   147
 src/dashboards/sources/llm.py          178      1    99%   285
 src/dashboards/sources/nut.py          413     20    95%   153, 172, 186, 198, 204, 217, 225, 241, 319-320, 322, 579, 591, 626, 657, 661, 718-719, 724, 736
 src/dashboards/sources/nut_sim.py      134      0   100%
 src/dashboards/store.py                304      2    99%   510-511
 ------------------------------------------------------------------
 TOTAL                                 3038     38    99%
 Required test coverage of 94% reached. Total coverage: 98.75%
 =========================== short test summary info ============================
 FAILED tests/test_store.py::test_a_second_maintain_reads_only_the_window_since_the_last_run
 1 failed, 1239 passed, 12 skipped, 1 warning in 111.30s (0:01:51)
 == uv run ruff check src tests agent ==
 All checks passed!
 == uv run pyright ==
 0 errors, 0 warnings, 0 informations
 == node --check src/dashboards/web/static/*.js ==
 advisory: dependencies are out of date (AGENTS.md rule 8 — update on purpose):
  certifi    2026.7.22 2026.10.18 wheel
 fastapi    0.143.5   0.150.0    wheel
 starlette  1.8.0     1.9.0      wheel
 websockets 17.2      17.3       wheel
 validation FAILED (see output above)
@@ -0,0 +1,19 @@
 **Phase 14 / Task 03 — re-run after the harness failure: green.**
 What was fixed
 - The failing test was mine, and it was wall-clock luck, not the store: `test_a_second_maintain_reads_only_the_window_since_the_last_run` asserted the 1m→1h step's floor as a **minute**-aligned number while `_rollup_floor` snaps to an **hour**. `pytest.approx`'s relative tolerance (≈±1 800 s at epoch scale) covered the half of the hour where the two are within that, so it passed twice and failed at 19:4x.
 - Fix: `WINDOW_CLOCK_OFFSETS = (0, 1, 30, 59, 1 800, 3 539, 3 599)` over a base epoch that sits exactly on both boundaries — every edge is now a case instead of a coin flip — and each bound is snapped to *its own step's* bucket; the same parametrization went on the first-pass and catch-up tests so no window assertion in this task reads `time.time()`.
 - Found and fixed a second latent flake in the same test: the "due" bucket sat one step behind the ceiling, so a clock within a second of a boundary put the pair's second row outside the window and rolled 40.0 instead of 45.0 (now two steps back). `db.query()` defaults `end` to the wall clock — an explicit `end=` is now passed where the test runs on its own clock.
 - Added `test_the_window_floor_is_always_aligned_to_its_own_bucket` (aligned, never above the bound, never a whole window back, and the two steps really differ). Mutation-checked: removing the `floor(…/bucket)*bucket` snap fails 6 of 7 clock cases plus that test; restored code passes.
 Results
 - `bash .agents/validate.sh`: **1259 passed, 12 skipped, 0 failed** · coverage **98.75%** (floor 94, unchanged) · `ruff check src tests agent` all passed · `pyright` 0 errors · `node --check` ok · hygiene ok · no `validation FAILED`.
 - `uv run pytest -m store_perf --store-perf`: **1 passed** (window plans seek, catch-up ≤ 2× steady + slack, the whole-window control still contends, the cadence race is 0 `database is locked`).
 - Dev server healthy: all 9 sources `ok`, real-store WAL holding at ~7.6 MB under the 64 MB cap.
 Notable decisions
 - No production-code change this pass — the defect was in the test's expectation, so the implementation (windowed rollups, per-resolution prune, `PRAGMA optimize` + `automatic_analyze=0`, WAL pragmas, `busy_timeout` untouched) stands as measured: `maintain()` 5 368 ms → 2.7 ms steady / 48 ms catch-up.
 - Preferred explicit clocks and a property test over widening a tolerance, and recorded the flake plus both root causes in `data/logs/NOTES-store-perf.md` ("One window assertion was wall-clock luck").
 - Left alone: the pre-existing `uv pip list --outdated` advisory (certifi/fastapi/starlette/websockets) — no dependencies touched in this task.
 Next pending task: `.agents/phases/todo/14_store_performance/04_budget_gates_and_live_proof.md`.
@@ -0,0 +1 @@
 ok
@@ -0,0 +1,64 @@
 == uv run pytest --cov (hermetic: no network, live tests skip) ==
 ........................................................................ [  5%]
 ........................................................................ [ 11%]
 ........................................................................ [ 16%]
 ........................................................................ [ 22%]
 ........................................................................ [ 28%]
 ........................................................................ [ 33%]
 ........................................................................ [ 39%]
 ........................................................................ [ 45%]
 ........................................................................ [ 50%]
 ........................................................................ [ 56%]
 ........................................................................ [ 62%]
 .....................sssssssssss........................................ [ 67%]
 ........................................................................ [ 73%]
 ........................................................................ [ 79%]
 ........................................................................ [ 84%]
 ........................................................................ [ 90%]
 ...............................................................s........ [ 96%]
 ...............................................                          [100%]
 =============================== warnings summary ===============================
 .venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
  /var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
    from starlette.testclient import TestClient as TestClient  # noqa
 -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
 ================================ tests coverage ================================
 _______________ coverage: platform linux, python 3.14.4-final-0 ________________
 Name                                 Stmts   Miss  Cover   Missing
 ------------------------------------------------------------------
 src/dashboards/__init__.py               3      0   100%
 src/dashboards/__main__.py              14      1    93%   32
 src/dashboards/alerts.py               176      0   100%
 src/dashboards/api.py                  356      0   100%
 src/dashboards/app.py                  178      0   100%
 src/dashboards/auth.py                 265      0   100%
 src/dashboards/collectors.py           155      5    97%   141-142, 146-147, 164
 src/dashboards/config.py               120      0   100%
 src/dashboards/debug.py                 14      0   100%
 src/dashboards/poller.py               165      3    98%   192-194, 279
 src/dashboards/prom.py                 112      5    96%   126, 163, 177, 180, 183
 src/dashboards/rates.py                 73      0   100%
 src/dashboards/severity.py              49      0   100%
 src/dashboards/sources/__init__.py       0      0   100%
 src/dashboards/sources/disks.py        130      0   100%
 src/dashboards/sources/host.py         199      1    99%   147
 src/dashboards/sources/llm.py          178      1    99%   285
 src/dashboards/sources/nut.py          413     20    95%   153, 172, 186, 198, 204, 217, 225, 241, 319-320, 322, 579, 591, 626, 657, 661, 718-719, 724, 736
 src/dashboards/sources/nut_sim.py      134      0   100%
 src/dashboards/store.py                304      2    99%   510-511
 ------------------------------------------------------------------
 TOTAL                                 3038     38    99%
 Required test coverage of 94% reached. Total coverage: 98.75%
 1259 passed, 12 skipped, 1 warning in 112.04s (0:01:52)
 == uv run ruff check src tests agent ==
 All checks passed!
 == uv run pyright ==
 0 errors, 0 warnings, 0 informations
 == node --check src/dashboards/web/static/*.js ==
 advisory: dependencies are out of date (AGENTS.md rule 8 — update on purpose):
  certifi    2026.7.22 2026.10.18 wheel
 fastapi    0.143.5   0.150.0    wheel
 starlette  1.8.0     1.9.0      wheel
 websockets 17.2      17.3       wheel
@@ -0,0 +1,14 @@
 Task 04 is complete and everything is green.
 **What shipped**
 - `tests/store_perf.py` — `BUDGETS`: 14 pinned ceilings (2‑3× the measured "after"), `budget_rows`/``budget_violations`/`missing_budget_keys`, `measure_budget_reads/_http`, and the budget table rendered into the probe report.
 - Budget tests: 2 default-run pins (a budget whose measurement was renamed fails; the table is internally coherent) + 4 `store_perf` passes (reads, HTTP, `maintain`, race). `uv run pytest -m store_perf --store-perf` → 4 passed, **14/14 budgets inside**.
 - **The watch found a real defect in task 02's fix** and I fixed it: the series cache was invalidated by *every* sample write, so on a store nine pollers write to it never served and `/api/metrics` paid the full `SELECT DISTINCT` scan (measured 535 ms/call; the scan alone is 572 ms on the live store). `store.py` now invalidates only the write that can change the answer (`_note_series_write` + a capped `_known_series` existence set; `DELETE` paths still invalidate unconditionally). Pinned by statement counts, not clocks, in `tests/test_store.py` and through the served endpoint in `tests/test_api.py`.
 - Live proof (`data/logs/watch-store-perf.py` → `store-perf-watch.{log,tsv,json}`, `store-perf-live.png`): 25 min / 74 passes / 296 signed-in requests on the operator's 636 MB store — **0 new `store-error` rows** (190 total, newest 2026‑10‑18 06:33), **WAL 6.5 → 7.77 MB** (cap 67 MB), 0 non‑200, medians 1.8–3.3 ms for `/api/health`, `/api/summary`, `/api/metrics` and a 3×9 90‑day `/api/system/history`.
 - Docs: `data/logs/NOTES-store-perf.md` gained the task‑04 section (baseline vs after side by side, EXPLAIN shapes, the watch table, the defect write-up); `README.md` and `AGENTS.md` updated (gap 4 now reads: closed, with the honest remainder).
 - Gates: `bash .agents/validate.sh` exit 0 — 1267 passed, 15 skipped, coverage 98.75%, ruff + pyright + `node --check` clean. Dev server restarted, all 9 sources `ok`.
 **Honest leftovers**
 1. The p95 is the cache window, not the request: on the live store a cold `series_names('host')` costs 599 ms and `stats()`' counts 729 ms, once per 60 s / 30 s (37 of 74 passes had one request >250 ms). Closing that needs a counts table or a bounded series index — task 03's note already names it, and it is a phase, not this one.
 2. `uv pip list --outdated` is non-empty (fastapi 0.143.5→0.150.0, starlette, certifi, websockets) — pre-existing drift, untouched here; rule 8 wants a deliberate decision.
 3. Per AGENTS rule 4 I did **not** modify `.agents/phases/todo/14_store_performance/04_...md` — say the word and I'll tick the completion criteria and move it to `complete/`.
@@ -0,0 +1 @@
 ok
@@ -0,0 +1,64 @@
 == uv run pytest --cov (hermetic: no network, live tests skip) ==
 ........................................................................ [  5%]
 ........................................................................ [ 11%]
 ........................................................................ [ 16%]
 ........................................................................ [ 22%]
 .............................................s.......................... [ 28%]
 ........................................................................ [ 33%]
 ........................................................................ [ 39%]
 ........................................................................ [ 44%]
 ........................................................................ [ 50%]
 ........................................................................ [ 56%]
 ........................................................................ [ 61%]
 ........................sssssssssss..................................... [ 67%]
 ........................................................................ [ 73%]
 ........................................................................ [ 78%]
 ........................................................................ [ 84%]
 ........................................................................ [ 89%]
 ........................................................................ [ 95%]
 sss.......................................................               [100%]
 =============================== warnings summary ===============================
 .venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
  /var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
    from starlette.testclient import TestClient as TestClient  # noqa
 -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
 ================================ tests coverage ================================
 _______________ coverage: platform linux, python 3.14.4-final-0 ________________
 Name                                 Stmts   Miss  Cover   Missing
 ------------------------------------------------------------------
 src/dashboards/__init__.py               3      0   100%
 src/dashboards/__main__.py              14      1    93%   32
 src/dashboards/alerts.py               176      0   100%
 src/dashboards/api.py                  356      0   100%
 src/dashboards/app.py                  178      0   100%
 src/dashboards/auth.py                 265      0   100%
 src/dashboards/collectors.py           155      5    97%   141-142, 146-147, 164
 src/dashboards/config.py               120      0   100%
 src/dashboards/debug.py                 14      0   100%
 src/dashboards/poller.py               165      3    98%   192-194, 279
 src/dashboards/prom.py                 112      5    96%   126, 163, 177, 180, 183
 src/dashboards/rates.py                 73      0   100%
 src/dashboards/severity.py              49      0   100%
 src/dashboards/sources/__init__.py       0      0   100%
 src/dashboards/sources/disks.py        130      0   100%
 src/dashboards/sources/host.py         199      1    99%   147
 src/dashboards/sources/llm.py          178      1    99%   285
 src/dashboards/sources/nut.py          413     20    95%   153, 172, 186, 198, 204, 217, 225, 241, 319-320, 322, 579, 591, 626, 657, 661, 718-719, 724, 736
 src/dashboards/sources/nut_sim.py      134      0   100%
 src/dashboards/store.py                313      2    99%   553-554
 ------------------------------------------------------------------
 TOTAL                                 3047     38    99%
 Required test coverage of 94% reached. Total coverage: 98.75%
 1267 passed, 15 skipped, 1 warning in 113.83s (0:01:53)
 == uv run ruff check src tests agent ==
 All checks passed!
 == uv run pyright ==
 0 errors, 0 warnings, 0 informations
 == node --check src/dashboards/web/static/*.js ==
 advisory: dependencies are out of date (AGENTS.md rule 8 — update on purpose):
  certifi    2026.7.22 2026.10.19 wheel
 fastapi    0.143.5   0.150.0    wheel
 starlette  1.8.0     1.9.0      wheel
 websockets 17.2      17.3       wheel
@@ -181,15 +181,16 @@ Responsibilities, one line each:
  `offline_uncorrectable`, hex-packed vendor raws are never decoded, and an attribute row is
  read from its **RAW_VALUE column** rather than the end of the line because this binary
  annotates that column — `35 (Min/Max 34/44)`, `43894h+43m+32.519s` — so its last token is
-  not the value), 5 s per device /
+  not the value), 5 s per device / a pass budget whose 20 s floor scales to 5 s × the disk count
-  20 s per pass / 240 s cache, attached as `disk_io.devices[].smart`. Missing binary →
+  / 240 s cache, attached as `disk_io.devices[].smart` (gap 10: a 16-drive shelf takes longer
  than the floor, and truncating the pass would null the tail drives). Missing binary →
  **one** `errors` line for the pass (the word `not found` in it is what
  `sources/disks.py` reads to answer "is smartctl installed", and
  `tests/test_agent.py` pins that the line still says it); one denied/wedged device → that
  disk's `smart` is `null` with its own line; `--no-smart` opts out silently. Never fails
  the snapshot. **An expired cache refreshes on a background thread while the request is
-  served the previous result** — only the first pass after startup runs inline — because 20 s
+  served the previous result** — only the first pass after startup runs inline — because a wedged
-  of wedged disk on the request path blows `HOST_TIMEOUT_SECONDS` and would turn the
+  pass on the request path blows `HOST_TIMEOUT_SECONDS` and would turn the
  *system* source red every four minutes over an add-on that page never draws. The refresher
  is single-flight and keeps the old cache on failure, so a bad pass retries on the next
  expiry instead of hammering the disk.
@@ -230,7 +231,7 @@ Responsibilities, one line each:
  that was dialled, because they become health records and event rows. `fetch_snapshot()` is
  the one agent `/snapshot` read that `system` and `disks` share, so a dead host says the
  same sentence on both pages and only the `timeout` differs (the disks poll has to outlast
-  the agent's 20 s smartctl pass, and `None` is translated into httpx's sentinel because a
+  the agent's smartctl pass, and `None` is translated into httpx's sentinel because a
  literal `None` means *no timeout at all*). Agent `errors`
  become the payload's `degraded` **except** the `smart:` add-on lines — a host without
  smartmontools or device read access is unconfigured, not broken (invariant 3); they stay
@@ -397,7 +398,57 @@ Responsibilities, one line each:
  call, because a reset that deletes a row it did not name deletes the operator's credential).
  `stats()` reads `last_maintain` through `get_meta()` too — outside its own lock, which
  `get_meta` takes itself and `threading.Lock` is not reentrant — while `maintain()`'s write
-  stays inline inside that method's transaction.
+  stays inline inside that method's transaction. **No per-request path may pay a whole-table
  scan** (phase 14 task 02): `series_names()` (1.4 s of `SCAN sample` on a production-sized store,
  behind `/api/metrics`) and the two `COUNT`s behind `stats()` (1.25 s, run by `app.js`'s
  `/api/health` poll on every tab every 15 s) answer from short-TTL caches — 60 s and 30 s, the
  series list additionally invalidated **only by a write that can change it** (task 04: an insert
  cannot remove a name, so `_note_series_write()` ends the cache on a `(source, host, metric)` the
  store has never written and leaves a poll of an existing series alone — invalidating on every
  sample was task 02's defect, and it put the 572 ms scan back on nearly every `/api/metrics` call
  on a store nine pollers write to, behind a cache that never served; `delete_series()`/`maintain()`
  invalidate unconditionally because a `DELETE` can take a name, `delete_series()` also forgetting
  the triples it emptied so a re-created series counts as new again), through a generation counter so
  a write racing a read cannot park a stale answer under a fresh timestamp, with the name cache
  bounded to 256 `(source, host)` keys because `/api/metrics` takes both as free text and the
  existence tracking capped at `SERIES_TRACKED_SERIES_MAX`, below which it falls back to
  invalidating on every write. A tracking set made wrong by *another* process deleting rows costs at
  most one TTL window, which is the reason the cache has a TTL and not only the tracking. `last_maintain` and `size_bytes` are never
  cached: a `last_maintain` that stops moving is how an operator spots the invariant-4 failure
  mode, and it costs a three-row lookup. The resolution decision is one compound statement of
  per-`(host, metric, res)` branches whose `COUNT` is capped by `LIMIT max_points + 1` — exact,
  because the decision only ever asks whether a count is `0`, `≤ max_points`, or neither — and
  `pick_res_many()`/`history_many()` are the batched door the N × M endpoints go through. The rows
  stay a per-series seek **measuredly**: a `ROW_NUMBER()` window or a compound of the 27 tail
  selects both cost more than 27 index seeks on the seeded store (`data/logs/NOTES-store-perf.md`
  records the table), so batching the *decision* is the fix and batching the *fetch* would be a
  round trip traded for work.
  **The write side is windowed too** (phase 14 task 03): `maintain()` reads `last_maintain` through
  `get_meta()` *before* its transaction (same non-reentrant-lock rule as `stats()`) and bounds each
  rollup step to what became due since, instead of averaging every row the resolution retains —
  3.97 M rows of 1-minute history per minute, a 5.4 s write transaction that made a poller's
  `add_many` outlive `busy_timeout=5000` (gap 4's `store-error` line). Two properties make that a
  cost bound and not a data change, and both are pinned: the lower bound reaches **past** the
  previous pass (one bucket plus slack, so ten quiet minutes are a catch-up scan rather than ten
  minutes of empty charts), and it is snapped **down to a bucket boundary** — the step's ceiling is
  a row bound, so the newest bucket is partial and a later pass re-rolls it as its rows land, and a
  floor that cut into a bucket would overwrite the fuller average with the mean of its tail rows
  (2.5 becoming 4.5 on a six-sample minute: rule 2 with a chart's clothes on). A window is only a
  cost bound if the plan can seek to it, and two things were in the way. The prune keeps its full
  retention bound — a rollup may window because it only adds rows, a prune must reach anything that
  expired — but it is **one `DELETE` per resolution**, because one
  `WHERE (res=? AND ts<?) OR (res=? AND ts<?) OR (…)` is unreadable to the planner and read the
  whole 6.4 M-row table to delete nothing (554 ms of a 562 ms pass; 0.3 ms per statement now). And
  the store had no query statistics, so every window was priced as a scan however few rows were
  inside it: `maintain()` ends with `PRAGMA optimize` and `__init__` sets `automatic_analyze=0`,
  because SQLite's own refresh lands inside whichever poller's commit trips its counter — the exact
  path this shortens — while a tick a minute costs ~100 ms once and 0.0 ms afterwards. Measured on
  the seeded store: **5,368 ms → 2.7 ms** steady, 48 ms for a ten-minute catch-up, and the
  two-connection race at the app's cadence goes from 5 of 9 inserts raised to 100 of 100 landing
  with a 21 ms worst wait. `__init__` also sets `wal_autocheckpoint` (SQLite's own 1 000 pages,
  written down so the knob is visible) plus `journal_size_limit = 64 MB`, the cap that truncates a
  runaway WAL at the next checkpoint. `busy_timeout=5000` is unchanged: the fix is a shorter write,
  not a longer wait.
 - `api.py` — GETs everywhere plus the one POST that dials an upstream, and the `auth_router()`
  that creates, describes, **re-credentials**, and ends a session (`POST /api/auth/login` public;
  `GET /api/auth/status`, `POST /api/auth/logout` and `POST /api/auth/password` protected). Those
@@ -441,6 +492,14 @@ Responsibilities, one line each:
  write of its own — a fourth dashboard must not quietly become a second one, so the app's
  only POST stays `/api/ups/command` (invariant 1, pinned by enumerating every served route's
  verbs).
  All four `history` routes reach the store through one door, `api._history_batch()` →
  `store.history_many()` (phase 14 task 02). That is a cost change and nothing else: the series
  list is host-major and metric-minor in request order, each entry the same
  `{source, host, metric, range_seconds, points}` document the per-series loop built, with the
  same empty-series answer for a name that has no rows. `tests/test_api.py` pins it for all four
  routes against points read straight out of the store with `query()` — a third path, in neither
  the loop nor the batch — so a future "optimisation" that moves a point, an order, or a key
  fails a test instead of redrawing a chart wrong.
  `/api/summary` is the overview's single call and the cheapest whole-fleet probe there is:
  the page renders only `alerts` plus `system.fleet`/`system.hosts`, while `disks.hosts`,
  `llm.engines`, `ups.ups`, the `states` maps and `recent_events` stay for programmatic
@@ -567,10 +626,11 @@ That scheme is the history API's key — treat it as a public interface.
  `--host 10.1.0.126 --port 9120 --token-file /etc/dashboards-agent.token` (`640
  root:dashboards-agent`, the mode `deploy/install-agent.sh` writes) with `User=dashboards-agent`,
  and `getent group disk` → `disk:x:6:dashboards-agent`.
-  **Two things about that install are not what this repo writes, and both cost SMART data on
+  **The deployed unit has since diverged from the template in the three ways the SMART
-  `/disks` (see the SMART block below): `systemctl show -p DevicePolicy dashboards-agent` answers
+  block below records** (the `DevicePolicy`/`DeviceAllow` lines removed, `User=root`,
-  `auto` with no `DeviceAllow` lines** (the installer's `strict` + node allow-list is not what is
+  `CapabilityBoundingSet=CAP_SYS_RAWIO CAP_SYS_ADMIN`) — re-running `deploy/install-agent.sh`
-  running there), **and its `smartctl` cannot read drives as that user.**
+  on driveripper would clobber the working unit, because it rewrites the unit unconditionally
  (gap 10).
  `/api/health` today reads **all nine sources `ok`**: `host:driveripper`, `host:deskwork`,
  `host:gateway`, `disks:driveripper`, `disks:deskwork`, `disks:gateway`, `nut`, `llm:gufo`,
  `llm:vllm`.
@@ -657,15 +717,29 @@ That scheme is the history API's key — treat it as a public interface.
  `nvme0n1`…`nvme7n1` + `sda`…`sdh`) and `smart` is non-null for the 8 NVMes, but every one of
  them carries only identity — `health`, `temp_c`, `power_on_hours`, `percentage_used` and the
  counters are all `null` — while the 8 spinning drives come back `smart: null` each with the
-  agent's own line `smart: sda: Try an additional '-d ata' or '-d sat' argument.` Measured
+  agent's own line `smart: sda: Try an additional '-d ata' or '-d sat' argument.` Pinned on
-  read-only on 2026-10-19: the same `smartctl -i -A -H /dev/nvme0n1` **as root** prints
+  2026-10-19 with read-only probes (`data/logs/NOTES-driveripper-agent.md`): the `disk`-group
-  `PASSED`, 55 °C and 14 % used, so the drives answer and the *agent user* is what is locked out
+  **open of the block nodes works** (agent user, `O_RDWR`), there is no SELinux denier (zero
-  — `/dev/nvme0`, `/dev/ng0n1` and `/dev/bsg/*` (the nodes NVMe admin and SATA pass-through
+  AVCs with `auditd` running), and the install's `DevicePolicy=strict`/`DeviceAllow` lines were
-  actually go through) are `crw------- root root`, and only `/dev/nvme0n1` / `/dev/sg0` carry the
+  removed from the unit (they matched nothing useful anyway — systemd resolves `DeviceAllow`
-  `disk` group the agent is in. So `/disks` shows driveripper's drive inventory with `n/a` values
+  globs by name against `/proc/devices`, where this kernel lists NVMe as `blkext`, so
-  and a why-line, which is the designed degradation, and **lighting it up is a new phase, not a
+  `nvme*` matches nothing). What locks the agent out is **kernel 7.2's unprivileged
-  token**: a udev rule granting those nodes, and/or `host_agent.smart()` learning the
+  passthrough gate**: NVMe admin via the block node is unprivileged-only for five identify
-  `-d sat` / `/dev/ngXnY` forms smartctl needs when auto-detection fails.
+  types; the SMART-log read needs the drive's ACEL to mark it supported (the Samsung 980
  PRO's 2020-era `5B2QGXA7` does not — `NVME_IOCTL_ADMIN_CMD: Permission denied`) or
  `CAP_SYS_ADMIN`, and SATA SMART behind a SAT layer is ATA pass-through, which the
  unprivileged `SG_IO` allow-list (`scsi_cmd_allowed()`: read-safe opcodes only) does not
  include — explicit `-d sat`/`-d ata` fails identically, so no `host_agent.smart()` retry
  form can reach it. `deskwork` reads live with the same user/unit because its drive's
  firmware does advertise the page. So `/disks` shows driveripper's drive inventory with
  `n/a` values and a why-line — the designed degradation, **resolved the same day (gap 10):
  the unit now runs `User=root` with
  `CapabilityBoundingSet=CAP_SYS_RAWIO CAP_SYS_ADMIN`, and all 16 drives read live** (the
  980 PRO's 59 °C / 6328 h / 14 % row, `nvme1n1` at 58 % wear, the red `13 uncorrectable`
  alarm chip on `nvme5n1`; verified in `data/logs/disks-driveripper-live.png`). A root
  process with an **empty** capability bounding set is still capability-less (`CapEff: 0`)
  because the kernel gates check capabilities, not uid — the named bounding set is what
  unblocked it.
  **Real data is in the repo now:** `tests/fixtures/smartctl_driveripper_<dev>.txt` is all 16
  drives (8 NVMe, 8 spinning) read through that sudo rule, and `tests/test_agent.py` parses
  every one — which is what found the parser's one real bug: this binary annotates the
@@ -681,9 +755,8 @@ That scheme is the history API's key — treat it as a public interface.
  *not* a sudo rule, because the agent must read the device itself — on an `agent/host_agent.py`
  from phase 09 or later (never redeploy without the grant: 16 permission-denied lines would age
  nothing but are pure noise, while with the grant they are 16 drives' worth of temperature,
-  hours and sector counters). **The agent and the `disk` group are now both in place on
+  hours and sector counters). **The agent and the `disk` group are both in place on
-  `driveripper`, and it still reads no SMART numbers** — the group only covers nodes that carry
+  `driveripper`** — the `disk` group alone is not enough on kernel 7.2 (the passthrough gate, see above), so the unit additionally runs as root with the two gate capabilities bounded (gap 10) — all 16 drives now read live. `gateway` needs smartmontools first and will still read no SMART
  it (see above), so what is left is the node grant / `-d sat` work, not an install. `gateway` needs smartmontools first and will still read no SMART
  after that, being a VM with no physical disk behind `vda`. A host that cannot read its drives
  renders the designed degradation: one dimmed card per physical drive, every number `n/a`, and
  the one-line why.
@@ -711,9 +784,26 @@ set -a; . ./.env.dev; set +a              # dev dials the fleet the prod way, no
 ./scripts/dev-server.sh status            # per-source ok/DOWN — unauthenticated: /api/health
 ./scripts/dev-server.sh logs
-uv run pytest                             # hermetic: 1195 passed + 11 live-skipped, no network
+uv run pytest                             # hermetic: 1267 passed + 15 skipped (11 live,
-uv run pytest --cov                       # coverage gate (98.7%; floor 94%)
+                                          # 4 store_perf), no network
 uv run pytest --cov                       # coverage gate (98.75%; floor 94%)
 uv run pytest -m live --live              # opt-in: real fleet / engines / upsd
 uv run pytest -m store_perf --store-perf  # opt-in: the store hot-path timings over the seeded
                                          # store, priced against `store_perf.BUDGETS` (the cached
                                          # reads report cold *and* warm, so a before/after never
                                          # books a cache hit as a speed-up)
 uv run python scripts/watch-store-perf.py
                                          # ... and the live proof: 25 signed-in minutes against the
                                          # running dev server, timing the page paths while counting
                                          # `store-error` rows and the WAL size (DASH_WATCH_SECONDS
                                          # shortens it; writes store-perf-watch.{log,tsv,json})
 uv run python scripts/store_perf_probe.py # ... as a report (data/logs/store-perf-baseline.{md,json},
                                          # then store-perf-task03.* for the write side and
                                          # store-perf-task04.* for the budget table);
                                          # --rebuild re-seeds, --dev-db <file> also times a real
                                          # store read-only, DASH_PERF_STORE_DIR moves the cache, and
                                          # every pass that *writes* runs on a scratch copy so the
                                          # seeded baseline never moves under the next measurement
 uv run ruff check src tests agent && uv run pyright
 ./scripts/capture-fixtures.sh             # re-record ground truth (agents + UPS + gufo /slots)
 SMART_HOSTS="deskwork-ops driveripper-ops" ./scripts/capture-fixtures.sh
@@ -837,6 +927,28 @@ branch `main`); there is no CI on it, so `bash .agents/validate.sh` is the only
  verbatim, the four config rows, the `DASH_COOKIE_SECURE=1` deployment note, the three auth
  gaps). No server starts and no request is made: a doc that stops naming the reset command is a
  locked-out admin, and that is worth a test even though it asserts nothing about Python.
 - **A cost claim is pinned as a shape and a control, never as a wall clock** (phase 14). The default
  run keeps the seeded-store tests hermetic and asserts **shape and determinism** — counts,
  retention windows, the statements a plan is taken from, idempotence — while the timings live behind
  `-m store_perf --store-perf` and in `scripts/store_perf_probe.py`, because a budget asserted on
  first writing is a number fitted to the machine it was written on. Two rules came out of it.
  (1) A cheaper query is proven *equal*, not just faster:
  `test_windowed_rollups_answer_exactly_what_one_full_window_rollup_answers` compares every stored
  row of a windowed store against a second store rolled the old way, and
  `test_a_re_rolled_bucket_only_ever_gains_rows` pins the one bug that equivalence alone misses (a
  window floor that cuts into a bucket re-rolls it from its tail rows). (2) A race that comes back
  clean has to be shown capable of going dirty: `store_perf` runs the pre-fix whole-window
  `maintain()` as a **control** beside the windowed one and `tests/test_store.py` fails if the control
  stops contending, because "0 `database is locked`" from a race that never met is the invariant-4
  failure mode with a green checkmark attached. (3) **A budget is a table, not a test body**
  (task 04): the 14 ceilings live in `store_perf.BUDGETS` with the measured figure and the margin in
  each row's `basis`, the probe renders that table, and two default-run tests fail if a budget stops
  being priced or a ceiling contradicts its own ratio partner — a ceiling nobody measures is a
  comment. (4) A cache's cost claim needs the *write* cadence in the measurement: task 02's series
  cache reported 0.0 ms on a quiet seeded store and served almost nothing on a live one, because
  every sample write invalidated it. The pinned version is statement-counting, not timing — a poll of
  existing series issues no `SELECT`, a write of a name nobody wrote ends the cache — and the live
  version is `scripts/watch-store-perf.py`, which is where the defect showed up.
 - Coverage gate: 80% is the repo floor, the phase gate in `.agents/validate.sh` enforces
  94% (`--cov-fail-under`), and ~98% is where we are; keep it there. New/modified Python
  still has to clear >90% on its own. API tests use `create_app()` with an in-memory DB,
@@ -1221,7 +1333,7 @@ memory plot; the `cpu.usage_pct@coreN` series is still stored and still queryabl
   field map is written.
 3. **UPS history depth.** Events and the numeric series are stored; long-window battery
   health (capacity fade over months) has the rollups but no dedicated view.
-4. **The dev DB is big enough to refuse a write now and then.** The dev store (~640 MB on
+4. **The dev DB's `database is locked` line is fixed (phase 14); a cold-cache tail is left.** The dev store (~640 MB on
   2026-10-19, its WAL grown past 140 MB at its worst) logged `store-error: OperationalError:
   database is locked` a few times an hour across `system`/`llm`/`ups` through 2026-10-18; the
   poller handles it exactly as invariant 4 requires — the source goes red with the real reason,
@@ -1232,8 +1344,52 @@ memory plot; the `cpu.usage_pct@coreN` series is still stored and still queryabl
   phase 12's re-check on 2026-10-19 found the 190 stored `store-error` rows all older than
   2026-10-18 06:33 with none in the previous 24 h of the running dev store — the absence of a
   sample, not the absence of the pressure, since the file only keeps growing.
-   Retention/`wal_checkpoint` tuning for a long-running store is its own phase, not a
+   Retention/`wal_checkpoint` tuning for a long-running store was its own phase rather than a
-   phase-09 follow-up.
+   phase-09 follow-up — that phase is `14_store_performance`, and the two paragraphs below are
   what it measured and changed.
   **Phase 14 has since split this in two, and the read half is closed.** The in-process half was
   never a SQLite lock at all: `SampleStore._lock` serialises everything, so a 1.25 s `stats()`
   from a browser tab's `/api/health` poll was a poller waiting up to 1.45 s to insert 30 rows
   (measured, `data/logs/NOTES-store-perf.md`). Task 02 took the two whole-table reads off the
   per-request path, and `/api/health` now answers in ~1.2 ms instead of ~1.27 s, so the store
   mutex is no longer held by a reader for seconds at a time. The **write half is closed too**
   (task 03): `maintain()` rolls up only what became due since `last_maintain`, its prune is one
   `DELETE` per resolution instead of one `OR` the planner could not use, and `PRAGMA optimize` runs
   there (with SQLite's automatic ANALYZE switched off, so no poller's commit pays for it). On the
   seeded 6.4 M-row store that is 5,368 ms → **2.7 ms** steady / 48 ms for a ten-minute catch-up, and
   the two-connection race at the app's cadence is **100 of 100 inserts, zero `database is locked`,
   max wait 21 ms** where the pre-fix control still waits 4.76 s (task 04's re-run of the same
   budget: 6.8 ms steady / 46 ms catch-up / 8.5 ms worst wait, so the row is a range, not a
   digit). The WAL follows the same shape: a steady pass leaves ~0 MB and a catch-up tick 4 MB
   instead of 38 MB that never comes back down, and `journal_size_limit=64 MB` caps what a
   long-lived reader can make it grow to (`data/logs/NOTES-store-perf.md` has the tables).
   **The live proof is in** (task 04, 25 minutes / 74 passes / 296 signed-in requests against the
   running dev server on the operator's own 636 MB store): **0 new `store-error` rows** (still 190,
   newest 2026-10-18 06:33:30, i.e. 39 h before the watch), **WAL 6.5 → 7.77 MB** against the 67.1 MB
   cap and a main file that did not move, 0 non-200 responses, and page paths at 1.8-3.3 ms median
   (`/api/health`, `/api/summary`, `/api/metrics`, a 3-host × 9-metric 90-day `/api/system/history`).
   The watch also caught a defect in task 02's fix — the series cache was invalidated by *every*
   sample write, so on a store the hub writes to every few seconds it never served and `/api/metrics`
   paid the 572 ms scan on nearly every call (measured: 535 ms per call pre-fix, 1.8 ms median
   after) — and that is what the store bullet above now describes.
   What is **left** is the cost of the two cold answers themselves, measured on that store: 599 ms
   for `series_names('host')`, 729 ms for `stats()`' counts, now once per cache window (~1.3 s of
   store mutex per minute) instead of once per call — which is why the watch's medians are 1.8-3.3 ms
   and its p95s are 620/729 ms: the p95 is the window boundary, and 37 of 74 passes had one request
   over 250 ms, every one of them a cold re-derivation. No cache policy reaches *that*; the fix is
   the bounded `(source, metric)` structure or a counts table named in task 03's note. Two cheaper
   knobs exist and were deliberately not pulled, because both change what `/api/health` answers and
   move rather than remove the mutex hold: raising `STATS_CACHE_TTL_SECONDS`, and refreshing both
   caches from the maintenance tick so the miss lands on a poller instead of a page load. That is a
   phase, not a cleanup. Two servers on one file can still contend, and `busy_timeout=5000` is
   unchanged on purpose so a second one stays visible as a slower write; what they contend with is
   now milliseconds instead of 5.4 s.
   One consequence worth naming, because two other gaps pointed here for it: the **~10 s
   process-wide stall** (gap 9's `Δ`-window dilution, `poller`'s catch-up gaps) is gone with the
   mutex holds — coincident multi-second gaps in the 2 s `llm` and 5 s `ups` raw streams went from
   393 and 198 in the two halves of 2026-10-19 to **0** across the 110 minutes the phase-14 store
   has run (re-measured in this phase's verification pass from the `sample` table; the signature
   separates a process stall from one slow engine).
 5. **The login throttle keys on the socket peer address** (phase 11), and the one
   `LoginThrottle` instance now holds two keys per address: the bare one for
   `POST /api/auth/login` and `password:<address>` for `POST /api/auth/password`. Behind nginx
@@ -1315,7 +1471,14 @@ memory plot; the `cpu.usage_pct@coreN` series is still stored and still queryabl
   pass line and what the watch above stayed under. The `Δ<window>` label stayed honest throughout —
   both before and after the fix it names the window actually used; do not "fix" anything
   here by hiding or rewriting it. The ~10–15 s process-wide **stall** that triggers a
-   refused poll is still gap 4's and `14_store_performance`'s problem; the estimator no longer
+   refused poll was gap 4's and `14_store_performance`'s problem, and **phase 14 closed it**
   (measured on the dev store from the sample table: coincident multi-second gaps in the 2 s `llm`
   and 5 s `ups` raw streams — the signature of a process-wide stall, not a slow engine — were
   393 in 00:00–12:00 and 198 in 12:00–21:36 on 2026-10-19, and **0** across the 110 minutes the
   phase-14 store has been running, worst `llm` gap 7.1 s and that one the recorded gufo timeout).
   The mechanism is the one phase 14 removed: a whole-table `stats()` count held the store mutex
   for ~1.25 s on every browser tab's `/api/health` poll, and `maintain()` held the write lock for
   5.4 s. The estimator no longer
   turns a slow fetch into 0.2 s arithmetic, and one `n/a` per stall is the designed degradation
   (the residual the verification pass measured is the paragraph below).
   **The verification pass found one residual path and closed it** (same day, live rows in the
@@ -1341,11 +1504,47 @@ memory plot; the `cpu.usage_pct@coreN` series is still stored and still queryabl
   35.6 once the freed slot settled — every value labelled, **0 of 540 above the 300 pass line**,
   and no stored raw row over it since the restart either. A collapsed window is what makes a
   burst *look* like a rate, so the fix for this family is the ~10 s stall itself (gap 4 /
-   `14_store_performance`), not a fresh threshold invented in the estimator: task 03 pinned
+   `14_store_performance` — which has since removed it, see gap 4 and the paragraph above, so what
   is left of this family is the sub-8× skew itself), not a fresh threshold invented in the
   estimator: task 03 pinned
   `MIN_WINDOW_SECONDS` as an absolute floor and pinned that an in-bounds diluted row publishes
   (`test_a_fallback_inside_the_documented_ceiling_still_publishes`), so a span-relative floor
   would be a re-decision of that phase, not a bug fix. Do not "fix" the residual by hiding the
   short window or its label.
 10. **`driveripper`'s 16 drives read `n/a`: kernel 7.2's unprivileged-passthrough gate (2026-10-19).**
    The agent is installed, the `disk` group grant is in place, the block nodes open, and the
    page renders all 16 real drive rows — but kernel 7.2.11 blocks the *agent user's* `smartctl`
    at the command level (not the open; the probe matrix is in
    `data/logs/NOTES-driveripper-agent.md`): the 8 SATA drives' SMART is ATA pass-through
    behind a SAT layer, and the kernel's unprivileged `SG_IO` allow-list
    (`scsi_cmd_allowed()`, read-safe opcodes only) does not include it — `CAP_SYS_RAWIO` or
    root unblocks it, with or without `-d sat`/`-d ata`; the 8 Samsung/WD NVMes' SMART-log
    read needs the drive's ACEL to advertise the page (the 980 PRO's `5B2QGXA7`, 2020-era,
    does not: `NVME_IOCTL_ADMIN_CMD: Permission denied`) or `CAP_SYS_ADMIN`; `deskwork` reads
    live because its firmware does. Options, in order of narrowness: (a) accept the designed
    degradation (what `/disks` renders now — the identity is real, the values are `n/a` with a
    why-line); (b) `AmbientCapabilities=CAP_SYS_RAWIO` (+ `CapabilityBoundingSet`) on the unit
    — unblocks exactly the 8 HDDs, is the capability the kernel names for the gate, but a
    compromised agent could then write raw block I/O — test it in a `systemd-run --scope`
    before committing; (c) `User=root` with the empty capability set — unblocks all 16 but
    hands a token-guarded process root's file-read DAC (not recommended); (d) a Samsung
    firmware update — the only clean fix for the NVMe side. **Resolved the same day with a
    tighter variant of (c): `User=root` + `CapabilityBoundingSet=CAP_SYS_RAWIO CAP_SYS_ADMIN`
    — all 16 drives read live** (root DAC is the price; the bounding set keeps the grant
    named rather than full). Two empirical notes: a root process under the unit's *empty*
    bounding set is capability-less (`CapEff: 0`), because the kernel gates check
    capabilities, not uid — that is why the first `User=root` attempt still read `n/a`; and
    `CapEff` came back `0000000000220000` (SYS_ADMIN + systemd's auto CHROOT; RAWIO in the
    bounding set but not effective) — the 8 SATA drives' reads were unblocked by SYS_ADMIN
    alone, so the effective grant is {SYS_ADMIN}; the RAWIO in the line stays as the named
    intent for the `SG_IO` gate. The agent also gained the pass-budget scale-up (20 s floor
    → 5 s × disk count; an explicit override still wins, the budget test is untouched),
    because a 16-disk pass takes more than 20 s when the drives answer. The `deploy/install-agent.sh`
    template still ships the `DevicePolicy`/`DeviceAllow` lines that misbehaved on this host
    (a deny-all filter; the `nvme*` glob matches nothing on this kernel because
    `/proc/devices` lists NVMe as `blkext`) — the template change is its own phase.
    `tests/fixtures/smartctl_driveripper_*.txt` (captured via `sudo`) remain the ground truth
    for the drives' shape.
 ## Locked decisions (and why)
@@ -91,7 +91,8 @@ a labelled window, so the agent never has to remember who polled it last.
 Per-disk `smart` health is a best-effort add-on on the whole-disk `/sys/block` list (no
 partitions): the agent runs `smartctl -i -A -H` per physical disk — 5 s bound per device,
-20 s per pass, results cached 4 minutes — and parses health, temperature, power-on hours,
+a pass budget of 20 s as a floor that scales to 5 s × the disk count, results cached 4 minutes —
 and parses health, temperature, power-on hours,
 the sector alarms and SSD wear into `disk_io.devices[].smart`. Refreshing an expired cache
 happens on a background thread while the request is served the previous result, so a wedged
 disk can never cost a poll more than its own timeout (only the very first pass after the
@@ -334,7 +335,7 @@ documentation and is never loaded, and `scripts/dev-server.sh` exports `.env.dev
 | `HOST_INTERVAL_SECONDS` / `LLM_INTERVAL_SECONDS` / `NUT_INTERVAL_SECONDS` | how often each source is polled (10 / 2 / 5 s) |
 | `DISKS_INTERVAL_SECONDS` | how often the disk-health source re-reads each agent for SMART (default 300 s — SMART is not a per-second signal). No targets of its own: the `disks:<host>` jobs exist exactly when `HOST_TARGETS` does |
 | `HOST_TIMEOUT_SECONDS` / `LLM_TIMEOUT_SECONDS` / `NUT_TIMEOUT_SECONDS` | per-source fetch timeout — one slow source never holds up the others |
-| `DISKS_TIMEOUT_SECONDS` | the disks fetch's own budget (default 30 s), because the agent runs `smartctl` inside its snapshot handler and 20 s of wedged disk must not read as a dead host |
+| `DISKS_TIMEOUT_SECONDS` | the disks fetch's own budget (default 30 s), because the agent runs `smartctl` inside its snapshot handler and a wedged SMART pass must not read as a dead host |
 | `GUFO_API_KEY` / `VLLM_API_KEY`       | only if an engine is behind a bearer token; they never leave the server |
 | `NUT_HOST` / `NUT_PORT`               | upsd address — **whatever port it really listens on**, not always 3493. Reads are unauthenticated in stock NUT |
 | `NUT_USER` / `NUT_PASSWORD`           | only needed for instant commands (battery tests)                |
@@ -383,9 +384,27 @@ auth.py ── verifier + session key, loaded once at startup (no row and no see
  poller, so a SQLite write that gets refused is reported (`ok: false`, the real reason, the
  numbers aging, one line in the event log) instead of freezing the source under a green dot.
 * `store.py` — raw samples roll up to 1-minute and 1-hour averages and the raw rows are
-  pruned, so a year of history is a few hundred MB rather than a disk emergency. Its `meta`
+  pruned, so a year of history is a few hundred MB rather than a disk emergency. Nothing a
-  table is the one place state that must outlive a restart goes (`get_meta`/`set_meta`/
+  per-request path asks is answered by a whole-table scan **on every call** (phase 14): the two
-  `delete_meta`), and its first user is the admin credential below.
+  answers that used to be scanned on every call — `series_names()` behind `/api/metrics` and the
  row counts behind `/api/health` — come from a 60 s / 30 s cache, so the scan is paid once per
  window (its cost is the honest remainder in the known-gaps list) instead of once per request. `series_names`' is ended only by a write that can
  change it: a `(source, host, metric)` the store has never written, or any `DELETE`. A poll of an
  existing series leaves it alone, which is what makes the cache serve at all on a store nine pollers
  write to (task 04's live watch is the proof: 535 ms per call before, 1.8 ms median after), and the
  counts age on the clock. A chart's resolution comes from one capped compound statement instead of
  three full `COUNT`s per series. `last_maintain` and `size_bytes` are deliberately never cached —
  the first is how an operator tells a stalled rollup loop from a healthy one.
  The write side is windowed too: `maintain()` rolls up only the buckets that became due since its
  last run (read from `last_maintain`, overlapped by one bucket and snapped down to a bucket
  boundary, so a gap in polling is a catch-up scan rather than missing charts) and prunes on its
  full retention bound, one statement per resolution. It ends with `PRAGMA optimize` — the store's
  only statistics refresh, deliberately not inside a poller's commit — and the WAL is kept honest by
  `wal_autocheckpoint` plus a 64 MB `journal_size_limit` cap. On a 6.4 M-row store that took the
  once-a-minute write transaction from 5.4 s to a few milliseconds, which is what stopped a poller's
  insert losing its 5 s `busy_timeout` (known gap 4).
  Its `meta` table is the one place state that must outlive a restart goes (`get_meta`/
  `set_meta`/`delete_meta`), and its first user is the admin credential below.
 * `auth.py` — the admin login's decisions, in stdlib and without HTTP: a self-describing
  PBKDF2-HMAC-SHA256 verifier, an HMAC-signed stateless session cookie (no session table, and
  rotating the stored key is what evicts every other browser), a session-bound CSRF token, and
@@ -423,7 +442,12 @@ auth.py ── verifier + session key, loaded once at startup (no row and no see
  no env config answers `404 {"configured": false}` **without dialling** (so a page says
  "not configured" instead of going red), and a history query for a series that does not
  exist answers an empty series instead of an error (so a card asks for every plot it could
-  draw and reads `n/a` for the ones the drive never reported). `/api/disks/snapshot` is the
+  draw and reads `n/a` for the ones the drive never reported). The four `history` routes take
  one batched door into the store (`api._history_batch` → `store.history_many`), so a page's
  chart request is one grouped resolution query plus one index seek per series rather than four
  queries per series — and the JSON it answers with is the same document the per-series loop
  produced, series for series and in the same order (`tests/test_api.py` pins that against
  points read straight out of the store with `query()`). `/api/disks/snapshot` is the
  drive-health surface: one entry per host with its physical disk rows, the agent's `smart`
  block verbatim, `smartctl_available` (`true` / `false` = install smartmontools / `null` =
  the agent never said), a `severity` beside each row's rated numbers, and a `fleet` block
@@ -557,6 +581,11 @@ uv run pytest                                  # unit + API tests, fully mocked,
 uv run pytest --cov                            # coverage gate
 set -a; . ./.env.dev; set +a                 # the live suite reads the same env as the app
 uv run pytest -m live --live                   # optional: real fleet / engines / upsd
 uv run pytest -m store_perf --store-perf -s    # optional: the store's hot-path timings (phase 14)
 uv run python scripts/store_perf_probe.py      # the same measurements as a report in data/logs/
 DASH_WATCH_SECONDS=1500 uv run python scripts/watch-store-perf.py
                                               # ... and the live watch: 25 signed-in minutes against
                                               # the running dev server, counting store-error and WAL
 uv run ruff check src tests agent && uv run pyright            # lint + types
 ./scripts/capture-fixtures.sh                  # re-record ground truth: host agents, UPS replies, gufo /slots
 ./scripts/capture-fixtures.sh --with-llm        # ... and the engines' /metrics while they are up
@@ -611,6 +640,38 @@ local-testing word stays legal in prose, which is the only place it can be usefu
 the four public paths, the reset command, the four variables and the `DASH_COOKIE_SECURE=1`
 deployment note. Nothing starts a server there; a sentence that quietly disappears is the bug.
 `tests/store_perf.py` is the second opt-in marker's machinery (`live` is the other). It seeds a
 **production-shaped
 SQLite file** — `SampleStore`'s own retention (`raw 48 h / 1m 30 d / 1h 400 d`), the fleet's real
 metric names, the real 10 s cadence: ~6.4 M rows and ~720 MB, the dev DB's order of magnitude — and
 times every hot path of `store.py`/`api.py` against it, with `EXPLAIN QUERY PLAN` for the queries
 phase 14 changes and a two-connection race that reproduces the `database is locked` line in the
 known-gaps list — beside the **pre-fix control** of the same race, because a clean race that never
 met a lock would prove nothing. Every pass that *writes* (a rollup catch-up, a race) runs on a
 scratch copy of the seeded file, so the baseline the next measurement compares against comes back
 exactly as it went in. The default suite builds that store and asserts its **shape and determinism**
 (row counts, retention windows, rollups that are the mean of the raw rows they came from, a second
 `maintain()` that changes nothing) but never its wall time — except behind the marker, where
 `store_perf.BUDGETS` is the phase's pinned regression table (14 ceilings, 2-3x the measured "after",
 `maintain()` < 1 s, a racing insert at zero `database is locked`) and a stale budget pointer fails a
 default-run test. The cached reads are reported twice —
 **cold** (the cache invalidated before every run) and **cached** — because comparing a cache hit
 against the baseline's full scan would book a 1 000x speed-up that is really just a cache. Only
 `-m store_perf --store-perf` runs the timing passes, and `scripts/store_perf_probe.py` is the CLI
 form that writes `data/logs/store-perf-baseline.{md,json}` (task 03's write-side run is
 `store-perf-task03.*`, task 04's budget run `store-perf-task04.*`) for `data/logs/NOTES-store-perf.md`. Two rules keep it
 honest: **no test ever opens `data/dashboards.sqlite3`** (the `--dev-db` flag exists for a human
 running the probe read-only), and the seeded file is a cache under `data/perf/`
 (`DASH_PERF_STORE_DIR` relocates it) because a before/after has to measure the same file — pytest's
 `tmp_path` on this box is `tmpfs`, i.e. RAM, which times a store nobody deploys.
 The non-timing half of that lives in `tests/test_store.py` and runs every time: a cache hit
 issues **no** SQL at all (counted through `sqlite3`'s trace callback, never through a wall clock),
 the series list is invalidated by the write that introduces a name and not by the poll that repeats
 one (and the tracking that decides has a cap, below which it falls back to invalidating),
 `last_maintain` is never served from a cache, and `pick_res_many()`/`history_many()` answer exactly
 what the per-series calls answer — on a small store and on the seeded one, at every window.
 One more file is worth knowing about: `tests/card_contract.py` is the table of
 `/api/system/snapshot` paths the system page's per-host cards are allowed to read. It is
 checked against all three captured host shapes (discrete GPU, APU, no GPU), over HTTP, and
@@ -22,7 +22,8 @@ Design rules
 * **Every read is guarded.** A vanished /proc entry (process exited mid-scan) skips that
  item; the snapshot still succeeds. The poll is degraded, never failed.
 * **SMART is a best-effort add-on.** The per-disk `smart` block runs `smartctl -i -A -H`
-  in a subprocess: bounded 5 s per device and 20 s per pass, cached 240 s so a fast
+  in a subprocess: bounded 5 s per device and a pass budget of 20 s scaled to the disk
  count, cached 240 s so a fast
  dashboard poll cannot re-issue ATA commands every few seconds. Refreshing an expired
  cache happens on a background thread while the request is served the previous result —
  20 s of wedged disk must never cost a poll more than its own timeout. No binary, a denied
@@ -122,10 +123,12 @@ VIRTUAL_NET_PREFIXES = ("lo", "veth", "docker", "br-", "virbr", "br", "tap", "vn
 VIRTUAL_DISK_PREFIXES = ("loop", "ram", "zram", "sr", "dm-", "md127")
 # SMART collection bounds. 5 s per device is generous for a healthy drive and caps how
-# long one wedged device can hold a snapshot hostage; 20 s caps the whole pass so even a
+# long one wedged device can hold a snapshot hostage; the pass budget is a 20 s floor
-# shelf of bad disks cannot stall the HTTP handler; 240 s is below the dashboard's
+# scaled to the per-device bound (5 s x disk count), so a 16-disk shelf gets the 80 s a
-# 5-minute disks cadence, so the slow disks job gets fresh data while the fast system
+# real read takes while a single-disk host keeps 20 s and a shelf of bad disks cannot
-# poll never re-runs smartctl every 10 s.
+# stall the HTTP handler; 240 s is below the dashboard's 5-minute disks cadence, so the
 # slow disks job gets fresh data while the fast system poll never re-runs smartctl
 # every 10 s.
 SMARTCTL_TIMEOUT_SECONDS = 5.0
 SMARTCTL_BUDGET_SECONDS = 20.0
 SMARTCTL_CACHE_SECONDS = 240.0
@@ -522,7 +525,8 @@ class Collector:
        same error lines, so a snapshot always states why `smart` is null.
        An **expired** cache is refreshed on a background thread and the request is served
-        the previous result, because the pass budget (20 s) is five times the dashboard's
+        the previous result, because the pass budget (20 s floor, 5 s per disk) is five
        times the dashboard's
        `HOST_TIMEOUT_SECONDS` (4 s): run inline on the request path, one wedged disk would
        turn the *system* source red every `smart_cache_seconds` for a fault in an add-on
        (AGENTS.md invariant 4 — SMART must not blink the page that does not use it). The
@@ -569,14 +573,20 @@ class Collector:
        now = time.monotonic()
        results: dict[str, dict[str, Any] | None] = {}
        errors: list[str] = []
-        deadline = now + self.smart_budget_seconds
+        # 20 s floor scaled to the per-device bound: a 16-disk shelf takes more than 20 s
        # when the drives answer, and truncating the pass would null the tail drives.
        # An explicit override (the budget test sets this to 0) wins over the scale.
        budget = (max(SMARTCTL_BUDGET_SECONDS, self.smart_timeout_seconds * len(device_names))
                  if self.smart_budget_seconds == SMARTCTL_BUDGET_SECONDS
                  else self.smart_budget_seconds)
        deadline = now + budget
        for name in device_names:
            left = deadline - time.monotonic()
            if left <= 0:
                remaining = [n for n in device_names if n not in results]
                for skipped_name in remaining:
                    results[skipped_name] = None
-                errors.append(f"smart: pass budget of {self.smart_budget_seconds:.0f}s "
+                errors.append(f"smart: pass budget of {budget:.0f}s "
                              f"exhausted, {len(remaining)} disk(s) skipped")
                break
            try:
@@ -47,6 +47,11 @@ markers = [
    # `-m live --live` runs them; without --live they SKIP, so `uv run pytest` stays
    # network-free. See tests/test_live_fleet.py for the env vars each one needs.
    "live: needs real hardware (fleet hosts, LLM engines, or upsd) — opt in with --live",
    # `-m store_perf --store-perf` runs them: the phase-14 timing measurements over the seeded
    # production-shaped store (tests/store_perf.py). The seed itself runs in the default suite;
    # only the multi-minute measurement passes are behind this flag. Underscored because Python
    # decorators cannot carry a hyphen; the CLI flag stays `--store-perf`.
    "store_perf: measures the seeded store's hot paths — opt in with --store-perf",
 ]
 # auto: async tests run without a decorator on every single one
 asyncio_mode = "auto"
@@ -0,0 +1,95 @@
 #!/usr/bin/env python3
 """Measure the store's hot paths against a production-shaped SQLite file.
 Phase 14 exists because `TODO.md:7` says "pages take forever to load and seem to lock up the
 database". Task 01's job is the **before**: one run of this script is the baseline in
 `data/logs/NOTES-store-perf.md`, and task 04 runs it again as the "after". Nothing here changes
 behaviour — it reads, times, `EXPLAIN`s, and cleans up after itself.
    uv run python scripts/store_perf_probe.py                 # baseline (builds the store once)
    uv run python scripts/store_perf_probe.py --rebuild       # force a fresh seed
    uv run python scripts/store_perf_probe.py --dir /tmp/perf # measure on tmpfs (RAM: faster)
    uv run python scripts/store_perf_probe.py --out data/logs/store-perf-after
 It writes two files beside the numbers it prints: `<out>.json` (everything, machine-readable)
 and `<out>.md` (the same tables as markdown, to paste into the NOTES). The seeded store itself
 lives in `data/perf/` (gitignored, like the dev DB) and is reused while it is fresh — the same
 physical file has to be the one timed before and after the fixes, or the two numbers describe
 two databases.
 The shape, the timings and the machine specs all come from `tests/store_perf.py`, which is also
 what the pytest budget tests of this phase use: one definition of "production-shaped", because
 two of those drift.
 """
 from __future__ import annotations
 import argparse
 import sys
 import time
 from pathlib import Path
 ROOT = Path(__file__).resolve().parents[1]
 for entry in (ROOT / "tests", ROOT / "src"):
    # `tests/store_perf.py` is not a package (pytest imports it the same way via conftest), and
    # a bare checkout has no installed `dashboards` either. Both paths, or nothing works.
    if str(entry) not in sys.path:
        sys.path.insert(0, str(entry))
 import store_perf  # noqa: E402 - after the sys.path fixup above, by design
 def log(message: str) -> None:
    print(f"[{time.strftime('%H:%M:%S')}] {message}", flush=True)
 def main(argv: list[str] | None = None) -> int:
    parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
    parser.add_argument("--dir", default=None,
                        help="where the seeded store lives (default: $DASH_PERF_STORE_DIR "
                             "or data/perf — a real block device, unlike pytest's tmpfs)")
    parser.add_argument("--rebuild", action="store_true",
                        help="throw the cached store away and seed a new one")
    parser.add_argument("--out", default=str(ROOT / "data" / "logs" / "store-perf-baseline"),
                        help="prefix for the report files (default: data/logs/store-perf-baseline)")
    parser.add_argument("--race-iterations", type=int, default=100,
                        help="how many 30-sample inserts to race against stats()/maintain() "
                             "(task 01 says ~100)")
    parser.add_argument("--race-seconds", type=float, default=store_perf.MAX_RACE_SECONDS,
                        help=f"wall-clock budget per race pass (default "
                             f"{store_perf.MAX_RACE_SECONDS:.0f} s — one stats() pass costs "
                             "seconds on this store, so the pass is bounded by time as well as "
                             "by the insert count)")
    parser.add_argument("--dev-db", default=None,
                        help="also time the three hot reads against this *real* database file, "
                             "opened read-only (e.g. data/dashboards.sqlite3). Nothing here "
                             "writes to it and no test ever passes this flag — the hermetic "
                             "suite never opens the operator's store.")
    parser.add_argument("--no-http", action="store_true",
                        help="skip the TestClient pass (/api/health, /api/metrics, "
                             "/api/system/history) — store-only numbers")
    args = parser.parse_args(argv)
    path = store_perf.ensure_seeded(args.dir, force=args.rebuild, log=log)
    store = store_perf.open_store(path)
    try:
        report = store_perf.measure(store, race_iterations=args.race_iterations,
                                    race_max_seconds=args.race_seconds,
                                    with_http=not args.no_http, log=log)
    finally:
        store.close()
    if args.dev_db:
        log(f"read-only pass over the real store: {args.dev_db}")
        report["dev_db"] = store_perf.measure_dev_db(args.dev_db)
    written = store_perf.write_report(report, Path(args.out))
    print()
    print(store_perf.render_markdown(report))
    print()
    print(f"report: {written['md']}\n        {written['json']}")
    return 0
 if __name__ == "__main__":
    raise SystemExit(main())
@@ -0,0 +1,194 @@
 #!/usr/bin/env python3
 """Phase 14 task 04: watch the running dev server for `database is locked`.
 Sampled, not exhaustive: every ~20 s a signed-in pass hits the paths a page actually loads
 (``/api/health``, ``/api/summary``, ``/api/metrics``, a 3-host x 9-metric 90-day
 ``/api/system/history``) and records the wall time beside the store's own evidence — the
 ``store-error`` rows in the ``event`` table, the WAL file size, the database size.
 Run it in the background while the dev server keeps polling the fleet:
    set -a; . ./.env.dev; set +a
    setsid nohup uv run python scripts/watch-store-perf.py > /tmp/watch.out 2>&1 < /dev/null &
 Writes ``data/logs/store-perf-watch.log`` (human), ``store-perf-watch.tsv`` (machine) and
 ``store-perf-watch.json`` (the summary every path ran against). ``data/`` is gitignored, so those
 are local evidence for ``data/logs/NOTES-store-perf.md``, never tracked artifacts.
 """
 from __future__ import annotations
 import json
 import os
 import sqlite3
 import statistics
 import sys
 import time
 from pathlib import Path
 import httpx
 #: Repo root, so the paths below mean the same tree wherever the watch is invoked from
 #: (the same anchor ``scripts/store_perf_probe.py`` uses).
 ROOT = Path(__file__).resolve().parents[1]
 BASE_URL = os.environ.get("DASH_WATCH_URL", "http://127.0.0.1:8090")
 ADMIN_PASSWORD = os.environ.get("DASH_ADMIN_PASSWORD", "")
 DB_PATH = Path(os.environ.get("DASH_WATCH_DB", ROOT / "data" / "dashboards.sqlite3"))
 OUT = Path(os.environ.get("DASH_WATCH_OUT", ROOT / "data" / "logs"))
 DURATION_SECONDS = float(os.environ.get("DASH_WATCH_SECONDS", "1500"))
 INTERVAL_SECONDS = float(os.environ.get("DASH_WATCH_INTERVAL", "20"))
 #: What a page load may cost. Everything here is a panel the operator is looking at.
 PAGE_BUDGET_MS = 250.0
 METRICS = [
    "cpu.usage_pct", "cpu.temp_c", "cpu.load1", "mem.used_pct", "disk.used_pct@/",
    "disk.used_pct@/home", "net.rx_bps", "net.tx_bps", "gpu.busy_pct@0",
 ]
 def store_evidence(db: sqlite3.Connection, since: float) -> dict[str, float | int | str]:
    """The store's own record, read on a separate connection while the server runs."""
    total = db.execute("SELECT COUNT(*) FROM event WHERE kind='store-error'").fetchone()[0]
    recent = db.execute(
        "SELECT COUNT(*) FROM event WHERE kind='store-error' AND ts>=?", (since,)
    ).fetchone()[0]
    last = db.execute(
        "SELECT message FROM event WHERE kind='store-error' ORDER BY ts DESC LIMIT 1"
    ).fetchone()
    return {
        "store_errors_total": total,
        "store_errors_in_window": recent,
        "last_store_error": (last[0] if last else ""),
    }
 def timed(client: httpx.Client, path: str) -> tuple[float, int]:
    started = time.perf_counter()
    try:
        response = client.get(path)
        code = response.status_code
    except httpx.HTTPError:  # a refused request is a sample too
        return (time.perf_counter() - started) * 1000.0, -1
    return (time.perf_counter() - started) * 1000.0, code
 def main() -> int:
    started = time.perf_counter()
    OUT.mkdir(parents=True, exist_ok=True)
    log_path = OUT / "store-perf-watch.log"
    tsv_path = OUT / "store-perf-watch.tsv"
    json_path = OUT / "store-perf-watch.json"
    samples: list[dict[str, float | int | str]] = []
    client = httpx.Client(base_url=BASE_URL, timeout=60.0, follow_redirects=False)
    login = client.post("/api/auth/login", json={"username": "admin", "password": ADMIN_PASSWORD})
    if login.status_code != 200:
        print(f"login failed: {login.status_code} {login.text[:200]}", flush=True)
        return 1
    client.headers["X-CSRF-Token"] = login.json()["csrf"]
    hosts = [t["name"] for t in client.get("/api/config").json()["system"]["targets"]]
    host_query = "".join(f"hosts={h}&" for h in hosts) or "hosts=driveripper&"
    metrics_query = "".join(f"metrics={m}&" for m in METRICS)
    paths = {
        "health": "/api/health",
        "summary": "/api/summary",
        "metrics": "/api/metrics?source=host",
        "history_90d": f"/api/system/history?{host_query}{metrics_query}range=90d",
    }
    db = sqlite3.connect(f"file:{DB_PATH}?mode=ro", uri=True, timeout=30.0)
    started_wall = time.time()
    with log_path.open("a", buffering=1) as log:
        log.write(f"# watch start {time.strftime('%Y-%m-%dT%H:%M:%S')} "
                  f"budget={PAGE_BUDGET_MS:.0f} ms interval={INTERVAL_SECONDS:.0f} s "
                  f"duration={DURATION_SECONDS:.0f} s hosts={hosts}\n")
        while time.perf_counter() - started < DURATION_SECONDS:
            row: dict[str, float | int | str] = {
                "t": round(time.time(), 1),
                "elapsed_s": round(time.perf_counter() - started, 1),
            }
            for name, path in paths.items():
                ms, code = timed(client, path)
                row[f"{name}_ms"] = round(ms, 1)
                row[f"{name}_code"] = code
            wal = DB_PATH.with_name(DB_PATH.name + "-wal")
            row["wal_mb"] = round((wal.stat().st_size if wal.exists() else 0) / 1e6, 2)
            row["db_mb"] = round(DB_PATH.stat().st_size / 1e6, 1)
            try:
                row.update(store_evidence(db, started_wall))
            except sqlite3.Error as exc:
                row["store_errors_total"] = -1
                row["store_errors_in_window"] = -1
                row["last_store_error"] = f"read failed: {exc}"
            samples.append(row)
            worst = max(float(row[f"{n}_ms"]) for n in paths)
            log.write(
                f"{row['t']} elapsed={row['elapsed_s']:7.1f}s "
                + " ".join(f"{n}={row[f'{n}_ms']:7.1f}ms" for n in paths)
                + f" worst={worst:7.1f}ms wal={row['wal_mb']}MB db={row['db_mb']}MB"
                + f" store-error(total={row['store_errors_total']},"
                  f" in-window={row['store_errors_in_window']})\n"
            )
            time.sleep(INTERVAL_SECONDS)
    # ---- summary
    def stat(name: str, fn) -> float:
        values = [float(s[f"{name}_ms"]) for s in samples]
        return round(fn(values), 1)
    summary: dict[str, object] = {
        "started": time.strftime("%Y-%m-%dT%H:%M:%S", time.localtime(started_wall)),
        "duration_seconds": round(time.perf_counter() - started, 1),
        "passes": len(samples),
        "db": str(DB_PATH),
        "db_mb_first": samples[0]["db_mb"],
        "db_mb_last": samples[-1]["db_mb"],
        "wal_mb_first": samples[0]["wal_mb"],
        "wal_mb_max": max(float(s["wal_mb"]) for s in samples),
        "wal_mb_last": samples[-1]["wal_mb"],
        "store_errors_total": samples[-1]["store_errors_total"],
        "store_errors_in_window": samples[-1]["store_errors_in_window"],
        "last_store_error": samples[-1]["last_store_error"],
        "paths": {
            name: {
                "median_ms": stat(name, statistics.median),
                "p95_ms": stat(name, lambda v: sorted(v)[max(0, int(len(v) * 0.95) - 1)]),
                "max_ms": stat(name, max),
            }
            for name in paths
        },
        "budget_ms": PAGE_BUDGET_MS,
        "passes_over_budget": sum(
            1 for s in samples if max(float(s[f"{n}_ms"]) for n in paths) > PAGE_BUDGET_MS
        ),
        "non_200_responses": sum(
            1 for s in samples for n in paths if int(s[f"{n}_code"]) not in (200, 404)
        ),
    }
    json_path.write_text(json.dumps({"summary": summary, "samples": samples}, indent=2) + "\n")
    with log_path.open("a", buffering=1) as log:
        log.write("# summary\n")
        for name, values in summary["paths"].items():  # type: ignore[index]
            log.write(f"#   {name:12s} median={values['median_ms']:8.1f} ms "
                      f"p95={values['p95_ms']:8.1f} ms max={values['max_ms']:8.1f} ms\n")
        log.write(f"#   store-error rows in window: {summary['store_errors_in_window']} "
                  f"(total {summary['store_errors_total']})\n")
        log.write(f"#   WAL first/max/last MB: {summary['wal_mb_first']} / "
                  f"{summary['wal_mb_max']} / {summary['wal_mb_last']}\n")
        log.write(f"#   passes over the {PAGE_BUDGET_MS:.0f} ms budget: "
                  f"{summary['passes_over_budget']} of {summary['passes']}\n")
    header = ["t"] + [f"{n}_ms" for n in paths] + ["wal_mb", "db_mb", "store_errors_in_window"]
    with tsv_path.open("w") as tsv:
        tsv.write("\t".join(header) + "\n")
        for sample in samples:
            tsv.write("\t".join(str(sample.get(key, "")) for key in header) + "\n")
    print(json.dumps(summary, indent=2))
    return 0
 if __name__ == "__main__":
    sys.exit(main())
@@ -16,6 +16,7 @@ from __future__ import annotations
 import logging
 import math
 import time
 from collections.abc import Sequence
 from dataclasses import dataclass, field, replace
 from typing import Any
@@ -109,9 +110,9 @@ def parse_range(value: str) -> float:
    raise HTTPException(400, f"unknown range {value!r}; try 5m 1h 24h 7d 30d")
-def _history(store: SampleStore, source: str, host: str, metric: str, span: float,
+def _series_dict(source: str, host: str, metric: str, span: float,
-             points: int) -> dict[str, Any]:
+                 series: list[tuple[float, float]]) -> dict[str, Any]:
-    series = store.history(source, host, metric, span=span, max_points=points)
+    """The one series shape every `/api/*/history` route has always answered with."""
    return {
        "source": source,
        "host": host,
@@ -121,6 +122,30 @@ def _history(store: SampleStore, source: str, host: str, metric: str, span: floa
    }
 def _history(store: SampleStore, source: str, host: str, metric: str, span: float,
             points: int) -> dict[str, Any]:
    return _series_dict(source, host, metric, span,
                        store.history(source, host, metric, span=span, max_points=points))
 def _history_batch(store: SampleStore, source: str, hosts: Sequence[str], metrics: Sequence[str],
                   span: float, points: int) -> list[dict[str, Any]]:
    """N hosts x M metrics through the store's one batched door, host-major and in request order.
    Every `/api/*/history` route that used to loop `store.history()` per pair calls this instead.
    The list it returns is identical, entry for entry, to what that loop built (same series, same
    order, same `points`); what changed is the cost — the resolution choice for the whole batch is
    one capped statement rather than three full `COUNT`s per pair (`store.history_many`).
    """
    wanted = [(host, metric) for host in hosts for metric in metrics]
    if not wanted:
        return []
    answers = store.history_many(source, [host for host, _ in wanted],
                                 [metric for _, metric in wanted], span=span, max_points=points)
    return [_series_dict(source, host, metric, span, answers.get((host, metric), []))
            for host, metric in wanted]
 def _host_severity(host: dict[str, Any]) -> dict[str, str | None]:
    """Verdicts for the overview's fleet-table columns, from the one threshold table.
@@ -722,8 +747,7 @@ def system_router() -> APIRouter:
        wanted_metrics = [item.strip() for item in metrics.split(",") if item.strip()]
        wanted_hosts = [item.strip() for item in hosts.split(",") if item.strip()] or [
            target.name for target in ctx.settings.host_targets]
-        series = [_history(ctx.store, "host", host, metric, span, points)
+        series = _history_batch(ctx.store, "host", wanted_hosts, wanted_metrics, span, points)
                  for host in wanted_hosts for metric in wanted_metrics]
        return {"hosts": wanted_hosts, "range_seconds": span, "series": series}
    return router
@@ -816,8 +840,8 @@ def disks_router() -> APIRouter:
        wanted_hosts = [item.strip() for item in hosts.split(",") if item.strip()] or [
            target.name for target in ctx.settings.host_targets]
        return {"hosts": wanted_hosts, "range_seconds": span,
-                "series": [_history(ctx.store, DISKS_SOURCE, host, metric, span, points)
+                "series": _history_batch(ctx.store, DISKS_SOURCE, wanted_hosts, wanted_metrics,
-                           for host in wanted_hosts for metric in wanted_metrics]}
+                                         span, points)}
    return router
@@ -854,8 +878,9 @@ def llm_router() -> APIRouter:
        wanted_engines = [item.strip() for item in engines.split(",") if item.strip()] or [
            name for name, url in (("gufo", ctx.settings.gufo_url), ("vllm", ctx.settings.vllm_url)) if url]
        return {"engines": wanted_engines, "range_seconds": span,
-                "series": [_history(ctx.store, "llm", engine, f"llm.{metric}", span, points)
+                "series": _history_batch(ctx.store, "llm", wanted_engines,
-                           for engine in wanted_engines for metric in wanted_metrics]}
+                                         [f"llm.{metric}" for metric in wanted_metrics],
                                         span, points)}
    return router
@@ -893,8 +918,7 @@ def ups_router() -> APIRouter:
        wanted_ups = [item.strip() for item in ups.split(",") if item.strip()] or [
            card["name"] for card in ((job.state.payload if job else None) or {}).get("ups", [])]
        return {"ups": wanted_ups, "range_seconds": span,
-                "series": [_history(ctx.store, "ups", name, metric, span, points)
+                "series": _history_batch(ctx.store, "ups", wanted_ups, wanted_metrics, span, points)}
                           for name in wanted_ups for metric in wanted_metrics]}
    @router.get("/commands")
    async def commands(ctx: AppContext = Depends(get_context)) -> dict[str, Any]:
@@ -28,11 +28,30 @@ first start (phase 11). Retention never touches it; it holds a handful of rows.
 The connection is guarded by a lock and every method is synchronous: async callers wrap
 them in `anyio.to_thread.run_sync`, sync FastAPI handlers run in the threadpool already.
 Phase 14 task 02 added the two short-TTL read caches (`series_names`, `stats`) and the
 capped per-resolution count. Both exist for the same reason: a per-request answer that costs
 a whole-table pass is a page load, and on a 600 MB store the two answers that used to be
 scanned on every request cost ~1.3 s each (`data/logs/NOTES-store-perf.md`), serialized
 behind the very mutex the poller writes under. A cached answer is a *bounded* answer; the
 TTL is how stale it may get before it is re-derived, and a write invalidates the one whose
 staleness would be a lie about the data (`series_names`) rather than about a counter.
 Phase 14 task 03 bounded the *write* side. Each rollup step now scans only the rows that
 became due since the previous pass (`maintain()` reads `last_maintain` first and windows each
 step to it), because the whole-table rollup was a 5.4 s write transaction that held the WAL
 write lock past a poller's `busy_timeout` — the `store-error: database is locked` line of
 AGENTS.md known gap 4. Two properties make the window a cost bound rather than a data-bound
 change, and both are pinned in `tests/test_store.py`: its lower bound reaches **past** the
 previous pass (one bucket plus slack, so a missed interval is a catch-up scan and not a gap), and
 it is snapped down to a **bucket boundary** (so a bucket any pass re-reads is a bucket it reads
 whole, which is what keeps the averaged values identical to a whole-table rollup's).
 """
 from __future__ import annotations
 import json
 import logging
 import math
 import sqlite3
 import threading
@@ -42,9 +61,94 @@ from dataclasses import dataclass
 from pathlib import Path
 RES_RAW, RES_MIN, RES_HOUR = "raw", "1m", "1h"
 #: The three resolutions, finest first. Every resolution decision in the app walks this tuple,
 #: so `pick_res` and `pick_res_many` cannot disagree about which one is finest.
 RESOLUTIONS: tuple[str, ...] = (RES_RAW, RES_MIN, RES_HOUR)
 MINUTE = 60.0
 HOUR = 3600.0
 #: How long `series_names()` may answer from its cache (phase 14 task 02). The set of distinct
 #: metric names changes when a source starts or stops publishing one — minutes, not seconds —
 #: and a write that can introduce one invalidates it anyway (task 04: *that* write, not every
 #: sample the hub stores), so the TTL is only the ceiling on a miss we did not notice. Instance
 #: attribute `series_ttl_seconds` overrides it (tests, probes).
 SERIES_CACHE_TTL_SECONDS = 60.0
 #: How long `stats()` may answer from its cache. `/api/health` is polled by every open tab every
 #: 15 s and by the container healthcheck, and its counts are operator trivia, not a control
 #: surface: 30 s of staleness buys a 1.3 s full-table scan off every poll but one.
 STATS_CACHE_TTL_SECONDS = 30.0
 #: How many cached `(source, host)` answers `series_names()` keeps before starting over. The
 #: endpoint takes both as free-text query parameters, so the key space is the caller's; the
 #: cache is bounded rather than grown by whatever a stranger types.
 SERIES_CACHE_MAX_ENTRIES = 256
 #: How many `(source, host, metric)` triples the store remembers having written, which is what
 #: lets `add_many` leave the series cache alone (see `_note_series_write`). The fleet's whole
 #: name space is a few hundred triples, so this is a generous ceiling on a bounded namespace,
 #: not a growing structure: past it the store goes back to invalidating on every write, because
 #: memory tracked on a namespace a caller can influence needs a cap either way.
 SERIES_TRACKED_SERIES_MAX = 8192
 #: Branches per compound count statement. SQLite's `SQLITE_LIMIT_COMPOUND_SELECT` defaults to
 #: 500 terms and the variable limit to 32 766; 100 keeps one statement per 33 series while
 #: leaving 5x headroom on both limits, and it is where the statement-preparation cost stops
 #: being noise (measured on the seeded store: 81 branches ≈ 0.5 ms of preparation).
 COMPOUND_CHUNK = 100
 #: Slack on a bucket's completeness, and on how far behind the previous pass a rollup looks.
 #: A minute bucket is only rolled once its whole 60 s is in the past *plus* this slack, so a
 #: rollup never averages a partial minute and gets rewritten by the next pass; the same 60 s on
 #: the window's lower bound is what makes a missed interval a catch-up scan rather than a gap.
 MAINTAIN_SLACK_SECONDS = 60.0
 #: The `meta` key holding the previous `maintain()` pass's clock. Written by `maintain()` and read
 #: by it on the next run to window the rollups; `stats()` surfaces it in `/api/health` (uncached)
 #: precisely so an operator can see it stop moving.
 LAST_MAINTAIN_KEY = "last_maintain"
 #: WAL auto-checkpoint threshold, in pages. This is SQLite's own default (1 000 pages ≈ 4 MB) and
 #: is written down so the knob is visible next to the one that actually bites: a WAL that reaches
 #: 140 MB (AGENTS.md known gap 4) is a checkpoint that keeps losing to the write rate, and
 #: asking for one more often does not help — a checkpoint cannot run *inside* a long transaction.
 WAL_AUTOCHECKPOINT_PAGES = 1000
 #: Cap on the WAL file, in bytes. At a checkpoint a WAL larger than this is truncated to it
 #: instead of staying at its high-water mark forever, so the file has a ceiling even when a
 #: reader kept a snapshot open long enough to make it grow. 64 MB sits comfortably above what the
 #: write path asks for (the whole-retention pass, the biggest thing this does, left a 41 MB WAL on
 #: the seeded store) and far below the 140 MB gap 4 recorded, so the cap binds on runaways and not
 #: on maintenance. It is a per-connection pragma, not a migration, so moving it is a one-line
 #: change (task 04's live watch decides whether to).
 JOURNAL_SIZE_LIMIT_BYTES = 64 * 1024 * 1024
 #: SQLite's background ANALYZE, deliberately off. It refreshes statistics inside whichever commit
 #: trips its change counter, which on this app means inside a poller's `add_many` — the write path
 #: this task exists to shorten. `maintain()` calls `PRAGMA optimize` instead: once a minute, on the
 #: tick that already owns bookkeeping, outside the write transaction.
 AUTOMATIC_ANALYZE = 0
 #: One rollup step: average `res_from` rows into `res_to` buckets, complete buckets only.
 #: Bindings: `(res_to, bucket, bucket, res_from, ceiling, floor, bucket)`. Public because
 #: `tests/store_perf.py` EXPLAINs the same statement — the plan in the NOTES has to be the plan
 #: that runs (the same reason `RES_COUNT_BRANCH` is a constant).
 ROLLUP_INSERT = (
    "INSERT OR REPLACE INTO sample (source, host, metric, res, ts, value)"
    " SELECT source, host, metric, ?, CAST(CAST(ts AS INT)/? AS INT)*?, AVG(value)"
    " FROM sample WHERE res=? AND ts < ? AND ts >= ?"
    " GROUP BY source, host, metric, CAST(CAST(ts AS INT)/? AS INT)"
 )
 logger = logging.getLogger("dashboards.store")
 #: One `(host, metric, res)` row of the capped count. Public because `tests/store_perf.py`
 #: EXPLAINs it: the plan the NOTES record has to be the plan that runs, not a lookalike.
 #:
 #: The count is capped by `LIMIT` inside the subquery because the only question ever asked of it
 #: is "is this <= max_points", and counting 43 200 minute rows to answer "no" is what made a 90 d
 #: chart request expensive. The three label columns come first so the answer does not depend on
 #: compound-row ordering.
 RES_COUNT_BRANCH = (
    "SELECT ?, ?, ?, COUNT(*) FROM (SELECT 1 FROM sample"
    " WHERE source=? AND host=? AND metric=? AND res=? AND ts>=? LIMIT ?)"
 )
 _SCHEMA = """
 CREATE TABLE IF NOT EXISTS sample (
    source TEXT NOT NULL,
@@ -104,6 +208,47 @@ class Event:
        }
 def _pairs(hosts: Sequence[str], metrics: Sequence[str]) -> list[tuple[str, str]]:
    """The host-major series list a request asks for, de-duplicated but otherwise in order.
    `dict.fromkeys` rather than a set: the order is the order the endpoints emit their series in,
    and a set would shuffle a response that a browser (and a test pin) expects in request order.
    """
    return [(host, metric) for host in dict.fromkeys(hosts) for metric in dict.fromkeys(metrics)]
 def _choose_res(counts: dict[str, int], max_points: int) -> str:
    """The resolution decision, from one series' capped per-`res` counts.
    Counts are capped at `max_points + 1`, which leaves both predicates below exactly as true as
    they were against a full `COUNT(*)`: `count` is 0 exactly when the real count is, and
    `≤ max_points` exactly when the real count is.
    """
    finest_with_data: str | None = None
    for res in RESOLUTIONS:
        count = counts.get(res, 0)
        if not count:
            continue
        if finest_with_data is None:
            # too many points for the budget is fine — history() thins. Returning a
            # resolution with *no* rows in the window reads as a dead sensor.
            finest_with_data = res
        if count <= max_points:
            return res
    return finest_with_data or RES_HOUR
 def _thin(points: list[tuple[float, float]], max_points: int) -> list[tuple[float, float]]:
    """Stride-thin a series to the point budget without ever losing the newest point."""
    if len(points) <= max_points:
        return points
    stride = math.ceil(len(points) / max_points)
    thinned = points[::stride]
    if thinned[-1] != points[-1]:
        thinned.append(points[-1])  # never lose "now" to the thinning
    return thinned
 class SampleStore:
    def __init__(self, path: str | Path, *, raw_keep_hours: float = 48,
                 minute_keep_days: float = 30, hour_keep_days: float = 400) -> None:
@@ -118,8 +263,82 @@ class SampleStore:
        self._db.execute("PRAGMA journal_mode=WAL")
        self._db.execute("PRAGMA synchronous=NORMAL")  # a lost sample is fine, corruption is not
        self._db.execute("PRAGMA busy_timeout=5000")
        # WAL housekeeping (phase 14 task 03): keep trying to checkpoint at the default threshold
        # and cap the file, so a WAL that does grow — a long-lived reader holding a snapshot open
        # is enough — stops at the cap at the next checkpoint instead of at 140 MB. Then take
        # SQLite's statistics upkeep away from the poll path: without statistics the planner reads
        # the table for every windowed rollup anyway (`SCAN sample`), and with SQLite's own
        # automatic ANALYZE the refresh lands inside whichever poller's commit trips its counter.
        # All three are per-connection pragmas, so they are re-asserted on every open, not migrated.
        self._db.execute(f"PRAGMA wal_autocheckpoint={WAL_AUTOCHECKPOINT_PAGES}")
        self._db.execute(f"PRAGMA journal_size_limit={JOURNAL_SIZE_LIMIT_BYTES}")
        self._db.execute(f"PRAGMA automatic_analyze={AUTOMATIC_ANALYZE}")
        self._db.executescript(_SCHEMA)
        self._db.commit()
        # The two read caches (see the TTL constants above). Both are instance attributes so a
        # test or a probe can force expiry with `store.stats_ttl_seconds = 0` instead of waiting
        # for a clock or monkeypatching a module-wide `time`.
        self.series_ttl_seconds = SERIES_CACHE_TTL_SECONDS
        self.stats_ttl_seconds = STATS_CACHE_TTL_SECONDS
        # `(argument tuple) -> (generation, monotonic, answer)`. The generation is what makes a
        # concurrent write safe: a thread that read the table *before* a write and stores its
        # answer *after* it would otherwise park a stale list under a fresh timestamp.
        # The `set` beside each cached list is what `add_many` consults to decide whether the
        # write can change the answer at all (see `_note_series_write`); membership in a list of
        # every metric name is not a check to make per sample.
        self._series_cache: dict[tuple[str | None, str | None],
                                 tuple[int, float, list[str], set[str]]] = {}
        self._stats_cache: tuple[float, dict[str, int], int] | None = None
        self._generation = 0
        #: `(source, host, metric)` triples this store has written rows for. Tracks *existence*,
        #: not rows, and is capped (see `SERIES_TRACKED_SERIES_MAX`).
        self._known_series: set[tuple[str, str, str]] = set()
    # -- the read caches -------------------------------------------------------- #
    def invalidate_read_caches(self) -> None:
        """Drop every cached read answer.
        `add_many` invalidates the name list only when it introduces a name
        (`_note_series_write`); every write path that can *remove* one invalidates unconditionally.
        This is the blunt instrument for the callers that need a cold measurement (the phase-14
        probe) or a test that wants to be sure it is not reading its own previous answer.
        """
        self._generation += 1
        self._series_cache.clear()
        self._stats_cache = None
    def _invalidate_series(self) -> None:
        """Called from inside a write's own lock: never takes `self._lock` (it is not reentrant)."""
        self._generation += 1
        self._series_cache.clear()
    def _note_series_write(self, series: Sequence[tuple[str, str, str]]) -> None:
        """Invalidate the series-name cache only for a write that can change it.
        Every sample used to clear it, and the live watch is what caught the consequence: the hub
        writes every 2-300 s, so on the operator's store `/api/metrics` answered with its full
        `SELECT DISTINCT` scan on nearly every call (535 ms of store mutex, measured) — the same
        per-request scan task 02 existed to remove, surviving behind a cache that never served.
        An insert cannot *remove* a name, so the cached answers change only when the write
        introduces a `(source, host, metric)` this store has never written. A poll of an existing
        series is therefore free, and a genuinely new name still costs exactly one invalidation,
        which is what `tests/test_store.py` pins in both directions.
        `delete_series()` and `maintain()` keep the unconditional invalidation they have always
        had, because a `DELETE` can take the last row of a name and existence-tracking cannot see
        that. The tracking can also be wrong in the same direction if *another* process deletes
        rows this one believes exist (two servers on one file, AGENTS.md gap 4's configuration
        mistake): the cost is that a name reappearing inside one TTL window is not in the list
        until the window ends. `series_ttl_seconds` is the ceiling on that, and it is the reason
        the cache has a TTL rather than trusting the tracking alone.
        """
        if any(key not in self._known_series for key in series):
            self._invalidate_series()
        if len(self._known_series) + len(series) <= SERIES_TRACKED_SERIES_MAX:
            self._known_series.update(series)
        else:  # cap reached: forget the tracking, so from the next write on every one invalidates
            self._known_series.clear()
    def close(self) -> None:
        with self._lock:
@@ -144,6 +363,7 @@ class SampleStore:
            return 0
        with self._lock, self._db:
            self._db.executemany("INSERT OR REPLACE INTO sample VALUES (?,?,?,?,?,?)", rows)
            self._note_series_write([(row[0], row[1], row[2]) for row in rows])
        return len(rows)
    def log_event(self, event: Event) -> None:
@@ -182,48 +402,126 @@ class SampleStore:
            return cur.fetchall()  # type: ignore[no-any-return]
    def pick_res(self, source: str, host: str, metric: str, span: float, max_points: int) -> str:
-        """Finest resolution that still fits in `max_points` for this window."""
+        """Finest resolution that still fits in `max_points` for this window.
-        now = time.time()
+
-        start = now - span
+        One statement, not three: the counts for all three resolutions come back from a single
-        finest_with_data: str | None = None
+        compound query whose per-`res` `LIMIT` caps the work (phase 14 task 02). Capping is exact,
-        for res in (RES_RAW, RES_MIN, RES_HOUR):
+        not an approximation, because the only thing the decision asks of a count is whether it is
-            with self._lock:
+        `0`, `≤ max_points`, or neither — so counting 43 200 minute rows to answer "neither" was
-                (count,) = self._db.execute(
+        pure waste. Measured on the seeded store: 90 d, one series, 2.97 ms → 0.10 ms; the
-                    "SELECT COUNT(*) FROM sample WHERE source=? AND host=? AND metric=? AND res=? AND ts>=?",
+        `tests/test_store.py` pin is that the *answer* is unchanged for every window.
-                    (source, host, metric, res, start),
+        """
-                ).fetchone()
+        counts = self._counts_by_res(source, ((host, metric),), span, max_points)
-            if not count:
+        return _choose_res(counts[(host, metric)], max_points)
-                continue
+
-            if finest_with_data is None:
+    def pick_res_many(self, source: str, hosts: Sequence[str], metrics: Sequence[str],
-                # too many points for the budget is fine — history() thins. Returning a
+                      span: float, max_points: int) -> dict[tuple[str, str], str]:
-                # resolution with *no* rows in the window reads as a dead sensor.
+        """`pick_res` for every `(host, metric)` pair, in one grouped round trip per 33 series.
-                finest_with_data = res
+
-            if count <= max_points:
+        The batched door the N x M history endpoints go through. The decision is made by the same
-                return res
+        `_choose_res` on the same capped counts, so a pair's answer here is identical to the answer
-        return finest_with_data or RES_HOUR
+        `pick_res` gives for it alone — that equality is a test, not a hope.
        """
        counts = self._counts_by_res(source, _pairs(hosts, metrics), span, max_points)
        return {pair: _choose_res(seen, max_points) for pair, seen in counts.items()}
    def history(self, source: str, host: str, metric: str, *, span: float = 3600.0,
                max_points: int = 600) -> list[tuple[float, float]]:
        """Windowed series, thinned to `max_points` so the browser never gets 100k rows."""
-        res = self.pick_res(source, host, metric, span, max_points)
+        return self.history_many(source, [host], [metric], span=span,
-        window_start = time.time() - span
+                                 max_points=max_points)[(host, metric)]
-        points = self.query(source, host, metric, res=res, start=window_start,
+
-                            limit=max_points * 4, tail=True)
+    def history_many(self, source: str, hosts: Sequence[str], metrics: Sequence[str], *,
-        if not points and res != RES_RAW:
+                     span: float = 3600.0,
-            # Rollups lag behind (they only cover buckets that are fully in the past, and a
+                     max_points: int = 600) -> dict[tuple[str, str], list[tuple[float, float]]]:
-            # dashboard that has been up for ten minutes has none at all). Falling back to
+        """`history` for many series at once, keyed by `(host, metric)`.
-            # raw turns "no samples yet" into a real chart instead of a lie by omission.
+
-            points = self.query(source, host, metric, res=RES_RAW, start=window_start,
+        Same per-series answer as `history()` — resolution choice, tail semantics, raw fallback,
        thinning, and the point list that keeps "now" — with the resolution choice made in one
        grouped statement instead of three counts per series.
        The *rows* are still fetched per series, deliberately and measuredly: the tail select is
        already a covering-index seek that reads at most `max_points x 4` rows, and every shape
        that folds the whole batch into one statement (a `ROW_NUMBER()` window, a compound of the
        per-series tails) was slower on the seeded store — 73 ms and 38 ms against 29 ms for 27
        series at 90 d — because it numbers or sorts rows the seek never has to touch. Batching
        the *decision* was where the cost was; batching the fetch is a round trip traded for work.
        """
        pairs = _pairs(hosts, metrics)
        if not pairs:
            return {}
        now = time.time()
        window_start = now - span
        counts = self._counts_by_res(source, pairs, span, max_points)
        out: dict[tuple[str, str], list[tuple[float, float]]] = {}
        for host, metric in pairs:
            res = _choose_res(counts[(host, metric)], max_points)
            points = self.query(source, host, metric, res=res, start=window_start, end=now,
                                limit=max_points * 4, tail=True)
-        if len(points) <= max_points:
+            if not points and res != RES_RAW:
-            return points
+                # Rollups lag behind (they only cover buckets that are fully in the past, and a
-        stride = math.ceil(len(points) / max_points)
+                # dashboard that has been up for ten minutes has none at all). Falling back to
-        thinned = points[::stride]
+                # raw turns "no samples yet" into a real chart instead of a lie by omission.
-        if thinned[-1] != points[-1]:
+                points = self.query(source, host, metric, res=RES_RAW, start=window_start,
-            thinned.append(points[-1])  # never lose "now" to the thinning
+                                    end=now, limit=max_points * 4, tail=True)
-        return thinned
+            out[(host, metric)] = _thin(points, max_points)
        return out
    def _counts_by_res(self, source: str, pairs: Sequence[tuple[str, str]], span: float,
                       max_points: int) -> dict[tuple[str, str], dict[str, int]]:
        """Capped per-resolution row counts for many series, in as few statements as possible.
        One compound statement of `(host, metric, res)` branches, chunked at `COMPOUND_CHUNK`
        terms, each branch an index seek on the full primary-key prefix (`tests/test_store.py`
        pins the plan) that stops after `max_points + 1` rows. Nothing here scans a table: the
        work is proportional to `series x max_points`, never to rows of history.
        """
        start = time.time() - span
        cap = max_points + 1
        out: dict[tuple[str, str], dict[str, int]] = {pair: {} for pair in pairs}
        branches: list[str] = []
        args: list[object] = []
        def flush() -> None:
            if not branches:
                return
            with self._lock:
                rows = self._db.execute(" UNION ALL ".join(branches), tuple(args)).fetchall()
            for host, metric, res, count in rows:
                out[(host, metric)][res] = int(count)
            branches.clear()
            args.clear()
        for host, metric in pairs:
            for res in RESOLUTIONS:
                branches.append(RES_COUNT_BRANCH)
                args += [host, metric, res, source, host, metric, res, start, cap]
                if len(branches) >= COMPOUND_CHUNK:
                    flush()
        flush()
        return out
    def series_names(self, source: str | None = None, host: str | None = None) -> list[str]:
        """Distinct metric names, from a cache that only a write which can change them ends.
        The uncached answer is `SELECT DISTINCT metric` over the table — 1.4 s of `SCAN sample`
        on a production-sized store, on a route the browser polls alongside the rest. The set of
        names is also the slowest-moving fact in the database (it changes when a source gains a
        metric, i.e. on a deploy), which is what a 60 s TTL plus write invalidation is sized for.
        """
        key = (source, host)
        cached = self._series_cache.get(key)
        generation, built_at = self._generation, time.monotonic()
        if cached is not None and cached[0] == generation and built_at - cached[1] < self.series_ttl_seconds:
            return list(cached[2])
        names = self._scan_series_names(source, host)
        if len(self._series_cache) >= SERIES_CACHE_MAX_ENTRIES:
            self._series_cache.clear()
        self._series_cache[key] = (generation, built_at, names, set(names))
        return list(names)
    def _scan_series_names(self, source: str | None, host: str | None) -> list[str]:
        """The uncached answer behind `series_names()`: one pass, no cache, no TTL."""
        sql = "SELECT DISTINCT metric FROM sample"
        args: list[object] = []
        clauses = []
@@ -304,48 +602,166 @@ class SampleStore:
            return self._db.execute("DELETE FROM meta WHERE key=?", (key,)).rowcount
    # -- maintenance ----------------------------------------------------------- #
    def _previous_maintain(self) -> float | None:
        """The clock of the last `maintain()` pass, or `None` for "no idea — scan the retention".
        Missing is the normal case on a fresh database. A row that is not a number should not
        exist (`maintain()` is the only writer) and is treated as missing rather than fatal, with
        one line said about it: the expensive answer is the safe one here, because a rollup that
        re-runs over buckets it already rolled rewrites identical rows. Never interpolate the
        value — it is data the operator did not write, and this app's logs end up in an `event`
        table a browser reads.
        """
        raw = self.get_meta(LAST_MAINTAIN_KEY)
        if raw is None:
            return None
        try:
            return float(raw)
        except ValueError:
            logger.warning("meta row '%s' is not a number; the next rollup scans the full "
                           "retention window", LAST_MAINTAIN_KEY)
            return None
    def _rollup_floor(self, since: float | None, retention_start: float, bucket: float) -> float:
        """The oldest row a rollup step has to read, snapped down to a bucket boundary.
        With no previous pass that is the whole retention window, exactly as the step always
        cost. With one, it is the window that became due since: `since - bucket - slack`, one
        bucket plus the completeness slack behind the previous pass's clock. That overlap is what
        turns "the app was down for ten minutes" into a ten-minute catch-up scan instead of nine
        minutes of silently missing buckets.
        The snap down to a bucket boundary is not cosmetic, it is what makes the window safe. The
        step's ceiling is a *row* bound, so the newest bucket it touches is usually a partial one
        that a later pass re-rolls as more of its rows land — that progressive correction is how
        the whole-table version always got the right average. Re-reading a bucket from anywhere
        other than its own start therefore overwrites a fuller average with a thinner one: with a
        floor of exactly `since - bucket - slack` a minute bucket split by the boundary ends up as
        the mean of its own tail rows (`tests/test_store.py` pins both numbers). Aligning the floor
        down means every bucket this pass reads again is one it reads *whole*, so each re-roll can
        only add rows to it — which is the same guarantee the unwindowed scan had.
        """
        if since is None:
            return retention_start
        aligned = math.floor((since - bucket - MAINTAIN_SLACK_SECONDS) / bucket) * bucket
        return max(retention_start, aligned)
    def maintain(self, now: float | None = None) -> dict[str, int]:
-        """Roll up and prune. Safe to call every minute; each step is idempotent.
+        """Roll up and prune, windowed to what became due since the last pass. Idempotent.
        A bucket is only rolled once its whole window is in the past (+60s slack), so a
        rollup never averages a partial minute and then gets rewritten by a later one.
        Both rollup steps were `INSERT … SELECT … GROUP BY` over everything their resolution
        retains, which on a store with 30 days of 1-minute rows meant a 3.97 M-row scan inside the
        one write transaction (task 01's measurement: 5.4 s, and 5 of 9 concurrent poller inserts
        then outlived `busy_timeout=5000` and surfaced as `store-error`). Windowing each step to
        `last_maintain` is what makes the transaction short: a run once a minute aggregates the
        buckets that filled in the last minute, and the first run after a restart — or after a
        night with the app stopped — scans back to the previous pass instead of back to retention.
        What a window costs, stated plainly: a raw row that lands *behind* the window (a
        multi-minute poller stall, or a host whose clock put the sample far in the past) is not
        re-read by the next pass, so the bucket it belongs to keeps the average the earlier pass
        computed. The poller stamps samples with its own wall clock, so reaching that case costs a
        stall measured in minutes; re-scanning 30 days on every tick to cover it is the thing this
        task exists to stop. `_rollup_floor` explains the bucket alignment that keeps every bucket
        inside the window honest, and
        `test_windowed_rollups_answer_exactly_what_one_full_window_rollup_answers` is the proof.
        """
        now = time.time() if now is None else now
        # Outside the transaction and before the lock below: `get_meta` takes `self._lock` and
        # `threading.Lock` is not reentrant (the same reason `stats()` reads meta from outside).
        since = self._previous_maintain()
        minute_ceiling = now - MINUTE - MAINTAIN_SLACK_SECONDS
        hour_ceiling = now - HOUR - MAINTAIN_SLACK_SECONDS
        counts = {"to_minute": 0, "to_hour": 0, "pruned": 0}
        with self._lock, self._db:
            counts["to_minute"] = self._db.execute(
-                "INSERT OR REPLACE INTO sample (source, host, metric, res, ts, value)"
+                ROLLUP_INSERT,
-                " SELECT source, host, metric, ?, CAST(CAST(ts AS INT)/? AS INT)*?, AVG(value)"
+                (RES_MIN, MINUTE, MINUTE, RES_RAW, minute_ceiling,
-                " FROM sample WHERE res=? AND ts < ? AND ts >= ?"
+                 self._rollup_floor(since, now - self.raw_keep_seconds, MINUTE), MINUTE),
                " GROUP BY source, host, metric, CAST(CAST(ts AS INT)/? AS INT)",
                (RES_MIN, MINUTE, MINUTE, RES_RAW, now - MINUTE - 60,
                 now - self.raw_keep_seconds, MINUTE),
            ).rowcount
            counts["to_hour"] = self._db.execute(
-                "INSERT OR REPLACE INTO sample (source, host, metric, res, ts, value)"
+                ROLLUP_INSERT,
-                " SELECT source, host, metric, ?, CAST(CAST(ts AS INT)/? AS INT)*?, AVG(value)"
+                (RES_HOUR, HOUR, HOUR, RES_MIN, hour_ceiling,
-                " FROM sample WHERE res=? AND ts < ? AND ts >= ?"
+                 self._rollup_floor(since, now - self.minute_keep_seconds, HOUR), HOUR),
                " GROUP BY source, host, metric, CAST(CAST(ts AS INT)/? AS INT)",
                (RES_HOUR, HOUR, HOUR, RES_MIN, now - HOUR - 60,
                 now - self.minute_keep_seconds, HOUR),
            ).rowcount
-            counts["pruned"] = self._db.execute(
+            # The prune keeps its full retention bound on purpose. A rollup may window because it
-                "DELETE FROM sample WHERE (res=? AND ts < ?) OR (res=? AND ts < ?) OR (res=? AND ts < ?)",
+            # only ever *adds* rows and re-adding one is a rewrite; a prune must be able to reach
-                (RES_RAW, now - self.raw_keep_seconds,
+            # anything that expired, whenever it expired, or gap 9's "the file stays bounded" stops
-                 RES_MIN, now - self.minute_keep_seconds,
+            # being true. It is a bounded delete per resolution, not a GROUP BY.
-                 RES_HOUR, now - self.hour_keep_seconds),
+            #
-            ).rowcount
+            # One DELETE per resolution, not one `OR` of the three, because the composition is the
-            self._db.execute("INSERT OR REPLACE INTO meta (key, value) VALUES ('last_maintain', ?)", (str(now),))
+            # part the planner cannot use. `WHERE (res=? AND ts<?) OR (res=? AND ts<?) OR (res=?
            # AND ts<?)` reads every row of the table to delete none of them — 554 ms of the 562 ms
            # a steady `maintain()` cost on the seeded 6.4 M-row store (`SCAN sample`, recorded in
            # `data/logs/NOTES-store-perf.md`) — while each half of that same predicate is a seek on
            # the primary-key index at a third of a millisecond. Same three bounds, same rows gone.
            pruned = 0
            for res, keep_seconds in ((RES_RAW, self.raw_keep_seconds),
                                      (RES_MIN, self.minute_keep_seconds),
                                      (RES_HOUR, self.hour_keep_seconds)):
                pruned += max(self._db.execute("DELETE FROM sample WHERE res = ? AND ts < ?",
                                               (res, now - keep_seconds)).rowcount, 0)
            counts["pruned"] = pruned
            self._db.execute("INSERT OR REPLACE INTO meta (key, value) VALUES (?, ?)",
                             (LAST_MAINTAIN_KEY, str(now)))
            # Maintenance writes rollup rows and deletes pruned ones, so it can both add and
            # remove the last row of a metric name. The counts stay behind `stats()`'s TTL (they
            # are an operator-facing total, and `maintain()` is the thing that would have to
            # re-derive them at the same cost); the name set is invalidated, because a chart that
            # asks for a name the pruner just deleted has to be told, not served yesterday's list.
            self._invalidate_series()
        self._optimize()
        return counts
    def _optimize(self) -> None:
        """`PRAGMA optimize`, best effort, outside the write transaction.
        This is what makes the windowing measurable. Without query statistics SQLite cannot tell a
        two-minute window from a whole-retention scan and picks the scan for both — the windowed
        rollup cost 328 ms and the prune 554 ms with `SCAN sample` either way, and 1 ms and 0 ms
        with `SEARCH sample USING INDEX sqlite_autoindex_sample_1` after one `optimize`. Statistics
        are the other half of "cheaper for the same rows", not an optional extra.
        SQLite's own automatic ANALYZE is switched off in `__init__`, so this is the only place
        statistics are refreshed: automatic analyse runs inside whichever poll's commit trips its
        change counter, which is exactly the poll path this task is shortening. Here it is once a
        minute, on the tick that already owns maintenance — a sampled full analyse the first time
        (~100 ms on the seeded store) and 0.0 ms on every tick after, because SQLite only re-samples
        tables whose change counter moved. A failure is logged and dropped: stale statistics are a
        slower query, and `store-error` on a maintenance tick because of a planner hint would be a
        worse report to the operator.
        """
        try:
            with self._lock:
                self._db.execute("PRAGMA optimize")
        except sqlite3.Error:
            logger.warning("PRAGMA optimize failed; the query planner keeps its existing statistics")
    def stats(self) -> dict[str, object]:
        """Store size and row counts for `/api/health`, with the two COUNT scans cached.
        The payload keys are a contract with deployment (`/api/health` is the one public data
        route, AGENTS.md invariant 12), so nothing here changed shape — only how often it is
        derived from the table. `last_maintain` and `size_bytes` are deliberately **not** cached:
        the first is how an operator tells a stalled rollup loop from a healthy one and costs
        a three-row lookup, and the second is a `stat()` call, not a query.
        """
        cached = self._stats_cache
        now = time.monotonic()
        if cached is not None and now - cached[0] < self.stats_ttl_seconds:
            by_res, events = dict(cached[1]), cached[2]
        else:
            with self._lock:
                by_res = {row[0]: row[1] for row in self._db.execute(
                    "SELECT res, COUNT(*) FROM sample GROUP BY res")}
                (events,) = self._db.execute("SELECT COUNT(*) FROM event").fetchone()
            self._stats_cache = (time.monotonic(), dict(by_res), events)
        # Read through `get_meta` so there is one path to the table — but *outside* the lock
-        # below, which `get_meta` takes itself (`threading.Lock` is not reentrant).
+        # above, which `get_meta` takes itself (`threading.Lock` is not reentrant).
-        last_raw = self.get_meta("last_maintain")
+        last_raw = self.get_meta(LAST_MAINTAIN_KEY)
        with self._lock:
            by_res = {row[0]: row[1] for row in self._db.execute(
                "SELECT res, COUNT(*) FROM sample GROUP BY res")}
            (events,) = self._db.execute("SELECT COUNT(*) FROM event").fetchone()
        size = 0
        if str(self.path) != ":memory:" and self.path.exists():
            size = self.path.stat().st_size
@@ -367,7 +783,18 @@ class SampleStore:
            sql += " AND metric=?"
            args.append(metric)
        with self._lock, self._db:
-            return self._db.execute(sql, args).rowcount
+            deleted = self._db.execute(sql, args).rowcount
            # Forget the triples this delete emptied, so the next write of one of them counts as
            # the new series it now is: `add_many` invalidates on a triple it has not seen, and a
            # deleted-then-recreated series that stayed "known" would be invisible to
            # `/api/metrics` for a TTL window. One `DELETE` cannot name the triples it emptied,
            # so this drops whatever the predicate could have covered.
            self._known_series = {
                key for key in self._known_series
                if not (key[0] == source and key[1] == host and (metric is None or key[2] == metric))
            }
            self._invalidate_series()
            return deleted
 def samples_from_flat(source: str, host: str, values: dict[str, float | None],
@@ -27,15 +27,33 @@ def pytest_addoption(parser: pytest.Parser) -> None:
        default=False,
        help="run tests marked `live` against the real fleet/engines/upsd (they skip otherwise)",
    )
    parser.addoption(
        "--store-perf",
        action="store_true",
        default=False,
        help="run tests marked `store_perf`: the phase-14 timing measurements against the "
             "seeded production-shaped store (they skip otherwise — they cost minutes, not ms)",
    )
 def pytest_collection_modifyitems(config: pytest.Config, items: list[pytest.Item]) -> None:
-    if config.getoption("--live"):
+    # Two opt-ins, one shape: a marker whose tests are real but expensive, skipped by default
-        return
+    # so `uv run pytest` stays fast and hermetic. `--store-perf` is the phase-14 half of the
-    skip = pytest.mark.skip(reason="needs --live and a reachable target (see tests/test_live_fleet.py)")
+    # rule `--live` established; `scripts/store_perf_probe.py` is the CLI for the same numbers.
    skips = {
        "live": pytest.mark.skip(
            reason="needs --live and a reachable target (see tests/test_live_fleet.py)"),
        "store_perf": pytest.mark.skip(
            reason="needs --store-perf (a seeded ~6 M-row store; see tests/store_perf.py)"),
    }
    enabled = {
        "live": bool(config.getoption("--live")),
        "store_perf": bool(config.getoption("--store-perf")),
    }
    for item in items:
-        if "live" in item.keywords:
+        for marker, skip in skips.items():
-            item.add_marker(skip)
+            if not enabled[marker] and marker in item.keywords:
                item.add_marker(skip)
 ROOT = Path(__file__).resolve().parents[1]
 SRC = ROOT / "src"
@@ -228,3 +246,40 @@ def metrics_text() -> Any:
@pytest.fixture
 def rates() -> RateEstimator:
    return RateEstimator(span=10.0)
 # ---- phase 14: the production-shaped store the hot-path budgets measure ---- #
 #
 # Here rather than in tests/test_store.py because two files price the same budgets — the store
 # paths in `test_store.py`, the HTTP paths in `test_api.py` — and a second seeded 6.4 M-row file
 # (or, worse, a second *differently shaped* one) would make the two sets of numbers describe two
 # databases. Session-scoped because seeding costs ~25 s, and cached on disk (`data/perf/`, see
 # `store_perf.cache_dir`) because the before/after has to be the same physical file both times.
@pytest.fixture(scope="session")
 def perf_store_path() -> Path:
    """The cached seeded store, built on first use and re-verified on every reuse.
    Opt-in tests only: nothing in the default run asks for this fixture, so `uv run pytest` never
    pays for the seed. `DASH_PERF_STORE_DIR` relocates the file (see `store_perf.cache_dir`).
    """
    import store_perf  # noqa: PLC0415 - only the phase-14 tests seed a store
    try:
        return store_perf.ensure_seeded(log=lambda line: print(line, flush=True))
    except OSError as exc:  # pragma: no cover - only a read-only checkout hits this
        pytest.skip(f"cannot build the seeded store ({exc}) — set DASH_PERF_STORE_DIR "
                    "to a writable directory to run the phase-14 measurements")
@pytest.fixture
 def perf_store(perf_store_path: Path) -> Any:
    """The seeded store, opened per test and closed after (one handle per test, one file)."""
    import store_perf  # noqa: PLC0415 - see perf_store_path
    store = store_perf.open_store(perf_store_path)
    try:
        yield store
    finally:
        store.close()
@@ -19,6 +19,7 @@ import pytest
 from fastapi import HTTPException
 from fastapi.testclient import TestClient
 import store_perf
 from auth_client import sign_in
 from auth_contract import ADMIN_PASSWORD, FAST_PBKDF2
 from dashboards import alerts, auth
@@ -34,7 +35,7 @@ from dashboards.sources.llm import (
    parse_gufo_metrics,
    parse_vllm_metrics,
 )
-from dashboards.store import Event, Sample, SampleStore
+from dashboards.store import RES_RAW, Event, Sample, SampleStore
 from slots_contract import (
    CANONICAL_TASK_ID_FIELD,
    OPTIONAL_NEXT_TOKEN_FIELDS,
@@ -3153,6 +3154,84 @@ def test_one_request_per_host_answers_every_per_host_chart(
        assert gap["series"][0]["points"] == []
 def test_every_history_endpoint_answers_exactly_what_the_per_series_path_did(
        tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
    """Phase 14 task 02: the batched store door moved the cost of an N x M request, not its answer.
    All four history routes now ask `store.history_many` for the whole batch instead of looping
    `store.history` per pair. That is only safe if the JSON is the same document the loop built,
    so this compares every route against points read straight out of the store with `query()` —
    a third path, in neither the loop nor the batch — and pins the series order the pages parse
    (host-major, metric-minor) plus the exact key set of one series entry.
    """
    with build_client(tmp_path, monkeypatch, simulate=True, nut_host="127.0.0.1",
                      host_targets=(Target("driveripper", DEAD_URL),
                                    Target("deskwork", DEAD_URL)),
                      gufo_url=DEAD_URL) as test_client:
        store = test_client.app.state.ctx.store  # type: ignore[attr-defined]
        now = time.time()
        writes = [
            ("host", "driveripper", "cpu.usage_pct", [1.0, 2.0, 3.0]),
            ("host", "driveripper", "mem.used_pct", [10.0, 20.0]),
            ("host", "deskwork", "cpu.usage_pct", [4.0, 5.0, 6.0]),
            ("host", "deskwork", "mem.used_pct", [30.0]),
            ("disks", "driveripper", "disk.temp_c@nvme0n1", [38.0, 39.0]),
            ("llm", "gufo", "llm.decode_tokens_per_second", [40.0, 45.0]),
            # a UPS name the simulator does not export, so its 5-second poll cannot add rows to
            # the series this assertion is about
            ("ups", "notaserial", "ups.charge_pct", [99.0, 100.0]),
        ]
        for source, host, metric, values in writes:
            store.add_many([Sample(source, host, metric, value, now - (len(values) - index) * 60)
                            for index, value in enumerate(values)])
        def expected(source: str, host: str, metric: str) -> list[dict[str, float]]:
            rows = store.query(source, host, metric, res=RES_RAW, start=now - 3_600.0, tail=True)
            return [{"ts": ts, "value": value} for ts, value in rows]
        routes = [
            ("/api/system/history", {"hosts": "driveripper,deskwork",
                                    "metrics": "cpu.usage_pct,mem.used_pct", "range": "1h"},
             [("host", host, metric) for host in ("driveripper", "deskwork")
              for metric in ("cpu.usage_pct", "mem.used_pct")]),
            ("/api/disks/history", {"hosts": "driveripper",
                                    "metrics": "disk.temp_c@nvme0n1", "range": "1h"},
             [("disks", "driveripper", "disk.temp_c@nvme0n1")]),
            ("/api/llm/history", {"engines": "gufo", "metrics": "decode_tokens_per_second",
                                  "range": "1h"},
             [("llm", "gufo", "llm.decode_tokens_per_second")]),
            ("/api/ups/history", {"ups": "notaserial", "metrics": "ups.charge_pct",
                                  "range": "1h"},
             [("ups", "notaserial", "ups.charge_pct")]),
        ]
        for path, params, pairs in routes:
            body = test_client.get(path, params=params).json()
            assert len(body["series"]) == len(pairs), path
            for series, (source, host, metric) in zip(body["series"], pairs, strict=True):
                assert set(series) == {"source", "host", "metric", "range_seconds", "points"}
                assert (series["source"], series["host"], series["metric"]) == (source, host, metric)
                assert series["range_seconds"] == 3_600.0
                assert series["points"] == expected(source, host, metric), f"{path} {metric}"
        # and an empty answer is still an answer: a name with no rows, in a batch with names that
        # do, comes back as an empty series rather than a missing entry or a 404
        body = test_client.get("/api/system/history", params={
            "hosts": "driveripper", "metrics": "cpu.usage_pct,gpu.busy_pct@7",
            "range": "1h"}).json()
        assert [series["points"] for series in body["series"]] == [expected(
            "host", "driveripper", "cpu.usage_pct"), []]
        # no metric named is no series asked for: the batch door is never opened with an empty
        # list (SQL `IN ()` is a syntax error waiting for that, and 500 is not the answer anyway)
        empty = test_client.get("/api/system/history",
                                params={"hosts": "driveripper", "metrics": ""}).json()
        assert empty["hosts"] == ["driveripper"] and empty["series"] == []
        no_hosts = test_client.get("/api/system/history",
                                   params={"hosts": "", "metrics": "cpu.usage_pct"}).json()
        assert no_hosts["hosts"] == ["driveripper", "deskwork"], "empty means every configured host"
        assert len(no_hosts["series"]) == 2
 def test_llm_snapshot_keeps_an_engine_that_never_answered_in_the_list(
        tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
    """A configured-but-dead engine is listed as unreachable, not dropped (invariant 4)."""
@@ -4079,3 +4158,93 @@ def test_samples_written_by_the_poller_are_queryable(tmp_path: Path,
                                                   "range": "15m"}).json()["points"]
        assert [p["value"] for p in points] == [40.0, 45.0]
    store.close()
 # --------------------------------------------------------------------------- #
 # phase 14 task 04: the endpoint budgets
 #
 # `store_perf.BUDGETS` is the one table of ceilings (it is what the probe's report and the notes
 # render, so a ceiling relaxed in one place is relaxed everywhere). The ceilings are 2-3x the
 # measured "after" on the seeded production-shaped store, which makes them a regression gate
 # rather than a stopwatch: the `/system` page's own chart request costs ~2 ms and still gets
 # 100 ms, while putting a full-table scan back on a per-request path crosses by three orders of
 # magnitude. Wall time lives behind `--store-perf` for that reason; what runs in every `uv run
 # pytest` is the key-name pin below, because a budget that silently stops being priced is as
 # useless as a budget that is crossed.
 # --------------------------------------------------------------------------- #
 def test_every_http_budget_names_a_route_the_probe_prices(tmp_path: Path) -> None:
    """The stale-pointer check for the HTTP half, on a throwaway store.
    `measure_http` is what the priced test runs, and it owns the request shapes; if a key there is
    renamed, or a route stops being measured, the priced test would report the gap rather than the
    cost — which is why this asserts the key set here, cheaply, on every run.
    """
    report = store_perf.measure_budget_http(tmp_path / "budget-keys.sqlite3",
                                            hosts=("driveripper",), metrics=("cpu.usage_pct",))
    assert set(report["http"]) == set(store_perf.HTTP_MEASURED_KEYS), (
        "the HTTP pass and the budget table no longer describe the same requests")
    assert store_perf.missing_budget_keys(report, ("http",)) == []
    # and the request the budgets describe is the one the task names: 3 hosts x 9 series
    nine = store_perf.HISTORY_METRICS
    assert len(nine) == 9, "the 90 d budget is priced against a 3x9 request, not some other shape"
 def test_metrics_stops_scanning_for_series_names_once_the_hub_is_polling(
    tmp_path: Path, monkeypatch: pytest.MonkeyPatch
 ) -> None:
    """The live shape behind one budgeted endpoint, guarded by statements rather than a clock.
    Task 04's watch caught `/api/metrics` answering with its full `SELECT DISTINCT` scan on
    practically every call of the running server (535 ms, measured): task 02's cache was invalidated
    by every sample write, and the hub writes every 2-300 s, so the cache never lived long enough to
    serve. The store-level half of this lives in `tests/test_store.py`; this is the half that says
    the *endpoint* is what stops scanning, on the app the operator runs.
    """
    db = str(tmp_path / "metrics-busy.sqlite3")
    store = SampleStore(db)
    store.add_many([Sample("host", "driveripper", "cpu.usage_pct", 1.0, time.time()),
                    Sample("host", "deskwork", "cpu.usage_pct", 1.0, time.time())])
    app = create_app(Settings(db_path=db, admin_password=ADMIN_PASSWORD,
                              pbkdf2_iterations=FAST_PBKDF2), store)
    try:
        with TestClient(app) as http:
            sign_in(http)
            assert "cpu.usage_pct" in http.get("/api/metrics", params={"source": "host"}).json()["metrics"]
            # a poll of series the store already has, then the same request with the log open
            store.add_many([Sample("host", "driveripper", "cpu.usage_pct", 2.0, time.time()),
                            Sample("host", "deskwork", "cpu.usage_pct", 2.0, time.time())])
            seen: list[str] = []
            # the statement log is the test: `SampleStore._db` is the handle the request runs on
            store._db.set_trace_callback(seen.append)
            try:
                assert "cpu.usage_pct" in http.get("/api/metrics",
                                                   params={"source": "host"}).json()["metrics"]
            finally:
                store._db.set_trace_callback(None)
            assert [s for s in seen if "DISTINCT metric" in s] == [], (
                f"`/api/metrics` went back to the table while nothing changed: {seen}")
            # and a source that starts publishing something new is in the very next answer
            store.add("host", "driveripper", "gpu.busy_pct@0", 1.0)
            assert "gpu.busy_pct@0" in http.get("/api/metrics",
                                                params={"source": "host"}).json()["metrics"]
    finally:
        store.close()
@pytest.mark.store_perf
 def test_the_budgeted_http_paths_fit_their_pinned_margins(perf_store_path: Path) -> None:
    """`/api/health`, `/api/metrics`, `/api/system/history` (1 h and 90 d), `/api/summary`.
    Priced through the real ASGI gate with a signed-in session, on the seeded 6.4 M-row store —
    the same call the operator's browser makes, in-process so no network is measured. This is the
    line `TODO.md:7` is about: `/api/health` used to answer in 1.27 s because it counted the whole
    `sample` table on every poll, on every tab.
    """
    report = store_perf.measure_budget_http(perf_store_path)
    print()
    print(store_perf.render_budgets_markdown(store_perf.budget_rows(report, ("http",))))
    assert store_perf.budget_violations(report, ("http",)) == []
		`@@ -0,0 +1 @@`
							`Now let me write the baseline NOTES document.`