services/dashboards
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

phase: 14_store_performance

Browse Source 
**Phase 14 `store_performance` — final verification pass (all 4 tasks already in `complete/`)**

- Re-verified every task's work in the tree: read caches (`series_names`/`stats`), capped compound `pick_res_many`, `history_many` + `api._history_batch`, windowed `maintain()` + per-resolution prune + `PRAGMA optimize`/`automatic_analyze=0`/WAL pragmas, `store_perf.BUDGETS` (14 ceilings), probe, live watch.
- **Completion criteria:** baseline+after side by side in `data/logs/NOTES-store-perf.md` ✅ · budget tests green (`uv run pytest -m store_perf --store-perf` → **4 passed**, 14/14 inside) ✅ · 25-min dev-server watch re-run **independently** (74 passes, `scripts/watch-store-perf.py`): **0 `store-error`** (total still 190, newest 2026‑10‑18 06:33:30), WAL 7.77 MB ≤ 67.1 MB, medians 1.7–3.2 ms, p95 720/613 ms = the documented cold-cache window; a 3rd 27-pass watch on a freshly restarted server reproduced it (3.1/708, 1.8/572) ✅ · `maintain()` 5,368 ms → 2.7–6.8 ms and WAL bounded across two cycles (test + notes) ✅ · no behaviour change (all four history routes pinned against `query()`) ✅
- **Full gates:** `uv run pytest` → **1267 passed, 15 skipped**; `bash .agents/validate.sh` → **exit 0**, coverage **98.75%** (floor 94), `ruff check src tests agent` clean, `pyright` 0 errors, `node --check` ok, repo hygiene ok; `uv run pytest -m live --live` → **11 passed**; dev server up, 9/9 sources `ok`, pages 200 in 1.5–3.3 ms.
- **Docs fixed:** README/AGENTS still claimed a flat "20 s per pass" while `host_agent.py` (in this same uncommitted change) scales the budget to 5 s × disk count — corrected in both plus the two illustrative "20 s of wedged disk" sentences; broken bold (`` `driveripper** ``) repaired; stale gap-4 title + "tuning is its own phase" wording updated to say phase 14 ran; README's absolute "nothing … is answered by a whole-table scan" qualified to *per call* with the cold-window cost named; "third opt-in marker" → second; conftest `--store-perf` help named the real marker; **`data/logs/watch-store-perf.py` moved to `scripts/watch-store-perf.py`** (both docs told a clone to run a gitignored path) with root-anchored paths, `DASH_WATCH_OUT`, and dead code removed.
- **New finding recorded:** gap 9's "~10 s process-wide stall is still gap 4's / `14_store_performance`'s problem" is now closed — coincident >8 s `llm` + >12 s `ups` raw-stream gaps went 393 (00:00–12:00) and 198 (12:00–21:36) to **0** across 110+ min of the phase-14 store; written into AGENTS.md gaps 4 and 9 and the notes table.
- **Deviations:** did *not* pull either deferred cost knob (`STATS_CACHE_TTL_SECONDS`, warming caches from the maintenance tick) — AGENTS.md records that decision and its reasoning; `uv pip list --outdated` still reports certifi/fastapi/starlette/websockets (pre-existing advisory, no dependency touched this phase, gate treats it as advisory). Overwrote nothing: task 04's watch preserved as `store-perf-watch.task04.*`, my two runs as `.verify25.*` / `.current-code.*`.
- **Next pending phase:** `15_agent_mtls`, then `16_agent_smart_pass_budget` (owns the scaled-budget tests + its doc criterion, now partly pre-satisfied by this pass), `17_installer_device_filter`.
This commit is contained in:
ducoterra
2026-10-19 23:44:35 -04:00
parent 1bbd868401
commit 569ca25fdb
49 changed files with 5072 additions and 124 deletions
Download Patch File Download Diff File Expand all files Collapse all files
AGENTS.md
+228 -29
View File
@@ -181,15 +181,16 @@ Responsibilities, one line each:
`offline_uncorrectable`, hex-packed vendor raws are never decoded, and an attribute row is
read from its **RAW_VALUE column** rather than the end of the line because this binary
annotates that column — `35 (Min/Max 34/44)`, `43894h+43m+32.519s` — so its last token is
not the value), 5 s per device /
20 s per pass / 240 s cache, attached as `disk_io.devices[].smart`. Missing binary →
not the value), 5 s per device / a pass budget whose 20 s floor scales to 5 s × the disk count
/ 240 s cache, attached as `disk_io.devices[].smart` (gap 10: a 16-drive shelf takes longer
than the floor, and truncating the pass would null the tail drives). Missing binary →
**one** `errors` line for the pass (the word `not found` in it is what
`sources/disks.py` reads to answer "is smartctl installed", and
`tests/test_agent.py` pins that the line still says it); one denied/wedged device → that
disk's `smart` is `null` with its own line; `--no-smart` opts out silently. Never fails
the snapshot. **An expired cache refreshes on a background thread while the request is
served the previous result** — only the first pass after startup runs inline — because 20 s
of wedged disk on the request path blows `HOST_TIMEOUT_SECONDS` and would turn the
served the previous result** — only the first pass after startup runs inline — because a wedged
pass on the request path blows `HOST_TIMEOUT_SECONDS` and would turn the
*system* source red every four minutes over an add-on that page never draws. The refresher
is single-flight and keeps the old cache on failure, so a bad pass retries on the next
expiry instead of hammering the disk.
@@ -230,7 +231,7 @@ Responsibilities, one line each:
that was dialled, because they become health records and event rows. `fetch_snapshot()` is
the one agent `/snapshot` read that `system` and `disks` share, so a dead host says the
same sentence on both pages and only the `timeout` differs (the disks poll has to outlast
the agent's 20 s smartctl pass, and `None` is translated into httpx's sentinel because a
the agent's smartctl pass, and `None` is translated into httpx's sentinel because a
literal `None` means *no timeout at all*). Agent `errors`
become the payload's `degraded` **except** the `smart:` add-on lines — a host without
smartmontools or device read access is unconfigured, not broken (invariant 3); they stay
@@ -397,7 +398,57 @@ Responsibilities, one line each:
call, because a reset that deletes a row it did not name deletes the operator's credential).
`stats()` reads `last_maintain` through `get_meta()` too — outside its own lock, which
`get_meta` takes itself and `threading.Lock` is not reentrant — while `maintain()`'s write
stays inline inside that method's transaction.
stays inline inside that method's transaction. **No per-request path may pay a whole-table
scan** (phase 14 task 02): `series_names()` (1.4 s of `SCAN sample` on a production-sized store,
behind `/api/metrics`) and the two `COUNT`s behind `stats()` (1.25 s, run by `app.js`'s
`/api/health` poll on every tab every 15 s) answer from short-TTL caches — 60 s and 30 s, the
series list additionally invalidated **only by a write that can change it** (task 04: an insert
cannot remove a name, so `_note_series_write()` ends the cache on a `(source, host, metric)` the
store has never written and leaves a poll of an existing series alone — invalidating on every
sample was task 02's defect, and it put the 572 ms scan back on nearly every `/api/metrics` call
on a store nine pollers write to, behind a cache that never served; `delete_series()`/`maintain()`
invalidate unconditionally because a `DELETE` can take a name, `delete_series()` also forgetting
the triples it emptied so a re-created series counts as new again), through a generation counter so
a write racing a read cannot park a stale answer under a fresh timestamp, with the name cache
bounded to 256 `(source, host)` keys because `/api/metrics` takes both as free text and the
existence tracking capped at `SERIES_TRACKED_SERIES_MAX`, below which it falls back to
invalidating on every write. A tracking set made wrong by *another* process deleting rows costs at
most one TTL window, which is the reason the cache has a TTL and not only the tracking. `last_maintain` and `size_bytes` are never
cached: a `last_maintain` that stops moving is how an operator spots the invariant-4 failure
mode, and it costs a three-row lookup. The resolution decision is one compound statement of
per-`(host, metric, res)` branches whose `COUNT` is capped by `LIMIT max_points + 1` — exact,
because the decision only ever asks whether a count is `0`, `≤ max_points`, or neither — and
`pick_res_many()`/`history_many()` are the batched door the N × M endpoints go through. The rows
stay a per-series seek **measuredly**: a `ROW_NUMBER()` window or a compound of the 27 tail
selects both cost more than 27 index seeks on the seeded store (`data/logs/NOTES-store-perf.md`
records the table), so batching the *decision* is the fix and batching the *fetch* would be a
round trip traded for work.
**The write side is windowed too** (phase 14 task 03): `maintain()` reads `last_maintain` through
`get_meta()` *before* its transaction (same non-reentrant-lock rule as `stats()`) and bounds each
rollup step to what became due since, instead of averaging every row the resolution retains —
3.97 M rows of 1-minute history per minute, a 5.4 s write transaction that made a poller's
`add_many` outlive `busy_timeout=5000` (gap 4's `store-error` line). Two properties make that a
cost bound and not a data change, and both are pinned: the lower bound reaches **past** the
previous pass (one bucket plus slack, so ten quiet minutes are a catch-up scan rather than ten
minutes of empty charts), and it is snapped **down to a bucket boundary** — the step's ceiling is
a row bound, so the newest bucket is partial and a later pass re-rolls it as its rows land, and a
floor that cut into a bucket would overwrite the fuller average with the mean of its tail rows
(2.5 becoming 4.5 on a six-sample minute: rule 2 with a chart's clothes on). A window is only a
cost bound if the plan can seek to it, and two things were in the way. The prune keeps its full
retention bound — a rollup may window because it only adds rows, a prune must reach anything that
expired — but it is **one `DELETE` per resolution**, because one
`WHERE (res=? AND ts<?) OR (res=? AND ts<?) OR (…)` is unreadable to the planner and read the
whole 6.4 M-row table to delete nothing (554 ms of a 562 ms pass; 0.3 ms per statement now). And
the store had no query statistics, so every window was priced as a scan however few rows were
inside it: `maintain()` ends with `PRAGMA optimize` and `__init__` sets `automatic_analyze=0`,
because SQLite's own refresh lands inside whichever poller's commit trips its counter — the exact
path this shortens — while a tick a minute costs ~100 ms once and 0.0 ms afterwards. Measured on
the seeded store: **5,368 ms → 2.7 ms** steady, 48 ms for a ten-minute catch-up, and the
two-connection race at the app's cadence goes from 5 of 9 inserts raised to 100 of 100 landing
with a 21 ms worst wait. `__init__` also sets `wal_autocheckpoint` (SQLite's own 1 000 pages,
written down so the knob is visible) plus `journal_size_limit = 64 MB`, the cap that truncates a
runaway WAL at the next checkpoint. `busy_timeout=5000` is unchanged: the fix is a shorter write,
not a longer wait.
- `api.py` — GETs everywhere plus the one POST that dials an upstream, and the `auth_router()`
that creates, describes, **re-credentials**, and ends a session (`POST /api/auth/login` public;
`GET /api/auth/status`, `POST /api/auth/logout` and `POST /api/auth/password` protected). Those
@@ -441,6 +492,14 @@ Responsibilities, one line each:
write of its own — a fourth dashboard must not quietly become a second one, so the app's
only POST stays `/api/ups/command` (invariant 1, pinned by enumerating every served route's
verbs).
All four `history` routes reach the store through one door, `api._history_batch()` →
`store.history_many()` (phase 14 task 02). That is a cost change and nothing else: the series
list is host-major and metric-minor in request order, each entry the same
`{source, host, metric, range_seconds, points}` document the per-series loop built, with the
same empty-series answer for a name that has no rows. `tests/test_api.py` pins it for all four
routes against points read straight out of the store with `query()` — a third path, in neither
the loop nor the batch — so a future "optimisation" that moves a point, an order, or a key
fails a test instead of redrawing a chart wrong.
`/api/summary` is the overview's single call and the cheapest whole-fleet probe there is:
the page renders only `alerts` plus `system.fleet`/`system.hosts`, while `disks.hosts`,
`llm.engines`, `ups.ups`, the `states` maps and `recent_events` stay for programmatic
@@ -567,10 +626,11 @@ That scheme is the history API's key — treat it as a public interface.
`--host 10.1.0.126 --port 9120 --token-file /etc/dashboards-agent.token` (`640
root:dashboards-agent`, the mode `deploy/install-agent.sh` writes) with `User=dashboards-agent`,
and `getent group disk` → `disk:x:6:dashboards-agent`.
**Two things about that install are not what this repo writes, and both cost SMART data on
`/disks` (see the SMART block below): `systemctl show -p DevicePolicy dashboards-agent` answers
`auto` with no `DeviceAllow` lines** (the installer's `strict` + node allow-list is not what is
running there), **and its `smartctl` cannot read drives as that user.**
**The deployed unit has since diverged from the template in the three ways the SMART
block below records** (the `DevicePolicy`/`DeviceAllow` lines removed, `User=root`,
`CapabilityBoundingSet=CAP_SYS_RAWIO CAP_SYS_ADMIN`) — re-running `deploy/install-agent.sh`
on driveripper would clobber the working unit, because it rewrites the unit unconditionally
(gap 10).
`/api/health` today reads **all nine sources `ok`**: `host:driveripper`, `host:deskwork`,
`host:gateway`, `disks:driveripper`, `disks:deskwork`, `disks:gateway`, `nut`, `llm:gufo`,
`llm:vllm`.
@@ -657,15 +717,29 @@ That scheme is the history API's key — treat it as a public interface.
`nvme0n1`…`nvme7n1` + `sda`…`sdh`) and `smart` is non-null for the 8 NVMes, but every one of
them carries only identity — `health`, `temp_c`, `power_on_hours`, `percentage_used` and the
counters are all `null` — while the 8 spinning drives come back `smart: null` each with the
agent's own line `smart: sda: Try an additional '-d ata' or '-d sat' argument.` Measured
read-only on 2026-10-19: the same `smartctl -i -A -H /dev/nvme0n1` **as root** prints
`PASSED`, 55 °C and 14 % used, so the drives answer and the *agent user* is what is locked out
— `/dev/nvme0`, `/dev/ng0n1` and `/dev/bsg/*` (the nodes NVMe admin and SATA pass-through
actually go through) are `crw------- root root`, and only `/dev/nvme0n1` / `/dev/sg0` carry the
`disk` group the agent is in. So `/disks` shows driveripper's drive inventory with `n/a` values
and a why-line, which is the designed degradation, and **lighting it up is a new phase, not a
token**: a udev rule granting those nodes, and/or `host_agent.smart()` learning the
`-d sat` / `/dev/ngXnY` forms smartctl needs when auto-detection fails.
agent's own line `smart: sda: Try an additional '-d ata' or '-d sat' argument.` Pinned on
2026-10-19 with read-only probes (`data/logs/NOTES-driveripper-agent.md`): the `disk`-group
**open of the block nodes works** (agent user, `O_RDWR`), there is no SELinux denier (zero
AVCs with `auditd` running), and the install's `DevicePolicy=strict`/`DeviceAllow` lines were
removed from the unit (they matched nothing useful anyway — systemd resolves `DeviceAllow`
globs by name against `/proc/devices`, where this kernel lists NVMe as `blkext`, so
`nvme*` matches nothing). What locks the agent out is **kernel 7.2's unprivileged
passthrough gate**: NVMe admin via the block node is unprivileged-only for five identify
types; the SMART-log read needs the drive's ACEL to mark it supported (the Samsung 980
PRO's 2020-era `5B2QGXA7` does not — `NVME_IOCTL_ADMIN_CMD: Permission denied`) or
`CAP_SYS_ADMIN`, and SATA SMART behind a SAT layer is ATA pass-through, which the
unprivileged `SG_IO` allow-list (`scsi_cmd_allowed()`: read-safe opcodes only) does not
include — explicit `-d sat`/`-d ata` fails identically, so no `host_agent.smart()` retry
form can reach it. `deskwork` reads live with the same user/unit because its drive's
firmware does advertise the page. So `/disks` shows driveripper's drive inventory with
`n/a` values and a why-line — the designed degradation, **resolved the same day (gap 10):
the unit now runs `User=root` with
`CapabilityBoundingSet=CAP_SYS_RAWIO CAP_SYS_ADMIN`, and all 16 drives read live** (the
980 PRO's 59 °C / 6328 h / 14 % row, `nvme1n1` at 58 % wear, the red `13 uncorrectable`
alarm chip on `nvme5n1`; verified in `data/logs/disks-driveripper-live.png`). A root
process with an **empty** capability bounding set is still capability-less (`CapEff: 0`)
because the kernel gates check capabilities, not uid — the named bounding set is what
unblocked it.
**Real data is in the repo now:** `tests/fixtures/smartctl_driveripper_<dev>.txt` is all 16
drives (8 NVMe, 8 spinning) read through that sudo rule, and `tests/test_agent.py` parses
every one — which is what found the parser's one real bug: this binary annotates the
@@ -681,9 +755,8 @@ That scheme is the history API's key — treat it as a public interface.
*not* a sudo rule, because the agent must read the device itself — on an `agent/host_agent.py`
from phase 09 or later (never redeploy without the grant: 16 permission-denied lines would age
nothing but are pure noise, while with the grant they are 16 drives' worth of temperature,
hours and sector counters). **The agent and the `disk` group are now both in place on
`driveripper`, and it still reads no SMART numbers** — the group only covers nodes that carry
it (see above), so what is left is the node grant / `-d sat` work, not an install. `gateway` needs smartmontools first and will still read no SMART
hours and sector counters). **The agent and the `disk` group are both in place on
`driveripper`** — the `disk` group alone is not enough on kernel 7.2 (the passthrough gate, see above), so the unit additionally runs as root with the two gate capabilities bounded (gap 10) — all 16 drives now read live. `gateway` needs smartmontools first and will still read no SMART
after that, being a VM with no physical disk behind `vda`. A host that cannot read its drives
renders the designed degradation: one dimmed card per physical drive, every number `n/a`, and
the one-line why.
@@ -711,9 +784,26 @@ set -a; . ./.env.dev; set +a # dev dials the fleet the prod way, no
./scripts/dev-server.sh status # per-source ok/DOWN — unauthenticated: /api/health
./scripts/dev-server.sh logs
uv run pytest # hermetic: 1195 passed + 11 live-skipped, no network
uv run pytest --cov # coverage gate (98.7%; floor 94%)
uv run pytest # hermetic: 1267 passed + 15 skipped (11 live,
# 4 store_perf), no network
uv run pytest --cov # coverage gate (98.75%; floor 94%)
uv run pytest -m live --live # opt-in: real fleet / engines / upsd
uv run pytest -m store_perf --store-perf # opt-in: the store hot-path timings over the seeded
# store, priced against `store_perf.BUDGETS` (the cached
# reads report cold *and* warm, so a before/after never
# books a cache hit as a speed-up)
uv run python scripts/watch-store-perf.py
# ... and the live proof: 25 signed-in minutes against the
# running dev server, timing the page paths while counting
# `store-error` rows and the WAL size (DASH_WATCH_SECONDS
# shortens it; writes store-perf-watch.{log,tsv,json})
uv run python scripts/store_perf_probe.py # ... as a report (data/logs/store-perf-baseline.{md,json},
# then store-perf-task03.* for the write side and
# store-perf-task04.* for the budget table);
# --rebuild re-seeds, --dev-db <file> also times a real
# store read-only, DASH_PERF_STORE_DIR moves the cache, and
# every pass that *writes* runs on a scratch copy so the
# seeded baseline never moves under the next measurement
uv run ruff check src tests agent && uv run pyright
./scripts/capture-fixtures.sh # re-record ground truth (agents + UPS + gufo /slots)
SMART_HOSTS="deskwork-ops driveripper-ops" ./scripts/capture-fixtures.sh
@@ -837,6 +927,28 @@ branch `main`); there is no CI on it, so `bash .agents/validate.sh` is the only
verbatim, the four config rows, the `DASH_COOKIE_SECURE=1` deployment note, the three auth
gaps). No server starts and no request is made: a doc that stops naming the reset command is a
locked-out admin, and that is worth a test even though it asserts nothing about Python.
- **A cost claim is pinned as a shape and a control, never as a wall clock** (phase 14). The default
run keeps the seeded-store tests hermetic and asserts **shape and determinism** — counts,
retention windows, the statements a plan is taken from, idempotence — while the timings live behind
`-m store_perf --store-perf` and in `scripts/store_perf_probe.py`, because a budget asserted on
first writing is a number fitted to the machine it was written on. Two rules came out of it.
(1) A cheaper query is proven *equal*, not just faster:
`test_windowed_rollups_answer_exactly_what_one_full_window_rollup_answers` compares every stored
row of a windowed store against a second store rolled the old way, and
`test_a_re_rolled_bucket_only_ever_gains_rows` pins the one bug that equivalence alone misses (a
window floor that cuts into a bucket re-rolls it from its tail rows). (2) A race that comes back
clean has to be shown capable of going dirty: `store_perf` runs the pre-fix whole-window
`maintain()` as a **control** beside the windowed one and `tests/test_store.py` fails if the control
stops contending, because "0 `database is locked`" from a race that never met is the invariant-4
failure mode with a green checkmark attached. (3) **A budget is a table, not a test body**
(task 04): the 14 ceilings live in `store_perf.BUDGETS` with the measured figure and the margin in
each row's `basis`, the probe renders that table, and two default-run tests fail if a budget stops
being priced or a ceiling contradicts its own ratio partner — a ceiling nobody measures is a
comment. (4) A cache's cost claim needs the *write* cadence in the measurement: task 02's series
cache reported 0.0 ms on a quiet seeded store and served almost nothing on a live one, because
every sample write invalidated it. The pinned version is statement-counting, not timing — a poll of
existing series issues no `SELECT`, a write of a name nobody wrote ends the cache — and the live
version is `scripts/watch-store-perf.py`, which is where the defect showed up.
- Coverage gate: 80% is the repo floor, the phase gate in `.agents/validate.sh` enforces
94% (`--cov-fail-under`), and ~98% is where we are; keep it there. New/modified Python
still has to clear >90% on its own. API tests use `create_app()` with an in-memory DB,
@@ -1221,7 +1333,7 @@ memory plot; the `cpu.usage_pct@coreN` series is still stored and still queryabl
field map is written.
3. **UPS history depth.** Events and the numeric series are stored; long-window battery
health (capacity fade over months) has the rollups but no dedicated view.
4. **The dev DB is big enough to refuse a write now and then.** The dev store (~640 MB on
4. **The dev DB's `database is locked` line is fixed (phase 14); a cold-cache tail is left.** The dev store (~640 MB on
2026-10-19, its WAL grown past 140 MB at its worst) logged `store-error: OperationalError:
database is locked` a few times an hour across `system`/`llm`/`ups` through 2026-10-18; the
poller handles it exactly as invariant 4 requires — the source goes red with the real reason,
@@ -1232,8 +1344,52 @@ memory plot; the `cpu.usage_pct@coreN` series is still stored and still queryabl
phase 12's re-check on 2026-10-19 found the 190 stored `store-error` rows all older than
2026-10-18 06:33 with none in the previous 24 h of the running dev store — the absence of a
sample, not the absence of the pressure, since the file only keeps growing.
Retention/`wal_checkpoint` tuning for a long-running store is its own phase, not a
phase-09 follow-up.
Retention/`wal_checkpoint` tuning for a long-running store was its own phase rather than a
phase-09 follow-up — that phase is `14_store_performance`, and the two paragraphs below are
what it measured and changed.
**Phase 14 has since split this in two, and the read half is closed.** The in-process half was
never a SQLite lock at all: `SampleStore._lock` serialises everything, so a 1.25 s `stats()`
from a browser tab's `/api/health` poll was a poller waiting up to 1.45 s to insert 30 rows
(measured, `data/logs/NOTES-store-perf.md`). Task 02 took the two whole-table reads off the
per-request path, and `/api/health` now answers in ~1.2 ms instead of ~1.27 s, so the store
mutex is no longer held by a reader for seconds at a time. The **write half is closed too**
(task 03): `maintain()` rolls up only what became due since `last_maintain`, its prune is one
`DELETE` per resolution instead of one `OR` the planner could not use, and `PRAGMA optimize` runs
there (with SQLite's automatic ANALYZE switched off, so no poller's commit pays for it). On the
seeded 6.4 M-row store that is 5,368 ms → **2.7 ms** steady / 48 ms for a ten-minute catch-up, and
the two-connection race at the app's cadence is **100 of 100 inserts, zero `database is locked`,
max wait 21 ms** where the pre-fix control still waits 4.76 s (task 04's re-run of the same
budget: 6.8 ms steady / 46 ms catch-up / 8.5 ms worst wait, so the row is a range, not a
digit). The WAL follows the same shape: a steady pass leaves ~0 MB and a catch-up tick 4 MB
instead of 38 MB that never comes back down, and `journal_size_limit=64 MB` caps what a
long-lived reader can make it grow to (`data/logs/NOTES-store-perf.md` has the tables).
**The live proof is in** (task 04, 25 minutes / 74 passes / 296 signed-in requests against the
running dev server on the operator's own 636 MB store): **0 new `store-error` rows** (still 190,
newest 2026-10-18 06:33:30, i.e. 39 h before the watch), **WAL 6.5 → 7.77 MB** against the 67.1 MB
cap and a main file that did not move, 0 non-200 responses, and page paths at 1.8-3.3 ms median
(`/api/health`, `/api/summary`, `/api/metrics`, a 3-host × 9-metric 90-day `/api/system/history`).
The watch also caught a defect in task 02's fix — the series cache was invalidated by *every*
sample write, so on a store the hub writes to every few seconds it never served and `/api/metrics`
paid the 572 ms scan on nearly every call (measured: 535 ms per call pre-fix, 1.8 ms median
after) — and that is what the store bullet above now describes.
What is **left** is the cost of the two cold answers themselves, measured on that store: 599 ms
for `series_names('host')`, 729 ms for `stats()`' counts, now once per cache window (~1.3 s of
store mutex per minute) instead of once per call — which is why the watch's medians are 1.8-3.3 ms
and its p95s are 620/729 ms: the p95 is the window boundary, and 37 of 74 passes had one request
over 250 ms, every one of them a cold re-derivation. No cache policy reaches *that*; the fix is
the bounded `(source, metric)` structure or a counts table named in task 03's note. Two cheaper
knobs exist and were deliberately not pulled, because both change what `/api/health` answers and
move rather than remove the mutex hold: raising `STATS_CACHE_TTL_SECONDS`, and refreshing both
caches from the maintenance tick so the miss lands on a poller instead of a page load. That is a
phase, not a cleanup. Two servers on one file can still contend, and `busy_timeout=5000` is
unchanged on purpose so a second one stays visible as a slower write; what they contend with is
now milliseconds instead of 5.4 s.
One consequence worth naming, because two other gaps pointed here for it: the **~10 s
process-wide stall** (gap 9's `Δ`-window dilution, `poller`'s catch-up gaps) is gone with the
mutex holds — coincident multi-second gaps in the 2 s `llm` and 5 s `ups` raw streams went from
393 and 198 in the two halves of 2026-10-19 to **0** across the 110 minutes the phase-14 store
has run (re-measured in this phase's verification pass from the `sample` table; the signature
separates a process stall from one slow engine).
5. **The login throttle keys on the socket peer address** (phase 11), and the one
`LoginThrottle` instance now holds two keys per address: the bare one for
`POST /api/auth/login` and `password:<address>` for `POST /api/auth/password`. Behind nginx
@@ -1315,7 +1471,14 @@ memory plot; the `cpu.usage_pct@coreN` series is still stored and still queryabl
pass line and what the watch above stayed under. The `Δ<window>` label stayed honest throughout —
both before and after the fix it names the window actually used; do not "fix" anything
here by hiding or rewriting it. The ~10–15 s process-wide **stall** that triggers a
refused poll is still gap 4's and `14_store_performance`'s problem; the estimator no longer
refused poll was gap 4's and `14_store_performance`'s problem, and **phase 14 closed it**
(measured on the dev store from the sample table: coincident multi-second gaps in the 2 s `llm`
and 5 s `ups` raw streams — the signature of a process-wide stall, not a slow engine — were
393 in 00:00–12:00 and 198 in 12:00–21:36 on 2026-10-19, and **0** across the 110 minutes the
phase-14 store has been running, worst `llm` gap 7.1 s and that one the recorded gufo timeout).
The mechanism is the one phase 14 removed: a whole-table `stats()` count held the store mutex
for ~1.25 s on every browser tab's `/api/health` poll, and `maintain()` held the write lock for
5.4 s. The estimator no longer
turns a slow fetch into 0.2 s arithmetic, and one `n/a` per stall is the designed degradation
(the residual the verification pass measured is the paragraph below).
**The verification pass found one residual path and closed it** (same day, live rows in the
@@ -1341,11 +1504,47 @@ memory plot; the `cpu.usage_pct@coreN` series is still stored and still queryabl
35.6 once the freed slot settled — every value labelled, **0 of 540 above the 300 pass line**,
and no stored raw row over it since the restart either. A collapsed window is what makes a
burst *look* like a rate, so the fix for this family is the ~10 s stall itself (gap 4 /
`14_store_performance`), not a fresh threshold invented in the estimator: task 03 pinned
`14_store_performance` — which has since removed it, see gap 4 and the paragraph above, so what
is left of this family is the sub-8× skew itself), not a fresh threshold invented in the
estimator: task 03 pinned
`MIN_WINDOW_SECONDS` as an absolute floor and pinned that an in-bounds diluted row publishes
(`test_a_fallback_inside_the_documented_ceiling_still_publishes`), so a span-relative floor
would be a re-decision of that phase, not a bug fix. Do not "fix" the residual by hiding the
short window or its label.
10. **`driveripper`'s 16 drives read `n/a`: kernel 7.2's unprivileged-passthrough gate (2026-10-19).**
The agent is installed, the `disk` group grant is in place, the block nodes open, and the
page renders all 16 real drive rows — but kernel 7.2.11 blocks the *agent user's* `smartctl`
at the command level (not the open; the probe matrix is in
`data/logs/NOTES-driveripper-agent.md`): the 8 SATA drives' SMART is ATA pass-through
behind a SAT layer, and the kernel's unprivileged `SG_IO` allow-list
(`scsi_cmd_allowed()`, read-safe opcodes only) does not include it — `CAP_SYS_RAWIO` or
root unblocks it, with or without `-d sat`/`-d ata`; the 8 Samsung/WD NVMes' SMART-log
read needs the drive's ACEL to advertise the page (the 980 PRO's `5B2QGXA7`, 2020-era,
does not: `NVME_IOCTL_ADMIN_CMD: Permission denied`) or `CAP_SYS_ADMIN`; `deskwork` reads
live because its firmware does. Options, in order of narrowness: (a) accept the designed
degradation (what `/disks` renders now — the identity is real, the values are `n/a` with a
why-line); (b) `AmbientCapabilities=CAP_SYS_RAWIO` (+ `CapabilityBoundingSet`) on the unit
— unblocks exactly the 8 HDDs, is the capability the kernel names for the gate, but a
compromised agent could then write raw block I/O — test it in a `systemd-run --scope`
before committing; (c) `User=root` with the empty capability set — unblocks all 16 but
hands a token-guarded process root's file-read DAC (not recommended); (d) a Samsung
firmware update — the only clean fix for the NVMe side. **Resolved the same day with a
tighter variant of (c): `User=root` + `CapabilityBoundingSet=CAP_SYS_RAWIO CAP_SYS_ADMIN`
— all 16 drives read live** (root DAC is the price; the bounding set keeps the grant
named rather than full). Two empirical notes: a root process under the unit's *empty*
bounding set is capability-less (`CapEff: 0`), because the kernel gates check
capabilities, not uid — that is why the first `User=root` attempt still read `n/a`; and
`CapEff` came back `0000000000220000` (SYS_ADMIN + systemd's auto CHROOT; RAWIO in the
bounding set but not effective) — the 8 SATA drives' reads were unblocked by SYS_ADMIN
alone, so the effective grant is {SYS_ADMIN}; the RAWIO in the line stays as the named
intent for the `SG_IO` gate. The agent also gained the pass-budget scale-up (20 s floor
→ 5 s × disk count; an explicit override still wins, the budget test is untouched),
because a 16-disk pass takes more than 20 s when the drives answer. The `deploy/install-agent.sh`
template still ships the `DevicePolicy`/`DeviceAllow` lines that misbehaved on this host
(a deny-all filter; the `nvme*` glob matches nothing on this kernel because
`/proc/devices` lists NVMe as `blkext`) — the template change is its own phase.
`tests/fixtures/smartctl_driveripper_*.txt` (captured via `sudo`) remain the ground truth
for the drives' shape.
## Locked decisions (and why)