services/dashboards
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

phase: 14_store_performance

Browse Source 
**Phase 14 `store_performance` — final verification pass (all 4 tasks already in `complete/`)**

- Re-verified every task's work in the tree: read caches (`series_names`/`stats`), capped compound `pick_res_many`, `history_many` + `api._history_batch`, windowed `maintain()` + per-resolution prune + `PRAGMA optimize`/`automatic_analyze=0`/WAL pragmas, `store_perf.BUDGETS` (14 ceilings), probe, live watch.
- **Completion criteria:** baseline+after side by side in `data/logs/NOTES-store-perf.md` ✅ · budget tests green (`uv run pytest -m store_perf --store-perf` → **4 passed**, 14/14 inside) ✅ · 25-min dev-server watch re-run **independently** (74 passes, `scripts/watch-store-perf.py`): **0 `store-error`** (total still 190, newest 2026‑10‑18 06:33:30), WAL 7.77 MB ≤ 67.1 MB, medians 1.7–3.2 ms, p95 720/613 ms = the documented cold-cache window; a 3rd 27-pass watch on a freshly restarted server reproduced it (3.1/708, 1.8/572) ✅ · `maintain()` 5,368 ms → 2.7–6.8 ms and WAL bounded across two cycles (test + notes) ✅ · no behaviour change (all four history routes pinned against `query()`) ✅
- **Full gates:** `uv run pytest` → **1267 passed, 15 skipped**; `bash .agents/validate.sh` → **exit 0**, coverage **98.75%** (floor 94), `ruff check src tests agent` clean, `pyright` 0 errors, `node --check` ok, repo hygiene ok; `uv run pytest -m live --live` → **11 passed**; dev server up, 9/9 sources `ok`, pages 200 in 1.5–3.3 ms.
- **Docs fixed:** README/AGENTS still claimed a flat "20 s per pass" while `host_agent.py` (in this same uncommitted change) scales the budget to 5 s × disk count — corrected in both plus the two illustrative "20 s of wedged disk" sentences; broken bold (`` `driveripper** ``) repaired; stale gap-4 title + "tuning is its own phase" wording updated to say phase 14 ran; README's absolute "nothing … is answered by a whole-table scan" qualified to *per call* with the cold-window cost named; "third opt-in marker" → second; conftest `--store-perf` help named the real marker; **`data/logs/watch-store-perf.py` moved to `scripts/watch-store-perf.py`** (both docs told a clone to run a gitignored path) with root-anchored paths, `DASH_WATCH_OUT`, and dead code removed.
- **New finding recorded:** gap 9's "~10 s process-wide stall is still gap 4's / `14_store_performance`'s problem" is now closed — coincident >8 s `llm` + >12 s `ups` raw-stream gaps went 393 (00:00–12:00) and 198 (12:00–21:36) to **0** across 110+ min of the phase-14 store; written into AGENTS.md gaps 4 and 9 and the notes table.
- **Deviations:** did *not* pull either deferred cost knob (`STATS_CACHE_TTL_SECONDS`, warming caches from the maintenance tick) — AGENTS.md records that decision and its reasoning; `uv pip list --outdated` still reports certifi/fastapi/starlette/websockets (pre-existing advisory, no dependency touched this phase, gate treats it as advisory). Overwrote nothing: task 04's watch preserved as `store-perf-watch.task04.*`, my two runs as `.verify25.*` / `.current-code.*`.
- **Next pending phase:** `15_agent_mtls`, then `16_agent_smart_pass_budget` (owns the scaled-budget tests + its doc criterion, now partly pre-satisfied by this pass), `17_installer_device_filter`.
This commit is contained in:
ducoterra
2026-10-19 23:44:35 -04:00
parent 1bbd868401
commit 569ca25fdb
49 changed files with 5072 additions and 124 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.agents/phases/todo/14_store_performance/00_phase.md → .agents/phases/complete/14_store_performance/00_phase.md
View File
.agents/phases/todo/14_store_performance/01_baseline_store_costs.md → .agents/phases/complete/14_store_performance/01_baseline_store_costs.md
View File
.agents/phases/todo/14_store_performance/02_no_full_scan_on_hot_paths.md → .agents/phases/complete/14_store_performance/02_no_full_scan_on_hot_paths.md
View File
.agents/phases/todo/14_store_performance/03_windowed_rollups_and_wal_tuning.md → .agents/phases/complete/14_store_performance/03_windowed_rollups_and_wal_tuning.md
View File
.agents/phases/todo/14_store_performance/04_budget_gates_and_live_proof.md → .agents/phases/complete/14_store_performance/04_budget_gates_and_live_proof.md
View File
.agents/phases/todo/16_agent_smart_pass_budget/00_phase.md
+42
View File
@@ -0,0 +1,42 @@
# Phase 16 — Pin the agent's scaled smart-pass budget
**Source:** chat 2026-10-19 — the `driveripper` 16-drive host: the flat 20 s smart-pass
budget truncates a 16-device pass (a real read takes ~30–80 s), and because the deadline
resets each pass, the tail drives would have stayed `smart: null` every pass. The fix —
budget = 20 s floor scaled to 5 s × disk count, an explicit override still winning — was
applied to the working tree on 2026-10-19 (operator-confirmed, verified on 16/16 live
drives). This phase pins that already-applied change with tests and documentation.
**Story:** n/a
**Context:** `agent/host_agent.py` (`_smart_pass` — the scaled `budget = max(SMARTCTL_BUDGET_SECONDS, self.smart_timeout_seconds * len(device_names)) when the default is in force, else the explicit override; the `pass budget of {budget:.0f}s exhausted` error line; the module docstring "5 s per device and a pass budget of 20 s scaled to the disk count"), `tests/test_agent.py::test_the_pass_budget_skips_the_rest_with_one_error` (sets `collector.smart_budget_seconds = 0.0` — the override-wins branch), `AGENTS.md` line 185 ("5 s per device / 20 s per pass / 240 s cache") and the `deploy/install-agent.sh` header comment (line 25 area, "bounded 5 s per device and 20 s per pass" if still present there), `README.md` line 94 ("20 s per pass, results cached 4 minutes")
## Objective
The smart pass budget scale-up that is already in the working tree is pinned: a test proves the *default* scales (16 devices → 80 s deadline, one device → 20 s floor, the exhaustion line names the scaled figure), the override-wins branch stays covered, and every doc that still says "20 s per pass" says the floor-scaled truth. No new behaviour — the code change is already deployed on `driveripper` and verified live.
## Dependencies
- `09_disks_smartctl` (complete) — the smart pass, its bounds, and `tests/test_agent.py`'s in-process agent harness
## Tasks
1. `01_pin_the_scaled_budget.md` — the tests that the default budget scales with the disk count and that the error line names the scaled figure
2. `02_docs_and_gate.md` — every "20 s per pass" mention in `AGENTS.md`/`README.md`/`agent/host_agent.py` says the floor-scaled truth; the gate is green
## Testing & Quality
- Unit: `tests/test_agent.py` — the scaled default (a default-attribute collector over 16 device names exhausts only when the 80 s budget runs out; over one device the floor is 20 s; the exhaustion error line carries the scaled figure, not the floor)
- Coverage: **>90%** on the modified code (the `_smart_pass` branch)
- The phase is "Complete" only when the **entire** project test suite passes 100% (every test in the project, not just this phase's)
## Documentation & Consistency
- The final pass reviews `README.md` and `AGENTS.md` in full against the project as it stands after this phase — fixing every inconsistency found, **including pre-existing ones not caused by this phase**.
- Tasks update `README.md`/`AGENTS.md` immediately when their own changes are user-visible (the smart-pass bounds are a documented contract of `host_agent.py`).
## Completion Criteria
- [ ] `uv run pytest tests/test_agent.py` green, including the new scaled-budget tests and the untouched override test
- [ ] `grep -rn "20 s per pass" AGENTS.md README.md agent/ deploy/` returns nothing (the wording is floor-scaled everywhere)
- [ ] `bash .agents/validate.sh` green (pytest --cov --cov-fail-under=94, ruff, pyright, node --check)
- [ ] README.md and AGENTS.md are accurate and consistent with the whole project (verified in the final pass)
- [ ] no behavior change in completed work: the deployed agent on `driveripper` is byte-identical to the repo's `agent/host_agent.py` (md5), the live suite is green
.agents/phases/todo/16_agent_smart_pass_budget/01_pin_the_scaled_budget.md
+26
View File
@@ -0,0 +1,26 @@
# Task 01 — Pin the scaled smart-pass budget
**Phase:** `16_agent_smart_pass_budget` · **Story:** n/a
## Objective
The default smart-pass budget (20 s floor, scaled to 5 s × disk count) is proven by tests: the deadline actually moves with the disk count, the exhaustion error line names the *scaled* figure, and the existing override-wins branch (a test sets `smart_budget_seconds = 0.0`) keeps passing untouched. The code change is already in `agent/host_agent.py::_smart_pass` — this task adds the tests, it does not rewrite the code.
## Work
1. `tests/test_agent.py` — add a helper that builds the in-process collector with **default** smart attributes (do not touch `smart_budget_seconds`/`smart_timeout_seconds`) and a `smartctl` fake that records each device's deadline window, in the same style as `test_the_pass_budget_skips_the_rest_with_one_error` (which stays untouched):
- `test_the_pass_budget_scales_with_the_disk_count`: 16 device names → the last device's `smartctl` run happens with `left` (its deadline margin) consistent with a ~80 s budget (assert the deadline the code computes: `max(20, 5 × 16)`; the fake binary is instant, so assert via the recorded `min(left, 5.0)` timeout being 5.0 for a late device and the pass *not* truncating — every device got a `smartctl` run); 1 device name → budget 20 s (assert the recorded timeout is 5.0 and, with the fake made slow enough to exhaust a 20 s but not an 80 s budget in a *separate* case if feasible — if the harness cannot fake wall time, assert the computed `budget` expression via the error-line route instead: force exhaustion with a `time.monotonic` monkeypatch offset).
- `test_the_pass_budget_error_line_names_the_scaled_figure`: force exhaustion on 16 devices (monkeypatch the clock forward past the 80 s deadline between two device runs, as the existing test's `0.0` override does for its 0 s budget) → the error line reads `smart: pass budget of 80s exhausted, N disk(s) skipped` — the scaled figure, not the 20 s floor.
2. `agent/host_agent.py` — **no code change expected.** If a test exposes a real bug in the already-applied change (e.g. the `== SMARTCTL_BUDGET_SECONDS` default check), fix it minimally and record the diff in the task's completion note; do not redesign.
## Testing & Quality
- Unit: the two new tests above + the untouched `test_the_pass_budget_skips_the_rest_with_one_error` (override wins: `0.0` still exhausts immediately on 3 devices with one error line)
- Coverage: **>90%** on `agent/host_agent.py::_smart_pass`'s budget branch (the default-scaled path must now be exercised, not just the override)
## Completion Criteria
- [ ] `uv run pytest tests/test_agent.py -q` green: the two new tests, the untouched override test, and every pre-existing agent test
- [ ] the 80 s case asserts the *scaled* figure in the error line (a test that passed with a 20 s budget would fail on the scaled code — no tautological assertions)
- [ ] `uv run ruff check agent tests` and `uv run pyright` clean
- [ ] no behavior change in completed work: `agent/host_agent.py` unchanged unless a real bug was found (then the fix is minimal and recorded)
.agents/phases/todo/16_agent_smart_pass_budget/02_docs_and_gate.md
+25
View File
@@ -0,0 +1,25 @@
# Task 02 — Docs and gate for the scaled budget
**Phase:** `16_agent_smart_pass_budget` · **Story:** n/a
## Objective
Every place that still says the smart pass is "20 s per pass" says the floor-scaled truth (20 s floor, 5 s × disk count), and the phase gate is green. The agent's module docstring was already updated on 2026-10-19 to the scaled wording — this task verifies it and fixes the two remaining doc lines.
## Work
1. `AGENTS.md` line ~185 (the `host_agent.py` bullet): `5 s per device / 20 s per pass / 240 s cache` → `5 s per device / a pass budget of 20 s scaled to 5 s × disk count / 240 s cache` (keep the bullet's voice; one line).
2. `README.md` line ~94: `20 s per pass, results cached 4 minutes` → `a pass budget of 20 s scaled to the disk count (5 s each), results cached 4 minutes` (one line).
3. `agent/host_agent.py` module docstring — verify it reads `bounded 5 s per device and a pass budget of 20 s scaled to the disk count, cached 240 s`; fix only if it drifted.
4. `bash .agents/validate.sh` — the four checks (pytest --cov --cov-fail-under=94, ruff, pyright, node --check) + repo hygiene, all `uv run`.
## Testing & Quality
- No new logic in this task; the gate run is the check.
- Coverage: **>90%** gate holds (task 01's tests carry the new branch).
## Completion Criteria
- [ ] `grep -rn "20 s per pass" AGENTS.md README.md agent/ deploy/` returns nothing
- [ ] `bash .agents/validate.sh` green
- [ ] the two edited doc lines read the floor-scaled truth and nothing else changed in either file
.agents/phases/todo/17_installer_device_filter/00_phase.md
+59
View File
@@ -0,0 +1,59 @@
# Phase 17 — The installer's broken device filter and the kernel-gate variant
**Source:** chat 2026-10-19 — the `driveripper` install: the template's
`DevicePolicy=strict` + `DeviceAllow` lines produced a **deny-all device filter** on
kernel 7.2 (systemd resolves `DeviceAllow` globs by name against `/proc/devices`, where
this kernel lists NVMe as `blkext` (major 259), so `nvme*` matched nothing and the whole
device namespace was locked). The operator removed the lines by hand; the unit now also
runs `User=root` + `CapabilityBoundingSet=CAP_SYS_RAWIO CAP_SYS_ADMIN` (kernel 7.2's
unprivileged-passthrough gate — `data/logs/NOTES-driveripper-agent.md`). The template
still ships the broken lines, `User=$AGENT_USER`, and an empty bounding set, and re-running
the installer on `driveripper` would **clobber the working unit** (unconditional
`cat > "$UNIT"`). This phase makes the template the single source of the working unit.
**Story:** n/a
**Context:** `deploy/install-agent.sh` (`do_install` — the unconditional unit heredoc with `DevicePolicy=strict` + nine `DeviceAllow` lines, the `PrivateDevices=false` rationale comment block, `User=$AGENT_USER`, `CapabilityBoundingSet=`, the `--bind/--port/--token/--rotate-token/--status/--uninstall` flag set), `tests/test_deploy_scripts.py` (the fake-root install sandbox `test_install_path_installs_into_its_own_paths_and_prints_the_token_once`, the token-file permission test, the `bash -n` + function-naming pins), `AGENTS.md` (the fleet bullet "The deployed unit has since diverged from the template in the three ways the SMART block below records", gap 10's "the template change is its own phase"), `README.md` (the install section, line ~46 `sudo deploy/install-agent.sh --bind <addr> --port 9120`)
## Objective
Re-running `deploy/install-agent.sh` on any host — including `driveripper`, with
`--as-root --caps CAP_SYS_RAWIO,CAP_SYS_ADMIN --force` — produces the working unit: no
`DevicePolicy`/`DeviceAllow` lines (device access falls back to DAC via the `disk` group
+ SELinux, the standard SMART-monitoring arrangement), the kernel-gate variant expressible
by flags, and a re-run that would change an existing unit prints the diff and refuses
without `--force`. The default install (dedicated user, empty bounding set) is otherwise
byte-identical to what the template wrote before this phase.
## Dependencies
- `09_disks_smartctl` (complete) — the smart pass the unit exists to serve
- `15_agent_mtls` (todo) — **shared-file, run-order note**: 15's task 06 adds `--cert-bundle`/`RuntimeDirectory` to the same `do_install`; apply on top of the file state at the time it runs, never restore a line the other phase owns. If 15 lands first, its new lines survive this phase's filter-line removal untouched.
## Tasks
1. `01_drop_the_broken_device_filter.md` — remove `DevicePolicy=strict` + all nine `DeviceAllow` lines and rewrite the rationale comment block
2. `02_root_and_caps_flags.md` — `--as-root` and `--caps CAP1,CAP2` write the kernel-gate unit lines
3. `03_no_silent_clobber.md` — a re-run that would change an existing unit prints the diff and refuses without `--force`
4. `04_tests_docs_gate.md` — the unit pins in `tests/test_deploy_scripts.py`, `AGENTS.md`/`README.md` updated, gate green
## Testing & Quality
- Integration: `tests/test_deploy_scripts.py` — the template unit carries **no** `DevicePolicy`/`DeviceAllow` line; a default-install sandbox unit is byte-identical to the pre-phase template unit minus the filter lines; `--as-root --caps CAP_SYS_RAWIO,CAP_SYS_ADMIN` writes `User=root` and `CapabilityBoundingSet=CAP_SYS_RAWIO CAP_SYS_ADMIN` exactly; the re-run guard refuses (exit 1, diff on stdout) and overwrites with `--force`; `bash -n` and the existing function-naming pin still pass
- Live (task 04, operator-verified): `driveripper`'s deployed unit is reproduced by `sudo deploy/install-agent.sh --bind 10.1.0.126 --port 9120 --as-root --caps CAP_SYS_RAWIO,CAP_SYS_ADMIN --force` — `systemctl diff` empty, `CapEff` still `0000000000220000`, 16/16 drives still live; `deskwork`/`gateway` untouched (their units are not re-run)
- Coverage: the script itself is pinned by text assertions + `bash -n` (house pattern; >90% applies to the new Python test code)
- The phase is "Complete" only when the **entire** project test suite passes 100% (every test in the project, not just this phase's)
## Documentation & Consistency
- The final pass reviews `README.md` and `AGENTS.md` in full against the project as it stands after this phase — fixing every inconsistency found, **including pre-existing ones not caused by this phase**.
- Tasks update `README.md`/`AGENTS.md` immediately when their own changes are user-visible (the new flags are operator-facing; gap 10's "the template change is its own phase" becomes "done in phase 17" with the `driveripper` flag line).
## Completion Criteria
- [ ] `grep -n "DevicePolicy\|DeviceAllow" deploy/install-agent.sh` returns nothing (the `PrivateDevices=false` line stays)
- [ ] `sudo deploy/install-agent.sh --help` names `--as-root` and `--caps` with the kernel-gate sentence
- [ ] a fresh sandbox install (fake root, no flags) produces a unit byte-identical to the pre-phase template unit minus the filter lines (pinned in `tests/test_deploy_scripts.py`)
- [ ] a re-run that would change an existing unit exits 1 with the diff on stdout; with `--force` it overwrites
- [ ] operator-verified on `driveripper`: the flags reproduce the working unit (`systemctl diff` empty, 16/16 drives live after restart)
- [ ] `uv run pytest` green, `bash .agents/validate.sh` green
- [ ] README.md and AGENTS.md are accurate and consistent with the whole project (verified in the final pass)
- [ ] no behavior change in completed work: the default install's unit (minus the filter lines) and every existing flag behave as before; the live suite is green
.agents/phases/todo/17_installer_device_filter/01_drop_the_broken_device_filter.md
+25
View File
@@ -0,0 +1,25 @@
# Task 01 — Drop the broken device filter
**Phase:** `17_installer_device_filter` · **Story:** n/a
## Objective
The template unit no longer carries `DevicePolicy=strict` or any `DeviceAllow` line, and the comment block that justified them is rewritten to say why the *absence* is the fix. The `PrivateDevices=false` line stays (explicit, default behaviour, and the rest of the lockdown is unchanged).
## Work
1. `deploy/install-agent.sh` — in the `do_install` unit heredoc: delete the `DevicePolicy=strict` line and all nine `DeviceAllow=…` lines (`/dev/null`, `/dev/zero`, `/dev/full`, `/dev/random`, `/dev/urandom`, `/dev/nvme*`, `/dev/sd*`, `/dev/vd*`, `/dev/sg*`).
2. `deploy/install-agent.sh` — rewrite the comment block above `PrivateDevices=false` (the "The pseudo nodes are not decoration…" paragraph) to the new truth: **no device filter on purpose** — systemd resolves `DeviceAllow` globs by name against `/proc/devices` (fnmatch in `bpf_devices_allow_list_major`), where kernel 7.2 lists NVMe as `blkext` (major 259), so `nvme*` matched nothing and `DevicePolicy=strict` denied the *whole* device namespace (the `driveripper` install went deny-all, root included; `data/logs/NOTES-driveripper-agent.md`). Without the policy, device access falls back to DAC (the `disk` group the script already grants) + SELinux — the standard SMART-monitoring arrangement. Keep the one operational fact that survives: `PrivateDevices=false` is the default (a `true` sandbox with no `DeviceAllow` list is what dies at spawn with `status=208/STDIN` — keep that warning sentence, it is still why the line is written explicitly).
3. `deploy/install-agent.sh` — the script's top-of-file header comment (line ~25: "the unit device-allows the disk nodes") → "the unit applies no device filter (kernel 7.2's `/proc/devices` name matching made the old `nvme*` glob deny-all); device access is DAC via the `disk` group + SELinux".
## Testing & Quality
- Integration: `bash -n deploy/install-agent.sh` clean (the existing pin covers it); a quick sandbox run (the `test_install_path_installs_into_its_own_paths_and_prints_the_token_once` pattern, run manually) shows no `DevicePolicy`/`DeviceAllow` in the written unit — the formal pins are task 04, this task keeps the change small and reviewable.
- Coverage: **>90%** n/a (shell script; the pins are text assertions per the house pattern).
## Completion Criteria
- [ ] `grep -n "DevicePolicy\|DeviceAllow" deploy/install-agent.sh` returns nothing
- [ ] `bash -n deploy/install-agent.sh` clean
- [ ] a manual sandbox install (fake root, `--bind 127.0.0.1 --port 9199`) writes a unit with the filter lines gone, `PrivateDevices=false` present, and every other line byte-identical to the pre-phase template unit
- [ ] the rewritten comment names the `/proc/devices`/`blkext` mechanism in one sentence (an executor reading only the script can understand *why* there is no filter)
.agents/phases/todo/17_installer_device_filter/02_root_and_caps_flags.md
+27
View File
@@ -0,0 +1,27 @@
# Task 02 — `--as-root` and `--caps` flags
**Phase:** `17_installer_device_filter` · **Story:** n/a
## Objective
The kernel-gate variant of the unit (what `driveripper` runs) is expressible from the template: `--as-root` writes `User=root` instead of `User=$AGENT_USER`, and `--caps CAP1,CAP2` (comma-separated, e.g. `--caps CAP_SYS_RAWIO,CAP_SYS_ADMIN`) writes `CapabilityBoundingSet=CAP1 CAP2` instead of the empty `CapabilityBoundingSet=` line. Both default off; a flag without its friend is refused with a message.
## Work
1. `deploy/install-agent.sh` — flag parsing: add `--as-root` (sets `AS_ROOT=1`) and `--caps <list>` (sets `CAPS="<list>"`, validates non-empty, rejects an empty list with exit 1) in the existing `case`/flag block, and `--help` text:
- `--as-root run the service as root (for a kernel-gated host: kernel 7.2 gates unprivileged smartctl passthrough behind capabilities, not uid — a root process under the default empty bounding set is capability-less, so this flag needs --caps to mean anything)`
- `--caps LIST CapabilityBoundingSet line for the unit (comma-separated, e.g. CAP_SYS_RAWIO,CAP_SYS_ADMIN); default: empty (capability-less service)`
2. `deploy/install-agent.sh` — validation: `--as-root` without `--caps` → print "note: --as-root with the default empty CapabilityBoundingSet leaves the process capability-less (the kernel gates check capabilities, not uid); pair it with --caps for a kernel-gated host" and **still proceed** (a capability-less root is a legitimate, if unusual, choice — do not hard-refuse; the note is the operator's reminder). `--caps` without `--as-root` → refuse, exit 1 ("--caps needs --as-root: a capability grant on the service user is not the kernel-gate fix").
3. `deploy/install-agent.sh` — the unit heredoc: `User=$AGENT_USER` → `User=$( [ "$AS_ROOT" -eq 1 ] && echo root || echo "$AGENT_USER" )`; the empty `CapabilityBoundingSet=` line → the `CAPS` list converted comma→space when set. Add one comment line above: `# --caps: kernel 7.2 gates unprivileged smartctl passthrough (NVMe admin log: CAP_SYS_ADMIN; SG_IO allow-list: CAP_SYS_RAWIO); a root process needs them named, because an empty bounding set bounds even root to CapEff 0`.
4. `deploy/install-agent.sh` — the `--status` probe is unchanged (it probes the agent over HTTP; the user the unit runs as is invisible to it). The token file stays `0640 root:$AGENT_USER` — with `--as-root` the group read is irrelevant (root reads it), so do **not** change the token-file logic.
## Testing & Quality
- Integration: manual sandbox runs (fake root): default → `User=dashboards-agent` + empty `CapabilityBoundingSet=`; `--as-root --caps CAP_SYS_RAWIO,CAP_SYS_ADMIN` → `User=root` + `CapabilityBoundingSet=CAP_SYS_RAWIO CAP_SYS_ADMIN`; `--caps` alone → exit 1 with the message; `--as-root` alone → proceeds with the note on stdout. (Formal pins: task 04.)
- Coverage: **>90%** n/a (shell script; text pins per the house pattern).
## Completion Criteria
- [ ] `sudo deploy/install-agent.sh --help` names both flags with the kernel-gate sentence (a fresh clone's executor can learn the `driveripper` recipe from `--help` alone)
- [ ] the two sandbox units match the exact lines above (byte-for-byte on the `User=` and `CapabilityBoundingSet=` lines)
- [ ] `bash -n deploy/install-agent.sh` clean; every pre-existing flag (`--bind/--port/--token/--rotate-token/--status/--uninstall`) behaves as before in a sandbox run
.agents/phases/todo/17_installer_device_filter/03_no_silent_clobber.md
+27
View File
@@ -0,0 +1,27 @@
# Task 03 — No silent clobber on re-run
**Phase:** `17_installer_device_filter` · **Story:** n/a
## Objective
A re-run that would change an already-installed unit stops: it prints the diff between the installed unit and the unit the script would write, and exits 1 without touching the file. `--force` proceeds with the overwrite. A re-run whose generated unit is identical to the installed one proceeds silently (idempotent re-runs stay the zero-noise path the script has always had).
## Work
1. `deploy/install-agent.sh` — add a `--force` flag to the flag block (`FORCE=1`) and `--help` text: `--force overwrite an existing unit even if the generated unit differs (prints the diff first)`.
2. `deploy/install-agent.sh` — in `do_install`, immediately before the `cat > "$UNIT"` heredoc write: build the would-be unit into a temp file (the heredoc already exists — render it once to `$(mktemp)`), and if `"$UNIT"` exists and `! cmp -s "$UNIT" "$tmp_unit"`:
- if `FORCE=1`: print the diff (`diff -u "$UNIT" "$tmp_unit"` on stdout) and proceed with the overwrite;
- else: print `refusing to overwrite $UNIT (generated unit differs); run with --force to apply:` + the same diff, and `exit 1` **before** the token-file write, the `systemctl daemon-reload`, and the `systemctl enable --now` — a refused run must leave the host exactly as it was (token file, unit, and running service all untouched; note the token-reuse logic runs earlier in `do_install` and only *reads* the existing token, so ordering the guard before the unit write is enough).
- identical unit → no output, normal proceed.
3. `deploy/install-agent.sh` — `--uninstall` is unchanged (it already removes the unit deliberately; the guard applies to `do_install` only).
## Testing & Quality
- Integration: manual sandbox sequence (fake root): install → modify the installed unit by one line → re-run with the same flags → exit 1, diff on stdout, installed unit byte-identical to the modified version, service still the old one; re-run with `--force` → diff printed, unit overwritten; re-run again unchanged → silent, exit 0. (Formal pins: task 04.)
- Coverage: **>90%** n/a (shell script; text/behaviour pins per the house pattern).
## Completion Criteria
- [ ] the three-way sequence above (refuse → force → silent) holds in the sandbox, with the token file and the running service untouched by a refused run
- [ ] `--force` prints the diff *before* overwriting (an operator sees exactly what changed)
- [ ] `bash -n deploy/install-agent.sh` clean; a fresh install (no existing unit) is unaffected by the guard
.agents/phases/todo/17_installer_device_filter/04_tests_docs_gate.md
+34
View File
@@ -0,0 +1,34 @@
# Task 04 — Pins, docs, gate
**Phase:** `17_installer_device_filter` · **Story:** n/a
## Objective
The template's new shape is pinned in `tests/test_deploy_scripts.py` (no filter lines, the flags' exact lines, the clobber guard's refuse/force/silent sequence), `AGENTS.md` and `README.md` name the `driveripper` recipe, and the gate is green. The operator-verified `driveripper` reproduction (the completion criterion in `00_phase.md`) is the last check.
## Work
1. `tests/test_deploy_scripts.py` — extend the existing sandbox pattern:
- `test_the_template_unit_has_no_device_filter_lines` — read `deploy/install-agent.sh`'s unit heredoc region and assert no line starts with `DevicePolicy=` or `DeviceAllow=`, and `PrivateDevices=false` is still present (the filter's absence is the fix; a regression that re-adds `DeviceAllow=/dev/nvme*` is the deny-all returning).
- `test_a_default_install_unit_is_the_pre_phase_unit_minus_the_filter_lines` — the sandbox-installed unit equals the known pre-phase template unit text minus the ten filter lines (pin the whole unit, so a future template line drift fails here).
- `test_as_root_and_caps_write_the_kernel_gate_unit_lines` — `--as-root --caps CAP_SYS_RAWIO,CAP_SYS_ADMIN` → `User=root` and `CapabilityBoundingSet=CAP_SYS_RAWIO CAP_SYS_ADMIN` exactly; `--caps` alone → exit 1 with the `--as-root` message; `--as-root` alone → proceeds with the capability-less note on stdout.
- `test_a_re_run_that_differs_refuses_without_force_and_applies_with_it` — the refuse/force/silent sequence (exit 1 + diff on stdout + installed unit untouched; `--force` overwrites with the diff printed; an unchanged re-run is silent).
2. `AGENTS.md` — the fleet bullet's "re-running `deploy/install-agent.sh` on driveripper would clobber the working unit, because it rewrites the unit unconditionally (gap 10)" → the post-phase truth: the installer no longer writes the device-filter lines, and the `driveripper` unit is reproducible with `sudo deploy/install-agent.sh --bind 10.1.0.126 --port 9120 --as-root --caps CAP_SYS_RAWIO,CAP_SYS_ADMIN --force` (a re-run that would change an existing unit refuses without `--force`). Update gap 10's "the template change is its own phase" → "done in phase 17" with the flag line.
3. `README.md` — the install section: one line naming the flags' purpose (the kernel-gated-host recipe), keeping the how-to voice.
4. `bash .agents/validate.sh` — all four checks green.
5. **Operator-verified (live, read-only + one restart):** on `driveripper`, `sudo deploy/install-agent.sh --bind 10.1.0.126 --port 9120 --as-root --caps CAP_SYS_RAWIO,CAP_SYS_ADMIN --force` → `systemctl diff dashboards-agent` empty, `systemctl is-active` `active`, `CapEff` still `0000000000220000`, and 16/16 drives still live in the next `/snapshot` (the restart is the only fleet action; `deskwork`/`gateway` are not re-run). Screenshot `data/logs/disks-17-installer-repro.png` per invariant 10.
## Testing & Quality
- Integration: the four new pins above + the pre-existing pins (sudoers, `bash -n`, function naming, token-file permissions) all green.
- Coverage: **>90%** on the new test code; the 94% gate holds.
- Live: the operator-verified reproduction (item 5) — the phase's `driveripper` criterion.
## Completion Criteria
- [ ] the four new pins pass and every pre-existing `test_deploy_scripts.py` pin passes
- [ ] `grep -n "DevicePolicy\|DeviceAllow" deploy/install-agent.sh` returns nothing
- [ ] `AGENTS.md` fleet bullet + gap 10 and `README.md` name the `driveripper` flag recipe
- [ ] the operator-verified reproduction: `systemctl diff` empty, 16/16 drives live, screenshot in `data/logs/`
- [ ] `uv run pytest` green, `bash .agents/validate.sh` green
- [ ] the live suite (`set -a; . ./.env.dev; set +a; uv run pytest -m live --live`) is green
.agents/reports/14_store_performance/14_store_performance__00_phase.a1.err
View File
.agents/reports/14_store_performance/14_store_performance__00_phase.a1.md
+9
View File
@@ -0,0 +1,9 @@
**Phase 14 `store_performance` — final verification pass (all 4 tasks already in `complete/`)**
- Re-verified every task's work in the tree: read caches (`series_names`/`stats`), capped compound `pick_res_many`, `history_many` + `api._history_batch`, windowed `maintain()` + per-resolution prune + `PRAGMA optimize`/`automatic_analyze=0`/WAL pragmas, `store_perf.BUDGETS` (14 ceilings), probe, live watch.
- **Completion criteria:** baseline+after side by side in `data/logs/NOTES-store-perf.md` ✅ · budget tests green (`uv run pytest -m store_perf --store-perf` → **4 passed**, 14/14 inside) ✅ · 25-min dev-server watch re-run **independently** (74 passes, `scripts/watch-store-perf.py`): **0 `store-error`** (total still 190, newest 2026‑10‑18 06:33:30), WAL 7.77 MB ≤ 67.1 MB, medians 1.7–3.2 ms, p95 720/613 ms = the documented cold-cache window; a 3rd 27-pass watch on a freshly restarted server reproduced it (3.1/708, 1.8/572) ✅ · `maintain()` 5,368 ms → 2.7–6.8 ms and WAL bounded across two cycles (test + notes) ✅ · no behaviour change (all four history routes pinned against `query()`) ✅
- **Full gates:** `uv run pytest` → **1267 passed, 15 skipped**; `bash .agents/validate.sh` → **exit 0**, coverage **98.75%** (floor 94), `ruff check src tests agent` clean, `pyright` 0 errors, `node --check` ok, repo hygiene ok; `uv run pytest -m live --live` → **11 passed**; dev server up, 9/9 sources `ok`, pages 200 in 1.5–3.3 ms.
- **Docs fixed:** README/AGENTS still claimed a flat "20 s per pass" while `host_agent.py` (in this same uncommitted change) scales the budget to 5 s × disk count — corrected in both plus the two illustrative "20 s of wedged disk" sentences; broken bold (`` `driveripper** ``) repaired; stale gap-4 title + "tuning is its own phase" wording updated to say phase 14 ran; README's absolute "nothing … is answered by a whole-table scan" qualified to *per call* with the cold-window cost named; "third opt-in marker" → second; conftest `--store-perf` help named the real marker; **`data/logs/watch-store-perf.py` moved to `scripts/watch-store-perf.py`** (both docs told a clone to run a gitignored path) with root-anchored paths, `DASH_WATCH_OUT`, and dead code removed.
- **New finding recorded:** gap 9's "~10 s process-wide stall is still gap 4's / `14_store_performance`'s problem" is now closed — coincident >8 s `llm` + >12 s `ups` raw-stream gaps went 393 (00:00–12:00) and 198 (12:00–21:36) to **0** across 110+ min of the phase-14 store; written into AGENTS.md gaps 4 and 9 and the notes table.
- **Deviations:** did *not* pull either deferred cost knob (`STATS_CACHE_TTL_SECONDS`, warming caches from the maintenance tick) — AGENTS.md records that decision and its reasoning; `uv pip list --outdated` still reports certifi/fastapi/starlette/websockets (pre-existing advisory, no dependency touched this phase, gate treats it as advisory). Overwrote nothing: task 04's watch preserved as `store-perf-watch.task04.*`, my two runs as `.verify25.*` / `.current-code.*`.
- **Next pending phase:** `15_agent_mtls`, then `16_agent_smart_pass_budget` (owns the scaled-budget tests + its doc criterion, now partly pre-satisfied by this pass), `17_installer_device_filter`.
.agents/reports/14_store_performance/14_store_performance__00_phase.a1.md.status
+1
View File
@@ -0,0 +1 @@
ok
.agents/reports/14_store_performance/14_store_performance__00_phase.a1.validate
+64
View File
@@ -0,0 +1,64 @@
== uv run pytest --cov (hermetic: no network, live tests skip) ==
........................................................................ [ 5%]
........................................................................ [ 11%]
........................................................................ [ 16%]
........................................................................ [ 22%]
.............................................s.......................... [ 28%]
........................................................................ [ 33%]
........................................................................ [ 39%]
........................................................................ [ 44%]
........................................................................ [ 50%]
........................................................................ [ 56%]
........................................................................ [ 61%]
........................sssssssssss..................................... [ 67%]
........................................................................ [ 73%]
........................................................................ [ 78%]
........................................................................ [ 84%]
........................................................................ [ 89%]
........................................................................ [ 95%]
sss....................................................... [100%]
=============================== warnings summary ===============================
.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.14.4-final-0 ________________
Name Stmts Miss Cover Missing
------------------------------------------------------------------
src/dashboards/__init__.py 3 0 100%
src/dashboards/__main__.py 14 1 93% 32
src/dashboards/alerts.py 176 0 100%
src/dashboards/api.py 356 0 100%
src/dashboards/app.py 178 0 100%
src/dashboards/auth.py 265 0 100%
src/dashboards/collectors.py 155 5 97% 141-142, 146-147, 164
src/dashboards/config.py 120 0 100%
src/dashboards/debug.py 14 0 100%
src/dashboards/poller.py 165 3 98% 192-194, 279
src/dashboards/prom.py 112 5 96% 126, 163, 177, 180, 183
src/dashboards/rates.py 73 0 100%
src/dashboards/severity.py 49 0 100%
src/dashboards/sources/__init__.py 0 0 100%
src/dashboards/sources/disks.py 130 0 100%
src/dashboards/sources/host.py 199 1 99% 147
src/dashboards/sources/llm.py 178 1 99% 285
src/dashboards/sources/nut.py 413 20 95% 153, 172, 186, 198, 204, 217, 225, 241, 319-320, 322, 579, 591, 626, 657, 661, 718-719, 724, 736
src/dashboards/sources/nut_sim.py 134 0 100%
src/dashboards/store.py 313 2 99% 553-554
------------------------------------------------------------------
TOTAL 3047 38 99%
Required test coverage of 94% reached. Total coverage: 98.75%
1267 passed, 15 skipped, 1 warning in 110.62s (0:01:50)
== uv run ruff check src tests agent ==
All checks passed!
== uv run pyright ==
0 errors, 0 warnings, 0 informations
== node --check src/dashboards/web/static/*.js ==
advisory: dependencies are out of date (AGENTS.md rule 8 — update on purpose):
certifi 2026.7.22 2026.10.19 wheel
fastapi 0.143.5 0.150.0 wheel
starlette 1.8.0 1.9.0 wheel
websockets 17.2 17.3 wheel
.agents/reports/14_store_performance/14_store_performance__01_baseline_store_costs.a1.err
View File
.agents/reports/14_store_performance/14_store_performance__01_baseline_store_costs.a1.md
+1
View File
@@ -0,0 +1 @@
Now let me write the baseline NOTES document.
.agents/reports/14_store_performance/14_store_performance__01_baseline_store_costs.a1.md.status
+1
View File
@@ -0,0 +1 @@
ok
.agents/reports/14_store_performance/14_store_performance__01_baseline_store_costs.a1.validate
+63
View File
@@ -0,0 +1,63 @@
== uv run pytest --cov (hermetic: no network, live tests skip) ==
........................................................................ [ 5%]
........................................................................ [ 11%]
........................................................................ [ 17%]
........................................................................ [ 23%]
........................................................................ [ 29%]
........................................................................ [ 35%]
........................................................................ [ 41%]
........................................................................ [ 47%]
........................................................................ [ 53%]
........................................................................ [ 59%]
........................................................................ [ 65%]
....................sssssssssss......................................... [ 71%]
........................................................................ [ 77%]
........................................................................ [ 83%]
........................................................................ [ 89%]
........................................................................ [ 95%]
...s....................................................... [100%]
=============================== warnings summary ===============================
.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.14.4-final-0 ________________
Name Stmts Miss Cover Missing
------------------------------------------------------------------
src/dashboards/__init__.py 3 0 100%
src/dashboards/__main__.py 14 1 93% 32
src/dashboards/alerts.py 176 0 100%
src/dashboards/api.py 348 0 100%
src/dashboards/app.py 178 0 100%
src/dashboards/auth.py 265 0 100%
src/dashboards/collectors.py 155 5 97% 141-142, 146-147, 164
src/dashboards/config.py 120 0 100%
src/dashboards/debug.py 14 0 100%
src/dashboards/poller.py 165 3 98% 192-194, 279
src/dashboards/prom.py 112 5 96% 126, 163, 177, 180, 183
src/dashboards/rates.py 73 0 100%
src/dashboards/severity.py 49 0 100%
src/dashboards/sources/__init__.py 0 0 100%
src/dashboards/sources/disks.py 130 0 100%
src/dashboards/sources/host.py 199 1 99% 147
src/dashboards/sources/llm.py 178 1 99% 285
src/dashboards/sources/nut.py 413 20 95% 153, 172, 186, 198, 204, 217, 225, 241, 319-320, 322, 579, 591, 626, 657, 661, 718-719, 724, 736
src/dashboards/sources/nut_sim.py 134 0 100%
src/dashboards/store.py 188 2 99% 255-256
------------------------------------------------------------------
TOTAL 2914 38 99%
Required test coverage of 94% reached. Total coverage: 98.70%
1199 passed, 12 skipped, 1 warning in 117.25s (0:01:57)
== uv run ruff check src tests agent ==
All checks passed!
== uv run pyright ==
0 errors, 0 warnings, 0 informations
== node --check src/dashboards/web/static/*.js ==
advisory: dependencies are out of date (AGENTS.md rule 8 — update on purpose):
certifi 2026.7.22 2026.10.18 wheel
fastapi 0.143.5 0.149.0 wheel
starlette 1.8.0 1.9.0 wheel
websockets 17.2 17.3 wheel
.agents/reports/14_store_performance/14_store_performance__02_no_full_scan_on_hot_paths.a1.err
View File
.agents/reports/14_store_performance/14_store_performance__02_no_full_scan_on_hot_paths.a1.md
+17
View File
@@ -0,0 +1,17 @@
**Task 02 — no full table scan on a per-request path: done, all green.**
**Implemented**
- `store.py`: `series_names()` cached 60 s (invalidated by `add`/`add_many`/`delete_series`/`maintain` via a generation counter; bounded to 256 `(source,host)` keys since `/api/metrics` takes both as free text); `stats()` counts cached 30 s with `last_maintain`/`size_bytes` deliberately never cached; `invalidate_read_caches()` for cold probes.
- `pick_res()` now one compound statement of per-`res` `COUNT`s capped by `LIMIT max_points+1` (exact: the decision only asks `0` / `≤max_points` / neither), plus new `pick_res_many()` and `history_many()` (chunked at `COMPOUND_CHUNK`, index-range seek per branch).
- `api.py`: all four history routes go through `_history_batch()` → `store.history_many()`; response dicts unchanged.
- `tests/store_perf.py`: stats/`series_names` measured **cold and cached**, batch-vs-loop measured at 27 series × 24 h/90 d, EXPLAIN of the statement that actually runs.
- Fixed a task-01 defect found in the full suite: the seeded determinism test pinned `SUM(value)`, whose float accumulation order changes when `INSERT OR REPLACE` moves rowids — now counts/windows exact, sum `rel=1e-9`, plus an exact PK-ordered row slice.
- Docs: `AGENTS.md` (store.py/api.py bullets, gap 4 split into read-half-closed, commands), `README.md` (store.py, api.py, store_perf, new test pins), `data/logs/NOTES-store-perf.md` task-02 section (before/after, rejected shapes, EXPLAIN, live curl proof).
**Measured** (`uv run pytest -m store_perf --store-perf`): HTTP `/api/health` median 1 272 ms → **1.2 ms**; `/api/metrics?source=host` 1 415 → **1.6 ms**; `/api/system/history` 3×9 at 90 d 100.3 → **49.8 ms**; `pick_res()`×3 3.0 → **0.3 ms**. Live dev server on the real 636 MB store: `/api/health` 0.700 s → **1.5 ms**, all nine sources `ok`.
**Tests/lint** (exact): `bash .agents/validate.sh` → pytest `--cov --cov-fail-under=94` **1214 passed, 12 skipped, coverage 98.73%**; `ruff` All checks passed; `pyright` 0 errors; `node --check` ok. Final confirmation run `uv run pytest -q --cov --cov-fail-under=94` → **EXIT=0**, 0 FAILED (log `data/logs/pytest-final2.log`). No dependencies touched (validate's outdated-dep line is pre-existing advisory).
**Deviations**: batched the resolution *decision*, not the row fetch — `ROW_NUMBER()` window (73 ms) and a compound of the 27 tail selects (38 ms) both measured slower than 27 index seeks (27 ms) on the seeded store, and task 01 requires the numbers not to regress; recorded with its table in the notes. The literal `SELECT res, COUNT(*) … GROUP BY res` shape was also measured slower (4.3 vs 2.7 ms) than the capped compound that shipped. No live-infra changes beyond `./scripts/dev-server.sh restart` (local dev, invariant 10).
**Next pending task**: `.agents/phases/todo/14_store_performance/03_windowed_rollups_and_wal_tuning.md`.
.agents/reports/14_store_performance/14_store_performance__02_no_full_scan_on_hot_paths.a1.md.status
+1
View File
@@ -0,0 +1 @@
ok
.agents/reports/14_store_performance/14_store_performance__02_no_full_scan_on_hot_paths.a1.validate
+64
View File
@@ -0,0 +1,64 @@
== uv run pytest --cov (hermetic: no network, live tests skip) ==
........................................................................ [ 5%]
........................................................................ [ 11%]
........................................................................ [ 17%]
........................................................................ [ 23%]
........................................................................ [ 29%]
........................................................................ [ 35%]
........................................................................ [ 41%]
........................................................................ [ 46%]
........................................................................ [ 52%]
........................................................................ [ 58%]
........................................................................ [ 64%]
.....................sssssssssss........................................ [ 70%]
........................................................................ [ 76%]
........................................................................ [ 82%]
........................................................................ [ 88%]
........................................................................ [ 93%]
..................s..................................................... [ 99%]
.. [100%]
=============================== warnings summary ===============================
.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.14.4-final-0 ________________
Name Stmts Miss Cover Missing
------------------------------------------------------------------
src/dashboards/__init__.py 3 0 100%
src/dashboards/__main__.py 14 1 93% 32
src/dashboards/alerts.py 176 0 100%
src/dashboards/api.py 356 0 100%
src/dashboards/app.py 178 0 100%
src/dashboards/auth.py 265 0 100%
src/dashboards/collectors.py 155 5 97% 141-142, 146-147, 164
src/dashboards/config.py 120 0 100%
src/dashboards/debug.py 14 0 100%
src/dashboards/poller.py 165 3 98% 192-194, 279
src/dashboards/prom.py 112 5 96% 126, 163, 177, 180, 183
src/dashboards/rates.py 73 0 100%
src/dashboards/severity.py 49 0 100%
src/dashboards/sources/__init__.py 0 0 100%
src/dashboards/sources/disks.py 130 0 100%
src/dashboards/sources/host.py 199 1 99% 147
src/dashboards/sources/llm.py 178 1 99% 285
src/dashboards/sources/nut.py 413 20 95% 153, 172, 186, 198, 204, 217, 225, 241, 319-320, 322, 579, 591, 626, 657, 661, 718-719, 724, 736
src/dashboards/sources/nut_sim.py 134 0 100%
src/dashboards/store.py 266 2 99% 445-446
------------------------------------------------------------------
TOTAL 3000 38 99%
Required test coverage of 94% reached. Total coverage: 98.73%
1214 passed, 12 skipped, 1 warning in 122.10s (0:02:02)
== uv run ruff check src tests agent ==
All checks passed!
== uv run pyright ==
0 errors, 0 warnings, 0 informations
== node --check src/dashboards/web/static/*.js ==
advisory: dependencies are out of date (AGENTS.md rule 8 — update on purpose):
certifi 2026.7.22 2026.10.18 wheel
fastapi 0.143.5 0.149.0 wheel
starlette 1.8.0 1.9.0 wheel
websockets 17.2 17.3 wheel
.agents/reports/14_store_performance/14_store_performance__03_windowed_rollups_and_wal_tuning.a1.err
View File
.agents/reports/14_store_performance/14_store_performance__03_windowed_rollups_and_wal_tuning.a1.md
+22
View File
@@ -0,0 +1,22 @@
**Phase 14 / Task 03 — windowed rollups + WAL tuning: done.**
Implementation (`src/dashboards/store.py`)
- `maintain()` now windows both rollup steps off `last_maintain` (read via `get_meta()` *outside* the lock), overlapped by one bucket + `MAINTAIN_SLACK_SECONDS=60` and **snapped down to a bucket boundary** — a floor that cut into a bucket re-rolled a partial average from its tail rows (2.5 → 4.5); missing/non-numeric row falls back to full retention with a fixed-line warning.
- Prune keeps its **full retention bound** but is now one `DELETE` per resolution: the old single `OR` of the three was unplannable (`SCAN sample`, 554 ms of a 562 ms pass while deleting nothing).
- Statistics are the other half of the cost bound: `maintain()` ends with `PRAGMA optimize`, `__init__` sets `automatic_analyze=0` (SQLite's own refresh lands inside a poller's commit). WAL: `wal_autocheckpoint=1000`, `journal_size_limit=64 MB`; `busy_timeout=5000` deliberately unchanged.
- Measured (seeded 6.4 M-row store): `maintain()` **5,368 ms → 2.7 ms** steady, 48 ms ten-minute catch-up; race at the app's cadence **100/100 inserts, 0 `database is locked`, max wait 21 ms** where the pre-fix control still waits 4.76 s; WAL 38 MB-forever → 0–4 MB. Live store: WAL 52.3 → 5.9 MB, `sqlite_stat1` created, real-store passes ~40 ms, `/api/health` 3.5 ms.
Tests / gate
- `tests/test_store.py`: +9 tests (window bounds from a trace callback, catch-up-not-a-gap, windowed-vs-full-window row-for-row equivalence, re-roll only gains rows, corrupt-row fallback, `maintain` leaves statistics, optimize failure is a log line not a failed tick, WAL pragmas, two-cycle WAL cap).
- `tests/store_perf.py`: write passes and races moved to a `scratch_copy()` (baseline never moves), added `RACE_MAINTAIN_GAP_SECONDS` cadence races + the whole-window **control**, per-window plans derived from `ROLLUP_INSERT`.
- `bash .agents/validate.sh`: **1240 passed, 12 skipped**, coverage **98.75%** (floor 94), ruff + pyright + `node --check` clean; `-m store_perf --store-perf` 1 passed; dev server restarted, all 9 sources `ok`.
Docs
- `AGENTS.md` (store.py bullet incl. the bucket-alignment rule, known gap 4 → write half closed, Commands, new testing-policy rule "a cost claim is pinned as a shape and a control"), `README.md` (store.py + `store_perf` paragraphs), `data/logs/NOTES-store-perf.md` (full task-03 tables, race-shape key, no-read-regression table, live smoke check).
Deviations
- The task's "windowing + WAL tuning" needed two extra fixes to actually reach the budget (prune `OR`, missing statistics); both are measured and recorded rather than assumed, and the prune's retention bound is unchanged.
- The saturated task-01 race is kept for comparability but the pass line moved to the cadence-shaped race (a maintainer looping with no gap starves a writer at *any* pass length); the control beside it proves the race can still see the failure.
- Left alone: `uv pip list --outdated` advisory (certifi/fastapi/starlette/websockets) — pre-existing, no dependencies touched.
Next pending task: `.agents/phases/todo/14_store_performance/04_budget_gates_and_live_proof.md`.
.agents/reports/14_store_performance/14_store_performance__03_windowed_rollups_and_wal_tuning.a1.md.status
+1
View File
@@ -0,0 +1 @@
ok
.agents/reports/14_store_performance/14_store_performance__03_windowed_rollups_and_wal_tuning.a1.validate
+111
View File
@@ -0,0 +1,111 @@
== uv run pytest --cov (hermetic: no network, live tests skip) ==
........................................................................ [ 5%]
........................................................................ [ 11%]
........................................................................ [ 17%]
........................................................................ [ 23%]
........................................................................ [ 28%]
........................................................................ [ 34%]
........................................................................ [ 40%]
........................................................................ [ 46%]
........................................................................ [ 51%]
........................................................................ [ 57%]
........................................................................ [ 63%]
.....................sssssssssss........................................ [ 69%]
........................................................................ [ 74%]
........................................................................ [ 80%]
........................................................................ [ 86%]
........................................................................ [ 92%]
..............F.............................s........................... [ 97%]
............................ [100%]
=================================== FAILURES ===================================
_______ test_a_second_maintain_reads_only_the_window_since_the_last_run ________
db = <dashboards.store.SampleStore object at 0x7fe8d2126c50>
def test_a_second_maintain_reads_only_the_window_since_the_last_run(db) -> None:
"""The cost change, pinned as a bound and as behaviour.
A bucket that filled after the previous pass is inside the next window and gets rolled. A raw
row that arrives for a bucket ten minutes behind it is not, and that bucket keeps the average
the earlier pass computed — the documented cost of a window, stated here as a passing test
rather than as a surprise. Reaching it in production costs a poller stall measured in minutes
(the poller stamps samples with its own wall clock); the alternative it replaced was
re-averaging 3.97 M rows every minute.
"""
now = time.time()
bucket_a = bucket_start(now - 10 * MINUTE)
add_series(db, "host", "h", "m", [10, 20], start=bucket_a, step=1)
db.maintain(now=now)
assert db.query("host", "h", "m", res=RES_MIN, start=bucket_a - 1,
end=bucket_a + MINUTE) == [(bucket_a, 15.0)]
second = now + 60
bucket_b = bucket_start(second - 2 * MINUTE)
add_series(db, "host", "h", "m", [40, 50], start=bucket_b, step=1)
add_series(db, "host", "h", "m", [1000], start=bucket_a + 30, step=1)
with query_log(db) as seen:
counts = db.maintain(now=second)
windows = rollup_windows(seen)
# the bound: one minute (one hour) of rows plus the overlap, not 48 h of raw (not 30 d of 1m)
assert windows[RES_RAW] == pytest.approx(
(second - MINUTE - MAINTAIN_SLACK_SECONDS, bucket_start(now - MINUTE - MAINTAIN_SLACK_SECONDS)))
> assert windows[RES_MIN] == pytest.approx(
(second - HOUR - MAINTAIN_SLACK_SECONDS, bucket_start(now - HOUR - MAINTAIN_SLACK_SECONDS)))
E assert (1792449272.5... 1792447200.0) == approx((17924....0 ± 1.8e+03))
E
E comparison failed. Mismatched elements: 1 / 2:
E Max absolute difference: 1980.0
E Max relative difference: 1.104635048664195e-06
E Index | Obtained | Expected
E 1 | 1792447200.0 | 1792449180.0 ± 1.8e+03
tests/test_store.py:802: AssertionError
=============================== warnings summary ===============================
.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.14.4-final-0 ________________
Name Stmts Miss Cover Missing
------------------------------------------------------------------
src/dashboards/__init__.py 3 0 100%
src/dashboards/__main__.py 14 1 93% 32
src/dashboards/alerts.py 176 0 100%
src/dashboards/api.py 356 0 100%
src/dashboards/app.py 178 0 100%
src/dashboards/auth.py 265 0 100%
src/dashboards/collectors.py 155 5 97% 141-142, 146-147, 164
src/dashboards/config.py 120 0 100%
src/dashboards/debug.py 14 0 100%
src/dashboards/poller.py 165 3 98% 192-194, 279
src/dashboards/prom.py 112 5 96% 126, 163, 177, 180, 183
src/dashboards/rates.py 73 0 100%
src/dashboards/severity.py 49 0 100%
src/dashboards/sources/__init__.py 0 0 100%
src/dashboards/sources/disks.py 130 0 100%
src/dashboards/sources/host.py 199 1 99% 147
src/dashboards/sources/llm.py 178 1 99% 285
src/dashboards/sources/nut.py 413 20 95% 153, 172, 186, 198, 204, 217, 225, 241, 319-320, 322, 579, 591, 626, 657, 661, 718-719, 724, 736
src/dashboards/sources/nut_sim.py 134 0 100%
src/dashboards/store.py 304 2 99% 510-511
------------------------------------------------------------------
TOTAL 3038 38 99%
Required test coverage of 94% reached. Total coverage: 98.75%
=========================== short test summary info ============================
FAILED tests/test_store.py::test_a_second_maintain_reads_only_the_window_since_the_last_run
1 failed, 1239 passed, 12 skipped, 1 warning in 111.30s (0:01:51)
== uv run ruff check src tests agent ==
All checks passed!
== uv run pyright ==
0 errors, 0 warnings, 0 informations
== node --check src/dashboards/web/static/*.js ==
advisory: dependencies are out of date (AGENTS.md rule 8 — update on purpose):
certifi 2026.7.22 2026.10.18 wheel
fastapi 0.143.5 0.150.0 wheel
starlette 1.8.0 1.9.0 wheel
websockets 17.2 17.3 wheel
validation FAILED (see output above)
.agents/reports/14_store_performance/14_store_performance__03_windowed_rollups_and_wal_tuning.a2.err
View File
.agents/reports/14_store_performance/14_store_performance__03_windowed_rollups_and_wal_tuning.a2.md
+19
View File
@@ -0,0 +1,19 @@
**Phase 14 / Task 03 — re-run after the harness failure: green.**
What was fixed
- The failing test was mine, and it was wall-clock luck, not the store: `test_a_second_maintain_reads_only_the_window_since_the_last_run` asserted the 1m→1h step's floor as a **minute**-aligned number while `_rollup_floor` snaps to an **hour**. `pytest.approx`'s relative tolerance (≈±1 800 s at epoch scale) covered the half of the hour where the two are within that, so it passed twice and failed at 19:4x.
- Fix: `WINDOW_CLOCK_OFFSETS = (0, 1, 30, 59, 1 800, 3 539, 3 599)` over a base epoch that sits exactly on both boundaries — every edge is now a case instead of a coin flip — and each bound is snapped to *its own step's* bucket; the same parametrization went on the first-pass and catch-up tests so no window assertion in this task reads `time.time()`.
- Found and fixed a second latent flake in the same test: the "due" bucket sat one step behind the ceiling, so a clock within a second of a boundary put the pair's second row outside the window and rolled 40.0 instead of 45.0 (now two steps back). `db.query()` defaults `end` to the wall clock — an explicit `end=` is now passed where the test runs on its own clock.
- Added `test_the_window_floor_is_always_aligned_to_its_own_bucket` (aligned, never above the bound, never a whole window back, and the two steps really differ). Mutation-checked: removing the `floor(…/bucket)*bucket` snap fails 6 of 7 clock cases plus that test; restored code passes.
Results
- `bash .agents/validate.sh`: **1259 passed, 12 skipped, 0 failed** · coverage **98.75%** (floor 94, unchanged) · `ruff check src tests agent` all passed · `pyright` 0 errors · `node --check` ok · hygiene ok · no `validation FAILED`.
- `uv run pytest -m store_perf --store-perf`: **1 passed** (window plans seek, catch-up ≤ 2× steady + slack, the whole-window control still contends, the cadence race is 0 `database is locked`).
- Dev server healthy: all 9 sources `ok`, real-store WAL holding at ~7.6 MB under the 64 MB cap.
Notable decisions
- No production-code change this pass — the defect was in the test's expectation, so the implementation (windowed rollups, per-resolution prune, `PRAGMA optimize` + `automatic_analyze=0`, WAL pragmas, `busy_timeout` untouched) stands as measured: `maintain()` 5 368 ms → 2.7 ms steady / 48 ms catch-up.
- Preferred explicit clocks and a property test over widening a tolerance, and recorded the flake plus both root causes in `data/logs/NOTES-store-perf.md` ("One window assertion was wall-clock luck").
- Left alone: the pre-existing `uv pip list --outdated` advisory (certifi/fastapi/starlette/websockets) — no dependencies touched in this task.
Next pending task: `.agents/phases/todo/14_store_performance/04_budget_gates_and_live_proof.md`.
.agents/reports/14_store_performance/14_store_performance__03_windowed_rollups_and_wal_tuning.a2.md.status
+1
View File
@@ -0,0 +1 @@
ok
.agents/reports/14_store_performance/14_store_performance__03_windowed_rollups_and_wal_tuning.a2.validate
+64
View File
@@ -0,0 +1,64 @@
== uv run pytest --cov (hermetic: no network, live tests skip) ==
........................................................................ [ 5%]
........................................................................ [ 11%]
........................................................................ [ 16%]
........................................................................ [ 22%]
........................................................................ [ 28%]
........................................................................ [ 33%]
........................................................................ [ 39%]
........................................................................ [ 45%]
........................................................................ [ 50%]
........................................................................ [ 56%]
........................................................................ [ 62%]
.....................sssssssssss........................................ [ 67%]
........................................................................ [ 73%]
........................................................................ [ 79%]
........................................................................ [ 84%]
........................................................................ [ 90%]
...............................................................s........ [ 96%]
............................................... [100%]
=============================== warnings summary ===============================
.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.14.4-final-0 ________________
Name Stmts Miss Cover Missing
------------------------------------------------------------------
src/dashboards/__init__.py 3 0 100%
src/dashboards/__main__.py 14 1 93% 32
src/dashboards/alerts.py 176 0 100%
src/dashboards/api.py 356 0 100%
src/dashboards/app.py 178 0 100%
src/dashboards/auth.py 265 0 100%
src/dashboards/collectors.py 155 5 97% 141-142, 146-147, 164
src/dashboards/config.py 120 0 100%
src/dashboards/debug.py 14 0 100%
src/dashboards/poller.py 165 3 98% 192-194, 279
src/dashboards/prom.py 112 5 96% 126, 163, 177, 180, 183
src/dashboards/rates.py 73 0 100%
src/dashboards/severity.py 49 0 100%
src/dashboards/sources/__init__.py 0 0 100%
src/dashboards/sources/disks.py 130 0 100%
src/dashboards/sources/host.py 199 1 99% 147
src/dashboards/sources/llm.py 178 1 99% 285
src/dashboards/sources/nut.py 413 20 95% 153, 172, 186, 198, 204, 217, 225, 241, 319-320, 322, 579, 591, 626, 657, 661, 718-719, 724, 736
src/dashboards/sources/nut_sim.py 134 0 100%
src/dashboards/store.py 304 2 99% 510-511
------------------------------------------------------------------
TOTAL 3038 38 99%
Required test coverage of 94% reached. Total coverage: 98.75%
1259 passed, 12 skipped, 1 warning in 112.04s (0:01:52)
== uv run ruff check src tests agent ==
All checks passed!
== uv run pyright ==
0 errors, 0 warnings, 0 informations
== node --check src/dashboards/web/static/*.js ==
advisory: dependencies are out of date (AGENTS.md rule 8 — update on purpose):
certifi 2026.7.22 2026.10.18 wheel
fastapi 0.143.5 0.150.0 wheel
starlette 1.8.0 1.9.0 wheel
websockets 17.2 17.3 wheel
.agents/reports/14_store_performance/14_store_performance__04_budget_gates_and_live_proof.a1.err
View File
.agents/reports/14_store_performance/14_store_performance__04_budget_gates_and_live_proof.a1.md
+14
View File
@@ -0,0 +1,14 @@
Task 04 is complete and everything is green.
**What shipped**
- `tests/store_perf.py` — `BUDGETS`: 14 pinned ceilings (2‑3× the measured "after"), `budget_rows`/``budget_violations`/`missing_budget_keys`, `measure_budget_reads/_http`, and the budget table rendered into the probe report.
- Budget tests: 2 default-run pins (a budget whose measurement was renamed fails; the table is internally coherent) + 4 `store_perf` passes (reads, HTTP, `maintain`, race). `uv run pytest -m store_perf --store-perf` → 4 passed, **14/14 budgets inside**.
- **The watch found a real defect in task 02's fix** and I fixed it: the series cache was invalidated by *every* sample write, so on a store nine pollers write to it never served and `/api/metrics` paid the full `SELECT DISTINCT` scan (measured 535 ms/call; the scan alone is 572 ms on the live store). `store.py` now invalidates only the write that can change the answer (`_note_series_write` + a capped `_known_series` existence set; `DELETE` paths still invalidate unconditionally). Pinned by statement counts, not clocks, in `tests/test_store.py` and through the served endpoint in `tests/test_api.py`.
- Live proof (`data/logs/watch-store-perf.py` → `store-perf-watch.{log,tsv,json}`, `store-perf-live.png`): 25 min / 74 passes / 296 signed-in requests on the operator's 636 MB store — **0 new `store-error` rows** (190 total, newest 2026‑10‑18 06:33), **WAL 6.5 → 7.77 MB** (cap 67 MB), 0 non‑200, medians 1.8–3.3 ms for `/api/health`, `/api/summary`, `/api/metrics` and a 3×9 90‑day `/api/system/history`.
- Docs: `data/logs/NOTES-store-perf.md` gained the task‑04 section (baseline vs after side by side, EXPLAIN shapes, the watch table, the defect write-up); `README.md` and `AGENTS.md` updated (gap 4 now reads: closed, with the honest remainder).
- Gates: `bash .agents/validate.sh` exit 0 — 1267 passed, 15 skipped, coverage 98.75%, ruff + pyright + `node --check` clean. Dev server restarted, all 9 sources `ok`.
**Honest leftovers**
1. The p95 is the cache window, not the request: on the live store a cold `series_names('host')` costs 599 ms and `stats()`' counts 729 ms, once per 60 s / 30 s (37 of 74 passes had one request >250 ms). Closing that needs a counts table or a bounded series index — task 03's note already names it, and it is a phase, not this one.
2. `uv pip list --outdated` is non-empty (fastapi 0.143.5→0.150.0, starlette, certifi, websockets) — pre-existing drift, untouched here; rule 8 wants a deliberate decision.
3. Per AGENTS rule 4 I did **not** modify `.agents/phases/todo/14_store_performance/04_...md` — say the word and I'll tick the completion criteria and move it to `complete/`.
.agents/reports/14_store_performance/14_store_performance__04_budget_gates_and_live_proof.a1.md.status
+1
View File
@@ -0,0 +1 @@
ok
.agents/reports/14_store_performance/14_store_performance__04_budget_gates_and_live_proof.a1.validate
+64
View File
@@ -0,0 +1,64 @@
== uv run pytest --cov (hermetic: no network, live tests skip) ==
........................................................................ [ 5%]
........................................................................ [ 11%]
........................................................................ [ 16%]
........................................................................ [ 22%]
.............................................s.......................... [ 28%]
........................................................................ [ 33%]
........................................................................ [ 39%]
........................................................................ [ 44%]
........................................................................ [ 50%]
........................................................................ [ 56%]
........................................................................ [ 61%]
........................sssssssssss..................................... [ 67%]
........................................................................ [ 73%]
........................................................................ [ 78%]
........................................................................ [ 84%]
........................................................................ [ 89%]
........................................................................ [ 95%]
sss....................................................... [100%]
=============================== warnings summary ===============================
.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.14.4-final-0 ________________
Name Stmts Miss Cover Missing
------------------------------------------------------------------
src/dashboards/__init__.py 3 0 100%
src/dashboards/__main__.py 14 1 93% 32
src/dashboards/alerts.py 176 0 100%
src/dashboards/api.py 356 0 100%
src/dashboards/app.py 178 0 100%
src/dashboards/auth.py 265 0 100%
src/dashboards/collectors.py 155 5 97% 141-142, 146-147, 164
src/dashboards/config.py 120 0 100%
src/dashboards/debug.py 14 0 100%
src/dashboards/poller.py 165 3 98% 192-194, 279
src/dashboards/prom.py 112 5 96% 126, 163, 177, 180, 183
src/dashboards/rates.py 73 0 100%
src/dashboards/severity.py 49 0 100%
src/dashboards/sources/__init__.py 0 0 100%
src/dashboards/sources/disks.py 130 0 100%
src/dashboards/sources/host.py 199 1 99% 147
src/dashboards/sources/llm.py 178 1 99% 285
src/dashboards/sources/nut.py 413 20 95% 153, 172, 186, 198, 204, 217, 225, 241, 319-320, 322, 579, 591, 626, 657, 661, 718-719, 724, 736
src/dashboards/sources/nut_sim.py 134 0 100%
src/dashboards/store.py 313 2 99% 553-554
------------------------------------------------------------------
TOTAL 3047 38 99%
Required test coverage of 94% reached. Total coverage: 98.75%
1267 passed, 15 skipped, 1 warning in 113.83s (0:01:53)
== uv run ruff check src tests agent ==
All checks passed!
== uv run pyright ==
0 errors, 0 warnings, 0 informations
== node --check src/dashboards/web/static/*.js ==
advisory: dependencies are out of date (AGENTS.md rule 8 — update on purpose):
certifi 2026.7.22 2026.10.19 wheel
fastapi 0.143.5 0.150.0 wheel
starlette 1.8.0 1.9.0 wheel
websockets 17.2 17.3 wheel