phase: 14_store_performance
**Phase 14 `store_performance` — final verification pass (all 4 tasks already in `complete/`)** - Re-verified every task's work in the tree: read caches (`series_names`/`stats`), capped compound `pick_res_many`, `history_many` + `api._history_batch`, windowed `maintain()` + per-resolution prune + `PRAGMA optimize`/`automatic_analyze=0`/WAL pragmas, `store_perf.BUDGETS` (14 ceilings), probe, live watch. - **Completion criteria:** baseline+after side by side in `data/logs/NOTES-store-perf.md` ✅ · budget tests green (`uv run pytest -m store_perf --store-perf` → **4 passed**, 14/14 inside) ✅ · 25-min dev-server watch re-run **independently** (74 passes, `scripts/watch-store-perf.py`): **0 `store-error`** (total still 190, newest 2026‑10‑18 06:33:30), WAL 7.77 MB ≤ 67.1 MB, medians 1.7–3.2 ms, p95 720/613 ms = the documented cold-cache window; a 3rd 27-pass watch on a freshly restarted server reproduced it (3.1/708, 1.8/572) ✅ · `maintain()` 5,368 ms → 2.7–6.8 ms and WAL bounded across two cycles (test + notes) ✅ · no behaviour change (all four history routes pinned against `query()`) ✅ - **Full gates:** `uv run pytest` → **1267 passed, 15 skipped**; `bash .agents/validate.sh` → **exit 0**, coverage **98.75%** (floor 94), `ruff check src tests agent` clean, `pyright` 0 errors, `node --check` ok, repo hygiene ok; `uv run pytest -m live --live` → **11 passed**; dev server up, 9/9 sources `ok`, pages 200 in 1.5–3.3 ms. - **Docs fixed:** README/AGENTS still claimed a flat "20 s per pass" while `host_agent.py` (in this same uncommitted change) scales the budget to 5 s × disk count — corrected in both plus the two illustrative "20 s of wedged disk" sentences; broken bold (`` `driveripper** ``) repaired; stale gap-4 title + "tuning is its own phase" wording updated to say phase 14 ran; README's absolute "nothing … is answered by a whole-table scan" qualified to *per call* with the cold-window cost named; "third opt-in marker" → second; conftest `--store-perf` help named the real marker; **`data/logs/watch-store-perf.py` moved to `scripts/watch-store-perf.py`** (both docs told a clone to run a gitignored path) with root-anchored paths, `DASH_WATCH_OUT`, and dead code removed. - **New finding recorded:** gap 9's "~10 s process-wide stall is still gap 4's / `14_store_performance`'s problem" is now closed — coincident >8 s `llm` + >12 s `ups` raw-stream gaps went 393 (00:00–12:00) and 198 (12:00–21:36) to **0** across 110+ min of the phase-14 store; written into AGENTS.md gaps 4 and 9 and the notes table. - **Deviations:** did *not* pull either deferred cost knob (`STATS_CACHE_TTL_SECONDS`, warming caches from the maintenance tick) — AGENTS.md records that decision and its reasoning; `uv pip list --outdated` still reports certifi/fastapi/starlette/websockets (pre-existing advisory, no dependency touched this phase, gate treats it as advisory). Overwrote nothing: task 04's watch preserved as `store-perf-watch.task04.*`, my two runs as `.verify25.*` / `.current-code.*`. - **Next pending phase:** `15_agent_mtls`, then `16_agent_smart_pass_budget` (owns the scaled-budget tests + its doc criterion, now partly pre-satisfied by this pass), `17_installer_device_filter`.
This commit is contained in:
@@ -0,0 +1,42 @@
|
||||
# Phase 16 — Pin the agent's scaled smart-pass budget
|
||||
|
||||
**Source:** chat 2026-10-19 — the `driveripper` 16-drive host: the flat 20 s smart-pass
|
||||
budget truncates a 16-device pass (a real read takes ~30–80 s), and because the deadline
|
||||
resets each pass, the tail drives would have stayed `smart: null` every pass. The fix —
|
||||
budget = 20 s floor scaled to 5 s × disk count, an explicit override still winning — was
|
||||
applied to the working tree on 2026-10-19 (operator-confirmed, verified on 16/16 live
|
||||
drives). This phase pins that already-applied change with tests and documentation.
|
||||
**Story:** n/a
|
||||
**Context:** `agent/host_agent.py` (`_smart_pass` — the scaled `budget = max(SMARTCTL_BUDGET_SECONDS, self.smart_timeout_seconds * len(device_names)) when the default is in force, else the explicit override; the `pass budget of {budget:.0f}s exhausted` error line; the module docstring "5 s per device and a pass budget of 20 s scaled to the disk count"), `tests/test_agent.py::test_the_pass_budget_skips_the_rest_with_one_error` (sets `collector.smart_budget_seconds = 0.0` — the override-wins branch), `AGENTS.md` line 185 ("5 s per device / 20 s per pass / 240 s cache") and the `deploy/install-agent.sh` header comment (line 25 area, "bounded 5 s per device and 20 s per pass" if still present there), `README.md` line 94 ("20 s per pass, results cached 4 minutes")
|
||||
|
||||
## Objective
|
||||
|
||||
The smart pass budget scale-up that is already in the working tree is pinned: a test proves the *default* scales (16 devices → 80 s deadline, one device → 20 s floor, the exhaustion line names the scaled figure), the override-wins branch stays covered, and every doc that still says "20 s per pass" says the floor-scaled truth. No new behaviour — the code change is already deployed on `driveripper` and verified live.
|
||||
|
||||
## Dependencies
|
||||
|
||||
- `09_disks_smartctl` (complete) — the smart pass, its bounds, and `tests/test_agent.py`'s in-process agent harness
|
||||
|
||||
## Tasks
|
||||
|
||||
1. `01_pin_the_scaled_budget.md` — the tests that the default budget scales with the disk count and that the error line names the scaled figure
|
||||
2. `02_docs_and_gate.md` — every "20 s per pass" mention in `AGENTS.md`/`README.md`/`agent/host_agent.py` says the floor-scaled truth; the gate is green
|
||||
|
||||
## Testing & Quality
|
||||
|
||||
- Unit: `tests/test_agent.py` — the scaled default (a default-attribute collector over 16 device names exhausts only when the 80 s budget runs out; over one device the floor is 20 s; the exhaustion error line carries the scaled figure, not the floor)
|
||||
- Coverage: **>90%** on the modified code (the `_smart_pass` branch)
|
||||
- The phase is "Complete" only when the **entire** project test suite passes 100% (every test in the project, not just this phase's)
|
||||
|
||||
## Documentation & Consistency
|
||||
|
||||
- The final pass reviews `README.md` and `AGENTS.md` in full against the project as it stands after this phase — fixing every inconsistency found, **including pre-existing ones not caused by this phase**.
|
||||
- Tasks update `README.md`/`AGENTS.md` immediately when their own changes are user-visible (the smart-pass bounds are a documented contract of `host_agent.py`).
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
- [ ] `uv run pytest tests/test_agent.py` green, including the new scaled-budget tests and the untouched override test
|
||||
- [ ] `grep -rn "20 s per pass" AGENTS.md README.md agent/ deploy/` returns nothing (the wording is floor-scaled everywhere)
|
||||
- [ ] `bash .agents/validate.sh` green (pytest --cov --cov-fail-under=94, ruff, pyright, node --check)
|
||||
- [ ] README.md and AGENTS.md are accurate and consistent with the whole project (verified in the final pass)
|
||||
- [ ] no behavior change in completed work: the deployed agent on `driveripper` is byte-identical to the repo's `agent/host_agent.py` (md5), the live suite is green
|
||||
@@ -0,0 +1,26 @@
|
||||
# Task 01 — Pin the scaled smart-pass budget
|
||||
|
||||
**Phase:** `16_agent_smart_pass_budget` · **Story:** n/a
|
||||
|
||||
## Objective
|
||||
|
||||
The default smart-pass budget (20 s floor, scaled to 5 s × disk count) is proven by tests: the deadline actually moves with the disk count, the exhaustion error line names the *scaled* figure, and the existing override-wins branch (a test sets `smart_budget_seconds = 0.0`) keeps passing untouched. The code change is already in `agent/host_agent.py::_smart_pass` — this task adds the tests, it does not rewrite the code.
|
||||
|
||||
## Work
|
||||
|
||||
1. `tests/test_agent.py` — add a helper that builds the in-process collector with **default** smart attributes (do not touch `smart_budget_seconds`/`smart_timeout_seconds`) and a `smartctl` fake that records each device's deadline window, in the same style as `test_the_pass_budget_skips_the_rest_with_one_error` (which stays untouched):
|
||||
- `test_the_pass_budget_scales_with_the_disk_count`: 16 device names → the last device's `smartctl` run happens with `left` (its deadline margin) consistent with a ~80 s budget (assert the deadline the code computes: `max(20, 5 × 16)`; the fake binary is instant, so assert via the recorded `min(left, 5.0)` timeout being 5.0 for a late device and the pass *not* truncating — every device got a `smartctl` run); 1 device name → budget 20 s (assert the recorded timeout is 5.0 and, with the fake made slow enough to exhaust a 20 s but not an 80 s budget in a *separate* case if feasible — if the harness cannot fake wall time, assert the computed `budget` expression via the error-line route instead: force exhaustion with a `time.monotonic` monkeypatch offset).
|
||||
- `test_the_pass_budget_error_line_names_the_scaled_figure`: force exhaustion on 16 devices (monkeypatch the clock forward past the 80 s deadline between two device runs, as the existing test's `0.0` override does for its 0 s budget) → the error line reads `smart: pass budget of 80s exhausted, N disk(s) skipped` — the scaled figure, not the 20 s floor.
|
||||
2. `agent/host_agent.py` — **no code change expected.** If a test exposes a real bug in the already-applied change (e.g. the `== SMARTCTL_BUDGET_SECONDS` default check), fix it minimally and record the diff in the task's completion note; do not redesign.
|
||||
|
||||
## Testing & Quality
|
||||
|
||||
- Unit: the two new tests above + the untouched `test_the_pass_budget_skips_the_rest_with_one_error` (override wins: `0.0` still exhausts immediately on 3 devices with one error line)
|
||||
- Coverage: **>90%** on `agent/host_agent.py::_smart_pass`'s budget branch (the default-scaled path must now be exercised, not just the override)
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
- [ ] `uv run pytest tests/test_agent.py -q` green: the two new tests, the untouched override test, and every pre-existing agent test
|
||||
- [ ] the 80 s case asserts the *scaled* figure in the error line (a test that passed with a 20 s budget would fail on the scaled code — no tautological assertions)
|
||||
- [ ] `uv run ruff check agent tests` and `uv run pyright` clean
|
||||
- [ ] no behavior change in completed work: `agent/host_agent.py` unchanged unless a real bug was found (then the fix is minimal and recorded)
|
||||
@@ -0,0 +1,25 @@
|
||||
# Task 02 — Docs and gate for the scaled budget
|
||||
|
||||
**Phase:** `16_agent_smart_pass_budget` · **Story:** n/a
|
||||
|
||||
## Objective
|
||||
|
||||
Every place that still says the smart pass is "20 s per pass" says the floor-scaled truth (20 s floor, 5 s × disk count), and the phase gate is green. The agent's module docstring was already updated on 2026-10-19 to the scaled wording — this task verifies it and fixes the two remaining doc lines.
|
||||
|
||||
## Work
|
||||
|
||||
1. `AGENTS.md` line ~185 (the `host_agent.py` bullet): `5 s per device / 20 s per pass / 240 s cache` → `5 s per device / a pass budget of 20 s scaled to 5 s × disk count / 240 s cache` (keep the bullet's voice; one line).
|
||||
2. `README.md` line ~94: `20 s per pass, results cached 4 minutes` → `a pass budget of 20 s scaled to the disk count (5 s each), results cached 4 minutes` (one line).
|
||||
3. `agent/host_agent.py` module docstring — verify it reads `bounded 5 s per device and a pass budget of 20 s scaled to the disk count, cached 240 s`; fix only if it drifted.
|
||||
4. `bash .agents/validate.sh` — the four checks (pytest --cov --cov-fail-under=94, ruff, pyright, node --check) + repo hygiene, all `uv run`.
|
||||
|
||||
## Testing & Quality
|
||||
|
||||
- No new logic in this task; the gate run is the check.
|
||||
- Coverage: **>90%** gate holds (task 01's tests carry the new branch).
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
- [ ] `grep -rn "20 s per pass" AGENTS.md README.md agent/ deploy/` returns nothing
|
||||
- [ ] `bash .agents/validate.sh` green
|
||||
- [ ] the two edited doc lines read the floor-scaled truth and nothing else changed in either file
|
||||
@@ -0,0 +1,59 @@
|
||||
# Phase 17 — The installer's broken device filter and the kernel-gate variant
|
||||
|
||||
**Source:** chat 2026-10-19 — the `driveripper` install: the template's
|
||||
`DevicePolicy=strict` + `DeviceAllow` lines produced a **deny-all device filter** on
|
||||
kernel 7.2 (systemd resolves `DeviceAllow` globs by name against `/proc/devices`, where
|
||||
this kernel lists NVMe as `blkext` (major 259), so `nvme*` matched nothing and the whole
|
||||
device namespace was locked). The operator removed the lines by hand; the unit now also
|
||||
runs `User=root` + `CapabilityBoundingSet=CAP_SYS_RAWIO CAP_SYS_ADMIN` (kernel 7.2's
|
||||
unprivileged-passthrough gate — `data/logs/NOTES-driveripper-agent.md`). The template
|
||||
still ships the broken lines, `User=$AGENT_USER`, and an empty bounding set, and re-running
|
||||
the installer on `driveripper` would **clobber the working unit** (unconditional
|
||||
`cat > "$UNIT"`). This phase makes the template the single source of the working unit.
|
||||
**Story:** n/a
|
||||
**Context:** `deploy/install-agent.sh` (`do_install` — the unconditional unit heredoc with `DevicePolicy=strict` + nine `DeviceAllow` lines, the `PrivateDevices=false` rationale comment block, `User=$AGENT_USER`, `CapabilityBoundingSet=`, the `--bind/--port/--token/--rotate-token/--status/--uninstall` flag set), `tests/test_deploy_scripts.py` (the fake-root install sandbox `test_install_path_installs_into_its_own_paths_and_prints_the_token_once`, the token-file permission test, the `bash -n` + function-naming pins), `AGENTS.md` (the fleet bullet "The deployed unit has since diverged from the template in the three ways the SMART block below records", gap 10's "the template change is its own phase"), `README.md` (the install section, line ~46 `sudo deploy/install-agent.sh --bind <addr> --port 9120`)
|
||||
|
||||
## Objective
|
||||
|
||||
Re-running `deploy/install-agent.sh` on any host — including `driveripper`, with
|
||||
`--as-root --caps CAP_SYS_RAWIO,CAP_SYS_ADMIN --force` — produces the working unit: no
|
||||
`DevicePolicy`/`DeviceAllow` lines (device access falls back to DAC via the `disk` group
|
||||
+ SELinux, the standard SMART-monitoring arrangement), the kernel-gate variant expressible
|
||||
by flags, and a re-run that would change an existing unit prints the diff and refuses
|
||||
without `--force`. The default install (dedicated user, empty bounding set) is otherwise
|
||||
byte-identical to what the template wrote before this phase.
|
||||
|
||||
## Dependencies
|
||||
|
||||
- `09_disks_smartctl` (complete) — the smart pass the unit exists to serve
|
||||
- `15_agent_mtls` (todo) — **shared-file, run-order note**: 15's task 06 adds `--cert-bundle`/`RuntimeDirectory` to the same `do_install`; apply on top of the file state at the time it runs, never restore a line the other phase owns. If 15 lands first, its new lines survive this phase's filter-line removal untouched.
|
||||
|
||||
## Tasks
|
||||
|
||||
1. `01_drop_the_broken_device_filter.md` — remove `DevicePolicy=strict` + all nine `DeviceAllow` lines and rewrite the rationale comment block
|
||||
2. `02_root_and_caps_flags.md` — `--as-root` and `--caps CAP1,CAP2` write the kernel-gate unit lines
|
||||
3. `03_no_silent_clobber.md` — a re-run that would change an existing unit prints the diff and refuses without `--force`
|
||||
4. `04_tests_docs_gate.md` — the unit pins in `tests/test_deploy_scripts.py`, `AGENTS.md`/`README.md` updated, gate green
|
||||
|
||||
## Testing & Quality
|
||||
|
||||
- Integration: `tests/test_deploy_scripts.py` — the template unit carries **no** `DevicePolicy`/`DeviceAllow` line; a default-install sandbox unit is byte-identical to the pre-phase template unit minus the filter lines; `--as-root --caps CAP_SYS_RAWIO,CAP_SYS_ADMIN` writes `User=root` and `CapabilityBoundingSet=CAP_SYS_RAWIO CAP_SYS_ADMIN` exactly; the re-run guard refuses (exit 1, diff on stdout) and overwrites with `--force`; `bash -n` and the existing function-naming pin still pass
|
||||
- Live (task 04, operator-verified): `driveripper`'s deployed unit is reproduced by `sudo deploy/install-agent.sh --bind 10.1.0.126 --port 9120 --as-root --caps CAP_SYS_RAWIO,CAP_SYS_ADMIN --force` — `systemctl diff` empty, `CapEff` still `0000000000220000`, 16/16 drives still live; `deskwork`/`gateway` untouched (their units are not re-run)
|
||||
- Coverage: the script itself is pinned by text assertions + `bash -n` (house pattern; >90% applies to the new Python test code)
|
||||
- The phase is "Complete" only when the **entire** project test suite passes 100% (every test in the project, not just this phase's)
|
||||
|
||||
## Documentation & Consistency
|
||||
|
||||
- The final pass reviews `README.md` and `AGENTS.md` in full against the project as it stands after this phase — fixing every inconsistency found, **including pre-existing ones not caused by this phase**.
|
||||
- Tasks update `README.md`/`AGENTS.md` immediately when their own changes are user-visible (the new flags are operator-facing; gap 10's "the template change is its own phase" becomes "done in phase 17" with the `driveripper` flag line).
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
- [ ] `grep -n "DevicePolicy\|DeviceAllow" deploy/install-agent.sh` returns nothing (the `PrivateDevices=false` line stays)
|
||||
- [ ] `sudo deploy/install-agent.sh --help` names `--as-root` and `--caps` with the kernel-gate sentence
|
||||
- [ ] a fresh sandbox install (fake root, no flags) produces a unit byte-identical to the pre-phase template unit minus the filter lines (pinned in `tests/test_deploy_scripts.py`)
|
||||
- [ ] a re-run that would change an existing unit exits 1 with the diff on stdout; with `--force` it overwrites
|
||||
- [ ] operator-verified on `driveripper`: the flags reproduce the working unit (`systemctl diff` empty, 16/16 drives live after restart)
|
||||
- [ ] `uv run pytest` green, `bash .agents/validate.sh` green
|
||||
- [ ] README.md and AGENTS.md are accurate and consistent with the whole project (verified in the final pass)
|
||||
- [ ] no behavior change in completed work: the default install's unit (minus the filter lines) and every existing flag behave as before; the live suite is green
|
||||
@@ -0,0 +1,25 @@
|
||||
# Task 01 — Drop the broken device filter
|
||||
|
||||
**Phase:** `17_installer_device_filter` · **Story:** n/a
|
||||
|
||||
## Objective
|
||||
|
||||
The template unit no longer carries `DevicePolicy=strict` or any `DeviceAllow` line, and the comment block that justified them is rewritten to say why the *absence* is the fix. The `PrivateDevices=false` line stays (explicit, default behaviour, and the rest of the lockdown is unchanged).
|
||||
|
||||
## Work
|
||||
|
||||
1. `deploy/install-agent.sh` — in the `do_install` unit heredoc: delete the `DevicePolicy=strict` line and all nine `DeviceAllow=…` lines (`/dev/null`, `/dev/zero`, `/dev/full`, `/dev/random`, `/dev/urandom`, `/dev/nvme*`, `/dev/sd*`, `/dev/vd*`, `/dev/sg*`).
|
||||
2. `deploy/install-agent.sh` — rewrite the comment block above `PrivateDevices=false` (the "The pseudo nodes are not decoration…" paragraph) to the new truth: **no device filter on purpose** — systemd resolves `DeviceAllow` globs by name against `/proc/devices` (fnmatch in `bpf_devices_allow_list_major`), where kernel 7.2 lists NVMe as `blkext` (major 259), so `nvme*` matched nothing and `DevicePolicy=strict` denied the *whole* device namespace (the `driveripper` install went deny-all, root included; `data/logs/NOTES-driveripper-agent.md`). Without the policy, device access falls back to DAC (the `disk` group the script already grants) + SELinux — the standard SMART-monitoring arrangement. Keep the one operational fact that survives: `PrivateDevices=false` is the default (a `true` sandbox with no `DeviceAllow` list is what dies at spawn with `status=208/STDIN` — keep that warning sentence, it is still why the line is written explicitly).
|
||||
3. `deploy/install-agent.sh` — the script's top-of-file header comment (line ~25: "the unit device-allows the disk nodes") → "the unit applies no device filter (kernel 7.2's `/proc/devices` name matching made the old `nvme*` glob deny-all); device access is DAC via the `disk` group + SELinux".
|
||||
|
||||
## Testing & Quality
|
||||
|
||||
- Integration: `bash -n deploy/install-agent.sh` clean (the existing pin covers it); a quick sandbox run (the `test_install_path_installs_into_its_own_paths_and_prints_the_token_once` pattern, run manually) shows no `DevicePolicy`/`DeviceAllow` in the written unit — the formal pins are task 04, this task keeps the change small and reviewable.
|
||||
- Coverage: **>90%** n/a (shell script; the pins are text assertions per the house pattern).
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
- [ ] `grep -n "DevicePolicy\|DeviceAllow" deploy/install-agent.sh` returns nothing
|
||||
- [ ] `bash -n deploy/install-agent.sh` clean
|
||||
- [ ] a manual sandbox install (fake root, `--bind 127.0.0.1 --port 9199`) writes a unit with the filter lines gone, `PrivateDevices=false` present, and every other line byte-identical to the pre-phase template unit
|
||||
- [ ] the rewritten comment names the `/proc/devices`/`blkext` mechanism in one sentence (an executor reading only the script can understand *why* there is no filter)
|
||||
@@ -0,0 +1,27 @@
|
||||
# Task 02 — `--as-root` and `--caps` flags
|
||||
|
||||
**Phase:** `17_installer_device_filter` · **Story:** n/a
|
||||
|
||||
## Objective
|
||||
|
||||
The kernel-gate variant of the unit (what `driveripper` runs) is expressible from the template: `--as-root` writes `User=root` instead of `User=$AGENT_USER`, and `--caps CAP1,CAP2` (comma-separated, e.g. `--caps CAP_SYS_RAWIO,CAP_SYS_ADMIN`) writes `CapabilityBoundingSet=CAP1 CAP2` instead of the empty `CapabilityBoundingSet=` line. Both default off; a flag without its friend is refused with a message.
|
||||
|
||||
## Work
|
||||
|
||||
1. `deploy/install-agent.sh` — flag parsing: add `--as-root` (sets `AS_ROOT=1`) and `--caps <list>` (sets `CAPS="<list>"`, validates non-empty, rejects an empty list with exit 1) in the existing `case`/flag block, and `--help` text:
|
||||
- `--as-root run the service as root (for a kernel-gated host: kernel 7.2 gates unprivileged smartctl passthrough behind capabilities, not uid — a root process under the default empty bounding set is capability-less, so this flag needs --caps to mean anything)`
|
||||
- `--caps LIST CapabilityBoundingSet line for the unit (comma-separated, e.g. CAP_SYS_RAWIO,CAP_SYS_ADMIN); default: empty (capability-less service)`
|
||||
2. `deploy/install-agent.sh` — validation: `--as-root` without `--caps` → print "note: --as-root with the default empty CapabilityBoundingSet leaves the process capability-less (the kernel gates check capabilities, not uid); pair it with --caps for a kernel-gated host" and **still proceed** (a capability-less root is a legitimate, if unusual, choice — do not hard-refuse; the note is the operator's reminder). `--caps` without `--as-root` → refuse, exit 1 ("--caps needs --as-root: a capability grant on the service user is not the kernel-gate fix").
|
||||
3. `deploy/install-agent.sh` — the unit heredoc: `User=$AGENT_USER` → `User=$( [ "$AS_ROOT" -eq 1 ] && echo root || echo "$AGENT_USER" )`; the empty `CapabilityBoundingSet=` line → the `CAPS` list converted comma→space when set. Add one comment line above: `# --caps: kernel 7.2 gates unprivileged smartctl passthrough (NVMe admin log: CAP_SYS_ADMIN; SG_IO allow-list: CAP_SYS_RAWIO); a root process needs them named, because an empty bounding set bounds even root to CapEff 0`.
|
||||
4. `deploy/install-agent.sh` — the `--status` probe is unchanged (it probes the agent over HTTP; the user the unit runs as is invisible to it). The token file stays `0640 root:$AGENT_USER` — with `--as-root` the group read is irrelevant (root reads it), so do **not** change the token-file logic.
|
||||
|
||||
## Testing & Quality
|
||||
|
||||
- Integration: manual sandbox runs (fake root): default → `User=dashboards-agent` + empty `CapabilityBoundingSet=`; `--as-root --caps CAP_SYS_RAWIO,CAP_SYS_ADMIN` → `User=root` + `CapabilityBoundingSet=CAP_SYS_RAWIO CAP_SYS_ADMIN`; `--caps` alone → exit 1 with the message; `--as-root` alone → proceeds with the note on stdout. (Formal pins: task 04.)
|
||||
- Coverage: **>90%** n/a (shell script; text pins per the house pattern).
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
- [ ] `sudo deploy/install-agent.sh --help` names both flags with the kernel-gate sentence (a fresh clone's executor can learn the `driveripper` recipe from `--help` alone)
|
||||
- [ ] the two sandbox units match the exact lines above (byte-for-byte on the `User=` and `CapabilityBoundingSet=` lines)
|
||||
- [ ] `bash -n deploy/install-agent.sh` clean; every pre-existing flag (`--bind/--port/--token/--rotate-token/--status/--uninstall`) behaves as before in a sandbox run
|
||||
@@ -0,0 +1,27 @@
|
||||
# Task 03 — No silent clobber on re-run
|
||||
|
||||
**Phase:** `17_installer_device_filter` · **Story:** n/a
|
||||
|
||||
## Objective
|
||||
|
||||
A re-run that would change an already-installed unit stops: it prints the diff between the installed unit and the unit the script would write, and exits 1 without touching the file. `--force` proceeds with the overwrite. A re-run whose generated unit is identical to the installed one proceeds silently (idempotent re-runs stay the zero-noise path the script has always had).
|
||||
|
||||
## Work
|
||||
|
||||
1. `deploy/install-agent.sh` — add a `--force` flag to the flag block (`FORCE=1`) and `--help` text: `--force overwrite an existing unit even if the generated unit differs (prints the diff first)`.
|
||||
2. `deploy/install-agent.sh` — in `do_install`, immediately before the `cat > "$UNIT"` heredoc write: build the would-be unit into a temp file (the heredoc already exists — render it once to `$(mktemp)`), and if `"$UNIT"` exists and `! cmp -s "$UNIT" "$tmp_unit"`:
|
||||
- if `FORCE=1`: print the diff (`diff -u "$UNIT" "$tmp_unit"` on stdout) and proceed with the overwrite;
|
||||
- else: print `refusing to overwrite $UNIT (generated unit differs); run with --force to apply:` + the same diff, and `exit 1` **before** the token-file write, the `systemctl daemon-reload`, and the `systemctl enable --now` — a refused run must leave the host exactly as it was (token file, unit, and running service all untouched; note the token-reuse logic runs earlier in `do_install` and only *reads* the existing token, so ordering the guard before the unit write is enough).
|
||||
- identical unit → no output, normal proceed.
|
||||
3. `deploy/install-agent.sh` — `--uninstall` is unchanged (it already removes the unit deliberately; the guard applies to `do_install` only).
|
||||
|
||||
## Testing & Quality
|
||||
|
||||
- Integration: manual sandbox sequence (fake root): install → modify the installed unit by one line → re-run with the same flags → exit 1, diff on stdout, installed unit byte-identical to the modified version, service still the old one; re-run with `--force` → diff printed, unit overwritten; re-run again unchanged → silent, exit 0. (Formal pins: task 04.)
|
||||
- Coverage: **>90%** n/a (shell script; text/behaviour pins per the house pattern).
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
- [ ] the three-way sequence above (refuse → force → silent) holds in the sandbox, with the token file and the running service untouched by a refused run
|
||||
- [ ] `--force` prints the diff *before* overwriting (an operator sees exactly what changed)
|
||||
- [ ] `bash -n deploy/install-agent.sh` clean; a fresh install (no existing unit) is unaffected by the guard
|
||||
@@ -0,0 +1,34 @@
|
||||
# Task 04 — Pins, docs, gate
|
||||
|
||||
**Phase:** `17_installer_device_filter` · **Story:** n/a
|
||||
|
||||
## Objective
|
||||
|
||||
The template's new shape is pinned in `tests/test_deploy_scripts.py` (no filter lines, the flags' exact lines, the clobber guard's refuse/force/silent sequence), `AGENTS.md` and `README.md` name the `driveripper` recipe, and the gate is green. The operator-verified `driveripper` reproduction (the completion criterion in `00_phase.md`) is the last check.
|
||||
|
||||
## Work
|
||||
|
||||
1. `tests/test_deploy_scripts.py` — extend the existing sandbox pattern:
|
||||
- `test_the_template_unit_has_no_device_filter_lines` — read `deploy/install-agent.sh`'s unit heredoc region and assert no line starts with `DevicePolicy=` or `DeviceAllow=`, and `PrivateDevices=false` is still present (the filter's absence is the fix; a regression that re-adds `DeviceAllow=/dev/nvme*` is the deny-all returning).
|
||||
- `test_a_default_install_unit_is_the_pre_phase_unit_minus_the_filter_lines` — the sandbox-installed unit equals the known pre-phase template unit text minus the ten filter lines (pin the whole unit, so a future template line drift fails here).
|
||||
- `test_as_root_and_caps_write_the_kernel_gate_unit_lines` — `--as-root --caps CAP_SYS_RAWIO,CAP_SYS_ADMIN` → `User=root` and `CapabilityBoundingSet=CAP_SYS_RAWIO CAP_SYS_ADMIN` exactly; `--caps` alone → exit 1 with the `--as-root` message; `--as-root` alone → proceeds with the capability-less note on stdout.
|
||||
- `test_a_re_run_that_differs_refuses_without_force_and_applies_with_it` — the refuse/force/silent sequence (exit 1 + diff on stdout + installed unit untouched; `--force` overwrites with the diff printed; an unchanged re-run is silent).
|
||||
2. `AGENTS.md` — the fleet bullet's "re-running `deploy/install-agent.sh` on driveripper would clobber the working unit, because it rewrites the unit unconditionally (gap 10)" → the post-phase truth: the installer no longer writes the device-filter lines, and the `driveripper` unit is reproducible with `sudo deploy/install-agent.sh --bind 10.1.0.126 --port 9120 --as-root --caps CAP_SYS_RAWIO,CAP_SYS_ADMIN --force` (a re-run that would change an existing unit refuses without `--force`). Update gap 10's "the template change is its own phase" → "done in phase 17" with the flag line.
|
||||
3. `README.md` — the install section: one line naming the flags' purpose (the kernel-gated-host recipe), keeping the how-to voice.
|
||||
4. `bash .agents/validate.sh` — all four checks green.
|
||||
5. **Operator-verified (live, read-only + one restart):** on `driveripper`, `sudo deploy/install-agent.sh --bind 10.1.0.126 --port 9120 --as-root --caps CAP_SYS_RAWIO,CAP_SYS_ADMIN --force` → `systemctl diff dashboards-agent` empty, `systemctl is-active` `active`, `CapEff` still `0000000000220000`, and 16/16 drives still live in the next `/snapshot` (the restart is the only fleet action; `deskwork`/`gateway` are not re-run). Screenshot `data/logs/disks-17-installer-repro.png` per invariant 10.
|
||||
|
||||
## Testing & Quality
|
||||
|
||||
- Integration: the four new pins above + the pre-existing pins (sudoers, `bash -n`, function naming, token-file permissions) all green.
|
||||
- Coverage: **>90%** on the new test code; the 94% gate holds.
|
||||
- Live: the operator-verified reproduction (item 5) — the phase's `driveripper` criterion.
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
- [ ] the four new pins pass and every pre-existing `test_deploy_scripts.py` pin passes
|
||||
- [ ] `grep -n "DevicePolicy\|DeviceAllow" deploy/install-agent.sh` returns nothing
|
||||
- [ ] `AGENTS.md` fleet bullet + gap 10 and `README.md` name the `driveripper` flag recipe
|
||||
- [ ] the operator-verified reproduction: `systemctl diff` empty, 16/16 drives live, screenshot in `data/logs/`
|
||||
- [ ] `uv run pytest` green, `bash .agents/validate.sh` green
|
||||
- [ ] the live suite (`set -a; . ./.env.dev; set +a; uv run pytest -m live --live`) is green
|
||||
@@ -0,0 +1,9 @@
|
||||
**Phase 14 `store_performance` — final verification pass (all 4 tasks already in `complete/`)**
|
||||
|
||||
- Re-verified every task's work in the tree: read caches (`series_names`/`stats`), capped compound `pick_res_many`, `history_many` + `api._history_batch`, windowed `maintain()` + per-resolution prune + `PRAGMA optimize`/`automatic_analyze=0`/WAL pragmas, `store_perf.BUDGETS` (14 ceilings), probe, live watch.
|
||||
- **Completion criteria:** baseline+after side by side in `data/logs/NOTES-store-perf.md` ✅ · budget tests green (`uv run pytest -m store_perf --store-perf` → **4 passed**, 14/14 inside) ✅ · 25-min dev-server watch re-run **independently** (74 passes, `scripts/watch-store-perf.py`): **0 `store-error`** (total still 190, newest 2026‑10‑18 06:33:30), WAL 7.77 MB ≤ 67.1 MB, medians 1.7–3.2 ms, p95 720/613 ms = the documented cold-cache window; a 3rd 27-pass watch on a freshly restarted server reproduced it (3.1/708, 1.8/572) ✅ · `maintain()` 5,368 ms → 2.7–6.8 ms and WAL bounded across two cycles (test + notes) ✅ · no behaviour change (all four history routes pinned against `query()`) ✅
|
||||
- **Full gates:** `uv run pytest` → **1267 passed, 15 skipped**; `bash .agents/validate.sh` → **exit 0**, coverage **98.75%** (floor 94), `ruff check src tests agent` clean, `pyright` 0 errors, `node --check` ok, repo hygiene ok; `uv run pytest -m live --live` → **11 passed**; dev server up, 9/9 sources `ok`, pages 200 in 1.5–3.3 ms.
|
||||
- **Docs fixed:** README/AGENTS still claimed a flat "20 s per pass" while `host_agent.py` (in this same uncommitted change) scales the budget to 5 s × disk count — corrected in both plus the two illustrative "20 s of wedged disk" sentences; broken bold (`` `driveripper** ``) repaired; stale gap-4 title + "tuning is its own phase" wording updated to say phase 14 ran; README's absolute "nothing … is answered by a whole-table scan" qualified to *per call* with the cold-window cost named; "third opt-in marker" → second; conftest `--store-perf` help named the real marker; **`data/logs/watch-store-perf.py` moved to `scripts/watch-store-perf.py`** (both docs told a clone to run a gitignored path) with root-anchored paths, `DASH_WATCH_OUT`, and dead code removed.
|
||||
- **New finding recorded:** gap 9's "~10 s process-wide stall is still gap 4's / `14_store_performance`'s problem" is now closed — coincident >8 s `llm` + >12 s `ups` raw-stream gaps went 393 (00:00–12:00) and 198 (12:00–21:36) to **0** across 110+ min of the phase-14 store; written into AGENTS.md gaps 4 and 9 and the notes table.
|
||||
- **Deviations:** did *not* pull either deferred cost knob (`STATS_CACHE_TTL_SECONDS`, warming caches from the maintenance tick) — AGENTS.md records that decision and its reasoning; `uv pip list --outdated` still reports certifi/fastapi/starlette/websockets (pre-existing advisory, no dependency touched this phase, gate treats it as advisory). Overwrote nothing: task 04's watch preserved as `store-perf-watch.task04.*`, my two runs as `.verify25.*` / `.current-code.*`.
|
||||
- **Next pending phase:** `15_agent_mtls`, then `16_agent_smart_pass_budget` (owns the scaled-budget tests + its doc criterion, now partly pre-satisfied by this pass), `17_installer_device_filter`.
|
||||
@@ -0,0 +1 @@
|
||||
ok
|
||||
@@ -0,0 +1,64 @@
|
||||
== uv run pytest --cov (hermetic: no network, live tests skip) ==
|
||||
........................................................................ [ 5%]
|
||||
........................................................................ [ 11%]
|
||||
........................................................................ [ 16%]
|
||||
........................................................................ [ 22%]
|
||||
.............................................s.......................... [ 28%]
|
||||
........................................................................ [ 33%]
|
||||
........................................................................ [ 39%]
|
||||
........................................................................ [ 44%]
|
||||
........................................................................ [ 50%]
|
||||
........................................................................ [ 56%]
|
||||
........................................................................ [ 61%]
|
||||
........................sssssssssss..................................... [ 67%]
|
||||
........................................................................ [ 73%]
|
||||
........................................................................ [ 78%]
|
||||
........................................................................ [ 84%]
|
||||
........................................................................ [ 89%]
|
||||
........................................................................ [ 95%]
|
||||
sss....................................................... [100%]
|
||||
=============================== warnings summary ===============================
|
||||
.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
|
||||
/var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
================================ tests coverage ================================
|
||||
_______________ coverage: platform linux, python 3.14.4-final-0 ________________
|
||||
|
||||
Name Stmts Miss Cover Missing
|
||||
------------------------------------------------------------------
|
||||
src/dashboards/__init__.py 3 0 100%
|
||||
src/dashboards/__main__.py 14 1 93% 32
|
||||
src/dashboards/alerts.py 176 0 100%
|
||||
src/dashboards/api.py 356 0 100%
|
||||
src/dashboards/app.py 178 0 100%
|
||||
src/dashboards/auth.py 265 0 100%
|
||||
src/dashboards/collectors.py 155 5 97% 141-142, 146-147, 164
|
||||
src/dashboards/config.py 120 0 100%
|
||||
src/dashboards/debug.py 14 0 100%
|
||||
src/dashboards/poller.py 165 3 98% 192-194, 279
|
||||
src/dashboards/prom.py 112 5 96% 126, 163, 177, 180, 183
|
||||
src/dashboards/rates.py 73 0 100%
|
||||
src/dashboards/severity.py 49 0 100%
|
||||
src/dashboards/sources/__init__.py 0 0 100%
|
||||
src/dashboards/sources/disks.py 130 0 100%
|
||||
src/dashboards/sources/host.py 199 1 99% 147
|
||||
src/dashboards/sources/llm.py 178 1 99% 285
|
||||
src/dashboards/sources/nut.py 413 20 95% 153, 172, 186, 198, 204, 217, 225, 241, 319-320, 322, 579, 591, 626, 657, 661, 718-719, 724, 736
|
||||
src/dashboards/sources/nut_sim.py 134 0 100%
|
||||
src/dashboards/store.py 313 2 99% 553-554
|
||||
------------------------------------------------------------------
|
||||
TOTAL 3047 38 99%
|
||||
Required test coverage of 94% reached. Total coverage: 98.75%
|
||||
1267 passed, 15 skipped, 1 warning in 110.62s (0:01:50)
|
||||
== uv run ruff check src tests agent ==
|
||||
All checks passed!
|
||||
== uv run pyright ==
|
||||
0 errors, 0 warnings, 0 informations
|
||||
== node --check src/dashboards/web/static/*.js ==
|
||||
advisory: dependencies are out of date (AGENTS.md rule 8 — update on purpose):
|
||||
certifi 2026.7.22 2026.10.19 wheel
|
||||
fastapi 0.143.5 0.150.0 wheel
|
||||
starlette 1.8.0 1.9.0 wheel
|
||||
websockets 17.2 17.3 wheel
|
||||
+1
@@ -0,0 +1 @@
|
||||
Now let me write the baseline NOTES document.
|
||||
+1
@@ -0,0 +1 @@
|
||||
ok
|
||||
+63
@@ -0,0 +1,63 @@
|
||||
== uv run pytest --cov (hermetic: no network, live tests skip) ==
|
||||
........................................................................ [ 5%]
|
||||
........................................................................ [ 11%]
|
||||
........................................................................ [ 17%]
|
||||
........................................................................ [ 23%]
|
||||
........................................................................ [ 29%]
|
||||
........................................................................ [ 35%]
|
||||
........................................................................ [ 41%]
|
||||
........................................................................ [ 47%]
|
||||
........................................................................ [ 53%]
|
||||
........................................................................ [ 59%]
|
||||
........................................................................ [ 65%]
|
||||
....................sssssssssss......................................... [ 71%]
|
||||
........................................................................ [ 77%]
|
||||
........................................................................ [ 83%]
|
||||
........................................................................ [ 89%]
|
||||
........................................................................ [ 95%]
|
||||
...s....................................................... [100%]
|
||||
=============================== warnings summary ===============================
|
||||
.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
|
||||
/var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
================================ tests coverage ================================
|
||||
_______________ coverage: platform linux, python 3.14.4-final-0 ________________
|
||||
|
||||
Name Stmts Miss Cover Missing
|
||||
------------------------------------------------------------------
|
||||
src/dashboards/__init__.py 3 0 100%
|
||||
src/dashboards/__main__.py 14 1 93% 32
|
||||
src/dashboards/alerts.py 176 0 100%
|
||||
src/dashboards/api.py 348 0 100%
|
||||
src/dashboards/app.py 178 0 100%
|
||||
src/dashboards/auth.py 265 0 100%
|
||||
src/dashboards/collectors.py 155 5 97% 141-142, 146-147, 164
|
||||
src/dashboards/config.py 120 0 100%
|
||||
src/dashboards/debug.py 14 0 100%
|
||||
src/dashboards/poller.py 165 3 98% 192-194, 279
|
||||
src/dashboards/prom.py 112 5 96% 126, 163, 177, 180, 183
|
||||
src/dashboards/rates.py 73 0 100%
|
||||
src/dashboards/severity.py 49 0 100%
|
||||
src/dashboards/sources/__init__.py 0 0 100%
|
||||
src/dashboards/sources/disks.py 130 0 100%
|
||||
src/dashboards/sources/host.py 199 1 99% 147
|
||||
src/dashboards/sources/llm.py 178 1 99% 285
|
||||
src/dashboards/sources/nut.py 413 20 95% 153, 172, 186, 198, 204, 217, 225, 241, 319-320, 322, 579, 591, 626, 657, 661, 718-719, 724, 736
|
||||
src/dashboards/sources/nut_sim.py 134 0 100%
|
||||
src/dashboards/store.py 188 2 99% 255-256
|
||||
------------------------------------------------------------------
|
||||
TOTAL 2914 38 99%
|
||||
Required test coverage of 94% reached. Total coverage: 98.70%
|
||||
1199 passed, 12 skipped, 1 warning in 117.25s (0:01:57)
|
||||
== uv run ruff check src tests agent ==
|
||||
All checks passed!
|
||||
== uv run pyright ==
|
||||
0 errors, 0 warnings, 0 informations
|
||||
== node --check src/dashboards/web/static/*.js ==
|
||||
advisory: dependencies are out of date (AGENTS.md rule 8 — update on purpose):
|
||||
certifi 2026.7.22 2026.10.18 wheel
|
||||
fastapi 0.143.5 0.149.0 wheel
|
||||
starlette 1.8.0 1.9.0 wheel
|
||||
websockets 17.2 17.3 wheel
|
||||
+17
@@ -0,0 +1,17 @@
|
||||
**Task 02 — no full table scan on a per-request path: done, all green.**
|
||||
|
||||
**Implemented**
|
||||
- `store.py`: `series_names()` cached 60 s (invalidated by `add`/`add_many`/`delete_series`/`maintain` via a generation counter; bounded to 256 `(source,host)` keys since `/api/metrics` takes both as free text); `stats()` counts cached 30 s with `last_maintain`/`size_bytes` deliberately never cached; `invalidate_read_caches()` for cold probes.
|
||||
- `pick_res()` now one compound statement of per-`res` `COUNT`s capped by `LIMIT max_points+1` (exact: the decision only asks `0` / `≤max_points` / neither), plus new `pick_res_many()` and `history_many()` (chunked at `COMPOUND_CHUNK`, index-range seek per branch).
|
||||
- `api.py`: all four history routes go through `_history_batch()` → `store.history_many()`; response dicts unchanged.
|
||||
- `tests/store_perf.py`: stats/`series_names` measured **cold and cached**, batch-vs-loop measured at 27 series × 24 h/90 d, EXPLAIN of the statement that actually runs.
|
||||
- Fixed a task-01 defect found in the full suite: the seeded determinism test pinned `SUM(value)`, whose float accumulation order changes when `INSERT OR REPLACE` moves rowids — now counts/windows exact, sum `rel=1e-9`, plus an exact PK-ordered row slice.
|
||||
- Docs: `AGENTS.md` (store.py/api.py bullets, gap 4 split into read-half-closed, commands), `README.md` (store.py, api.py, store_perf, new test pins), `data/logs/NOTES-store-perf.md` task-02 section (before/after, rejected shapes, EXPLAIN, live curl proof).
|
||||
|
||||
**Measured** (`uv run pytest -m store_perf --store-perf`): HTTP `/api/health` median 1 272 ms → **1.2 ms**; `/api/metrics?source=host` 1 415 → **1.6 ms**; `/api/system/history` 3×9 at 90 d 100.3 → **49.8 ms**; `pick_res()`×3 3.0 → **0.3 ms**. Live dev server on the real 636 MB store: `/api/health` 0.700 s → **1.5 ms**, all nine sources `ok`.
|
||||
|
||||
**Tests/lint** (exact): `bash .agents/validate.sh` → pytest `--cov --cov-fail-under=94` **1214 passed, 12 skipped, coverage 98.73%**; `ruff` All checks passed; `pyright` 0 errors; `node --check` ok. Final confirmation run `uv run pytest -q --cov --cov-fail-under=94` → **EXIT=0**, 0 FAILED (log `data/logs/pytest-final2.log`). No dependencies touched (validate's outdated-dep line is pre-existing advisory).
|
||||
|
||||
**Deviations**: batched the resolution *decision*, not the row fetch — `ROW_NUMBER()` window (73 ms) and a compound of the 27 tail selects (38 ms) both measured slower than 27 index seeks (27 ms) on the seeded store, and task 01 requires the numbers not to regress; recorded with its table in the notes. The literal `SELECT res, COUNT(*) … GROUP BY res` shape was also measured slower (4.3 vs 2.7 ms) than the capped compound that shipped. No live-infra changes beyond `./scripts/dev-server.sh restart` (local dev, invariant 10).
|
||||
|
||||
**Next pending task**: `.agents/phases/todo/14_store_performance/03_windowed_rollups_and_wal_tuning.md`.
|
||||
+1
@@ -0,0 +1 @@
|
||||
ok
|
||||
+64
@@ -0,0 +1,64 @@
|
||||
== uv run pytest --cov (hermetic: no network, live tests skip) ==
|
||||
........................................................................ [ 5%]
|
||||
........................................................................ [ 11%]
|
||||
........................................................................ [ 17%]
|
||||
........................................................................ [ 23%]
|
||||
........................................................................ [ 29%]
|
||||
........................................................................ [ 35%]
|
||||
........................................................................ [ 41%]
|
||||
........................................................................ [ 46%]
|
||||
........................................................................ [ 52%]
|
||||
........................................................................ [ 58%]
|
||||
........................................................................ [ 64%]
|
||||
.....................sssssssssss........................................ [ 70%]
|
||||
........................................................................ [ 76%]
|
||||
........................................................................ [ 82%]
|
||||
........................................................................ [ 88%]
|
||||
........................................................................ [ 93%]
|
||||
..................s..................................................... [ 99%]
|
||||
.. [100%]
|
||||
=============================== warnings summary ===============================
|
||||
.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
|
||||
/var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
================================ tests coverage ================================
|
||||
_______________ coverage: platform linux, python 3.14.4-final-0 ________________
|
||||
|
||||
Name Stmts Miss Cover Missing
|
||||
------------------------------------------------------------------
|
||||
src/dashboards/__init__.py 3 0 100%
|
||||
src/dashboards/__main__.py 14 1 93% 32
|
||||
src/dashboards/alerts.py 176 0 100%
|
||||
src/dashboards/api.py 356 0 100%
|
||||
src/dashboards/app.py 178 0 100%
|
||||
src/dashboards/auth.py 265 0 100%
|
||||
src/dashboards/collectors.py 155 5 97% 141-142, 146-147, 164
|
||||
src/dashboards/config.py 120 0 100%
|
||||
src/dashboards/debug.py 14 0 100%
|
||||
src/dashboards/poller.py 165 3 98% 192-194, 279
|
||||
src/dashboards/prom.py 112 5 96% 126, 163, 177, 180, 183
|
||||
src/dashboards/rates.py 73 0 100%
|
||||
src/dashboards/severity.py 49 0 100%
|
||||
src/dashboards/sources/__init__.py 0 0 100%
|
||||
src/dashboards/sources/disks.py 130 0 100%
|
||||
src/dashboards/sources/host.py 199 1 99% 147
|
||||
src/dashboards/sources/llm.py 178 1 99% 285
|
||||
src/dashboards/sources/nut.py 413 20 95% 153, 172, 186, 198, 204, 217, 225, 241, 319-320, 322, 579, 591, 626, 657, 661, 718-719, 724, 736
|
||||
src/dashboards/sources/nut_sim.py 134 0 100%
|
||||
src/dashboards/store.py 266 2 99% 445-446
|
||||
------------------------------------------------------------------
|
||||
TOTAL 3000 38 99%
|
||||
Required test coverage of 94% reached. Total coverage: 98.73%
|
||||
1214 passed, 12 skipped, 1 warning in 122.10s (0:02:02)
|
||||
== uv run ruff check src tests agent ==
|
||||
All checks passed!
|
||||
== uv run pyright ==
|
||||
0 errors, 0 warnings, 0 informations
|
||||
== node --check src/dashboards/web/static/*.js ==
|
||||
advisory: dependencies are out of date (AGENTS.md rule 8 — update on purpose):
|
||||
certifi 2026.7.22 2026.10.18 wheel
|
||||
fastapi 0.143.5 0.149.0 wheel
|
||||
starlette 1.8.0 1.9.0 wheel
|
||||
websockets 17.2 17.3 wheel
|
||||
+22
@@ -0,0 +1,22 @@
|
||||
**Phase 14 / Task 03 — windowed rollups + WAL tuning: done.**
|
||||
|
||||
Implementation (`src/dashboards/store.py`)
|
||||
- `maintain()` now windows both rollup steps off `last_maintain` (read via `get_meta()` *outside* the lock), overlapped by one bucket + `MAINTAIN_SLACK_SECONDS=60` and **snapped down to a bucket boundary** — a floor that cut into a bucket re-rolled a partial average from its tail rows (2.5 → 4.5); missing/non-numeric row falls back to full retention with a fixed-line warning.
|
||||
- Prune keeps its **full retention bound** but is now one `DELETE` per resolution: the old single `OR` of the three was unplannable (`SCAN sample`, 554 ms of a 562 ms pass while deleting nothing).
|
||||
- Statistics are the other half of the cost bound: `maintain()` ends with `PRAGMA optimize`, `__init__` sets `automatic_analyze=0` (SQLite's own refresh lands inside a poller's commit). WAL: `wal_autocheckpoint=1000`, `journal_size_limit=64 MB`; `busy_timeout=5000` deliberately unchanged.
|
||||
- Measured (seeded 6.4 M-row store): `maintain()` **5,368 ms → 2.7 ms** steady, 48 ms ten-minute catch-up; race at the app's cadence **100/100 inserts, 0 `database is locked`, max wait 21 ms** where the pre-fix control still waits 4.76 s; WAL 38 MB-forever → 0–4 MB. Live store: WAL 52.3 → 5.9 MB, `sqlite_stat1` created, real-store passes ~40 ms, `/api/health` 3.5 ms.
|
||||
|
||||
Tests / gate
|
||||
- `tests/test_store.py`: +9 tests (window bounds from a trace callback, catch-up-not-a-gap, windowed-vs-full-window row-for-row equivalence, re-roll only gains rows, corrupt-row fallback, `maintain` leaves statistics, optimize failure is a log line not a failed tick, WAL pragmas, two-cycle WAL cap).
|
||||
- `tests/store_perf.py`: write passes and races moved to a `scratch_copy()` (baseline never moves), added `RACE_MAINTAIN_GAP_SECONDS` cadence races + the whole-window **control**, per-window plans derived from `ROLLUP_INSERT`.
|
||||
- `bash .agents/validate.sh`: **1240 passed, 12 skipped**, coverage **98.75%** (floor 94), ruff + pyright + `node --check` clean; `-m store_perf --store-perf` 1 passed; dev server restarted, all 9 sources `ok`.
|
||||
|
||||
Docs
|
||||
- `AGENTS.md` (store.py bullet incl. the bucket-alignment rule, known gap 4 → write half closed, Commands, new testing-policy rule "a cost claim is pinned as a shape and a control"), `README.md` (store.py + `store_perf` paragraphs), `data/logs/NOTES-store-perf.md` (full task-03 tables, race-shape key, no-read-regression table, live smoke check).
|
||||
|
||||
Deviations
|
||||
- The task's "windowing + WAL tuning" needed two extra fixes to actually reach the budget (prune `OR`, missing statistics); both are measured and recorded rather than assumed, and the prune's retention bound is unchanged.
|
||||
- The saturated task-01 race is kept for comparability but the pass line moved to the cadence-shaped race (a maintainer looping with no gap starves a writer at *any* pass length); the control beside it proves the race can still see the failure.
|
||||
- Left alone: `uv pip list --outdated` advisory (certifi/fastapi/starlette/websockets) — pre-existing, no dependencies touched.
|
||||
|
||||
Next pending task: `.agents/phases/todo/14_store_performance/04_budget_gates_and_live_proof.md`.
|
||||
+1
@@ -0,0 +1 @@
|
||||
ok
|
||||
+111
@@ -0,0 +1,111 @@
|
||||
== uv run pytest --cov (hermetic: no network, live tests skip) ==
|
||||
........................................................................ [ 5%]
|
||||
........................................................................ [ 11%]
|
||||
........................................................................ [ 17%]
|
||||
........................................................................ [ 23%]
|
||||
........................................................................ [ 28%]
|
||||
........................................................................ [ 34%]
|
||||
........................................................................ [ 40%]
|
||||
........................................................................ [ 46%]
|
||||
........................................................................ [ 51%]
|
||||
........................................................................ [ 57%]
|
||||
........................................................................ [ 63%]
|
||||
.....................sssssssssss........................................ [ 69%]
|
||||
........................................................................ [ 74%]
|
||||
........................................................................ [ 80%]
|
||||
........................................................................ [ 86%]
|
||||
........................................................................ [ 92%]
|
||||
..............F.............................s........................... [ 97%]
|
||||
............................ [100%]
|
||||
=================================== FAILURES ===================================
|
||||
_______ test_a_second_maintain_reads_only_the_window_since_the_last_run ________
|
||||
|
||||
db = <dashboards.store.SampleStore object at 0x7fe8d2126c50>
|
||||
|
||||
def test_a_second_maintain_reads_only_the_window_since_the_last_run(db) -> None:
|
||||
"""The cost change, pinned as a bound and as behaviour.
|
||||
|
||||
A bucket that filled after the previous pass is inside the next window and gets rolled. A raw
|
||||
row that arrives for a bucket ten minutes behind it is not, and that bucket keeps the average
|
||||
the earlier pass computed — the documented cost of a window, stated here as a passing test
|
||||
rather than as a surprise. Reaching it in production costs a poller stall measured in minutes
|
||||
(the poller stamps samples with its own wall clock); the alternative it replaced was
|
||||
re-averaging 3.97 M rows every minute.
|
||||
"""
|
||||
now = time.time()
|
||||
bucket_a = bucket_start(now - 10 * MINUTE)
|
||||
add_series(db, "host", "h", "m", [10, 20], start=bucket_a, step=1)
|
||||
db.maintain(now=now)
|
||||
assert db.query("host", "h", "m", res=RES_MIN, start=bucket_a - 1,
|
||||
end=bucket_a + MINUTE) == [(bucket_a, 15.0)]
|
||||
|
||||
second = now + 60
|
||||
bucket_b = bucket_start(second - 2 * MINUTE)
|
||||
add_series(db, "host", "h", "m", [40, 50], start=bucket_b, step=1)
|
||||
add_series(db, "host", "h", "m", [1000], start=bucket_a + 30, step=1)
|
||||
with query_log(db) as seen:
|
||||
counts = db.maintain(now=second)
|
||||
|
||||
windows = rollup_windows(seen)
|
||||
# the bound: one minute (one hour) of rows plus the overlap, not 48 h of raw (not 30 d of 1m)
|
||||
assert windows[RES_RAW] == pytest.approx(
|
||||
(second - MINUTE - MAINTAIN_SLACK_SECONDS, bucket_start(now - MINUTE - MAINTAIN_SLACK_SECONDS)))
|
||||
> assert windows[RES_MIN] == pytest.approx(
|
||||
(second - HOUR - MAINTAIN_SLACK_SECONDS, bucket_start(now - HOUR - MAINTAIN_SLACK_SECONDS)))
|
||||
E assert (1792449272.5... 1792447200.0) == approx((17924....0 ± 1.8e+03))
|
||||
E
|
||||
E comparison failed. Mismatched elements: 1 / 2:
|
||||
E Max absolute difference: 1980.0
|
||||
E Max relative difference: 1.104635048664195e-06
|
||||
E Index | Obtained | Expected
|
||||
E 1 | 1792447200.0 | 1792449180.0 ± 1.8e+03
|
||||
|
||||
tests/test_store.py:802: AssertionError
|
||||
=============================== warnings summary ===============================
|
||||
.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
|
||||
/var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
================================ tests coverage ================================
|
||||
_______________ coverage: platform linux, python 3.14.4-final-0 ________________
|
||||
|
||||
Name Stmts Miss Cover Missing
|
||||
------------------------------------------------------------------
|
||||
src/dashboards/__init__.py 3 0 100%
|
||||
src/dashboards/__main__.py 14 1 93% 32
|
||||
src/dashboards/alerts.py 176 0 100%
|
||||
src/dashboards/api.py 356 0 100%
|
||||
src/dashboards/app.py 178 0 100%
|
||||
src/dashboards/auth.py 265 0 100%
|
||||
src/dashboards/collectors.py 155 5 97% 141-142, 146-147, 164
|
||||
src/dashboards/config.py 120 0 100%
|
||||
src/dashboards/debug.py 14 0 100%
|
||||
src/dashboards/poller.py 165 3 98% 192-194, 279
|
||||
src/dashboards/prom.py 112 5 96% 126, 163, 177, 180, 183
|
||||
src/dashboards/rates.py 73 0 100%
|
||||
src/dashboards/severity.py 49 0 100%
|
||||
src/dashboards/sources/__init__.py 0 0 100%
|
||||
src/dashboards/sources/disks.py 130 0 100%
|
||||
src/dashboards/sources/host.py 199 1 99% 147
|
||||
src/dashboards/sources/llm.py 178 1 99% 285
|
||||
src/dashboards/sources/nut.py 413 20 95% 153, 172, 186, 198, 204, 217, 225, 241, 319-320, 322, 579, 591, 626, 657, 661, 718-719, 724, 736
|
||||
src/dashboards/sources/nut_sim.py 134 0 100%
|
||||
src/dashboards/store.py 304 2 99% 510-511
|
||||
------------------------------------------------------------------
|
||||
TOTAL 3038 38 99%
|
||||
Required test coverage of 94% reached. Total coverage: 98.75%
|
||||
=========================== short test summary info ============================
|
||||
FAILED tests/test_store.py::test_a_second_maintain_reads_only_the_window_since_the_last_run
|
||||
1 failed, 1239 passed, 12 skipped, 1 warning in 111.30s (0:01:51)
|
||||
== uv run ruff check src tests agent ==
|
||||
All checks passed!
|
||||
== uv run pyright ==
|
||||
0 errors, 0 warnings, 0 informations
|
||||
== node --check src/dashboards/web/static/*.js ==
|
||||
advisory: dependencies are out of date (AGENTS.md rule 8 — update on purpose):
|
||||
certifi 2026.7.22 2026.10.18 wheel
|
||||
fastapi 0.143.5 0.150.0 wheel
|
||||
starlette 1.8.0 1.9.0 wheel
|
||||
websockets 17.2 17.3 wheel
|
||||
validation FAILED (see output above)
|
||||
+19
@@ -0,0 +1,19 @@
|
||||
**Phase 14 / Task 03 — re-run after the harness failure: green.**
|
||||
|
||||
What was fixed
|
||||
- The failing test was mine, and it was wall-clock luck, not the store: `test_a_second_maintain_reads_only_the_window_since_the_last_run` asserted the 1m→1h step's floor as a **minute**-aligned number while `_rollup_floor` snaps to an **hour**. `pytest.approx`'s relative tolerance (≈±1 800 s at epoch scale) covered the half of the hour where the two are within that, so it passed twice and failed at 19:4x.
|
||||
- Fix: `WINDOW_CLOCK_OFFSETS = (0, 1, 30, 59, 1 800, 3 539, 3 599)` over a base epoch that sits exactly on both boundaries — every edge is now a case instead of a coin flip — and each bound is snapped to *its own step's* bucket; the same parametrization went on the first-pass and catch-up tests so no window assertion in this task reads `time.time()`.
|
||||
- Found and fixed a second latent flake in the same test: the "due" bucket sat one step behind the ceiling, so a clock within a second of a boundary put the pair's second row outside the window and rolled 40.0 instead of 45.0 (now two steps back). `db.query()` defaults `end` to the wall clock — an explicit `end=` is now passed where the test runs on its own clock.
|
||||
- Added `test_the_window_floor_is_always_aligned_to_its_own_bucket` (aligned, never above the bound, never a whole window back, and the two steps really differ). Mutation-checked: removing the `floor(…/bucket)*bucket` snap fails 6 of 7 clock cases plus that test; restored code passes.
|
||||
|
||||
Results
|
||||
- `bash .agents/validate.sh`: **1259 passed, 12 skipped, 0 failed** · coverage **98.75%** (floor 94, unchanged) · `ruff check src tests agent` all passed · `pyright` 0 errors · `node --check` ok · hygiene ok · no `validation FAILED`.
|
||||
- `uv run pytest -m store_perf --store-perf`: **1 passed** (window plans seek, catch-up ≤ 2× steady + slack, the whole-window control still contends, the cadence race is 0 `database is locked`).
|
||||
- Dev server healthy: all 9 sources `ok`, real-store WAL holding at ~7.6 MB under the 64 MB cap.
|
||||
|
||||
Notable decisions
|
||||
- No production-code change this pass — the defect was in the test's expectation, so the implementation (windowed rollups, per-resolution prune, `PRAGMA optimize` + `automatic_analyze=0`, WAL pragmas, `busy_timeout` untouched) stands as measured: `maintain()` 5 368 ms → 2.7 ms steady / 48 ms catch-up.
|
||||
- Preferred explicit clocks and a property test over widening a tolerance, and recorded the flake plus both root causes in `data/logs/NOTES-store-perf.md` ("One window assertion was wall-clock luck").
|
||||
- Left alone: the pre-existing `uv pip list --outdated` advisory (certifi/fastapi/starlette/websockets) — no dependencies touched in this task.
|
||||
|
||||
Next pending task: `.agents/phases/todo/14_store_performance/04_budget_gates_and_live_proof.md`.
|
||||
+1
@@ -0,0 +1 @@
|
||||
ok
|
||||
+64
@@ -0,0 +1,64 @@
|
||||
== uv run pytest --cov (hermetic: no network, live tests skip) ==
|
||||
........................................................................ [ 5%]
|
||||
........................................................................ [ 11%]
|
||||
........................................................................ [ 16%]
|
||||
........................................................................ [ 22%]
|
||||
........................................................................ [ 28%]
|
||||
........................................................................ [ 33%]
|
||||
........................................................................ [ 39%]
|
||||
........................................................................ [ 45%]
|
||||
........................................................................ [ 50%]
|
||||
........................................................................ [ 56%]
|
||||
........................................................................ [ 62%]
|
||||
.....................sssssssssss........................................ [ 67%]
|
||||
........................................................................ [ 73%]
|
||||
........................................................................ [ 79%]
|
||||
........................................................................ [ 84%]
|
||||
........................................................................ [ 90%]
|
||||
...............................................................s........ [ 96%]
|
||||
............................................... [100%]
|
||||
=============================== warnings summary ===============================
|
||||
.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
|
||||
/var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
================================ tests coverage ================================
|
||||
_______________ coverage: platform linux, python 3.14.4-final-0 ________________
|
||||
|
||||
Name Stmts Miss Cover Missing
|
||||
------------------------------------------------------------------
|
||||
src/dashboards/__init__.py 3 0 100%
|
||||
src/dashboards/__main__.py 14 1 93% 32
|
||||
src/dashboards/alerts.py 176 0 100%
|
||||
src/dashboards/api.py 356 0 100%
|
||||
src/dashboards/app.py 178 0 100%
|
||||
src/dashboards/auth.py 265 0 100%
|
||||
src/dashboards/collectors.py 155 5 97% 141-142, 146-147, 164
|
||||
src/dashboards/config.py 120 0 100%
|
||||
src/dashboards/debug.py 14 0 100%
|
||||
src/dashboards/poller.py 165 3 98% 192-194, 279
|
||||
src/dashboards/prom.py 112 5 96% 126, 163, 177, 180, 183
|
||||
src/dashboards/rates.py 73 0 100%
|
||||
src/dashboards/severity.py 49 0 100%
|
||||
src/dashboards/sources/__init__.py 0 0 100%
|
||||
src/dashboards/sources/disks.py 130 0 100%
|
||||
src/dashboards/sources/host.py 199 1 99% 147
|
||||
src/dashboards/sources/llm.py 178 1 99% 285
|
||||
src/dashboards/sources/nut.py 413 20 95% 153, 172, 186, 198, 204, 217, 225, 241, 319-320, 322, 579, 591, 626, 657, 661, 718-719, 724, 736
|
||||
src/dashboards/sources/nut_sim.py 134 0 100%
|
||||
src/dashboards/store.py 304 2 99% 510-511
|
||||
------------------------------------------------------------------
|
||||
TOTAL 3038 38 99%
|
||||
Required test coverage of 94% reached. Total coverage: 98.75%
|
||||
1259 passed, 12 skipped, 1 warning in 112.04s (0:01:52)
|
||||
== uv run ruff check src tests agent ==
|
||||
All checks passed!
|
||||
== uv run pyright ==
|
||||
0 errors, 0 warnings, 0 informations
|
||||
== node --check src/dashboards/web/static/*.js ==
|
||||
advisory: dependencies are out of date (AGENTS.md rule 8 — update on purpose):
|
||||
certifi 2026.7.22 2026.10.18 wheel
|
||||
fastapi 0.143.5 0.150.0 wheel
|
||||
starlette 1.8.0 1.9.0 wheel
|
||||
websockets 17.2 17.3 wheel
|
||||
+14
@@ -0,0 +1,14 @@
|
||||
Task 04 is complete and everything is green.
|
||||
|
||||
**What shipped**
|
||||
- `tests/store_perf.py` — `BUDGETS`: 14 pinned ceilings (2‑3× the measured "after"), `budget_rows`/``budget_violations`/`missing_budget_keys`, `measure_budget_reads/_http`, and the budget table rendered into the probe report.
|
||||
- Budget tests: 2 default-run pins (a budget whose measurement was renamed fails; the table is internally coherent) + 4 `store_perf` passes (reads, HTTP, `maintain`, race). `uv run pytest -m store_perf --store-perf` → 4 passed, **14/14 budgets inside**.
|
||||
- **The watch found a real defect in task 02's fix** and I fixed it: the series cache was invalidated by *every* sample write, so on a store nine pollers write to it never served and `/api/metrics` paid the full `SELECT DISTINCT` scan (measured 535 ms/call; the scan alone is 572 ms on the live store). `store.py` now invalidates only the write that can change the answer (`_note_series_write` + a capped `_known_series` existence set; `DELETE` paths still invalidate unconditionally). Pinned by statement counts, not clocks, in `tests/test_store.py` and through the served endpoint in `tests/test_api.py`.
|
||||
- Live proof (`data/logs/watch-store-perf.py` → `store-perf-watch.{log,tsv,json}`, `store-perf-live.png`): 25 min / 74 passes / 296 signed-in requests on the operator's 636 MB store — **0 new `store-error` rows** (190 total, newest 2026‑10‑18 06:33), **WAL 6.5 → 7.77 MB** (cap 67 MB), 0 non‑200, medians 1.8–3.3 ms for `/api/health`, `/api/summary`, `/api/metrics` and a 3×9 90‑day `/api/system/history`.
|
||||
- Docs: `data/logs/NOTES-store-perf.md` gained the task‑04 section (baseline vs after side by side, EXPLAIN shapes, the watch table, the defect write-up); `README.md` and `AGENTS.md` updated (gap 4 now reads: closed, with the honest remainder).
|
||||
- Gates: `bash .agents/validate.sh` exit 0 — 1267 passed, 15 skipped, coverage 98.75%, ruff + pyright + `node --check` clean. Dev server restarted, all 9 sources `ok`.
|
||||
|
||||
**Honest leftovers**
|
||||
1. The p95 is the cache window, not the request: on the live store a cold `series_names('host')` costs 599 ms and `stats()`' counts 729 ms, once per 60 s / 30 s (37 of 74 passes had one request >250 ms). Closing that needs a counts table or a bounded series index — task 03's note already names it, and it is a phase, not this one.
|
||||
2. `uv pip list --outdated` is non-empty (fastapi 0.143.5→0.150.0, starlette, certifi, websockets) — pre-existing drift, untouched here; rule 8 wants a deliberate decision.
|
||||
3. Per AGENTS rule 4 I did **not** modify `.agents/phases/todo/14_store_performance/04_...md` — say the word and I'll tick the completion criteria and move it to `complete/`.
|
||||
+1
@@ -0,0 +1 @@
|
||||
ok
|
||||
+64
@@ -0,0 +1,64 @@
|
||||
== uv run pytest --cov (hermetic: no network, live tests skip) ==
|
||||
........................................................................ [ 5%]
|
||||
........................................................................ [ 11%]
|
||||
........................................................................ [ 16%]
|
||||
........................................................................ [ 22%]
|
||||
.............................................s.......................... [ 28%]
|
||||
........................................................................ [ 33%]
|
||||
........................................................................ [ 39%]
|
||||
........................................................................ [ 44%]
|
||||
........................................................................ [ 50%]
|
||||
........................................................................ [ 56%]
|
||||
........................................................................ [ 61%]
|
||||
........................sssssssssss..................................... [ 67%]
|
||||
........................................................................ [ 73%]
|
||||
........................................................................ [ 78%]
|
||||
........................................................................ [ 84%]
|
||||
........................................................................ [ 89%]
|
||||
........................................................................ [ 95%]
|
||||
sss....................................................... [100%]
|
||||
=============================== warnings summary ===============================
|
||||
.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
|
||||
/var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
================================ tests coverage ================================
|
||||
_______________ coverage: platform linux, python 3.14.4-final-0 ________________
|
||||
|
||||
Name Stmts Miss Cover Missing
|
||||
------------------------------------------------------------------
|
||||
src/dashboards/__init__.py 3 0 100%
|
||||
src/dashboards/__main__.py 14 1 93% 32
|
||||
src/dashboards/alerts.py 176 0 100%
|
||||
src/dashboards/api.py 356 0 100%
|
||||
src/dashboards/app.py 178 0 100%
|
||||
src/dashboards/auth.py 265 0 100%
|
||||
src/dashboards/collectors.py 155 5 97% 141-142, 146-147, 164
|
||||
src/dashboards/config.py 120 0 100%
|
||||
src/dashboards/debug.py 14 0 100%
|
||||
src/dashboards/poller.py 165 3 98% 192-194, 279
|
||||
src/dashboards/prom.py 112 5 96% 126, 163, 177, 180, 183
|
||||
src/dashboards/rates.py 73 0 100%
|
||||
src/dashboards/severity.py 49 0 100%
|
||||
src/dashboards/sources/__init__.py 0 0 100%
|
||||
src/dashboards/sources/disks.py 130 0 100%
|
||||
src/dashboards/sources/host.py 199 1 99% 147
|
||||
src/dashboards/sources/llm.py 178 1 99% 285
|
||||
src/dashboards/sources/nut.py 413 20 95% 153, 172, 186, 198, 204, 217, 225, 241, 319-320, 322, 579, 591, 626, 657, 661, 718-719, 724, 736
|
||||
src/dashboards/sources/nut_sim.py 134 0 100%
|
||||
src/dashboards/store.py 313 2 99% 553-554
|
||||
------------------------------------------------------------------
|
||||
TOTAL 3047 38 99%
|
||||
Required test coverage of 94% reached. Total coverage: 98.75%
|
||||
1267 passed, 15 skipped, 1 warning in 113.83s (0:01:53)
|
||||
== uv run ruff check src tests agent ==
|
||||
All checks passed!
|
||||
== uv run pyright ==
|
||||
0 errors, 0 warnings, 0 informations
|
||||
== node --check src/dashboards/web/static/*.js ==
|
||||
advisory: dependencies are out of date (AGENTS.md rule 8 — update on purpose):
|
||||
certifi 2026.7.22 2026.10.19 wheel
|
||||
fastapi 0.143.5 0.150.0 wheel
|
||||
starlette 1.8.0 1.9.0 wheel
|
||||
websockets 17.2 17.3 wheel
|
||||
@@ -181,15 +181,16 @@ Responsibilities, one line each:
|
||||
`offline_uncorrectable`, hex-packed vendor raws are never decoded, and an attribute row is
|
||||
read from its **RAW_VALUE column** rather than the end of the line because this binary
|
||||
annotates that column — `35 (Min/Max 34/44)`, `43894h+43m+32.519s` — so its last token is
|
||||
not the value), 5 s per device /
|
||||
20 s per pass / 240 s cache, attached as `disk_io.devices[].smart`. Missing binary →
|
||||
not the value), 5 s per device / a pass budget whose 20 s floor scales to 5 s × the disk count
|
||||
/ 240 s cache, attached as `disk_io.devices[].smart` (gap 10: a 16-drive shelf takes longer
|
||||
than the floor, and truncating the pass would null the tail drives). Missing binary →
|
||||
**one** `errors` line for the pass (the word `not found` in it is what
|
||||
`sources/disks.py` reads to answer "is smartctl installed", and
|
||||
`tests/test_agent.py` pins that the line still says it); one denied/wedged device → that
|
||||
disk's `smart` is `null` with its own line; `--no-smart` opts out silently. Never fails
|
||||
the snapshot. **An expired cache refreshes on a background thread while the request is
|
||||
served the previous result** — only the first pass after startup runs inline — because 20 s
|
||||
of wedged disk on the request path blows `HOST_TIMEOUT_SECONDS` and would turn the
|
||||
served the previous result** — only the first pass after startup runs inline — because a wedged
|
||||
pass on the request path blows `HOST_TIMEOUT_SECONDS` and would turn the
|
||||
*system* source red every four minutes over an add-on that page never draws. The refresher
|
||||
is single-flight and keeps the old cache on failure, so a bad pass retries on the next
|
||||
expiry instead of hammering the disk.
|
||||
@@ -230,7 +231,7 @@ Responsibilities, one line each:
|
||||
that was dialled, because they become health records and event rows. `fetch_snapshot()` is
|
||||
the one agent `/snapshot` read that `system` and `disks` share, so a dead host says the
|
||||
same sentence on both pages and only the `timeout` differs (the disks poll has to outlast
|
||||
the agent's 20 s smartctl pass, and `None` is translated into httpx's sentinel because a
|
||||
the agent's smartctl pass, and `None` is translated into httpx's sentinel because a
|
||||
literal `None` means *no timeout at all*). Agent `errors`
|
||||
become the payload's `degraded` **except** the `smart:` add-on lines — a host without
|
||||
smartmontools or device read access is unconfigured, not broken (invariant 3); they stay
|
||||
@@ -397,7 +398,57 @@ Responsibilities, one line each:
|
||||
call, because a reset that deletes a row it did not name deletes the operator's credential).
|
||||
`stats()` reads `last_maintain` through `get_meta()` too — outside its own lock, which
|
||||
`get_meta` takes itself and `threading.Lock` is not reentrant — while `maintain()`'s write
|
||||
stays inline inside that method's transaction.
|
||||
stays inline inside that method's transaction. **No per-request path may pay a whole-table
|
||||
scan** (phase 14 task 02): `series_names()` (1.4 s of `SCAN sample` on a production-sized store,
|
||||
behind `/api/metrics`) and the two `COUNT`s behind `stats()` (1.25 s, run by `app.js`'s
|
||||
`/api/health` poll on every tab every 15 s) answer from short-TTL caches — 60 s and 30 s, the
|
||||
series list additionally invalidated **only by a write that can change it** (task 04: an insert
|
||||
cannot remove a name, so `_note_series_write()` ends the cache on a `(source, host, metric)` the
|
||||
store has never written and leaves a poll of an existing series alone — invalidating on every
|
||||
sample was task 02's defect, and it put the 572 ms scan back on nearly every `/api/metrics` call
|
||||
on a store nine pollers write to, behind a cache that never served; `delete_series()`/`maintain()`
|
||||
invalidate unconditionally because a `DELETE` can take a name, `delete_series()` also forgetting
|
||||
the triples it emptied so a re-created series counts as new again), through a generation counter so
|
||||
a write racing a read cannot park a stale answer under a fresh timestamp, with the name cache
|
||||
bounded to 256 `(source, host)` keys because `/api/metrics` takes both as free text and the
|
||||
existence tracking capped at `SERIES_TRACKED_SERIES_MAX`, below which it falls back to
|
||||
invalidating on every write. A tracking set made wrong by *another* process deleting rows costs at
|
||||
most one TTL window, which is the reason the cache has a TTL and not only the tracking. `last_maintain` and `size_bytes` are never
|
||||
cached: a `last_maintain` that stops moving is how an operator spots the invariant-4 failure
|
||||
mode, and it costs a three-row lookup. The resolution decision is one compound statement of
|
||||
per-`(host, metric, res)` branches whose `COUNT` is capped by `LIMIT max_points + 1` — exact,
|
||||
because the decision only ever asks whether a count is `0`, `≤ max_points`, or neither — and
|
||||
`pick_res_many()`/`history_many()` are the batched door the N × M endpoints go through. The rows
|
||||
stay a per-series seek **measuredly**: a `ROW_NUMBER()` window or a compound of the 27 tail
|
||||
selects both cost more than 27 index seeks on the seeded store (`data/logs/NOTES-store-perf.md`
|
||||
records the table), so batching the *decision* is the fix and batching the *fetch* would be a
|
||||
round trip traded for work.
|
||||
**The write side is windowed too** (phase 14 task 03): `maintain()` reads `last_maintain` through
|
||||
`get_meta()` *before* its transaction (same non-reentrant-lock rule as `stats()`) and bounds each
|
||||
rollup step to what became due since, instead of averaging every row the resolution retains —
|
||||
3.97 M rows of 1-minute history per minute, a 5.4 s write transaction that made a poller's
|
||||
`add_many` outlive `busy_timeout=5000` (gap 4's `store-error` line). Two properties make that a
|
||||
cost bound and not a data change, and both are pinned: the lower bound reaches **past** the
|
||||
previous pass (one bucket plus slack, so ten quiet minutes are a catch-up scan rather than ten
|
||||
minutes of empty charts), and it is snapped **down to a bucket boundary** — the step's ceiling is
|
||||
a row bound, so the newest bucket is partial and a later pass re-rolls it as its rows land, and a
|
||||
floor that cut into a bucket would overwrite the fuller average with the mean of its tail rows
|
||||
(2.5 becoming 4.5 on a six-sample minute: rule 2 with a chart's clothes on). A window is only a
|
||||
cost bound if the plan can seek to it, and two things were in the way. The prune keeps its full
|
||||
retention bound — a rollup may window because it only adds rows, a prune must reach anything that
|
||||
expired — but it is **one `DELETE` per resolution**, because one
|
||||
`WHERE (res=? AND ts<?) OR (res=? AND ts<?) OR (…)` is unreadable to the planner and read the
|
||||
whole 6.4 M-row table to delete nothing (554 ms of a 562 ms pass; 0.3 ms per statement now). And
|
||||
the store had no query statistics, so every window was priced as a scan however few rows were
|
||||
inside it: `maintain()` ends with `PRAGMA optimize` and `__init__` sets `automatic_analyze=0`,
|
||||
because SQLite's own refresh lands inside whichever poller's commit trips its counter — the exact
|
||||
path this shortens — while a tick a minute costs ~100 ms once and 0.0 ms afterwards. Measured on
|
||||
the seeded store: **5,368 ms → 2.7 ms** steady, 48 ms for a ten-minute catch-up, and the
|
||||
two-connection race at the app's cadence goes from 5 of 9 inserts raised to 100 of 100 landing
|
||||
with a 21 ms worst wait. `__init__` also sets `wal_autocheckpoint` (SQLite's own 1 000 pages,
|
||||
written down so the knob is visible) plus `journal_size_limit = 64 MB`, the cap that truncates a
|
||||
runaway WAL at the next checkpoint. `busy_timeout=5000` is unchanged: the fix is a shorter write,
|
||||
not a longer wait.
|
||||
- `api.py` — GETs everywhere plus the one POST that dials an upstream, and the `auth_router()`
|
||||
that creates, describes, **re-credentials**, and ends a session (`POST /api/auth/login` public;
|
||||
`GET /api/auth/status`, `POST /api/auth/logout` and `POST /api/auth/password` protected). Those
|
||||
@@ -441,6 +492,14 @@ Responsibilities, one line each:
|
||||
write of its own — a fourth dashboard must not quietly become a second one, so the app's
|
||||
only POST stays `/api/ups/command` (invariant 1, pinned by enumerating every served route's
|
||||
verbs).
|
||||
All four `history` routes reach the store through one door, `api._history_batch()` →
|
||||
`store.history_many()` (phase 14 task 02). That is a cost change and nothing else: the series
|
||||
list is host-major and metric-minor in request order, each entry the same
|
||||
`{source, host, metric, range_seconds, points}` document the per-series loop built, with the
|
||||
same empty-series answer for a name that has no rows. `tests/test_api.py` pins it for all four
|
||||
routes against points read straight out of the store with `query()` — a third path, in neither
|
||||
the loop nor the batch — so a future "optimisation" that moves a point, an order, or a key
|
||||
fails a test instead of redrawing a chart wrong.
|
||||
`/api/summary` is the overview's single call and the cheapest whole-fleet probe there is:
|
||||
the page renders only `alerts` plus `system.fleet`/`system.hosts`, while `disks.hosts`,
|
||||
`llm.engines`, `ups.ups`, the `states` maps and `recent_events` stay for programmatic
|
||||
@@ -567,10 +626,11 @@ That scheme is the history API's key — treat it as a public interface.
|
||||
`--host 10.1.0.126 --port 9120 --token-file /etc/dashboards-agent.token` (`640
|
||||
root:dashboards-agent`, the mode `deploy/install-agent.sh` writes) with `User=dashboards-agent`,
|
||||
and `getent group disk` → `disk:x:6:dashboards-agent`.
|
||||
**Two things about that install are not what this repo writes, and both cost SMART data on
|
||||
`/disks` (see the SMART block below): `systemctl show -p DevicePolicy dashboards-agent` answers
|
||||
`auto` with no `DeviceAllow` lines** (the installer's `strict` + node allow-list is not what is
|
||||
running there), **and its `smartctl` cannot read drives as that user.**
|
||||
**The deployed unit has since diverged from the template in the three ways the SMART
|
||||
block below records** (the `DevicePolicy`/`DeviceAllow` lines removed, `User=root`,
|
||||
`CapabilityBoundingSet=CAP_SYS_RAWIO CAP_SYS_ADMIN`) — re-running `deploy/install-agent.sh`
|
||||
on driveripper would clobber the working unit, because it rewrites the unit unconditionally
|
||||
(gap 10).
|
||||
`/api/health` today reads **all nine sources `ok`**: `host:driveripper`, `host:deskwork`,
|
||||
`host:gateway`, `disks:driveripper`, `disks:deskwork`, `disks:gateway`, `nut`, `llm:gufo`,
|
||||
`llm:vllm`.
|
||||
@@ -657,15 +717,29 @@ That scheme is the history API's key — treat it as a public interface.
|
||||
`nvme0n1`…`nvme7n1` + `sda`…`sdh`) and `smart` is non-null for the 8 NVMes, but every one of
|
||||
them carries only identity — `health`, `temp_c`, `power_on_hours`, `percentage_used` and the
|
||||
counters are all `null` — while the 8 spinning drives come back `smart: null` each with the
|
||||
agent's own line `smart: sda: Try an additional '-d ata' or '-d sat' argument.` Measured
|
||||
read-only on 2026-10-19: the same `smartctl -i -A -H /dev/nvme0n1` **as root** prints
|
||||
`PASSED`, 55 °C and 14 % used, so the drives answer and the *agent user* is what is locked out
|
||||
— `/dev/nvme0`, `/dev/ng0n1` and `/dev/bsg/*` (the nodes NVMe admin and SATA pass-through
|
||||
actually go through) are `crw------- root root`, and only `/dev/nvme0n1` / `/dev/sg0` carry the
|
||||
`disk` group the agent is in. So `/disks` shows driveripper's drive inventory with `n/a` values
|
||||
and a why-line, which is the designed degradation, and **lighting it up is a new phase, not a
|
||||
token**: a udev rule granting those nodes, and/or `host_agent.smart()` learning the
|
||||
`-d sat` / `/dev/ngXnY` forms smartctl needs when auto-detection fails.
|
||||
agent's own line `smart: sda: Try an additional '-d ata' or '-d sat' argument.` Pinned on
|
||||
2026-10-19 with read-only probes (`data/logs/NOTES-driveripper-agent.md`): the `disk`-group
|
||||
**open of the block nodes works** (agent user, `O_RDWR`), there is no SELinux denier (zero
|
||||
AVCs with `auditd` running), and the install's `DevicePolicy=strict`/`DeviceAllow` lines were
|
||||
removed from the unit (they matched nothing useful anyway — systemd resolves `DeviceAllow`
|
||||
globs by name against `/proc/devices`, where this kernel lists NVMe as `blkext`, so
|
||||
`nvme*` matches nothing). What locks the agent out is **kernel 7.2's unprivileged
|
||||
passthrough gate**: NVMe admin via the block node is unprivileged-only for five identify
|
||||
types; the SMART-log read needs the drive's ACEL to mark it supported (the Samsung 980
|
||||
PRO's 2020-era `5B2QGXA7` does not — `NVME_IOCTL_ADMIN_CMD: Permission denied`) or
|
||||
`CAP_SYS_ADMIN`, and SATA SMART behind a SAT layer is ATA pass-through, which the
|
||||
unprivileged `SG_IO` allow-list (`scsi_cmd_allowed()`: read-safe opcodes only) does not
|
||||
include — explicit `-d sat`/`-d ata` fails identically, so no `host_agent.smart()` retry
|
||||
form can reach it. `deskwork` reads live with the same user/unit because its drive's
|
||||
firmware does advertise the page. So `/disks` shows driveripper's drive inventory with
|
||||
`n/a` values and a why-line — the designed degradation, **resolved the same day (gap 10):
|
||||
the unit now runs `User=root` with
|
||||
`CapabilityBoundingSet=CAP_SYS_RAWIO CAP_SYS_ADMIN`, and all 16 drives read live** (the
|
||||
980 PRO's 59 °C / 6328 h / 14 % row, `nvme1n1` at 58 % wear, the red `13 uncorrectable`
|
||||
alarm chip on `nvme5n1`; verified in `data/logs/disks-driveripper-live.png`). A root
|
||||
process with an **empty** capability bounding set is still capability-less (`CapEff: 0`)
|
||||
because the kernel gates check capabilities, not uid — the named bounding set is what
|
||||
unblocked it.
|
||||
**Real data is in the repo now:** `tests/fixtures/smartctl_driveripper_<dev>.txt` is all 16
|
||||
drives (8 NVMe, 8 spinning) read through that sudo rule, and `tests/test_agent.py` parses
|
||||
every one — which is what found the parser's one real bug: this binary annotates the
|
||||
@@ -681,9 +755,8 @@ That scheme is the history API's key — treat it as a public interface.
|
||||
*not* a sudo rule, because the agent must read the device itself — on an `agent/host_agent.py`
|
||||
from phase 09 or later (never redeploy without the grant: 16 permission-denied lines would age
|
||||
nothing but are pure noise, while with the grant they are 16 drives' worth of temperature,
|
||||
hours and sector counters). **The agent and the `disk` group are now both in place on
|
||||
`driveripper`, and it still reads no SMART numbers** — the group only covers nodes that carry
|
||||
it (see above), so what is left is the node grant / `-d sat` work, not an install. `gateway` needs smartmontools first and will still read no SMART
|
||||
hours and sector counters). **The agent and the `disk` group are both in place on
|
||||
`driveripper`** — the `disk` group alone is not enough on kernel 7.2 (the passthrough gate, see above), so the unit additionally runs as root with the two gate capabilities bounded (gap 10) — all 16 drives now read live. `gateway` needs smartmontools first and will still read no SMART
|
||||
after that, being a VM with no physical disk behind `vda`. A host that cannot read its drives
|
||||
renders the designed degradation: one dimmed card per physical drive, every number `n/a`, and
|
||||
the one-line why.
|
||||
@@ -711,9 +784,26 @@ set -a; . ./.env.dev; set +a # dev dials the fleet the prod way, no
|
||||
./scripts/dev-server.sh status # per-source ok/DOWN — unauthenticated: /api/health
|
||||
./scripts/dev-server.sh logs
|
||||
|
||||
uv run pytest # hermetic: 1195 passed + 11 live-skipped, no network
|
||||
uv run pytest --cov # coverage gate (98.7%; floor 94%)
|
||||
uv run pytest # hermetic: 1267 passed + 15 skipped (11 live,
|
||||
# 4 store_perf), no network
|
||||
uv run pytest --cov # coverage gate (98.75%; floor 94%)
|
||||
uv run pytest -m live --live # opt-in: real fleet / engines / upsd
|
||||
uv run pytest -m store_perf --store-perf # opt-in: the store hot-path timings over the seeded
|
||||
# store, priced against `store_perf.BUDGETS` (the cached
|
||||
# reads report cold *and* warm, so a before/after never
|
||||
# books a cache hit as a speed-up)
|
||||
uv run python scripts/watch-store-perf.py
|
||||
# ... and the live proof: 25 signed-in minutes against the
|
||||
# running dev server, timing the page paths while counting
|
||||
# `store-error` rows and the WAL size (DASH_WATCH_SECONDS
|
||||
# shortens it; writes store-perf-watch.{log,tsv,json})
|
||||
uv run python scripts/store_perf_probe.py # ... as a report (data/logs/store-perf-baseline.{md,json},
|
||||
# then store-perf-task03.* for the write side and
|
||||
# store-perf-task04.* for the budget table);
|
||||
# --rebuild re-seeds, --dev-db <file> also times a real
|
||||
# store read-only, DASH_PERF_STORE_DIR moves the cache, and
|
||||
# every pass that *writes* runs on a scratch copy so the
|
||||
# seeded baseline never moves under the next measurement
|
||||
uv run ruff check src tests agent && uv run pyright
|
||||
./scripts/capture-fixtures.sh # re-record ground truth (agents + UPS + gufo /slots)
|
||||
SMART_HOSTS="deskwork-ops driveripper-ops" ./scripts/capture-fixtures.sh
|
||||
@@ -837,6 +927,28 @@ branch `main`); there is no CI on it, so `bash .agents/validate.sh` is the only
|
||||
verbatim, the four config rows, the `DASH_COOKIE_SECURE=1` deployment note, the three auth
|
||||
gaps). No server starts and no request is made: a doc that stops naming the reset command is a
|
||||
locked-out admin, and that is worth a test even though it asserts nothing about Python.
|
||||
- **A cost claim is pinned as a shape and a control, never as a wall clock** (phase 14). The default
|
||||
run keeps the seeded-store tests hermetic and asserts **shape and determinism** — counts,
|
||||
retention windows, the statements a plan is taken from, idempotence — while the timings live behind
|
||||
`-m store_perf --store-perf` and in `scripts/store_perf_probe.py`, because a budget asserted on
|
||||
first writing is a number fitted to the machine it was written on. Two rules came out of it.
|
||||
(1) A cheaper query is proven *equal*, not just faster:
|
||||
`test_windowed_rollups_answer_exactly_what_one_full_window_rollup_answers` compares every stored
|
||||
row of a windowed store against a second store rolled the old way, and
|
||||
`test_a_re_rolled_bucket_only_ever_gains_rows` pins the one bug that equivalence alone misses (a
|
||||
window floor that cuts into a bucket re-rolls it from its tail rows). (2) A race that comes back
|
||||
clean has to be shown capable of going dirty: `store_perf` runs the pre-fix whole-window
|
||||
`maintain()` as a **control** beside the windowed one and `tests/test_store.py` fails if the control
|
||||
stops contending, because "0 `database is locked`" from a race that never met is the invariant-4
|
||||
failure mode with a green checkmark attached. (3) **A budget is a table, not a test body**
|
||||
(task 04): the 14 ceilings live in `store_perf.BUDGETS` with the measured figure and the margin in
|
||||
each row's `basis`, the probe renders that table, and two default-run tests fail if a budget stops
|
||||
being priced or a ceiling contradicts its own ratio partner — a ceiling nobody measures is a
|
||||
comment. (4) A cache's cost claim needs the *write* cadence in the measurement: task 02's series
|
||||
cache reported 0.0 ms on a quiet seeded store and served almost nothing on a live one, because
|
||||
every sample write invalidated it. The pinned version is statement-counting, not timing — a poll of
|
||||
existing series issues no `SELECT`, a write of a name nobody wrote ends the cache — and the live
|
||||
version is `scripts/watch-store-perf.py`, which is where the defect showed up.
|
||||
- Coverage gate: 80% is the repo floor, the phase gate in `.agents/validate.sh` enforces
|
||||
94% (`--cov-fail-under`), and ~98% is where we are; keep it there. New/modified Python
|
||||
still has to clear >90% on its own. API tests use `create_app()` with an in-memory DB,
|
||||
@@ -1221,7 +1333,7 @@ memory plot; the `cpu.usage_pct@coreN` series is still stored and still queryabl
|
||||
field map is written.
|
||||
3. **UPS history depth.** Events and the numeric series are stored; long-window battery
|
||||
health (capacity fade over months) has the rollups but no dedicated view.
|
||||
4. **The dev DB is big enough to refuse a write now and then.** The dev store (~640 MB on
|
||||
4. **The dev DB's `database is locked` line is fixed (phase 14); a cold-cache tail is left.** The dev store (~640 MB on
|
||||
2026-10-19, its WAL grown past 140 MB at its worst) logged `store-error: OperationalError:
|
||||
database is locked` a few times an hour across `system`/`llm`/`ups` through 2026-10-18; the
|
||||
poller handles it exactly as invariant 4 requires — the source goes red with the real reason,
|
||||
@@ -1232,8 +1344,52 @@ memory plot; the `cpu.usage_pct@coreN` series is still stored and still queryabl
|
||||
phase 12's re-check on 2026-10-19 found the 190 stored `store-error` rows all older than
|
||||
2026-10-18 06:33 with none in the previous 24 h of the running dev store — the absence of a
|
||||
sample, not the absence of the pressure, since the file only keeps growing.
|
||||
Retention/`wal_checkpoint` tuning for a long-running store is its own phase, not a
|
||||
phase-09 follow-up.
|
||||
Retention/`wal_checkpoint` tuning for a long-running store was its own phase rather than a
|
||||
phase-09 follow-up — that phase is `14_store_performance`, and the two paragraphs below are
|
||||
what it measured and changed.
|
||||
**Phase 14 has since split this in two, and the read half is closed.** The in-process half was
|
||||
never a SQLite lock at all: `SampleStore._lock` serialises everything, so a 1.25 s `stats()`
|
||||
from a browser tab's `/api/health` poll was a poller waiting up to 1.45 s to insert 30 rows
|
||||
(measured, `data/logs/NOTES-store-perf.md`). Task 02 took the two whole-table reads off the
|
||||
per-request path, and `/api/health` now answers in ~1.2 ms instead of ~1.27 s, so the store
|
||||
mutex is no longer held by a reader for seconds at a time. The **write half is closed too**
|
||||
(task 03): `maintain()` rolls up only what became due since `last_maintain`, its prune is one
|
||||
`DELETE` per resolution instead of one `OR` the planner could not use, and `PRAGMA optimize` runs
|
||||
there (with SQLite's automatic ANALYZE switched off, so no poller's commit pays for it). On the
|
||||
seeded 6.4 M-row store that is 5,368 ms → **2.7 ms** steady / 48 ms for a ten-minute catch-up, and
|
||||
the two-connection race at the app's cadence is **100 of 100 inserts, zero `database is locked`,
|
||||
max wait 21 ms** where the pre-fix control still waits 4.76 s (task 04's re-run of the same
|
||||
budget: 6.8 ms steady / 46 ms catch-up / 8.5 ms worst wait, so the row is a range, not a
|
||||
digit). The WAL follows the same shape: a steady pass leaves ~0 MB and a catch-up tick 4 MB
|
||||
instead of 38 MB that never comes back down, and `journal_size_limit=64 MB` caps what a
|
||||
long-lived reader can make it grow to (`data/logs/NOTES-store-perf.md` has the tables).
|
||||
**The live proof is in** (task 04, 25 minutes / 74 passes / 296 signed-in requests against the
|
||||
running dev server on the operator's own 636 MB store): **0 new `store-error` rows** (still 190,
|
||||
newest 2026-10-18 06:33:30, i.e. 39 h before the watch), **WAL 6.5 → 7.77 MB** against the 67.1 MB
|
||||
cap and a main file that did not move, 0 non-200 responses, and page paths at 1.8-3.3 ms median
|
||||
(`/api/health`, `/api/summary`, `/api/metrics`, a 3-host × 9-metric 90-day `/api/system/history`).
|
||||
The watch also caught a defect in task 02's fix — the series cache was invalidated by *every*
|
||||
sample write, so on a store the hub writes to every few seconds it never served and `/api/metrics`
|
||||
paid the 572 ms scan on nearly every call (measured: 535 ms per call pre-fix, 1.8 ms median
|
||||
after) — and that is what the store bullet above now describes.
|
||||
What is **left** is the cost of the two cold answers themselves, measured on that store: 599 ms
|
||||
for `series_names('host')`, 729 ms for `stats()`' counts, now once per cache window (~1.3 s of
|
||||
store mutex per minute) instead of once per call — which is why the watch's medians are 1.8-3.3 ms
|
||||
and its p95s are 620/729 ms: the p95 is the window boundary, and 37 of 74 passes had one request
|
||||
over 250 ms, every one of them a cold re-derivation. No cache policy reaches *that*; the fix is
|
||||
the bounded `(source, metric)` structure or a counts table named in task 03's note. Two cheaper
|
||||
knobs exist and were deliberately not pulled, because both change what `/api/health` answers and
|
||||
move rather than remove the mutex hold: raising `STATS_CACHE_TTL_SECONDS`, and refreshing both
|
||||
caches from the maintenance tick so the miss lands on a poller instead of a page load. That is a
|
||||
phase, not a cleanup. Two servers on one file can still contend, and `busy_timeout=5000` is
|
||||
unchanged on purpose so a second one stays visible as a slower write; what they contend with is
|
||||
now milliseconds instead of 5.4 s.
|
||||
One consequence worth naming, because two other gaps pointed here for it: the **~10 s
|
||||
process-wide stall** (gap 9's `Δ`-window dilution, `poller`'s catch-up gaps) is gone with the
|
||||
mutex holds — coincident multi-second gaps in the 2 s `llm` and 5 s `ups` raw streams went from
|
||||
393 and 198 in the two halves of 2026-10-19 to **0** across the 110 minutes the phase-14 store
|
||||
has run (re-measured in this phase's verification pass from the `sample` table; the signature
|
||||
separates a process stall from one slow engine).
|
||||
5. **The login throttle keys on the socket peer address** (phase 11), and the one
|
||||
`LoginThrottle` instance now holds two keys per address: the bare one for
|
||||
`POST /api/auth/login` and `password:<address>` for `POST /api/auth/password`. Behind nginx
|
||||
@@ -1315,7 +1471,14 @@ memory plot; the `cpu.usage_pct@coreN` series is still stored and still queryabl
|
||||
pass line and what the watch above stayed under. The `Δ<window>` label stayed honest throughout —
|
||||
both before and after the fix it names the window actually used; do not "fix" anything
|
||||
here by hiding or rewriting it. The ~10–15 s process-wide **stall** that triggers a
|
||||
refused poll is still gap 4's and `14_store_performance`'s problem; the estimator no longer
|
||||
refused poll was gap 4's and `14_store_performance`'s problem, and **phase 14 closed it**
|
||||
(measured on the dev store from the sample table: coincident multi-second gaps in the 2 s `llm`
|
||||
and 5 s `ups` raw streams — the signature of a process-wide stall, not a slow engine — were
|
||||
393 in 00:00–12:00 and 198 in 12:00–21:36 on 2026-10-19, and **0** across the 110 minutes the
|
||||
phase-14 store has been running, worst `llm` gap 7.1 s and that one the recorded gufo timeout).
|
||||
The mechanism is the one phase 14 removed: a whole-table `stats()` count held the store mutex
|
||||
for ~1.25 s on every browser tab's `/api/health` poll, and `maintain()` held the write lock for
|
||||
5.4 s. The estimator no longer
|
||||
turns a slow fetch into 0.2 s arithmetic, and one `n/a` per stall is the designed degradation
|
||||
(the residual the verification pass measured is the paragraph below).
|
||||
**The verification pass found one residual path and closed it** (same day, live rows in the
|
||||
@@ -1341,11 +1504,47 @@ memory plot; the `cpu.usage_pct@coreN` series is still stored and still queryabl
|
||||
35.6 once the freed slot settled — every value labelled, **0 of 540 above the 300 pass line**,
|
||||
and no stored raw row over it since the restart either. A collapsed window is what makes a
|
||||
burst *look* like a rate, so the fix for this family is the ~10 s stall itself (gap 4 /
|
||||
`14_store_performance`), not a fresh threshold invented in the estimator: task 03 pinned
|
||||
`14_store_performance` — which has since removed it, see gap 4 and the paragraph above, so what
|
||||
is left of this family is the sub-8× skew itself), not a fresh threshold invented in the
|
||||
estimator: task 03 pinned
|
||||
`MIN_WINDOW_SECONDS` as an absolute floor and pinned that an in-bounds diluted row publishes
|
||||
(`test_a_fallback_inside_the_documented_ceiling_still_publishes`), so a span-relative floor
|
||||
would be a re-decision of that phase, not a bug fix. Do not "fix" the residual by hiding the
|
||||
short window or its label.
|
||||
10. **`driveripper`'s 16 drives read `n/a`: kernel 7.2's unprivileged-passthrough gate (2026-10-19).**
|
||||
The agent is installed, the `disk` group grant is in place, the block nodes open, and the
|
||||
page renders all 16 real drive rows — but kernel 7.2.11 blocks the *agent user's* `smartctl`
|
||||
at the command level (not the open; the probe matrix is in
|
||||
`data/logs/NOTES-driveripper-agent.md`): the 8 SATA drives' SMART is ATA pass-through
|
||||
behind a SAT layer, and the kernel's unprivileged `SG_IO` allow-list
|
||||
(`scsi_cmd_allowed()`, read-safe opcodes only) does not include it — `CAP_SYS_RAWIO` or
|
||||
root unblocks it, with or without `-d sat`/`-d ata`; the 8 Samsung/WD NVMes' SMART-log
|
||||
read needs the drive's ACEL to advertise the page (the 980 PRO's `5B2QGXA7`, 2020-era,
|
||||
does not: `NVME_IOCTL_ADMIN_CMD: Permission denied`) or `CAP_SYS_ADMIN`; `deskwork` reads
|
||||
live because its firmware does. Options, in order of narrowness: (a) accept the designed
|
||||
degradation (what `/disks` renders now — the identity is real, the values are `n/a` with a
|
||||
why-line); (b) `AmbientCapabilities=CAP_SYS_RAWIO` (+ `CapabilityBoundingSet`) on the unit
|
||||
— unblocks exactly the 8 HDDs, is the capability the kernel names for the gate, but a
|
||||
compromised agent could then write raw block I/O — test it in a `systemd-run --scope`
|
||||
before committing; (c) `User=root` with the empty capability set — unblocks all 16 but
|
||||
hands a token-guarded process root's file-read DAC (not recommended); (d) a Samsung
|
||||
firmware update — the only clean fix for the NVMe side. **Resolved the same day with a
|
||||
tighter variant of (c): `User=root` + `CapabilityBoundingSet=CAP_SYS_RAWIO CAP_SYS_ADMIN`
|
||||
— all 16 drives read live** (root DAC is the price; the bounding set keeps the grant
|
||||
named rather than full). Two empirical notes: a root process under the unit's *empty*
|
||||
bounding set is capability-less (`CapEff: 0`), because the kernel gates check
|
||||
capabilities, not uid — that is why the first `User=root` attempt still read `n/a`; and
|
||||
`CapEff` came back `0000000000220000` (SYS_ADMIN + systemd's auto CHROOT; RAWIO in the
|
||||
bounding set but not effective) — the 8 SATA drives' reads were unblocked by SYS_ADMIN
|
||||
alone, so the effective grant is {SYS_ADMIN}; the RAWIO in the line stays as the named
|
||||
intent for the `SG_IO` gate. The agent also gained the pass-budget scale-up (20 s floor
|
||||
→ 5 s × disk count; an explicit override still wins, the budget test is untouched),
|
||||
because a 16-disk pass takes more than 20 s when the drives answer. The `deploy/install-agent.sh`
|
||||
template still ships the `DevicePolicy`/`DeviceAllow` lines that misbehaved on this host
|
||||
(a deny-all filter; the `nvme*` glob matches nothing on this kernel because
|
||||
`/proc/devices` lists NVMe as `blkext`) — the template change is its own phase.
|
||||
`tests/fixtures/smartctl_driveripper_*.txt` (captured via `sudo`) remain the ground truth
|
||||
for the drives' shape.
|
||||
|
||||
## Locked decisions (and why)
|
||||
|
||||
|
||||
@@ -91,7 +91,8 @@ a labelled window, so the agent never has to remember who polled it last.
|
||||
|
||||
Per-disk `smart` health is a best-effort add-on on the whole-disk `/sys/block` list (no
|
||||
partitions): the agent runs `smartctl -i -A -H` per physical disk — 5 s bound per device,
|
||||
20 s per pass, results cached 4 minutes — and parses health, temperature, power-on hours,
|
||||
a pass budget of 20 s as a floor that scales to 5 s × the disk count, results cached 4 minutes —
|
||||
and parses health, temperature, power-on hours,
|
||||
the sector alarms and SSD wear into `disk_io.devices[].smart`. Refreshing an expired cache
|
||||
happens on a background thread while the request is served the previous result, so a wedged
|
||||
disk can never cost a poll more than its own timeout (only the very first pass after the
|
||||
@@ -334,7 +335,7 @@ documentation and is never loaded, and `scripts/dev-server.sh` exports `.env.dev
|
||||
| `HOST_INTERVAL_SECONDS` / `LLM_INTERVAL_SECONDS` / `NUT_INTERVAL_SECONDS` | how often each source is polled (10 / 2 / 5 s) |
|
||||
| `DISKS_INTERVAL_SECONDS` | how often the disk-health source re-reads each agent for SMART (default 300 s — SMART is not a per-second signal). No targets of its own: the `disks:<host>` jobs exist exactly when `HOST_TARGETS` does |
|
||||
| `HOST_TIMEOUT_SECONDS` / `LLM_TIMEOUT_SECONDS` / `NUT_TIMEOUT_SECONDS` | per-source fetch timeout — one slow source never holds up the others |
|
||||
| `DISKS_TIMEOUT_SECONDS` | the disks fetch's own budget (default 30 s), because the agent runs `smartctl` inside its snapshot handler and 20 s of wedged disk must not read as a dead host |
|
||||
| `DISKS_TIMEOUT_SECONDS` | the disks fetch's own budget (default 30 s), because the agent runs `smartctl` inside its snapshot handler and a wedged SMART pass must not read as a dead host |
|
||||
| `GUFO_API_KEY` / `VLLM_API_KEY` | only if an engine is behind a bearer token; they never leave the server |
|
||||
| `NUT_HOST` / `NUT_PORT` | upsd address — **whatever port it really listens on**, not always 3493. Reads are unauthenticated in stock NUT |
|
||||
| `NUT_USER` / `NUT_PASSWORD` | only needed for instant commands (battery tests) |
|
||||
@@ -383,9 +384,27 @@ auth.py ── verifier + session key, loaded once at startup (no row and no see
|
||||
poller, so a SQLite write that gets refused is reported (`ok: false`, the real reason, the
|
||||
numbers aging, one line in the event log) instead of freezing the source under a green dot.
|
||||
* `store.py` — raw samples roll up to 1-minute and 1-hour averages and the raw rows are
|
||||
pruned, so a year of history is a few hundred MB rather than a disk emergency. Its `meta`
|
||||
table is the one place state that must outlive a restart goes (`get_meta`/`set_meta`/
|
||||
`delete_meta`), and its first user is the admin credential below.
|
||||
pruned, so a year of history is a few hundred MB rather than a disk emergency. Nothing a
|
||||
per-request path asks is answered by a whole-table scan **on every call** (phase 14): the two
|
||||
answers that used to be scanned on every call — `series_names()` behind `/api/metrics` and the
|
||||
row counts behind `/api/health` — come from a 60 s / 30 s cache, so the scan is paid once per
|
||||
window (its cost is the honest remainder in the known-gaps list) instead of once per request. `series_names`' is ended only by a write that can
|
||||
change it: a `(source, host, metric)` the store has never written, or any `DELETE`. A poll of an
|
||||
existing series leaves it alone, which is what makes the cache serve at all on a store nine pollers
|
||||
write to (task 04's live watch is the proof: 535 ms per call before, 1.8 ms median after), and the
|
||||
counts age on the clock. A chart's resolution comes from one capped compound statement instead of
|
||||
three full `COUNT`s per series. `last_maintain` and `size_bytes` are deliberately never cached —
|
||||
the first is how an operator tells a stalled rollup loop from a healthy one.
|
||||
The write side is windowed too: `maintain()` rolls up only the buckets that became due since its
|
||||
last run (read from `last_maintain`, overlapped by one bucket and snapped down to a bucket
|
||||
boundary, so a gap in polling is a catch-up scan rather than missing charts) and prunes on its
|
||||
full retention bound, one statement per resolution. It ends with `PRAGMA optimize` — the store's
|
||||
only statistics refresh, deliberately not inside a poller's commit — and the WAL is kept honest by
|
||||
`wal_autocheckpoint` plus a 64 MB `journal_size_limit` cap. On a 6.4 M-row store that took the
|
||||
once-a-minute write transaction from 5.4 s to a few milliseconds, which is what stopped a poller's
|
||||
insert losing its 5 s `busy_timeout` (known gap 4).
|
||||
Its `meta` table is the one place state that must outlive a restart goes (`get_meta`/
|
||||
`set_meta`/`delete_meta`), and its first user is the admin credential below.
|
||||
* `auth.py` — the admin login's decisions, in stdlib and without HTTP: a self-describing
|
||||
PBKDF2-HMAC-SHA256 verifier, an HMAC-signed stateless session cookie (no session table, and
|
||||
rotating the stored key is what evicts every other browser), a session-bound CSRF token, and
|
||||
@@ -423,7 +442,12 @@ auth.py ── verifier + session key, loaded once at startup (no row and no see
|
||||
no env config answers `404 {"configured": false}` **without dialling** (so a page says
|
||||
"not configured" instead of going red), and a history query for a series that does not
|
||||
exist answers an empty series instead of an error (so a card asks for every plot it could
|
||||
draw and reads `n/a` for the ones the drive never reported). `/api/disks/snapshot` is the
|
||||
draw and reads `n/a` for the ones the drive never reported). The four `history` routes take
|
||||
one batched door into the store (`api._history_batch` → `store.history_many`), so a page's
|
||||
chart request is one grouped resolution query plus one index seek per series rather than four
|
||||
queries per series — and the JSON it answers with is the same document the per-series loop
|
||||
produced, series for series and in the same order (`tests/test_api.py` pins that against
|
||||
points read straight out of the store with `query()`). `/api/disks/snapshot` is the
|
||||
drive-health surface: one entry per host with its physical disk rows, the agent's `smart`
|
||||
block verbatim, `smartctl_available` (`true` / `false` = install smartmontools / `null` =
|
||||
the agent never said), a `severity` beside each row's rated numbers, and a `fleet` block
|
||||
@@ -557,6 +581,11 @@ uv run pytest # unit + API tests, fully mocked,
|
||||
uv run pytest --cov # coverage gate
|
||||
set -a; . ./.env.dev; set +a # the live suite reads the same env as the app
|
||||
uv run pytest -m live --live # optional: real fleet / engines / upsd
|
||||
uv run pytest -m store_perf --store-perf -s # optional: the store's hot-path timings (phase 14)
|
||||
uv run python scripts/store_perf_probe.py # the same measurements as a report in data/logs/
|
||||
DASH_WATCH_SECONDS=1500 uv run python scripts/watch-store-perf.py
|
||||
# ... and the live watch: 25 signed-in minutes against
|
||||
# the running dev server, counting store-error and WAL
|
||||
uv run ruff check src tests agent && uv run pyright # lint + types
|
||||
./scripts/capture-fixtures.sh # re-record ground truth: host agents, UPS replies, gufo /slots
|
||||
./scripts/capture-fixtures.sh --with-llm # ... and the engines' /metrics while they are up
|
||||
@@ -611,6 +640,38 @@ local-testing word stays legal in prose, which is the only place it can be usefu
|
||||
the four public paths, the reset command, the four variables and the `DASH_COOKIE_SECURE=1`
|
||||
deployment note. Nothing starts a server there; a sentence that quietly disappears is the bug.
|
||||
|
||||
`tests/store_perf.py` is the second opt-in marker's machinery (`live` is the other). It seeds a
|
||||
**production-shaped
|
||||
SQLite file** — `SampleStore`'s own retention (`raw 48 h / 1m 30 d / 1h 400 d`), the fleet's real
|
||||
metric names, the real 10 s cadence: ~6.4 M rows and ~720 MB, the dev DB's order of magnitude — and
|
||||
times every hot path of `store.py`/`api.py` against it, with `EXPLAIN QUERY PLAN` for the queries
|
||||
phase 14 changes and a two-connection race that reproduces the `database is locked` line in the
|
||||
known-gaps list — beside the **pre-fix control** of the same race, because a clean race that never
|
||||
met a lock would prove nothing. Every pass that *writes* (a rollup catch-up, a race) runs on a
|
||||
scratch copy of the seeded file, so the baseline the next measurement compares against comes back
|
||||
exactly as it went in. The default suite builds that store and asserts its **shape and determinism**
|
||||
(row counts, retention windows, rollups that are the mean of the raw rows they came from, a second
|
||||
`maintain()` that changes nothing) but never its wall time — except behind the marker, where
|
||||
`store_perf.BUDGETS` is the phase's pinned regression table (14 ceilings, 2-3x the measured "after",
|
||||
`maintain()` < 1 s, a racing insert at zero `database is locked`) and a stale budget pointer fails a
|
||||
default-run test. The cached reads are reported twice —
|
||||
**cold** (the cache invalidated before every run) and **cached** — because comparing a cache hit
|
||||
against the baseline's full scan would book a 1 000x speed-up that is really just a cache. Only
|
||||
`-m store_perf --store-perf` runs the timing passes, and `scripts/store_perf_probe.py` is the CLI
|
||||
form that writes `data/logs/store-perf-baseline.{md,json}` (task 03's write-side run is
|
||||
`store-perf-task03.*`, task 04's budget run `store-perf-task04.*`) for `data/logs/NOTES-store-perf.md`. Two rules keep it
|
||||
honest: **no test ever opens `data/dashboards.sqlite3`** (the `--dev-db` flag exists for a human
|
||||
running the probe read-only), and the seeded file is a cache under `data/perf/`
|
||||
(`DASH_PERF_STORE_DIR` relocates it) because a before/after has to measure the same file — pytest's
|
||||
`tmp_path` on this box is `tmpfs`, i.e. RAM, which times a store nobody deploys.
|
||||
|
||||
The non-timing half of that lives in `tests/test_store.py` and runs every time: a cache hit
|
||||
issues **no** SQL at all (counted through `sqlite3`'s trace callback, never through a wall clock),
|
||||
the series list is invalidated by the write that introduces a name and not by the poll that repeats
|
||||
one (and the tracking that decides has a cap, below which it falls back to invalidating),
|
||||
`last_maintain` is never served from a cache, and `pick_res_many()`/`history_many()` answer exactly
|
||||
what the per-series calls answer — on a small store and on the seeded one, at every window.
|
||||
|
||||
One more file is worth knowing about: `tests/card_contract.py` is the table of
|
||||
`/api/system/snapshot` paths the system page's per-host cards are allowed to read. It is
|
||||
checked against all three captured host shapes (discrete GPU, APU, no GPU), over HTTP, and
|
||||
|
||||
+18
-8
@@ -22,7 +22,8 @@ Design rules
|
||||
* **Every read is guarded.** A vanished /proc entry (process exited mid-scan) skips that
|
||||
item; the snapshot still succeeds. The poll is degraded, never failed.
|
||||
* **SMART is a best-effort add-on.** The per-disk `smart` block runs `smartctl -i -A -H`
|
||||
in a subprocess: bounded 5 s per device and 20 s per pass, cached 240 s so a fast
|
||||
in a subprocess: bounded 5 s per device and a pass budget of 20 s scaled to the disk
|
||||
count, cached 240 s so a fast
|
||||
dashboard poll cannot re-issue ATA commands every few seconds. Refreshing an expired
|
||||
cache happens on a background thread while the request is served the previous result —
|
||||
20 s of wedged disk must never cost a poll more than its own timeout. No binary, a denied
|
||||
@@ -122,10 +123,12 @@ VIRTUAL_NET_PREFIXES = ("lo", "veth", "docker", "br-", "virbr", "br", "tap", "vn
|
||||
VIRTUAL_DISK_PREFIXES = ("loop", "ram", "zram", "sr", "dm-", "md127")
|
||||
|
||||
# SMART collection bounds. 5 s per device is generous for a healthy drive and caps how
|
||||
# long one wedged device can hold a snapshot hostage; 20 s caps the whole pass so even a
|
||||
# shelf of bad disks cannot stall the HTTP handler; 240 s is below the dashboard's
|
||||
# 5-minute disks cadence, so the slow disks job gets fresh data while the fast system
|
||||
# poll never re-runs smartctl every 10 s.
|
||||
# long one wedged device can hold a snapshot hostage; the pass budget is a 20 s floor
|
||||
# scaled to the per-device bound (5 s x disk count), so a 16-disk shelf gets the 80 s a
|
||||
# real read takes while a single-disk host keeps 20 s and a shelf of bad disks cannot
|
||||
# stall the HTTP handler; 240 s is below the dashboard's 5-minute disks cadence, so the
|
||||
# slow disks job gets fresh data while the fast system poll never re-runs smartctl
|
||||
# every 10 s.
|
||||
SMARTCTL_TIMEOUT_SECONDS = 5.0
|
||||
SMARTCTL_BUDGET_SECONDS = 20.0
|
||||
SMARTCTL_CACHE_SECONDS = 240.0
|
||||
@@ -522,7 +525,8 @@ class Collector:
|
||||
same error lines, so a snapshot always states why `smart` is null.
|
||||
|
||||
An **expired** cache is refreshed on a background thread and the request is served
|
||||
the previous result, because the pass budget (20 s) is five times the dashboard's
|
||||
the previous result, because the pass budget (20 s floor, 5 s per disk) is five
|
||||
times the dashboard's
|
||||
`HOST_TIMEOUT_SECONDS` (4 s): run inline on the request path, one wedged disk would
|
||||
turn the *system* source red every `smart_cache_seconds` for a fault in an add-on
|
||||
(AGENTS.md invariant 4 — SMART must not blink the page that does not use it). The
|
||||
@@ -569,14 +573,20 @@ class Collector:
|
||||
now = time.monotonic()
|
||||
results: dict[str, dict[str, Any] | None] = {}
|
||||
errors: list[str] = []
|
||||
deadline = now + self.smart_budget_seconds
|
||||
# 20 s floor scaled to the per-device bound: a 16-disk shelf takes more than 20 s
|
||||
# when the drives answer, and truncating the pass would null the tail drives.
|
||||
# An explicit override (the budget test sets this to 0) wins over the scale.
|
||||
budget = (max(SMARTCTL_BUDGET_SECONDS, self.smart_timeout_seconds * len(device_names))
|
||||
if self.smart_budget_seconds == SMARTCTL_BUDGET_SECONDS
|
||||
else self.smart_budget_seconds)
|
||||
deadline = now + budget
|
||||
for name in device_names:
|
||||
left = deadline - time.monotonic()
|
||||
if left <= 0:
|
||||
remaining = [n for n in device_names if n not in results]
|
||||
for skipped_name in remaining:
|
||||
results[skipped_name] = None
|
||||
errors.append(f"smart: pass budget of {self.smart_budget_seconds:.0f}s "
|
||||
errors.append(f"smart: pass budget of {budget:.0f}s "
|
||||
f"exhausted, {len(remaining)} disk(s) skipped")
|
||||
break
|
||||
try:
|
||||
|
||||
@@ -47,6 +47,11 @@ markers = [
|
||||
# `-m live --live` runs them; without --live they SKIP, so `uv run pytest` stays
|
||||
# network-free. See tests/test_live_fleet.py for the env vars each one needs.
|
||||
"live: needs real hardware (fleet hosts, LLM engines, or upsd) — opt in with --live",
|
||||
# `-m store_perf --store-perf` runs them: the phase-14 timing measurements over the seeded
|
||||
# production-shaped store (tests/store_perf.py). The seed itself runs in the default suite;
|
||||
# only the multi-minute measurement passes are behind this flag. Underscored because Python
|
||||
# decorators cannot carry a hyphen; the CLI flag stays `--store-perf`.
|
||||
"store_perf: measures the seeded store's hot paths — opt in with --store-perf",
|
||||
]
|
||||
# auto: async tests run without a decorator on every single one
|
||||
asyncio_mode = "auto"
|
||||
|
||||
Executable
+95
@@ -0,0 +1,95 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Measure the store's hot paths against a production-shaped SQLite file.
|
||||
|
||||
Phase 14 exists because `TODO.md:7` says "pages take forever to load and seem to lock up the
|
||||
database". Task 01's job is the **before**: one run of this script is the baseline in
|
||||
`data/logs/NOTES-store-perf.md`, and task 04 runs it again as the "after". Nothing here changes
|
||||
behaviour — it reads, times, `EXPLAIN`s, and cleans up after itself.
|
||||
|
||||
uv run python scripts/store_perf_probe.py # baseline (builds the store once)
|
||||
uv run python scripts/store_perf_probe.py --rebuild # force a fresh seed
|
||||
uv run python scripts/store_perf_probe.py --dir /tmp/perf # measure on tmpfs (RAM: faster)
|
||||
uv run python scripts/store_perf_probe.py --out data/logs/store-perf-after
|
||||
|
||||
It writes two files beside the numbers it prints: `<out>.json` (everything, machine-readable)
|
||||
and `<out>.md` (the same tables as markdown, to paste into the NOTES). The seeded store itself
|
||||
lives in `data/perf/` (gitignored, like the dev DB) and is reused while it is fresh — the same
|
||||
physical file has to be the one timed before and after the fixes, or the two numbers describe
|
||||
two databases.
|
||||
|
||||
The shape, the timings and the machine specs all come from `tests/store_perf.py`, which is also
|
||||
what the pytest budget tests of this phase use: one definition of "production-shaped", because
|
||||
two of those drift.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import sys
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
for entry in (ROOT / "tests", ROOT / "src"):
|
||||
# `tests/store_perf.py` is not a package (pytest imports it the same way via conftest), and
|
||||
# a bare checkout has no installed `dashboards` either. Both paths, or nothing works.
|
||||
if str(entry) not in sys.path:
|
||||
sys.path.insert(0, str(entry))
|
||||
|
||||
import store_perf # noqa: E402 - after the sys.path fixup above, by design
|
||||
|
||||
|
||||
def log(message: str) -> None:
|
||||
print(f"[{time.strftime('%H:%M:%S')}] {message}", flush=True)
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
|
||||
parser.add_argument("--dir", default=None,
|
||||
help="where the seeded store lives (default: $DASH_PERF_STORE_DIR "
|
||||
"or data/perf — a real block device, unlike pytest's tmpfs)")
|
||||
parser.add_argument("--rebuild", action="store_true",
|
||||
help="throw the cached store away and seed a new one")
|
||||
parser.add_argument("--out", default=str(ROOT / "data" / "logs" / "store-perf-baseline"),
|
||||
help="prefix for the report files (default: data/logs/store-perf-baseline)")
|
||||
parser.add_argument("--race-iterations", type=int, default=100,
|
||||
help="how many 30-sample inserts to race against stats()/maintain() "
|
||||
"(task 01 says ~100)")
|
||||
parser.add_argument("--race-seconds", type=float, default=store_perf.MAX_RACE_SECONDS,
|
||||
help=f"wall-clock budget per race pass (default "
|
||||
f"{store_perf.MAX_RACE_SECONDS:.0f} s — one stats() pass costs "
|
||||
"seconds on this store, so the pass is bounded by time as well as "
|
||||
"by the insert count)")
|
||||
parser.add_argument("--dev-db", default=None,
|
||||
help="also time the three hot reads against this *real* database file, "
|
||||
"opened read-only (e.g. data/dashboards.sqlite3). Nothing here "
|
||||
"writes to it and no test ever passes this flag — the hermetic "
|
||||
"suite never opens the operator's store.")
|
||||
parser.add_argument("--no-http", action="store_true",
|
||||
help="skip the TestClient pass (/api/health, /api/metrics, "
|
||||
"/api/system/history) — store-only numbers")
|
||||
args = parser.parse_args(argv)
|
||||
|
||||
path = store_perf.ensure_seeded(args.dir, force=args.rebuild, log=log)
|
||||
store = store_perf.open_store(path)
|
||||
try:
|
||||
report = store_perf.measure(store, race_iterations=args.race_iterations,
|
||||
race_max_seconds=args.race_seconds,
|
||||
with_http=not args.no_http, log=log)
|
||||
finally:
|
||||
store.close()
|
||||
|
||||
if args.dev_db:
|
||||
log(f"read-only pass over the real store: {args.dev_db}")
|
||||
report["dev_db"] = store_perf.measure_dev_db(args.dev_db)
|
||||
|
||||
written = store_perf.write_report(report, Path(args.out))
|
||||
print()
|
||||
print(store_perf.render_markdown(report))
|
||||
print()
|
||||
print(f"report: {written['md']}\n {written['json']}")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
Executable
+194
@@ -0,0 +1,194 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Phase 14 task 04: watch the running dev server for `database is locked`.
|
||||
|
||||
Sampled, not exhaustive: every ~20 s a signed-in pass hits the paths a page actually loads
|
||||
(``/api/health``, ``/api/summary``, ``/api/metrics``, a 3-host x 9-metric 90-day
|
||||
``/api/system/history``) and records the wall time beside the store's own evidence — the
|
||||
``store-error`` rows in the ``event`` table, the WAL file size, the database size.
|
||||
|
||||
Run it in the background while the dev server keeps polling the fleet:
|
||||
|
||||
set -a; . ./.env.dev; set +a
|
||||
setsid nohup uv run python scripts/watch-store-perf.py > /tmp/watch.out 2>&1 < /dev/null &
|
||||
|
||||
Writes ``data/logs/store-perf-watch.log`` (human), ``store-perf-watch.tsv`` (machine) and
|
||||
``store-perf-watch.json`` (the summary every path ran against). ``data/`` is gitignored, so those
|
||||
are local evidence for ``data/logs/NOTES-store-perf.md``, never tracked artifacts.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import sqlite3
|
||||
import statistics
|
||||
import sys
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
import httpx
|
||||
|
||||
#: Repo root, so the paths below mean the same tree wherever the watch is invoked from
|
||||
#: (the same anchor ``scripts/store_perf_probe.py`` uses).
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
|
||||
BASE_URL = os.environ.get("DASH_WATCH_URL", "http://127.0.0.1:8090")
|
||||
ADMIN_PASSWORD = os.environ.get("DASH_ADMIN_PASSWORD", "")
|
||||
DB_PATH = Path(os.environ.get("DASH_WATCH_DB", ROOT / "data" / "dashboards.sqlite3"))
|
||||
OUT = Path(os.environ.get("DASH_WATCH_OUT", ROOT / "data" / "logs"))
|
||||
DURATION_SECONDS = float(os.environ.get("DASH_WATCH_SECONDS", "1500"))
|
||||
INTERVAL_SECONDS = float(os.environ.get("DASH_WATCH_INTERVAL", "20"))
|
||||
|
||||
#: What a page load may cost. Everything here is a panel the operator is looking at.
|
||||
PAGE_BUDGET_MS = 250.0
|
||||
|
||||
METRICS = [
|
||||
"cpu.usage_pct", "cpu.temp_c", "cpu.load1", "mem.used_pct", "disk.used_pct@/",
|
||||
"disk.used_pct@/home", "net.rx_bps", "net.tx_bps", "gpu.busy_pct@0",
|
||||
]
|
||||
|
||||
|
||||
def store_evidence(db: sqlite3.Connection, since: float) -> dict[str, float | int | str]:
|
||||
"""The store's own record, read on a separate connection while the server runs."""
|
||||
total = db.execute("SELECT COUNT(*) FROM event WHERE kind='store-error'").fetchone()[0]
|
||||
recent = db.execute(
|
||||
"SELECT COUNT(*) FROM event WHERE kind='store-error' AND ts>=?", (since,)
|
||||
).fetchone()[0]
|
||||
last = db.execute(
|
||||
"SELECT message FROM event WHERE kind='store-error' ORDER BY ts DESC LIMIT 1"
|
||||
).fetchone()
|
||||
return {
|
||||
"store_errors_total": total,
|
||||
"store_errors_in_window": recent,
|
||||
"last_store_error": (last[0] if last else ""),
|
||||
}
|
||||
|
||||
|
||||
def timed(client: httpx.Client, path: str) -> tuple[float, int]:
|
||||
started = time.perf_counter()
|
||||
try:
|
||||
response = client.get(path)
|
||||
code = response.status_code
|
||||
except httpx.HTTPError: # a refused request is a sample too
|
||||
return (time.perf_counter() - started) * 1000.0, -1
|
||||
return (time.perf_counter() - started) * 1000.0, code
|
||||
|
||||
|
||||
def main() -> int:
|
||||
started = time.perf_counter()
|
||||
OUT.mkdir(parents=True, exist_ok=True)
|
||||
log_path = OUT / "store-perf-watch.log"
|
||||
tsv_path = OUT / "store-perf-watch.tsv"
|
||||
json_path = OUT / "store-perf-watch.json"
|
||||
samples: list[dict[str, float | int | str]] = []
|
||||
|
||||
client = httpx.Client(base_url=BASE_URL, timeout=60.0, follow_redirects=False)
|
||||
login = client.post("/api/auth/login", json={"username": "admin", "password": ADMIN_PASSWORD})
|
||||
if login.status_code != 200:
|
||||
print(f"login failed: {login.status_code} {login.text[:200]}", flush=True)
|
||||
return 1
|
||||
client.headers["X-CSRF-Token"] = login.json()["csrf"]
|
||||
|
||||
hosts = [t["name"] for t in client.get("/api/config").json()["system"]["targets"]]
|
||||
host_query = "".join(f"hosts={h}&" for h in hosts) or "hosts=driveripper&"
|
||||
metrics_query = "".join(f"metrics={m}&" for m in METRICS)
|
||||
paths = {
|
||||
"health": "/api/health",
|
||||
"summary": "/api/summary",
|
||||
"metrics": "/api/metrics?source=host",
|
||||
"history_90d": f"/api/system/history?{host_query}{metrics_query}range=90d",
|
||||
}
|
||||
|
||||
db = sqlite3.connect(f"file:{DB_PATH}?mode=ro", uri=True, timeout=30.0)
|
||||
started_wall = time.time()
|
||||
with log_path.open("a", buffering=1) as log:
|
||||
log.write(f"# watch start {time.strftime('%Y-%m-%dT%H:%M:%S')} "
|
||||
f"budget={PAGE_BUDGET_MS:.0f} ms interval={INTERVAL_SECONDS:.0f} s "
|
||||
f"duration={DURATION_SECONDS:.0f} s hosts={hosts}\n")
|
||||
while time.perf_counter() - started < DURATION_SECONDS:
|
||||
row: dict[str, float | int | str] = {
|
||||
"t": round(time.time(), 1),
|
||||
"elapsed_s": round(time.perf_counter() - started, 1),
|
||||
}
|
||||
for name, path in paths.items():
|
||||
ms, code = timed(client, path)
|
||||
row[f"{name}_ms"] = round(ms, 1)
|
||||
row[f"{name}_code"] = code
|
||||
wal = DB_PATH.with_name(DB_PATH.name + "-wal")
|
||||
row["wal_mb"] = round((wal.stat().st_size if wal.exists() else 0) / 1e6, 2)
|
||||
row["db_mb"] = round(DB_PATH.stat().st_size / 1e6, 1)
|
||||
try:
|
||||
row.update(store_evidence(db, started_wall))
|
||||
except sqlite3.Error as exc:
|
||||
row["store_errors_total"] = -1
|
||||
row["store_errors_in_window"] = -1
|
||||
row["last_store_error"] = f"read failed: {exc}"
|
||||
samples.append(row)
|
||||
worst = max(float(row[f"{n}_ms"]) for n in paths)
|
||||
log.write(
|
||||
f"{row['t']} elapsed={row['elapsed_s']:7.1f}s "
|
||||
+ " ".join(f"{n}={row[f'{n}_ms']:7.1f}ms" for n in paths)
|
||||
+ f" worst={worst:7.1f}ms wal={row['wal_mb']}MB db={row['db_mb']}MB"
|
||||
+ f" store-error(total={row['store_errors_total']},"
|
||||
f" in-window={row['store_errors_in_window']})\n"
|
||||
)
|
||||
time.sleep(INTERVAL_SECONDS)
|
||||
|
||||
# ---- summary
|
||||
def stat(name: str, fn) -> float:
|
||||
values = [float(s[f"{name}_ms"]) for s in samples]
|
||||
return round(fn(values), 1)
|
||||
|
||||
summary: dict[str, object] = {
|
||||
"started": time.strftime("%Y-%m-%dT%H:%M:%S", time.localtime(started_wall)),
|
||||
"duration_seconds": round(time.perf_counter() - started, 1),
|
||||
"passes": len(samples),
|
||||
"db": str(DB_PATH),
|
||||
"db_mb_first": samples[0]["db_mb"],
|
||||
"db_mb_last": samples[-1]["db_mb"],
|
||||
"wal_mb_first": samples[0]["wal_mb"],
|
||||
"wal_mb_max": max(float(s["wal_mb"]) for s in samples),
|
||||
"wal_mb_last": samples[-1]["wal_mb"],
|
||||
"store_errors_total": samples[-1]["store_errors_total"],
|
||||
"store_errors_in_window": samples[-1]["store_errors_in_window"],
|
||||
"last_store_error": samples[-1]["last_store_error"],
|
||||
"paths": {
|
||||
name: {
|
||||
"median_ms": stat(name, statistics.median),
|
||||
"p95_ms": stat(name, lambda v: sorted(v)[max(0, int(len(v) * 0.95) - 1)]),
|
||||
"max_ms": stat(name, max),
|
||||
}
|
||||
for name in paths
|
||||
},
|
||||
"budget_ms": PAGE_BUDGET_MS,
|
||||
"passes_over_budget": sum(
|
||||
1 for s in samples if max(float(s[f"{n}_ms"]) for n in paths) > PAGE_BUDGET_MS
|
||||
),
|
||||
"non_200_responses": sum(
|
||||
1 for s in samples for n in paths if int(s[f"{n}_code"]) not in (200, 404)
|
||||
),
|
||||
}
|
||||
json_path.write_text(json.dumps({"summary": summary, "samples": samples}, indent=2) + "\n")
|
||||
with log_path.open("a", buffering=1) as log:
|
||||
log.write("# summary\n")
|
||||
for name, values in summary["paths"].items(): # type: ignore[index]
|
||||
log.write(f"# {name:12s} median={values['median_ms']:8.1f} ms "
|
||||
f"p95={values['p95_ms']:8.1f} ms max={values['max_ms']:8.1f} ms\n")
|
||||
log.write(f"# store-error rows in window: {summary['store_errors_in_window']} "
|
||||
f"(total {summary['store_errors_total']})\n")
|
||||
log.write(f"# WAL first/max/last MB: {summary['wal_mb_first']} / "
|
||||
f"{summary['wal_mb_max']} / {summary['wal_mb_last']}\n")
|
||||
log.write(f"# passes over the {PAGE_BUDGET_MS:.0f} ms budget: "
|
||||
f"{summary['passes_over_budget']} of {summary['passes']}\n")
|
||||
|
||||
header = ["t"] + [f"{n}_ms" for n in paths] + ["wal_mb", "db_mb", "store_errors_in_window"]
|
||||
with tsv_path.open("w") as tsv:
|
||||
tsv.write("\t".join(header) + "\n")
|
||||
for sample in samples:
|
||||
tsv.write("\t".join(str(sample.get(key, "")) for key in header) + "\n")
|
||||
print(json.dumps(summary, indent=2))
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
+35
-11
@@ -16,6 +16,7 @@ from __future__ import annotations
|
||||
import logging
|
||||
import math
|
||||
import time
|
||||
from collections.abc import Sequence
|
||||
from dataclasses import dataclass, field, replace
|
||||
from typing import Any
|
||||
|
||||
@@ -109,9 +110,9 @@ def parse_range(value: str) -> float:
|
||||
raise HTTPException(400, f"unknown range {value!r}; try 5m 1h 24h 7d 30d")
|
||||
|
||||
|
||||
def _history(store: SampleStore, source: str, host: str, metric: str, span: float,
|
||||
points: int) -> dict[str, Any]:
|
||||
series = store.history(source, host, metric, span=span, max_points=points)
|
||||
def _series_dict(source: str, host: str, metric: str, span: float,
|
||||
series: list[tuple[float, float]]) -> dict[str, Any]:
|
||||
"""The one series shape every `/api/*/history` route has always answered with."""
|
||||
return {
|
||||
"source": source,
|
||||
"host": host,
|
||||
@@ -121,6 +122,30 @@ def _history(store: SampleStore, source: str, host: str, metric: str, span: floa
|
||||
}
|
||||
|
||||
|
||||
def _history(store: SampleStore, source: str, host: str, metric: str, span: float,
|
||||
points: int) -> dict[str, Any]:
|
||||
return _series_dict(source, host, metric, span,
|
||||
store.history(source, host, metric, span=span, max_points=points))
|
||||
|
||||
|
||||
def _history_batch(store: SampleStore, source: str, hosts: Sequence[str], metrics: Sequence[str],
|
||||
span: float, points: int) -> list[dict[str, Any]]:
|
||||
"""N hosts x M metrics through the store's one batched door, host-major and in request order.
|
||||
|
||||
Every `/api/*/history` route that used to loop `store.history()` per pair calls this instead.
|
||||
The list it returns is identical, entry for entry, to what that loop built (same series, same
|
||||
order, same `points`); what changed is the cost — the resolution choice for the whole batch is
|
||||
one capped statement rather than three full `COUNT`s per pair (`store.history_many`).
|
||||
"""
|
||||
wanted = [(host, metric) for host in hosts for metric in metrics]
|
||||
if not wanted:
|
||||
return []
|
||||
answers = store.history_many(source, [host for host, _ in wanted],
|
||||
[metric for _, metric in wanted], span=span, max_points=points)
|
||||
return [_series_dict(source, host, metric, span, answers.get((host, metric), []))
|
||||
for host, metric in wanted]
|
||||
|
||||
|
||||
def _host_severity(host: dict[str, Any]) -> dict[str, str | None]:
|
||||
"""Verdicts for the overview's fleet-table columns, from the one threshold table.
|
||||
|
||||
@@ -722,8 +747,7 @@ def system_router() -> APIRouter:
|
||||
wanted_metrics = [item.strip() for item in metrics.split(",") if item.strip()]
|
||||
wanted_hosts = [item.strip() for item in hosts.split(",") if item.strip()] or [
|
||||
target.name for target in ctx.settings.host_targets]
|
||||
series = [_history(ctx.store, "host", host, metric, span, points)
|
||||
for host in wanted_hosts for metric in wanted_metrics]
|
||||
series = _history_batch(ctx.store, "host", wanted_hosts, wanted_metrics, span, points)
|
||||
return {"hosts": wanted_hosts, "range_seconds": span, "series": series}
|
||||
|
||||
return router
|
||||
@@ -816,8 +840,8 @@ def disks_router() -> APIRouter:
|
||||
wanted_hosts = [item.strip() for item in hosts.split(",") if item.strip()] or [
|
||||
target.name for target in ctx.settings.host_targets]
|
||||
return {"hosts": wanted_hosts, "range_seconds": span,
|
||||
"series": [_history(ctx.store, DISKS_SOURCE, host, metric, span, points)
|
||||
for host in wanted_hosts for metric in wanted_metrics]}
|
||||
"series": _history_batch(ctx.store, DISKS_SOURCE, wanted_hosts, wanted_metrics,
|
||||
span, points)}
|
||||
|
||||
return router
|
||||
|
||||
@@ -854,8 +878,9 @@ def llm_router() -> APIRouter:
|
||||
wanted_engines = [item.strip() for item in engines.split(",") if item.strip()] or [
|
||||
name for name, url in (("gufo", ctx.settings.gufo_url), ("vllm", ctx.settings.vllm_url)) if url]
|
||||
return {"engines": wanted_engines, "range_seconds": span,
|
||||
"series": [_history(ctx.store, "llm", engine, f"llm.{metric}", span, points)
|
||||
for engine in wanted_engines for metric in wanted_metrics]}
|
||||
"series": _history_batch(ctx.store, "llm", wanted_engines,
|
||||
[f"llm.{metric}" for metric in wanted_metrics],
|
||||
span, points)}
|
||||
|
||||
return router
|
||||
|
||||
@@ -893,8 +918,7 @@ def ups_router() -> APIRouter:
|
||||
wanted_ups = [item.strip() for item in ups.split(",") if item.strip()] or [
|
||||
card["name"] for card in ((job.state.payload if job else None) or {}).get("ups", [])]
|
||||
return {"ups": wanted_ups, "range_seconds": span,
|
||||
"series": [_history(ctx.store, "ups", name, metric, span, points)
|
||||
for name in wanted_ups for metric in wanted_metrics]}
|
||||
"series": _history_batch(ctx.store, "ups", wanted_ups, wanted_metrics, span, points)}
|
||||
|
||||
@router.get("/commands")
|
||||
async def commands(ctx: AppContext = Depends(get_context)) -> dict[str, Any]:
|
||||
|
||||
+481
-54
@@ -28,11 +28,30 @@ first start (phase 11). Retention never touches it; it holds a handful of rows.
|
||||
|
||||
The connection is guarded by a lock and every method is synchronous: async callers wrap
|
||||
them in `anyio.to_thread.run_sync`, sync FastAPI handlers run in the threadpool already.
|
||||
|
||||
Phase 14 task 02 added the two short-TTL read caches (`series_names`, `stats`) and the
|
||||
capped per-resolution count. Both exist for the same reason: a per-request answer that costs
|
||||
a whole-table pass is a page load, and on a 600 MB store the two answers that used to be
|
||||
scanned on every request cost ~1.3 s each (`data/logs/NOTES-store-perf.md`), serialized
|
||||
behind the very mutex the poller writes under. A cached answer is a *bounded* answer; the
|
||||
TTL is how stale it may get before it is re-derived, and a write invalidates the one whose
|
||||
staleness would be a lie about the data (`series_names`) rather than about a counter.
|
||||
|
||||
Phase 14 task 03 bounded the *write* side. Each rollup step now scans only the rows that
|
||||
became due since the previous pass (`maintain()` reads `last_maintain` first and windows each
|
||||
step to it), because the whole-table rollup was a 5.4 s write transaction that held the WAL
|
||||
write lock past a poller's `busy_timeout` — the `store-error: database is locked` line of
|
||||
AGENTS.md known gap 4. Two properties make the window a cost bound rather than a data-bound
|
||||
change, and both are pinned in `tests/test_store.py`: its lower bound reaches **past** the
|
||||
previous pass (one bucket plus slack, so a missed interval is a catch-up scan and not a gap), and
|
||||
it is snapped down to a **bucket boundary** (so a bucket any pass re-reads is a bucket it reads
|
||||
whole, which is what keeps the averaged values identical to a whole-table rollup's).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
import math
|
||||
import sqlite3
|
||||
import threading
|
||||
@@ -42,9 +61,94 @@ from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
|
||||
RES_RAW, RES_MIN, RES_HOUR = "raw", "1m", "1h"
|
||||
#: The three resolutions, finest first. Every resolution decision in the app walks this tuple,
|
||||
#: so `pick_res` and `pick_res_many` cannot disagree about which one is finest.
|
||||
RESOLUTIONS: tuple[str, ...] = (RES_RAW, RES_MIN, RES_HOUR)
|
||||
MINUTE = 60.0
|
||||
HOUR = 3600.0
|
||||
|
||||
#: How long `series_names()` may answer from its cache (phase 14 task 02). The set of distinct
|
||||
#: metric names changes when a source starts or stops publishing one — minutes, not seconds —
|
||||
#: and a write that can introduce one invalidates it anyway (task 04: *that* write, not every
|
||||
#: sample the hub stores), so the TTL is only the ceiling on a miss we did not notice. Instance
|
||||
#: attribute `series_ttl_seconds` overrides it (tests, probes).
|
||||
SERIES_CACHE_TTL_SECONDS = 60.0
|
||||
#: How long `stats()` may answer from its cache. `/api/health` is polled by every open tab every
|
||||
#: 15 s and by the container healthcheck, and its counts are operator trivia, not a control
|
||||
#: surface: 30 s of staleness buys a 1.3 s full-table scan off every poll but one.
|
||||
STATS_CACHE_TTL_SECONDS = 30.0
|
||||
#: How many cached `(source, host)` answers `series_names()` keeps before starting over. The
|
||||
#: endpoint takes both as free-text query parameters, so the key space is the caller's; the
|
||||
#: cache is bounded rather than grown by whatever a stranger types.
|
||||
SERIES_CACHE_MAX_ENTRIES = 256
|
||||
#: How many `(source, host, metric)` triples the store remembers having written, which is what
|
||||
#: lets `add_many` leave the series cache alone (see `_note_series_write`). The fleet's whole
|
||||
#: name space is a few hundred triples, so this is a generous ceiling on a bounded namespace,
|
||||
#: not a growing structure: past it the store goes back to invalidating on every write, because
|
||||
#: memory tracked on a namespace a caller can influence needs a cap either way.
|
||||
SERIES_TRACKED_SERIES_MAX = 8192
|
||||
#: Branches per compound count statement. SQLite's `SQLITE_LIMIT_COMPOUND_SELECT` defaults to
|
||||
#: 500 terms and the variable limit to 32 766; 100 keeps one statement per 33 series while
|
||||
#: leaving 5x headroom on both limits, and it is where the statement-preparation cost stops
|
||||
#: being noise (measured on the seeded store: 81 branches ≈ 0.5 ms of preparation).
|
||||
COMPOUND_CHUNK = 100
|
||||
|
||||
#: Slack on a bucket's completeness, and on how far behind the previous pass a rollup looks.
|
||||
#: A minute bucket is only rolled once its whole 60 s is in the past *plus* this slack, so a
|
||||
#: rollup never averages a partial minute and gets rewritten by the next pass; the same 60 s on
|
||||
#: the window's lower bound is what makes a missed interval a catch-up scan rather than a gap.
|
||||
MAINTAIN_SLACK_SECONDS = 60.0
|
||||
|
||||
#: The `meta` key holding the previous `maintain()` pass's clock. Written by `maintain()` and read
|
||||
#: by it on the next run to window the rollups; `stats()` surfaces it in `/api/health` (uncached)
|
||||
#: precisely so an operator can see it stop moving.
|
||||
LAST_MAINTAIN_KEY = "last_maintain"
|
||||
|
||||
#: WAL auto-checkpoint threshold, in pages. This is SQLite's own default (1 000 pages ≈ 4 MB) and
|
||||
#: is written down so the knob is visible next to the one that actually bites: a WAL that reaches
|
||||
#: 140 MB (AGENTS.md known gap 4) is a checkpoint that keeps losing to the write rate, and
|
||||
#: asking for one more often does not help — a checkpoint cannot run *inside* a long transaction.
|
||||
WAL_AUTOCHECKPOINT_PAGES = 1000
|
||||
#: Cap on the WAL file, in bytes. At a checkpoint a WAL larger than this is truncated to it
|
||||
#: instead of staying at its high-water mark forever, so the file has a ceiling even when a
|
||||
#: reader kept a snapshot open long enough to make it grow. 64 MB sits comfortably above what the
|
||||
#: write path asks for (the whole-retention pass, the biggest thing this does, left a 41 MB WAL on
|
||||
#: the seeded store) and far below the 140 MB gap 4 recorded, so the cap binds on runaways and not
|
||||
#: on maintenance. It is a per-connection pragma, not a migration, so moving it is a one-line
|
||||
#: change (task 04's live watch decides whether to).
|
||||
JOURNAL_SIZE_LIMIT_BYTES = 64 * 1024 * 1024
|
||||
|
||||
#: SQLite's background ANALYZE, deliberately off. It refreshes statistics inside whichever commit
|
||||
#: trips its change counter, which on this app means inside a poller's `add_many` — the write path
|
||||
#: this task exists to shorten. `maintain()` calls `PRAGMA optimize` instead: once a minute, on the
|
||||
#: tick that already owns bookkeeping, outside the write transaction.
|
||||
AUTOMATIC_ANALYZE = 0
|
||||
|
||||
#: One rollup step: average `res_from` rows into `res_to` buckets, complete buckets only.
|
||||
#: Bindings: `(res_to, bucket, bucket, res_from, ceiling, floor, bucket)`. Public because
|
||||
#: `tests/store_perf.py` EXPLAINs the same statement — the plan in the NOTES has to be the plan
|
||||
#: that runs (the same reason `RES_COUNT_BRANCH` is a constant).
|
||||
ROLLUP_INSERT = (
|
||||
"INSERT OR REPLACE INTO sample (source, host, metric, res, ts, value)"
|
||||
" SELECT source, host, metric, ?, CAST(CAST(ts AS INT)/? AS INT)*?, AVG(value)"
|
||||
" FROM sample WHERE res=? AND ts < ? AND ts >= ?"
|
||||
" GROUP BY source, host, metric, CAST(CAST(ts AS INT)/? AS INT)"
|
||||
)
|
||||
|
||||
logger = logging.getLogger("dashboards.store")
|
||||
|
||||
#: One `(host, metric, res)` row of the capped count. Public because `tests/store_perf.py`
|
||||
#: EXPLAINs it: the plan the NOTES record has to be the plan that runs, not a lookalike.
|
||||
#:
|
||||
#: The count is capped by `LIMIT` inside the subquery because the only question ever asked of it
|
||||
#: is "is this <= max_points", and counting 43 200 minute rows to answer "no" is what made a 90 d
|
||||
#: chart request expensive. The three label columns come first so the answer does not depend on
|
||||
#: compound-row ordering.
|
||||
RES_COUNT_BRANCH = (
|
||||
"SELECT ?, ?, ?, COUNT(*) FROM (SELECT 1 FROM sample"
|
||||
" WHERE source=? AND host=? AND metric=? AND res=? AND ts>=? LIMIT ?)"
|
||||
)
|
||||
|
||||
_SCHEMA = """
|
||||
CREATE TABLE IF NOT EXISTS sample (
|
||||
source TEXT NOT NULL,
|
||||
@@ -104,6 +208,47 @@ class Event:
|
||||
}
|
||||
|
||||
|
||||
def _pairs(hosts: Sequence[str], metrics: Sequence[str]) -> list[tuple[str, str]]:
|
||||
"""The host-major series list a request asks for, de-duplicated but otherwise in order.
|
||||
|
||||
`dict.fromkeys` rather than a set: the order is the order the endpoints emit their series in,
|
||||
and a set would shuffle a response that a browser (and a test pin) expects in request order.
|
||||
"""
|
||||
return [(host, metric) for host in dict.fromkeys(hosts) for metric in dict.fromkeys(metrics)]
|
||||
|
||||
|
||||
def _choose_res(counts: dict[str, int], max_points: int) -> str:
|
||||
"""The resolution decision, from one series' capped per-`res` counts.
|
||||
|
||||
Counts are capped at `max_points + 1`, which leaves both predicates below exactly as true as
|
||||
they were against a full `COUNT(*)`: `count` is 0 exactly when the real count is, and
|
||||
`≤ max_points` exactly when the real count is.
|
||||
"""
|
||||
finest_with_data: str | None = None
|
||||
for res in RESOLUTIONS:
|
||||
count = counts.get(res, 0)
|
||||
if not count:
|
||||
continue
|
||||
if finest_with_data is None:
|
||||
# too many points for the budget is fine — history() thins. Returning a
|
||||
# resolution with *no* rows in the window reads as a dead sensor.
|
||||
finest_with_data = res
|
||||
if count <= max_points:
|
||||
return res
|
||||
return finest_with_data or RES_HOUR
|
||||
|
||||
|
||||
def _thin(points: list[tuple[float, float]], max_points: int) -> list[tuple[float, float]]:
|
||||
"""Stride-thin a series to the point budget without ever losing the newest point."""
|
||||
if len(points) <= max_points:
|
||||
return points
|
||||
stride = math.ceil(len(points) / max_points)
|
||||
thinned = points[::stride]
|
||||
if thinned[-1] != points[-1]:
|
||||
thinned.append(points[-1]) # never lose "now" to the thinning
|
||||
return thinned
|
||||
|
||||
|
||||
class SampleStore:
|
||||
def __init__(self, path: str | Path, *, raw_keep_hours: float = 48,
|
||||
minute_keep_days: float = 30, hour_keep_days: float = 400) -> None:
|
||||
@@ -118,8 +263,82 @@ class SampleStore:
|
||||
self._db.execute("PRAGMA journal_mode=WAL")
|
||||
self._db.execute("PRAGMA synchronous=NORMAL") # a lost sample is fine, corruption is not
|
||||
self._db.execute("PRAGMA busy_timeout=5000")
|
||||
# WAL housekeeping (phase 14 task 03): keep trying to checkpoint at the default threshold
|
||||
# and cap the file, so a WAL that does grow — a long-lived reader holding a snapshot open
|
||||
# is enough — stops at the cap at the next checkpoint instead of at 140 MB. Then take
|
||||
# SQLite's statistics upkeep away from the poll path: without statistics the planner reads
|
||||
# the table for every windowed rollup anyway (`SCAN sample`), and with SQLite's own
|
||||
# automatic ANALYZE the refresh lands inside whichever poller's commit trips its counter.
|
||||
# All three are per-connection pragmas, so they are re-asserted on every open, not migrated.
|
||||
self._db.execute(f"PRAGMA wal_autocheckpoint={WAL_AUTOCHECKPOINT_PAGES}")
|
||||
self._db.execute(f"PRAGMA journal_size_limit={JOURNAL_SIZE_LIMIT_BYTES}")
|
||||
self._db.execute(f"PRAGMA automatic_analyze={AUTOMATIC_ANALYZE}")
|
||||
self._db.executescript(_SCHEMA)
|
||||
self._db.commit()
|
||||
# The two read caches (see the TTL constants above). Both are instance attributes so a
|
||||
# test or a probe can force expiry with `store.stats_ttl_seconds = 0` instead of waiting
|
||||
# for a clock or monkeypatching a module-wide `time`.
|
||||
self.series_ttl_seconds = SERIES_CACHE_TTL_SECONDS
|
||||
self.stats_ttl_seconds = STATS_CACHE_TTL_SECONDS
|
||||
# `(argument tuple) -> (generation, monotonic, answer)`. The generation is what makes a
|
||||
# concurrent write safe: a thread that read the table *before* a write and stores its
|
||||
# answer *after* it would otherwise park a stale list under a fresh timestamp.
|
||||
# The `set` beside each cached list is what `add_many` consults to decide whether the
|
||||
# write can change the answer at all (see `_note_series_write`); membership in a list of
|
||||
# every metric name is not a check to make per sample.
|
||||
self._series_cache: dict[tuple[str | None, str | None],
|
||||
tuple[int, float, list[str], set[str]]] = {}
|
||||
self._stats_cache: tuple[float, dict[str, int], int] | None = None
|
||||
self._generation = 0
|
||||
#: `(source, host, metric)` triples this store has written rows for. Tracks *existence*,
|
||||
#: not rows, and is capped (see `SERIES_TRACKED_SERIES_MAX`).
|
||||
self._known_series: set[tuple[str, str, str]] = set()
|
||||
|
||||
# -- the read caches -------------------------------------------------------- #
|
||||
def invalidate_read_caches(self) -> None:
|
||||
"""Drop every cached read answer.
|
||||
|
||||
`add_many` invalidates the name list only when it introduces a name
|
||||
(`_note_series_write`); every write path that can *remove* one invalidates unconditionally.
|
||||
This is the blunt instrument for the callers that need a cold measurement (the phase-14
|
||||
probe) or a test that wants to be sure it is not reading its own previous answer.
|
||||
"""
|
||||
self._generation += 1
|
||||
self._series_cache.clear()
|
||||
self._stats_cache = None
|
||||
|
||||
def _invalidate_series(self) -> None:
|
||||
"""Called from inside a write's own lock: never takes `self._lock` (it is not reentrant)."""
|
||||
self._generation += 1
|
||||
self._series_cache.clear()
|
||||
|
||||
def _note_series_write(self, series: Sequence[tuple[str, str, str]]) -> None:
|
||||
"""Invalidate the series-name cache only for a write that can change it.
|
||||
|
||||
Every sample used to clear it, and the live watch is what caught the consequence: the hub
|
||||
writes every 2-300 s, so on the operator's store `/api/metrics` answered with its full
|
||||
`SELECT DISTINCT` scan on nearly every call (535 ms of store mutex, measured) — the same
|
||||
per-request scan task 02 existed to remove, surviving behind a cache that never served.
|
||||
|
||||
An insert cannot *remove* a name, so the cached answers change only when the write
|
||||
introduces a `(source, host, metric)` this store has never written. A poll of an existing
|
||||
series is therefore free, and a genuinely new name still costs exactly one invalidation,
|
||||
which is what `tests/test_store.py` pins in both directions.
|
||||
|
||||
`delete_series()` and `maintain()` keep the unconditional invalidation they have always
|
||||
had, because a `DELETE` can take the last row of a name and existence-tracking cannot see
|
||||
that. The tracking can also be wrong in the same direction if *another* process deletes
|
||||
rows this one believes exist (two servers on one file, AGENTS.md gap 4's configuration
|
||||
mistake): the cost is that a name reappearing inside one TTL window is not in the list
|
||||
until the window ends. `series_ttl_seconds` is the ceiling on that, and it is the reason
|
||||
the cache has a TTL rather than trusting the tracking alone.
|
||||
"""
|
||||
if any(key not in self._known_series for key in series):
|
||||
self._invalidate_series()
|
||||
if len(self._known_series) + len(series) <= SERIES_TRACKED_SERIES_MAX:
|
||||
self._known_series.update(series)
|
||||
else: # cap reached: forget the tracking, so from the next write on every one invalidates
|
||||
self._known_series.clear()
|
||||
|
||||
def close(self) -> None:
|
||||
with self._lock:
|
||||
@@ -144,6 +363,7 @@ class SampleStore:
|
||||
return 0
|
||||
with self._lock, self._db:
|
||||
self._db.executemany("INSERT OR REPLACE INTO sample VALUES (?,?,?,?,?,?)", rows)
|
||||
self._note_series_write([(row[0], row[1], row[2]) for row in rows])
|
||||
return len(rows)
|
||||
|
||||
def log_event(self, event: Event) -> None:
|
||||
@@ -182,48 +402,126 @@ class SampleStore:
|
||||
return cur.fetchall() # type: ignore[no-any-return]
|
||||
|
||||
def pick_res(self, source: str, host: str, metric: str, span: float, max_points: int) -> str:
|
||||
"""Finest resolution that still fits in `max_points` for this window."""
|
||||
now = time.time()
|
||||
start = now - span
|
||||
finest_with_data: str | None = None
|
||||
for res in (RES_RAW, RES_MIN, RES_HOUR):
|
||||
with self._lock:
|
||||
(count,) = self._db.execute(
|
||||
"SELECT COUNT(*) FROM sample WHERE source=? AND host=? AND metric=? AND res=? AND ts>=?",
|
||||
(source, host, metric, res, start),
|
||||
).fetchone()
|
||||
if not count:
|
||||
continue
|
||||
if finest_with_data is None:
|
||||
# too many points for the budget is fine — history() thins. Returning a
|
||||
# resolution with *no* rows in the window reads as a dead sensor.
|
||||
finest_with_data = res
|
||||
if count <= max_points:
|
||||
return res
|
||||
return finest_with_data or RES_HOUR
|
||||
"""Finest resolution that still fits in `max_points` for this window.
|
||||
|
||||
One statement, not three: the counts for all three resolutions come back from a single
|
||||
compound query whose per-`res` `LIMIT` caps the work (phase 14 task 02). Capping is exact,
|
||||
not an approximation, because the only thing the decision asks of a count is whether it is
|
||||
`0`, `≤ max_points`, or neither — so counting 43 200 minute rows to answer "neither" was
|
||||
pure waste. Measured on the seeded store: 90 d, one series, 2.97 ms → 0.10 ms; the
|
||||
`tests/test_store.py` pin is that the *answer* is unchanged for every window.
|
||||
"""
|
||||
counts = self._counts_by_res(source, ((host, metric),), span, max_points)
|
||||
return _choose_res(counts[(host, metric)], max_points)
|
||||
|
||||
def pick_res_many(self, source: str, hosts: Sequence[str], metrics: Sequence[str],
|
||||
span: float, max_points: int) -> dict[tuple[str, str], str]:
|
||||
"""`pick_res` for every `(host, metric)` pair, in one grouped round trip per 33 series.
|
||||
|
||||
The batched door the N x M history endpoints go through. The decision is made by the same
|
||||
`_choose_res` on the same capped counts, so a pair's answer here is identical to the answer
|
||||
`pick_res` gives for it alone — that equality is a test, not a hope.
|
||||
"""
|
||||
counts = self._counts_by_res(source, _pairs(hosts, metrics), span, max_points)
|
||||
return {pair: _choose_res(seen, max_points) for pair, seen in counts.items()}
|
||||
|
||||
def history(self, source: str, host: str, metric: str, *, span: float = 3600.0,
|
||||
max_points: int = 600) -> list[tuple[float, float]]:
|
||||
"""Windowed series, thinned to `max_points` so the browser never gets 100k rows."""
|
||||
res = self.pick_res(source, host, metric, span, max_points)
|
||||
window_start = time.time() - span
|
||||
points = self.query(source, host, metric, res=res, start=window_start,
|
||||
return self.history_many(source, [host], [metric], span=span,
|
||||
max_points=max_points)[(host, metric)]
|
||||
|
||||
def history_many(self, source: str, hosts: Sequence[str], metrics: Sequence[str], *,
|
||||
span: float = 3600.0,
|
||||
max_points: int = 600) -> dict[tuple[str, str], list[tuple[float, float]]]:
|
||||
"""`history` for many series at once, keyed by `(host, metric)`.
|
||||
|
||||
Same per-series answer as `history()` — resolution choice, tail semantics, raw fallback,
|
||||
thinning, and the point list that keeps "now" — with the resolution choice made in one
|
||||
grouped statement instead of three counts per series.
|
||||
|
||||
The *rows* are still fetched per series, deliberately and measuredly: the tail select is
|
||||
already a covering-index seek that reads at most `max_points x 4` rows, and every shape
|
||||
that folds the whole batch into one statement (a `ROW_NUMBER()` window, a compound of the
|
||||
per-series tails) was slower on the seeded store — 73 ms and 38 ms against 29 ms for 27
|
||||
series at 90 d — because it numbers or sorts rows the seek never has to touch. Batching
|
||||
the *decision* was where the cost was; batching the fetch is a round trip traded for work.
|
||||
"""
|
||||
pairs = _pairs(hosts, metrics)
|
||||
if not pairs:
|
||||
return {}
|
||||
now = time.time()
|
||||
window_start = now - span
|
||||
counts = self._counts_by_res(source, pairs, span, max_points)
|
||||
out: dict[tuple[str, str], list[tuple[float, float]]] = {}
|
||||
for host, metric in pairs:
|
||||
res = _choose_res(counts[(host, metric)], max_points)
|
||||
points = self.query(source, host, metric, res=res, start=window_start, end=now,
|
||||
limit=max_points * 4, tail=True)
|
||||
if not points and res != RES_RAW:
|
||||
# Rollups lag behind (they only cover buckets that are fully in the past, and a
|
||||
# dashboard that has been up for ten minutes has none at all). Falling back to
|
||||
# raw turns "no samples yet" into a real chart instead of a lie by omission.
|
||||
points = self.query(source, host, metric, res=RES_RAW, start=window_start,
|
||||
limit=max_points * 4, tail=True)
|
||||
if len(points) <= max_points:
|
||||
return points
|
||||
stride = math.ceil(len(points) / max_points)
|
||||
thinned = points[::stride]
|
||||
if thinned[-1] != points[-1]:
|
||||
thinned.append(points[-1]) # never lose "now" to the thinning
|
||||
return thinned
|
||||
end=now, limit=max_points * 4, tail=True)
|
||||
out[(host, metric)] = _thin(points, max_points)
|
||||
return out
|
||||
|
||||
def _counts_by_res(self, source: str, pairs: Sequence[tuple[str, str]], span: float,
|
||||
max_points: int) -> dict[tuple[str, str], dict[str, int]]:
|
||||
"""Capped per-resolution row counts for many series, in as few statements as possible.
|
||||
|
||||
One compound statement of `(host, metric, res)` branches, chunked at `COMPOUND_CHUNK`
|
||||
terms, each branch an index seek on the full primary-key prefix (`tests/test_store.py`
|
||||
pins the plan) that stops after `max_points + 1` rows. Nothing here scans a table: the
|
||||
work is proportional to `series x max_points`, never to rows of history.
|
||||
"""
|
||||
start = time.time() - span
|
||||
cap = max_points + 1
|
||||
out: dict[tuple[str, str], dict[str, int]] = {pair: {} for pair in pairs}
|
||||
branches: list[str] = []
|
||||
args: list[object] = []
|
||||
|
||||
def flush() -> None:
|
||||
if not branches:
|
||||
return
|
||||
with self._lock:
|
||||
rows = self._db.execute(" UNION ALL ".join(branches), tuple(args)).fetchall()
|
||||
for host, metric, res, count in rows:
|
||||
out[(host, metric)][res] = int(count)
|
||||
branches.clear()
|
||||
args.clear()
|
||||
|
||||
for host, metric in pairs:
|
||||
for res in RESOLUTIONS:
|
||||
branches.append(RES_COUNT_BRANCH)
|
||||
args += [host, metric, res, source, host, metric, res, start, cap]
|
||||
if len(branches) >= COMPOUND_CHUNK:
|
||||
flush()
|
||||
flush()
|
||||
return out
|
||||
|
||||
def series_names(self, source: str | None = None, host: str | None = None) -> list[str]:
|
||||
"""Distinct metric names, from a cache that only a write which can change them ends.
|
||||
|
||||
The uncached answer is `SELECT DISTINCT metric` over the table — 1.4 s of `SCAN sample`
|
||||
on a production-sized store, on a route the browser polls alongside the rest. The set of
|
||||
names is also the slowest-moving fact in the database (it changes when a source gains a
|
||||
metric, i.e. on a deploy), which is what a 60 s TTL plus write invalidation is sized for.
|
||||
"""
|
||||
key = (source, host)
|
||||
cached = self._series_cache.get(key)
|
||||
generation, built_at = self._generation, time.monotonic()
|
||||
if cached is not None and cached[0] == generation and built_at - cached[1] < self.series_ttl_seconds:
|
||||
return list(cached[2])
|
||||
names = self._scan_series_names(source, host)
|
||||
if len(self._series_cache) >= SERIES_CACHE_MAX_ENTRIES:
|
||||
self._series_cache.clear()
|
||||
self._series_cache[key] = (generation, built_at, names, set(names))
|
||||
return list(names)
|
||||
|
||||
def _scan_series_names(self, source: str | None, host: str | None) -> list[str]:
|
||||
"""The uncached answer behind `series_names()`: one pass, no cache, no TTL."""
|
||||
sql = "SELECT DISTINCT metric FROM sample"
|
||||
args: list[object] = []
|
||||
clauses = []
|
||||
@@ -304,48 +602,166 @@ class SampleStore:
|
||||
return self._db.execute("DELETE FROM meta WHERE key=?", (key,)).rowcount
|
||||
|
||||
# -- maintenance ----------------------------------------------------------- #
|
||||
def _previous_maintain(self) -> float | None:
|
||||
"""The clock of the last `maintain()` pass, or `None` for "no idea — scan the retention".
|
||||
|
||||
Missing is the normal case on a fresh database. A row that is not a number should not
|
||||
exist (`maintain()` is the only writer) and is treated as missing rather than fatal, with
|
||||
one line said about it: the expensive answer is the safe one here, because a rollup that
|
||||
re-runs over buckets it already rolled rewrites identical rows. Never interpolate the
|
||||
value — it is data the operator did not write, and this app's logs end up in an `event`
|
||||
table a browser reads.
|
||||
"""
|
||||
raw = self.get_meta(LAST_MAINTAIN_KEY)
|
||||
if raw is None:
|
||||
return None
|
||||
try:
|
||||
return float(raw)
|
||||
except ValueError:
|
||||
logger.warning("meta row '%s' is not a number; the next rollup scans the full "
|
||||
"retention window", LAST_MAINTAIN_KEY)
|
||||
return None
|
||||
|
||||
def _rollup_floor(self, since: float | None, retention_start: float, bucket: float) -> float:
|
||||
"""The oldest row a rollup step has to read, snapped down to a bucket boundary.
|
||||
|
||||
With no previous pass that is the whole retention window, exactly as the step always
|
||||
cost. With one, it is the window that became due since: `since - bucket - slack`, one
|
||||
bucket plus the completeness slack behind the previous pass's clock. That overlap is what
|
||||
turns "the app was down for ten minutes" into a ten-minute catch-up scan instead of nine
|
||||
minutes of silently missing buckets.
|
||||
|
||||
The snap down to a bucket boundary is not cosmetic, it is what makes the window safe. The
|
||||
step's ceiling is a *row* bound, so the newest bucket it touches is usually a partial one
|
||||
that a later pass re-rolls as more of its rows land — that progressive correction is how
|
||||
the whole-table version always got the right average. Re-reading a bucket from anywhere
|
||||
other than its own start therefore overwrites a fuller average with a thinner one: with a
|
||||
floor of exactly `since - bucket - slack` a minute bucket split by the boundary ends up as
|
||||
the mean of its own tail rows (`tests/test_store.py` pins both numbers). Aligning the floor
|
||||
down means every bucket this pass reads again is one it reads *whole*, so each re-roll can
|
||||
only add rows to it — which is the same guarantee the unwindowed scan had.
|
||||
"""
|
||||
if since is None:
|
||||
return retention_start
|
||||
aligned = math.floor((since - bucket - MAINTAIN_SLACK_SECONDS) / bucket) * bucket
|
||||
return max(retention_start, aligned)
|
||||
|
||||
def maintain(self, now: float | None = None) -> dict[str, int]:
|
||||
"""Roll up and prune. Safe to call every minute; each step is idempotent.
|
||||
"""Roll up and prune, windowed to what became due since the last pass. Idempotent.
|
||||
|
||||
A bucket is only rolled once its whole window is in the past (+60s slack), so a
|
||||
rollup never averages a partial minute and then gets rewritten by a later one.
|
||||
|
||||
Both rollup steps were `INSERT … SELECT … GROUP BY` over everything their resolution
|
||||
retains, which on a store with 30 days of 1-minute rows meant a 3.97 M-row scan inside the
|
||||
one write transaction (task 01's measurement: 5.4 s, and 5 of 9 concurrent poller inserts
|
||||
then outlived `busy_timeout=5000` and surfaced as `store-error`). Windowing each step to
|
||||
`last_maintain` is what makes the transaction short: a run once a minute aggregates the
|
||||
buckets that filled in the last minute, and the first run after a restart — or after a
|
||||
night with the app stopped — scans back to the previous pass instead of back to retention.
|
||||
|
||||
What a window costs, stated plainly: a raw row that lands *behind* the window (a
|
||||
multi-minute poller stall, or a host whose clock put the sample far in the past) is not
|
||||
re-read by the next pass, so the bucket it belongs to keeps the average the earlier pass
|
||||
computed. The poller stamps samples with its own wall clock, so reaching that case costs a
|
||||
stall measured in minutes; re-scanning 30 days on every tick to cover it is the thing this
|
||||
task exists to stop. `_rollup_floor` explains the bucket alignment that keeps every bucket
|
||||
inside the window honest, and
|
||||
`test_windowed_rollups_answer_exactly_what_one_full_window_rollup_answers` is the proof.
|
||||
"""
|
||||
now = time.time() if now is None else now
|
||||
# Outside the transaction and before the lock below: `get_meta` takes `self._lock` and
|
||||
# `threading.Lock` is not reentrant (the same reason `stats()` reads meta from outside).
|
||||
since = self._previous_maintain()
|
||||
minute_ceiling = now - MINUTE - MAINTAIN_SLACK_SECONDS
|
||||
hour_ceiling = now - HOUR - MAINTAIN_SLACK_SECONDS
|
||||
counts = {"to_minute": 0, "to_hour": 0, "pruned": 0}
|
||||
with self._lock, self._db:
|
||||
counts["to_minute"] = self._db.execute(
|
||||
"INSERT OR REPLACE INTO sample (source, host, metric, res, ts, value)"
|
||||
" SELECT source, host, metric, ?, CAST(CAST(ts AS INT)/? AS INT)*?, AVG(value)"
|
||||
" FROM sample WHERE res=? AND ts < ? AND ts >= ?"
|
||||
" GROUP BY source, host, metric, CAST(CAST(ts AS INT)/? AS INT)",
|
||||
(RES_MIN, MINUTE, MINUTE, RES_RAW, now - MINUTE - 60,
|
||||
now - self.raw_keep_seconds, MINUTE),
|
||||
ROLLUP_INSERT,
|
||||
(RES_MIN, MINUTE, MINUTE, RES_RAW, minute_ceiling,
|
||||
self._rollup_floor(since, now - self.raw_keep_seconds, MINUTE), MINUTE),
|
||||
).rowcount
|
||||
counts["to_hour"] = self._db.execute(
|
||||
"INSERT OR REPLACE INTO sample (source, host, metric, res, ts, value)"
|
||||
" SELECT source, host, metric, ?, CAST(CAST(ts AS INT)/? AS INT)*?, AVG(value)"
|
||||
" FROM sample WHERE res=? AND ts < ? AND ts >= ?"
|
||||
" GROUP BY source, host, metric, CAST(CAST(ts AS INT)/? AS INT)",
|
||||
(RES_HOUR, HOUR, HOUR, RES_MIN, now - HOUR - 60,
|
||||
now - self.minute_keep_seconds, HOUR),
|
||||
ROLLUP_INSERT,
|
||||
(RES_HOUR, HOUR, HOUR, RES_MIN, hour_ceiling,
|
||||
self._rollup_floor(since, now - self.minute_keep_seconds, HOUR), HOUR),
|
||||
).rowcount
|
||||
counts["pruned"] = self._db.execute(
|
||||
"DELETE FROM sample WHERE (res=? AND ts < ?) OR (res=? AND ts < ?) OR (res=? AND ts < ?)",
|
||||
(RES_RAW, now - self.raw_keep_seconds,
|
||||
RES_MIN, now - self.minute_keep_seconds,
|
||||
RES_HOUR, now - self.hour_keep_seconds),
|
||||
).rowcount
|
||||
self._db.execute("INSERT OR REPLACE INTO meta (key, value) VALUES ('last_maintain', ?)", (str(now),))
|
||||
# The prune keeps its full retention bound on purpose. A rollup may window because it
|
||||
# only ever *adds* rows and re-adding one is a rewrite; a prune must be able to reach
|
||||
# anything that expired, whenever it expired, or gap 9's "the file stays bounded" stops
|
||||
# being true. It is a bounded delete per resolution, not a GROUP BY.
|
||||
#
|
||||
# One DELETE per resolution, not one `OR` of the three, because the composition is the
|
||||
# part the planner cannot use. `WHERE (res=? AND ts<?) OR (res=? AND ts<?) OR (res=?
|
||||
# AND ts<?)` reads every row of the table to delete none of them — 554 ms of the 562 ms
|
||||
# a steady `maintain()` cost on the seeded 6.4 M-row store (`SCAN sample`, recorded in
|
||||
# `data/logs/NOTES-store-perf.md`) — while each half of that same predicate is a seek on
|
||||
# the primary-key index at a third of a millisecond. Same three bounds, same rows gone.
|
||||
pruned = 0
|
||||
for res, keep_seconds in ((RES_RAW, self.raw_keep_seconds),
|
||||
(RES_MIN, self.minute_keep_seconds),
|
||||
(RES_HOUR, self.hour_keep_seconds)):
|
||||
pruned += max(self._db.execute("DELETE FROM sample WHERE res = ? AND ts < ?",
|
||||
(res, now - keep_seconds)).rowcount, 0)
|
||||
counts["pruned"] = pruned
|
||||
self._db.execute("INSERT OR REPLACE INTO meta (key, value) VALUES (?, ?)",
|
||||
(LAST_MAINTAIN_KEY, str(now)))
|
||||
# Maintenance writes rollup rows and deletes pruned ones, so it can both add and
|
||||
# remove the last row of a metric name. The counts stay behind `stats()`'s TTL (they
|
||||
# are an operator-facing total, and `maintain()` is the thing that would have to
|
||||
# re-derive them at the same cost); the name set is invalidated, because a chart that
|
||||
# asks for a name the pruner just deleted has to be told, not served yesterday's list.
|
||||
self._invalidate_series()
|
||||
self._optimize()
|
||||
return counts
|
||||
|
||||
def _optimize(self) -> None:
|
||||
"""`PRAGMA optimize`, best effort, outside the write transaction.
|
||||
|
||||
This is what makes the windowing measurable. Without query statistics SQLite cannot tell a
|
||||
two-minute window from a whole-retention scan and picks the scan for both — the windowed
|
||||
rollup cost 328 ms and the prune 554 ms with `SCAN sample` either way, and 1 ms and 0 ms
|
||||
with `SEARCH sample USING INDEX sqlite_autoindex_sample_1` after one `optimize`. Statistics
|
||||
are the other half of "cheaper for the same rows", not an optional extra.
|
||||
|
||||
SQLite's own automatic ANALYZE is switched off in `__init__`, so this is the only place
|
||||
statistics are refreshed: automatic analyse runs inside whichever poll's commit trips its
|
||||
change counter, which is exactly the poll path this task is shortening. Here it is once a
|
||||
minute, on the tick that already owns maintenance — a sampled full analyse the first time
|
||||
(~100 ms on the seeded store) and 0.0 ms on every tick after, because SQLite only re-samples
|
||||
tables whose change counter moved. A failure is logged and dropped: stale statistics are a
|
||||
slower query, and `store-error` on a maintenance tick because of a planner hint would be a
|
||||
worse report to the operator.
|
||||
"""
|
||||
try:
|
||||
with self._lock:
|
||||
self._db.execute("PRAGMA optimize")
|
||||
except sqlite3.Error:
|
||||
logger.warning("PRAGMA optimize failed; the query planner keeps its existing statistics")
|
||||
|
||||
def stats(self) -> dict[str, object]:
|
||||
# Read through `get_meta` so there is one path to the table — but *outside* the lock
|
||||
# below, which `get_meta` takes itself (`threading.Lock` is not reentrant).
|
||||
last_raw = self.get_meta("last_maintain")
|
||||
"""Store size and row counts for `/api/health`, with the two COUNT scans cached.
|
||||
|
||||
The payload keys are a contract with deployment (`/api/health` is the one public data
|
||||
route, AGENTS.md invariant 12), so nothing here changed shape — only how often it is
|
||||
derived from the table. `last_maintain` and `size_bytes` are deliberately **not** cached:
|
||||
the first is how an operator tells a stalled rollup loop from a healthy one and costs
|
||||
a three-row lookup, and the second is a `stat()` call, not a query.
|
||||
"""
|
||||
cached = self._stats_cache
|
||||
now = time.monotonic()
|
||||
if cached is not None and now - cached[0] < self.stats_ttl_seconds:
|
||||
by_res, events = dict(cached[1]), cached[2]
|
||||
else:
|
||||
with self._lock:
|
||||
by_res = {row[0]: row[1] for row in self._db.execute(
|
||||
"SELECT res, COUNT(*) FROM sample GROUP BY res")}
|
||||
(events,) = self._db.execute("SELECT COUNT(*) FROM event").fetchone()
|
||||
self._stats_cache = (time.monotonic(), dict(by_res), events)
|
||||
# Read through `get_meta` so there is one path to the table — but *outside* the lock
|
||||
# above, which `get_meta` takes itself (`threading.Lock` is not reentrant).
|
||||
last_raw = self.get_meta(LAST_MAINTAIN_KEY)
|
||||
size = 0
|
||||
if str(self.path) != ":memory:" and self.path.exists():
|
||||
size = self.path.stat().st_size
|
||||
@@ -367,7 +783,18 @@ class SampleStore:
|
||||
sql += " AND metric=?"
|
||||
args.append(metric)
|
||||
with self._lock, self._db:
|
||||
return self._db.execute(sql, args).rowcount
|
||||
deleted = self._db.execute(sql, args).rowcount
|
||||
# Forget the triples this delete emptied, so the next write of one of them counts as
|
||||
# the new series it now is: `add_many` invalidates on a triple it has not seen, and a
|
||||
# deleted-then-recreated series that stayed "known" would be invisible to
|
||||
# `/api/metrics` for a TTL window. One `DELETE` cannot name the triples it emptied,
|
||||
# so this drops whatever the predicate could have covered.
|
||||
self._known_series = {
|
||||
key for key in self._known_series
|
||||
if not (key[0] == source and key[1] == host and (metric is None or key[2] == metric))
|
||||
}
|
||||
self._invalidate_series()
|
||||
return deleted
|
||||
|
||||
|
||||
def samples_from_flat(source: str, host: str, values: dict[str, float | None],
|
||||
|
||||
+59
-4
@@ -27,14 +27,32 @@ def pytest_addoption(parser: pytest.Parser) -> None:
|
||||
default=False,
|
||||
help="run tests marked `live` against the real fleet/engines/upsd (they skip otherwise)",
|
||||
)
|
||||
parser.addoption(
|
||||
"--store-perf",
|
||||
action="store_true",
|
||||
default=False,
|
||||
help="run tests marked `store_perf`: the phase-14 timing measurements against the "
|
||||
"seeded production-shaped store (they skip otherwise — they cost minutes, not ms)",
|
||||
)
|
||||
|
||||
|
||||
def pytest_collection_modifyitems(config: pytest.Config, items: list[pytest.Item]) -> None:
|
||||
if config.getoption("--live"):
|
||||
return
|
||||
skip = pytest.mark.skip(reason="needs --live and a reachable target (see tests/test_live_fleet.py)")
|
||||
# Two opt-ins, one shape: a marker whose tests are real but expensive, skipped by default
|
||||
# so `uv run pytest` stays fast and hermetic. `--store-perf` is the phase-14 half of the
|
||||
# rule `--live` established; `scripts/store_perf_probe.py` is the CLI for the same numbers.
|
||||
skips = {
|
||||
"live": pytest.mark.skip(
|
||||
reason="needs --live and a reachable target (see tests/test_live_fleet.py)"),
|
||||
"store_perf": pytest.mark.skip(
|
||||
reason="needs --store-perf (a seeded ~6 M-row store; see tests/store_perf.py)"),
|
||||
}
|
||||
enabled = {
|
||||
"live": bool(config.getoption("--live")),
|
||||
"store_perf": bool(config.getoption("--store-perf")),
|
||||
}
|
||||
for item in items:
|
||||
if "live" in item.keywords:
|
||||
for marker, skip in skips.items():
|
||||
if not enabled[marker] and marker in item.keywords:
|
||||
item.add_marker(skip)
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
@@ -228,3 +246,40 @@ def metrics_text() -> Any:
|
||||
@pytest.fixture
|
||||
def rates() -> RateEstimator:
|
||||
return RateEstimator(span=10.0)
|
||||
|
||||
|
||||
# ---- phase 14: the production-shaped store the hot-path budgets measure ---- #
|
||||
#
|
||||
# Here rather than in tests/test_store.py because two files price the same budgets — the store
|
||||
# paths in `test_store.py`, the HTTP paths in `test_api.py` — and a second seeded 6.4 M-row file
|
||||
# (or, worse, a second *differently shaped* one) would make the two sets of numbers describe two
|
||||
# databases. Session-scoped because seeding costs ~25 s, and cached on disk (`data/perf/`, see
|
||||
# `store_perf.cache_dir`) because the before/after has to be the same physical file both times.
|
||||
|
||||
|
||||
@pytest.fixture(scope="session")
|
||||
def perf_store_path() -> Path:
|
||||
"""The cached seeded store, built on first use and re-verified on every reuse.
|
||||
|
||||
Opt-in tests only: nothing in the default run asks for this fixture, so `uv run pytest` never
|
||||
pays for the seed. `DASH_PERF_STORE_DIR` relocates the file (see `store_perf.cache_dir`).
|
||||
"""
|
||||
import store_perf # noqa: PLC0415 - only the phase-14 tests seed a store
|
||||
|
||||
try:
|
||||
return store_perf.ensure_seeded(log=lambda line: print(line, flush=True))
|
||||
except OSError as exc: # pragma: no cover - only a read-only checkout hits this
|
||||
pytest.skip(f"cannot build the seeded store ({exc}) — set DASH_PERF_STORE_DIR "
|
||||
"to a writable directory to run the phase-14 measurements")
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def perf_store(perf_store_path: Path) -> Any:
|
||||
"""The seeded store, opened per test and closed after (one handle per test, one file)."""
|
||||
import store_perf # noqa: PLC0415 - see perf_store_path
|
||||
|
||||
store = store_perf.open_store(perf_store_path)
|
||||
try:
|
||||
yield store
|
||||
finally:
|
||||
store.close()
|
||||
|
||||
+1696
File diff suppressed because it is too large
Load Diff
+170
-1
@@ -19,6 +19,7 @@ import pytest
|
||||
from fastapi import HTTPException
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
import store_perf
|
||||
from auth_client import sign_in
|
||||
from auth_contract import ADMIN_PASSWORD, FAST_PBKDF2
|
||||
from dashboards import alerts, auth
|
||||
@@ -34,7 +35,7 @@ from dashboards.sources.llm import (
|
||||
parse_gufo_metrics,
|
||||
parse_vllm_metrics,
|
||||
)
|
||||
from dashboards.store import Event, Sample, SampleStore
|
||||
from dashboards.store import RES_RAW, Event, Sample, SampleStore
|
||||
from slots_contract import (
|
||||
CANONICAL_TASK_ID_FIELD,
|
||||
OPTIONAL_NEXT_TOKEN_FIELDS,
|
||||
@@ -3153,6 +3154,84 @@ def test_one_request_per_host_answers_every_per_host_chart(
|
||||
assert gap["series"][0]["points"] == []
|
||||
|
||||
|
||||
def test_every_history_endpoint_answers_exactly_what_the_per_series_path_did(
|
||||
tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""Phase 14 task 02: the batched store door moved the cost of an N x M request, not its answer.
|
||||
|
||||
All four history routes now ask `store.history_many` for the whole batch instead of looping
|
||||
`store.history` per pair. That is only safe if the JSON is the same document the loop built,
|
||||
so this compares every route against points read straight out of the store with `query()` —
|
||||
a third path, in neither the loop nor the batch — and pins the series order the pages parse
|
||||
(host-major, metric-minor) plus the exact key set of one series entry.
|
||||
"""
|
||||
with build_client(tmp_path, monkeypatch, simulate=True, nut_host="127.0.0.1",
|
||||
host_targets=(Target("driveripper", DEAD_URL),
|
||||
Target("deskwork", DEAD_URL)),
|
||||
gufo_url=DEAD_URL) as test_client:
|
||||
store = test_client.app.state.ctx.store # type: ignore[attr-defined]
|
||||
now = time.time()
|
||||
writes = [
|
||||
("host", "driveripper", "cpu.usage_pct", [1.0, 2.0, 3.0]),
|
||||
("host", "driveripper", "mem.used_pct", [10.0, 20.0]),
|
||||
("host", "deskwork", "cpu.usage_pct", [4.0, 5.0, 6.0]),
|
||||
("host", "deskwork", "mem.used_pct", [30.0]),
|
||||
("disks", "driveripper", "disk.temp_c@nvme0n1", [38.0, 39.0]),
|
||||
("llm", "gufo", "llm.decode_tokens_per_second", [40.0, 45.0]),
|
||||
# a UPS name the simulator does not export, so its 5-second poll cannot add rows to
|
||||
# the series this assertion is about
|
||||
("ups", "notaserial", "ups.charge_pct", [99.0, 100.0]),
|
||||
]
|
||||
for source, host, metric, values in writes:
|
||||
store.add_many([Sample(source, host, metric, value, now - (len(values) - index) * 60)
|
||||
for index, value in enumerate(values)])
|
||||
|
||||
def expected(source: str, host: str, metric: str) -> list[dict[str, float]]:
|
||||
rows = store.query(source, host, metric, res=RES_RAW, start=now - 3_600.0, tail=True)
|
||||
return [{"ts": ts, "value": value} for ts, value in rows]
|
||||
|
||||
routes = [
|
||||
("/api/system/history", {"hosts": "driveripper,deskwork",
|
||||
"metrics": "cpu.usage_pct,mem.used_pct", "range": "1h"},
|
||||
[("host", host, metric) for host in ("driveripper", "deskwork")
|
||||
for metric in ("cpu.usage_pct", "mem.used_pct")]),
|
||||
("/api/disks/history", {"hosts": "driveripper",
|
||||
"metrics": "disk.temp_c@nvme0n1", "range": "1h"},
|
||||
[("disks", "driveripper", "disk.temp_c@nvme0n1")]),
|
||||
("/api/llm/history", {"engines": "gufo", "metrics": "decode_tokens_per_second",
|
||||
"range": "1h"},
|
||||
[("llm", "gufo", "llm.decode_tokens_per_second")]),
|
||||
("/api/ups/history", {"ups": "notaserial", "metrics": "ups.charge_pct",
|
||||
"range": "1h"},
|
||||
[("ups", "notaserial", "ups.charge_pct")]),
|
||||
]
|
||||
for path, params, pairs in routes:
|
||||
body = test_client.get(path, params=params).json()
|
||||
assert len(body["series"]) == len(pairs), path
|
||||
for series, (source, host, metric) in zip(body["series"], pairs, strict=True):
|
||||
assert set(series) == {"source", "host", "metric", "range_seconds", "points"}
|
||||
assert (series["source"], series["host"], series["metric"]) == (source, host, metric)
|
||||
assert series["range_seconds"] == 3_600.0
|
||||
assert series["points"] == expected(source, host, metric), f"{path} {metric}"
|
||||
|
||||
# and an empty answer is still an answer: a name with no rows, in a batch with names that
|
||||
# do, comes back as an empty series rather than a missing entry or a 404
|
||||
body = test_client.get("/api/system/history", params={
|
||||
"hosts": "driveripper", "metrics": "cpu.usage_pct,gpu.busy_pct@7",
|
||||
"range": "1h"}).json()
|
||||
assert [series["points"] for series in body["series"]] == [expected(
|
||||
"host", "driveripper", "cpu.usage_pct"), []]
|
||||
|
||||
# no metric named is no series asked for: the batch door is never opened with an empty
|
||||
# list (SQL `IN ()` is a syntax error waiting for that, and 500 is not the answer anyway)
|
||||
empty = test_client.get("/api/system/history",
|
||||
params={"hosts": "driveripper", "metrics": ""}).json()
|
||||
assert empty["hosts"] == ["driveripper"] and empty["series"] == []
|
||||
no_hosts = test_client.get("/api/system/history",
|
||||
params={"hosts": "", "metrics": "cpu.usage_pct"}).json()
|
||||
assert no_hosts["hosts"] == ["driveripper", "deskwork"], "empty means every configured host"
|
||||
assert len(no_hosts["series"]) == 2
|
||||
|
||||
|
||||
def test_llm_snapshot_keeps_an_engine_that_never_answered_in_the_list(
|
||||
tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""A configured-but-dead engine is listed as unreachable, not dropped (invariant 4)."""
|
||||
@@ -4079,3 +4158,93 @@ def test_samples_written_by_the_poller_are_queryable(tmp_path: Path,
|
||||
"range": "15m"}).json()["points"]
|
||||
assert [p["value"] for p in points] == [40.0, 45.0]
|
||||
store.close()
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# phase 14 task 04: the endpoint budgets
|
||||
#
|
||||
# `store_perf.BUDGETS` is the one table of ceilings (it is what the probe's report and the notes
|
||||
# render, so a ceiling relaxed in one place is relaxed everywhere). The ceilings are 2-3x the
|
||||
# measured "after" on the seeded production-shaped store, which makes them a regression gate
|
||||
# rather than a stopwatch: the `/system` page's own chart request costs ~2 ms and still gets
|
||||
# 100 ms, while putting a full-table scan back on a per-request path crosses by three orders of
|
||||
# magnitude. Wall time lives behind `--store-perf` for that reason; what runs in every `uv run
|
||||
# pytest` is the key-name pin below, because a budget that silently stops being priced is as
|
||||
# useless as a budget that is crossed.
|
||||
# --------------------------------------------------------------------------- #
|
||||
|
||||
|
||||
def test_every_http_budget_names_a_route_the_probe_prices(tmp_path: Path) -> None:
|
||||
"""The stale-pointer check for the HTTP half, on a throwaway store.
|
||||
|
||||
`measure_http` is what the priced test runs, and it owns the request shapes; if a key there is
|
||||
renamed, or a route stops being measured, the priced test would report the gap rather than the
|
||||
cost — which is why this asserts the key set here, cheaply, on every run.
|
||||
"""
|
||||
report = store_perf.measure_budget_http(tmp_path / "budget-keys.sqlite3",
|
||||
hosts=("driveripper",), metrics=("cpu.usage_pct",))
|
||||
assert set(report["http"]) == set(store_perf.HTTP_MEASURED_KEYS), (
|
||||
"the HTTP pass and the budget table no longer describe the same requests")
|
||||
assert store_perf.missing_budget_keys(report, ("http",)) == []
|
||||
# and the request the budgets describe is the one the task names: 3 hosts x 9 series
|
||||
nine = store_perf.HISTORY_METRICS
|
||||
assert len(nine) == 9, "the 90 d budget is priced against a 3x9 request, not some other shape"
|
||||
|
||||
|
||||
def test_metrics_stops_scanning_for_series_names_once_the_hub_is_polling(
|
||||
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
"""The live shape behind one budgeted endpoint, guarded by statements rather than a clock.
|
||||
|
||||
Task 04's watch caught `/api/metrics` answering with its full `SELECT DISTINCT` scan on
|
||||
practically every call of the running server (535 ms, measured): task 02's cache was invalidated
|
||||
by every sample write, and the hub writes every 2-300 s, so the cache never lived long enough to
|
||||
serve. The store-level half of this lives in `tests/test_store.py`; this is the half that says
|
||||
the *endpoint* is what stops scanning, on the app the operator runs.
|
||||
"""
|
||||
db = str(tmp_path / "metrics-busy.sqlite3")
|
||||
store = SampleStore(db)
|
||||
store.add_many([Sample("host", "driveripper", "cpu.usage_pct", 1.0, time.time()),
|
||||
Sample("host", "deskwork", "cpu.usage_pct", 1.0, time.time())])
|
||||
app = create_app(Settings(db_path=db, admin_password=ADMIN_PASSWORD,
|
||||
pbkdf2_iterations=FAST_PBKDF2), store)
|
||||
try:
|
||||
with TestClient(app) as http:
|
||||
sign_in(http)
|
||||
assert "cpu.usage_pct" in http.get("/api/metrics", params={"source": "host"}).json()["metrics"]
|
||||
|
||||
# a poll of series the store already has, then the same request with the log open
|
||||
store.add_many([Sample("host", "driveripper", "cpu.usage_pct", 2.0, time.time()),
|
||||
Sample("host", "deskwork", "cpu.usage_pct", 2.0, time.time())])
|
||||
seen: list[str] = []
|
||||
# the statement log is the test: `SampleStore._db` is the handle the request runs on
|
||||
store._db.set_trace_callback(seen.append)
|
||||
try:
|
||||
assert "cpu.usage_pct" in http.get("/api/metrics",
|
||||
params={"source": "host"}).json()["metrics"]
|
||||
finally:
|
||||
store._db.set_trace_callback(None)
|
||||
assert [s for s in seen if "DISTINCT metric" in s] == [], (
|
||||
f"`/api/metrics` went back to the table while nothing changed: {seen}")
|
||||
|
||||
# and a source that starts publishing something new is in the very next answer
|
||||
store.add("host", "driveripper", "gpu.busy_pct@0", 1.0)
|
||||
assert "gpu.busy_pct@0" in http.get("/api/metrics",
|
||||
params={"source": "host"}).json()["metrics"]
|
||||
finally:
|
||||
store.close()
|
||||
|
||||
|
||||
@pytest.mark.store_perf
|
||||
def test_the_budgeted_http_paths_fit_their_pinned_margins(perf_store_path: Path) -> None:
|
||||
"""`/api/health`, `/api/metrics`, `/api/system/history` (1 h and 90 d), `/api/summary`.
|
||||
|
||||
Priced through the real ASGI gate with a signed-in session, on the seeded 6.4 M-row store —
|
||||
the same call the operator's browser makes, in-process so no network is measured. This is the
|
||||
line `TODO.md:7` is about: `/api/health` used to answer in 1.27 s because it counted the whole
|
||||
`sample` table on every poll, on every tab.
|
||||
"""
|
||||
report = store_perf.measure_budget_http(perf_store_path)
|
||||
print()
|
||||
print(store_perf.render_budgets_markdown(store_perf.budget_rows(report, ("http",))))
|
||||
assert store_perf.budget_violations(report, ("http",)) == []
|
||||
|
||||
+1232
-2
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user