phase: 14_store_performance
**Phase 14 `store_performance` — final verification pass (all 4 tasks already in `complete/`)** - Re-verified every task's work in the tree: read caches (`series_names`/`stats`), capped compound `pick_res_many`, `history_many` + `api._history_batch`, windowed `maintain()` + per-resolution prune + `PRAGMA optimize`/`automatic_analyze=0`/WAL pragmas, `store_perf.BUDGETS` (14 ceilings), probe, live watch. - **Completion criteria:** baseline+after side by side in `data/logs/NOTES-store-perf.md` ✅ · budget tests green (`uv run pytest -m store_perf --store-perf` → **4 passed**, 14/14 inside) ✅ · 25-min dev-server watch re-run **independently** (74 passes, `scripts/watch-store-perf.py`): **0 `store-error`** (total still 190, newest 2026‑10‑18 06:33:30), WAL 7.77 MB ≤ 67.1 MB, medians 1.7–3.2 ms, p95 720/613 ms = the documented cold-cache window; a 3rd 27-pass watch on a freshly restarted server reproduced it (3.1/708, 1.8/572) ✅ · `maintain()` 5,368 ms → 2.7–6.8 ms and WAL bounded across two cycles (test + notes) ✅ · no behaviour change (all four history routes pinned against `query()`) ✅ - **Full gates:** `uv run pytest` → **1267 passed, 15 skipped**; `bash .agents/validate.sh` → **exit 0**, coverage **98.75%** (floor 94), `ruff check src tests agent` clean, `pyright` 0 errors, `node --check` ok, repo hygiene ok; `uv run pytest -m live --live` → **11 passed**; dev server up, 9/9 sources `ok`, pages 200 in 1.5–3.3 ms. - **Docs fixed:** README/AGENTS still claimed a flat "20 s per pass" while `host_agent.py` (in this same uncommitted change) scales the budget to 5 s × disk count — corrected in both plus the two illustrative "20 s of wedged disk" sentences; broken bold (`` `driveripper** ``) repaired; stale gap-4 title + "tuning is its own phase" wording updated to say phase 14 ran; README's absolute "nothing … is answered by a whole-table scan" qualified to *per call* with the cold-window cost named; "third opt-in marker" → second; conftest `--store-perf` help named the real marker; **`data/logs/watch-store-perf.py` moved to `scripts/watch-store-perf.py`** (both docs told a clone to run a gitignored path) with root-anchored paths, `DASH_WATCH_OUT`, and dead code removed. - **New finding recorded:** gap 9's "~10 s process-wide stall is still gap 4's / `14_store_performance`'s problem" is now closed — coincident >8 s `llm` + >12 s `ups` raw-stream gaps went 393 (00:00–12:00) and 198 (12:00–21:36) to **0** across 110+ min of the phase-14 store; written into AGENTS.md gaps 4 and 9 and the notes table. - **Deviations:** did *not* pull either deferred cost knob (`STATS_CACHE_TTL_SECONDS`, warming caches from the maintenance tick) — AGENTS.md records that decision and its reasoning; `uv pip list --outdated` still reports certifi/fastapi/starlette/websockets (pre-existing advisory, no dependency touched this phase, gate treats it as advisory). Overwrote nothing: task 04's watch preserved as `store-perf-watch.task04.*`, my two runs as `.verify25.*` / `.current-code.*`. - **Next pending phase:** `15_agent_mtls`, then `16_agent_smart_pass_budget` (owns the scaled-budget tests + its doc criterion, now partly pre-satisfied by this pass), `17_installer_device_filter`.
This commit is contained in:
+18
-8
@@ -22,7 +22,8 @@ Design rules
|
||||
* **Every read is guarded.** A vanished /proc entry (process exited mid-scan) skips that
|
||||
item; the snapshot still succeeds. The poll is degraded, never failed.
|
||||
* **SMART is a best-effort add-on.** The per-disk `smart` block runs `smartctl -i -A -H`
|
||||
in a subprocess: bounded 5 s per device and 20 s per pass, cached 240 s so a fast
|
||||
in a subprocess: bounded 5 s per device and a pass budget of 20 s scaled to the disk
|
||||
count, cached 240 s so a fast
|
||||
dashboard poll cannot re-issue ATA commands every few seconds. Refreshing an expired
|
||||
cache happens on a background thread while the request is served the previous result —
|
||||
20 s of wedged disk must never cost a poll more than its own timeout. No binary, a denied
|
||||
@@ -122,10 +123,12 @@ VIRTUAL_NET_PREFIXES = ("lo", "veth", "docker", "br-", "virbr", "br", "tap", "vn
|
||||
VIRTUAL_DISK_PREFIXES = ("loop", "ram", "zram", "sr", "dm-", "md127")
|
||||
|
||||
# SMART collection bounds. 5 s per device is generous for a healthy drive and caps how
|
||||
# long one wedged device can hold a snapshot hostage; 20 s caps the whole pass so even a
|
||||
# shelf of bad disks cannot stall the HTTP handler; 240 s is below the dashboard's
|
||||
# 5-minute disks cadence, so the slow disks job gets fresh data while the fast system
|
||||
# poll never re-runs smartctl every 10 s.
|
||||
# long one wedged device can hold a snapshot hostage; the pass budget is a 20 s floor
|
||||
# scaled to the per-device bound (5 s x disk count), so a 16-disk shelf gets the 80 s a
|
||||
# real read takes while a single-disk host keeps 20 s and a shelf of bad disks cannot
|
||||
# stall the HTTP handler; 240 s is below the dashboard's 5-minute disks cadence, so the
|
||||
# slow disks job gets fresh data while the fast system poll never re-runs smartctl
|
||||
# every 10 s.
|
||||
SMARTCTL_TIMEOUT_SECONDS = 5.0
|
||||
SMARTCTL_BUDGET_SECONDS = 20.0
|
||||
SMARTCTL_CACHE_SECONDS = 240.0
|
||||
@@ -522,7 +525,8 @@ class Collector:
|
||||
same error lines, so a snapshot always states why `smart` is null.
|
||||
|
||||
An **expired** cache is refreshed on a background thread and the request is served
|
||||
the previous result, because the pass budget (20 s) is five times the dashboard's
|
||||
the previous result, because the pass budget (20 s floor, 5 s per disk) is five
|
||||
times the dashboard's
|
||||
`HOST_TIMEOUT_SECONDS` (4 s): run inline on the request path, one wedged disk would
|
||||
turn the *system* source red every `smart_cache_seconds` for a fault in an add-on
|
||||
(AGENTS.md invariant 4 — SMART must not blink the page that does not use it). The
|
||||
@@ -569,14 +573,20 @@ class Collector:
|
||||
now = time.monotonic()
|
||||
results: dict[str, dict[str, Any] | None] = {}
|
||||
errors: list[str] = []
|
||||
deadline = now + self.smart_budget_seconds
|
||||
# 20 s floor scaled to the per-device bound: a 16-disk shelf takes more than 20 s
|
||||
# when the drives answer, and truncating the pass would null the tail drives.
|
||||
# An explicit override (the budget test sets this to 0) wins over the scale.
|
||||
budget = (max(SMARTCTL_BUDGET_SECONDS, self.smart_timeout_seconds * len(device_names))
|
||||
if self.smart_budget_seconds == SMARTCTL_BUDGET_SECONDS
|
||||
else self.smart_budget_seconds)
|
||||
deadline = now + budget
|
||||
for name in device_names:
|
||||
left = deadline - time.monotonic()
|
||||
if left <= 0:
|
||||
remaining = [n for n in device_names if n not in results]
|
||||
for skipped_name in remaining:
|
||||
results[skipped_name] = None
|
||||
errors.append(f"smart: pass budget of {self.smart_budget_seconds:.0f}s "
|
||||
errors.append(f"smart: pass budget of {budget:.0f}s "
|
||||
f"exhausted, {len(remaining)} disk(s) skipped")
|
||||
break
|
||||
try:
|
||||
|
||||
Reference in New Issue
Block a user