phase: 14_store_performance
**Phase 14 `store_performance` — final verification pass (all 4 tasks already in `complete/`)** - Re-verified every task's work in the tree: read caches (`series_names`/`stats`), capped compound `pick_res_many`, `history_many` + `api._history_batch`, windowed `maintain()` + per-resolution prune + `PRAGMA optimize`/`automatic_analyze=0`/WAL pragmas, `store_perf.BUDGETS` (14 ceilings), probe, live watch. - **Completion criteria:** baseline+after side by side in `data/logs/NOTES-store-perf.md` ✅ · budget tests green (`uv run pytest -m store_perf --store-perf` → **4 passed**, 14/14 inside) ✅ · 25-min dev-server watch re-run **independently** (74 passes, `scripts/watch-store-perf.py`): **0 `store-error`** (total still 190, newest 2026‑10‑18 06:33:30), WAL 7.77 MB ≤ 67.1 MB, medians 1.7–3.2 ms, p95 720/613 ms = the documented cold-cache window; a 3rd 27-pass watch on a freshly restarted server reproduced it (3.1/708, 1.8/572) ✅ · `maintain()` 5,368 ms → 2.7–6.8 ms and WAL bounded across two cycles (test + notes) ✅ · no behaviour change (all four history routes pinned against `query()`) ✅ - **Full gates:** `uv run pytest` → **1267 passed, 15 skipped**; `bash .agents/validate.sh` → **exit 0**, coverage **98.75%** (floor 94), `ruff check src tests agent` clean, `pyright` 0 errors, `node --check` ok, repo hygiene ok; `uv run pytest -m live --live` → **11 passed**; dev server up, 9/9 sources `ok`, pages 200 in 1.5–3.3 ms. - **Docs fixed:** README/AGENTS still claimed a flat "20 s per pass" while `host_agent.py` (in this same uncommitted change) scales the budget to 5 s × disk count — corrected in both plus the two illustrative "20 s of wedged disk" sentences; broken bold (`` `driveripper** ``) repaired; stale gap-4 title + "tuning is its own phase" wording updated to say phase 14 ran; README's absolute "nothing … is answered by a whole-table scan" qualified to *per call* with the cold-window cost named; "third opt-in marker" → second; conftest `--store-perf` help named the real marker; **`data/logs/watch-store-perf.py` moved to `scripts/watch-store-perf.py`** (both docs told a clone to run a gitignored path) with root-anchored paths, `DASH_WATCH_OUT`, and dead code removed. - **New finding recorded:** gap 9's "~10 s process-wide stall is still gap 4's / `14_store_performance`'s problem" is now closed — coincident >8 s `llm` + >12 s `ups` raw-stream gaps went 393 (00:00–12:00) and 198 (12:00–21:36) to **0** across 110+ min of the phase-14 store; written into AGENTS.md gaps 4 and 9 and the notes table. - **Deviations:** did *not* pull either deferred cost knob (`STATS_CACHE_TTL_SECONDS`, warming caches from the maintenance tick) — AGENTS.md records that decision and its reasoning; `uv pip list --outdated` still reports certifi/fastapi/starlette/websockets (pre-existing advisory, no dependency touched this phase, gate treats it as advisory). Overwrote nothing: task 04's watch preserved as `store-perf-watch.task04.*`, my two runs as `.verify25.*` / `.current-code.*`. - **Next pending phase:** `15_agent_mtls`, then `16_agent_smart_pass_budget` (owns the scaled-budget tests + its doc criterion, now partly pre-satisfied by this pass), `17_installer_device_filter`.
This commit is contained in:
@@ -1,66 +0,0 @@
|
||||
# Phase 14 — The store: fast reads, short writes, no "database is locked"
|
||||
|
||||
**Source:** `TODO.md` L7 — "Pages take forever to load and seem to lock up the database. Fix that."
|
||||
**Story:** n/a
|
||||
**Context:** `src/dashboards/store.py` (the `sample`/`event`/`meta` schema, `pick_res`, `history`, `series_names`, `stats`, `maintain`, the single `self._lock`, `busy_timeout=5000`), `src/dashboards/api.py` (the N-hosts × M-metrics history loops, `/api/metrics`, the polled `/api/health`), `src/dashboards/poller.py` (`add_many` on every successful poll — the write side that loses the `busy_timeout` race), `tests/test_store.py`, `tests/test_api.py`, `data/logs/` (the ~610 MB dev DB per AGENTS.md known gap 4 — the `store-error: OperationalError: database is locked` line)
|
||||
|
||||
## Objective
|
||||
|
||||
The page load time and the `database is locked` lines are one disease: the hot read
|
||||
paths of a 610 MB / 140 MB-WAL database are **full-table operations on every request**
|
||||
(`series_names()` scans for `DISTINCT metric`, the browser-polled `/api/health` runs
|
||||
`stats()` which counts the whole `sample` table, `pick_res()` pays three COUNT
|
||||
round-trips per series), while `maintain()`'s rollups are long whole-table write
|
||||
transactions that hold the WAL write lock long enough for a poller's `add_many` to
|
||||
outlive `busy_timeout=5000` and surface as the reported `store-error` (AGENTS.md
|
||||
known gap 4). This phase makes every hot path bounded (indexes, one grouped query,
|
||||
short-TTL caches) and every write short (windowed rollups, WAL checkpoint tuning),
|
||||
with timing budgets pinned as tests so a regression is a red test, not a slow page.
|
||||
|
||||
## Dependencies
|
||||
|
||||
- the foundation (complete) — `store.py`/`api.py`/`poller.py` as shipped; the rollup
|
||||
and retention contract in AGENTS.md invariant 9.
|
||||
- `11_admin_login` (todo) — **shared files, run-order note**: 11 rewrites `api.py`
|
||||
(the session gate) and adds `get_meta`/`set_meta`/`delete_meta` to `store.py`. Apply
|
||||
this phase's `store.py`/`api.py` changes on top of the file state at the time it
|
||||
runs; never restore a line the other phase owns. The two phases touch disjoint
|
||||
*regions* (11: the meta helpers + the auth routes; 14: the query paths + the
|
||||
maintain rollups) — the coupling is the file, not the logic.
|
||||
|
||||
## Tasks
|
||||
|
||||
1. `01_baseline_store_costs.md` — a production-shaped seeded store + the timing/EXPLAIN baseline for every hot path, recorded in `data/logs/`.
|
||||
2. `02_no_full_scan_on_hot_paths.md` — `series_names`/`stats` off the per-request full scan (bounded caches), `pick_res` to one grouped query, the N×M history endpoints to one grouped round trip.
|
||||
3. `03_windowed_rollups_and_wal_tuning.md` — `maintain()` rolls up only the window since `last_maintain`; WAL checkpoint + journal-size tuning so the 140 MB WAL stops growing between checkpoints.
|
||||
4. `04_budget_gates_and_live_proof.md` — the timing budgets pinned as tests, the live dev-server proof (page load measured, zero `store-error` over a watch window), docs updated.
|
||||
|
||||
## Testing & Quality
|
||||
|
||||
- Unit/integration: `tests/test_store.py` — the budget tests (a seeded
|
||||
production-shaped store: the task-01 recipe), the cache-invalidation behaviour, the
|
||||
windowed-rollup correctness (a rollup run twice rolls the new window only, and the
|
||||
1m/1h values match a whole-table rollup on the same data — the windowing must not
|
||||
change a single averaged value).
|
||||
- Integration: `tests/test_api.py` — the N×M history endpoints' response shape is
|
||||
byte-identical to today (only the cost changes), under the new budget.
|
||||
- Live: the dev-server watch per task 04 (AGENTS.md invariant 10 — the dev server stays
|
||||
up while work happens; invariant 9 — the DB stays bounded, on a volume in
|
||||
deployment).
|
||||
- Coverage: **>90%** on new/modified code; the 94% gate must hold.
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
- [ ] the task-01 baseline and the task-04 "after" numbers are both in
|
||||
`data/logs/NOTES-store-perf.md`, side by side
|
||||
- [ ] the budget tests are green: the full `/system` history request, `/api/metrics`,
|
||||
`/api/health` and a 90d `history()` call all inside their pinned budgets on the
|
||||
seeded store
|
||||
- [ ] a 25-minute dev-server watch (the task-04 recipe) shows **zero** `store-error`
|
||||
events and page loads inside the budget — the known-gap-4 failure mode is gone,
|
||||
not hidden
|
||||
- [ ] `maintain()`'s write transaction is measurably shorter (the task-03 note) and
|
||||
the WAL stays bounded across two maintain cycles
|
||||
- [ ] `uv run pytest` green, `bash .agents/validate.sh` green
|
||||
- [ ] no behavior change in completed phases: every history payload, rollup value and
|
||||
event answer is identical to today's shape (the `tests/test_api.py` pins)
|
||||
@@ -1,68 +0,0 @@
|
||||
# Task 01 — What the hot paths actually cost
|
||||
|
||||
**Phase:** `14_store_performance` · **Source:** `TODO.md:7` — "Pages take forever to load and seem to lock up the database. Fix that."
|
||||
|
||||
## Objective
|
||||
|
||||
A before/after exists only if the "before" is measured, not remembered. Seed a
|
||||
production-shaped store, time every hot path of `store.py`/`api.py` against it,
|
||||
`EXPLAIN QUERY PLAN` the queries that will change, and record the baseline in
|
||||
`data/logs/NOTES-store-perf.md` — the numbers task 04 compares against.
|
||||
|
||||
## Work
|
||||
|
||||
1. `tests/test_store.py` — the seeded-store recipe (a fixture the budget tests of this
|
||||
phase all reuse): an on-disk SQLite file (WAL, like production — the in-memory DB
|
||||
hides the lock behaviour) seeded to the dev DB's *shape and retention semantics*
|
||||
(`raw_keep_hours=48`, `minute_keep_days=30`, `hour_keep_days=400` — the
|
||||
`SampleStore` defaults, i.e. what a 90 d chart actually reads: 1m/1h, never raw
|
||||
beyond 48 h):
|
||||
- ~3 hosts × ~30 series per host, 10 s cadence for the 48 h raw window;
|
||||
- their 1m rollups for the 30 d window and 1h rollups for the 400 d window
|
||||
(generated by calling `maintain()` — a few hundred MB, the dev DB's order of
|
||||
magnitude, a temp file, never the dev DB itself: tests stay hermetic per the
|
||||
AGENTS.md testing policy);
|
||||
- ~2 000 event rows and the `meta` rows the app writes (`last_maintain` included).
|
||||
The test asserts the row counts, not the wall time of seeding.
|
||||
- ASSUMPTION: "production-shaped" means the dev DB's *shape and magnitude* (~610 MB,
|
||||
per AGENTS.md known gap 4), reproduced in a temp file — the dev DB itself is
|
||||
never opened by a test (tests stay hermetic, the AGENTS.md testing policy), and
|
||||
a temp DB of the same shape is the honest substitute.
|
||||
2. Measure (a script or the test's `--store-perf` marker; record in
|
||||
`data/logs/NOTES-store-perf.md` with the machine's specs):
|
||||
- `store.series_names()` (what `/api/metrics` runs per call) — wall time +
|
||||
`EXPLAIN QUERY PLAN` (expected: a full scan of `sample`);
|
||||
- `store.stats()` (what the browser-polled `/api/health` runs) — wall time +
|
||||
EXPLAIN (expected: a full COUNT scan of `sample` + `event`);
|
||||
- `store.pick_res()` × 3 + `store.history()` for one 90 d window (what one chart
|
||||
series costs) — wall time + EXPLAIN;
|
||||
- the full `/api/system/history` request shape (3 hosts × 9 series — the
|
||||
`tests/test_system_contract.py` table, 27 `_history` calls) through the test app
|
||||
— total wall time;
|
||||
- a `maintain()` run on the seeded store — the write transaction's wall time
|
||||
(the lock-holding window; note the `to_hour` step already scans the *entire*
|
||||
30 d 1m window on every run — the suspected long write) and the WAL file's
|
||||
size before/after;
|
||||
3. Record the baseline table in `data/logs/NOTES-store-perf.md`: path, wall time,
|
||||
EXPLAIN shape, and the wall time of the poller's `add_many` (a 30-sample insert)
|
||||
issued **concurrently** with a `stats()`/`series_names()` call — the race known gap
|
||||
4 describes, made visible as a number (how often does the insert wait, and does it
|
||||
ever outlive `busy_timeout=5000`? run the race loop ~100× and count the waits).
|
||||
|
||||
## Testing & Quality
|
||||
|
||||
- Unit: the seeded-store fixture in `tests/test_store.py` (asserts row counts and the
|
||||
rollup values match a second `maintain()` pass — determinism check).
|
||||
- No behaviour changes in this task — measurement only.
|
||||
- Coverage: **>90%** on the fixture code; the 94% gate must hold.
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
- [ ] `data/logs/NOTES-store-perf.md` exists: the full baseline table (wall times,
|
||||
EXPLAIN shapes, the concurrent-insert wait count), the machine's specs, and the
|
||||
seeded DB's row counts
|
||||
- [ ] the baseline confirms (or refutes — record the refutation) the three suspect
|
||||
mechanisms: `series_names` full scan, `stats` full scan, `maintain`'s long write
|
||||
window
|
||||
- [ ] the fixture is green and deterministic (two runs, same rollup values)
|
||||
- [ ] `bash .agents/validate.sh` green
|
||||
@@ -1,70 +0,0 @@
|
||||
# Task 02 — No full table scan on a per-request path
|
||||
|
||||
**Phase:** `14_store_performance` · **Source:** `TODO.md:7` — "Pages take forever to load and seem to lock up the database. Fix that."
|
||||
|
||||
## Objective
|
||||
|
||||
Three per-request paths pay a whole-table price on a 610 MB database:
|
||||
`series_names()` (a `SELECT DISTINCT metric` scan, on every `/api/metrics` call),
|
||||
`stats()` (a `COUNT(*) GROUP BY res` over all of `sample` plus `event`, on every
|
||||
browser-polled `/api/health`), and the N-hosts × M-metrics history endpoints
|
||||
(`pick_res()` is three COUNT round-trips per series, so one `/system` chart request
|
||||
is ~108 serialized round-trips through the single `self._lock`). Make every hot path
|
||||
bounded: short-TTL caches for the two scanned answers, one grouped query where three
|
||||
stand, and one grouped round trip where N×M stand. The response shapes are byte-
|
||||
identical — only the cost moves.
|
||||
|
||||
## Work
|
||||
|
||||
1. `src/dashboards/store.py` — `series_names()`: an in-memory cache of the distinct
|
||||
metric set (per `source`/`host` argument tuple, a 60 s TTL, invalidated on
|
||||
`add`/`add_many`/`delete_series` via a dirty flag — a cache that outlives a write
|
||||
answers "why is this chart empty" with yesterday's truth). `stats()`: cache the
|
||||
`sample`/`event` counts the same way (30 s TTL) — the `/api/health` poll (the
|
||||
browser's `app.js` masthead loop) must never run a full COUNT scan; `last_maintain`
|
||||
stays read through `get_meta()` as today (the meta table is tiny; do not cache
|
||||
what a three-row table costs nothing to read).
|
||||
2. `src/dashboards/store.py` — `pick_res()`: replace the three per-`res` COUNT queries
|
||||
with **one** `SELECT res, COUNT(*) … WHERE … ts >= ? GROUP BY res` (the three
|
||||
resolutions in a single index-range pass), same decision logic, same return value.
|
||||
3. `src/dashboards/api.py` — the four history endpoints' loops (system/disks/llm/ups —
|
||||
the `[host, metric]` nesting in each) call `store.history()` per pair, which is
|
||||
`pick_res()` (now 1 query) + the thinning query. Add the batched door in
|
||||
`store.py`: `pick_res_many(hosts, metrics, span, max_points)` and
|
||||
`history_many(hosts, metrics, span, max_points)` — one grouped COUNT over the whole
|
||||
`host IN (…) AND metric IN (…) AND res IN (…) AND ts >= ?` range, and one
|
||||
per-resolution thinning query (the per-series `LIMIT ?` tail-thinning kept by a
|
||||
window function or a per-series subquery — the shape of each series' answer must
|
||||
stay identical: ascending `(ts, value)`, `tail=True` semantics preserved). The
|
||||
endpoints' response dicts are unchanged (the same series, the same order); the
|
||||
`tests/test_api.py` pins are the contract.
|
||||
- ASSUMPTION: one grouped round trip (plus one thinning query per resolution
|
||||
present) is the target for the N×M endpoints — the alternative (N×M separate
|
||||
queries, each now cheaper) leaves ~100 serialized round-trips on a 3-host request,
|
||||
which on a 610 MB file is still the page-load tail; the grouped shape is the fix
|
||||
the TODO asks for. The per-series thinning keeps `tail=True` (the "the chart must
|
||||
have 'now'" docstring) exactly as today.
|
||||
4. `tests/test_store.py` — the pins: `series_names()`/`stats()` answer from cache
|
||||
within the TTL (a write after the read invalidates `series_names`; `stats` expires
|
||||
on the clock); `pick_res_many`/`history_many` equal the per-series answers on the
|
||||
seeded store (the byte-identical-shape test); a cache hit makes a second
|
||||
`series_names()` call measurably not scan (assert via a query-counter hook or the
|
||||
wall-time budget of task 04).
|
||||
|
||||
## Testing & Quality
|
||||
|
||||
- Unit/integration: per the work list, on the task-01 seeded store.
|
||||
- Regression: every pre-existing `tests/test_store.py` / `tests/test_api.py` test
|
||||
passes unmodified in intent — the response shapes are the contract.
|
||||
- Coverage: **>90%** on this task's new/modified code; the 94% gate must hold.
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
- [ ] no per-request path in `api.py` triggers a full-table scan of `sample` or
|
||||
`event` (the task-04 budgets are the proof; `EXPLAIN QUERY PLAN` of the grouped
|
||||
queries shows index-range access, recorded in the notes file)
|
||||
- [ ] the N×M endpoints' answers are byte-identical to the per-series path on the
|
||||
seeded store (the equality test)
|
||||
- [ ] a fresh `series_names()`/`stats()` call costs the full scan (documented — the
|
||||
first poll after a process start pays it once), a repeated call costs the cache
|
||||
- [ ] `bash .agents/validate.sh` green
|
||||
@@ -1,78 +0,0 @@
|
||||
# Task 03 — A rollup that holds the write lock for seconds, not minutes
|
||||
|
||||
**Phase:** `14_store_performance` · **Source:** `TODO.md:7` — "Pages take forever to load and seem to lock up the database. Fix that."
|
||||
|
||||
## Objective
|
||||
|
||||
`maintain()` is "safe to call every minute" and is idempotent — but on a real-sized
|
||||
store the `to_hour` step's `SELECT … GROUP BY` scans the **entire 30-day 1m window**
|
||||
on every run (and `to_minute` the entire 48 h raw window), and that GROUP BY is
|
||||
inside the one `with self._lock, self._db:` transaction: a multi-second write
|
||||
transaction that holds the WAL write lock long enough for a poller's `add_many` to
|
||||
outlive `busy_timeout=5000` and surface as the `store-error: OperationalError:
|
||||
database is locked` line (AGENTS.md known gap 4). Window each step to *what became
|
||||
due since the last run* — the idempotent `INSERT OR REPLACE` makes a slightly wider
|
||||
re-scan safe, so the window is a cost bound, not a correctness device — and tune the
|
||||
WAL so its 140 MB growth stops being a per-run surprise.
|
||||
|
||||
## Work
|
||||
|
||||
1. `src/dashboards/store.py` — `maintain()`, both rollup steps:
|
||||
- read `last_maintain` through `get_meta()` **before** the transaction (the
|
||||
existing `stats()` note applies: the meta read is outside the non-reentrant
|
||||
lock);
|
||||
- `to_minute`: the raw scan bound moves from `ts >= now - raw_keep_seconds` to
|
||||
`ts >= max(now - raw_keep_seconds, last_maintain - MINUTE - 60)` — steady state
|
||||
(a run per minute) scans ~one minute of raw rows; a missed interval scans the
|
||||
catch-up window; a first run on a fresh DB (no `last_maintain` row) scans the
|
||||
full retention window exactly as today;
|
||||
- `to_hour`: the same pattern against the 1m rows (`ts >= max(now -
|
||||
minute_keep_seconds, last_maintain - HOUR - 60)`). The `ts < now - {MINUTE|HOUR}
|
||||
- 60` incompleteness guard stays — a bucket is still only rolled once its whole
|
||||
window is in the past.
|
||||
- the `DELETE … pruned` step is unchanged (the retention window is the correct
|
||||
bound for a prune; it is `DELETE`s on indexed ranges, not a GROUP BY).
|
||||
2. `src/dashboards/store.py` — the WAL tuning in `__init__` (after the existing
|
||||
pragmas): `PRAGMA wal_autocheckpoint = 1000` (the default 1000 pages ≈ 4 MB
|
||||
auto-checkpoint is already the default — the 140 MB WAL means checkpoints lose to
|
||||
the write rate; the real lever is the journal-size cap) and
|
||||
`PRAGMA journal_size_limit = 67108864` (64 MB — the WAL stops growing past it;
|
||||
checkpoints then truncate it). Keep `journal_mode=WAL` and
|
||||
`synchronous=NORMAL` (the "a lost sample is fine, corruption is not" comment
|
||||
stands); keep `busy_timeout=5000` — the fix is a shorter write, not a longer wait.
|
||||
- ASSUMPTION: 64 MB is the journal-size cap — the dev DB's 140 MB WAL is the
|
||||
measured "too big" (AGENTS.md known gap 4), 64 MB is the nearest power of two
|
||||
below it, and the cap is a pragma, not a migration; if the live watch (task 04)
|
||||
shows checkpoint truncation starving writers, the cap moves (named constant in
|
||||
`__init__`, one-line change).
|
||||
3. `tests/test_store.py` — the pins:
|
||||
- **windowing correctness**: seed a store, run `maintain()` (full window, no
|
||||
`last_maintain` yet), advance time and add new 1m buckets, run `maintain()`
|
||||
again — the second run's `to_hour`/`to_minute` scan (assert via the EXPLAIN/row
|
||||
count hook, or a `sqlite3` trace) touches only the since-`last_maintain` window,
|
||||
**and** the rolled 1m/1h values are byte-identical to a reference whole-window
|
||||
rollup on the same data (the windowing must not change a single averaged value —
|
||||
`INSERT OR REPLACE` idempotency is what makes this safe, prove it);
|
||||
- **catch-up**: no `maintain` for 10 minutes, then one run — the 10 minutes of
|
||||
buckets are all rolled (the wider window, not a gap);
|
||||
- **the lock race**: the task-01 concurrent loop (a poller-style `add_many` during
|
||||
a `maintain()` run) now completes with **zero** `OperationalError`s over the
|
||||
100× loop (the baseline's wait count is the before, this is the after).
|
||||
|
||||
## Testing & Quality
|
||||
|
||||
- Unit: per the work list, on the task-01 seeded store.
|
||||
- Regression: every pre-existing `tests/test_store.py` test passes unmodified in
|
||||
intent (the idempotency/retention tests are the load-bearing ones).
|
||||
- Coverage: **>90%** on this task's new/modified code; the 94% gate must hold.
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
- [ ] the second `maintain()` run's write transaction is measured shorter than the
|
||||
task-01 baseline (both numbers in `data/logs/NOTES-store-perf.md`), and the
|
||||
rollup values are unchanged (the equality test)
|
||||
- [ ] the 100× concurrent `add_many`-during-`maintain` loop: zero `OperationalError`
|
||||
(baseline wait count vs after, in the notes file)
|
||||
- [ ] two `maintain()` cycles on the seeded store: the WAL file size stays bounded by
|
||||
the 64 MB cap (before/after sizes recorded)
|
||||
- [ ] `bash .agents/validate.sh` green
|
||||
@@ -1,78 +0,0 @@
|
||||
# Task 04 — The budgets, and the dev server that proves them
|
||||
|
||||
**Phase:** `14_store_performance` · **Source:** `TODO.md:7` — "Pages take forever to load and seem to lock up the database. Fix that."
|
||||
|
||||
## Objective
|
||||
|
||||
The "after" numbers, pinned as tests so a regression is red instead of slow: the
|
||||
task-01 baseline vs the fixed store, side by side in `data/logs/NOTES-store-perf.md`,
|
||||
plus the live dev-server proof on the real 610 MB DB — page loads inside budget, zero
|
||||
`store-error` events over a watch window (the known-gap-4 failure mode gone, not
|
||||
hidden), and the docs updated.
|
||||
|
||||
## Work
|
||||
|
||||
1. `tests/test_store.py` / `tests/test_api.py` — the budget tests (the task-01 recipe
|
||||
on the seeded store; budgets are the task-01 baseline's *target* — record the
|
||||
chosen numbers in the notes file with the reasoning):
|
||||
- the full `/api/system/history` request (3 hosts × 9 series, 1 h and 90 d):
|
||||
wall time budget (the 90 d case reads the 1m/1h rollups — the one that used to
|
||||
be the page-load tail);
|
||||
- `store.series_names()` / `store.stats()` second call (cache hit): wall time
|
||||
budget, asserted to be a fraction of the first call's (the scan is gone);
|
||||
- one 90 d `store.history()` call: wall time budget;
|
||||
- the budgets are generous (2–3× the measured "after"), so a slower CI machine does
|
||||
not flake red — the notes file records both the machine's specs and the margin.
|
||||
2. The live proof (AGENTS.md invariant 10 — the dev server stays up while work
|
||||
happens, and a fix is screenshotted, not claimed; invariant 9 — the DB is the
|
||||
volume's business in deployment, the dev DB is the test bench):
|
||||
- `./scripts/dev-server.sh restart` (its `stop` waits for the old pid to die —
|
||||
known gap 4's "two servers on one file" cause stays fixed), `status` shows the
|
||||
sources;
|
||||
- page loads: `curl -o /dev/null -s -w '%{time_total}'` on `/` and on the
|
||||
history-heavy pages' API calls (signed in, per phase 11), five runs each,
|
||||
recorded; the browser's `/llm` or `/system` page with a 90 d window selected is
|
||||
the worst case — screenshot `data/logs/store-perf-live.png` (the page loaded,
|
||||
the charts drawn, the masthead dots `ok`);
|
||||
- the watch: ≥ 25 minutes of `./scripts/dev-server.sh logs` (longer than the
|
||||
gap-4 baseline's "a few times an hour" rate) with **zero**
|
||||
`store-error: OperationalError: database is locked` lines; the `store.stats()`
|
||||
`event` count for `kind='store-error'` over the window is 0 (the `sqlite3`
|
||||
check on the dev DB — read-only);
|
||||
- the WAL file's size after the watch (the cap from task 03, in the real DB).
|
||||
- ASSUMPTION: the watch window is 25 minutes against a "few times an hour"
|
||||
baseline rate — enough to cover > 0.4 of the baseline's expected occurrences at
|
||||
the mean rate; if the baseline was at the mean, a zero-count 25-minute window
|
||||
already moves the needle, and the `sqlite3` event-table check over *the whole
|
||||
dev DB's* `store-error` rows (not just the window) shows when the last
|
||||
occurrence predates this phase.
|
||||
3. Docs: `README.md` — the *Configuration*/*Development* notes gain the two new
|
||||
`PRAGMA`s' one-line "why" (the journal cap, and that it is a cap not a target);
|
||||
`AGENTS.md` — known gap 4's entry is updated to record the fix (the windowed
|
||||
rollups, the journal cap, the budget tests) and the gap's "what remains" line
|
||||
(checkpoint starvation was never shown to be a cause — the controlled 25 s read
|
||||
transaction produced none — so the entry's honest wording stays, with the
|
||||
`database is locked` line now attributed to the rollup's write window). (The
|
||||
AGENTS.md/README update is the phase's *normal work* per the repo's definition of
|
||||
done — this instruction is for the phase's executor, not for the conversion.)
|
||||
|
||||
## Testing & Quality
|
||||
|
||||
- Unit/integration: the budget tests per step 1.
|
||||
- Live: the step-2 watch, screenshots and log evidence in `data/logs/`.
|
||||
- Coverage: **>90%** on new/modified code; the 94% gate must hold.
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
- [ ] `data/logs/NOTES-store-perf.md` has the baseline and the after table side by
|
||||
side (every task-01 path, wall time before/after, EXPLAIN shapes)
|
||||
- [ ] the budget tests are green with the recorded margins; `bash .agents/validate.sh`
|
||||
green
|
||||
- [ ] the 25-minute watch: zero `store-error` lines in the dev-server logs and zero
|
||||
new `store-error` rows in the dev DB's `event` table; `data/logs/` holds the
|
||||
page-load numbers and `store-perf-live.png`
|
||||
- [ ] the WAL's post-watch size is inside the 64 MB cap on the real DB
|
||||
- [ ] `README.md`/`AGENTS.md` carry the updated, accurate wording (gap 4: fixed, with
|
||||
its "what remains" honest)
|
||||
- [ ] no behavior change in completed phases: the `tests/test_api.py` payload pins all
|
||||
pass — the answers are the same, only faster
|
||||
@@ -0,0 +1,42 @@
|
||||
# Phase 16 — Pin the agent's scaled smart-pass budget
|
||||
|
||||
**Source:** chat 2026-10-19 — the `driveripper` 16-drive host: the flat 20 s smart-pass
|
||||
budget truncates a 16-device pass (a real read takes ~30–80 s), and because the deadline
|
||||
resets each pass, the tail drives would have stayed `smart: null` every pass. The fix —
|
||||
budget = 20 s floor scaled to 5 s × disk count, an explicit override still winning — was
|
||||
applied to the working tree on 2026-10-19 (operator-confirmed, verified on 16/16 live
|
||||
drives). This phase pins that already-applied change with tests and documentation.
|
||||
**Story:** n/a
|
||||
**Context:** `agent/host_agent.py` (`_smart_pass` — the scaled `budget = max(SMARTCTL_BUDGET_SECONDS, self.smart_timeout_seconds * len(device_names)) when the default is in force, else the explicit override; the `pass budget of {budget:.0f}s exhausted` error line; the module docstring "5 s per device and a pass budget of 20 s scaled to the disk count"), `tests/test_agent.py::test_the_pass_budget_skips_the_rest_with_one_error` (sets `collector.smart_budget_seconds = 0.0` — the override-wins branch), `AGENTS.md` line 185 ("5 s per device / 20 s per pass / 240 s cache") and the `deploy/install-agent.sh` header comment (line 25 area, "bounded 5 s per device and 20 s per pass" if still present there), `README.md` line 94 ("20 s per pass, results cached 4 minutes")
|
||||
|
||||
## Objective
|
||||
|
||||
The smart pass budget scale-up that is already in the working tree is pinned: a test proves the *default* scales (16 devices → 80 s deadline, one device → 20 s floor, the exhaustion line names the scaled figure), the override-wins branch stays covered, and every doc that still says "20 s per pass" says the floor-scaled truth. No new behaviour — the code change is already deployed on `driveripper` and verified live.
|
||||
|
||||
## Dependencies
|
||||
|
||||
- `09_disks_smartctl` (complete) — the smart pass, its bounds, and `tests/test_agent.py`'s in-process agent harness
|
||||
|
||||
## Tasks
|
||||
|
||||
1. `01_pin_the_scaled_budget.md` — the tests that the default budget scales with the disk count and that the error line names the scaled figure
|
||||
2. `02_docs_and_gate.md` — every "20 s per pass" mention in `AGENTS.md`/`README.md`/`agent/host_agent.py` says the floor-scaled truth; the gate is green
|
||||
|
||||
## Testing & Quality
|
||||
|
||||
- Unit: `tests/test_agent.py` — the scaled default (a default-attribute collector over 16 device names exhausts only when the 80 s budget runs out; over one device the floor is 20 s; the exhaustion error line carries the scaled figure, not the floor)
|
||||
- Coverage: **>90%** on the modified code (the `_smart_pass` branch)
|
||||
- The phase is "Complete" only when the **entire** project test suite passes 100% (every test in the project, not just this phase's)
|
||||
|
||||
## Documentation & Consistency
|
||||
|
||||
- The final pass reviews `README.md` and `AGENTS.md` in full against the project as it stands after this phase — fixing every inconsistency found, **including pre-existing ones not caused by this phase**.
|
||||
- Tasks update `README.md`/`AGENTS.md` immediately when their own changes are user-visible (the smart-pass bounds are a documented contract of `host_agent.py`).
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
- [ ] `uv run pytest tests/test_agent.py` green, including the new scaled-budget tests and the untouched override test
|
||||
- [ ] `grep -rn "20 s per pass" AGENTS.md README.md agent/ deploy/` returns nothing (the wording is floor-scaled everywhere)
|
||||
- [ ] `bash .agents/validate.sh` green (pytest --cov --cov-fail-under=94, ruff, pyright, node --check)
|
||||
- [ ] README.md and AGENTS.md are accurate and consistent with the whole project (verified in the final pass)
|
||||
- [ ] no behavior change in completed work: the deployed agent on `driveripper` is byte-identical to the repo's `agent/host_agent.py` (md5), the live suite is green
|
||||
@@ -0,0 +1,26 @@
|
||||
# Task 01 — Pin the scaled smart-pass budget
|
||||
|
||||
**Phase:** `16_agent_smart_pass_budget` · **Story:** n/a
|
||||
|
||||
## Objective
|
||||
|
||||
The default smart-pass budget (20 s floor, scaled to 5 s × disk count) is proven by tests: the deadline actually moves with the disk count, the exhaustion error line names the *scaled* figure, and the existing override-wins branch (a test sets `smart_budget_seconds = 0.0`) keeps passing untouched. The code change is already in `agent/host_agent.py::_smart_pass` — this task adds the tests, it does not rewrite the code.
|
||||
|
||||
## Work
|
||||
|
||||
1. `tests/test_agent.py` — add a helper that builds the in-process collector with **default** smart attributes (do not touch `smart_budget_seconds`/`smart_timeout_seconds`) and a `smartctl` fake that records each device's deadline window, in the same style as `test_the_pass_budget_skips_the_rest_with_one_error` (which stays untouched):
|
||||
- `test_the_pass_budget_scales_with_the_disk_count`: 16 device names → the last device's `smartctl` run happens with `left` (its deadline margin) consistent with a ~80 s budget (assert the deadline the code computes: `max(20, 5 × 16)`; the fake binary is instant, so assert via the recorded `min(left, 5.0)` timeout being 5.0 for a late device and the pass *not* truncating — every device got a `smartctl` run); 1 device name → budget 20 s (assert the recorded timeout is 5.0 and, with the fake made slow enough to exhaust a 20 s but not an 80 s budget in a *separate* case if feasible — if the harness cannot fake wall time, assert the computed `budget` expression via the error-line route instead: force exhaustion with a `time.monotonic` monkeypatch offset).
|
||||
- `test_the_pass_budget_error_line_names_the_scaled_figure`: force exhaustion on 16 devices (monkeypatch the clock forward past the 80 s deadline between two device runs, as the existing test's `0.0` override does for its 0 s budget) → the error line reads `smart: pass budget of 80s exhausted, N disk(s) skipped` — the scaled figure, not the 20 s floor.
|
||||
2. `agent/host_agent.py` — **no code change expected.** If a test exposes a real bug in the already-applied change (e.g. the `== SMARTCTL_BUDGET_SECONDS` default check), fix it minimally and record the diff in the task's completion note; do not redesign.
|
||||
|
||||
## Testing & Quality
|
||||
|
||||
- Unit: the two new tests above + the untouched `test_the_pass_budget_skips_the_rest_with_one_error` (override wins: `0.0` still exhausts immediately on 3 devices with one error line)
|
||||
- Coverage: **>90%** on `agent/host_agent.py::_smart_pass`'s budget branch (the default-scaled path must now be exercised, not just the override)
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
- [ ] `uv run pytest tests/test_agent.py -q` green: the two new tests, the untouched override test, and every pre-existing agent test
|
||||
- [ ] the 80 s case asserts the *scaled* figure in the error line (a test that passed with a 20 s budget would fail on the scaled code — no tautological assertions)
|
||||
- [ ] `uv run ruff check agent tests` and `uv run pyright` clean
|
||||
- [ ] no behavior change in completed work: `agent/host_agent.py` unchanged unless a real bug was found (then the fix is minimal and recorded)
|
||||
@@ -0,0 +1,25 @@
|
||||
# Task 02 — Docs and gate for the scaled budget
|
||||
|
||||
**Phase:** `16_agent_smart_pass_budget` · **Story:** n/a
|
||||
|
||||
## Objective
|
||||
|
||||
Every place that still says the smart pass is "20 s per pass" says the floor-scaled truth (20 s floor, 5 s × disk count), and the phase gate is green. The agent's module docstring was already updated on 2026-10-19 to the scaled wording — this task verifies it and fixes the two remaining doc lines.
|
||||
|
||||
## Work
|
||||
|
||||
1. `AGENTS.md` line ~185 (the `host_agent.py` bullet): `5 s per device / 20 s per pass / 240 s cache` → `5 s per device / a pass budget of 20 s scaled to 5 s × disk count / 240 s cache` (keep the bullet's voice; one line).
|
||||
2. `README.md` line ~94: `20 s per pass, results cached 4 minutes` → `a pass budget of 20 s scaled to the disk count (5 s each), results cached 4 minutes` (one line).
|
||||
3. `agent/host_agent.py` module docstring — verify it reads `bounded 5 s per device and a pass budget of 20 s scaled to the disk count, cached 240 s`; fix only if it drifted.
|
||||
4. `bash .agents/validate.sh` — the four checks (pytest --cov --cov-fail-under=94, ruff, pyright, node --check) + repo hygiene, all `uv run`.
|
||||
|
||||
## Testing & Quality
|
||||
|
||||
- No new logic in this task; the gate run is the check.
|
||||
- Coverage: **>90%** gate holds (task 01's tests carry the new branch).
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
- [ ] `grep -rn "20 s per pass" AGENTS.md README.md agent/ deploy/` returns nothing
|
||||
- [ ] `bash .agents/validate.sh` green
|
||||
- [ ] the two edited doc lines read the floor-scaled truth and nothing else changed in either file
|
||||
@@ -0,0 +1,59 @@
|
||||
# Phase 17 — The installer's broken device filter and the kernel-gate variant
|
||||
|
||||
**Source:** chat 2026-10-19 — the `driveripper` install: the template's
|
||||
`DevicePolicy=strict` + `DeviceAllow` lines produced a **deny-all device filter** on
|
||||
kernel 7.2 (systemd resolves `DeviceAllow` globs by name against `/proc/devices`, where
|
||||
this kernel lists NVMe as `blkext` (major 259), so `nvme*` matched nothing and the whole
|
||||
device namespace was locked). The operator removed the lines by hand; the unit now also
|
||||
runs `User=root` + `CapabilityBoundingSet=CAP_SYS_RAWIO CAP_SYS_ADMIN` (kernel 7.2's
|
||||
unprivileged-passthrough gate — `data/logs/NOTES-driveripper-agent.md`). The template
|
||||
still ships the broken lines, `User=$AGENT_USER`, and an empty bounding set, and re-running
|
||||
the installer on `driveripper` would **clobber the working unit** (unconditional
|
||||
`cat > "$UNIT"`). This phase makes the template the single source of the working unit.
|
||||
**Story:** n/a
|
||||
**Context:** `deploy/install-agent.sh` (`do_install` — the unconditional unit heredoc with `DevicePolicy=strict` + nine `DeviceAllow` lines, the `PrivateDevices=false` rationale comment block, `User=$AGENT_USER`, `CapabilityBoundingSet=`, the `--bind/--port/--token/--rotate-token/--status/--uninstall` flag set), `tests/test_deploy_scripts.py` (the fake-root install sandbox `test_install_path_installs_into_its_own_paths_and_prints_the_token_once`, the token-file permission test, the `bash -n` + function-naming pins), `AGENTS.md` (the fleet bullet "The deployed unit has since diverged from the template in the three ways the SMART block below records", gap 10's "the template change is its own phase"), `README.md` (the install section, line ~46 `sudo deploy/install-agent.sh --bind <addr> --port 9120`)
|
||||
|
||||
## Objective
|
||||
|
||||
Re-running `deploy/install-agent.sh` on any host — including `driveripper`, with
|
||||
`--as-root --caps CAP_SYS_RAWIO,CAP_SYS_ADMIN --force` — produces the working unit: no
|
||||
`DevicePolicy`/`DeviceAllow` lines (device access falls back to DAC via the `disk` group
|
||||
+ SELinux, the standard SMART-monitoring arrangement), the kernel-gate variant expressible
|
||||
by flags, and a re-run that would change an existing unit prints the diff and refuses
|
||||
without `--force`. The default install (dedicated user, empty bounding set) is otherwise
|
||||
byte-identical to what the template wrote before this phase.
|
||||
|
||||
## Dependencies
|
||||
|
||||
- `09_disks_smartctl` (complete) — the smart pass the unit exists to serve
|
||||
- `15_agent_mtls` (todo) — **shared-file, run-order note**: 15's task 06 adds `--cert-bundle`/`RuntimeDirectory` to the same `do_install`; apply on top of the file state at the time it runs, never restore a line the other phase owns. If 15 lands first, its new lines survive this phase's filter-line removal untouched.
|
||||
|
||||
## Tasks
|
||||
|
||||
1. `01_drop_the_broken_device_filter.md` — remove `DevicePolicy=strict` + all nine `DeviceAllow` lines and rewrite the rationale comment block
|
||||
2. `02_root_and_caps_flags.md` — `--as-root` and `--caps CAP1,CAP2` write the kernel-gate unit lines
|
||||
3. `03_no_silent_clobber.md` — a re-run that would change an existing unit prints the diff and refuses without `--force`
|
||||
4. `04_tests_docs_gate.md` — the unit pins in `tests/test_deploy_scripts.py`, `AGENTS.md`/`README.md` updated, gate green
|
||||
|
||||
## Testing & Quality
|
||||
|
||||
- Integration: `tests/test_deploy_scripts.py` — the template unit carries **no** `DevicePolicy`/`DeviceAllow` line; a default-install sandbox unit is byte-identical to the pre-phase template unit minus the filter lines; `--as-root --caps CAP_SYS_RAWIO,CAP_SYS_ADMIN` writes `User=root` and `CapabilityBoundingSet=CAP_SYS_RAWIO CAP_SYS_ADMIN` exactly; the re-run guard refuses (exit 1, diff on stdout) and overwrites with `--force`; `bash -n` and the existing function-naming pin still pass
|
||||
- Live (task 04, operator-verified): `driveripper`'s deployed unit is reproduced by `sudo deploy/install-agent.sh --bind 10.1.0.126 --port 9120 --as-root --caps CAP_SYS_RAWIO,CAP_SYS_ADMIN --force` — `systemctl diff` empty, `CapEff` still `0000000000220000`, 16/16 drives still live; `deskwork`/`gateway` untouched (their units are not re-run)
|
||||
- Coverage: the script itself is pinned by text assertions + `bash -n` (house pattern; >90% applies to the new Python test code)
|
||||
- The phase is "Complete" only when the **entire** project test suite passes 100% (every test in the project, not just this phase's)
|
||||
|
||||
## Documentation & Consistency
|
||||
|
||||
- The final pass reviews `README.md` and `AGENTS.md` in full against the project as it stands after this phase — fixing every inconsistency found, **including pre-existing ones not caused by this phase**.
|
||||
- Tasks update `README.md`/`AGENTS.md` immediately when their own changes are user-visible (the new flags are operator-facing; gap 10's "the template change is its own phase" becomes "done in phase 17" with the `driveripper` flag line).
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
- [ ] `grep -n "DevicePolicy\|DeviceAllow" deploy/install-agent.sh` returns nothing (the `PrivateDevices=false` line stays)
|
||||
- [ ] `sudo deploy/install-agent.sh --help` names `--as-root` and `--caps` with the kernel-gate sentence
|
||||
- [ ] a fresh sandbox install (fake root, no flags) produces a unit byte-identical to the pre-phase template unit minus the filter lines (pinned in `tests/test_deploy_scripts.py`)
|
||||
- [ ] a re-run that would change an existing unit exits 1 with the diff on stdout; with `--force` it overwrites
|
||||
- [ ] operator-verified on `driveripper`: the flags reproduce the working unit (`systemctl diff` empty, 16/16 drives live after restart)
|
||||
- [ ] `uv run pytest` green, `bash .agents/validate.sh` green
|
||||
- [ ] README.md and AGENTS.md are accurate and consistent with the whole project (verified in the final pass)
|
||||
- [ ] no behavior change in completed work: the default install's unit (minus the filter lines) and every existing flag behave as before; the live suite is green
|
||||
@@ -0,0 +1,25 @@
|
||||
# Task 01 — Drop the broken device filter
|
||||
|
||||
**Phase:** `17_installer_device_filter` · **Story:** n/a
|
||||
|
||||
## Objective
|
||||
|
||||
The template unit no longer carries `DevicePolicy=strict` or any `DeviceAllow` line, and the comment block that justified them is rewritten to say why the *absence* is the fix. The `PrivateDevices=false` line stays (explicit, default behaviour, and the rest of the lockdown is unchanged).
|
||||
|
||||
## Work
|
||||
|
||||
1. `deploy/install-agent.sh` — in the `do_install` unit heredoc: delete the `DevicePolicy=strict` line and all nine `DeviceAllow=…` lines (`/dev/null`, `/dev/zero`, `/dev/full`, `/dev/random`, `/dev/urandom`, `/dev/nvme*`, `/dev/sd*`, `/dev/vd*`, `/dev/sg*`).
|
||||
2. `deploy/install-agent.sh` — rewrite the comment block above `PrivateDevices=false` (the "The pseudo nodes are not decoration…" paragraph) to the new truth: **no device filter on purpose** — systemd resolves `DeviceAllow` globs by name against `/proc/devices` (fnmatch in `bpf_devices_allow_list_major`), where kernel 7.2 lists NVMe as `blkext` (major 259), so `nvme*` matched nothing and `DevicePolicy=strict` denied the *whole* device namespace (the `driveripper` install went deny-all, root included; `data/logs/NOTES-driveripper-agent.md`). Without the policy, device access falls back to DAC (the `disk` group the script already grants) + SELinux — the standard SMART-monitoring arrangement. Keep the one operational fact that survives: `PrivateDevices=false` is the default (a `true` sandbox with no `DeviceAllow` list is what dies at spawn with `status=208/STDIN` — keep that warning sentence, it is still why the line is written explicitly).
|
||||
3. `deploy/install-agent.sh` — the script's top-of-file header comment (line ~25: "the unit device-allows the disk nodes") → "the unit applies no device filter (kernel 7.2's `/proc/devices` name matching made the old `nvme*` glob deny-all); device access is DAC via the `disk` group + SELinux".
|
||||
|
||||
## Testing & Quality
|
||||
|
||||
- Integration: `bash -n deploy/install-agent.sh` clean (the existing pin covers it); a quick sandbox run (the `test_install_path_installs_into_its_own_paths_and_prints_the_token_once` pattern, run manually) shows no `DevicePolicy`/`DeviceAllow` in the written unit — the formal pins are task 04, this task keeps the change small and reviewable.
|
||||
- Coverage: **>90%** n/a (shell script; the pins are text assertions per the house pattern).
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
- [ ] `grep -n "DevicePolicy\|DeviceAllow" deploy/install-agent.sh` returns nothing
|
||||
- [ ] `bash -n deploy/install-agent.sh` clean
|
||||
- [ ] a manual sandbox install (fake root, `--bind 127.0.0.1 --port 9199`) writes a unit with the filter lines gone, `PrivateDevices=false` present, and every other line byte-identical to the pre-phase template unit
|
||||
- [ ] the rewritten comment names the `/proc/devices`/`blkext` mechanism in one sentence (an executor reading only the script can understand *why* there is no filter)
|
||||
@@ -0,0 +1,27 @@
|
||||
# Task 02 — `--as-root` and `--caps` flags
|
||||
|
||||
**Phase:** `17_installer_device_filter` · **Story:** n/a
|
||||
|
||||
## Objective
|
||||
|
||||
The kernel-gate variant of the unit (what `driveripper` runs) is expressible from the template: `--as-root` writes `User=root` instead of `User=$AGENT_USER`, and `--caps CAP1,CAP2` (comma-separated, e.g. `--caps CAP_SYS_RAWIO,CAP_SYS_ADMIN`) writes `CapabilityBoundingSet=CAP1 CAP2` instead of the empty `CapabilityBoundingSet=` line. Both default off; a flag without its friend is refused with a message.
|
||||
|
||||
## Work
|
||||
|
||||
1. `deploy/install-agent.sh` — flag parsing: add `--as-root` (sets `AS_ROOT=1`) and `--caps <list>` (sets `CAPS="<list>"`, validates non-empty, rejects an empty list with exit 1) in the existing `case`/flag block, and `--help` text:
|
||||
- `--as-root run the service as root (for a kernel-gated host: kernel 7.2 gates unprivileged smartctl passthrough behind capabilities, not uid — a root process under the default empty bounding set is capability-less, so this flag needs --caps to mean anything)`
|
||||
- `--caps LIST CapabilityBoundingSet line for the unit (comma-separated, e.g. CAP_SYS_RAWIO,CAP_SYS_ADMIN); default: empty (capability-less service)`
|
||||
2. `deploy/install-agent.sh` — validation: `--as-root` without `--caps` → print "note: --as-root with the default empty CapabilityBoundingSet leaves the process capability-less (the kernel gates check capabilities, not uid); pair it with --caps for a kernel-gated host" and **still proceed** (a capability-less root is a legitimate, if unusual, choice — do not hard-refuse; the note is the operator's reminder). `--caps` without `--as-root` → refuse, exit 1 ("--caps needs --as-root: a capability grant on the service user is not the kernel-gate fix").
|
||||
3. `deploy/install-agent.sh` — the unit heredoc: `User=$AGENT_USER` → `User=$( [ "$AS_ROOT" -eq 1 ] && echo root || echo "$AGENT_USER" )`; the empty `CapabilityBoundingSet=` line → the `CAPS` list converted comma→space when set. Add one comment line above: `# --caps: kernel 7.2 gates unprivileged smartctl passthrough (NVMe admin log: CAP_SYS_ADMIN; SG_IO allow-list: CAP_SYS_RAWIO); a root process needs them named, because an empty bounding set bounds even root to CapEff 0`.
|
||||
4. `deploy/install-agent.sh` — the `--status` probe is unchanged (it probes the agent over HTTP; the user the unit runs as is invisible to it). The token file stays `0640 root:$AGENT_USER` — with `--as-root` the group read is irrelevant (root reads it), so do **not** change the token-file logic.
|
||||
|
||||
## Testing & Quality
|
||||
|
||||
- Integration: manual sandbox runs (fake root): default → `User=dashboards-agent` + empty `CapabilityBoundingSet=`; `--as-root --caps CAP_SYS_RAWIO,CAP_SYS_ADMIN` → `User=root` + `CapabilityBoundingSet=CAP_SYS_RAWIO CAP_SYS_ADMIN`; `--caps` alone → exit 1 with the message; `--as-root` alone → proceeds with the note on stdout. (Formal pins: task 04.)
|
||||
- Coverage: **>90%** n/a (shell script; text pins per the house pattern).
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
- [ ] `sudo deploy/install-agent.sh --help` names both flags with the kernel-gate sentence (a fresh clone's executor can learn the `driveripper` recipe from `--help` alone)
|
||||
- [ ] the two sandbox units match the exact lines above (byte-for-byte on the `User=` and `CapabilityBoundingSet=` lines)
|
||||
- [ ] `bash -n deploy/install-agent.sh` clean; every pre-existing flag (`--bind/--port/--token/--rotate-token/--status/--uninstall`) behaves as before in a sandbox run
|
||||
@@ -0,0 +1,27 @@
|
||||
# Task 03 — No silent clobber on re-run
|
||||
|
||||
**Phase:** `17_installer_device_filter` · **Story:** n/a
|
||||
|
||||
## Objective
|
||||
|
||||
A re-run that would change an already-installed unit stops: it prints the diff between the installed unit and the unit the script would write, and exits 1 without touching the file. `--force` proceeds with the overwrite. A re-run whose generated unit is identical to the installed one proceeds silently (idempotent re-runs stay the zero-noise path the script has always had).
|
||||
|
||||
## Work
|
||||
|
||||
1. `deploy/install-agent.sh` — add a `--force` flag to the flag block (`FORCE=1`) and `--help` text: `--force overwrite an existing unit even if the generated unit differs (prints the diff first)`.
|
||||
2. `deploy/install-agent.sh` — in `do_install`, immediately before the `cat > "$UNIT"` heredoc write: build the would-be unit into a temp file (the heredoc already exists — render it once to `$(mktemp)`), and if `"$UNIT"` exists and `! cmp -s "$UNIT" "$tmp_unit"`:
|
||||
- if `FORCE=1`: print the diff (`diff -u "$UNIT" "$tmp_unit"` on stdout) and proceed with the overwrite;
|
||||
- else: print `refusing to overwrite $UNIT (generated unit differs); run with --force to apply:` + the same diff, and `exit 1` **before** the token-file write, the `systemctl daemon-reload`, and the `systemctl enable --now` — a refused run must leave the host exactly as it was (token file, unit, and running service all untouched; note the token-reuse logic runs earlier in `do_install` and only *reads* the existing token, so ordering the guard before the unit write is enough).
|
||||
- identical unit → no output, normal proceed.
|
||||
3. `deploy/install-agent.sh` — `--uninstall` is unchanged (it already removes the unit deliberately; the guard applies to `do_install` only).
|
||||
|
||||
## Testing & Quality
|
||||
|
||||
- Integration: manual sandbox sequence (fake root): install → modify the installed unit by one line → re-run with the same flags → exit 1, diff on stdout, installed unit byte-identical to the modified version, service still the old one; re-run with `--force` → diff printed, unit overwritten; re-run again unchanged → silent, exit 0. (Formal pins: task 04.)
|
||||
- Coverage: **>90%** n/a (shell script; text/behaviour pins per the house pattern).
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
- [ ] the three-way sequence above (refuse → force → silent) holds in the sandbox, with the token file and the running service untouched by a refused run
|
||||
- [ ] `--force` prints the diff *before* overwriting (an operator sees exactly what changed)
|
||||
- [ ] `bash -n deploy/install-agent.sh` clean; a fresh install (no existing unit) is unaffected by the guard
|
||||
@@ -0,0 +1,34 @@
|
||||
# Task 04 — Pins, docs, gate
|
||||
|
||||
**Phase:** `17_installer_device_filter` · **Story:** n/a
|
||||
|
||||
## Objective
|
||||
|
||||
The template's new shape is pinned in `tests/test_deploy_scripts.py` (no filter lines, the flags' exact lines, the clobber guard's refuse/force/silent sequence), `AGENTS.md` and `README.md` name the `driveripper` recipe, and the gate is green. The operator-verified `driveripper` reproduction (the completion criterion in `00_phase.md`) is the last check.
|
||||
|
||||
## Work
|
||||
|
||||
1. `tests/test_deploy_scripts.py` — extend the existing sandbox pattern:
|
||||
- `test_the_template_unit_has_no_device_filter_lines` — read `deploy/install-agent.sh`'s unit heredoc region and assert no line starts with `DevicePolicy=` or `DeviceAllow=`, and `PrivateDevices=false` is still present (the filter's absence is the fix; a regression that re-adds `DeviceAllow=/dev/nvme*` is the deny-all returning).
|
||||
- `test_a_default_install_unit_is_the_pre_phase_unit_minus_the_filter_lines` — the sandbox-installed unit equals the known pre-phase template unit text minus the ten filter lines (pin the whole unit, so a future template line drift fails here).
|
||||
- `test_as_root_and_caps_write_the_kernel_gate_unit_lines` — `--as-root --caps CAP_SYS_RAWIO,CAP_SYS_ADMIN` → `User=root` and `CapabilityBoundingSet=CAP_SYS_RAWIO CAP_SYS_ADMIN` exactly; `--caps` alone → exit 1 with the `--as-root` message; `--as-root` alone → proceeds with the capability-less note on stdout.
|
||||
- `test_a_re_run_that_differs_refuses_without_force_and_applies_with_it` — the refuse/force/silent sequence (exit 1 + diff on stdout + installed unit untouched; `--force` overwrites with the diff printed; an unchanged re-run is silent).
|
||||
2. `AGENTS.md` — the fleet bullet's "re-running `deploy/install-agent.sh` on driveripper would clobber the working unit, because it rewrites the unit unconditionally (gap 10)" → the post-phase truth: the installer no longer writes the device-filter lines, and the `driveripper` unit is reproducible with `sudo deploy/install-agent.sh --bind 10.1.0.126 --port 9120 --as-root --caps CAP_SYS_RAWIO,CAP_SYS_ADMIN --force` (a re-run that would change an existing unit refuses without `--force`). Update gap 10's "the template change is its own phase" → "done in phase 17" with the flag line.
|
||||
3. `README.md` — the install section: one line naming the flags' purpose (the kernel-gated-host recipe), keeping the how-to voice.
|
||||
4. `bash .agents/validate.sh` — all four checks green.
|
||||
5. **Operator-verified (live, read-only + one restart):** on `driveripper`, `sudo deploy/install-agent.sh --bind 10.1.0.126 --port 9120 --as-root --caps CAP_SYS_RAWIO,CAP_SYS_ADMIN --force` → `systemctl diff dashboards-agent` empty, `systemctl is-active` `active`, `CapEff` still `0000000000220000`, and 16/16 drives still live in the next `/snapshot` (the restart is the only fleet action; `deskwork`/`gateway` are not re-run). Screenshot `data/logs/disks-17-installer-repro.png` per invariant 10.
|
||||
|
||||
## Testing & Quality
|
||||
|
||||
- Integration: the four new pins above + the pre-existing pins (sudoers, `bash -n`, function naming, token-file permissions) all green.
|
||||
- Coverage: **>90%** on the new test code; the 94% gate holds.
|
||||
- Live: the operator-verified reproduction (item 5) — the phase's `driveripper` criterion.
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
- [ ] the four new pins pass and every pre-existing `test_deploy_scripts.py` pin passes
|
||||
- [ ] `grep -n "DevicePolicy\|DeviceAllow" deploy/install-agent.sh` returns nothing
|
||||
- [ ] `AGENTS.md` fleet bullet + gap 10 and `README.md` name the `driveripper` flag recipe
|
||||
- [ ] the operator-verified reproduction: `systemctl diff` empty, 16/16 drives live, screenshot in `data/logs/`
|
||||
- [ ] `uv run pytest` green, `bash .agents/validate.sh` green
|
||||
- [ ] the live suite (`set -a; . ./.env.dev; set +a; uv run pytest -m live --live`) is green
|
||||
Reference in New Issue
Block a user