Compare commits

..
49 Commits
Author SHA1 Message Date
ducoterra 883e36c727 phase: 35_llm_equal_window_heights
**Phase 35 final verification pass — complete.**

- Both task files (01, 02) already in `complete/`; verified the work in-tree: `#engines` = `grid cols-2`, `.strip` has no `align-items` (keeps `display: grid`, `gap: .85rem`, `auto-fit minmax(min(340px, 100%), 1fr)`), `.grid.items-start` rule + corrected comment intact, rewritten pin `test_llm_engine_windows_stretch_to_a_common_row_height` in place.
- Live re-verified in browser (fresh `35-llm-verify.cjs` run, screenshots re-captured): both engines answering; strip cards 456 px / panel cards 422 px with flat equal bottoms; row gaps 13.6 px (uniform `.85rem`); `n/a` cells preserved; 360 px viewport single-column, no overflow/clip. `/system` no-regress: `grid auto items-start`, expanded disclosure does not stretch collapsed siblings (325.4 → 325.4).
- Completion criteria: served `/llm` markup ✓ · live equalized rows + screenshot ✓ · `/system` bytes unchanged ✓ · full suite/coverage/lint ✓ · README/AGENTS.md ✓ · no behaviour change (all existing pins pass) ✓.
- Tests: `bash .agents/validate.sh` → **exit 0**: `uv run pytest --cov` **2031 passed, 19 skipped** (15 live + 4 store_perf, opt-in), coverage **98.71%** (94% floor reached); `ruff` all checks passed; `pyright` 0 errors; `node --check` + repo hygiene clean.
- Live suite (`--live`, read-only): 4/15 pass (LLM engines); 5 fail solely on host-agent/upsd **connectivity timeouts** (9120/3051 unreachable from this machine — same errors as `dev-server.sh status`); environmental, no code defect. No live changes made (all verification read-only).
- Docs: README.md reviewed in full — verified quickstart, installer/agent flags, config defaults, Containerfile facts, script flags, fixtures, test files, polling cadences: **consistent, no changes needed**. AGENTS.md: fixed one stale statement (hermetic skip count "12 live" → "15 live"); nothing else stale.
- Dev server up (pid 811266), `/api/health` 200; hosts/upsd DOWN is the pre-existing environmental state, LLM source ok.
- Next pending phase: **none** — `todo/` contains only this phase (35).
2026-11-02 21:25:13 -05:00
ducoterra 219d7a7c3b phase: 34_radiance_llm_engine
Everything is verified and green. Final report:

**Phase 34 final verification pass — complete**

- Verified all 6 completed tasks' work against the live fleet and hermetic suite: `llm:radiance` job registered, `llm_engines()` registry drives 404 gate / history defaults / `/api/config` (additive, byte-identical gufo/vllm keys), radiance field map + `/stats` sidecar + 7-position spec curve, `/llm` strip/curve/rows.
- **Defect found & fixed (pre-existing, task 05):** the vLLM-only sentence "vLLM exposes no per-slot JSON…" rendered on the **radiance** panel (`slotSection`'s non-gufo branch). Fix: vLLM line gated to `engine === 'vllm'`; radiance's panel slot section renders nothing (its session home is the strip) — `page-llm.js` + the served-byte pin in `tests/test_api.py` (now asserts the radiance branch is empty and the sentence can't leak). Browser-verified; re-shot `data/logs/34-llm-radiance-rest.png` (the 4.5-min wait found no in-flight request, so the busy shot remains the earlier pass's — its strip rows still stand; only the panel text predates the fix).
- Tests: `uv run pytest` → **2031 passed, 19 skipped** (100%); `uv run pytest --cov` → **98.73%** (floor 94%; config 100%, llm 99%, collectors 98%, api 99%); `uv run ruff check src tests agent && uv run pyright` → clean; `node --check` OK; `uv pip list --outdated` → empty.
- `uv run pytest -m live --live` → **13 passed, 2 skipped** (vLLM unset — correct; live-write opt-in — correct); the read-only radiance test passed.
- Completion criteria: (1) engines `gufo`+`radiance`, `fleet.engines_live` 2, `config.llm.radiance.enabled` true — **pass (live curl)**; (2) unset → 404 no-dial + no job — **pass (spy-pinned, green)**; (3) fixtures present, both families (79 `radiance:` / 198 `vllm:` lines), drift pin — **pass**; (4) panel/curve/rows — **pass (re-shot rest screenshot)**; (5) live radiance test — **pass**; (6) suite/coverage/lint/`bash .agents/validate.sh` (exit 0) — **pass**; (7) docs — below; (8) gufo/vLLM behaviour unchanged — **pass (existing pins untouched)**.
- Documentation: **README.md** full review — env table, LLM-engines section, architecture diagram, capture-script and `/llm` section all name radiance correctly; no inconsistencies found, no changes needed. **AGENTS.md** — the three task-06 fixes (fleet line, route row, module bullets) are in place and all other statements match reality; no changes needed.
- Dev server healthy (`./scripts/dev-server.sh status`: all sources ok, `llm:radiance` answering); all changes left in the working tree for the harness.
- Next pending phase: none — 34 was the last phase in `todo/`.
2026-11-01 18:53:46 -05:00
ducoterra 86c8bf4992 phase: 33_gateway_route_switching
Phase 33 completion pass is done. Final report:

## Phase 33 — completion report

**All 8 task files complete** (`.agents/phases/complete/33_gateway_route_switching/`), full gate green, docs verified.

### Work in this pass

1. **`tests/test_routectl.py` added** (17 tests, all green) — exercises the *real* helper binary unprivileged against a fake tree (`--root`), the real agent parser, the captured gateway fixtures, and stubbed `nginx`/`haproxy`/`systemctl` on `PATH`. Pins the full protocol (locate → state → dry-run → temp-file/fsync/`os.replace` write → checker → restore on checker failure → reload-never-restart → re-read proof → sidecar record), all three route shapes (haproxy `server`, nginx `proxy_pass`, nginx stream `map` via its sink-value id), `drift` / unknown-id / `map-default` refusals, `list`/`verify`, and the `--serve` peer gate (`SO_PEERCRED` uid check, one JSON line per request, exact body shape).

2. **Real helper defect fixed** — `deploy/dashboards-routectl.py`: `main()`'s early-return paths returned `({"ok": False, …}, EXIT_*)` tuples, so `sys.exit()` printed a Python repr instead of a JSON line (usage errors, agent-import refusal). Now prints `json.dumps(…)` and returns only the code, keeping the one-JSON-line contract.

3. **Documentation consistency fixes** (task 08 criterion 6):
   - `README.md`: garbled verdict-colour sentence on the switch ("green and to the position") → "in the **on** position and tinted `--gate-on`" (the existence-pair contract).
   - `README.md`: audit sentence now names all three event kinds — `route-disabled` / `route-enabled` / **`route-denied`** (the code stores one per refused call; dry runs store none).
   - `page-gateway.js`: replaced the stale pre-33 module docstring ("the switch is not wired") with the two-arm `GATEWAY_ALLOW_WRITES` contract.
   - `AGENTS.md`: every phase-33 claim re-verified against code (the ten gates, the 35 s/30 s budget pair, `gateway.writes_enabled`, `/api/summary`'s `disabled`, the 1.4.0 agent bullet, the gap list) — no changes needed.

### Verification
- `uv run pytest --cov`: **1970 passed**, 18 skipped — coverage **98.74%** (floor 94%)
- `ruff` clean · `pyright` 0 errors · `bash .agents/validate.sh` **exit 0** (incl. `node --check` on every page script)
- Dev server running; all 10 sources `ok`; bundles exported

### Live infrastructure
**No live changes made.** The live drill remains unexecuted per `data/logs/NOTES-33-switching.md` (no operator approval; checker binaries unavailable in the podman proxy units). All routes stay enabled, config hashes unchanged, and the sandbox rehearsal + backup tree are in place for the operator's own `DANGER` call. All changes are in the working tree only.
2026-11-01 11:50:51 -05:00
ducoterra 0271387053 compact AGENTS.md 2026-10-28 00:02:17 -04:00
ducoterra 51772ffef2 phase: 32_gateway_resolved_target_liveness
Fixture and live stream.conf are byte-identical (modulo header comment). Let me test the parser directly:
2026-10-26 14:03:42 -04:00
ducoterra 25fd74b7c5 phase: 31_gateway_route_readability
**Phase 31 verification complete — all 5 tasks already in `complete/`; this was the final verification pass.**

Verified / changed
- Re-ran every phase-31 artifact live and green: `verify-31-page-only.py` 6 PASS/0 FAIL; browser pass `31-route-readability.cjs` **64 PASS / 0 FAIL** (exit 0); `27-task02-mobile-audit.cjs` **371 PASS / 0 FAIL / 0 FINDING** (exit 0); rebuilt `data/logs/31-route-readability.out` as the 4-part transcript.
- Added the 4 degradation-arm screenshots the criterion asked for but the pass lacked (`31-gateway-{unconfigured,unreachable,no-nginx,sockets-quiet}.png`) — 12 frames now, one per arm.
- Grep criteria: no gateway connector rule uses `border-strong`; no verdict token in the gateway CSS section or `page-gateway.js` beyond the `alerts.py`-owned `.warn`; exactly 4 `addEventListener` calls (none for the switch); `routeInConfig()` returns `true`; **no `.py` under `src/dashboards/` changed**.

Tests / lint (all 100% green)
- `uv run pytest` → **1796 passed, 16 skipped**; `--cov` → **98.94%** (floor 94); `ruff check src tests agent` → All checks passed; `pyright` → 0 errors; `node --check` clean; `bash .agents/validate.sh` → **exit 0**.
- `uv run pytest -m live --live` → **12 passed / 0 failed**; `./scripts/dev-server.sh status` → all **10 sources ok**.

Completion criteria: colour (33/33 rows every segment `--gate-on`, dark `rgb(63,185,80)` + light `rgb(26,127,55)`, zero grey dashes) ✔ · inbound-first rows + filter (33/33 line 1 = `out.selectors`, line 2 = `→ host:port · kind`; `stackexpected` → **6** rows, not the overview's "five" — the capture declares 6 such routes; tallies byte-identical `10/33/16/0`, zero requests) ✔ · trunk motion truth table + `null` arm prints `n/a` never `0` + reduced-motion stills 43 paths and every knob ✔ · screenshots/transcripts ✔ · payload page-only ✔ (literal byte-diff of a live socket table can never pass; answered by the no-`.py`-diff + shape-equality proof, as recorded in the transcript) · no behaviour change elsewhere ✔.

Docs: fixed 6 inconsistencies — two identifiers split across line breaks (`test_the_pipe_animation_stills_in_both_states_under_reduced_motion`, `32_gateway_resolved_target_liveness`), stale "sixteen routes" count, stale "four machines/hosts" in README **and** AGENTS (fleet is three), "three fields"→"three field types", gap-14 headline wording; updated the pass's screenshot list.

Deviations: `uv pip list --outdated` shows one advisory (`starlette 1.9.0 → 1.10.0`) — not upgraded, this phase moved no dependency (rule 8 / DoD apply "if deps moved"). `TODO.md` carries two new operator disk requests, left untouched.

Next pending phase: `32_gateway_resolved_target_liveness`.
2026-10-25 05:32:30 -04:00
ducoterra e366f599f3 phase: 30_gateway_connection_map
**Phase 30 final verification pass — all 6 tasks already in `complete/`, so this was the phase-completion pass.**

- **Verified (re-run, not trusted):** `/gateway` serves exactly two panels (nginx→haproxy, `GATEWAY_CARD_KEYS`); haproxy rows `.solo` straight across, nginx `:443` branched with one tick per leaf (18 live, fixture 17); every pipe computes the theme `--ok` in dark+light, animates only at count ≥1, fully static under `prefers-reduced-motion`; switch = `role="switch"`+`aria-checked`+`aria-disabled`, no listener, footer+title name the missing write path; zero non-GET requests anywhere.
- **Live read-only re-run** (`data/logs/30-final-live-verify.out`): 17 PASS — `warnings: []`, 6 haproxy frontends→own `server`, `:443` 18 map leaves incl. `default→127.0.0.1:4443`, `:4443` 6 / `:8443` 3, `:80` bare listener, no `unmatched`/`outbound`/`by_port`/`partial`, no `9120`; inbound reconciliation **page 16 = kernel 16**; `gateway` block = the 12 declared keys; no `gateway:config` alert on a clean config. `git diff --stat agent/ deploy/` empty; agents 1.2.0 (gateway) / 1.1.0 ×2, all `tls`, `bundle_rev 2`.
- **Tests/lint (exact):** `uv run pytest` → **1791 passed, 16 skipped**, exit 0 · `bash .agents/validate.sh` → **exit 0**, `--cov` **98.94%** (floor 94), `ruff check src tests agent` clean, `pyright` 0 errors, `node --check` all page scripts, hygiene clean · `uv run pytest -m live --live` → **12 passed** · `uv pip list --outdated` empty · `./scripts/dev-server.sh status` → **10 sources ok**, `gateway` included · browser pass `30-connection-map.out` 63 PASS/0 FAIL (5 screenshots at 1440/360) · `30-final-mobile-audit.out` 371 PASS/0 FAIL/0 FINDING, exit 0, nine pages.
- **Docs fixed:** README `alerts.py` bullet omitted phase 30's keyword-only `gateway_payload` and its one `warn` config-integrity row (+suppression) — added; overview ingress line labelled a dated reading. AGENTS gap 13 corrected 391 px → **361 px** (task 05's pill took 30 px), "task 06 measured" → "phase 29 measured", coverage figure un-pinned to 98.9 %, live-suite count noted (12 today), "all five pages" severity sentence clarified for `/gateway`'s existence-only palette. Everything else reviewed in full and **verified consistent**.
- **Defect fixed:** duplicated footer-assertion block at the end of `tests/test_api.py::test_the_switch_is_built_and_explicitly_unwired` (one copy removed).
- **Notable:** `NOTES.md` is empty in the working tree — both items in it (non-mTLS agents, disappearing admin links) landed in phases 24 and 20; left cleared. Fixture/live `map`-count drift (17 vs 18) is recorded in AGENTS.md, not silently re-captured.
- **Next pending phase:** none — `30_gateway_connection_map` was the only entry in `.agents/phases/todo/`.
2026-10-23 22:56:21 -04:00
ducoterra fa4f38c694 phase: 29_gateway_dashboard_page
Phase 29 completion pass is done. Final state:

**Verification (all re-run this pass, no task files touched)**
- `uv run pytest` → **1746 passed, 16 skipped**; `uv run pytest --cov` → **99.05 %** (gate 94 %); `bash .agents/validate.sh` → **rc=0** (ruff clean, pyright `0 errors, 0 warnings`, `node --check` clean); `uv run pytest -m live --live` → **12 passed**.
- Live, read-only: dev server **10/10 sources `ok`** (incl. `gateway`); `data/logs/29-task06-live.cjs` → **49 PASS / 0 FAIL** with per-port inbound counts reconciled against a direct `/proc/net/tcp{,6}` read (divergence 0); `data/logs/verify-29-task06-predates.py` → 0 failed (1.1.0 arm renders `WHY_PREDATES`, no pipes, every tally `n/a`, job still `ok`); phase-27 mobile audit re-run over all nine served pages → **371 PASS / 0 FAIL / 0 FINDING** (`data/logs/29-final-mobile-audit.out`).
- Criteria: (1) unconfigured `GATEWAY_HOST` → `404 {configured:false}` with dial spies + `#gate-empty` sentence ✅; (2) 1.1.0/`--no-proxy` → `supported:false`, no `0`s, never red ✅; (3) live gateway 1.2.0 → pipes animate, totals reconcile ✅; (4) no `severity`/verdict colour anywhere, validate green ✅; (5) docs updated + gap 13 logged ✅; (6) the five existing dashboards, jobs and `/api/summary` unchanged (live control leg) ✅.

**Docs (consistency pass, incl. pre-existing drift)**
- `README.md`: architecture diagram now lists `sources/gateway.py`, the 30 s gateway job and the third `/snapshot` reader; `GATEWAY_HOST` row says *a name, never a URL*; the `POLL_MS_*` row states `/gateway`'s fixed 15 s browser poll; "nine pollers" → per-source cadences; phase-28 "until phase 29" wording → current state; new Testing paragraph for `tests/test_gateway_source.py` (join pinned from the captures, reconciliation identity, `getaddrinfo` spy).
- `AGENTS.md`: phase-28 stop-point wording replaced with the shipped phase-29 state; suite/coverage figures → 1746 / 16 skipped / 99.08 %→99.05 % band and `~99 %`; window-chips paragraph records that `/gateway` has **no** chips (fixed 24 h); new testing-policy bullet for the join; stale page counts fixed (seven → eight tabbed pages / `MARKUP_PAGES`, clock-order and badge-count transcripts annotated); gap 12 now records the phase-29 nine-page audit re-run.
- `tests/test_api.py`: comment/docstring-only fixes for the "seven pages" wording that phase 29 made eight.

**Deviations / notes**
- Nothing implemented: all six task files were already in `complete/`, so this run was the final verification + docs pass; no git index or phase file was touched.
- Coverage wobbles 99.05–99.08 % between runs on a pre-existing timing-dependent line (`collectors.py` `/slots`-alias `except`/`break`), not a phase-29 path (`sources/gateway.py` is 100 %).
- `uv pip list --outdated` still lists upgrades; dependencies did not move in this phase, so they were left alone rather than upgraded mid-verification.
- Historical browser-transcript counts (phase 18/20/25 script runs) were left as recorded history and annotated instead of rewritten.

**Next phase:** none — `.agents/phases/todo/` is empty after 29; further work needs a new phase authored first.
2026-10-23 09:52:30 -04:00
ducoterra 44d61d3ab9 phase: 28_gateway_agent_proxy
**Phase 28 final pass — complete, 100 % green.** All five tasks were already `complete/`; this pass verified the criteria, re-ran the live half, and did the docs pass.

- Gate `bash .agents/validate.sh` **rc=0**: `1650 passed, 15 skipped`, coverage **98.97 %** (floor 94), `ruff` clean, `pyright` **0 errors**, `node --check` clean, hygiene clean (`/tmp/validate-done.log`).
- Live: `pytest -m live --live` **11 passed**; `./scripts/dev-server.sh status` nine sources `ok`; mTLS re-read of `gateway` → agent **1.2.0**, all three sections `available: true`, 6/6 frontends routed, 11 nginx servers + `$targetBackend`, **19** listeners, no core errors (`data/logs/28-finalpass-live.out`); installed agent file byte-identical to this tree (`b0e9ebf1b718…`).
- Criteria: 1 ✅ pytest/coverage · 2 ✅ three sections over a captured tree, `false`/`null` + one `errors` line each elsewhere with `cpu`/`mem`/`disk_io`/`system` intact · 3 ✅ `AGENT_VERSION` 1.2.0 + SCHEMA names all three · 4 ✅ six fixtures verbatim, capture refuses a bad read, 116 socket address columns hold no globally routable endpoint · 5 ✅ gateway runs 1.2.0 and answers over mTLS (transcript `data/logs/28-task05-rollout.md`, `28-finalpass-live.out`) · 6 ✅ validate · 7 ✅ additive: HEAD's 1.1.0 and this 1.2.0 `collect()`ed over one `--root` are key-for-key equal plus the three new sections, `--no-proxy` equal exactly (`data/logs/verify-28-final.py` → 41 PASS, `28-final-verify.out`).
- Docs fixed (README/AGENTS/Containerfile, all re-green): README's opening said "Four dashboards" above a five-row table and said the new sections are "stored in the snapshot" (they are in the **agent's** document — no `/api` payload, no `sample` row, nothing charts them until phase 29); `Containerfile`'s header still said "three dashboards"; AGENTS's coverage figure (98.94 → **98.97 %**), invariant 6 gained the captured-socket-table identifier rule, the agent bullet gained "phase 28 stops at the wire", and the fleet record names the stash-and-compare control plus the volatile listener count. Full record: `data/logs/NOTES-28-final.md`.
- Deviations: none functional; no git index/history touched, phase files untouched (`PLAN.md` and `complete/` unchanged). `uv pip list --outdated` still lists 4 packages (fastapi 0.153→0.161 etc.) — advisory only, dependencies untouched by this phase, left for a deliberate pass.
- Next pending phase: `.agents/phases/todo/29_gateway_dashboard_page/` (the page that consumes these three sections).
2026-10-22 20:24:01 -04:00
ducoterra 5d37444fb0 phase: 27_mobile_tables_and_audit
The audit re-ran green (exit 0). Now checking the docs for defects introduced by the phase.
2026-10-21 23:43:25 -04:00
ducoterra 9ff3c7889b phase: 26_ups_event_split
**Phase 26 `ups_event_split` — final verification pass (all 3 tasks already complete; nothing missing from `complete/`).**

Verified
- `page-ups.js`: `POWER_EVENT_KINDS = {status,test,action}` split, unconditional `refreshEvents()`, no `state.events`/toggle-fetch, shared `eventRow`, `#ups-errors-card.hidden = errors.length === 0`; `ups.html` error card above the disclosure, `hidden`, outside every `<details>`. Kind set cross-checked against the real writers (`collectors.py:393/409`, `api.py:1043/1058`, `poller.py:127/157/164/190`) — classification is correct.
- Re-ran the phase's browser pass live: `node data/logs/26-task03-verify.cjs` → **31 checks, 0 skipped, ALL PASS** (`data/logs/26-final-verify.out`) — poll-failure rows readable with the disclosure shut, card hidden on a power-only feed and appearing on the first `unreachable`, lists paging independently, 360 px clean.

Tests / lint (all green)
- `uv run pytest` → **1545 passed, 15 skipped**, exit 0 · `bash .agents/validate.sh` → exit 0, coverage **98.94%** (floor 94%) · `uv run ruff check src tests agent` clean · `uv run pyright` 0 errors · `node --check` clean · `uv run pytest -m live --live` (with `.env.dev`) → **11 passed**, dev server reads all 9 sources `ok`.

Completion criteria: poll-failure visible + power collapsed ✅ (pinned + browser); hidden-when-empty card, one `/api/events?source=ups` fetch, shared pager ✅; stale-last-good via the empty `catch` pin ✅; pytest/coverage/ruff/pyright ✅; `validate.sh` ✅; no behaviour change to prior work — `dash.css`/`app.js` untouched, disclosure CSS / pager / row-shape pins pass unchanged ✅.

Docs
- Fixed (pre-existing + this phase's drift): AGENTS.md "one helper with four call sites" → **five over four pages** and the `row(event)` sentence (phase 26 added a second `/ups` call site); invariant 5's `web/static/charts.js` → `src/dashboards/web/static/charts.js`; recorded the completion-pass transcript. README: "one helper renders every feed" now notes five lists. Full sweep otherwise consistent — all 38 env vars match `config.py` exactly, all served routes/scripts/contracts referenced exist, suite/coverage/live counts and marker splits verified.

Deviations: `fastapi 0.153.0 → 0.159.1` advisory left alone (validate.sh declares it advisory-not-a-gate; this phase touched no dependencies, matching prior phases' precedent). Next phase: **`27_mobile_tables_and_audit`**.
2026-10-21 19:42:19 -04:00
ducoterra 2bf236fd92 phase: 25_masthead_control_placement
**Phase 25 final verification pass** (all 3 tasks were already in `complete/`, so this was the phase-level verify + docs pass)

**Verified**
- `#nav-status` first child of `.masthead-right` on all 7 signed-in templates (incl. `/certs`, `/settings`); `app.js`/`charts.js` untouched
- `#nav-status` + `.masthead-right` both `justify-content: flex-end`, `margin-inline-start` still absent, row-break (`flex-wrap` + two `flex-basis: 100%`), non-sticky bar, dropdown slide + 4 close paths, `setInterval(refreshStatus, 15000)` all intact (dash.css diff = comments + the two alignment values)
- Clock is the last control on all 7 templates; no `order:` anywhere in the bar

**Fixed (defects found in this pass)**
- `tests/test_api.py::test_every_page_links_to_both_admin_pages` — duplicated `tabs = html[...]` statement (task 03 edit artifact)
- `certs.html` / `settings.html` comments said "all **eight** signed-in pages"; the signed-in set is seven (`/login` is the eighth, anonymous, and has no links/clock)
- `AGENTS.md` — suite tally `1542 → 1543 passed`; "the three phase-20 tests" re-worded, since task 02 renamed one to `test_the_masthead_is_three_rows_right_justified_at_every_width`

**Results**
- `uv run pytest` → **1543 passed, 15 skipped**, 0 failed
- `bash .agents/validate.sh` → **exit 0**: coverage **98.94 %** (floor 94 %), `ruff` All checks passed, `pyright` 0 errors, `node --check` clean, hygiene clean
- `uv run pytest -m live --live` (`.env.dev` sourced) → **11 passed**; `./scripts/dev-server.sh status` → 9/9 sources `ok`
- Browser: `node data/logs/25-task03-verify.cjs` → **235 checks, 0 FAIL**, exit 0 (`data/logs/25-task03-verify.out`, 16 `masthead-25-*.png`)

**Completion criteria**: badges pinned+measured (green `system/disks/llm/ups` on both admin pages, 1280 px and 360 px) ✅ · both rows flush right, three rows + dropdown unchanged (pinned + measured) ✅ · clock farthest-right (pinned markup; measured right edge 1260.0 at 1280 px, ≤900 px hidden by the long-standing readout rule with source order asserted) ✅ · pytest/coverage/ruff/pyright ✅ · validate.sh ✅ · no behaviour change in completed work ✅

**Docs**: README reviewed end-to-end against the code (served-route audit vs documented routes, env table vs `config.py`, commands vs `scripts/`, public-path table, `web/` masthead bullet) — consistent, no changes needed beyond the phase's own edits. AGENTS.md — two fixes above; spot-checked every doc-named test, function, script, threshold, `install-agent.sh` sha256 (`ffbcd91f…` matches) and agent version (1.1.0).

**Deviations**: left the pre-existing `fastapi 0.153.0 → 0.159.0` advisory (validate prints it as advisory; no dependency work in scope, AGENTS rule 8 triggers on touching deps). No live-infra changes. Overview's "all eight signed-in pages" for the clock is verified against seven templates + `/login` (which has no clock, pinned).

**Next pending phase**: `.agents/phases/todo/26_ups_event_split/` (3 tasks).
2026-10-21 16:42:14 -04:00
ducoterra 558a83d0be chore(agent): phase roadmap from TODO.md, 2 phases
Convert the gateway-visualisation TODO item into phases 28 (gateway_agent_proxy: agent sections for haproxy/nginx configs + live sockets, 1.2.0, capture, rollout) and 29 (gateway_dashboard_page: source, wiring, API, animated-pipe page, nav, docs). Clear TODO.md.

LLM-Generated: true
2026-10-21 16:24:01 -04:00
ducoterra 89d9fb9f89 phase: 24_dev_mtls_rollout
**Phase 24 final verification — all green**

- **Verified (read-only re-run, 62/62 PASS)** via `data/logs/verify-24-mtls.py`: nine sources `ok`; `/api/certs` all three `active` with `deployed_bundle_rev == bundle_rev == 2`; `agent.tls: true` + `agent.version 1.1.0` on every host; driveripper 16/16 drives live; gateway absent from `/disks` but `disks:gateway ok`; all served target URLs `https://`, no userinfo; pre-flip backup present; no transport/cert alert.
- **Criterion 1 (export):** `data/certs/{driveripper,deskwork,gateway}.bundle` re-checked byte-identical (sha256) to `GET /api/certs/<name>/bundle`, mode `0600`, under gitignored `data/`; live arms re-probed (issued→file, unknown/`issued:false`→one-line note + no file, wrong password→exit 2, nothing written).
- **Criterion 2 (`--fetch-bundle`):** 54 sandbox pins in `tests/test_deploy_scripts.py` pass — byte-identical `0640` install + TLS unit lines, refusals (unissued / 401 / non-bundle body / dead dashboard) before `need_root` with no write, token absent from token file, unit, stdout and sudoers hint.
- **Criterion 3–4:** `./scripts/dev-server.sh status` nine `ok` + bundles block; screenshots `data/logs/24-task04-{certs,disks,system}.png` exist.
- **Criterion 7:** no behavior change — `agent/` untouched, `src/` diff is a `certs.ensure_certs` docstring only; token-mode, revocation-before-dial and bundle-route tests green.
- **Tests/lint:** `bash .agents/validate.sh` → **EXIT=0**; `uv run pytest --cov --cov-fail-under=94` → **1542 passed, 15 skipped, coverage 98.94%**; `ruff check src tests agent` → All checks passed; `pyright` → 0 errors; `node --check` clean; `uv run pytest -m live --live` → **11 passed**.
- **Docs:** README.md (all 1144 lines) + AGENTS.md reviewed against the live project — every config.py env var is in README and both templates, route set / severity thresholds / test counts / coverage figure match, all repo paths, cited test names and `Dash.*`/`DashCharts.*` helpers resolve, phase-24 doc pins green. **Verified consistent — no changes needed.**
- **Notable:** left `fastapi 0.153.0 → 0.158.0` (validate reports it as advisory; no dependency was touched, rule 8 conditions the check on deps moving). Killed two abandoned `/tmp/fbtest` loopback stub processes left by task 02's manual probe; dev server unaffected.
- **Next pending phase:** `.agents/phases/todo/25_masthead_control_placement/`.
2026-10-21 13:47:45 -04:00
ducoterra 5f3ed9bcd4 phase: 23_llm_rate_responsiveness
Phase 23 final verification complete — all green.

```
VERIFIED (criteria)
1  /api/llm/snapshot windows ≈2 s live (1.98–2.00), /api/system window_seconds ≈10 s (9.996–10.005);
   /api/config llm.rate_window_seconds=2=LLM_INTERVAL_SECONDS; browser re-check ALL CHECKS PASSED
   (data/logs/23-window-{llm,system}.png — /llm prints Δ2.0s, /system footnote Δ10.0s)
2  Phase-13 pins unchanged: floor, 2×span ceiling, 8× jump gate + independent anchor — all pass
   (only import/span-line diffs in tests/; git diff confirms no assertion rewritten)
3  Live proof re-derived from artifacts: 780 rows, 0 unlabelled, windows 0.61–3.38 s, 5 starts +
   5 ends settled by poll 2 (0 unsettled), 1 m rollups max 42.3 vs raw 201.6, 0 over 300 tok/s;
   counterfactual 10 s window: median settle poll 3, |ln| 0.924 vs 0.037
4  uv run pytest: 1494 passed / 15 skipped · coverage 98.94% (floor 94) · ruff clean · pyright 0
   errors · node --check clean · bash .agents/validate.sh exit 0 · live 11 passed · store_perf 4 passed
   Dev server up, all nine sources ok. No git add/commit run; everything left in the working tree.

DOCS PASS — 4 inconsistencies fixed in AGENTS.md (last two runs above include them)
- UI conventions hero example still said the `.sub` footnote reads `Δ10.0s`; the only hero that
  prints a Δ is /llm's → now `Δ2.0s` on /llm, "at the current load" on /ups
- gap 9: "found 10 busy-edges" contradicted its own artifact (17 edges, 10 judged, 7 reported-not-
  judged) → 17; and the poll-2 ratios were a 3-of-5 cherry-pick → full five values both windows
- gap 12: the responsive-table fix is now scheduled — names 27_mobile_tables_and_audit
README, config table, .env.example, rates/prom docstrings, app/api bullets and the testing-policy
pins all re-checked against code and found consistent (store pragmas, budgets, throttle, fleet).

NOTES
- Advisory only, not phase 23: fastapi 0.153.0 → 0.156.8 available; dependencies untouched here,
  so rule 8's "update on purpose" is left to the operator.
- NOTES.md is empty in the working tree (its items became phases 24–27); left as-is, untracked-file
  rules forbid me removing it.
- Next phase: 24_dev_mtls_rollout (25–27 follow).
```
2026-10-21 08:04:21 -04:00
ducoterra 895f309f8e phase: 22_disks_hide_unconfigured
All green. Final report:

**Verified (phase 22 final pass — both task files already in `complete/`)**
- `api._disk_visible` (`is not False`) is the single gate, applied to the assembled host list in `/api/disks/snapshot` **and** `/api/summary`'s disks block before the fleet sum and the rows; `sources/disks.py`, `page-disks.js`, `poller.py`, `store.py` and `alerts.py` are byte-unchanged (`git diff --stat`: only `api.py`, `test_api.py`, docs).
- Re-ran the live browser pass `data/logs/22-task02-disks-live.cjs` against the running dev server: **34/34 PASS** — no gateway card/why-line, `2/2 hosts answering`, fleet header reads the API's `2 of 2` / `17-17-0-1`, deskwork `nvme0n1` OK 47 °C/19 202 h/11 %, driveripper 16 live cards + `13 uncorrectable` chip, both aria-labelled plots, `/` still 3 fleet rows. Screenshots + `22-task02-api.json` refreshed.
- `./scripts/dev-server.sh status`: all **nine sources `ok`**, `disks:gateway` included; live `/api/events?source=disks` still carries gateway transitions (10 of 45), proving the README/AGENTS claim the gate hides a row and not the job's history.

**Completion criteria**
- `false` host absent from both endpoints, `None` + `unreachable` stub present, fleet ≡ row count: **PASS** (6 pins in `tests/test_api.py`, incl. the single-hidden-host = 200-with-empty-list leg).
- Live `/disks` = deskwork + driveripper, header 2 hosts, cards unchanged, screenshot in `data/logs/`: **PASS** (re-run above).
- `uv run pytest` → **1477 passed, 15 skipped**; `--cov --cov-fail-under=94` → **98.94 %** (`api.py` 100 %); `ruff check src tests agent` → all passed; `pyright` → 0 errors; `bash .agents/validate.sh` → **EXIT 0**. `uv run pytest -m live --live` → **11 passed**.
- No behaviour change in completed work: **PASS** (unchanged files above; live numbers match the pre-phase fleet record).

**Docs** — README: `/disks` page-table row + the agent SMART paragraph now distinguish "cannot read drives" (dimmed cards kept) from "no smartctl" (host absent from the page/API lists). AGENTS.md: hermetic tally 1471→**1477**, and the `collectors.py`/`api.py` passages now note the `false` arm never reaches the `/disks` why-line while an overview alert from a hidden host's `disks` job still names it. Everything else (routes vs served table, env table vs `config.py`, commands, scripts/fixtures/tests references, masthead/POLL_MS/threshold/retention claims) **verified consistent**.

**Deviations** — `NOTES.md` left empty by the task runs (both notes are tracked: mTLS → phase 24 `todo/`, the admin-links bug → fixed in phase 20 task 04); `/disks`' `#hosts-empty` copy left as pinned markup (it already names smartmontools + the disk group, so an all-hidden fleet is still told the fix); validate's advisory `fastapi 0.153.0 → 0.156.4` untouched (no dependency changed this phase). No live fleet writes of any kind.

**Next pending phase:** `23_llm_rate_responsiveness`.
2026-10-21 03:27:08 -04:00
ducoterra 5e267737dc phase: 21_page_cleanups
**Phase 21 final verification pass** (all 3 task files already in `complete/`, working tree carries the implementation)

- **Verified** task 01: `#pause`/`Space`/`setPaused`/`_paused` gone — `grep -ri pause src/dashboards/web/` empty, 0 pause hits in all 8 served pages + 10 static assets; re-ran `data/logs/21-task01-nopause.cjs` → 30/30 PASS (hidden tab: 0 data fetches in 22 s, one poll 2 ms after return).
- **Verified** task 02: re-ran `21-task02-gpu-values.cjs` on the live fleet → all PASS (deskwork/driveripper show Model/VRAM/Pool/Temp at 16.3 px; gateway = 5 heroes, slot `hidden`, no `n/a`; hero order unchanged; GPU history chart still builds per host).
- **Verified** task 03: re-ran `21-task03-scroll.cjs` → 14/14 PASS (`class="data scroll"`, scrollWidth 1021 vs clientWidth 292 at 360 px, Rotate/Revoke/bundle reachable and hit-testable; unchanged at 1280 px). Screenshots refreshed in `data/logs/`.
- **Gate**: `uv run pytest` → **1471 passed, 15 skipped**; `--cov --cov-fail-under=94` → **98.94%**; `ruff check src tests agent` clean; `pyright` **0 errors**; `bash .agents/validate.sh` → **EXIT 0** (incl. `node --check`); `uv run pytest -m live --live` → **11 passed**; `/api/health` all 9 sources `ok`.
- **Docs fixed** (pre-existing + this phase): "all **eight** signed-in pages" was a miscount of `MARKUP_PAGES` (7 = 5 dashboards + `/settings` + `/certs`; `/login` is the 8th, unsigned) → corrected in `README.md`, `AGENTS.md` (×2) and the `tests/test_api.py` docstring/comment; `AGENTS.md` Commands hermetic tally 1470 → **1471**.
- **Docs reviewed, consistent**: Quickstart/commands/flags (`install-agent.sh`, `capture-fixtures.sh`, `dev-server.sh`), the env table vs `config.py`, every documented route vs the served route table, masthead/`/system` GPU/`/certs` scroll descriptions vs code, and all `data/logs/*` + `tests/fixtures/*` references.
- **Deviation**: `validate.sh`'s advisory `fastapi 0.153.0 → 0.156.4` left alone — no dependency was touched this phase, so AGENTS rule 8 ("check after touching dependencies") is not triggered; upgrading here would be an unrelated behaviour change.
- No live-infra changes made (repo-side + browser verification only).
- **Next pending phase**: `22_disks_hide_unconfigured`.
2026-10-21 01:44:52 -04:00
ducoterra 973b7299fe chore(agent): phase roadmap from TODO.md, 3 phases
Append phases 25-27 (masthead control placement, UPS event split, mobile
tables + full mobile audit) derived from TODO.md L3-L7; the TODO file is
cleared, its items now live in .agents/phases/todo/. Protocol B: PLAN.md
and AGENTS.md untouched.

LLM-Generated: true
2026-10-20 23:24:27 -04:00
ducoterra 6444a158f2 phase: 20_masthead_nav_polish
**Phase 20 final verification pass — all four tasks already complete; verified in full.**

- Verified: three-row masthead CSS (`.masthead` flex-wrap + two `flex-basis: 100%`, `min-height`, no `margin-inline-start`), un-sticky `position: relative` + `z-index: 40`, 180 ms `visibility/opacity/transform` slide (closed `nav.tabs` / open `.is-open`, both states in the file's single reduced-motion block), both admin links on all 8 signed-in pages, `/login` neither.
- `uv run pytest` → **1464 passed, 15 skipped** · `uv run pytest --cov --cov-fail-under=94` → **98.94%** (floor 94, unchanged) · `ruff check src tests agent` clean · `pyright` 0 errors · **`bash .agents/validate.sh` EXIT=0** (incl. `node --check` + repo hygiene) · `uv run pytest -m live --live` → **11 passed**.
- Browser (live dev server, invariant 10): `20-task01-rows.cjs` 121 PASS/0 FAIL · `20-task02-unsticky.cjs` 81/0 · `20-task03-slide.cjs` 32/0 · `20-task04-admin-links.cjs` 87/0 · phase-18 `nav-final-pass.cjs` 45/45, `nav-task03-verify` 77, `nav-task04-verify` 24/24 · phase-19 scripts 11/22/24/29 all pass. Screenshots refreshed in `data/logs/` (`masthead-rows*`, `masthead-unsticky*`, `slide-*`, `admin-links-*`). Dev server restarted; `/api/health` all nine sources `ok`.
- Completion criteria: 3-row bar @1280/360 ✓ · scroll leaves bar behind + dropdown under whole bar ✓ · slide out/in 180 ms, reduced-motion snap ✓ · `/certs`→`⚷ Certs`, `/settings`→`⚙ Account` before theme toggle, outside `.tabs` ✓ · pytest/coverage/ruff/pyright ✓ · validate ✓ · no regression in tab structure, token reveal, safe-area padding, four close paths, status dots, theme toggle ✓.
- Docs: **AGENTS.md** gap 12 still said phase 20 was "in `todo/`… admin-link half has landed" and attributed the wrap to the old fixed 58 px row — rewritten to the completed state (three rows/un-sticky/slide/both links; 24 still todo; 21's `/certs` `.scroll` noted as not reaching `/`'s table). **README.md** `web/` bullet enumerated the controls row without `⏸ Pause` and omitted `/login` from the two-row list — both fixed. Everything else (setup, commands, env table vs `config.py`, routes vs `api.py`, agent/installer flags, testing policy) verified consistent.
- Deviations: no dependency bump — `uv pip list --outdated` still reports the pre-existing advisory `fastapi 0.153.0 → 0.156.4` (deps untouched this phase, validate treats it as advisory); `NOTES.md` is empty in the working tree from an operator edit outside these tasks (both items live in phases 20/24) — left as-is; no live-infra changes made or needed.
- Next pending phase: `21_page_cleanups` (pause button removal, `/system` GPU values, `/certs` table scroll).
2026-10-20 22:49:37 -04:00
ducoterra 95df8b695b phase: 19_ups_events_expandable
**Phase 19 final verification pass — all tasks already complete; everything green, no code changes needed.**

- **Verified (all 4 task files present in `complete/`)**: `details.events-box` disclosure on `/ups` with no `open` attribute and `#events` inside it; lazy fetch (`if (state.events.open) await refreshEvents()`), toggle stale rule on `state.pollMs`, success-path-only `painted`/`fetchedAt`; `charts-box`-shaped CSS with the `> .body` padding restatement; 4 new served-byte pins in `tests/test_api.py`.
- **Criterion 1 (collapsed on load, no first-poll fetch)**: PASS — re-ran `data/logs/19-task04-verify.cjs` against the live dev server (real upsd, 2 UPSes): **28/28 PASS**, `0` `/api/events` requests across 2 closed polls vs 4 snapshot polls; expand → exactly 1 request, 50 rows with `Oct 20, 07:11` timestamps + severity chips + `1–50 of 200` pager.
- **Criterion 2 (cache/stale/poll-while-open)**: PASS — fresh re-open 2→2 requests, stale re-open 2→3, loop refreshes while open, close asks for nothing, failed fetch keeps rows and doesn't stamp the cache.
- **Criterion 3**: `uv run pytest` → **1458 passed, 15 skipped**; `uv run pytest --cov` → **98.94%** (floor 94, unchanged); `uv run ruff check src tests agent` → clean; `uv run pyright` → **0 errors**.
- **Criterion 4**: `bash .agents/validate.sh` → **exit 0** (incl. `node --check` of all page scripts + repo-hygiene gates).
- **Criterion 5**: `data/logs/ups-events-closed.png` (collapsed `▸ POWER EVENTS`, 0 events requests noted in `ups-events-verify.out`) and `ups-events-open.png` (`▾`, bounded 320px list, pager strip) regenerated and inspected this pass.
- **Criterion 6 (no regressions)**: full suite green incl. UPS contract/pager pins; `uv run pytest -m live --live` (with `.env.dev`) → **11 passed**; phase 18's `nav-final-pass.cjs` sweep re-run → **45/45** (this phase's markup/CSS didn't touch the masthead).
- **Documentation**: full README.md (980 L) + AGENTS.md (1992 L) review against the code — verified env templates vs `config.py`, API route table vs `api.py`, installer flags vs `deploy/install-agent.sh`, agent routes vs `host_agent.py`, test/coverage/live tallies, `data/logs` artifacts, and live fleet facts (`HOST_TARGETS` all `http://`, all three agents `version 1.0.0`, nine sources `ok`). AGENTS.md UI conventions and README's events paragraph already carry the disclosure/lazy/stale sentences from tasks 01–04. **Verified consistent — no changes needed.**
- **Notable**: made zero repo edits (working tree holds only this phase's task work). `uv pip list --outdated` couldn't complete (PyPI 503 for the project name) — no dependency moved this pass, so rule 8 is not triggered; validate.sh's earlier advisory still lists `fastapi 0.153.0 → 0.156.3`, deliberately not upgraded in a UI/verification phase.
- **Next pending phase**: `.agents/phases/todo/20_masthead_nav_polish/` (tasks 01–04).
2026-10-20 19:17:22 -04:00
ducoterra f9b1565abb chore(agent): phase roadmap from TODO.md, 5 phases
Convert the 11 TODO items to phases 20-24 (masthead/nav polish,
page cleanups, /disks hiding unconfigured hosts, LLM rate window,
dev mTLS rollout). TODO.md cleared; items now live in
.agents/phases/todo/.

LLM-Generated: true
2026-10-20 18:06:46 -04:00
ducoterra 10f86772ee phase: 18_nav_hamburger
**Phase 18 — collapse the nav bar into a hamburger: complete, all green.**

Verified (criteria):
- ≤640px: the tab links leave the bar, the ☰ is visible, and it opens a dropdown
  under the sticky masthead (`top: 100%`, `z-index: 41`, viewport-wide) with the
  five stacked 40px links; above 640px the one-row desktop masthead is unchanged
  and the button hidden — `data/logs/nav-task04-verify.cjs` 24/24 and
  `data/logs/nav-final-pass.cjs` 45/45; screenshots
  `data/logs/nav-hamburger-{closed,open}-360.png`, `data/logs/nav-masthead-desktop-1280.png` ✓
- The four close paths: the button again, a tab click (navigates and closes),
  Esc (focus back on the opener), and `matchMedia` resize back above the
  breakpoint (drops a stale `.is-open`; `aria-expanded` tracks state on every
  path through the single `setNavMenu()` funnel) ✓
- `uv run pytest` → 1454 passed, 15 skipped; `bash .agents/validate.sh` → exit 0
  (coverage 98.94% ≥ 94, ruff clean, pyright 0 errors, `node --check` + hygiene
  clean); served-page pins: button on all seven tabbed pages and nowhere on
  /login, CSS and binding compared on the 640px literal ✓

Real-device finding (mobile Chrome): the emulated 360px pass was blind to the
system top-edge zone — Android Chrome is edge-to-edge by default, so the sticky
bar started under the status bar and the first-row ☰ sat in a region the device
does not deliver taps to (unreachable, not unbound; the same symptom brain of
reese fixed in production). The ≤640px block now pads the bar by
`env(safe-area-inset-top, 0px)` (the block's `padding-block` value restated on
the top side), verified on the operator's phone; `env()` resolves to 0 on every
non-notch surface, so all other widths render byte-identical. The served-byte
pin's rule loop carries the line; CDP's `safeAreaInsets` override does not feed
`env()`, which is the honest limit of the emulated proof.

Docs: AGENTS.md *UI conventions* masthead entry extended with the three pins,
the drift-comparison, the safe-area rule, and the real-device verification
caveat; README's `web/` bullet names the ≤640px collapse with the desktop
masthead unchanged above the breakpoint.

Deviations: pinned the shipped token shape (`#nav-toggle { display:
var(--nav-toggle-display) }` + the `:root` token) instead of the task file's
literal `.nav-toggle { display: none }` — that literal loses to
`button.ghost { display: inline-flex }` (task 01 measured it); the test
docstring says so, so nobody "restores" the broken shape. `uv pip list
--outdated` still advisory (fastapi 0.153.0 → 0.156.2); this phase moved no
dependency. NOTES.md (emptied) / TODO.md (operator items added) are
harness-side changes — left in the working tree, as in phase 17.

Next pending phase: `.agents/phases/todo/19_ups_events_expandable/`.

LLM-Generated: true
2026-10-20 16:47:10 -04:00
ducoterra b02c13f7d3 phase: 17_installer_device_filter
**Phase 17 — final verification pass: complete, all green.**

Verified (criteria):
- `grep -n "DevicePolicy\|DeviceAllow" deploy/install-agent.sh` → no match; `PrivateDevices=false` retained (line 472) ✓
- `install-agent.sh --help` names `--as-root`/`--caps` with the kernel-gate sentence; `--force` documented ✓
- Fresh fake-root install = pre-phase unit minus the filter lines: `test_a_default_install_unit_is_the_pre_phase_unit_minus_the_filter_lines` + `test_the_template_unit_has_no_device_filter_lines` pass ✓
- Re-run that differs → exit 1 with diff on stdout, `--force` applies, identical re-run silent: `test_a_re_run_that_differs_refuses_without_force_and_applies_with_it` (+ 3 sibling guard pins) pass ✓
- driveripper reproduction — verified **read-only**, no live change: `systemctl cat` shows `User=root`, `CapabilityBoundingSet=CAP_SYS_RAWIO CAP_SYS_ADMIN`, no filter, same `ExecStart`; `active`, `CapEff 0000000000220000`; `/api/disks/snapshot` → 16/16 drives with temperatures ✓
- `uv run pytest` → **1451 passed, 15 skipped**; `bash .agents/validate.sh` → **exit 0** (coverage **98.94%** ≥ 94, `ruff` clean, `pyright` 0 errors, `node --check` + hygiene clean); `uv run pytest -m live --live` → **11 passed**; dev server 9/9 sources `ok` (one initial live failure was a transient gufo timeout, clean on re-run)

Docs (inconsistencies found + fixed, incl. pre-existing):
- `AGENTS.md` fleet record said driveripper's unit runs `User=dashboards-agent` while the same bullet (and the live host) says `User=root` — rewritten to separate the installed user from the running one.
- `AGENTS.md` gap 10 header still claimed "the installer template is what is left" → now records phase 17 as the fix.
- `README.md` "with no capabilities" → "with an empty `CapabilityBoundingSet` (unless `--caps`…)", since this phase added the capability flag.
- `README.md` battery-test "gated four times" now notes the two session gates in front, matching AGENTS invariant 1.
- `.env.example` was missing three README-documented settings (`DASH_ENV_FILE`, `DASH_RELOAD`, `NUT_SIMULATE`) and `AGENT_CERT_BUNDLE` in the agent-variables note — added, annotated; all template values re-checked against `config.py` defaults.

Deviations: no live `--force` run on driveripper (needs the Oct-18 sudo-staged installer re-staged — it lacks the flags — and would push agent 1.1.0 mid phase-15 rollout); command + expected diff + post-checks recorded in AGENTS/gap 10. `uv pip list --outdated` still advisory (fastapi 0.153→0.156), untouched as in tasks 02/04 — this phase moved no dependency.

Next pending phase: `.agents/phases/todo/18_nav_hamburger/`.
2026-10-20 13:48:40 -04:00
ducoterra 7e367a9014 phase: 16_agent_smart_pass_budget
**Report — phase 16 final verification pass** (`16_agent_smart_pass_budget`)

- Verified both tasks' work is present and non-tautological: `_smart_pass` scales `max(20 s, 5 s × disks)` with the override winning; `tests/test_agent.py` pins the 16-disk scale (no truncation), the 20 s floor, and the exact `pass budget of 80s exhausted` line, plus the untouched `0.0`-override test. No code change this pass (only the docstring reflow already in the tree).
- Completion criteria:
  - `uv run pytest tests/test_agent.py` → **105 passed** (4 pass-budget tests green, `-k pass_budget` → 4 passed) ✅
  - `grep -rn "20 s per pass" AGENTS.md README.md agent/ deploy/` → **no matches** (exit 1) ✅
  - `bash .agents/validate.sh` → **EXIT=0**: `uv run pytest --cov --cov-fail-under=94` **1427 passed, 15 skipped, coverage 98.94 %**; `ruff check src tests agent` clean; `pyright` 0 errors; `node --check` + repo hygiene clean ✅
  - `uv run pytest -m live --live` (with `.env.dev`) → **11 passed**; `./scripts/dev-server.sh status` → nine sources `ok`; live `/api/disks/snapshot` → driveripper **16/16 drives with temps, no budget line** ✅
  - driveripper md5 byte-identity → **not met, deliberately no live change**: deployed copy is 1.0.0 and differs from the repo only by phase 15's TLS block (verified by diff); `ops` there *is* sudo-granted a staged `install-agent.sh` (new finding), but re-running it rewrites that host's hand-diverged `User=root`+`CapabilityBoundingSet` unit and re-breaks the 16-drive SMART read (gap 10) — needs root + phase 17's template fix. Recorded in AGENTS.md gap 11.
- Docs (full README + AGENTS review, incl. pre-existing drift): fixed 4 inaccuracies — AGENTS fleet record claimed *neither* host sudo-grants the installer (driveripper does, and that grant is the trap); `/health` publishes `version`, not `agent_version`; deskwork's "55 °C amber" marked as a dated capture (48 °C `ok` on re-read); gap 10's title still read as an open defect though resolved the same day. README verified accurate end-to-end (SMART budget wording, TLS rollout, config table, routes, test philosophy) — no further change.
- Notable: left `NOTES.md` (untracked operator scratch) alone; its "admin buttons disappear on `/certs`/`/settings`" item is deliberate, test-pinned behaviour (`test_every_page_links_to_both_admin_pages`) and belongs to a future phase, not this one.
- Next pending phase: `17_installer_device_filter`.
2026-10-20 11:14:35 -04:00
ducoterra 4b0f53c499 chore(agent): phase roadmap from TODO.md, 2 phases
Convert the two TODO items into append-mode phases (Protocol B):
18_nav_hamburger (TODO L3) and 19_ups_events_expandable (TODO L4),
numbered after the existing todo phases. TODO.md is cleared to an
empty list; its items now live in .agents/phases/todo/.

LLM-Generated: true
2026-10-20 10:37:30 -04:00
ducoterra cffbac5771 phase: 15_agent_mtls
All green. Report:

**Verified / implemented (final pass of `15_agent_mtls`)**
- Wrote `data/logs/verify-15-tls-e2e.py`: a real `host_agent.py --cert-bundle` + a real dashboard process over a scratch DB (`HOST_TARGETS=loop=https://…`) — **27/27 PASS**, transcript `verify-15-tls-e2e.out`, notes `data/logs/NOTES-tls-rollout.md`.
- Read-only checks against the running dev store: 5 `certs.*` rows byte-identical across a restart; served bundle == the `meta` row (1430 B); 0 events containing `-----BEGIN`; unsigned `/certs`→`303 ?next=`, unsigned `/api/certs`→`401`, signed-in `200`; screenshots `data/logs/15-final-{certs,system}.png`.
- Docs (step 6, full review): README — added *Protecting an agent, host by host* (the rollout task-07 left out), a `tests/test_certs.py`/TLS-matrix/sandbox-installer paragraph, and fixed two stale “the one write endpoint in the app” claims (`NUT_ALLOW_ACTIONS`, Battery-test buttons). AGENTS.md — test count 1424→1427, bundle-fetch command in *Commands*, a Testing-policy bullet on structurally pinning a key-holding module, fleet transport bullet (token mode is now the compatibility path), gap 11 final state.

**Tests / lint (full suite 100% green)**
- `uv run pytest` → **1427 passed, 15 skipped**, exit 0 · `bash .agents/validate.sh` → **exit 0** (coverage **98.94%** vs 94% floor, `certs.py` 100%; ruff “All checks passed”; pyright 0 errors; `node --check` clean; hygiene clean)
- `uv run pytest -m live --live` → **11 passed / 0 failed** · `./scripts/dev-server.sh status` → all 9 sources `ok`
- `uv pip list --outdated` → empty after bumping `fastapi 0.152.0 → 0.153.0` (`uv.lock` only); full suite re-run green.

**Completion criteria**
- Fresh-DB root CA, idempotent: ✅ (tests + live store hashes) · agent TLS matrix incl. wrong-CA/certless 403/`/health` CA-only/revoked 403 + live denylist re-read: ✅
- `https` `host:`/`disks:` `ok` + revoked says “agent certificate revoked” with no dial: ✅ on loopback (real agent + real dashboard) and spy-pinned hermetically; **not** on the fleet (below)
- Signed-in `/certs` + rotate/revoke/root-rotate + byte-identical bundle download: ✅ · installer bundle/unit/`--status` over TLS: ✅ (0640, see deviation)
- `pytest`/`validate.sh`/`--outdated`: ✅ · token-mode byte-identical, engine/UPS untouched, public surface unchanged (route walk): ✅
- **Live fleet TLS (deskwork+gateway flip): NOT done** — no live changes. It needs root on hosts without the installer grant (`deskwork`), has no both-ways-working window (`http://` dashboard vs https-only agent), and would deploy the agent from this not-yet-committed tree. Exact procedure recorded in README + `data/logs/NOTES-tls-rollout.md`.

**Deviations**: bundle mode `0640 root:dashboards-agent` instead of the task file’s `0600` (the agent re-reads its own bundle for the denylist; 0600 breaks the unit — already recorded in AGENTS.md); the loopback dashboard substitutes for the fleet flip. Incidental: a stray `pkill` of mine killed the dev server mid-pass; restarted and verified healthy.

**Next pending phase**: `.agents/phases/todo/16_agent_smart_pass_budget` (then `17_installer_device_filter`).
2026-10-20 10:02:16 -04:00
ducoterra 569ca25fdb phase: 14_store_performance
**Phase 14 `store_performance` — final verification pass (all 4 tasks already in `complete/`)**

- Re-verified every task's work in the tree: read caches (`series_names`/`stats`), capped compound `pick_res_many`, `history_many` + `api._history_batch`, windowed `maintain()` + per-resolution prune + `PRAGMA optimize`/`automatic_analyze=0`/WAL pragmas, `store_perf.BUDGETS` (14 ceilings), probe, live watch.
- **Completion criteria:** baseline+after side by side in `data/logs/NOTES-store-perf.md` ✅ · budget tests green (`uv run pytest -m store_perf --store-perf` → **4 passed**, 14/14 inside) ✅ · 25-min dev-server watch re-run **independently** (74 passes, `scripts/watch-store-perf.py`): **0 `store-error`** (total still 190, newest 2026‑10‑18 06:33:30), WAL 7.77 MB ≤ 67.1 MB, medians 1.7–3.2 ms, p95 720/613 ms = the documented cold-cache window; a 3rd 27-pass watch on a freshly restarted server reproduced it (3.1/708, 1.8/572) ✅ · `maintain()` 5,368 ms → 2.7–6.8 ms and WAL bounded across two cycles (test + notes) ✅ · no behaviour change (all four history routes pinned against `query()`) ✅
- **Full gates:** `uv run pytest` → **1267 passed, 15 skipped**; `bash .agents/validate.sh` → **exit 0**, coverage **98.75%** (floor 94), `ruff check src tests agent` clean, `pyright` 0 errors, `node --check` ok, repo hygiene ok; `uv run pytest -m live --live` → **11 passed**; dev server up, 9/9 sources `ok`, pages 200 in 1.5–3.3 ms.
- **Docs fixed:** README/AGENTS still claimed a flat "20 s per pass" while `host_agent.py` (in this same uncommitted change) scales the budget to 5 s × disk count — corrected in both plus the two illustrative "20 s of wedged disk" sentences; broken bold (`` `driveripper** ``) repaired; stale gap-4 title + "tuning is its own phase" wording updated to say phase 14 ran; README's absolute "nothing … is answered by a whole-table scan" qualified to *per call* with the cold-window cost named; "third opt-in marker" → second; conftest `--store-perf` help named the real marker; **`data/logs/watch-store-perf.py` moved to `scripts/watch-store-perf.py`** (both docs told a clone to run a gitignored path) with root-anchored paths, `DASH_WATCH_OUT`, and dead code removed.
- **New finding recorded:** gap 9's "~10 s process-wide stall is still gap 4's / `14_store_performance`'s problem" is now closed — coincident >8 s `llm` + >12 s `ups` raw-stream gaps went 393 (00:00–12:00) and 198 (12:00–21:36) to **0** across 110+ min of the phase-14 store; written into AGENTS.md gaps 4 and 9 and the notes table.
- **Deviations:** did *not* pull either deferred cost knob (`STATS_CACHE_TTL_SECONDS`, warming caches from the maintenance tick) — AGENTS.md records that decision and its reasoning; `uv pip list --outdated` still reports certifi/fastapi/starlette/websockets (pre-existing advisory, no dependency touched this phase, gate treats it as advisory). Overwrote nothing: task 04's watch preserved as `store-perf-watch.task04.*`, my two runs as `.verify25.*` / `.current-code.*`.
- **Next pending phase:** `15_agent_mtls`, then `16_agent_smart_pass_budget` (owns the scaled-budget tests + its doc criterion, now partly pre-satisfied by this pass), `17_installer_device_filter`.
2026-10-19 23:44:35 -04:00
ducoterra 1bbd868401 phase: 13_llm_page_and_decode_rate
All green. Final verification summary:

**Verified / implemented (phase 13 final pass)**
- All 4 tasks already in `complete/`; this was the verification pass. Confirmed the CSS rhythm rules (`#strip`, `#banners:empty ~ #fleet`) are served and pinned, `data/logs/NOTES-decode-spike.md` names the mechanism, and the phase-07/04 surfaces still pass their pins.
- **Found and fixed a residual defect in this phase's own code**: a stalled fetch stamps its body with the poll's *finish* time, so that sample anchors every window for the rest of the span; the jump gate "confirmed" a jump measured from that same anchor → live gufo rows 232.28 tok/s (one request at ~50) + a decaying 146/114/95 tail, repeated at 200.6. `rates.py::_publish` now requires an **independent anchor** (`_pending` stores the base sample's ts; the hold continues until it ages out). Pinned by `test_a_held_jump_is_not_confirmed_by_a_window_on_the_same_anchor` + `test_gufo_a_stale_scrape_body_never_becomes_a_rate_above_the_engine` (both fail on task 03's gate).
- Re-verified live after a dev-server restart: 10 min / 540 signed-in `/api/llm/snapshot` polls + read-only SQLite pass; **0 values > 300, 0 without a `Δ<window>` label**; 0 post-fix stored rows over the ceiling except the two pre-fix-shape rows above.

**Tests / lint (exact, all 100 % green)**
- `bash .agents/validate.sh` → **EXIT=0**; `uv run pytest --cov --cov-fail-under=94` → **1195 passed, 11 skipped, coverage 98.70 %**; `ruff check src tests agent` → All checks passed; `pyright` → 0 errors; `node --check` clean; repo hygiene clean.
- `uv run pytest -m live --live` → **11 passed / 0 failed**. `/api/health` → **all nine sources ok**.

**Completion criteria**
- Fleet/strip + head/fleet gap with screenshots (`llm-fleet-spacing{,-top,-banner-case}.png`) — met (CSS unchanged this pass, pinned on served bytes).
- Task-02 findings file names the confirmed mechanism — met.
- `test_rates.py`/`test_llm_source.py` pin label-set, both window bounds and the always-honest `Δ` — met (+2 new pins).
- Sustained watch under the recorded ceiling (300 tok/s) — met (worst 203.05, all labelled).
- Full suite + validate green; no phase-07/04 behaviour change — met.

**Documentation**
- README: `RATE_WINDOW_SECONDS` row now states the hold/confirm rule (only change needed; rest verified consistent).
- AGENTS.md: `rates.py` bullet + gap 9 rewritten (residual closed *and* the remaining sub-8× skew measured, root-caused to gap 4, explicitly not fixed by inventing a new threshold).
- **Fleet facts changed under this pass and were corrected**: operator added driveripper's bearer token → `host:driveripper`/`disks:driveripper` `ok`, live suite 9/2→11/0; but driveripper reads **identity only, zero SMART numbers** (root `smartctl` gives 55 °C/14 %; `/dev/nvme0`, `/dev/ng0n1`, `/dev/bsg/*` are root-only; its unit is `DevicePolicy=auto`, not the installer's `strict`). Evidence appended to `data/logs/NOTES-agent-install.md` + `NOTES-decode-stable.md`.

**Notable decisions / deviations**
- Did **not** add a span-relative window floor: that would re-decide task 03's pinned `MIN_WINDOW_SECONDS` and its "in-bounds diluted row publishes" test. Recorded as the next phase's call, with the root cause left at gap 4 / `14_store_performance` (the ~10 s stall).
- No live fleet change made (read-only probes + one dev-server restart). Fixtures deliberately not re-captured (no parse change, task 02 proved no shape drift). Dependency-currency advisory left untouched (no dependency moved in this phase): certifi/fastapi/starlette/websockets have newer releases.

**Next pending phase:** `.agents/phases/todo/14_store_performance` (then `15_agent_mtls`).
2026-10-19 14:53:58 -04:00
ducoterra 04d934dd94 chore(agent): phase 15_agent_mtls from TODO.md (protected exporter web servers)
The single TODO item (exporters as protected web servers, certs as auth,
root certificate generated and managed by the collector) is now phase 15:
cert management in meta, agent TLS mode, https collector dials, gated
/api/certs surface, /certs admin page, installer + live rollout.

LLM-Generated: true
2026-10-19 13:25:45 -04:00
ducoterra 29524b3ebc re-add privileged actions to todo so we can re-plan the phase 2026-10-19 10:17:09 -04:00
ducoterra 28139df6d6 phase: 12_nut_self_test_verdict
Final state verified. Report:

**Verified (phase 12 — all 3 tasks already complete; this was the verification pass)**
- Re-ran the wire capture live: `ups.test.result` = `No test initiated` (apc) / `Done and passed` (cyberpower) — matches `LIVE_SELF_TEST_WORDS` + both fixtures; `/api/summary` has **no** self-test row, `/api/ups/snapshot` reads `cyberpower ok | Done and passed`, `apc unknown | …run a battery test`; both screenshots read back correct.
- Fixed a defect found in prior work while checking the "unrelated live noise" note: `NutMonitor.collect()` reused one upsd socket after a *partially-read* reply, so a 4 s stall produced bogus `expected 'BEGIN LIST VAR', got 'RW apc battery.charge.low "10"'` errors (same hazard on `INSTCMD`, where a late `OK` could be read as a different command's success). `NutClient` now marks a socket whose reply stopped early as `UNUSABLE`, naming the command that stalled; a complete `ERR …` line still leaves it usable (the per-UPS tolerance `collect()` depends on). +3 hermetic pins against the fake upsd; live re-check (09:33–09:44) now logs one line per stall, no cascade.

**Tests / lint / coverage (all green)**
- `bash .agents/validate.sh` → **exit 0**; `uv run pytest --cov --cov-fail-under=94` → **1172 passed, 11 skipped**, coverage **98.63 %** (floor 94); `uv run ruff check src tests agent` clean; `uv run pyright` 0 errors; `node --check` clean. Modified modules: `sources/nut.py` 95 %, `collectors.py` 97 % (no new line uncovered).
- `uv run pytest -m live --live` → **8 passed / 3 failed**, all three the pre-existing `driveripper` agent-transport failures AGENTS.md records (verified still true: unit `inactive`, no installed file, nothing on `:9120`).
- `./scripts/dev-server.sh status`: `nut ok`, `llm:* ok`, `host/disks:deskwork|gateway ok`; the two `driveripper` jobs DOWN.

**Completion criteria**
- Live capture recorded + asserts `ok` — met (re-confirmed from the wire today).
- `/ups` renders `ok`, overview silent, screenshots present — met.
- pytest / cov / ruff / pyright / validate — met (above).
- No other `health.checks[]` behaviour changed — met (diff confined to the `test_result` branch; all prior nut/ups-contract tests unmodified).

**Documentation**: README verified consistent (self-test wording, config table cross-checked against `config.py`, all documented commands/flags exist) — no changes needed beyond phase tasks' edit. AGENTS.md fixes: stale test count (1140 → 1172), `nvme0n1` power-on hours 19 158 → 19 160, gap 4 store-error claim corrected (last stored `store-error` 2026-10-18 06:33, none in the previous 24 h), new `collectors.py`/`sources/nut.py` socket-hygiene note, new gap-1 bullet on the upsd stall, plus live evidence appended to `data/logs/NOTES-self-test.md`.

**Deviations**: made no live *changes* (read-only probes only) — `driveripper`'s agent install needs root (`ops` has no installer/visudo grant) and a battery test stays the operator's call; the upsd 4 s stall itself is upstream-side. `uv pip list --outdated` still advisory (certifi/fastapi/starlette) — dependencies untouched this phase.

**Next pending phase**: `13_llm_page_and_decode_rate`.
2026-10-19 09:54:03 -04:00
ducoterra 1734d81e48 phase: 11_admin_login
**Verified (final pass, phase 11 — all 7 tasks already in `complete/`)**
- Gate from the wire on a restarted dev server: `/system`→`303 /login?next=/system` (`/llm?range=1h`→`/login?next=/llm%3Frange%3D1h`), `/api/summary`+`/api/config`→`401 {"ok":false,…}`, `/api/health`→`200`; signed-in `GET` of all six pages `200`, `/login`→`303 /`.
- Auth flows on a throwaway server (`:8199`, `/tmp` DB — dev `auth.verifier` untouched): wrong pw `401`, `testpass` `200`, password change `200`, **old cookie `401` / new cookie `200`**, old pw `401` / new pw `200`, short new `422`, wrong current `403`, cookie write without CSRF `403`, logout `200`; 5 failures then `429 retry-after: 60`; `login-denied`/`password-changed`/`password-denied` events; `meta` = the two auth keys.
- Fail-closed boot (no seed, empty DB): **exit 3**, message names `DASH_ADMIN_PASSWORD`, `auth.verifier`, and the `DELETE FROM meta …` reset. `dev-server.sh status` + the Containerfile `HEALTHCHECK` python one-liner both `200`/exit 0 unauthenticated.
- Live payloads unchanged (phases 01–10 walk): `/api/summary` keys + 8 alerts + hosts 3/2 live; UPS `simulated:false` with charge/load/runtime severity; deskwork `nvme0n1` 51 °C/19 159 h/11 % wear; gateway `smartctl_available:false`; driveripper `unreachable` stub.

**Tests/lint (all green)**: `bash .agents/validate.sh` → exit 0 — `1140 passed, 11 skipped`, coverage `98.62%` (floor 94), ruff `All checks passed!`, pyright `0 errors`, `node --check` clean, repo hygiene clean. `uv run pytest -m live --live` → `8 passed, 3 failed`, all `driveripper: 10.1.0.126:9120 … ConnectError` (pre-existing, auth-free; probe confirms nothing listening).

**Completion criteria**: all met except the live-suite caveat above — installing the driveripper agent needs root on that host (no `ops` grant), so per the live-infra rule I made **no live change**; task 07's five screenshots (`auth-login/-rejected/-locked-out/-signed-in-overview/-settings.png`) remain the browser evidence, and the login/rotation/lockout flows were re-proved over HTTP this pass.

**Docs**: README — fixed "only place the CSRF token is published" (login/password responses mint one), added the undocumented `DASH_RELOAD` row, stated that Sign out ends only this browser's cookie. `AGENTS.md` — new known gap 8 (stateless logout cannot revoke a copied token), gap-4 dev-store sizes dated, fleet section notes `data/` is gitignored. `api.py` — same `status` docstring imprecision fixed. Verified the 4 `DASH_*` rows, reset command, seed-wins rule and invariant 12 against `app.py`/`config.py`.

**Deviations**: no dependency bump (`uv pip list --outdated` advisory: certifi/fastapi/starlette — phase added none); verification notes appended to `data/logs/NOTES-auth.md` §10.

**Next pending phase**: `12_nut_self_test_verdict`.
2026-10-19 07:45:44 -04:00
ducoterra eb4788cba1 phases: convert TODO.md to phases 12-15 (store, NUT verdicts, LLM page, TLS)
The five TODO items become four phases: 12 NUT self-test verdict wording
(assess() passes through 'Done and passed' to warn), 13 LLM page fleet/strip
spacing plus the decode tok/s spike (labelled counter sums, unbounded
estimator span), 14 store performance (full-scan read paths, maintain()
rollup write window, WAL tuning), 15 NUT privileged-action security
(collector-managed root CA, STARTTLS, account gate documented).

TODO.md is cleared; .agents/phases/todo/ is the source of truth.

LLM-Generated: true
2026-10-19 01:36:07 -04:00
ducoterra a08e693698 phase: 10_dev_direct_dial
**Phase 10 final pass — repo green; live criteria blocked off-repo.**

- Verified: `gateway` runs `dashboards-agent` (active/enabled, `10.1.1.27:9120` → `/health` 200, `/snapshot` 401 no-token / 200 bearer on-host, token matches `/etc/dashboards-agent.token`); `driveripper`/`deskwork` inactive, `sudo -n` → "a password is required", `:9120` empty.
- New measurement (transient runtime rule on gateway `firewall-cmd --add-port=9120/tcp`, **reverted** — runtime+permanent port lists verified back to the original set, no rich rules): laptop still gets no SYN through, while `:22`/`:80` on the same address answer in ms ⇒ the block is the lab gateway `10.2.0.1` port filter, not this app/host firewall/agent. No other live change made (no sudoers, no upsd, no UPS command).
- Tests/lint (exact): `bash .agents/validate.sh` → **EXIT 0** — `uv run pytest --cov --cov-fail-under=94` **793 passed, 11 skipped, coverage 98.33 %**; `ruff check src tests agent` clean; `pyright` 0 errors; `node --check` clean; hygiene clean; `uv pip list --outdated` empty (bumped fastapi 0.143.3→0.143.5, `uv.lock` only).
- `uv run pytest -m live --live` (`.env.dev` sourced): **4 passed, 1 skipped, 6 failed** — all six are `:9120`/`:3051` connect timeouts; zero parse/contract failures.
- Dev server restarted on current code: engines `ok`; 3×`host` + 3×`disks` + `nut` `DOWN` with real reasons (designed degradation, stub rows intact, no credential leakage).
- Completion criteria: screenshots ✓ (`data/logs/phase10-{index,system,disks,llm,ups}-live.png`); tunnel-vocabulary grep ✓ outside `.agents/` history (PLAN.md locked line, phase overviews, past reports; fixtures/`uv.lock` hits are digit substrings — `tests/test_deploy_scripts.py` guards operational files); no page-payload change ✓ (contract tests). **Not met, operator-side:** agents on all three hosts, `LIST UPS` from the laptop, `dev-server.sh status` all-ok, live suite green.
- Fixed in this pass: `sources/nut.py` unreachable-upsd message no longer prints a bare `(TimeoutError: )` (now `(timed out after 4s)` / `(ConnectionRefusedError: …)`), pinned by a new test + extended refused-port test.
- Docs: README verified consistent (every env var in `config.py` documented, routes/flags/`--with-llm`/pager sizes checked) — no changes needed. AGENTS.md fixed 4 inconsistencies: per-host sudo reality (gateway grants `NOPASSWD ALL`, which is why it installed without `agent-sudoers`; driveripper scoped; deskwork unrelated), the now-proven firewall finding, `/system` table typo ("agent per host"), gap-4 DB/WAL sizes + test count. `.env.example` `upsd.users` sample now one `instcmds` per line (it contradicted README and the in-app copy). `.env.dev` (gitignored) comment corrected to match the evidence.
- Flagged, untouched (locked): `.agents/PLAN.md` L92–95 still says upsd is "forwarded to `127.0.0.1:3494` in dev" — false after this phase; evidence + exact operator commands appended to `data/logs/NOTES-verify.md` §11.
- Next pending phase: `11_admin_login`.
2026-10-18 05:44:13 -04:00
ducoterra fefbeb8227 phase: 09_disks_smartctl
**Phase 09 — final verification pass (all 4 task files already in `complete/`; nothing re-implemented)**

- Verified `GET /api/disks/snapshot` live: 3 hosts / 18 whole-`/sys/block` rows, `smartctl_available` three-way, per-row `severity`, API-summed `fleet`; 404 `{"configured": false}` proven *without dialling* (spies on `httpx.AsyncClient.get` + `collectors.fetch_snapshot`).
- Verified `/api/disks/history` live (`disk.temp_c@dev`, unknown names → empty series) + `test_store.py`'s 300 s rollup/prune buckets; `/api/metrics?source=disks` answers the `disks` family.
- Browser against live data: 18 drive cards (never a partition/mount), collapsed per-host `Charts` with `aria-label`, 3 `.stale` grids + one `.banner.info` why-line and **0 `.fail` classes / green `disks` dot** → criteria 3 and 4 hold; new shots `data/logs/09-verify-disks-live-dimmed.png`, `09-verify-overview-live.png` (13 total for the phase).
- Real-hardware criterion: read 4 driveripper drives through `ops`' `sudo -n smartctl -i -A -H` and ran the live output through `host_agent.parse_smart_output` → `sources/disks.parse_disks_snapshot`: `sda` 35 °C / 30 423 h, `nvme1n1` 58 % wear, `nvme5n1` 13 uncorrectable → `alarm_disks: 1`, temps graded `warn`; consistent with the 16 committed fixtures.
- `bash .agents/validate.sh` → **EXIT=0**; `uv run pytest --cov` → **769 passed, 11 skipped, coverage 98.33%** (floor 94); `ruff check src tests agent` clean; `pyright` 0 errors; `node --check` clean; hygiene + dependency advisories clean. `uv run pytest -m live --live` → 10 passed, 1 skipped (agents still predate the SMART pass). Dev server healthy: `/api/health` `ok`, 9/9 sources.
- Docs fixed: AGENTS.md test count 698→769 and coverage 98.2→98.33 %; `disks` added to the event dashboard-name list and to the "every feed renders `Dash.eventText`" sentence; README `POLL_MS_*` row now states `/disks`' deliberate fixed 30 s; new Known gap 4 records the dev-DB `database is locked` writes (reported/recovered per invariant 4, ~590 MB DB / ~70 MB WAL, not reproduced by a controlled 25 s external read txn).
- Deviations: **no live-infra change** — granting the agent user the `disk` group and redeploying `host_agent.py` on driveripper/deskwork stay the documented paired operator action (redeploying without the grant yields 16 permission-denied lines; a sudo-spawning agent would break the confirmed "direct read, no sudo" assumption). Store-lock issue left to its own phase rather than patched speculatively.
- Next pending phase: `10_dev_direct_dial`.
2026-10-18 00:45:01 -04:00
ducoterra 19cfedfddf phase: 08_events_paging_and_copy
All green. Final report:

**Verified (phase 08 — both task files already in `complete/`; this was the final verification pass)**
- `Dash.eventsPager()` in `app.js` (50/page, `1–50 of N` strip, aria-labelled `‹ prev`/`next ›` disabled at the ends, `WeakMap` page state, reset only when the event *set* changes, `EVENT_FEED_LIMIT`=200) with three call sites; `ul.events` bounded to 320px + `overflow-y:auto`; `/api/events` untouched (`le=500`, no silent clamp).
- Live browser against the real fleet: `/system` `1–50 of 121`, `/llm` `1–50 of 200`, `/ups` `1–50 of 200`; container 320px vs scrollHeight 1543/2368; page 2 → `51–100 of 200` and survived two polls; no empty `.sub`/`.faint` container visible on any page; footers carry only measurements (`rates over Δ11.9s · poll 10s`). Fresh artifacts: `data/logs/08-final-{system-head,llm-page,index-page,ups-feed-page2}.png` (32 phase screenshots already present).
- Grep of live served bytes for all 13 deleted flavour fragments + `actions = -`: none survive in HTML or JS.
- `AGENTS.md` carries both new UI-convention rules (flavour-text rule with the `TODO.md:7` cited examples; the event-feed-window rule) and the updated test count.

**Tests / lint (full project 100% green)**
- `bash .agents/validate.sh` → **EXIT=0**: `uv run pytest --cov --cov-fail-under=94` → **624 passed, 9 skipped, coverage 97.91%**; `ruff check src tests agent` → All checks passed; `pyright` → 0 errors; `node --check` ×5 clean; repo hygiene clean.
- `uv run pytest -m live --live` (`.env.dev` sourced) → **9 passed**; `uv pip list --outdated` → empty. Re-ran pytest/ruff/pyright/node after the doc edits → all exit 0.

**Completion criteria**: bounded scroll + 50/page + count footer ✅ (all three feeds, live) · no removed flavour string ✅ · AGENTS.md rule ✅ · `validate.sh` green + screenshots in `data/logs/` ✅.

**Documentation**: full README/AGENTS review against code (env table vs `config.py`/`.env*.example`, script flags, Containerfile uid 10001/8090, chart heights 120/110/160, test count, coverage) — one inaccuracy fixed: README said "Every page's event feed" (the overview has none) → now "The three event feeds (`/system`, `/llm`, `/ups`)"; also restored the missing trailing newline in `TODO.md`. Everything else verified consistent.

**Notes**: paging stays client-side (no API change); the task-02 defect fix in `/ups` setup copy (`NUT_HOST`/`NUT_PORT`, `LISTEN 127.0.0.1`, one `instcmds` per command, never `actions = -`) re-verified against README and live bytes. Pre-existing items intentionally left to their own phases: `ups.test.result: "Done and passed"` rated `warn` (operator's `TODO.md` entry) and the dev-DB `database is locked` retention issue.

**Next pending phase**: `.agents/phases/todo/09_disks_smartctl/`
2026-10-17 19:32:36 -04:00
ducoterra 3d5ccd7a39 phase: 07_llm_monitor_style_pages
**Phase 07 final verification pass — all tasks already in `complete/`; every criterion re-verified live.**

- **Old slot table gone**: no `#slots-card`/`#slots-table`/`renderSlots` in HTML/JS/CSS (only the new grid's `.slots-empty` *class*); pinned by `test_the_broken_slot_table_is_gone_from_html_js_and_css`.
- **`tests/fixtures/slots_gufo.json`**: present; live `GET ${GUFO_BASE_URL}/slots` this pass matches it key-for-key (row + `next_token[0]`), so the capture is reproducible; drift proven to fail (mutation tests + live `test_gufo_slots_answer_the_pinned_shape`, 9 live tests pass).
- **Aggregate block**: live page shows `2 of 2 engines`, running/waiting/prompt/decode/total-tokens straight from `fleet_totals()`, dot `ok`, no verdict colour; on a throwaway single-engine instance (:8091, private temp DB) `#fleet` is `hidden`, one full-width strip card + one panel, zero banners → "one clean half".
- **Strip + slot grid on live data**: 2 `.card.stat` (KV bars `bar ok` 13 %/0 %), gufo grid rendered real cards incl. a busy slot (`#2189`, 67849 prompt tok, 99 % cache share, `processing`), vLLM note-only with no grid; no `NaN`/`undefined` anywhere. Screenshots: `data/logs/llm-phase07-final-live-{two-engines,single-engine}.png`, `ups-charts-aria-labels.png`.
- **Defect found in earlier work and fixed**: `/ups`' four `LineChart`s shipped no `ariaLabel`, violating AGENTS.md's "every chart names itself for assistive tech" (`/system`+`/llm` were pinned, `/ups` wasn't) → added labels to `page-ups.js` and `test_every_ups_chart_names_itself_for_assistive_tech`; browser-verified `role="img"` + label on all four.
- **Gate**: `bash .agents/validate.sh` → **exit 0**; `uv run pytest --cov --cov-fail-under=94` → **604 passed, 9 skipped, coverage 97.91 %**; `uv run ruff check src tests agent` → All checks passed; `uv run pyright` → **0 errors**; `node --check` every page script ok; `uv run pytest -m live --live` → **9 passed**; `uv pip list --outdated` empty.
- **Docs**: README + AGENTS.md reviewed end-to-end; fixed test count (582→604), `/llm` source column + both architecture diagrams now name gufo `/slots`, AGENTS.md agent line gained `GET /schema` + status page, ariaLabel convention now covers all three chart pages, README Testing block now lists `bash .agents/validate.sh`. Env-var table, agent/install flags, Containerfile facts, `/api/*` shapes, range chips, remote URL all verified accurate — no other changes needed.
- **Live-infra**: no fleet/host/config change; the temp :8091 instance, its temp DB and the fake processes are removed; dev server restarted and `/api/health` `ok` (all 6 sources ok). One thing I deliberately did **not** touch: the gitignored local `.env.dev` still carries a comment claiming the NUT `[dashboards]` account has only battery-test `instcmds`, whereas AGENTS.md/README record `instcmds = all` (verified on hardware 2026-10-17) — operator-owned file, flag for the operator to reword.
- **Next pending phase**: `08_events_paging_and_copy`.
2026-10-17 17:50:19 -04:00
ducoterra de325caa49 phase: 06_system_page_gpu_verdicts
All criteria verified and the project is fully green.

**Verified (final pass — all 3 phase-06 tasks were already in `complete/`)**
- `classify("gpu.busy_pct", 99)` / `gpu.mem_used_pct` → `None`; `THRESHOLDS` has no `gpu.*` row; `alerts.py`/`api.py` carry no GPU verdict; live browser: deskwork GPU at **91 %** and **98 %** renders `class="hero"`, `color=rgb(22,25,31)` (plain text), no bar, no `.sev-*`.
- GPU temp on the hero `.sub` from live agents: `Radeon 8060S Graphics · VRAM 165.1 MiB / 512.0 MiB · pool 78% · 48 °C`, R9700 `· 40 °C`, gateway prints none (no `n/a`, no `0 °C`).
- Charts collapsed by default: fresh load = 3 panels / 3 toggles / **0 open / 0 `<svg>`**; expand → 3 SVGs (510 px wide), repaints across polls (legend 3.3 %→5.6 %), collapsed siblings keep their own height (803 vs 358/339).
- `gpu.busy_pct@0` still answers from `/api/system/history` (345 points) — verdict gone, data intact.

**Tests / lint / coverage**
- `bash .agents/validate.sh` → **EXIT=0**; `uv run pytest --cov --cov-fail-under=94` → **550 passed, 8 skipped, 97.91 %**; `uv run ruff check src tests agent` → clean; `uv run pyright` → 0 errors; `node --check` all scripts clean; `uv pip list --outdated` empty.
- `uv run pytest -m live --live` → **8 passed** (real agents/engines/upsd); dev server restarted, `/api/health` all sources `ok`.
- New screenshots: `data/logs/phase06-final-collapsed-default.png`, `data/logs/phase06-final-expanded-gpu-busy-uncoloured.png`.

**Completion criteria:** classify→None ✅ · pytest+94 % gate ✅ · ruff+pyright ✅ · validate.sh ✅ · screenshots of the three behaviours on live data ✅ · AGENTS.md severity section claims no GPU rows ✅

**Docs:** fixed 3 items — (1) `AGENTS.md` pytest count stale (546 → 550); (2) `app.js` `severityFor` docstring still claimed "`page-ups.js` still calls it (phase 05 converts it)", contradicting the code/tests/AGENTS.md (comment-only); rest of README/AGENTS reviewed in full (env table vs `config.py`+`.env.example`, agent routes incl. `/schema`, alerts rules, page descriptions, contracts) — consistent.

**Notable:** fixed a real pre-existing defect in `scripts/dev-server.sh` — `stop` slept 1 s and deleted the pidfile without waiting for exit, so `restart` with a browser open overlapped two servers on one SQLite file, producing a `database is locked` store refusal that `/system` rendered as a red event. Now waits (bounded 20 s) then SIGKILLs; verified a restart with live polling logs no such error/event. No live infra config was changed.

**Next pending phase:** `.agents/phases/todo/07_llm_monitor_style_pages`
2026-10-17 14:04:52 -04:00
ducoterra 9f3bf3d503 docs: close the no-remote gap and repoint the deleted phase-04 notes
`origin` is `git@git.reeseapps.com:services/dashboards.git` (`main`) and the history is
pushed, so gap 3 ("No remote yet") is gone and the remaining gaps renumber. The remote has
no CI, which is why the same paragraph now names `bash .agents/validate.sh` as the only gate.

The `/llm` layout note cited `NOTES-chart-set.md`, a pipeline stray that is deliberately not
tracked; it now cites the phase-04 task file where that decision was written.

LLM-Generated: true
2026-10-17 12:36:32 -04:00
ducoterra c5689a8bdd chore(agent): drop the verification notes of the completed phases 03-05
The three NOTES files recorded the browser-verification evidence for phases that are now in
.agents/phases/complete/; the conclusions they captured already live in AGENTS.md's UI
conventions. Their content stays retrievable from git history.

LLM-Generated: true
2026-10-17 12:28:05 -04:00
ducoterra 9e732c1f87 chore(agent): commit the validation gate the phase pipeline runs
AGENTS.md and PLAN.md already told every phase to `bash .agents/validate.sh`, but the script
itself was never tracked, so a fresh clone could not run the gate at all.

Four checks: `uv run pytest --cov --cov-fail-under=94`, `uv run ruff check src tests agent`,
`uv run pyright`, and a zero-dependency `node --check` of every served JS. Plus repo hygiene
(tracked files matching .gitignore, log drops/key material, non-placeholder secrets in the
tracked env templates) and an advisory `uv pip list --outdated`.

Two rules are baked in because breaking them made a false green: run the project toolchain
through `uv run` (a bare `python3`/`ruff` on PATH tests an interpreter nobody deploys and an
older linter than the pyproject floor), and treat a missing tool as a failure rather than an
`if command -v` skip.

LLM-Generated: true
2026-10-17 12:27:52 -04:00
ducoterra 046dc57b26 chore(agent): add phase 11 — single-admin login for the dashboards
Eight tasks to put a session gate in front of every page and API route: the auth primitives
(PBKDF2 verifier, HMAC-signed stateless session, session-bound CSRF token, login throttle,
origin check) in a new `auth.py`, the verifier in the store plus fail-closed startup, the
default-deny middleware with an explicit public list, `POST /api/auth/password` with session
rotation, a login page, a `/settings` page, and the dev-config/docs/browser verification pass.

Stdlib hashlib/hmac only — no new dependency, so no PLAN.md anchor moves. `/api/health`
stays public on purpose (the Containerfile HEALTHCHECK and dev-server.sh depend on it), and
one test enumerates every route and demands a session.

LLM-Generated: true
2026-10-17 12:27:40 -04:00
ducoterra 71900d903a chore(agent): add phase 10 — dev dials the fleet directly
Six tasks that make dev the prod topology: the documented `ops`-installable agent sudoers
grant, installing the agents as systemd services, the upsd ACL for the host and laptop
ranges, `.env.dev` pointing at the real hosts (token in the URL userinfo, `NUT_PORT=3051`),
retiring `scripts/dev-tunnels.sh`, and a live + browser verification pass.

No dashboard Python changes — deployment, config, scripts and docs only.

LLM-Generated: true
2026-10-17 12:27:27 -04:00
ducoterra 9015931860 docs: record the live UPS command state after real-hardware verification
AGENTS.md gap 1 said the buttons were read-only in practice because no upsd.users account
had `instcmds`. That is no longer true: the login verbs, the command catalog, and the
`[dashboards]` account have each been fixed and verified against both real UPSes.

The rewrite keeps the parts that stay load-bearing: `instcmds = all` means the account may
send every advertised command including the power-cutting ones, and `available_commands()`
filters `LIST CMD` (which upsd serves unfiltered) — NUT has no call for a user's
privileges, so the risk tiers and the typed `DANGER` are the control, not decoration. Also
records what was proven reversibly (`beeper.enable` -> OK, `driver.killpower` refused
without `DANGER`), why a battery test was not run, and the rules for editing upsd.users
without dropping the account `upsmon` authenticates as.

README: `instcmds` is one command per line per `man 5 upsd.users`, the fleet runs
`instcmds = all`, and authentication uses `USERNAME`/`PASSWORD`.

LLM-Generated: true
2026-10-17 12:27:11 -04:00
ducoterra 9041683179 fix(ups): offer every advertised command and gate its aliases too
Two halves of the same bug: the page offered less than the hardware does, and the gate
matched only the name it rendered.

`available_commands()` drops anything `COMMANDS.get()` misses, and 12 of the 23 commands
the two real UPSes advertise had no entry (`beeper.on/off`, `load.on/off.delay`,
`shutdown.default/reboot/stayoff`, `driver.reload*`, `driver.killpower`,
`test.panel.stop`) — an invisible button the operator had no way to learn about. Every one
is catalogued now with a risk tier, and the power-cutting ones are pinned to
`RISK_DANGEROUS` (`shutdown.stop` is the exception the name hides: it cancels a shutdown).

Both real drivers advertise `beeper.enable` *and* `beeper.on`, `beeper.disable` *and*
`beeper.off`, so rendering every advertised name drew "Beeper on" twice. `COMMAND_ALIASES`
collapses one action to one entry (named for a name this driver answers, so a driver that
only implements the alias still gets its button) and records the other names in `also`.

The POST gate resolved the request through that table instead of comparing the literal
string: previously `beeper.off` walked past a `disabled` verdict recorded against
`beeper.disable` — the same action, and the gate is the control, not the button.

LLM-Generated: true
2026-10-17 12:27:02 -04:00
ducoterra 1f20a5e57f fix(nut): log in with the verbs upsd 2.8.5 actually answers to
`NutClient.login()` sent `USER`/`PASS`. The deployed upsd 2.8.5 implements neither —
its own `HELP` advertises `USERNAME`/`PASSWORD` — so each line answered
`ERR UNKNOWN-COMMAND`, `_read_line()` raised, and `POST /api/ups/command` could only
ever 502 before `instcmd()`, credentials or not.

Values go out unquoted: upsd takes the rest of the line literally, so quoting them
would make the quotes part of the credential.

`ERROR_HINTS["ACCESS-DENIED"]` blamed the upsd.conf ACL/LISTEN lines, but that code is
only reachable on an authenticated session (an unauthenticated client gets
`ERR USERNAME-REQUIRED`), so the cause is the account's `instcmds` grant. The hint now
names the one thing the operator can change.

The fake upsd in tests has no `USER`/`PASS` branch, so a regression answers
`ERR UNKNOWN-COMMAND` exactly like the real server, and its `PASSWORD` branch checks the
secret so a wrong-credential refusal is expressible.

LLM-Generated: true
2026-10-17 12:26:47 -04:00
ducoterra 2460a0adf5 chore(agent): phase roadmap from TODO.md, 4 phases
06 system_page_gpu_verdicts — GPU stops carrying a verdict, GPU temp on the
   hero row, per-host history charts collapsed behind a toggle (TODO L3/L4/L9)
07 llm_monitor_style_pages — /slots schema pinned from the live server,
   broken slot table removed, fleet totals block, stats strip, slot grid
   (full restyle; TODO L5)
08 events_paging_and_copy — event feeds scrollable + paginated, flavour text
   stripped, rule pinned in AGENTS.md (TODO L6/L7)
09 disks_smartctl — smartctl physical-disk collection in the host agent,
   5-minute disks source/poller/store, /api/disks + severity rows, /disks
   page + nav tab + overview entry (TODO L8)

LLM-Generated: true
2026-10-17 11:39:29 -04:00
ducoterra 34854d32af phase: 05_ups_hero_battery_glyph
**Phase 05 final pass — all criteria met; whole project 100 % green.**

Verified (live upsd + isolated simulator):
- Real `apc`/`cyberpower`: Charge/Load/Runtime are the card's three largest elements (38.4 px vs 15.2 px next), coloured only by API `severity` — live right now 100 % ok / 79.0 % **warn** / 5m **fail**; glyph fill = charge (100/100 viewBox units, `--ok` token), `<title>` "apc: Charge 100 percent, on line"; SIMULATED hidden; battery °C `n/a` uncoloured; every button disabled with its reason; `confirm_text: DANGER` on power-cutting commands; **0 non-GET requests** → `data/logs/ups-verify-live-{card,full}.png`.
- Simulator on `:8091` (own `/tmp` DB, killed after): caution dialog → Cancel, dangerous dialog → Run disabled until `DANGER` → Cancel, **0 POSTs**; one deliberate quick test *to the sim only* flipped the card to `OB` (glyph `stroke-dasharray="7 5"`, "on battery", health "discharging, on battery") then `OL CHRG` (bolt) then `OL` → `ups-verify-sim-{dialog-caution,dialog-danger,onbattery-button,onbattery-page,charging-button}.png`; glyph matrix `ups-glyph-*.png` (100/mid/low/unknown/empty/unrated, dark/light/200 %) present; `grep -c "function battery"` = 1.

Defects found in earlier work and fixed:
1. **`poller.py` could stop polling in silence** — 4 of 6 dev-server sources had frozen `attempts` (ages 12–38 min) while `/api/health` still said `ok: true`: the store-write half of `run_once` was outside the `try`, and a dead task's exception is never retrieved. Now `_commit()` moves `last_ok` only after the write lands, `_store_refused()` reports it (`ok:false`, real reason, one `store-error` event + one `recording` on recovery, payload kept), and `_loop()` catches anything that escapes and keeps polling — 4 new tests in `tests/test_poller.py`.
2. **`nut_sim.py` (modified by this phase) sat at 83 % coverage**; its whole command path was untested. Added tests (every advertised command implemented, unknown → `ValueError`, alarms → card + `assess()` fail, `NUT_SIMULATE` truthy rule) and fixed two model lies: `beeper.*` answered OK while changing nothing (now publishes `ups.beeper.status` enabled/muted/disabled) and a finished self-test never wrote `ups.test.result` (aborted writes none). → 100 %.
3. `nut_sim.from_env()` was dead code duplicating `app._truthy`; `app.py` now builds the simulator through it, so the enable gate lives once.

Tests / lint (exact): `bash .agents/validate.sh` → **EXIT=0** — `uv run pytest --cov --cov-fail-under=94` **543 passed, 8 skipped, coverage 97.91 %**; `uv run ruff check src tests agent` clean; `uv run pyright` 0 errors; `node --check` clean; repo hygiene clean. `uv run pytest -m live --live` **with `.env.dev` sourced → 8 passed** (one earlier run: 2 transient `9121/snapshot timed out` against the 4 s `HOST_TIMEOUT`; three later runs 8/8). `uv pip list --outdated` empty. Dev server restarted on the final code: `health ok`, 6/6 sources fresh.

Docs: AGENTS.md — invariant 4 extended ("nothing may stop polling in silence" + store-refusal semantics), `poller.py`/`nut_sim.py` responsibility lines, 533→543 and 96.9 %→97.9 %, "~97 %"→"~98 %". README.md — poller bullet gained the same guarantee, Testing gained "source `.env.dev` before `-m live --live`" (bare runs skip 7 of 8 live tests, which reads like a pass), Battery-test section gained the `deploy/nut-sudoers` install line and the read-before-write warning.

Notes/deviations: no live instant command was ever sent to real hardware (the four gates leave them disabled — AGENTS gap 1 unchanged); the on-battery proof was driven through the real button against `NUT_SIMULATE=1`, which is stronger than the NOTES hand-driven states. `NOTES-verify.md` under `todo/` left untouched per instructions. Next phase: **none** — 05 is the last in the PLAN.md roadmap.
2026-10-17 10:27:23 -04:00
ducoterra f44a617ef9 phase: 04_llm_chart_readability
**Phase 04 final verification pass — all tasks already in `complete/`; verified + fixed 3 defects found in verification.**

- Verified `/llm` live (both engines, gufo-only instance on :8099, `llm_monitor` on :8095) in a real browser; dev server left running/healthy.
- Completed task-04's truncated layout fix: `#engines` now carries `.items-start` (the CSS rule existed but nothing used it) — gufo 544 px vs vLLM 715 px, no blank card; pinned by `test_llm_engine_panels_are_not_stretched_to_their_tallest_sibling`.
- Fixed: event rows said the subject twice ("gufo gufo is answering again"), and `/ups` mangled mid-message names ("sent to apc: OK" → "sent to  OK") → shared `Dash.eventText()` (strips `UpstreamError: ` + a *leading* name only, mirroring `alerts._clean_error`) used by `/system`, `/llm`, `/ups`; new test.
- Fixed: count sparklines autoscaled rollup averages (queue max 0.1379) into full-height spikes → `sparkline({minSpan})`, `minSpan: 1` on sessions/queued (spike 30 px → 4 px); assertions added.
- Tests: `bash .agents/validate.sh` → **EXIT=0** (`uv run pytest --cov --cov-fail-under=94`: **485 passed, 7 skipped**, coverage **96.87%**; `ruff check src tests agent`: All checks passed; `pyright`: 0 errors; `node --check` all static JS clean; repo hygiene clean; `uv pip list --outdated` empty). `uv run pytest -m live --live` → **7 passed**.
- Completion criteria: heroes largest per engine ✔ (38.4 px vs 16.8 px row values); sparkline per cell + selected window ✔ (5/5 gufo, 7/7 vLLM; 6 h chip moved KV max 53 %→58 %); mean+p95 without hover ✔ (243 / 48.8 / 285 ms feet); provenance tag ✔ (`Δ10.0s`/`Δ8.0s` + `title`=exact source); single-engine ✔ (1 panel, totals hidden, zero red); screenshots ✔ `data/logs/llm-verify-{both-engines,charts-and-slots,single-engine,monitor-reference}.png` + `NOTES-chart-set.md`; pytest/cov/ruff/pyright ✔; earlier phases unchanged-behaviour ✔ (all 01–03 tests pass).
- Deviation (documented, not code): throughput heroes stay **neutral ink** — `severity.py` has no throughput row and 0 tok/s idle is healthy; colour comes from rated cells (KV) + reachability. Kept per `NOTES-chart-set`.
- Deviation: could not append the task-04 written verdict to `NOTES-chart-set.md` (it lives in `todo/`, which I must not edit). Verdict: ours reads better than `:8095` — bigger throughput numbers, per-cell shape over the selected window, percentiles inline; `llm_monitor` only sparklines decode+KV and has no window control.
- Docs: AGENTS.md updated (485 test count, `sparkline` `minSpan` rule, `Dash.eventText` event-row rule, `/llm` panel `items-start`); README reviewed in full — commands, config table (all `config.py` vars present), architecture bullets, page descriptions verified accurate; no other inconsistencies found.
- Note for the harness: `.agents/phases/todo/03…/NOTES-verify.md` and `todo/04…/NOTES-chart-set.md` are untracked notes left behind by their phases (task files moved, notes did not).
- Next pending phase: `05_ups_hero_battery_glyph`.
2026-10-17 07:03:58 -04:00
985 changed files with 82362 additions and 1952 deletions
@@ -0,0 +1,42 @@
# Phase 06 — The system page stops grading GPU usage, shows GPU temp, and tucks charts away
**Source:** `TODO.md` L3, L4, L9 — "GPU verdicts + system-page cleanup": "High gpu usage is not a warning or error, these are LLM servers and that's expected" · "The historical charts on the system resources page should only be visible if expanded" · "Pull GPU temp for the system resources tab where you can."
**Story:** n/a
**Context:** `src/dashboards/severity.py` (`THRESHOLDS` — the only place thresholds live), `src/dashboards/sources/host.py` (per-GPU `severity` dict + `gpu.*` samples), `tests/card_contract.py` / `tests/test_system_contract.py` (the pinned card contract), `src/dashboards/web/static/page-system.js` (hero row, `gpuSub()`, the per-host `.host-charts` stack), `src/dashboards/web/static/dash.css`
## Objective
`/system` watches LLM servers, where a busy GPU is the job, not a fault: the GPU card stops
carrying any verdict (muted, never green and never red), the GPU temperature the agent
already reports moves up onto the hero row, and the per-host history charts collapse behind
a "Charts" toggle so the default view is current stats only.
## Dependencies
- `05_ups_hero_battery_glyph` (complete) — the shared `.hero` vocabulary and the per-host
chart stack this phase reworks; nothing in this phase may fight its conventions.
## Tasks
1. `01_drop_gpu_severity_verdicts.md` — remove `gpu.busy_pct` / `gpu.mem_used_pct` from the verdict table; the GPU card renders uncoloured; the contract pins the *absence*.
2. `02_gpu_temp_on_hero_row.md` — surface `gpus[].temp_c` (already in the payload) on the GPU hero's `.sub` line.
3. `03_collapse_host_history_charts.md` — wrap each host's `.host-charts` in a collapsed-by-default toggle; charts draw lazily on expand.
## Testing & Quality
- Unit: `tests/test_severity.py`, `tests/test_host_source.py`, `tests/test_system_contract.py`
(all three captures — discrete GPU, APU, no GPU — plus over HTTP) must pin the new
no-verdict shape; `tests/test_api.py` pins the chart-toggle markup on served bytes.
- Coverage: **>90%** on new/modified code, ≥94% overall gate maintained.
- Browser (DoD): dev server live at `127.0.0.1:8090`, screenshot per change in `data/logs/` —
a busy GPU (driveripper's R9700 under load) renders uncoloured, the temp is on the hero
row, charts are collapsed by default and draw on expand.
## Completion Criteria
- [ ] `classify("gpu.busy_pct", 99)` is `None` and no page script colours a GPU number from a threshold
- [ ] `uv run pytest` green; coverage gate (94%) holds
- [ ] `uv run ruff check src tests agent && uv run pyright` clean
- [ ] `bash .agents/validate.sh` green
- [ ] screenshots in `data/logs/` show the three behaviours on live data
- [ ] `AGENTS.md` severity section no longer claims GPU rows exist
@@ -0,0 +1,51 @@
# Task 01 — GPU usage and VRAM stop carrying a verdict
**Phase:** `06_system_page_gpu_verdicts` · **Source:** `TODO.md:3` — "High gpu usage is not a warning or error, these are LLM servers and that's expected"
**Story:** n/a
## Objective
Remove the GPU verdict entirely: a 95 %-busy card on an LLM server is the job, not a
warning, so the GPU card prints its numbers uncoloured at every utilisation.
## Work
1. `src/dashboards/severity.py` — delete the `gpu.busy_pct` and `gpu.mem_used_pct` rows
from `THRESHOLDS`. No verdict at all — not a raised threshold: "high usage is expected"
applies to every level, and the module's own rule is that a metric the table does not
know classifies to `None`, never `ok`.
2. `src/dashboards/sources/host.py` — stop classifying the GPU rows: drop the `busy_pct` /
`mem_used_pct` classifications from the per-GPU `severity` dict (remove the dict when it
becomes empty), leaving the `gpu.*` **samples** untouched — the history series stay
stored and queryable; removing the verdict is not removing the data.
3. `tests/card_contract.py` — pin the absence, the way it already pins `network`'s
colourlessness: the GPU card paths (`gpus[].model/busy_pct/vram_*/mem_used_pct`) stay,
the verdict paths go. `tests/test_system_contract.py` follows (all three captures + HTTP).
4. `tests/test_severity.py` — assert `classify("gpu.busy_pct", 99)` /
`classify("gpu.mem_used_pct", 99)` return `None`; keep the boundary-ordering test
consistent with the smaller table. `tests/test_host_source.py` — the GPU row no longer
carries a `severity` dict.
5. `src/dashboards/web/static/page-system.js` — the GPU hero renders without `severity` and
without the utilisation bar (a bar under a verdictless number is a verdict by shape);
drop the `data-sev="gpus[].severity.busy_pct"` hook. `n/a` stays uncoloured as before.
6. `AGENTS.md` — the `severity.py` section lists `gpus[].severity` among the rated cells;
update it so the file and the code agree (DoD: rules and fleet facts travel with the
change).
- ASSUMPTION: "no verdict at all (muted)" was the operator's confirmed reading of the item
— the card shows busy % / VRAM with no colour class at any level, rather than "only fail
above a new ceiling". If the operator later wants a ceiling, it is one table row back.
## Testing & Quality
- Unit: `tests/test_severity.py` (no GPU row in the table; `None` verdicts),
`tests/test_host_source.py` (payload shape), `tests/test_system_contract.py` (the pinned
contract over the three captures + over HTTP).
- Coverage: **>90%** on this task's modified code; overall gate (94%) maintained.
## Completion Criteria
- [ ] `uv run pytest` green with the updated contract
- [ ] a live busy GPU (driveripper, R9700) renders its numbers with no colour class and no bar
- [ ] `gpu.busy_pct@N` still answers from `/api/system/history` (data intact, verdict gone)
- [ ] `uv run ruff check src tests agent && uv run pyright` clean
@@ -0,0 +1,44 @@
# Task 02 — GPU temperature on the hero row
**Phase:** `06_system_page_gpu_verdicts` · **Source:** `TODO.md:9` — "Pull GPU temp for the system resources tab where you can."
**Story:** n/a
## Objective
The GPU temperature is already collected end-to-end (agent `gpus[].temps` → payload
`gpus[].temp_c`, max over the card's hwmon labels, and a `gpu.temp_c@N` sample) but the
hero row never shows it — it sits in the collapsed "More detail" table. Move it up so the
one number that matters while a card is busy is glanceable.
## Work
1. `src/dashboards/web/static/page-system.js` — `gpuSub()`: append the temperature to the
GPU hero's `.sub` line when `gpu.temp_c != null` (e.g.
`VRAM 31.2 GB / 32 GB · pool 98% · 63 °C`); when the agent cannot read it (the APU
build on deskwork or a card with no hwmon input), the sub line simply does not print a
temperature — `n/a` is reserved for numbers the card is supposed to have, and "where you
can" is the item's own wording.
2. Keep the "More detail" table exactly as-is (it carries per-label temps, power, and both
memory pools); this is a surface change, not a dedup.
3. No backend changes: `tests/fixtures/agent_deskwork.json` / `agent_driveripper.json`
already carry `gpus[].temps`, so the hermetic page tests can exercise both the
with-temp and without-temp paths.
- ASSUMPTION: the payload's `temp_c` (agent's max over hwmon labels) is "the" GPU temp —
the same max the agent already computes — so no new field, no new metric name, and the
`gpu.temp_c@N` history series keeps its existing name.
## Testing & Quality
- Unit: `node --check` on `page-system.js`; served-page assertions in `tests/test_api.py`
if the markup structure changes.
- Coverage: **>90%** on modified code (mostly JS; the Python side is unchanged).
- Browser (DoD): screenshot in `data/logs/` with live data — driveripper's R9700 shows a
temperature on the hero row; deskwork's APU row does not print a bogus one.
## Completion Criteria
- [ ] GPU hero `.sub` includes `· <n> °C` on hosts whose agent reports temps
- [ ] hosts without a readable sensor show no temperature (not `0 °C`, not red)
- [ ] `uv run pytest` green; `uv run ruff check src tests agent && uv run pyright` clean
- [ ] screenshot in `data/logs/` against live data
@@ -0,0 +1,49 @@
# Task 03 — Per-host history charts collapse behind a toggle
**Phase:** `06_system_page_gpu_verdicts` · **Source:** `TODO.md:4` — "The historical charts on the system resources page should only be visible if expanded. I only want to see current stats to keep things clean. There will be lots of hosts and historical charts make things look messy."
**Story:** n/a
## Objective
The default `/system` view is the hero row (current stats) only. Each host's `.host-charts`
stack (utilisation, network, GPU busy) moves behind a collapsed "Charts" toggle, so a
fleet that grows to many hosts reads clean at a glance — and a host that stops reporting
still loses its panel and charts (`pruneHosts`) exactly as today.
## Work
1. `src/dashboards/web/static/page-system.js` — wrap each host's `.host-charts` in a
native `<details>` (collapsed by default; `summary` reads "Charts" and carries the
window chip so the expanded state still says which window is drawn). Create the
`LineChart` instances **lazily on first expand** (the `state.charts` map already keys
them) and re-run the draw pass on refresh **only for hosts whose charts are open** —
drawing into a collapsed container is wasted work, and the page's one-
`/api/system/history`-request-per-host discipline is unchanged (the request still names
every series the panel could draw; only the paint is deferred).
2. `pruneHosts()` keeps destroying chart instances with the panel; add the `details`
element to the removal so a dead host leaves no orphan toggle.
3. `src/dashboards/web/static/dash.css` — style `summary` as a quiet toggle in the existing
vocabulary (no new palette, no `.sev-*` family); the collapsed stack takes no height.
4. `tests/test_api.py` — pin the served markup: one `<details>` per host panel, collapsed
by default (no `open` attribute), summary text "Charts"; the orphan-`id` rule keeps
passing (delete no rule that a live selector still uses).
5. `node --check` on both scripts.
- ASSUMPTION: the toggle is **per host** (not one global switch) and **collapsed by
default** — "only visible if expanded" is the item's own wording, and per-host matches
the page's one-panel-per-host discipline; the window chips (5 m … 30 d) stay where they
are and keep driving the same single history request.
## Testing & Quality
- Unit: `tests/test_api.py` served-bytes assertions (details present, closed by default).
- Coverage: **>90%** on modified code; overall gate (94%) maintained.
- Browser (DoD): screenshot in `data/logs/` — collapsed by default on a multi-host page,
one host expanded with its three charts drawn, a second refresh while expanded repaints.
## Completion Criteria
- [ ] default load shows hero rows only; no chart pixels before a toggle is opened
- [ ] expanding a host draws its charts on the first open and on every later refresh while open
- [ ] a pruned (dead) host removes its toggle along with its panel and charts
- [ ] `uv run pytest` green; `uv run ruff check src tests agent && uv run pyright` clean
@@ -0,0 +1,55 @@
# Phase 07 — The LLM page looks like llm_monitor again
**Source:** `TODO.md` L5 — "make this look like the llm_monitor project": "The LLM dashboard is nothing like ~/Experimental/llm_monitor. I'm missing the 'slots' tables (I really liked that). I'm missing the aggregate throughput at the top. Basically, make this look like the llm_monitor project. The gufo slots table that exists isn't even working and I want it gone anyway."
**Story:** n/a
**Context:** `~/Projects/Experimental/llm_monitor/static/index.html` (the reference: fleet-totals section ~L710, shared stats strip ~L743, gufo slot cards `buildSlotCard`/`renderSlots` ~L1179, `/slots` passthrough with `/v1/slots` fallback ~L750), `src/dashboards/web/pages/llm.html` + `src/dashboards/web/static/page-llm.js` (phase-04 layout: per-engine heroes, `CELLS` sparkline list, alarm chips, four trend charts), `src/dashboards/sources/llm.py` (`fleet_totals()` ~L327 — already sums decode/prompt tok/s, running/waiting, token totals; `_sessions()` passes gufo's raw `/slots` rows through as `sessions.list`), `scripts/capture-fixtures.sh` (today captures engine `/metrics` only — **no `/slots` capture exists**, which is why the slot table never matched the live shape)
## Objective
Port `llm_monitor`'s visual structure onto the LLM page — the aggregate fleet block at the
top, the per-engine stats strip, the slot-card grid — while keeping what phase 04 and the
locked UI conventions own: the per-engine panel (two throughput heroes, compact sparkline
cells, alarm chips, the four half-height trend charts) and every provenance/severity rule.
The broken gufo slot table is gone and its replacement is fed by a `/slots` shape pinned
from the live server, never from memory.
`llm_monitor` is the design reference, not a dependency: a separate app, nothing imported.
- ASSUMPTION (phase, confirmed with the operator): "full restyle" = port the three
llm_monitor structures above (aggregate block, stats strip, slot grid). The per-engine
panel and chart set stay; `llm_monitor`'s peak-ctx cell does **not** come over because
the payload publishes no peak-context number — a shape is not a verdict, and a missing
number prints `n/a`, never a borrowed one.
## Dependencies
- `06_system_page_gpu_verdicts` (todo) — same web assets (`dash.css`, `app.js`, page
scripts); land in that order so the CSS token vocabulary is settled first.
- `04_llm_chart_readability` (complete) — the layout this phase reworks; its NOTES
(`NOTES-chart-set.md`) and provenance/severity conventions bind.
## Tasks
1. `01_slots_schema_and_table_removal.md` — delete the broken slot table; extend `capture-fixtures.sh` to record gufo's live `/slots` and pin the shape in a fixture + contract test.
2. `02_fleet_totals_block.md` — render the aggregate block at the top from the existing `fleet_totals()` payload (hidden below two engines, as today).
3. `03_engine_stats_strip.md` — port the shared stats strip: one card per engine (KV bar, tok/s, sessions; gufo extras; alarm chips), fed only by payload fields that already exist.
4. `04_slot_grid.md` — the llm_monitor slot grid: one keyed card per gufo slot, updated in place, from `sessions.list`.
## Testing & Quality
- Hermetic: `tests/test_api.py` key-path pins (fleet block, strip, grid markup on served
bytes), `node --check` on every touched script, fixture-driven `sessions.list` shape test.
- Live: `uv run pytest -m live --live` — the `/slots` fixture test re-asserts against the
running gufo (`deskwork:8731`), and the page renders with both engines live and with a
single engine (one clean half, not a red panel).
- Browser (DoD): side-by-side with `llm_monitor` on its own port against the same live
engines; screenshots in `data/logs/` (two-engine, single-engine, gufo-busy states).
- Coverage: **>90%** on new/modified code, ≥94% overall gate maintained.
## Completion Criteria
- [ ] the old `#slots-card` table is gone from HTML, CSS and JS (and from any test that pinned it)
- [ ] `tests/fixtures/slots_gufo.json` exists, is produced by `./scripts/capture-fixtures.sh`, and the contract test fails if the live shape drifts
- [ ] top aggregate block shows summed decode/prompt tok/s, running/waiting, token totals — hidden below two engines
- [ ] stats strip + slot grid render on live data; single-engine page is one clean half
- [ ] `bash .agents/validate.sh` green; screenshots in `data/logs/`
@@ -0,0 +1,52 @@
# Task 01 — Remove the broken slot table; pin the /slots schema from the live server
**Phase:** `07_llm_monitor_style_pages` · **Source:** `TODO.md:5` — "The gufo slots table that exists isn't even working and I want it gone anyway."
**Story:** n/a
## Objective
Delete the slot table that has never matched gufo's live `/slots` shape, and replace
"trust a memory of an upstream" (rule 7) with a recorded ground truth: a fixture captured
from the running gufo, pinned by a contract test.
## Work
1. `scripts/capture-fixtures.sh` — add a gufo `/slots` capture: dial
`${GUFO_BASE_URL%/}/slots` with the `/v1/slots` fallback (the same order
`llm_monitor`'s passthrough at ~L750 uses — a server that only exposes the alias must
still be capturable), save verbatim as `tests/fixtures/slots_gufo.json`. Skip cleanly
(with a note) when `GUFO_BASE_URL` is unset, matching the script's existing engine
handling.
2. Delete the broken table end-to-end: `#slots-card` / `#slots-empty` / `#slots-table`
markup in `src/dashboards/web/pages/llm.html`, `renderSlots()` and the dead helpers in
`src/dashboards/web/static/page-llm.js`, and the CSS rules in `dash.css` that only that
table used (the orphan rule for both `id`s and selectors — `tests/test_api.py` enforces
both). Any test that pinned the table's markup or its guessed field names
(`slot.state`, `slot.n_prompt`, `slot.decode_per_second`, …) gets the pin removed or
turned into the new fixture-based assertion.
3. `tests/` — add the shape contract: a table of the `/slots` field paths the slot grid
(task 04) will read, asserted against `tests/fixtures/slots_gufo.json` hermetically and,
in a `live`-marked test, against the running gufo — the same device as
`tests/card_contract.py`. `tests/test_llm_source.py` keeps asserting that
`sessions.list` is the passthrough (raw rows, no re-mapping in `sources/llm.py`'s
`_sessions()` — the grid, not the source, decides what a slot field means).
4. `./scripts/capture-fixtures.sh` — run it against the live fleet and commit the new
fixture (serials/PII scrubbed, same as the existing captures).
- ASSUMPTION: the fixture is the *live* shape — whatever the running gufo answers today —
not a README's; if the live server is idle (zero busy slots) at capture time, the
capture still records the idle slot rows, which is the shape the grid must handle.
## Testing & Quality
- Unit: fixture-based shape test (hermetic), `node --check` on `page-llm.js`,
`tests/test_api.py` orphan-`id`/orphan-selector rules still pass after the deletion.
- Live: the `live`-marked shape test against `deskwork:8731`.
- Coverage: **>90%** on modified code; overall gate (94%) maintained.
## Completion Criteria
- [ ] no `#slots-table` / `#slots-card` / `renderSlots` remains in HTML, JS or CSS
- [ ] `tests/fixtures/slots_gufo.json` is in the repo and reproducible via `./scripts/capture-fixtures.sh`
- [ ] the shape contract fails when the fixture (or the live server) drifts
- [ ] `uv run pytest` green; `uv run ruff check src tests agent && uv run pyright` clean
@@ -0,0 +1,54 @@
# Task 02 — Aggregate throughput block at the top
**Phase:** `07_llm_monitor_style_pages` · **Source:** `TODO.md:5` — "I'm missing the aggregate throughput at the top."
**Story:** n/a
## Objective
The page leads with the fleet: one block above the engine panels with the summed numbers
llm_monitor's `#fleet` section shows — decode/prompt tok/s, running/waiting, token totals,
engines answering — which `fleet_totals()` in `sources/llm.py` already computes over the
engines that answered. Phase 04 retired the summed display to counts only; this task brings
the sums back in llm_monitor's shape.
## Work
1. `src/dashboards/web/pages/llm.html` + `page-llm.js` — replace the current `#totals`
card (counts only) with the llm_monitor-style fleet block: status dot, "all engines",
an `N engines · …` tag, the note "sums only engines answering now", and the
`running / waiting / prompt tok/s / decode tok/s / total tokens` row, rendered from
`fleet_totals()` via the existing payload (`/api/llm/snapshot` → `fleet`).
Provenance discipline stays: the sums carry no severity (throughput has no
good/bad — the same rule phase 04 wrote), and a dead engine's last-good values
contribute nothing to a total presented as current (`engines_live` vs `engines_total`
makes that visible, exactly as `fleet_totals()`'s docstring promises).
2. Keep the existing visibility rule — hidden below two engines (a cross-engine sum of one
engine duplicates that engine's own panel; the page comment at `#totals` says so and
`llm_monitor` hides the same way).
3. `tests/test_api.py` — pin the `fleet` key paths the block reads
(`engines_total`, `engines_live`, `decode_tokens_per_second`,
`prompt_tokens_per_second`, `requests_active`, `requests_waiting`, `tokens_total`) with
and without targets configured, and the served-markup presence/absence with one vs two
engines.
4. `dash.css` — block styling from the existing token vocabulary (`.metrics`/`.metric`
patterns already in the file; no new palette).
- ASSUMPTION: no backend change — `fleet_totals()` already sums exactly the numbers the
block prints; if a wanted number turns out to be missing from the payload, it stays
`n/a` rather than inventing a new metric (rule 2).
## Testing & Quality
- Unit: `tests/test_api.py` (key paths + markup, one vs two engines), `node --check`.
- Live: `uv run pytest -m live --live` with both engines; the block sums the two live
panels' numbers (a hand-check against the per-engine values on the same screen).
- Browser (DoD): screenshots in `data/logs/` — two-engine (block visible, sums correct),
single-engine (block absent, one clean half).
- Coverage: **>90%** on modified code; overall gate (94%) maintained.
## Completion Criteria
- [ ] the top block prints summed decode/prompt tok/s, running/waiting, token totals from `fleet_totals()`
- [ ] hidden below two engines; a dead engine dims the tag and contributes nothing to the sums
- [ ] no severity colour anywhere on the block
- [ ] `uv run pytest` green; `uv run ruff check src tests agent && uv run pyright` clean
@@ -0,0 +1,54 @@
# Task 03 — Per-engine stats strip
**Phase:** `07_llm_monitor_style_pages` · **Source:** `TODO.md:5` — "Basically, make this look like the llm_monitor project." (the shared stats strip, `index.html` ~L743)
**Story:** n/a
## Objective
Port llm_monitor's shared stats strip — one card per engine between the fleet block and
the engine panels — so the read-across-the-room numbers (throughput, KV pressure,
sessions, the gufo extras) sit in one row instead of being scattered inside the panels.
The panels keep their detail; the strip is the summary.
## Work
1. `src/dashboards/web/pages/llm.html` + `page-llm.js` — one stat card per configured
engine: status dot + name + timestamp, the KV-cache bar (the one rated cell — colour
from the payload's `severity`, the existing `scale: 100` ratio→percent conversion),
prompt tok/s and decode tok/s (with the provenance tags, uncoloured), total tokens,
and the session counts (gufo: active slots via `sessions.list` length/`active`; vLLM:
running/waiting from its count metrics).
2. Gufo-only extras on the gufo card, vLLM-only rows on the vLLM card — the existing
`only` scoping in `CELLS` is the model: prefix-cache hit %, spec-accept %
(gufo), queue time (vLLM). **No peak-ctx cell** — the payload publishes no
peak-context number (phase assumption; a missing number prints `n/a` or the row is
absent, never a borrowed one).
3. `preemptions` / `device_lost` remain **alarm chips that exist only while non-zero**
(phase-04 rule, `.chip.fail` pattern) — on the strip card, not in a permanent cell.
4. `dash.css` — the strip is a new layout of existing vocabulary (`.card`, `.metrics`,
the KV bar already has its tokens); no new palette, no `.sev-*` family.
5. `tests/test_api.py` — pin the strip's served markup (one card per engine, gufo/vLLM
scoping) and the payload field paths it reads; `node --check`.
6. The engine panels keep their hero row and the sparkline cells the strip *does not*
show (latency means + p95s, queue mean); cells that end up rendered in **both** strip
and panel get demoted from the panel to the strip — one value, one home (the
"one value drawn twice" rule from the UPS hero row).
- ASSUMPTION: the strip renders whatever the payload already sends — no new field in
`sources/llm.py` — so a single-engine deployment shows one card and the hidden
fleet block, and nothing on the strip is ever a number the upstream did not publish.
## Testing & Quality
- Unit: `tests/test_api.py` (markup + field paths), `node --check`.
- Live: both engines, then single-engine (one clean card, no red half).
- Browser (DoD): screenshot in `data/logs/` — strip visible, KV bar coloured from the
payload verdict, alarm chip present only in a non-zero state.
- Coverage: **>90%** on modified code; overall gate (94%) maintained.
## Completion Criteria
- [ ] one card per configured engine, between the fleet block and the panels, with KV bar + tok/s + sessions
- [ ] no value printed twice (strip vs panel); gufo/vLLM rows scoped correctly
- [ ] alarm chips absent at zero, present and red while non-zero
- [ ] `uv run pytest` green; `uv run ruff check src tests agent && uv run pyright` clean
@@ -0,0 +1,53 @@
# Task 04 — The gufo slot grid
**Phase:** `07_llm_monitor_style_pages` · **Source:** `TODO.md:5` — "I'm missing the 'slots' tables (I really liked that)."
**Story:** n/a
## Objective
Bring back what the operator liked about llm_monitor: one card per gufo slot, keyed and
created once, updated in place on refresh — fed by `sessions.list` (the raw `/slots`
passthrough whose shape task 01 pinned in `tests/fixtures/slots_gufo.json`).
## Work
1. `src/dashboards/web/pages/llm.html` + `page-llm.js` — implement the grid with
llm_monitor's `buildSlotCard`/`renderSlots` idiom (~L1179/L1209): a `.slot` card per
slot id (keyed `Map`, created once, patched in place — a busy slot does not get
rebuilt and re-announced on every poll), the card top shows the slot id + model, and
the per-slot fields come **only from the pinned fixture's shape** (state/busy
marker, prompt + generated token counts, cache share, per-slot speed — the exact keys
the fixture records; a field the live shape does not carry prints `n/a`, never a
guessed fallback chain like the old table's).
2. The grid lives in the gufo section under the "sessions (GET /slots)" label; the
empty state is the honest one (no slots answered → the panel says so, it does not
render a card of nothing). A queued (deferred) request is **not** in `/slots` — the
existing note under the grid stays (it is information, not flavour).
3. vLLM section: keep the one-line note that vLLM exposes no per-slot JSON and its
metrics are the equivalent of the session grid — no grid, no fake rows.
4. `dash.css` — `.slot` / `.slot-top` / `.slot.busy` styling ported to the existing token
vocabulary (busy state is a border/glyph, not a new palette entry); `node --check`.
5. `tests/test_api.py` — pin the grid markup on served bytes (container, empty state,
label); the fixture-driven render path is covered by the shape contract from task 01.
- ASSUMPTION: the slot card reads the fixture's fields one-to-one (no `a ?? b` fallback
chains — the old table's `slot.n_prompt ?? slot.prompt_tokens` style is exactly the
guessing that made it "not even working"); if a live capture shows two spellings of one
fact, the contract names the canonical one and the other is not rendered.
## Testing & Quality
- Unit: fixture-driven render test (the pinned shape through `renderSlots`-equivalent
logic, via served-bytes assertions), `node --check`.
- Live: gufo with at least one active request (a card with real numbers) and an idle
server (cards show idle, no invented speed).
- Browser (DoD): screenshots in `data/logs/` — idle, one busy slot, multiple busy slots;
refresh in place (cards do not flicker/rebuild).
- Coverage: **>90%** on modified code; overall gate (94%) maintained.
## Completion Criteria
- [ ] one keyed card per gufo slot, created once and updated in place across refreshes
- [ ] every printed field traces to a key in `tests/fixtures/slots_gufo.json` (no fallback chains)
- [ ] idle server: honest empty/idle state; vLLM: note only, no grid
- [ ] `uv run pytest` green; `uv run ruff check src tests agent && uv run pyright` clean
@@ -0,0 +1,44 @@
# Phase 08 — Event feeds page; the flavour text goes
**Source:** `TODO.md` L6, L7 — "page the event tables; strip the flavour text": "All the events tables need to be scrollable and paginated. They are causing the page to be massive." · "Get rid of most of the flavor text and add a note to AGENTS.md to not add it unless necessary. It just comes off as LLM generated slop, not helpful information."
**Story:** n/a
**Context:** the three `ul.events` feeds (`src/dashboards/web/pages/system.html` L58, `llm.html` L103, `ups.html` L70-72), `Dash.eventText()` in `src/dashboards/web/static/app.js`, the page-head sub-lines with the quoted flavour text (`system.html` L45, `index.html` L73), `AGENTS.md` → *UI conventions*, `tests/test_api.py` (served-bytes pins, orphan-`id` rule)
## Objective
A dead host's 500-event history must not make `/system` take a page to scroll: every
event feed becomes a bounded, scrollable, paginated window. The descriptive copy that
explains the app to the operator ("agent-reported values; rates are measured over the Δ
window…") goes away page by page, and `AGENTS.md` records the rule so the slop does not
crawl back in.
## Dependencies
- `07_llm_monitor_style_pages` (todo) — same page files and `app.js`/`dash.css`; this
phase lands on the restructured `/llm` page, not the phase-04 one.
- `06_system_page_gpu_verdicts` (todo) — rewrites `/system`'s panel structure; event-feed
work must not fight it for the same markup.
## Tasks
1. `01_paginate_event_feeds.md` — one shared pager: the three `ul.events` become scrollable containers with a 50-per-page window and a prev/next + count footer; no API change.
2. `02_strip_flavor_text.md` — audit all four pages and the page scripts for descriptive copy, delete what is not functional, and pin the rule in `AGENTS.md`'s UI conventions.
## Testing & Quality
- Hermetic: `tests/test_api.py` served-bytes pins (pager markup, no removed flavour
strings still present), `node --check` on touched scripts, the orphan-`id` rule
(the pager introduces `id`s — every one must be referenced).
- Live: `uv run pytest -m live --live` — the event endpoints still answer; a feed with
more than 50 stored events (the real event table has history) paginates instead of
dumping.
- Browser (DoD): screenshots in `data/logs/` — a long feed bounded and scrollable, page
2 reachable, the removed sub-lines gone from the page heads.
- Coverage: **>90%** on modified code, ≥94% overall gate maintained.
## Completion Criteria
- [ ] all three feeds scroll in a bounded container and paginate 50-per-page with a count footer
- [ ] no removed flavour string survives in served HTML or JS
- [ ] `AGENTS.md` carries the flavour-text rule under UI conventions
- [ ] `bash .agents/validate.sh` green; screenshots in `data/logs/`
@@ -0,0 +1,56 @@
# Task 01 — Event feeds: scrollable and paginated
**Phase:** `08_events_paging_and_copy` · **Source:** `TODO.md:6` — "All the events tables need to be scrollable and paginated. They are causing the page to be massive."
**Story:** n/a
## Objective
Every event feed (`/system`, `/llm`, `/ups`) renders a bounded, scrollable window over the
events instead of a growing `ul` that makes the page take a scroll to read. One shared
helper, three call sites — a pager that only works on one page is the bug this shape
prevents.
## Work
1. `src/dashboards/web/static/app.js` — add `Dash.eventsPager(el, events, opts)`: renders
the visible page of `events` into the feed's `ul` (keeping the existing
`Dash.eventText()` row rendering and per-row subject-once rule untouched), with a
footer: `1–50 of N` and prev/next controls (disabled at the ends, `aria`-labelled).
Page size defaults to 50; the feed container is the scrollable element, the page
stays short.
2. Call sites — `page-system.js`, `page-llm.js`, `page-upts.js` → `page-ups.js`: each
replaces its direct `ul#events` population with the pager (the "loading…" / empty /
not-configured states keep their current behaviour; pagination only applies when there
is more than one page of data). On a refresh that brings new events, the pager resets
to the first page (an operator reading page 3 of stale history should not silently land
on page 3 of the new set).
3. `src/dashboards/web/static/dash.css` — `.events` gets a bounded height (scrollable
container, existing token vocabulary); the pager footer styled as a quiet `.faint`
strip. No new palette.
4. No API change: `/api/events` keeps answering exactly what it does (server-side list,
no `page`/`limit` params added — the feeds are page-local and the endpoint's
contract is pinned by `tests/test_api.py`).
5. `tests/test_api.py` — pin the served markup (the three feeds keep their `id`s; the
pager's new `id`s/`aria-*` are all referenced — the orphan rule stays green), and
`node --check` on all touched scripts.
- ASSUMPTION: **client-side** pagination (50-per-page, scrollable container) is the
confirmed reading — the event tables grow with history and the server already serves
the list; a server-side `page`/`limit` API change would re-open a pinned endpoint
contract for a page-local problem.
## Testing & Quality
- Unit: `tests/test_api.py` (markup pins), `node --check`; a fixture-events render check
(page 1 of 3 pages, footer counts, disabled prev on page 1).
- Live: `uv run pytest -m live --live` — event endpoints unchanged.
- Browser (DoD): screenshot in `data/logs/` — a feed with >50 events bounded and
scrollable, page 2 showing rows 51–100.
- Coverage: **>90%** on modified code; overall gate (94%) maintained.
## Completion Criteria
- [ ] all three feeds paginate 50-per-page with a `1–50 of N` footer and prev/next
- [ ] the page height no longer grows with event history; the container scrolls
- [ ] refresh resets to page 1; loading/empty/not-configured states unchanged
- [ ] `uv run pytest` green; `uv run ruff check src tests agent && uv run pyright` clean
@@ -0,0 +1,56 @@
# Task 02 — Strip the flavour text; pin the rule in AGENTS.md
**Phase:** `08_events_paging_and_copy` · **Source:** `TODO.md:7` — "Get rid of most of the flavor text and add a note to AGENTS.md to not add it unless necessary. It just comes off as LLM generated slop, not helpful information. Like 'agent-reported values; rates are measured over the Δ window in the footer, and each panel's charts cover only that host' or 'every number is scraped from the box it belongs to · null renders as n/a'"
**Story:** n/a
## Objective
The pages stop narrating themselves. Copy that explains the app's own design to the
operator ("agent-reported values; rates are measured over the Δ window…", "every number is
scraped from the box it belongs to · null renders as n/a") is deleted; copy that is
*functional* — a source tag, a `Δ10.0s` provenance, a "not configured" explanation, the
SIMULATED badge, an error that says why a panel is empty — stays. `AGENTS.md` records
the rule so it is a convention, not a mood.
## Work
1. Audit, page by page: `src/dashboards/web/pages/index.html`, `system.html`, `llm.html`,
`ups.html` and the `src/dashboards/web/static/page-*.js` / `app.js` strings. The two
lines the TODO quotes (the `system.html` L45 sub-line, the `index.html` L73 sub-line)
are confirmed deletions; every other candidate is judged on one test: *would the
operator act differently without it?* No → delete. Kept by default: the `Δwindow`
footnotes (measurements, rule: "a Δ label is a measurement, never a config value"),
the per-rate provenance tags ("scraped / measured Δcounter/Δt / estimated" — dropping a
tag is a regression, not a simplification), `n/a`-and-why states, the "queued (deferred)
requests are not in /slots" note, and any string a test pins as functional.
2. Delete the confirmed flavour from markup *and* from JS that injects the same copy;
a sub-line that becomes empty removes its container, not an empty `<span>`.
3. `tests/test_api.py` — where a deleted string was pinned, update the pin; add an
assertion that the two quoted strings (and the other confirmed deletions) are **absent**
from served bytes, so the slop cannot creep back in silently.
4. `AGENTS.md` → *UI conventions* — add the rule: **Flavour text is not helpful
information.** Pages do not narrate their own data pipeline or design; copy exists
only when the operator needs it to act (a why-this-panel-is-empty, a provenance tag,
a measurement label). The two deleted lines are the cited examples. (DoD: AGENTS.md
updates with rule changes — this is the task that does it, not the conversion.)
5. `node --check` on every touched script.
- ASSUMPTION: "most of the flavour text" = the audit rule above (act-differently test),
with the two quoted lines as the floor of deletions — the operator's examples, not the
ceiling. If a borderline string survives the audit, it survives because it is a
measurement or a why, and the audit notes say which.
## Testing & Quality
- Unit: `tests/test_api.py` (absence pins for the deleted strings; served-bytes shape
unchanged otherwise), `node --check`.
- Browser (DoD): screenshot in `data/logs/` — the four page heads without the removed
sub-lines; no empty containers.
- Coverage: **>90%** on modified code; overall gate (94%) maintained.
## Completion Criteria
- [ ] the two quoted lines (and the rest of the audit's deletions) are gone from served HTML and JS
- [ ] functional copy (provenance tags, Δ footnotes, empty-state explanations) intact
- [ ] `AGENTS.md` carries the flavour-text rule with the deleted lines as cited examples
- [ ] `uv run pytest` green; `uv run ruff check src tests agent && uv run pyright` clean
@@ -0,0 +1,53 @@
# Phase 09 — A /disks page for physical disk health (smartctl)
**Source:** `TODO.md` L8 — "a truly physical disk monitoring page": "Add a 'disks' tab dedicated to smartctl and per-disk metrics. Note: not per-partition, but truly a physical disk monitoring page that collects data about all my physical disks across my fleet (should only be deskwork and driveripper right now)."
**Story:** n/a
**Context:** `agent/host_agent.py` (physical whole disks via `/sys/block`, ~L400 — already partition-free; stdlib-only, `ProtectSystem=strict`), `src/dashboards/config.py` (env only, sources default to unset), `src/dashboards/poller.py` (one task per source, stale/last-good, events under the dashboard name), `src/dashboards/store.py` (WAL, rollups raw→1m→1h, retention — rule 9), `src/dashboards/api.py` (GETs, 404-when-unconfigured without dialling), the three existing dashboards + `/` overview (a new tab joins the nav and the overview's "which of the three" table)
## Objective
A fourth dashboard: `/disks` — one card per **physical** disk (the agent's whole-
`/sys/block` enumeration, never partitions), each carrying `smartctl` health: overall
health, temperature, power-on hours, and the sector/wear alarms. Its own `disks` source
and poller task at a slow 5-minute cadence (SMART is not a per-second signal), its own
API (`/api/disks/snapshot`, `/api/disks/history`), rollups and retention, and a page that
reads `n/a` for what `smartctl` cannot see — never red for a missing binary.
- ASSUMPTION (phase, confirmed): the agent reads SMART **directly** (`smartctl` as the
agent user, e.g. `disk` group) — no sudo — and `DISKS_INTERVAL_SECONDS` defaults to
**300** s. `smartctl` absent or a disk unreadable → that disk's `smart` is `null`
with an error entry; the page says "smartctl not available" dimmed (invariants 3/4:
unconfigured is not broken; a dead source must not blink another).
## Dependencies
- `08_events_paging_and_copy` (todo) — sequence only: it settles `app.js`/`dash.css`
(the pager + copy audit) that the new page builds on; no hard code dependency.
## Tasks
1. `01_agent_smartctl_collection.md` — `host_agent.py` gains a `smartctl` pass over its existing whole-disk list (per-device `smartctl -i -A -H`, stdlib `subprocess`, timeout-bounded); fixture from the real disks; deploy docs.
2. `02_disks_source_poller_store.md` — the `disks` source (payload + `disk.*@dev` samples), its own poller job at `DISKS_INTERVAL_SECONDS` (default 300), rollups + retention in `store.py`.
3. `03_disks_api_and_config.md` — `/api/disks/snapshot` + `/api/disks/history`, config wiring, `404 {"configured": false}` without dialling, severity rows for the health/temperature metrics.
4. `04_disks_page_and_overview.md` — the `/disks` page (per-disk cards, alarm chips, per-host history charts), nav tab + overview entry + header status dot.
## Testing & Quality
- Hermetic: fixture-driven agent tests (`tests/test_agent.py` + recorded `smartctl`
output), `tests/test_disks_source.py` (new), `tests/test_api.py` key-path pins (with
and without config), `node --check`.
- Live: `uv run pytest -m live --live` — the agent on deskwork/driveripper answers
`smart` fields for its real disks; **live tests must not write** (no `smartctl -t`, no
self-test on hardware something is plugged into).
- Browser (DoD): screenshots in `data/logs/` — per-disk cards on live data, the
"smartctl not available" state, the overview with the new tab.
- Coverage: **>90%** on new/modified code, ≥94% overall gate maintained.
## Completion Criteria
- [ ] `GET /api/disks/snapshot` answers per-disk `smart` data for every physical disk the agents report; `404 {"configured": false}` with no agents configured, without dialling
- [ ] `/api/disks/history` serves the `disk.*@dev` series through the existing rollups
- [ ] the `/disks` page renders one card per physical disk (never per-partition) with health, temp, power-on hours and non-zero alarm chips
- [ ] a host without `smartctl` (or a denied read) shows dimmed `n/a` cards, never a red page
- [ ] `bash .agents/validate.sh` green; screenshots in `data/logs/`
- [ ] you have verified with real smartctl data from driveripper. The `ops` user has `sudo smartctl*` access via sudoers.
@@ -0,0 +1,71 @@
# Task 01 — The agent collects smartctl per physical disk
**Phase:** `09_disks_smartctl` · **Source:** `TODO.md:8` — "…collects data about all my physical disks across my fleet" (the collection half)
**Story:** n/a
## Objective
`agent/host_agent.py` answers SMART data for every physical disk it already enumerates
(whole `/sys/block` devices — the existing partition-free list), via a direct,
timeout-bounded `smartctl` call, without ever breaking the agent: no binary, a denied
read, or a wedgy device degrades that disk's `smart` to `null` with an `errors[]`
entry and leaves the rest of the snapshot intact.
## Work
1. `agent/host_agent.py` — a `smart()` pass inside the existing disk collection: for each
physical whole disk (the existing `/sys/block` enumeration, skipping the same
virtual/`dm-*` entries it already skips), run `smartctl -i -A -H <device>` via stdlib
`subprocess` with a per-call timeout (ASSUMPTION: **5 s per device**; the whole pass is
bounded so a wedgy disk cannot stall the snapshot). Parse, from the recorded output:
`model`/`serial` (from `-i`, falling back to the agent's existing disk identity when
smartctl's identity block is absent), `overall_health` (OK/FAILED), `temp_c`
(air-temperature line, absent on many SSDs → `None`), `power_on_hours`,
`reallocated_sector_ct`, `current_pending_sector_ct`, `offline_uncorrectable` (media
errors), `percentage_used` (SSD wear). Attach as `smart: {…}` per disk row in the
`GET /snapshot` docstring and payload.
2. Failure semantics: `smartctl` missing → **one** `errors[]` entry per snapshot (not one
per disk), every disk's `smart` is `null`; a single denied/wedgy device → only that
disk's `smart` is `null` with its own error entry. The agent's `/health` stays up:
SMART is a best-effort add-on, exactly like the existing per-section `_safe()`
handling.
3. Stdlib-only is kept (no new agent dependency; `subprocess` is stdlib, consistent with
the locked "our own stdlib host agent" decision). `GET /info` is unchanged (smartctl
is a per-snapshot capability, not identity).
4. Fixtures (rule 7 — never trust a memory of an upstream): record `smartctl -i -A -H`
output from the **real** disks on deskwork and driveripper (one representative SSD and
one HDD if the fleet has both) into `tests/fixtures/smartctl_<dev>.txt`, captured by
an extension of `scripts/capture-fixtures.sh` (same script, same PII/serial scrubbing
as the existing captures — a serial scrubbed in a fixture is not a serial in the
browser). The parse tests assert against these verbatim captures; a `smartctl` version
drift that breaks a field fails the fixture test before a dashboard does.
5. `tests/test_agent.py` — a fixture-rooted collector case: happy path (fields present),
binary-missing (one error entry, all `smart: null`), single-device failure (only that
disk null), non-physical device skipped.
6. Deployment docs: `deploy/install-agent.sh` (or its README section) and `README.md` note
the requirement — `smartmontools` installed and the agent user able to read SMART
(e.g. `disk` group); absence is a documented graceful degradation, not an install
failure (DoD: fleet facts travel with the change).
- ASSUMPTION: **direct read, no sudo** — `smartctl` runs as the agent user (confirmed
with the operator); if a host's permissions deny it, that host's disks degrade to the
"smartctl not available" state, and the fix is a group grant, not a sudoers entry
(command-scoped sudo exists for NUT, but SMART reads do not need root and the agent's
`ProtectSystem=strict` unit stays as-is).
## Testing & Quality
- Unit: `tests/test_agent.py` against the recorded fixtures (all four failure/happy modes
above); the real-`/proc` test keeps touching only the box running pytest (existing
policy — a box without `smartctl` exercises the binary-missing path, which is itself
a valid assertion).
- Live: `uv run pytest -m live --live` — the live agents' snapshots carry `smart` fields
for the real disks (read-only; no `smartctl -t` anywhere in the suite).
- Coverage: **>90%** on the agent's new code; overall gate (94%) maintained.
## Completion Criteria
- [ ] `GET /snapshot` disk rows carry `smart` (model/serial/health/temp/power-on/sectors/wear) on a host with a readable `smartctl`
- [ ] missing binary → one error entry + `smart: null` everywhere; one bad device → only that disk null; agent `/health` stays up
- [ ] `tests/fixtures/smartctl_*.txt` in the repo, reproducible via `./scripts/capture-fixtures.sh`, serials scrubbed
- [ ] `uv run pytest` green; `uv run ruff check src tests agent && uv run pyright` clean
@@ -0,0 +1,66 @@
# Task 02 — The `disks` source, its slow poller job, and the store's rollups
**Phase:** `09_disks_smartctl` · **Source:** `TODO.md:8` — the storage + polling half of the physical-disk dashboard
**Story:** n/a
## Objective
`disks` becomes a first-class source: one payload per host (the physical disks and their
`smart` fields), one `disk.*@dev` sample set per poll, its **own** asyncio poller task at
a slow 5-minute cadence — SMART is not a per-second signal, and a dead `disks` job must
age its numbers in the UI exactly like any other source (invariant 4), with the store's
existing rollups and retention doing the history work (rule 9).
## Work
1. `src/dashboards/config.py` — `DISKS_INTERVAL_SECONDS` (default **300**; env-only, the
existing pattern). The `disks` source is *configured* when at least one host agent
target is configured — it reads the same `/snapshot` the `system` source does, so it
has no targets of its own; with no host targets it is unconfigured and 404s without
dialling (invariant 3).
2. `src/dashboards/sources/disks.py` (new) — snapshot → payload per host: the physical
disk rows (device, model, size, rotation from the agent's existing `disks` fields)
plus `smart` (task 01's fields, `None` where the agent had nothing — the page prints
`n/a`, never `0`), plus a `smartctl_available` flag per host so the page can say
*why* a card is `n/a` instead of leaving it a mystery. Samples (dotted, `@dev` stream
suffix, the existing naming scheme — the history API's key):
`disk.health_ok@<dev>` is **not** a series (it is a verdict, not a number); the
series are `disk.temp_c@<dev>`, `disk.power_on_hours@<dev>`,
`disk.reallocated@<dev>`, `disk.pending@<dev>`, `disk.uncorrectable@<dev>`,
`disk.percentage_used@<dev>` (a count/percent metric, not a rate — no `Δ` window
applies). Events under the **dashboard** name `disks` (the phase-03 lesson: a page
filtering `/api/events` by source gets an empty feed otherwise).
3. `src/dashboards/poller.py` — register the job (one task per source, its own timeout
and health record; the 300 s cadence means the stale threshold — 3× its own interval —
is 15 min, and `alerts.py`'s existing ok-but-stale `warn` rule covers it with no
change: the alert reads the job's own interval).
4. `src/dashboards/store.py` — the new series follow the existing rollup path
(raw → 1m → 1h) and retention; at a 300 s cadence the raw rows are 288/day/host/disk —
the retention story is the existing one, but the task must verify the rollup window
arithmetic handles a 300 s raw resolution (a 1-minute rollup bucket straddles two
polls; the existing "latest wins" semantics must keep holding, asserted by a test).
5. `tests/test_disks_source.py` (new) — fixture-snapshot → payload/samples (fields present,
`null` where the agent had none, `smartctl_available` false → all `n/a`), the 300 s
cadence in the job registration, the unconfigured-without-dialling 404 path, the
rollup-bucket test.
- ASSUMPTION: `power_on_hours` and the sector counters are stored as **counters-as
reported** (monotonic by nature) — no `Δ` rate is derived; a rate the upstream does not
publish is `n/a` (rule 2). `disk.health_ok` (OK/FAILED) lives in the payload only;
colour comes from `severity.classify` on the numeric rows (task 03).
## Testing & Quality
- Unit: `tests/test_disks_source.py` (the full matrix above), `tests/test_poller.py`
additions (the job's own timeout/health, a failed poll keeps the last good payload and
marks stale — the invariant-4 behaviour on the new job), `tests/test_store.py`
(rollup/retention on the new series at 300 s resolution).
- Coverage: **>90%** on new/modified code; overall gate (94%) maintained.
## Completion Criteria
- [ ] `uv run pytest` green with the new source/job/store tests
- [ ] with no host targets, the `disks` job is unconfigured and 404s without dialling
- [ ] a failed `disks` poll ages the payload (stale in the UI) instead of zeroing it
- [ ] 1-minute rollup buckets are correct at 300 s raw resolution (asserted, not assumed)
- [ ] `uv run ruff check src tests agent && uv run pyright` clean
@@ -0,0 +1,62 @@
# Task 03 — The disks API, config wiring, and the severity rows
**Phase:** `09_disks_smartctl` · **Source:** `TODO.md:8` — the API half of the physical-disk dashboard
**Story:** n/a
## Objective
`/api/disks/snapshot` and `/api/disks/history` answer like every other dashboard's
endpoints (GETs only, 404-when-unconfigured, redacted error strings, the health record in
`/api/health`), and the numbers that *are* worth a verdict get one row in
`severity.py` — the only place thresholds live — while the alarm counters get none
(a non-zero sector count is an incident, not a graded number: presence, like the
`/llm` preemption chips, never a threshold).
## Work
1. `src/dashboards/api.py` — `GET /api/disks/snapshot` (per-host: disk rows + `smart` +
`smartctl_available`) and `GET /api/disks/history` (the `disk.*@dev` series through
the existing history mechanism; unknown names answer as empty series on purpose, the
same contract `/api/system/history` and `/api/llm/history` have). No POST anywhere on
this surface (invariant 1 — the one write in the app stays `POST /api/ups/command`).
2. `404 {"configured": false}` with **no dialling** when no host agent is configured
(invariant 3), asserted with a test that would catch a client being constructed before
the gate. `UpstreamError` messages carry redacted URLs only (rule 6 — the existing
`Target.display_url` path; the disks source reuses the host targets' redaction, so
nothing new leaves the process with a token).
3. `src/dashboards/severity.py` — the new rows (ASSUMPTION — proposed bounds, pinned by
test, adjustable in one place without touching the page):
- `disk.temp_c`: warn **50**, fail **70** (a HDD spinning at 70 °C is a fire; an SSD
without a sensor simply has no value and gets no verdict — no measurement, no
verdict);
- `disk.percentage_used`: warn **80**, fail **100** (a 100 %-used SSD is a
replacement, not a warning).
The sector counters (`disk.reallocated`, `disk.pending`,
`disk.uncorrectable`) get **no row**: non-zero is an alarm chip, zero is the
absence of one (never green).
4. `src/dashboards/alerts.py` — no change expected: the ok-but-stale `warn` and the
dead-source `fail` rules read the job's own interval and reachability, and the
unconfigured-never-alerts rule covers the unconfigured case. **If** a test shows the
new job needs nothing, say so in the report; an alert with its own threshold is a
bug (the module is a pure projection).
5. `src/dashboards/app.py` — assembly wiring (settings/store injection, so tests build an
app with an in-memory DB and no targets, the existing pattern).
6. `tests/test_api.py` — key-path pins for `/api/disks/snapshot` (with and without
targets; a configured job with no payload yet lists the hosts as stubs, `n/a`
everywhere — the same stub shape `/api/system/snapshot` has), `/api/disks/history`
(known + unknown series), the 404-without-dialling case, and `/api/health` carrying
the new job's record.
## Testing & Quality
- Unit: `tests/test_api.py` (the pins above), `tests/test_severity.py` (the two new rows,
boundary ordering, the counter metrics deliberately absent from `THRESHOLDS`),
`tests/test_alerts.py` (the new job's stale/dead/unconfigured states).
- Coverage: **>90%** on modified code; overall gate (94%) maintained.
## Completion Criteria
- [ ] `curl /api/disks/snapshot` answers per-host disk + smart data (or `404 {"configured": false}` with no dial)
- [ ] `/api/disks/history` serves the `disk.*@dev` series; unknown names → empty series
- [ ] `classify("disk.temp_c", 75)` is `fail`, `classify("disk.reallocated", 1)` is `None` (presence, not a verdict)
- [ ] `bash .agents/validate.sh` green
@@ -0,0 +1,74 @@
# Task 04 — The /disks page, the nav tab, and the overview entry
**Phase:** `09_disks_smartctl` · **Source:** `TODO.md:8` — "Add a 'disks' tab dedicated to smartctl and per-disk metrics" (the page half)
**Story:** n/a
## Objective
The fourth dashboard: `/disks` — one card per **physical** disk (the agent's whole-
`/sys/block` enumeration; a partition is never a card), led by what the operator needs
while a disk is acting up: health, temperature, power-on hours, and the sector alarms —
plus per-host history charts under the same discipline as `/system` (collapsed by
default, phase 06; one history request per host per refresh; charts belong to the host
they chart).
## Work
1. `src/dashboards/web/pages/disks.html` + `src/dashboards/web/static/page-disks.js`
(new) — the page, in the existing vocabulary (masthead nav + a `Disks` tab with
`aria-current` on itself, `.card`, `.heroes`-free layout — the disk card leads with
the disk identity, not a big number):
- **per-disk card**: host + device + model (+ size / rotation from the agent's
existing fields), a health line (OK/FAILED from `smart.overall_health`, coloured
from the payload's severity verdict — a FAILED health is `fail` by the existing
`disk.percentage_used`/`temp_c` rows or the explicit health field; **never** a
verdict the API did not send), temperature, power-on hours, percentage-used where
the device has it (HDDs may not — `n/a`, uncoloured);
- **alarm chips that exist only while non-zero** (the `/llm` `.chip.fail` idiom):
reallocated, pending, uncorrectable — a disk at zero shows no chip row at all,
because a permanent "0 pending sectors" row trains the eye to ignore it;
- **`smartctl` unavailable** (per host, `smartctl_available: false`): the host's
cards render dimmed (`.stale`) with `n/a` and a one-line why — "smartctl not
available on <host>" — never a red panel (invariant 3);
- **per-host chart section** (phase-06 discipline): collapsed `<details>` "Charts" by
default, lazy-drawn `LineChart`s for `disk.temp_c@<dev>` (and
`disk.percentage_used@<dev>` where present) through **one**
`/api/disks/history` request per host per refresh naming every series it could draw;
a host that stops reporting loses its cards *and* charts (`pruneHosts` pattern).
2. `src/dashboards/web/pages/*.html` — the `Disks` tab joins the masthead nav on all
four pages (the orphan-`id` rule: every new `id` is referenced); the overview's
"which dashboard do you need" table gains the `/disks` row
("are any of my physical disks failing, and which one") and the header's
per-source status dot picks up the `disks` job.
3. `src/dashboards/web/pages/index.html` + `page-index.js` — `/api/summary` is the
overview's single call; add the `disks` payload to its contract **additively**
(the existing key paths stay pinned — `tests/test_api.py` fails if a summary key
retires), and the Fleet/Warnings panels treat the `disks` source like the others
(an alert from `alerts.py` needs nothing new; a dead `disks` job is one line).
4. `src/dashboards/web/static/dash.css` — the disk-card + chip styling from existing
tokens (no new palette, no `.sev-*` family); `node --check` on every touched script.
5. `tests/test_api.py` — pin the served markup (nav tab on all four pages, overview row,
the page's card/empty/stale states) and the `/api/summary` additive key path.
- ASSUMPTION: the page's hero per card is **identity + health**, not a single big number
(a disk has no one-number story the way a UPS has charge %); the numbers that matter
(temp, hours, wear) are equal-size `.metric` cells, and the verdicts come only from
task 03's `severity` rows — `n/a` where the device does not publish the field.
## Testing & Quality
- Hermetic: `tests/test_api.py` (markup + summary contract), `node --check`, the
orphan-`id`/orphan-selector rules.
- Live: `uv run pytest -m live --live` (read-only); the real fleet today has physical
disks on deskwork and driveripper — the page must show both hosts' cards.
- Browser (DoD): screenshots in `data/logs/` — live cards on both hosts, the
"smartctl not available" state (a host with the binary removed, or a `smartctl_available:
false` fixture), collapsed charts expanded, the overview with the new tab + row.
- Coverage: **>90%** on new/modified code; overall gate (94%) maintained.
## Completion Criteria
- [ ] `/disks` renders one card per physical disk on live data (never per-partition), with health/ temp/hours/wear and non-zero-only alarm chips
- [ ] a `smartctl`-less host is dimmed with a one-line why — never red
- [ ] nav tab on all four pages; the overview's table lists `/disks`; the header dot tracks the job
- [ ] `bash .agents/validate.sh` green; screenshots in `data/logs/`
@@ -0,0 +1,104 @@
# Phase 10 — Dev dials the fleet directly (no more SSH tunnels)
**Source:** user request (2026-10) — "I don't want this separation of architectures. My laptop can
talk to these hosts directly via local network or vpn already. Just have dev work like prod."
**Story:** n/a
**Context:** `AGENTS.md` (dev workflow: `scripts/dev-tunnels.sh`, `DEV_TUNNELS`/`DEV_FORWARDS`),
`deploy/install-agent.sh`, `deploy/nut-sudoers`, `scripts/dev-server.sh`,
`scripts/capture-fixtures.sh`, `tests/test_live_fleet.py`, `.env.dev.example`, `.env.example`,
`README.md`
## Objective
Dev becomes the prod topology: each host's agent runs as a **systemd service** (installed by
`ops` through a documented minimal sudoers grant), bound to the address the dashboard dials,
token in the URL userinfo; driveripper's upsd ACLs the host and laptop IP ranges on port 3051;
`.env.dev` points at the real hosts with no `DEV_TUNNELS`/`DEV_FORWARDS`; `scripts/dev-tunnels.sh`
is retired. The dashboard's Python is untouched — this phase changes deployment, config, scripts
and docs, and proves parity with the live suite plus browser screenshots.
## Decisions (user, 2026-10)
- The agent is a **systemd service, installed by the `ops` user** (not root). The repo documents
the **simplest possible working sudoers addition** in `deploy/agent-sudoers`, and the user adds
it to each host.
- The upsd ACL allows the **host's IP range and the laptop's IP range** — both real ranges,
derived at install time and commented in `.env.dev` so they can be re-derived when the laptop's
IP changes.
## Precondition (user action, before task 02)
The sudoers line needs root, and the command-scoped sudo on driveripper does not include
`visudo`. The user adds the one line from `deploy/agent-sudoers` on each host. If it is missing
when task 02 runs, that task must **stop cleanly and print the exact instructions** — never loop,
never wait, never attempt to edit sudoers itself.
## Dependencies
- `05_ups_hero_battery_glyph` (complete) — the last UI phase; its pages are what this phase
re-verifies against live data after the transport change.
- `06_system_page_gpu_verdicts`, `07_llm_monitor_style_pages`, `08_events_paging_and_copy`,
`09_disks_smartctl` (todo) — independent: this phase changes transport and ops, not payloads or
page behaviour, and they do not depend on the transport.
## Tasks
1. `01_sudoers_grant_and_installer_hint.md` — `deploy/agent-sudoers` (the minimal working grant
+ install instructions, in the `nut-sudoers` house style); `install-agent.sh` run without
root prints the exact line to add.
2. `02_install_agents_on_fleet.md` — on all three hosts: kill the ad-hoc agent, stage the
installer, install the systemd service bound to the dialed address, capture the printed
tokens, verify from the laptop.
3. `03_upsd_acl_for_laptop.md` — driveripper: append ACL lines for the host range and the laptop
range to `upsd.conf` (read-first, `cmp`-after, never blind-write), restart `nut-server`, verify
the NUT wire from the laptop.
4. `04_env_dev_direct_dial.md` — stop the tunnels; rewrite the local `.env.dev` to direct targets
(userinfo tokens, `NUT_HOST`/`NUT_PORT=3051`); restart the dev server; sources ok.
5. `05_retire_tunnels_update_scripts.md` — delete `scripts/dev-tunnels.sh`; rewrite
`.env.dev.example` direct-shaped; fix `capture-fixtures.sh`, the live suite's docstring and
skip messages, `ups_contract.py`, the `alerts.py` docstring, `test_config.py`, `.env.example`'s
NUT note, and the dev-workflow sections of `README.md`/`AGENTS.md`.
6. `06_verify_direct_dial_live.md` — full live verification: `/api/health` ok over direct dial,
`uv run pytest -m live --live` green, browser screenshots of `/` + `/system` + `/ups` + `/llm`
on live data, no tunnel port needed on the laptop.
## Testing & Quality
- The Python changes in this phase are test-file strings and one docstring — no new logic. The
full suite must stay 100% green and overall coverage must not drop (gate: `--cov-fail-under=94`).
- `bash -n` on `deploy/install-agent.sh` and `scripts/capture-fixtures.sh` after their edits.
- The real proof of this phase is the live path: `uv run pytest -m live --live` against the
directly-dialed fleet (the same production code paths over the prod-shaped transport) plus a
browser screenshot per `AGENTS.md` rule 10.
- The phase is "Complete" only when the **entire** project test suite passes 100% (every test in
the project, not just the phase's — pre-existing failures included) and the coverage gate in
`.agents/validate.sh` holds.
## Documentation & Consistency
- The final pass reviews `README.md` and `AGENTS.md` in full against the project as it stands
after this phase — fixing every inconsistency found, **including pre-existing ones not caused
by the phase** (including any remaining mention of `dev-tunnels.sh`, `DEV_TUNNELS`,
`DEV_FORWARDS`, `9121`/`9122`/`9123` or `3494`).
- Task 05 updates `README.md`/`AGENTS.md` immediately for the user-visible dev-workflow change.
- `PLAN.md` is locked and must not be edited: its Data-Model note ("forwarded to
`127.0.0.1:3494` in dev") goes stale after this phase — flag it in the final pass's report to
the user, and leave the file untouched.
## Completion Criteria
- [ ] `systemctl status dashboards-agent` is active on all three hosts, each bound to the address
the dashboard dials, answering `/health` (unauthenticated) and `/snapshot` (bearer) from the
laptop
- [ ] driveripper's upsd answers `LIST UPS` from the laptop on 3051 — not via the host's loopback
- [ ] `git grep -n "dev-tunnels\|DEV_TUNNELS\|DEV_FORWARDS\|9121\|9122\|9123\|3494"` matches
nothing outside `.agents/phases/complete/`
- [ ] `./scripts/dev-server.sh status` shows every source `ok` over direct dial
- [ ] `uv run pytest -m live --live` (with `.env.dev` sourced) is green
- [ ] `data/logs/` holds `phase10-*.png` screenshots of `/`, `/system`, `/ups`, `/llm` on live
data over direct dial
- [ ] full test suite **100% passing**, coverage gate (≥94%) holds, `ruff` + `pyright` clean
- [ ] `README.md` and `AGENTS.md` describe the direct-dial dev workflow and mention no tunnels
(verified in the final pass)
- [ ] no behavior change in completed phases: the four pages render the same payloads as before
the transport change
@@ -0,0 +1,59 @@
# Task 01 — The agent sudoers grant and the installer's hint
**Phase:** `10_dev_direct_dial` · **Story:** n/a
## Objective
A user with the command-scoped `ops` sudo (no root) can install `deploy/install-agent.sh`
because the repo documents the single sudoers line that makes it work, and running the
installer without root tells you that exact line instead of the generic "run as root".
## Work
1. `deploy/agent-sudoers` (new) — house style: follow `deploy/nut-sudoers` (header with the
`visudo -cf` check-first install steps, a "WHY THESE ARE NEEDED" block, scope notes, the
grant, a "deliberately NOT granted" block). Content:
- **Why:** the agent is read-only by construction and every non-`/health` request needs the
bearer token, so the only privileged operation is installing its files and unit. The grant
is the whole script path, run as root, with no argument restriction — the flags
(`--bind`, `--port`, `--status`, `--uninstall`, `--rotate-token`) are non-sensitive, and
the documented flow never puts a token on argv (the installer generates one and prints it
once).
- **Canonical staging path:** the host has no repo checkout, so the host gets
`deploy/` + `agent/` staged to `~/agent-install/` (task 02 does the staging). The grant
line is therefore stable:
```
ops ALL=(root) NOPASSWD: /home/ops/agent-install/deploy/install-agent.sh
```
- **Install steps** (per host, the same two-stage check as `nut-sudoers`):
`sudo visudo -cf deploy/agent-sudoers` (adapt the path first), then install to
`/etc/sudoers.d/dashboards-agent` with `install -m 0440 -o root -g root`, and
`sudo visudo -cf` on the installed copy.
- **Deliberately NOT granted:** `visudo`/`vim`/`sh` (no self-amendment of the grant),
`systemctl *` other than what the script itself already needs as root, `cat` of
`/etc/dashboards-agent.token` (the token stays root-readable; the installer prints it
once), and the NUT instant-command clients.
2. `deploy/install-agent.sh` — replace the `need_root` failure message for the non-root case:
when `id -u != 0`, print (a) the exact NOPASSWD line for the script's **resolved absolute
path** (so a checkout at a different path still works), (b) the two install steps from
`deploy/agent-sudoers`, and (c) a pointer to `deploy/agent-sudoers`. Keep the exit code
(1) and the behaviour when already root unchanged. `bash -n` must stay clean.
## Testing & Quality
- `bash -n deploy/install-agent.sh` clean.
- Run the script **as the current (non-root) user** locally: `bash deploy/install-agent.sh
--bind 127.0.0.1` must exit 1 and print the hint with this checkout's absolute path; the
printed line must match the file's canonical line apart from the path (diff them in
`data/logs/NOTES-sudoers.md`).
- No Python changes this task; the full suite must stay green.
## Completion Criteria
- [ ] `deploy/agent-sudoers` exists, passes `visudo -cf` syntax locally, and names the canonical
staging path and the exact grant line
- [ ] non-root `bash deploy/install-agent.sh --bind 127.0.0.1` prints the exact line to add and
exits 1
- [ ] root behaviour unchanged (the script still refuses nothing new; `need_root` logic intact)
- [ ] `bash -n` clean on the script; `uv run pytest` 100% green; `ruff` + `pyright` clean
- [ ] README.md/AGENTS.md unchanged yet (task 05 is where the docs change)
@@ -0,0 +1,66 @@
# Task 02 — Install the agents as systemd services on the fleet
**Phase:** `10_dev_direct_dial` · **Story:** n/a
**Precondition (user action):** the one line from `deploy/agent-sudoers` is installed on each
host (root-only; the scoped sudo does not include `visudo`). If it is missing on a host, this
task must stop cleanly for that host and leave the exact instructions in
`data/logs/NOTES-agent-install.md` — never loop, never wait, never edit sudoers.
## Objective
Each of the three hosts (`driveripper`, `deskwork`, `gateway`) runs `dashboards-agent` as an
active systemd service — the prod shape — bound to the address the dashboard will dial, with
its bearer token captured for `.env.dev`. The old ad-hoc tunnel agent is gone.
## Work
1. Once, on the laptop: `./scripts/dev-tunnels.sh stop` (kills the local ssh legs; the remote
ad-hoc agents are orphans and are killed in step 2).
2. Per host (`ssh driveripper-ops | deskwork-ops | gateway-ops`):
- `ssh <host> "pkill -f 'host_agent.py --port 9120' || true"` — the orphaned ad-hoc agent
from the old dev flow (it runs as `ops`, so `ops` can kill it). Leave the
`~/lab-agent/` files on disk; they are simply no longer used.
- Stage the installer: `ssh <host> "mkdir -p ~/agent-install"` then
`rsync -a --delete deploy/ agent/ <host>:~/agent-install/`.
- Check the grant: `ssh <host> "sudo -n ~/agent-install/deploy/install-agent.sh --status"`.
On a permission failure: append to `data/logs/NOTES-agent-install.md` the host, the exact
line from `deploy/agent-sudoers` (with this host's staging path), and the two `visudo`
install steps; mark the host **BLOCKED**, continue with the next host, and end the task
with the blocker visible in the summary. Do not retry or wait.
- Install: `ssh <host> "sudo ~/agent-install/deploy/install-agent.sh --bind <addr> --port 9120"`.
- `<addr>` is the address the dashboard will dial. Derive it: the host's name from
`.env.dev` (the `*.reeselink.com` names used by the engines, or the LAN name the user
uses), resolved on the laptop (`getent hosts <name>`), then confirmed present on the
host (`ssh <host> "ip -o addr"`). If resolution is ambiguous, use `0.0.0.0` — the
installer requires a token off-loopback, which the flow always provides.
- The installer prints the new token exactly once on first install. Record it for task 04
(`data/logs/NOTES-agent-install.md` may hold `host:port + unit state` only — **no
tokens** in NOTES; the tokens go into `.env.dev` in task 04, which is `0600` and
gitignored, the same place `NUT_PASSWORD` already lives).
- Verify **from the laptop** (a host-side check could pass via loopback):
`curl -sf http://<host>:9120/health` → `{"ok": true}`, and
`curl -sf -H "Authorization: Bearer <token>" http://<host>:9120/snapshot | head -c 200`
shows the `cpu` object.
- `ssh <host> "systemctl status dashboards-agent --no-pager" | head -5` → `active (running)`.
3. If a firewall on any host blocks 9120 from the laptop (the `curl` in step 2 fails while
`systemctl status` is active), record it in `data/logs/NOTES-agent-install.md` with the host
and the port, and mark the host BLOCKED-with-details — the user's scoped sudo set may not
cover firewall edits on every host, and this task does not guess.
## Testing & Quality
- No Python changes; `uv run pytest` must stay 100% green (this is an ops task — its "tests"
are the verifications above).
- `data/logs/NOTES-agent-install.md` records per host: bind address, unit state, verification
outcome, and — for any BLOCKED host — the exact user instructions. No secrets in the file.
## Completion Criteria
- [ ] all three hosts: `dashboards-agent` active, verified from the laptop (or a host is
BLOCKED with the exact user instructions in `data/logs/NOTES-agent-install.md`)
- [ ] the old ad-hoc agent is dead on each host (the pkill in step 2 ran; no `9120` loopback
listener remains)
- [ ] tokens captured for task 04 (in the NOTES file only as host:port — the values go straight
into `.env.dev` there)
- [ ] `uv run pytest` 100% green; `ruff` + `pyright` clean
@@ -0,0 +1,54 @@
# Task 03 — upsd ACL for the host and laptop IP ranges
**Phase:** `10_dev_direct_dial` · **Story:** n/a
## Objective
driveripper's upsd (port 3051) accepts the NUT wire from the laptop over the network, using
the user-decided ranges — the host's IP range and the laptop's IP range — so the UPS source
can dial `NUT_HOST=<driveripper-addr> NUT_PORT=3051` directly. No `upsd.users` account is
touched, so `upsmon`'s orderly power-down is preserved.
## Work
1. Read first (the `NUT_READ` sudo grant on driveripper covers `cat /etc/ups/*`):
`ssh driveripper-ops "sudo cat /etc/ups/upsd.conf" > data/logs/upsd.conf.before`.
Confirm the file's current ACL lines (NUT 2.8 refuses non-local connections by default, so
there are likely none).
2. Derive the two ranges:
- **Laptop address as seen from driveripper:** `ssh driveripper-ops "ss -tn state
established '( sport = :22 )'"` — the source IP of this ssh session.
- **Host range:** driveripper's LAN subnet (`ssh driveripper-ops "ip -o addr"` — the
interface prefix, e.g. a `/24`).
3. Build the new file as `upsd.conf.before` **plus** exactly two appended NUT 2.8 ACL lines
(`ACL <ip-or-net> <netmask>`; a single host is mask `255.255.255.255`), write it back with
the granted `tee`, then prove the diff is exactly those two lines — the same read-first /
`cmp`-after care `AGENTS.md` mandates for `upsd.users` (never a blind write):
```
cat data/logs/upsd.conf.new | ssh driveripper-ops "sudo tee /etc/ups/upsd.conf"
ssh driveripper-ops "sudo cat /etc/ups/upsd.conf" | diff - data/logs/upsd.conf.before
```
4. `ssh driveripper-ops "sudo systemctl restart nut-server"` (the `NUT_WRITE` grant). `upsmon`
re-authenticates from `upsd.users`, which was not modified.
5. Firewall: `ssh driveripper-ops "sudo firewall-cmd --list-ports"` — if a firewall is active
and 3051 is not reachable from the laptop, add a rule limited to the laptop address (the
scoped sudo includes `firewall-cmd *` "only if upsd must serve the LAN"); record what was
added in `data/logs/NOTES-upsd-acl.md`. If no firewall is active, record that.
6. Verify **from the laptop** (a host-side check passes via loopback and proves nothing):
`printf 'LIST UPS\n' | nc -w 5 <driveripper-addr> 3051` → `BEGIN LIST UPS`. Task 04's
`/api/health` is the app-level proof.
## Testing & Quality
- No Python changes; `uv run pytest` must stay 100% green.
- `data/logs/NOTES-upsd-acl.md` records the two ACL lines, the diff output (must show exactly
the two added lines), the firewall state, and the laptop-side verification result.
## Completion Criteria
- [ ] `diff` between `upsd.conf.before` and the live file shows exactly the two ACL lines
- [ ] `nut-server` restarted cleanly (`systemctl status nut-server` active)
- [ ] from the laptop, `printf 'LIST UPS\n' | nc -w 5 <driveripper-addr> 3051` answers
`BEGIN LIST UPS` (with both UPS names following, or at least the BEGIN marker)
- [ ] no `upsd.users` modification (the diff is limited to `upsd.conf`)
- [ ] `uv run pytest` 100% green; `ruff` + `pyright` clean
@@ -0,0 +1,44 @@
# Task 04 — `.env.dev` goes direct
**Phase:** `10_dev_direct_dial` · **Story:** n/a
## Objective
The dev configuration dials the fleet the prod way: real host addresses with bearer tokens in
URL userinfo, upsd directly on 3051, and no `DEV_TUNNELS`/`DEV_FORWARDS` — with the dev server
showing every source healthy over the new transport.
## Work
1. `./scripts/dev-tunnels.sh stop` (idempotent; task 02 may have done it already).
2. Edit the local `.env.dev` (keep its `0600` mode; it is gitignored):
- delete the `DEV_TUNNELS` and `DEV_FORWARDS` lines
- `HOST_TARGETS="driveripper=http://<token>@<addr>:9120 deskwork=http://<token>@<addr>:9120
gateway=http://<token>@<addr>:9120"` — the addresses exactly as task 02 verified them,
the tokens task 02 captured.
- `NUT_HOST=<driveripper-addr>`, `NUT_PORT=3051`; keep `NUT_USER` / `NUT_PASSWORD` /
`NUT_ALLOW_ACTIONS` unchanged.
- `GUFO_BASE_URL` / `VLLM_BASE_URL` stay (already direct); `DASH_PORT` stays.
- add a short comment: where the upsd ACL lines live (`/etc/ups/upsd.conf` on
driveripper) and that a changed laptop IP means re-adding one ACL line (task 03's
method).
3. `./scripts/dev-server.sh restart`, then `./scripts/dev-server.sh status` — every source
`ok`. A host that was BLOCKED in task 02 may still be down; that is expected and must be
recorded, not "fixed" with a tunnel.
4. Watch the transition: a host that was tunnel-served should log an unreachable/ok (or
stale/ok) transition in `/api/events` — the poller's normal bookkeeping. Confirm the
transitions are the only new events and no source is stuck `DOWN`.
## Testing & Quality
- `uv run pytest` 100% green (no Python changes this task).
- `.env.dev` keeps mode `0600` and contains no token in any file git tracks.
## Completion Criteria
- [ ] `.env.dev` has no `DEV_TUNNELS`/`DEV_FORWARDS` and every target is a real address with
userinfo token (hosts) or direct `host:3051` (NUT)
- [ ] `./scripts/dev-server.sh status` shows every reachable source `ok` over direct dial
- [ ] `curl -s localhost:8090/api/system/snapshot` shows live data for every installed host
(a BLOCKED host is absent or stale — recorded, not hidden)
- [ ] the dev server shows no `DOWN` source that was healthy before the switchover
@@ -0,0 +1,69 @@
# Task 05 — Retire the tunnel machinery and update scripts and docs
**Phase:** `10_dev_direct_dial` · **Story:** n/a
## Objective
Everything that exists to serve the old architecture goes away or gets rewritten: the tunnel
script is deleted, the example env file is direct-shaped, and no tracked file mentions the
tunnel ports or variables any more.
## Work
1. `git rm scripts/dev-tunnels.sh` — the user decided against the separation; git history
keeps the script if it is ever needed.
2. `scripts/capture-fixtures.sh` — drop the "needs the tunnels up" line; the hard-coded
`HOSTS=(driveripper:9121 deskwork:9122 gateway:9123)` becomes a parse of `HOST_TARGETS`
from `.env.dev` (source it the way `dev-server.sh` does: `set -a; . .env.dev; set +a`,
then split `name=url` pairs and use the URL as-is, token userinfo and all — the app's own
`config.parse_targets` shows the shape). The NUT line already uses `$NUT_HOST`.
`bash -n` must stay clean.
3. `tests/test_live_fleet.py` — module docstring: replace the `./scripts/dev-tunnels.sh
start` line of the dev loop with `set -a; . .env.dev; set +a` (direct dial); both skip
messages become `pytest.skip("HOST_TARGETS is unset")`.
4. `tests/ups_contract.py` — the message that mentions `DEV_FORWARDS` (around L89): the UPS
hero row is contracted against the real drivers over direct dial; reword, drop the
forward reference.
5. `src/dashboards/alerts.py` — the docstring example `http://h:9121/snapshot` (around L138):
update to the direct-dial shape (`http://<host>:9120/snapshot`). Docstring only.
6. `tests/test_config.py` — the `HOST_TARGETS` monkeypatch `deskwork=http://127.0.0.1:9122`
(around L100): cosmetic, update to the direct shape (e.g.
`deskwork=http://tok@deskwork.example.net:9120`); the test stays hermetic.
7. `.env.example` — the NUT section's "reach it with an ssh forward or a tailnet address, or
bind a LAN address only if a firewall rule limits it": add the now-default third option —
ACL the dashboard's source range in `upsd.conf` (NUT refuses non-local connections by
default).
8. `.env.dev.example` — full rewrite, direct-shaped: the agent installed per host via
`deploy/agent-sudoers` + `deploy/install-agent.sh` (bound to the dialed address, token in
the URL userinfo), `NUT_HOST`/`NUT_PORT` pointing at the real upsd (keep the `ss -ltn` /
`nc 'LIST UPS'` port-discovery note — still the right first move), the upsd-ACL note, no
`DEV_TUNNELS`/`DEV_FORWARDS`, RFC 2606 placeholder hostnames (the file's existing
convention), and the "never commit this file" warning.
9. `README.md` — the dev section (around L40–49 and L275): replace the tunnel lines with the
direct-dial flow — the agent install (the `deploy/agent-sudoers` grant, the staged
installer, the token into `HOST_TARGETS` userinfo), the upsd ACL, `.env.dev`,
`./scripts/dev-server.sh restart`. State plainly that dev is the prod topology.
10. `AGENTS.md` — the fleet + Commands sections: the agents are systemd services installed by
`ops` via the `deploy/agent-sudoers` grant (bound to the dialed address, token in
userinfo); upsd serves the LAN for the ACLed ranges; the `dev-tunnels.sh` lines out of the
Commands block; "Nothing listens on the hosts in dev" is the opposite of the truth now and
must go. Keep the port-3051 discovery note (it is still why `3493` looks dead to a
default client).
## Testing & Quality
- `uv run pytest` 100% green (the edits are strings/docstrings; the suite must catch any
regression), `ruff` + `pyright` clean, `bash -n scripts/capture-fixtures.sh` clean.
- `git grep -n "dev-tunnels\|DEV_TUNNELS\|DEV_FORWARDS\|9121\|9122\|9123\|3494" --` must
match nothing outside `.agents/phases/complete/` (and this phase's own todo files, which
describe the change).
## Completion Criteria
- [ ] `scripts/dev-tunnels.sh` is deleted from the working tree
- [ ] `git grep` for the tunnel names/ports is clean (per Testing & Quality)
- [ ] `.env.dev.example` is direct-shaped and passes the same "copy and edit" workflow as
before
- [ ] `README.md` and `AGENTS.md` describe the direct-dial dev workflow with no tunnel
references (the phase's final pass re-checks both project-wide)
- [ ] full test suite 100% passing; `ruff` + `pyright` clean
@@ -0,0 +1,52 @@
# Task 06 — Verify the direct-dial fleet end to end
**Phase:** `10_dev_direct_dial` · **Story:** n/a
## Objective
Prove the claim this phase exists for — dev works like prod: every source healthy over the
direct transport, the live suite green against the real fleet, and the four pages rendering
live data in a browser, with no tunnel port existing on the laptop.
## Work
1. `./scripts/dev-server.sh restart`, then `curl -s localhost:8090/api/health` — every source
`ok` (a host BLOCKED in task 02 is the only acceptable `DOWN`, and it must be the one
recorded in `data/logs/NOTES-agent-install.md`).
2. Prove the laptop needs no tunnel port:
`ss -ltn | grep -E ':(9121|9122|9123|3494)\s'` → empty (the old tunnel ports are gone),
while the host sources in `/api/health` are still `ok`.
3. `set -a; . .env.dev; set +a; uv run pytest -m live --live` — green (or the normal
per-source skips; a source that is configured and reachable must not skip).
4. Browser verification (`AGENTS.md` rule 10) — screenshots to `data/logs/` as
`phase10-*.png`, all against live data over direct dial:
- `/` — the fleet table shows all three hosts with live numbers (any BLOCKED host dimmed
`n/a`, as designed), alerts consistent with the source states
- `/system` — three live host cards (CPU/MEM/TEMP/DISK/GPU/NET heroes populated where the
hardware reports it; gateway's absent GPU stays an uncoloured `n/a`)
- `/ups` — both real UPS cards, live charge/load/runtime, **no `SIMULATED` badge**
- `/llm` — the live engine(s) with throughput and latency cells
5. `data/logs/NOTES-verify.md` — what was verified, which hosts are BLOCKED (if any) with a
pointer to the task-02 notes, and the `PLAN.md` staleness flag: its Data-Model note
("forwarded to `127.0.0.1:3494` in dev", around L94) no longer describes reality —
**report it to the user; do not edit `PLAN.md`** (locked).
6. The full gate: `bash .agents/validate.sh`.
## Testing & Quality
- The live suite is read-only by policy — nothing in this task sends an instant command,
restarts a service, or writes to an upstream.
- `uv run pytest` (hermetic) 100% green; coverage gate (≥94%) holds; `ruff` + `pyright`
clean.
## Completion Criteria
- [ ] `/api/health`: every reachable source `ok` over direct dial; no `DOWN` source that was
healthy before the phase
- [ ] no tunnel port (9121/9122/9123/3494) listening on the laptop
- [ ] `uv run pytest -m live --live` green against the directly-dialed fleet
- [ ] `data/logs/phase10-*.png` exists for `/`, `/system`, `/ups`, `/llm`, showing live data
- [ ] `data/logs/NOTES-verify.md` written, including the `PLAN.md` staleness flag for the user
- [ ] `bash .agents/validate.sh` passes
- [ ] no behavior change in completed phases (same payloads, same pages — only the transport
underneath changed)
@@ -0,0 +1,110 @@
# Phase 11 — One admin, one password: the dashboards stop being public
**Source:** operator request in chat, 2026-10-17 — "password protect this dashboard for a single admin user… changeable once logged in, configured initially with an env var, not required to change… for local testing the default admin password should be `testpass`."
**Story:** n/a
**Context:** `src/dashboards/app.py` (route table, middleware, `PAGES`), `src/dashboards/api.py` (`AppContext`, `/api/config`, the one `POST /api/ups/command`), `src/dashboards/config.py` (env-only settings), `src/dashboards/store.py` (the existing `meta` table), `src/dashboards/web/static/app.js` (`fetchJSON`, `initChrome`), `.env.example` / `.env.dev.example`, `README.md` → *Configuration*, `AGENTS.md` → *Non-negotiable rules*
## Objective
Every page and every `/api` route except a small, explicit public list requires a signed-in
admin session; the admin password is seeded from `DASH_ADMIN_PASSWORD` on first start, is
changeable from a new `/settings` page, and does **not** have to be changed. The server refuses
to start when it has neither a stored password nor the env var — there is no way to run this
thing open by accident.
## Derived scope (from the operator, in chat)
- **Fail closed.** `DASH_ADMIN_PASSWORD` unset **and** no stored verifier → the process exits
with an operator-readable reason. `testpass` is the local-testing password (dev config +
the hermetic suite), never a production fallback.
- **`/api/health` stays public** on purpose — the Containerfile `HEALTHCHECK` and
`./scripts/dev-server.sh status` depend on it. Its shape must not change, and nothing new
may be added to it without treating it as a public endpoint.
- **Most secure, least likely to break** was the instruction for sessions/CSRF → stdlib
`hashlib`/`hmac` only: PBKDF2-HMAC-SHA256 verifier, HMAC-signed stateless session cookie,
a session-bound CSRF token on every write, and a default-deny gate. **No new dependency**,
so no `PLAN.md` anchor moves (`itsdangerous` + Starlette `SessionMiddleware` was the
alternative the operator declined by asking for "least likely to break").
## Dependencies
- `01_severity_language` … `05_ups_hero_battery_glyph` (complete) — the pages, the shared
`.masthead` chrome and `Dash.fetchJSON` that this phase wraps; nothing here may change what
they render once a session exists.
- `06_system_page_gpu_verdicts`, `07_llm_monitor_style_pages`, `08_events_paging_and_copy`,
`09_disks_smartctl` (todo) — **no** dependency in either direction. This phase touches
`app.py`, `config.py`, `store.py`, `app.js` and new files; those phases touch
`severity.py`, `sources/*`, page scripts. If they run first, re-read the `create_app(` call
sites in `tests/` before task 03 — that is the only overlap that can conflict.
- `10_dev_direct_dial` (todo) — **no functional dependency, four shared files**: it rewrites
`.env.dev.example`, edits `.env.example`'s NUT section, rewrites the local `.env.dev`, and
rewrites the dev sections of `README.md`/`AGENTS.md`. Run either order, with two rules: apply
this phase's `.env*` additions by **appending to whatever the file says at the time** (never
restore a `DEV_TUNNELS`/`DEV_FORWARDS` line that 10 deleted), and never delete a
`DASH_ADMIN_PASSWORD` line 10 touches. Whichever phase runs second re-checks the diff of the
other's doc edits rather than overwriting them.
## Tasks
1. `01_auth_primitives.md` — `src/dashboards/auth.py`: password hash/verify (self-describing PBKDF2), session mint/read, CSRF token, login throttle, origin check. Pure logic, no HTTP.
2. `02_verifier_config_fail_closed.md` — `store.get_meta`/`set_meta`/`delete_meta`, the four `DASH_*` auth settings, and `ensure_admin_verifier()` wired into the lifespan so a dashboard with no password anywhere refuses to start.
3. `03_session_gate_and_public_surface.md` — the default-deny middleware, `/login` route, `/api/auth/login|logout|status`, CSRF enforcement on every write, `fetchJSON` attaching the token and bouncing to `/login` on 401, plus the test that enumerates **every** route and demands a session.
4. `04_change_password.md` — `POST /api/auth/password`: current-password check, verifier replace, session-secret rotation (every other browser is out), event row, throttle reset.
5. `05_login_page.md` — `pages/login.html` + `static/page-login.js`: password field, inline accessible errors, lockout message, no CDN, theme parity with the other pages.
6. `06_settings_page.md` — `/settings`: change-password form, sign-out, session age; a `⚙` link in the masthead of all four dashboards.
7. `07_dev_config_docs_browser_verification.md` — env templates, the operator's gitignored `.env.dev`, README + AGENTS.md, then live browser proof with screenshots and the leak checks (`/api/summary` 401 anonymous, `/system` 303, `/api/health` 200).
## Testing & Quality
- Unit: `tests/test_auth.py` for every primitive — wrong password, tampered cookie, expired
cookie, cookie signed with a rotated secret, CSRF token from another session, throttle
backoff and reset, `needs_rehash` when the configured cost rises.
- Integration: `tests/test_auth_api.py` — anonymous 401 on every `/api/*` data route, 303 →
`/login?next=…` for HTML, login success/malfailure/lockout over HTTP, logout kills the
cookie, change-password invalidates the *old* session and the *other* browser's,
`POST /api/ups/command` refuses without the CSRF header even **with** a valid session.
- The default-deny guard: `test_every_route_that_is_not_public_requires_a_session` walks
`create_app().routes` and asserts nothing outside the public list answers anonymously. A
route added by a future phase that forgets the gate fails here, not in production.
- `tests/test_api.py` keeps passing unmodified in intent: `build_client()` logs in once with
`testpass`, so the ~200 existing HTTP assertions still assert what they assert. The nine
direct `create_app(Settings(...))` call sites in `tests/` get the same password.
- Coverage: **>90%** on new/modified code (`auth.py` should be near 100%); the overall gate
(94%, currently ~98%) must hold.
- The phase is "Complete" only when the **entire** project test suite passes 100% (every test
in the project, not just this phase's), including `uv run pytest -m live --live`.
## Documentation & Consistency
- The final pass reviews `README.md` and `AGENTS.md` in full against the project as it stands
after this phase — fixing every inconsistency found, **including pre-existing ones not
caused by this phase**.
- Tasks update `README.md`/`AGENTS.md` immediately when their own changes are user-visible
(setup, commands, configuration, features, conventions). This phase is user-visible in the
strongest sense: the first thing an operator meets is a login prompt.
## Completion Criteria
- [ ] `curl -si http://127.0.0.1:8090/system` → `303` to `/login?next=/system`; `curl -s
http://127.0.0.1:8090/api/summary` → `401 {"ok": false, …}`; `curl -s
http://127.0.0.1:8090/api/config` → `401`; `curl -s http://127.0.0.1:8090/api/health` →
`200` (the one deliberate public data route)
- [ ] login with `testpass` in a browser reaches `/`; a wrong password shows an inline error
and, after 5 failures, the lockout message with `Retry-After` honoured
- [ ] `/settings` changes the password; the same browser is signed out and gets back in with
the new one; a second browser (or a saved cookie) is rejected with 401
- [ ] with `DASH_ADMIN_PASSWORD` unset **and** no `auth.verifier` row, `uv run dashboards`
exits non-zero with a message naming the variable and the reset command
- [ ] `./scripts/dev-server.sh status` and the Containerfile `HEALTHCHECK` still work
unauthenticated (`/api/health` unchanged)
- [ ] `uv run pytest` green, `uv run pytest --cov --cov-fail-under=94` green,
`uv run ruff check src tests agent && uv run pyright` clean, `bash .agents/validate.sh`
green
- [ ] screenshots in `data/logs/`: `auth-login.png`, `auth-login-rejected.png`,
`auth-locked-out.png`, `auth-signed-in-overview.png`, `auth-settings.png` (AGENTS.md
invariant 10 — a UI change is screenshotted, not claimed)
- [ ] `README.md` documents the four new `DASH_*` variables, the reset procedure, and the fact
that a UI change wins over the env var; `AGENTS.md` carries the new invariant naming the
public surface; both are accurate and consistent with the whole project (final pass)
- [ ] no behavior change in completed phases: every payload, hero card, chart and gate the
five completed phases shipped still renders identically **once signed in**
@@ -0,0 +1,112 @@
# Task 01 — Auth primitives: hashes, session tokens, CSRF tokens, throttle
**Phase:** `11_admin_login` · **Source:** operator request 2026-10-17 — "password protect this dashboard for a single admin user"
**Story:** n/a
## Objective
`src/dashboards/auth.py`: every authentication decision as a pure, unit-testable function with
no HTTP and no I/O — password hash/verify, session token mint/read, CSRF token mint/verify,
login throttle, origin check. Nothing here knows about FastAPI; task 03 wires it.
## Work
1. New `src/dashboards/auth.py`, module docstring stating the design and **why** it is
hand-rolled (stdlib only — the same reasoning as `prom.py` and the NUT wire client; no
`itsdangerous`, no `bcrypt`, no session store to operate). Constants at the top:
```python
ADMIN_USER = "admin" # single admin; there is no user table and none is coming
SESSION_COOKIE = "dash_session"
CSRF_HEADER = "X-CSRF-Token"
SAFE_METHODS = frozenset({"GET", "HEAD", "OPTIONS"})
PBKDF2_ITERATIONS = 600_000 # OWASP's PBKDF2-HMAC-SHA256 figure; DASH_PBKDF2_ITERATIONS overrides
PBKDF2_ALGO = "pbkdf2_sha256"
```
2. Password verifier — self-describing so the cost can rise later without a migration:
- `hash_password(password: str, *, iterations: int = PBKDF2_ITERATIONS) -> str` →
`pbkdf2_sha256$<iterations>$<salt_hex>$<hash_hex>`, salt `secrets.token_bytes(16)`,
`hashlib.pbkdf2_hmac("sha256", password.encode(), salt, iterations)`.
- `verify_password(password: str, encoded: str) -> bool` — parses defensively and returns
`False` for `""`, a wrong field count, a non-numeric iteration count, an unknown algo
name, odd-length hex, or a length mismatch. **It must never raise**: a corrupt `meta`
row has to read as "wrong password", not a 500 on the login page. Compare with
`hmac.compare_digest`.
- `needs_rehash(encoded: str, *, iterations: int) -> bool` — true when the stored cost (or
algo) is below the configured one, so a login can transparently upgrade the hash.
- No password, hash, salt or session token may ever appear in a log line, an exception
message, or a return value used for display. Name this in the docstring: AGENTS.md
invariant 6 now covers the admin's own credential.
3. Sessions — stateless and signed, so there is no session table and no cleanup job:
- `new_secret() -> str` → `secrets.token_hex(32)`. Stored by task 02 under the meta key
`auth.session_secret`; rotating it invalidates every issued cookie, which is how changing
the password signs everyone else out.
- `@dataclass(frozen=True) Session: username: str; expires_at: float; token: str`
- `mint_session(secret: str, *, username: str = ADMIN_USER, hours: float = 720.0,
now: float | None = None) -> str` → `b64url(f"{username}|{expires_at:.0f}")` + `"."` +
`b64url(hmac_sha256(secret, payload))`. Use `base64.urlsafe_b64encode` and strip padding;
`hmac.new(secret.encode(), payload.encode(), hashlib.sha256)`.
- `read_session(secret: str, token: str, *, now: float | None = None) -> Session | None` —
signature first (constant-time), then `expires_at > now`. Returns `None` for: `""`,
no dot, non-base64 payload, payload without `|`, non-numeric expiry, an expired session,
a token signed by a rotated secret, and a payload whose *username* was tampered with.
- `mint_csrf(secret: str, session_token: str) -> str` →
`hmac_sha256(secret, "csrf|" + session_token).hexdigest()` — bound to the session, so a
token stolen from one session cannot authorise another's write, and it needs no expiry
of its own because the session has one.
- `verify_csrf(secret: str, session_token: str, provided: str | None) -> bool` —
`compare_digest`, `False` on `None`/empty.
4. Login throttle — in-process memory, because there is one admin and one uvicorn process:
- `class LoginThrottle: def __init__(self, *, max_attempts: int = 5, base_seconds: float =
60.0, max_seconds: float = 3600.0)`
- `retry_after(key: str, *, now: float | None = None) -> float` → seconds still blocked,
`0.0` when allowed.
- `failure(key: str, *, now: float | None = None) -> float` → the backoff just applied.
Exponential from `base_seconds` per key, capped at `max_seconds`; a key that gets a
successful login is cleared.
- `success(key: str) -> None`, and `prune(now)` (or an inline length guard) so a spoofed
swarm of source addresses cannot grow the dict without bound.
- Take `now` as an injectable seconds value on every method (`time.monotonic` at the call
site in task 03) — a test must not sleep 60 s to prove the backoff works.
- ASSUMPTION: the key is the socket peer address. Behind nginx every login shares one
bucket, so a determined attacker can lock the operator out of the dashboard (not of the
UPS, not of the data) for at most `max_seconds`. Accepted for a single-admin box; it goes
in `AGENTS.md` → Known gaps rather than becoming an `X-Forwarded-For` knob that is
trivially spoofed when someone forgets to trust the proxy.
5. Origin / Sec-Fetch gate as pure string functions (unit-testable without a `Request`):
- `origin_matches(origin: str, own_origin: str, allowed: Sequence[str] = ()) -> bool`
- `sec_fetch_site_allowed(value: str | None) -> bool` — allow `None`, `"same-origin"`,
`"none"`; reject `"cross-site"`, `"cross-origin"`, `"same-site"`(unknown → reject).
Anything unrecognised is rejected: a browser that invents a new value is a browser doing
something we did not reason about.
## Testing & Quality
- `tests/test_auth.py` (new) — the exhaustive negative table above. Minimum:
round-trip hash/verify, wrong password, unicode password (`"pässwörd"` — proves the utf-8
encode), five malformed encodings all `False` with no exception, `needs_rehash` on a
1_000-iteration hash against the default cost, session round-trip, tampered username,
flipped signature byte, truncated token, expired session (via `now`), cookie from a rotated
secret, CSRF ok/wrong-session/missing, throttle allows 5, blocks the 6th with
`retry_after > 0`, backoff doubles and caps at `max_seconds`, `success()` clears,
`origin_matches` accepts its own origin and the allow-list and rejects
`http://evil.example`, `sec_fetch_site_allowed("cross-site") is False`.
- Cover `auth.py` to ~100% — every branch here is a security decision, and a branch no test
reaches is a decision nobody checked.
- Coverage: **>90%** on this task's new code; overall gate (94%) maintained.
## Completion Criteria
- [ ] `uv run pytest tests/test_auth.py -v` green
- [ ] `python -c "from dashboards import auth; h=auth.hash_password('testpass'); print(h)"`
prints a `pbkdf2_sha256$600000$…` string and `auth.verify_password('testpass', h)` is
`True` while `verify_password('wrong', h)` and `verify_password('testpass', 'junk')` are
`False`
- [ ] `grep -n "print(\|logger\|logging" src/dashboards/auth.py` shows nothing that could
echo a credential
- [ ] no new dependency in `pyproject.toml` / `uv.lock` (`uv pip list --outdated` stays empty)
- [ ] `uv run pytest` green; `uv run ruff check src tests agent && uv run pyright` clean
@@ -0,0 +1,119 @@
# Task 02 — Where the password lives: `meta`, the four settings, and failing closed
**Phase:** `11_admin_login` · **Source:** operator request 2026-10-17 — "configured initially with an env var (but should not be required to change)"
**Story:** n/a
## Objective
Make the credential durable and the startup honest: the verifier and the signing secret live in
the SQLite `meta` table, the environment seeds them once, and a server with neither a stored
verifier nor `DASH_ADMIN_PASSWORD` **refuses to start** instead of serving the fleet open.
## Work
1. `src/dashboards/store.py` — three small public helpers over the `meta` table that already
exists (`CREATE TABLE IF NOT EXISTS meta (key TEXT PRIMARY KEY, value TEXT)`), each under
`self._lock`:
- `get_meta(key: str) -> str | None`
- `set_meta(key: str, value: str) -> None` (`INSERT OR REPLACE`)
- `delete_meta(key: str) -> int` (returns rowcount; the reset procedure needs it and a
test should prove it removes only the named key)
Route `stats()`'s `last_maintain` read through `get_meta()` so there is one path to the
table. Leave `maintain()`'s write inline — it is inside that method's transaction.
2. `src/dashboards/config.py` — four new `Settings` fields beside the server block, using the
existing `_env` / `_num` / `_flag` / `_int` helpers (field defaults are read once at import;
that is the established pattern, see the module docstring):
- `admin_password: str = _env("DASH_ADMIN_PASSWORD")` — the **seed**. `_env()` strips, so
`DASH_ADMIN_PASSWORD=` means *unset*, never "the password is empty".
- `session_hours: float = _num("DASH_SESSION_HOURS", 720)` — 30 days; a homelab dashboard
that logs the operator out every shift change gets bypassed, which is worse.
- `cookie_secure: bool = _flag("DASH_COOKIE_SECURE")` — off by default because dev is
`http://127.0.0.1`; task 03 also marks the cookie `Secure` when the request arrived over
`X-Forwarded-Proto: https`.
- `pbkdf2_iterations: int = _int("DASH_PBKDF2_ITERATIONS", PBKDF2_ITERATIONS)` — import the
constant from `.auth` (`auth.py` imports nothing from this package, so there is no cycle).
The knob exists so the hermetic suite can hash in microseconds instead of 0.3 s; the
comment must say that, and must not suggest lowering it in production.
3. `src/dashboards/auth.py` — the loader that turns settings + `meta` into request-time state.
Declare a `MetaStore` `Protocol` (`get_meta` / `set_meta` / `delete_meta`) so this module
still imports nothing from `config`/`store`:
```python
@dataclass(frozen=True)
class AuthState:
verifier: str # encoded PBKDF2, never the password
secret: str # session-signing key
session_hours: float
iterations: int
seeded: bool # True when this start wrote the verifier row
env_ignored: bool # True when a stored verifier beat DASH_ADMIN_PASSWORD
```
- `load_auth_state(store: MetaStore, *, admin_password: str, iterations: int =
PBKDF2_ITERATIONS, session_hours: float = 720.0, now: float | None = None) -> AuthState`
- `verifier = store.get_meta("auth.verifier")`.
- If absent and `admin_password` → `hash_password(...)`, `set_meta("auth.verifier", …)`,
`seeded=True`.
- If absent and no `admin_password` → raise `RuntimeError` with the exact operator text:
name `DASH_ADMIN_PASSWORD`, name the meta key, and give the reset command
(`sqlite3 <db> "DELETE FROM meta WHERE key IN ('auth.verifier','auth.session_secret')"`).
Fail closed here — not "start open with a warning": the domain is public and the fleet's
hostnames and UPS controls are behind it.
- `secret = store.get_meta("auth.session_secret")` or generate with `new_secret()` and
persist it. Persisting it is what makes a restart not invalidate every browser.
- `env_ignored = bool(admin_password) and not seeded`. When true,
`logging.getLogger("dashboards.auth").warning()` **once**, at startup, with the words:
`DASH_ADMIN_PASSWORD is ignored — the password set in /settings wins; delete the
'auth.verifier' row to reset from the environment`. Never log the value, the hash, or
any prefix of either.
- `update_verifier(store: MetaStore, state: AuthState, new_password: str) -> AuthState` —
hashes, writes both `auth.verifier` and a **fresh** `auth.session_secret`, returns the new
state (task 04 uses it; the rotation is what signs out every other browser). Empty /
whitespace-only `new_password` raises `ValueError` — an empty password must never be
storable.
4. `src/dashboards/app.py` — build the state during lifespan startup, before `start()`, for
both the owned-store and injected-store branches, and hang it off the context:
`ctx.auth = load_auth_state(store, admin_password=settings.admin_password,
iterations=settings.pbkdf2_iterations, session_hours=settings.session_hours)`.
`AppContext` (in `api.py`) gains `auth: Any = None` (typed as `AuthState | None`; import the
dataclass — that is how it stays honest under pyright). Task 03 is what *enforces* it; this
task only guarantees it exists or the process does not come up.
Startup failure must be loud and non-zero: a `RuntimeError` raised in the lifespan does that
for `uvicorn dashboards.app:app` and `uv run dashboards` alike. Do **not** swallow it.
5. Tests:
- `tests/test_store.py` — `get_meta` `None` for a missing key, round-trip, `set_meta`
replaces, `delete_meta` returns 1 for an existing key / 0 for a missing one and leaves the
other keys alone.
- `tests/test_config.py` — the four new fields pick up their env vars and their defaults;
`DASH_ADMIN_PASSWORD=` is `""`.
- `tests/test_auth.py` (or a new `tests/test_auth_bootstrap.py` if 01's file is getting
long) — seeds on first call and writes both rows; second call with `admin_password=""`
reuses the stored verifier (so a deployment can drop the env var after first boot);
`admin_password=""` **and** no row raises `RuntimeError` naming both the variable and the
key; a stored verifier makes a conflicting env password `env_ignored=True` and does **not**
overwrite the row; a corrupt `auth.session_secret`… (state the behaviour you choose:
regenerate and persist is fine, since the effect is "everyone signs in again", which is a
login prompt, not a lockout); `update_verifier` changes both rows and the new state's
secret, and refuses an empty password.
- `tests/test_api.py` — one test that `create_app(Settings(db_path=db), store)` (no
password anywhere) fails to start, and that with `admin_password="testpass"` it starts.
## Testing & Quality
- Coverage **>90%** on `auth.py`, `config.py`, `store.py`, `app.py` changes; overall gate
(94%) maintained.
- Prove the fail-closed branch and the seeded branch each run — a security default that no test
exercises is a rumour.
## Completion Criteria
- [ ] `uv run pytest tests/test_store.py tests/test_config.py tests/test_auth.py -v` green
- [ ] starting the real dev server with `DASH_ADMIN_PASSWORD` unset and a scratch
`DASH_DB=/tmp/auth-probe.sqlite3` exits non-zero and prints the message naming the
variable and the reset command (paste the line into the commit message)
- [ ] the same command with `DASH_ADMIN_PASSWORD=testpass` starts, and
`sqlite3 /tmp/auth-probe.sqlite3 "SELECT key FROM meta"` lists `auth.verifier` and
`auth.session_secret`
- [ ] the stored hash in `meta` is a `pbkdf2_sha256$…` string; `grep -c testpass` in that DB
is 0
- [ ] `uv run pytest` green; `uv run ruff check src tests agent && uv run pyright` clean
@@ -0,0 +1,163 @@
# Task 03 — The gate: default-deny middleware, `/login` route, `/api/auth/*`
**Phase:** `11_admin_login` · **Source:** operator request 2026-10-17 — "password protect this dashboard"
**Story:** n/a
## Objective
Put every route behind the session except four deliberately public ones, with the login,
logout and status endpoints that create and end it, and make the browser scripts carry the
credentials so no page has to think about auth.
## Work
1. `src/dashboards/app.py` — state the public surface **next to the route table** it guards:
```python
# Everything else requires a signed-in admin. This list is the whole attack surface, so
# adding a path here is a security decision, not a convenience: /api/health is here
# because the Containerfile HEALTHCHECK and ./scripts/dev-server.sh status poll it
# unauthenticated, and it is the only data route that stays public.
PUBLIC_PATHS = frozenset({"/login", "/api/auth/login", "/api/health"})
PUBLIC_PREFIXES = ("/static/",)
```
Add `"/login": "login.html"` to `PAGES`. Task 05 supplies the file; until then serve the
existing shape it will get (a minimal page is fine to create here and flesh out in 05 — do
**not** leave `/login` 404, or the gate redirects to a dead end).
2. Add the gate as an `@app.middleware("http")` in `create_app()`, beside the existing
`no_store_for_api`:
- `ctx = getattr(request.app.state, "ctx", None)`; no `ctx` → `503 {"ok": false, "detail":
"startup did not complete"}` for non-public paths (the app that failed to build its
context must not serve pages with `ctx.auth = None`).
- public path → straight through.
- session: `request.cookies.get(auth.SESSION_COOKIE)`, or `Authorization: Bearer <token>`
(so `curl -H "Authorization: Bearer …"` can script the API — document it in README).
Read with `auth.read_session(ctx.auth.secret, token)`.
- no/invalid session →
- `GET`/`HEAD` with `text/html` in `Accept` and a path not starting with `/api` →
`303` to `/login?next=<safe next>`;
- everything else → `401 {"ok": false, "detail": "sign in required", "login": "/login"}`.
- `safe_next(raw)` — keep it only if it starts with `/`, does not start with `//`, and
contains no `://`; else drop it. That is the open-redirect check; a `?next=` that sends
the operator to someone else's site would be a phishing kit with a homelab theme.
- `GET`/`HEAD` with a valid session → set `request.state.session = session` and continue.
- non-`GET`/`HEAD` (everything that can write, `POST` included) additionally:
- origin check with `auth.origin_matches(origin, own_origin, allowed)` where `own_origin`
is `f"{request.url.scheme}://{request.url.netloc}"` and `allowed` comes from
`DASH_ALLOWED_ORIGINS` — **no such setting is added**: same-origin is the only policy
this app needs, so pass `()`. Reject when `Origin` is present and mismatched, or when
`Sec-Fetch-Site` is present and `auth.sec_fetch_site_allowed()` rejects it;
- CSRF: required when the caller authenticated with the **cookie** (a bearer caller
cannot be forged by another site — that is what the header buys), validated with
`auth.verify_csrf(ctx.auth.secret, session.token, request.headers.get(auth.CSRF_HEADER))`;
- failures → `403 {"ok": false, "detail": "cross-origin request refused"}` /
`403 {"ok": false, "detail": "missing or invalid CSRF token"}`.
- **every response the gate produces carries `NO_STORE`** (the gate may short-circuit
before `no_store_for_api` runs, depending on middleware order — do not depend on the
order, pin the header in a test).
3. `POST /api/auth/login` (public; put it in a new `auth_router()` in
`src/dashboards/api.py`, included in `app.py` alongside the other routers, prefix `/api/auth`):
- body `{"password": "<string>"}` — there is no username field (single admin;
`auth.ADMIN_USER`). An absent/empty password is `422`, not a hash comparison.
- throttle first: `key = request.client.host if request.client else "unknown"`;
`retry_after = ctx.login_throttle.retry_after(key)`; blocked → `429` with
`Retry-After: <ceil>` and `detail` naming the lockout seconds — no "wrong password"
lie, and no hint about which account.
- `auth.verify_password(password, ctx.auth.verifier)`:
- no → `throttle.failure(key)`, log an event
(`Event(now, "auth", "admin", "login-denied", "warn", "sign-in refused (attempt N)")`)
and `401 {"ok": false, "detail": "wrong password"}` — one message for every failure
kind, because there is one account and the response must not rank failures.
`alerts.py` only turns `kind == "unreachable"` events into overview rows and the page
feeds filter by `source`, so an `auth` event cannot blink the overview — say so in the
code comment so nobody "fixes" it by hiding the event.
- yes → `throttle.success(key)`, `auth.mint_session(...)`, set the cookie
(`httponly=True`, `samesite="lax"`, `path="/"`, `max_age=int(hours*3600)`,
`secure=True` when `settings.cookie_secure` **or** the request's first
`X-Forwarded-Proto` value is `https` — marking it `Secure` for an https visit is safe
even if the header lies, because it only ever tightens), and answer
`{"ok": true, "username": "admin", "expires_at": ts, "csrf": <token>, "next": "/safe"}`.
Never echo the password, the verifier, or the session token in any other field.
- `GET /login` while already holding a valid session → `303` to `/` (or to a safe `next`).
4. Protected auth routes on the same router:
- `GET /api/auth/status` → `{"ok": true, "authenticated": true, "username": "admin",
"expires_at": ts, "seconds_left": n, "csrf": auth.mint_csrf(secret, session.token),
"session_hours": …}`. This is the only place the CSRF token is published; it is inside
the session for a reason.
- `POST /api/auth/logout` (session + CSRF required — a plain link would be a CSRF that
signs the operator out) → clears the cookie (`delete_cookie(path="/")`) and answers
`{"ok": true}`.
5. `src/dashboards/api.py` — `AppContext` gains `login_throttle: Any = None` next to the
`auth` field from task 02; `app.py` constructs one `auth.LoginThrottle()` per app (per
process, which is the documented model). Do not create it per request: a new throttle per
request is a throttle that never trips.
6. `src/dashboards/web/static/app.js` — one place learns the token, every request uses it:
- module-level `authState = null` + `async function ensureAuth()` that fetches
`/api/auth/status` once (memoise the promise, not just the result, or 4 page scripts race
four status calls), caching `{csrf}`;
- `fetchJSON(url, options)` attaches `X-CSRF-Token` for every method outside
`GET`/`HEAD`, awaiting `ensureAuth()` first;
- on `401`, `window.location.assign('/login?next=' + encodeURIComponent(location.pathname))`
and throw once — guard with a module flag so a 2 s poll loop whose session expired does
not spam the login page with redirects;
- keep the exported surface additive (`app.js`'s `return { … }` list): export `authStatus`
(the cached object) because task 06's `/settings` page prints the session expiry.
7. Tests — `tests/test_auth_api.py` (new) plus the plumbing this task forces on the suite:
- `tests/auth_client.py` (new helper module, imported the way `tests/card_contract.py`
is): `ADMIN_PASSWORD = "testpass"`, `sign_in(client, password=ADMIN_PASSWORD)` that posts
the login, asserts 200, and stashes the token on the client
(`client.headers[auth.CSRF_HEADER] = body["csrf"]`). Setting it on the TestClient is what
keeps the 12 existing `client.post(` assertions in `tests/test_api.py` unedited — they get
cookies *and* the CSRF header for free.
- `tests/test_api.py::build_client` logs in through that helper. The seven other
`create_app(` sites in `tests/test_api.py` (search `grep -n "create_app(" tests/`; there
are also two in `tests/test_system_contract.py` and `tests/test_ups_contract.py`) each get
`admin_password=ADMIN_PASSWORD` **and** a `sign_in()`, or their assertions read 401s.
- gate behaviour: every `/api/*` data route (at least `/api/config`, `/api/summary`,
`/api/events`, `/api/metrics`, `/api/system/snapshot`, `/api/system/history`,
`/api/llm/snapshot`, `/api/ups/snapshot`, `/api/ups/commands`) answers `401` with that
exact body shape anonymously; `GET /`, `/system`, `/llm`, `/ups` with `Accept:
text/html` → `303` with `Location` starting `/login?next=`; `/api/health`, `/login`,
`/static/dash.css` stay reachable anonymously; every gate response has
`cache-control: no-store`.
- `test_every_route_that_is_not_public_requires_a_session` — build the app, walk
`app.routes`, derive each path (`getattr(r, "path", "")`; a `Mount` counts as its prefix),
and assert: anything outside `PUBLIC_PATHS`/`PUBLIC_PREFIXES` is `401`/`303` for anonymous
`GET` **and** `POST`. This is the test that protects the next phase that adds a route.
- `?next=` handling: `/nope` passes through, `//evil.example` and
`http://evil.example` are dropped to `/` (assert the `Location` has no `evil` in it).
- login: wrong password 401 `wrong password`; empty 422; success sets a `Set-Cookie`
containing `HttpOnly`, `SameSite=lax` and `Max-Age`, and returns a usable `csrf`;
`X-Forwarded-Proto: https` adds `Secure`; `DASH_COOKIE_SECURE=1` adds it without the
header; 6th bad attempt from the same client → 429 with `Retry-After` and the event row
exists in `/api/events?source=auth` once signed in.
- CSRF: `POST /api/ups/command` with a valid session and **no** `X-CSRF-Token` → 403, and
prove it never reached the wire (the scripted `ScriptedNutClient` in `tests/test_api.py`
records nothing); with the header it behaves exactly as it did before this phase
(`DANGER` still required for `driver.killpower`); `Origin: http://evil.example` → 403 on
login and on a write; `Sec-Fetch-Site: cross-site` → 403; a bearer-token caller can write
without the CSRF header but not without the token.
- logout kills the cookie (subsequent `/api/summary` → 401).
- the served-pages tests in `tests/test_api.py` that scrape page scripts still pass — the
orphan-`id` test and the two-panel/hero pins must stay green without weakening them.
## Testing & Quality
- Coverage **>90%** on `app.py`, `api.py`, `auth.py` changes; every branch of the gate
(public, sessionless HTML, sessionless JSON, bad-session write, CSRF-missing,
origin-mismatch, no-ctx) needs a test — this module is the entire perimeter.
- `node --check src/dashboards/web/static/app.js` is part of `.agents/validate.sh`; run it.
## Completion Criteria
- [ ] `uv run pytest tests/test_auth_api.py -v` green, including the route-enumeration test
- [ ] `curl -si http://127.0.0.1:8090/system | head -5` shows `303` + `Location: /login?next=/system`
- [ ] `curl -s http://127.0.0.1:8090/api/summary` → the 401 JSON; `curl -s
http://127.0.0.1:8090/api/health` → 200; `curl -s -o /dev/null -w '%{http_code}'
http://127.0.0.1:8090/static/dash.css` → 200
- [ ] `curl -s -X POST -H 'content-type: application/json' -d '{"password":"testpass"}'
http://127.0.0.1:8090/api/auth/login` returns `ok: true` with a `csrf`, and the returned
token used as `X-CSRF-Token` + the cookie makes `POST /api/ups/command` behave as before
- [ ] `uv run pytest` fully green (all ~550 tests, not just the auth ones); `ruff` + `pyright`
clean; `bash .agents/validate.sh` green
@@ -0,0 +1,87 @@
# Task 04 — Change the password from the API, and mean it for every session
**Phase:** `11_admin_login` · **Source:** operator request 2026-10-17 — "the password should be changeable once logged in"
**Story:** n/a
## Objective
`POST /api/auth/password`: prove the current password, replace the verifier, rotate the signing
secret so the change takes effect everywhere at once. Optional in every sense — nothing forces
a change, no expiry, no first-login interstitial.
## Work
1. `src/dashboards/api.py` — add to `auth_router()`:
`POST /api/auth/password`, body `{"current": str, "new": str}` (session **and**
`X-CSRF-Token` required by task 03's gate):
- Validate `new` before touching anything: non-empty after `.strip()`, at least
`MIN_PASSWORD_LENGTH = 8`, and different from `current` → otherwise `422` with a `detail`
that says which. No composition rules (symbols/digits requirements produce sticky notes,
not passwords); length is the part that does work.
- Throttle the *current* password guesses on a **separate key** —
`f"password:{request.client.host}"` on the same `LoginThrottle` instance — so the
change-password form is not an unthrottled oracle for the old password once a cookie is
stolen. Blocked → `429` with `Retry-After`, same as login.
- `auth.verify_password(current, ctx.auth.verifier)`; wrong → `throttle.failure(that key)`
and `403 {"ok": false, "detail": "current password is wrong"}`. Nothing else changes and
an event is logged (`source="auth"`, `kind="password-denied"`, `severity="warn"`).
- Right → `auth.update_verifier(store, ctx.auth, new)` (task 02: writes `auth.verifier` and
a fresh `auth.session_secret`), assign `ctx.auth = new_state` (`AppContext` is a plain
mutable dataclass), `throttle.success()`, log
`Event(now, "auth", "admin", "password-changed", "warn", "admin password changed")`, and
respond with a **fresh cookie minted from the new secret** plus
`{"ok": true, "changed": true, "expires_at": ts, "csrf": <new token>,
"hint": "other browsers are now signed out"}`.
Cookie attributes identical to login (including the `X-Forwarded-Proto` /
`DASH_COOKIE_SECURE` rule).
- Rotating the secret is the whole point: sessions are stateless, so without it a stolen
cookie would outlive the password that "revoked" it. Say that in a comment — the next
reader will wonder why a password change touches the signing key.
2. Also in `auth_router()`: make login upgrade a cheap hash transparently. After a successful
`verify_password`, if `auth.needs_rehash(ctx.auth.verifier,
iterations=settings.pbkdf2_iterations)` → re-hash with the configured cost and write it
(keep the secret — this is not a logout). Without this, an operator who raises
`DASH_PBKDF2_ITERATIONS` never actually gets the new cost.
3. Order-of-operations guard: the response must be built from `new_state`, never from the old
`ctx.auth`. A response signed with the old secret is a browser that dies on its next
request — the exact bug this task can introduce.
4. Tests (extend `tests/test_auth_api.py`):
- wrong `current` → 403, and the `auth.verifier` string in the DB is byte-identical to
before; five wrong tries then a **correct** one is still `429` (proves the separate
throttle key actually throttles);
- `new` empty / whitespace / 7 chars / equal to `current` → 422 and both `meta` rows
unchanged;
- success → `auth.verifier` and `auth.session_secret` both changed; `verify_password`
confirms new-yes/old-no against the stored string; the **old** cookie now gets `401` from
`/api/summary`; the cookie in the response works and its returned `csrf` authorises
`POST /api/ups/command`; a second browser's cookie (captured earlier in the test) gets
`401`;
- restart semantics: after a change, rebuild the app from the same DB with
`admin_password="testpass"` in settings → login with the **new** password works and
`testpass` does not (the env var is a seed, not an override);
- rehash: seed a verifier at `iterations=1000`, start with `pbkdf2_iterations=2000`, log in
→ 200 and the stored string now says `pbkdf2_sha256$2000$…`, and the same password still
works afterwards;
- the `auth` events exist and are visible via `/api/events?source=auth`, and `alerts.build`
does **not** turn them into overview alerts (assert the summary's `alerts` list is
unchanged by them — invariant: a login failure is a fact for the feed, not a red panel).
5. `/api/config` and `/api/health` must not gain any auth detail: `/api/health` is public, and
"the admin password was changed at T" is not something to serve to an anonymous probe. The
fact lives in the event feed (authenticated) and in `/settings`.
## Testing & Quality
- Coverage **>90%** on the new route and `auth.update_verifier`; every rejection path above is
a test, not a comment.
- The suite's existing `/api/ups/command` tests (the `ScriptedNutClient` ones) must stay green
unchanged in behaviour.
## Completion Criteria
- [ ] `uv run pytest tests/test_auth_api.py -v` green for the eight scenarios above
- [ ] by hand on the dev server: sign in with `testpass`, change it to something else with
`curl`, and confirm the old cookie now gets `401` while the returned one works
- [ ] `sqlite3 data/dashboards.sqlite3 "SELECT value FROM meta WHERE key='auth.verifier'"`
still prints only an encoded PBKDF2 string — no plaintext, anywhere
- [ ] `uv run pytest` green; `ruff` + `pyright` clean; `bash .agents/validate.sh` green
@@ -0,0 +1,83 @@
# Task 05 — The login page
**Phase:** `11_admin_login` · **Source:** operator request 2026-10-17 — "password protect this dashboard"
**Story:** n/a
## Objective
A single-purpose page at `/login` that asks for the admin password, says honestly why it
refused, and lands the operator on `/` (or the page they were heading to). Hand-rolled HTML +
one small script, no CDN, dark/light parity with the dashboards.
## Work
1. `src/dashboards/web/pages/login.html` — modelled on the other four pages' `<head>`
(`/static/dash.css`, the same inline SVG favicon, `data-theme="dark"` so `app.js` can flip
it from `localStorage` before paint):
- title `Homelab · sign in`; the `.masthead` brand link, but **no** `.tabs` nav — the tabs
lead to pages that would bounce straight back here, and a nav of links that all redirect
is a lie about what is available.
- one `.card` in `main`: `<h1>Sign in</h1>`, a `.sub` line naming what this protects ("the
fleet's system, engine and battery dashboards — including the UPS controls"), then a form:
`<label for="admin-password">Admin password</label>` +
`<input id="admin-password" type="password" autocomplete="current-password" required>` +
`<button class="action" id="sign-in" type="submit">Sign in</button>`.
- a `<p id="login-error" class="login-error" role="alert" hidden>` for refusals and
`<p id="login-note" class="faint mono" hidden>` for the lockout countdown.
- scripts: `/static/app.js` then `/static/page-login.js`, `defer`.
- **no** username field: one admin, and a field that is always the same word teaches the
operator that half the form is theatre (`auth.ADMIN_USER` is the server's business).
2. `src/dashboards/web/static/page-login.js` — IIFE in the house style (see
`page-index.js`), no framework:
- `submit` handler → `POST /api/auth/login` with
`Dash.fetchJSON` (it adds the JSON accept header and turns a non-2xx into a thrown
`Error` with `status`/`body`); body `{password}`;
- on success `location.assign(body.next || '/')`;
- on `401` show `Wrong password.` and select the field's contents (fixing a typo without
retyping is the whole UX of a login form);
- on `429` show the `detail` plus the `Retry-After` seconds ticking down, and disable the
button until it reaches 0 — a submit that will be refused should not look pressable;
- on any other failure print the server's `detail` verbatim through the escape path the app
already uses for text (`textContent`, never `innerHTML`);
- honour `?next=` **only** if it starts with `/` and not `//` (the same rule the server
enforces; a client-side copy is defence in depth, and the server's check is the one that
counts);
- if the browser already holds a valid session, `GET /api/auth/status` succeeds → redirect
to `/` immediately (task 03 also does this server-side; this is what stops a flash of the
form).
3. `src/dashboards/web/static/dash.css`:
- extend the existing `input[type="text"]` rule (line ~642) to
`input[type="text"], input[type="password"]` — do not duplicate it;
- add `.login-card { max-width: 26rem; margin: 3rem auto; }` and `.login-error { color:
var(--fail); }`, `.login-card form { display: grid; gap: .6rem; }`, plus
`.login-card label { font-size: 10.5px; text-transform: uppercase; letter-spacing: .06em;
color: var(--text-faint); }` (the same caption vocabulary as `.hero .label`);
- only existing `:root` tokens (`--fail`, `--text-faint`, `--accent`, …) —
`tests/test_api.py` fails if a rule reaches for a token `:root` does not declare, and if a
`.sev-*` family appears.
4. Keep the invariant: nothing external. `tests/test_api.py` already asserts the pages load no
foreign asset — extend that assertion to `/login` (same test, one more path).
## Testing & Quality
- `tests/test_api.py` (the served-HTML section) — `/login` is served anonymously with
`HTTP/1.1 200`, contains `id="admin-password"`, `type="password"`,
`autocomplete="current-password"`, `role="alert"`, and **no** `/system`-style `.tabs`; no
`http://` or `https://` asset reference in the served bytes.
- Orphan-`id` pin: every new `id` (`admin-password`, `sign-in`, `login-error`, `login-note`)
must be referenced by `page-login.js` (or the CSS) — that test fails on a dead `id`, so name
what you use and use what you name.
- `node --check src/dashboards/web/static/page-login.js` (part of `validate.sh`).
- Coverage: no new Python; the served-bytes assertions are the tests. Do not add a JS test
runner (a locked anchor).
## Completion Criteria
- [ ] anonymous `GET /login` → 200 with the form; anonymous `GET /` → 303 to `/login?next=/`
- [ ] in the dev server: wrong password shows `Wrong password.` inline with no page reload;
`testpass` lands on `/` and the header status dots go green
- [ ] five wrong passwords then a correct one shows the lockout with a counting-down
`Retry-After` and a disabled button; it clears and the next attempt works
- [ ] `/login?next=/ups` after a successful sign-in lands on `/ups`
- [ ] dark and light both readable (screenshot both, or one plus a note in `NOTES-*.md`)
- [ ] `uv run pytest` green; `ruff` + `pyright` clean; `bash .agents/validate.sh` green
@@ -0,0 +1,89 @@
# Task 06 — `/settings`: change the password, sign out, see when the session ends
**Phase:** `11_admin_login` · **Source:** operator request 2026-10-17 — "the password should be changeable once logged in"
**Story:** n/a
## Objective
A fifth page the operator reaches from the masthead: the session facts, a working
change-password form, and a sign-out. It has to survive the secret rotation from task 04
without leaving the browser holding a stale CSRF token.
## Work
1. `src/dashboards/app.py` — add `"/settings": "settings.html"` to `PAGES`. It lands behind the
gate with no extra code (task 03's route-enumeration test proves it); do not add it to the
public list.
2. `src/dashboards/web/pages/settings.html` — same `<head>`/masthead as the other pages, `.tabs`
nav **without** `aria-current` (Settings is not one of the four dashboards), and in `main`:
- **Session** panel: `<dl>` with the signed-in name, the expiry as a local timestamp *and*
"in 29d 14h" via `Dash.duration(seconds_left)`, and `<button class="action"
id="sign-out" type="button">Sign out</button>`.
- **Change password** panel: three fields — current (`autocomplete="current-password"`),
new (`autocomplete="new-password"`, `id="new-password"`), confirm
(`autocomplete="new-password"`, `id="confirm-password"`) — each with a real `<label
for=…>` and an `aria-describedby` hint naming the 8-character floor, a `<button
class="action" id="change-password">Change password</button>`, and one
`<p id="settings-result" role="status" aria-live="polite">` for the outcome.
- A static `.sub` paragraph stating the precedence rule in the app's own voice: the password
was seeded from `DASH_ADMIN_PASSWORD` at first start, changing it here takes precedence,
the env var never overwrites a password set here, and resetting from the environment means
deleting the `auth.verifier` row and restarting. (This is the sentence that stops the
"I edited the env file and nothing happened" support call.)
- No `#nav-status` — `Dash.initChrome()` already guards its absence.
3. `src/dashboards/web/static/app.js`:
- `resetAuth()` — clears the memoised `authState`/promise added in task 03, exported in the
`return { … }` list. Needed twice: after a password change (the secret rotated, so the
cached `csrf` is now invalid) and after logout. Forgetting this is the bug where the
operator changes the password and then every UPS button 403s until they reload.
- also call it when a `401` bounces to `/login` (the flag from task 03 already stops the
redirect storm).
4. `src/dashboards/web/static/page-settings.js`:
- on load `Dash.fetchJSON('/api/auth/status')` → fill the session panel;
- sign-out → `POST /api/auth/logout` → `Dash.resetAuth()` → `location.assign('/login')`;
- change password → local checks first (new ≥ 8 chars, `new === confirm`, current non-empty),
reported inline without a round-trip; then
`POST /api/auth/password` with `{current, new}`; on success write
"Password changed. Other browsers are signed out." into `#settings-result`,
`Dash.resetAuth()` (so the next write re-reads the fresh `csrf`), clear all three fields
(`form.reset()` plus explicit blanking — password managers autofill them), and keep the
session cookie the response set; on `403` print "current password is wrong", on `422`
print the server's `detail`, on `429` the lockout text — every message via `textContent`;
- after a successful change, verify once by calling `/api/auth/status` again so the panel
cannot show a session the server no longer honours.
5. Masthead on all four dashboard pages (`index.html`, `system.html`, `llm.html`, `ups.html`) —
one link in `.masthead-right`, before the theme toggle:
`<a class="ghost account" href="/settings" title="Session and password">⚙ Account</a>`.
**No `id`** (the orphan-`id` test in `tests/test_api.py` flags ids nothing references — an
`<a href>` needs none), and it must sit outside `.tabs` so the pinned tab structure of the
four dashboards is untouched. Add `.ghost` styling for `a` alongside the existing
`button.ghost` rule (extend its selector to `a.ghost` rather than duplicating; existing
tokens only).
6. `src/dashboards/web/static/page-login.js` — a sign-in that carries `?next=/settings` is a
normal path once the session dies on the settings page; task 05's `next` handling covers it,
just verify it in the browser.
## Testing & Quality
- `tests/test_api.py` (served-bytes section, signed-in client) —
`GET /settings` is `200` and contains `id="new-password"`, `autocomplete="new-password"`,
`id="sign-out"`, `role="status"`; anonymous `GET /settings` is `401`/`303` (already covered by
the enumeration test — do not duplicate it there, but assert it once here for the page);
`test_every_dashboard_page_links_to_settings` over the four page paths; `/settings` carries no
external asset; the orphan-`id` test still passes with the four new ids.
- `node --check` for `page-settings.js` and `app.js` (in `validate.sh`).
- No new Python except the one `PAGES` line, so coverage moves only through `app.py`'s existing
route tests; keep the overall gate (94%) where it is.
## Completion Criteria
- [ ] in a browser: the `⚙ Account` link appears on all four dashboards and opens `/settings`
- [ ] signing out from `/settings` returns to `/login`, and `/system` afterwards bounces to
`/login` rather than rendering an empty page of `n/a`s
- [ ] changing the password on `/settings` shows the success line, keeps the operator signed
in, and a UPS button on `/ups` (e.g. beeper disable → confirm) still works **without a
page reload** — that is `resetAuth()` doing its job
- [ ] the same change in a second browser tab signed in earlier shows the 401 → `/login` path
- [ ] `/settings` renders correctly in dark and light; screenshots
`data/logs/auth-settings.png` and `data/logs/auth-account-link.png`
- [ ] `uv run pytest` green; `ruff` + `pyright` clean; `bash .agents/validate.sh` green
@@ -0,0 +1,125 @@
# Task 07 — Dev config, the two docs, and proof in a browser against live data
**Phase:** `11_admin_login` · **Source:** operator request 2026-10-17 — "for local testing the default admin password should be `testpass`"
**Story:** n/a
## Objective
Make `testpass` the password a developer types locally without ever putting it in a tracked
template, document the whole thing for the operator, and prove the gate in a real browser on
live fleet data (AGENTS.md invariant 10).
## Work
1. `.env.example` — new `Server` block lines, production-shaped and placeholder-only:
```
# Admin password. Seeds the SQLite verifier on first start; after that the password set
# in /settings wins and this line is ignored (delete the 'auth.verifier' row to reset).
# With neither this variable nor a stored verifier the app REFUSES to start.
DASH_ADMIN_PASSWORD=change-me
DASH_SESSION_HOURS=720
# 1 marks the session cookie Secure. Set it in deployment (TLS/nginx); the app also sets
# it automatically for requests that arrive with X-Forwarded-Proto: https.
DASH_COOKIE_SECURE=0
# Cost of the PBKDF2 hash. Lowering it weakens the one thing standing between the network
# and your fleet; it exists so the test suite can hash fast.
DASH_PBKDF2_ITERATIONS=600000
```
2. `.env.dev.example` — `export DASH_ADMIN_PASSWORD=change-me`, and the local-testing value in
prose **on a comment line**, e.g. `# Local throwaway value: the hermetic suite and every
dev box here use the word the tests sign in with (see tests/auth_client.py).`
- **The trap to avoid:** `.agents/validate.sh` scans every tracked env *template* and fails
the phase if a `*_PASSWORD=` / `*_TOKEN=` / `*_SECRET=` / `*_KEY=` line carries anything
but an allow-listed placeholder (`changeme`, `change-me`, `replace-me*`, `secret`,
`placeholder`, `dummy`, `example`, `x+`, empty). **`testpass` is not on that list**, so
`DASH_ADMIN_PASSWORD=testpass` in a tracked template fails the gate — correctly. Do not
edit `validate.sh` to let it through; the fix is a placeholder in the template and the
real word in the gitignored file. Comment lines are not matched (the pattern is anchored
at `^` and cannot start at `#`), which is why prose is safe and an assignment is not.
3. The developer's own `.env.dev` (gitignored, mode `0600`) — append
`export DASH_ADMIN_PASSWORD=testpass` if no `DASH_ADMIN_PASSWORD` line exists. Append with
`printf '…\n' >> .env.dev` (`tee` truncates in place; the mode is 0600 for a reason) and do
not reorder or reformat the rest of that file — it carries real upsd credentials.
Never print its contents into a log, a commit message, or a screenshot.
4. Optional one-line ergonomics in `scripts/dev-server.sh`'s `start()` — after the success
`echo`, add `echo " sign in with DASH_ADMIN_PASSWORD from .env.dev"`. `start`'s readiness
probe and `status` both use `/api/health`, which stayed public, so **no auth logic belongs in
this script** and no password belongs in it either.
5. `README.md`:
- **Quickstart** — after `cp .env.dev.example .env.dev`, one line: local dev signs in with
the dev file's password; the app refuses to start without one.
- New **Signing in** section (before *Configuration*): one admin; the four public paths and
why `/api/health` is one of them (container `HEALTHCHECK`, `dev-server.sh status`);
password seeded from `DASH_ADMIN_PASSWORD`, changed in `/settings`, never forced;
sessions are HMAC-signed cookies (`DASH_SESSION_HOURS`, default 30 days) invalidated
together by a password change; the reset procedure, verbatim:
`sqlite3 data/dashboards.sqlite3 "DELETE FROM meta WHERE key IN ('auth.verifier','auth.session_secret')"`;
scripting the API with `Authorization: Bearer <session token>`;
deployment note — set the password in the container env file (the deployment repo's quadlet
env template), set `DASH_COOKIE_SECURE=1` behind the TLS/nginx front, and note that nginx
config/HSTS live in the deployment repo like every other service here.
- **Configuration table** — the four new variables with their defaults, in the same
two-column style as the existing rows.
6. `AGENTS.md`:
- a new numbered rule in **Non-negotiable rules** — *"One admin, one password, four public
paths."* Everything else is behind the session; the public list is
`/login`, `/api/auth/login`, `/api/health`, `/static/*`; the verifier is PBKDF2 in the
SQLite `meta` table and the env var is a **seed, never an override**; no stored verifier
and no env var means the process refuses to start; a password change rotates the signing
secret so every browser is out; the login POST and every write are origin/CSRF-checked;
and `/api/health` is public by decision, which means nothing sensitive may be added to it
without re-deciding.
- `auth.py` in the one-line-per-responsibility architecture list (pure primitives: PBKDF2
verifier, signed sessions, CSRF tokens, login throttle — stdlib only, no new dependency).
- Architecture/API prose: the four dashboards and `/settings` now require a session.
- **Known gaps** — three entries: the login throttle keys the socket peer address, so behind
nginx a lockout is global (at worst `max_seconds`; accepted for a single admin);
no second factor and no per-user accounts by design; the throttle is per-process, so
running more than one uvicorn worker multiplies the buckets (and the session/CSRF design
itself is worker-safe because it is stateless).
- `PLAN.md`'s Architectural Anchors table has no auth row. This phase adds no dependency, so
it stays inside the existing anchors — but note in the phase `NOTES-*.md` that the operator
may want to ratify "in-house stdlib auth (PBKDF2 + HMAC cookie)" as an anchor; this skill
and this phase must not edit `PLAN.md`.
7. Live browser verification (dev server on `http://127.0.0.1:8090`, tunnels up, live data):
- `./scripts/dev-server.sh restart`, then confirm `./scripts/dev-server.sh status` still
prints per-source `ok`/`DOWN` **without** credentials;
- screenshots into `data/logs/`: `auth-login.png`, `auth-login-rejected.png`,
`auth-locked-out.png`, `auth-signed-in-overview.png`, `auth-settings.png`,
`auth-account-link.png` (the last two from task 06 — retake if a pixel moved);
- leak checks, recorded in `NOTES-verify.md` with the actual status codes:
`curl -si localhost:8090/system` (303 → `/login?next=/system`),
`curl -s localhost:8090/api/summary` (401), `/api/config` (401), `/api/events` (401),
`/api/system/history?...` (401), `/api/health` (200),
`curl -s localhost:8090/login | grep -c 'http'` (0 external refs);
- once signed in, walk all five pages and confirm the five completed phases look exactly as
they did before: the `/system` hero rows, the `/llm` sparklines and latency feet, the `/ups`
glyph and heroes, the overview's two panels, the `Δ<window>` footnotes;
- `uv run pytest -m live --live` — the live suite talks to the agents, engines and upsd
directly rather than to the dashboard, so it must be unaffected; run it and say so.
8. `NOTES-auth.md` in the phase directory — what was verified how (which screenshot shows
which behaviour), the `curl` outputs, the exact `meta` keys written, and anything the operator
must do in the deployment repo (set `DASH_ADMIN_PASSWORD`, set `DASH_COOKIE_SECURE=1`).
## Testing & Quality
- `bash .agents/validate.sh` green — this is the task where the tracked-template secret gate
bites, so run it before declaring the docs done.
- `git status --porcelain` must show no `.env` / `.env.dev` change staged (both are ignored) and
no screenshot except the `data/logs/` drops the repo already ignores.
- Coverage: no code change expected; the gate (94%) must not move.
## Completion Criteria
- [ ] `.env.example` and `.env.dev.example` are placeholder-clean and `bash .agents/validate.sh`
passes the template scan (quote the passing line in the commit message)
- [ ] `.env.dev` has the dev password, still mode `0600`, still containing its real NUT
credentials untouched
- [ ] `./scripts/dev-server.sh restart && ./scripts/dev-server.sh status` works unauthenticated
- [ ] all six screenshots exist in `data/logs/` and show live data, not the simulator
- [ ] the leak-check list above returns exactly the codes listed, recorded in `NOTES-auth.md`
- [ ] README's *Signing in* section and four config rows exist; `AGENTS.md` has the new
invariant, the `auth.py` line, and the three known gaps
- [ ] `uv run pytest` and `uv run pytest -m live --live` both green;
`uv run ruff check src tests agent && uv run pyright` clean; `bash .agents/validate.sh`
green
@@ -0,0 +1,55 @@
# Phase 12 — A passed battery self-test is news, not a warning
**Source:** `TODO.md` L3 — ""Last self-test: Done and passed" still shows up as a warning in the overview and UPS section"
**Story:** n/a
**Context:** `src/dashboards/sources/nut.py` (`assess()` — the explicit `health.checks[]` list, the "Last self-test" check at the `test_result` branch), `src/dashboards/alerts.py` (the overview alert rule that reads `health.checks[]`), `tests/test_nut.py` (the two existing "Last self-test" assertions, the `SimUps` test-result tests), `tests/fixtures/nut_apc.vars` / `nut_cyberpower.vars` (verbatim `LIST VAR` captures), `src/dashboards/sources/nut_sim.py` (the simulator's `PASSED`/`FAILED` window), `src/dashboards/web/pages/ups.html` + `src/dashboards/web/static/page-ups.js` (the check list the `/ups` card renders), `data/logs/` (screenshots)
## Objective
The `Last self-test` check grades the driver's own word about the last battery test, and a
passed test — the real drivers answer **`Done and passed`** — must read `ok`, not `warn`.
The current word lists (`"ok", "passed", "pass", "completed"` / `"failed", "fail", "error",
"aborted", "test incomplete"`) let the one phrasing this fleet's hardware actually uses fall
through to `warn`, so a *healthy* UPS carries a warning on the overview and the UPS page.
After this phase the verdict follows the test's outcome: pass → `ok`, fail → `fail`,
never-tested / unset → `unknown`, and the overview's alert feed stops naming it.
## Dependencies
- `05_ups_hero_battery_glyph` (complete) — the `health.checks[]` contract and the `/ups`
check list this phase changes.
- `11_admin_login` (todo) — **no shared files**: 11 touches `app.py`/`config.py`/`store.py`/
the auth surface; this phase touches `sources/nut.py` + its tests + fixtures. Either order
runs; the only coupling is the shared `tests/test_nut.py` — apply whichever runs second on
top of the file state at that time.
## Tasks
1. `01_pin_self_test_wording.md` — capture the real drivers' `ups.test.result` values and pin the current (wrong) verdict in a failing test, so the fix has a before/after.
2. `02_verdict_map_for_test_results.md` — the `assess()` word map: pass phrasings → `ok`, fail phrasings → `fail`, "No test initiated"/unset → `unknown` (never `warn`); fixtures + `tests/test_nut.py` + the simulator's wording stay consistent.
3. `03_no_false_warning_overview_and_ups.md` — end-to-end: a passed test produces no alert on `/api/summary` and an `ok` chip on `/ups`; pinned by API test and a live screenshot.
## Testing & Quality
- Unit: `tests/test_nut.py` — the full word map (each pass phrasing `ok`, each fail phrasing
`fail`, `No test initiated` and unset `unknown`, case/whitespace-insensitive), driven from
the verbatim fixture wording where it exists and synthetic `UpsReading`s elsewhere.
- Integration: `tests/test_api.py` — a `health.checks[]` entry with a passed self-test
produces no alert in the `/api/summary` payload; a failed one does.
- The `SimUps` test-result tests (`tests/test_nut.py`) keep passing: the simulator's
`PASSED`/`FAILED` window words were already in the right tiers.
- Coverage: **>90%** on new/modified code; the overall gate (94%) must hold.
## Completion Criteria
- [ ] `printf 'LIST VAR apc ups.test.result\n' | nc <driveripper> 3051` (run from a host
that can reach upsd — the laptop still cannot, per the fleet facts) captured and
recorded in `tests/test_nut.py` / the fixture; the captured word asserts `ok`
- [ ] the "Last self-test" check renders `ok` (dimmed, no warning class) on `/ups` and does
not appear in the overview's Warnings panel; screenshots in `data/logs/`
(`self-test-passed-overview.png`, `self-test-passed-ups.png`)
- [ ] `uv run pytest` green, `uv run pytest --cov --cov-fail-under=94` green,
`uv run ruff check src tests agent && uv run pyright` clean, `bash .agents/validate.sh`
green
- [ ] no behavior change in completed phases: every other `health.checks[]` check (power
path, charge, runtime, battery age/temperature, load, alarms) grades exactly as before
@@ -0,0 +1,50 @@
# Task 01 — Pin the self-test word the real drivers use
**Phase:** `12_nut_self_test_verdict` · **Source:** `TODO.md:3` — ""Last self-test: Done and passed" still shows up as a warning in the overview and UPS section"
## Objective
Prove the bug in a test before touching the fix: the `Last self-test` check in
`assess()` must be shown to grade the drivers' real wording (`Done and passed`) as
`warn`, and the verbatim `ups.test.result` value must be captured from the running
upsd so the fixture/test asserts the hardware's actual sentence, not a memory of it
(AGENTS.md rule 7 — verify against the deployed upstream; the factory state both UPSes
answer is `No test initiated`, pinned in `AGENTS.md`'s fleet facts).
## Work
1. `tests/test_nut.py` — add the pinning test **first**: build the `UpsReading` the way
the two existing "Last self-test" tests do (the `SimUps`/`vars` helpers in the file),
with `test_result="Done and passed"`, and assert the current behaviour
(`check["status"] == "warn"`) with a comment that the *fix* flips it to `ok`. This is
the red test the whole phase turns green.
2. Capture the real value — from a host that can reach upsd (the laptop cannot yet,
per the fleet facts; `driveripper` itself, or the deployed dashboard's network):
`printf 'LIST VAR apc ups.test.result\nLIST VAR cyberpower ups.test.result\n' | nc
<upsd-host> 3051` — record the exact words in `tests/test_nut.py` (an assertion per
UPS where the value is a stable driver constant) and note the capture date. If a test
has been run on the hardware, the value is the driver's pass word; if not, it is
`No test initiated` — either way it goes into the test, because the verdict map must
cover both.
3. `tests/fixtures/nut_apc.vars` / `nut_cyberpower.vars` — if the live `ups.test.result`
line differs from the captured fixture (a test has since run), re-capture with
`./scripts/capture-fixtures.sh` (the script scrubs serials; the test-result line is
not a secret) so the fixtures stay verbatim ground truth.
4. Confirm the symptom end-to-end while it is live: `curl -s
http://127.0.0.1:8090/api/summary` (signed in, per phase 11) shows the alert row for
the `Last self-test` check, and `/api/ups/snapshot`'s `health.checks[]` carries the
`warn` entry — the two surfaces the TODO names.
## Testing & Quality
- Unit: the pinning test (red now, green after task 02).
- Coverage: **>90%** on this task's new/modified code.
## Completion Criteria
- [ ] `uv run pytest tests/test_nut.py -k self_test` shows the new test **failing** on
the current code (or passing if the wording already graded `ok` — in which case
record that and task 02 narrows to the `No test initiated` branch only)
- [ ] the real `ups.test.result` word is captured from the running upsd and recorded in
the test suite with its capture date
- [ ] fixtures re-captured if the live value differs from the recorded one
@@ -0,0 +1,62 @@
# Task 02 — Grade the test outcome, not the phrasing
**Phase:** `12_nut_self_test_verdict` · **Source:** `TODO.md:3` — ""Last self-test: Done and passed" still shows up as a warning in the overview and UPS section"
## Objective
The `Last self-test` branch in `assess()` maps the driver's word to a verdict with two
hardcoded tuples, and any phrasing outside both falls through to `warn` — which is how
`Done and passed` (the real drivers' pass word) and `No test initiated` (the factory
state) both read as warnings. Rewrite the branch as an explicit outcome map: pass
phrasings → `ok`, fail phrasings → `fail`, "no test has run" phrasings and unset →
`unknown`. A warning for a *passed* test is the bug; a warning for a machine that never
ran a test is noise this phase removes too.
## Work
1. `src/dashboards/sources/nut.py` — the `test_result` branch in `assess()`:
- pass tier: the existing `("ok", "passed", "pass", "completed")` **plus**
`done and passed` (the live word captured in task 01, kept lowercase/trimmed —
keep the existing `strip().lower()` normalisation);
- fail tier: unchanged (`"failed", "fail", "error", "aborted", "test incomplete"`);
- new `unknown` tier: `no test initiated` (and the empty-string case) — reuse the
existing `None`-branch sentence ("no self-test result reported — run a battery
test") so the two "never tested" states say the same thing;
- the fall-through for any *other* phrasing stays `warn` (an unrecognised word is
the one thing worth a warning — a driver inventing a new word is drift worth
noticing), and the check keeps printing the raw word plus the `test_date` suffix.
- ASSUMPTION: `No test initiated` (the factory state on both UPSes, per AGENTS.md
fleet facts) maps to `unknown`, not `warn` — a pack that has never been
self-tested is a suggestion to run one, not a fault; the `None` branch already
says exactly that, so the two states read identically.
2. `tests/test_nut.py` — the task-01 red test now asserts `ok`; add the full map:
every pass phrasing (including the captured live word) → `ok`, every fail phrasing
→ `fail`, `No test initiated` / empty / unset → `unknown` with the run-a-test
sentence, an unknown phrasing (e.g. `mystery`) → `warn`; case and whitespace
variations of the live word.
3. `tests/fixtures/nut_apc.vars` / `nut_cyberpower.vars` — if the captured live value
(task 01) is the factory `No test initiated`, the fixtures already carry it; if a
test has run, the fixture carries `Done and passed` — either way the fixture-driven
tests in `tests/test_ups_contract.py` / `test_nut.py` must pass against the new map
(the fixture's word is part of the hero contract's check list).
4. `src/dashboards/sources/nut_sim.py` — no code change expected: the simulator's
`PASSED` / `FAILED` window words are already in the pass/fail tiers; run the
`SimUps` test-result tests to prove it. If the simulator ever answers a *third*
word, add it to the map — do not special-case the simulator.
## Testing & Quality
- Unit: the full word-map table in `tests/test_nut.py` (the tiers above, plus
`test_date` suffix still printed on the pass/fail detail line).
- Regression: every pre-existing `test_nut.py` / `test_ups_contract.py` test passes
unmodified in intent (the fixtures' words land in the right tier).
- Coverage: **>90%** on this task's new/modified code.
## Completion Criteria
- [ ] `uv run pytest tests/test_nut.py tests/test_ups_contract.py` green, including the
task-01 test now asserting `ok` for `Done and passed`
- [ ] no phrasing the two real drivers answer (verified against the task-01 capture)
falls into `warn`
- [ ] the `unknown` tier's detail sentence is the existing "run a battery test" one, so
unset and never-initiated read the same
@@ -0,0 +1,60 @@
# Task 03 — No false warning on the overview and the UPS page
**Phase:** `12_nut_self_test_verdict` · **Source:** `TODO.md:3` — ""Last self-test: Done and passed" still shows up as a warning in the overview and UPS section"
## Objective
Prove the fix end-to-end on the two surfaces the TODO names: a passed self-test must
produce **no alert** in `/api/summary` (the overview's Warnings panel) and an `ok`
entry in `/api/ups/snapshot`'s `health.checks[]` (the `/ups` check list), while a
*failed* test still alerts — the check keeps its reason for existing.
## Work
1. `tests/test_api.py` — the alert-absence pin: an app (the shared `create_app`
test harness, `auth_contract` seeding) whose UPS payload carries a `Last self-test`
check with a passed word answers `/api/summary` with **no** alert row naming it
(the `alerts` list is the thing the overview renders — a row there is the warning
the TODO sees); the same harness with a `fail`-tier word **does** produce the
alert, so the test pins both directions.
2. `src/dashboards/web/static/page-ups.js` / `src/dashboards/web/pages/ups.html` — no
code change expected (the check list renders `health.checks[]` verdicts verbatim);
verify the `ok` chip styling for the "Last self-test" row is the existing dim path
(`ok` stays quiet ink, per the shared verdict vocabulary). If the row needs no
markup, say so in the completion note — do not restyle.
3. Live verification (AGENTS.md invariant 10 — UI changes are screenshotted): the dev
server (`./scripts/dev-server.sh status` first; the source must be `ok`), signed in
with the dev password:
- `curl -s http://127.0.0.1:8090/api/summary` — no `Last self-test` alert while the
live UPSes report a passed (or never-initiated) result;
- the `/` overview: Warnings & errors panel shows no self-test row;
- the `/ups` page: the "Last self-test" row renders the driver's word with the `ok`
(or `unknown`) class.
4. `data/logs/` — `self-test-passed-overview.png` and `self-test-passed-ups.png`
(the overview with the Warnings panel visible, and the `/ups` card with the check
list visible). If the live UPSes are at the factory `No test initiated`, the
screenshots show the `unknown` dim state — the *absence of the warning* is the
point, in either state.
- ASSUMPTION: no battery test is run on the live hardware to force a `Done and
passed` state — a self-test puts real load on a battery things are plugged into,
which stays the operator's call (the AGENTS.md known-gaps rule). The hermetic
`fail`/`ok` pins in `tests/test_api.py` carry the behaviour; the live screenshots
carry whichever state the hardware currently reports.
## Testing & Quality
- Integration: the two-direction alert pin in `tests/test_api.py`.
- Live: the two surfaces checked per step 3, screenshots recorded.
- Coverage: **>90%** on this task's new/modified code (mostly test code — the overall
94% gate must hold).
## Completion Criteria
- [ ] `curl -s http://127.0.0.1:8090/api/summary | grep -i "self-test"` returns nothing
while the live `health.checks[]` entry is `ok`/`unknown`, and returns the alert
row when a `fail`-tier word is injected (the `tests/test_api.py` pin)
- [ ] `/` renders no self-test row in Warnings & errors; `/ups` renders the check with
its verdict class — both screenshotted in `data/logs/`
- [ ] `uv run pytest` green, `bash .agents/validate.sh` green
- [ ] the `Dev server stays up` rule held: `./scripts/dev-server.sh status` shows the
`ups` source `ok` before and after the screenshots
@@ -0,0 +1,60 @@
# Phase 13 — The LLM page: fleet spacing and an honest decode tok/s
**Source:** `TODO.md` L4–L5 — "In the LLM engines page the 'fleet' section has no spacing between it and the engine summaries" / "Decode tok/s sometimes jumps unrealistically high, could be a calculation error"
**Story:** n/a
**Context:** `src/dashboards/web/pages/llm.html` (the `#banners`/`#fleet`/`#strip`/engines section stack), `src/dashboards/web/static/dash.css` (`.strip`, the `.fleet` block, the shared `.85rem` rhythm), `src/dashboards/rates.py` (`RateEstimator` — the one piece of mutable state in the app), `src/dashboards/sources/llm.py` (`parse_gufo_metrics`/`parse_vllm_metrics`, the `measured()` Δcounter/Δt cells), `src/dashboards/prom.py` (`scalar(agg="sum")` over label sets), `tests/test_rates.py` / `tests/test_llm_source.py`, `tests/fixtures/metrics_gufo.txt` / `metrics_vllm.txt`, `data/logs/` (findings + screenshots)
## Objective
Two defects on the `/llm` page, one phase. (1) The `#fleet` aggregate block and the
`#strip` engine summaries sit flush — the page's shared `.85rem` vertical rhythm skips
the two sections that have no inline margin, so the page reads as one unbroken wall
above the engine panels. (2) The `decode tok/s` hero/chart occasionally prints values
far above anything the engines can produce: the measured Δcounter/Δt path is keyed
correctly by *name* (rule 2), but the *value* it diffs is a **sum over the label
set**, and the estimator's window-empty fallback divides by a `dt` it never bounds —
so a counter's carried value can leak into one interval's delta, or a stall can
rescale the window. The fix keeps every number measured and labelled (`Δ<window>`),
just makes the measurement honest.
## Dependencies
- `07_llm_monitor_style_pages` (complete) — the fleet block, the strip, and the
one-`/api/llm/history`-request-per-refresh contract this phase must not break.
- `04_llm_chart_readability` (complete) — the hero row and the `provenance()` tags.
- `11_admin_login` (todo) — no shared files (this phase touches `dash.css`, `llm.html`
optionally, `rates.py`, `sources/llm.py`, `prom.py` and their tests; 11 touches
`app.py`/`config.py`/`auth.py`). Either order runs.
## Tasks
1. `01_fleet_strip_spacing.md` — restore the shared `.85rem` rhythm between `#fleet` and `#strip` (and above `#fleet` when no banners are shown) in `dash.css`; screenshot.
2. `02_pin_decode_spike_mechanism.md` — diagnose the spike against the live engines and the stored history (AGENTS.md rule 7 — never guess the trigger); written findings in `data/logs/`.
3. `03_bounded_estimator_and_per_series_rates.md` — the `rates.py`/`sources/llm.py` fix: per-label-series diffing plus a bounded effective window, so a label-set change contributes zero and a stall contributes an `n/a`, never a spike.
4. `04_live_stable_verification.md` — the dev-server proof: sustained watch of the live `decode tok/s` with its `Δ<window>` label, screenshots, fixtures re-captured if the live shape changed.
## Testing & Quality
- Unit: `tests/test_rates.py` — the new window/label-set behaviours (see task 03); the
existing pinned behaviour (`test_window_is_labelled_with_the_span_actually_used`)
keeps passing — the labelled-window contract is load-bearing.
- Unit/integration: `tests/test_llm_source.py` — a metrics text whose label set gains a
series between two polls produces no spike in `decode_tokens_per_second`; the
fixture-driven parse tests pass unmodified in intent.
- Page: `tests/test_api.py` — the served `llm.html` + `dash.css` pin the spacing rule
(the same device that pins the other dash.css invariants).
- Coverage: **>90%** on new/modified code; the 94% gate must hold.
## Completion Criteria
- [ ] `/llm` shows the shared gap between the fleet block and the engine summaries, and
between the page head and the fleet block when no banners render — screenshots in
`data/logs/` (`llm-fleet-spacing.png`)
- [ ] the task-02 findings file exists and names the confirmed mechanism
- [ ] `tests/test_rates.py` + `tests/test_llm_source.py` pin: no spike on a label-set
change, no unbounded `dt`, the `Δ<window>` label always the window actually used
- [ ] a sustained live watch shows no `decode tok/s` value above the confirmed
physical ceiling for the fleet's cards (the task-02 note records the ceiling)
- [ ] `uv run pytest` green, `bash .agents/validate.sh` green
- [ ] no behavior change in completed phases: the strip, the fleet block, the slot grid
and the trend charts render exactly as phase 07/04 shipped them, apart from the gap
@@ -0,0 +1,59 @@
# Task 01 — The gap between the fleet block and the engine summaries
**Phase:** `13_llm_page_and_decode_rate` · **Source:** `TODO.md:4` — "In the LLM engines page the 'fleet' section has no spacing between it and the engine summaries"
## Objective
`#llm`'s top-level sections break the page's shared `.85rem` vertical rhythm:
`#banners` ends with `margin-bottom:.85rem`, the engine section below `#strip` carries
an inline `margin-top:.85rem`, but `#fleet` and `#strip` carry **no** margin at all —
so the fleet block and the engine summaries sit flush (and, with no banners, the fleet
block also sits flush against the page head). Restore the rhythm with two CSS rules;
no markup or JS change.
## Work
1. `src/dashboards/web/static/dash.css` — in the fleet/strip section:
- `#strip { margin-top: .85rem; }` — the gap the TODO names (fleet → engine
summaries). Unconditional on purpose: when `#fleet` is `hidden` (the
below-two-engines case) the strip is the first visible section and still needs
the same top gap.
- `#banners:empty ~ #fleet { margin-top: .85rem; }` — the gap above the fleet block
*only* when no banner occupies the slot above it; with a banner present, the
banner's own `margin-bottom:.85rem` already is the gap (never two stacked gaps).
- The engine section's existing inline `margin-top:.85rem` stays (it already is the
strip → panels gap).
2. `src/dashboards/web/pages/llm.html` — no change expected; if the executor finds the
sections are not siblings the way the markup suggests, fix the CSS selectors to the
actual structure rather than re-flowing the markup (the section order is a phase-07
contract).
3. `tests/test_api.py` — pin the two rules on the served bytes (the existing dash.css
pin device: a test that fails if the rules are missing from the served stylesheet),
so a later "cleanup" that drops them fails a test.
4. Screenshot (invariant 10): dev server up, `/llm` with **≥2 engines answering** (the
fleet block visible — below two engines it stays hidden and the fix is invisible),
full page top-to-bottom: `data/logs/llm-fleet-spacing.png`. If the live fleet cannot
show two engines (the transport gap in AGENTS.md's fleet facts), pin the spacing on
the simulated path is **not** an option for a layout claim — instead screenshot the
one-engine page (strip visible, `#fleet` hidden, strip's `margin-top` doing its job)
and record in the completion note that the two-engine gap was verified by the CSS
rule + the served-bytes test, with the live two-engine capture deferred to the
operator's engine restore.
- ASSUMPTION: the fix is CSS-only in `dash.css`, not an HTML wrapper change — the
page's section stack is a phase-07 layout contract and the page already uses
inline `margin-top:.85rem` for the same rhythm, so a rule (not a restructure) is
the change in the page's own vocabulary.
## Testing & Quality
- Page: the served-bytes pin in `tests/test_api.py`.
- Live: the screenshot per step 4.
- Coverage: **>90%** on new/modified code (CSS + test — the 94% gate must hold).
## Completion Criteria
- [ ] `#strip` has a `.85rem` top gap in all states (fleet visible, fleet hidden);
`#fleet` has the gap above it exactly when no banner is shown
- [ ] no double gap (banner + rule) and no flush pair anywhere in the section stack
- [ ] `tests/test_api.py` pin green; screenshot in `data/logs/`
- [ ] `bash .agents/validate.sh` green
@@ -0,0 +1,72 @@
# Task 02 — Pin the decode-tok/s spike against the live upstream
**Phase:** `13_llm_page_and_decode_rate` · **Source:** `TODO.md:5` — "Decode tok/s sometimes jumps unrealistically high, could be a calculation error"
## Objective
Rule 7: never guess the trigger. Before touching `rates.py`, confirm **which** mechanism
produces the spike, from the stored history and the live engines, and record the
findings in a file the fix task (03) and the verification task (04) both read.
## Work
1. Find the spikes in the stored history (the dev DB is the source of truth — AGENTS.md
known gap 4 records it at ~610 MB with a year of raw/1m/1h rollups):
`sqlite3 data/… "<db>" "SELECT ts, value FROM sample WHERE metric LIKE
'decode_tokens_per_second%' ORDER BY ts DESC LIMIT 40000"` — list the top-N values
and their timestamps. Note for each: which engine (`source`/`host`), and what the
neighbouring values look like (a one-poll spike vs a sustained step).
2. For each spike, read the **provenance** of the cell that produced it — the payload
carries it (rule 2): `computed:Δ<series>/Δt` + `Δ<window>` (our estimator) vs the
engine's own gauge name (the native `decode_tokens_per_second_native` source string).
A spike from the *engine's* gauge is the engine's rolling average, not our math —
the fix there is a label, not a code change. `sqlite3` the engine's native gauge
series alongside to separate the two paths.
3. For the estimator path, inspect the live exposure (from a host that can reach the
engines — the laptop can reach both, per the fleet facts):
- `curl -s <gufo>/metrics | grep -E "tokens_(predicted|generated)_total"` and the
vLLM equivalent — are the counters **plain** (as in `tests/fixtures/`), or
**labelled per slot/engine** in the live build? A label set that changes between
polls is the leading suspect: `prom.Prom.scalar(agg="sum")` sums whatever label
sets exist, so a series that appears with a carried cumulative value adds that
whole value to one interval's delta (a `value < samples[-1]` decrease would reset
the window — an *increase* from a new series does not).
- Cross-check the spike timestamps against `GET /slots` history (which slots were
starting/turning over) and the event feed (`/api/events`) for engine restarts in
the same minutes.
4. Record the findings in `data/logs/NOTES-decode-spike.md`: the confirmed mechanism
(one of the candidates below, or a new one), the evidence (values, timestamps,
captures), and the fleet's physical decode ceiling for the task-04 comparison —
the observed max of the native gauge over 90d **and** the cards' spec ceiling
(R9700 / Strix Halo 8060S), so "unrealistically high" has a number attached.
Candidates, in suspected order (the findings file must name the one that fired):
- (a) **label-set sum**: `scalar(agg="sum")` over a changing set of labelled
series — a new series' carried counter enters the delta as a one-interval spike;
- (b) **window-empty fallback**: `rates.py`'s `base = samples[-2]` when fewer than
two samples sit inside the span — the effective window is the gap to the previous
sample, unbounded above by 2×span (the trim keeps the last two however old);
- (c) **a lineage jump that is not a decrease**: a counter that restarts *above*
zero (a renamed/relabelled series resolving under the same key, a pre-loaded
counter) — `value < samples[-1]` never fires, so the carried value is diffed;
- (d) **the engine's own native gauge** (vLLM's `avg_*_throughput` rolling average) —
not our math; label-level fix only.
## Testing & Quality
- No code changes in this task — diagnostics only, with the findings file as the
artifact. If a candidate is confirmed hermetically reproducible, add the red test
here (a `tests/test_llm_source.py` / `tests/test_rates.py` test that fails on the
current code) so task 03 turns it green.
- Coverage: **>90%** on any test added (the 94% gate must hold).
## Completion Criteria
- [ ] `data/logs/NOTES-decode-spike.md` exists: confirmed mechanism, evidence, and the
physical ceiling with its two sources (observed native-gauge max + card spec)
- [ ] every spike in the stored top-N is classified (our estimator / the engine's
gauge / a real burst) — a spike that cannot be classified is a finding, not a
skipped row
- [ ] a red test exists for the confirmed mechanism if it is hermetically reproducible
- [ ] the fixtures were compared against the live exposure and any shape drift is
recorded (drift → `./scripts/capture-fixtures.sh` re-capture noted for task 04)
@@ -0,0 +1,85 @@
# Task 03 — A rate that cannot be a carried counter
**Phase:** `13_llm_page_and_decode_rate` · **Source:** `TODO.md:5` — "Decode tok/s sometimes jumps unrealistically high, could be a calculation error"
## Objective
Fix the confirmed mechanism from task 02, plus the two generic weaknesses that make
any mechanism possible: a summed counter whose *label set* changes between polls, and
an effective window the estimator never bounds. After this task, a delta that includes
a carried counter value is impossible by construction — a new series contributes
nothing until it has a baseline, a stalled window answers `n/a` for one poll instead of
rescaling, and a jump that looks like a lineage break is confirmed over two polls
before it is published. Every published number still carries its `Δ<window>` label and
`source` string (rule 2).
## Work
1. `src/dashboards/prom.py` — expose the **per-label-set** view the fix needs (the
`series` map already holds them; `by_label` shows the shape): a `values_by_series`
(or reuse `values` with labels) accessor that returns `[(labels, value)]` per
resolved name, so the source can diff each labelled series individually. Keep
`scalar` untouched (the gauges still sum/avg/max the same way — a *gauge's* sum is
a real aggregate; only *counter* deltas may not mix series).
2. `src/dashboards/sources/llm.py` — the `measured()` path (gufo's
`tokens_generated_total`/`tokens_prompt_total` and vLLM's
`generation_tokens_total`/`prompt_tokens_total`): resolve the per-label-set values
and run **each series through the estimator under its own key** (the key is the
resolved name *plus* its label set — `llamacpp:tokens_predicted_total{slot="2"}`),
then `rate = Σ per-series rates`, with the cell's `window` the max of the
contributing series' windows (the window label says the widest window actually
used). A series with no previous sample contributes 0 tokens this interval — never
its carried value. The `Δ<window>` label and the `computed:Δ…/Δt` source string stay
in the `provenance()` format the page already prints (the source string names the
resolved series, as today).
- ASSUMPTION: per-series diffing is the fix for candidates (a) and (c) — a
label-set change then changes *nothing* in the delta until the new series has two
samples, which is exactly the "a renamed counter starts a fresh window" doctrine
the estimator docstring already states, extended from names to label sets.
3. `src/dashboards/rates.py` — the two generic bounds:
- **stalled-window bound**: in the `len(inside) < 2` fallback, accept `samples[-2]`
as the base only while its age is ≤ 1.5 × `span`; older than that, return
`(None, None)` for this poll — one `n/a` on the poll after a stall is the honest
answer (a 20-second window on a 10-second panel is a different claim), and the
next poll recovers normally. The existing `test_window_is_labelled_with_the_span_actually_used`
(a 9s fallback) keeps passing: 9s ≤ 15s.
- **jump confirmation**: a published rate ≥ 8× the previous poll's published rate
(previous > 0) is held — the previous value is repeated for one more poll, and the
jump publishes only if *confirmed* by the next interval; an unconfirmed jump is
dropped (the cell prints the held value, its `Δ<window>` label unchanged). This
suppresses a single-interval artifact without flattening a real burst: a genuine
step-up is confirmed one interval later and then publishes, and the confirmation
state is per-key in the estimator.
- ASSUMPTION: the 8×-over-previous threshold and the one-poll hold are the right
confirmation rule — a real LLM decode step (a batch of long outputs starting)
persists across a 10s interval, while a carried-counter artifact does not; the
thresholds are named constants at the top of the file so they can move without a
logic change.
4. `tests/test_rates.py` — the pins: a stalled window (previous sample > 1.5×span old)
answers `(None, None)` for one poll and recovers; a jump ≥ 8× holds for one poll
and publishes when confirmed / drops when not; the pinned pre-existing tests all
pass unmodified.
5. `tests/test_llm_source.py` — the label-set pin: two parses of a metrics text whose
counter series set differs between polls (a `{slot="2"}` series appears carrying
50 000) — the delta of the second poll contains **no** 50 000; the flat-counter and
counter-reset tests keep passing.
## Testing & Quality
- Unit: `tests/test_rates.py` + `tests/test_llm_source.py` per the work list.
- Regression: every pre-existing `test_llm_source.py` / `test_rates.py` test passes
unmodified in intent; the fixture-driven parse tests (`metrics_gufo.txt`,
`metrics_vllm.txt`) are the shape pins.
- Coverage: **>90%** on this task's new/modified code; the 94% gate must hold.
## Completion Criteria
- [ ] no delta can contain a value that was not generated inside the window: the
label-set test, the reset test and the stall test all green
- [ ] a real step-up (sustained over two polls) still publishes — the confirmation
path is tested, not just the suppression path
- [ ] every published rate still carries `Δ<window>` (the window actually used, possibly
a sum-of-windows label) and its `source` string — `tests/test_llm_source.py`
asserts the provenance shape
- [ ] the task-02 red test is green
- [ ] `bash .agents/validate.sh` green
@@ -0,0 +1,50 @@
# Task 04 — Live proof: a decode rate that stays under its ceiling
**Phase:** `13_llm_page_and_decode_rate` · **Source:** `TODO.md:5` — "Decode tok/s sometimes jumps unrealistically high, could be a calculation error"
## Objective
The dev-server proof that the fix holds on the real engines: a sustained watch of the
live `decode tok/s` (payload cell + stored history) with its `Δ<window>` label, every
observed value under the physical ceiling task 02 recorded, and the page itself
screenshotted with the task-01 spacing in view.
## Work
1. Dev server up and the LLM sources `ok` (`./scripts/dev-server.sh status`); record
the start time. Watch for at least one full poll-cycle cluster with traffic if the
engines are idle (the operator's own requests are the traffic — do **not**
generate load: a live test that puts real work on the engines' batteries is not a
test, the AGENTS.md live-suite rule):
- `curl -s http://127.0.0.1:8090/api/llm/snapshot` every poll for ≥ 10 minutes:
log `decode_tokens_per_second` (value, `source`, `window`) per engine;
- then `sqlite3` the `decode_tokens_per_second` raw series for the same period and
confirm no stored value exceeds the task-02 ceiling.
2. `data/logs/llm-decode-stable.png` — the `/llm` page showing the hero row with the
`Δ<window>` provenance tag, the strip card, and (when ≥ 2 engines answer) the fleet
block with the task-01 gap; a second screenshot if a real burst was observed
mid-watch (the confirmed-and-published path — the burst's `Δ` label visible).
3. `./scripts/capture-fixtures.sh` — re-capture **only if** task 02 recorded a live
shape drift (a label set the fixtures lack); otherwise leave the fixtures
untouched and say so.
4. `uv run pytest -m live --live` — the opt-in live suite (read-only per its rule;
the LLM engine tests skip when their source is unset). Record the tally in the
completion note (the fleet's transport gap means the host/disks live tests may
still fail — the LLM and UPS ones are this task's bar).
## Testing & Quality
- Live: the watch + `sqlite3` comparison per step 1, screenshots per step 2.
- No code changes expected in this task — if the watch finds a residual spike the
fix missed, the findings go back into `data/logs/NOTES-decode-spike.md` and task 03
is re-opened (record the re-open in the completion note rather than fixing inline).
## Completion Criteria
- [ ] ≥ 10 minutes of live polls: no `decode_tokens_per_second` value (payload or
stored) above the task-02 ceiling; the `Δ<window>` label present on every
non-`n/a` value
- [ ] `data/logs/llm-decode-stable.png` (and the burst screenshot if one occurred)
- [ ] fixture re-capture decision recorded; `bash .agents/validate.sh` green
- [ ] the live tally recorded, with the known transport gap acknowledged as the reason
any host/disks live failures are out of scope here
@@ -0,0 +1,66 @@
# Phase 14 — The store: fast reads, short writes, no "database is locked"
**Source:** `TODO.md` L7 — "Pages take forever to load and seem to lock up the database. Fix that."
**Story:** n/a
**Context:** `src/dashboards/store.py` (the `sample`/`event`/`meta` schema, `pick_res`, `history`, `series_names`, `stats`, `maintain`, the single `self._lock`, `busy_timeout=5000`), `src/dashboards/api.py` (the N-hosts × M-metrics history loops, `/api/metrics`, the polled `/api/health`), `src/dashboards/poller.py` (`add_many` on every successful poll — the write side that loses the `busy_timeout` race), `tests/test_store.py`, `tests/test_api.py`, `data/logs/` (the ~610 MB dev DB per AGENTS.md known gap 4 — the `store-error: OperationalError: database is locked` line)
## Objective
The page load time and the `database is locked` lines are one disease: the hot read
paths of a 610 MB / 140 MB-WAL database are **full-table operations on every request**
(`series_names()` scans for `DISTINCT metric`, the browser-polled `/api/health` runs
`stats()` which counts the whole `sample` table, `pick_res()` pays three COUNT
round-trips per series), while `maintain()`'s rollups are long whole-table write
transactions that hold the WAL write lock long enough for a poller's `add_many` to
outlive `busy_timeout=5000` and surface as the reported `store-error` (AGENTS.md
known gap 4). This phase makes every hot path bounded (indexes, one grouped query,
short-TTL caches) and every write short (windowed rollups, WAL checkpoint tuning),
with timing budgets pinned as tests so a regression is a red test, not a slow page.
## Dependencies
- the foundation (complete) — `store.py`/`api.py`/`poller.py` as shipped; the rollup
and retention contract in AGENTS.md invariant 9.
- `11_admin_login` (todo) — **shared files, run-order note**: 11 rewrites `api.py`
(the session gate) and adds `get_meta`/`set_meta`/`delete_meta` to `store.py`. Apply
this phase's `store.py`/`api.py` changes on top of the file state at the time it
runs; never restore a line the other phase owns. The two phases touch disjoint
*regions* (11: the meta helpers + the auth routes; 14: the query paths + the
maintain rollups) — the coupling is the file, not the logic.
## Tasks
1. `01_baseline_store_costs.md` — a production-shaped seeded store + the timing/EXPLAIN baseline for every hot path, recorded in `data/logs/`.
2. `02_no_full_scan_on_hot_paths.md` — `series_names`/`stats` off the per-request full scan (bounded caches), `pick_res` to one grouped query, the N×M history endpoints to one grouped round trip.
3. `03_windowed_rollups_and_wal_tuning.md` — `maintain()` rolls up only the window since `last_maintain`; WAL checkpoint + journal-size tuning so the 140 MB WAL stops growing between checkpoints.
4. `04_budget_gates_and_live_proof.md` — the timing budgets pinned as tests, the live dev-server proof (page load measured, zero `store-error` over a watch window), docs updated.
## Testing & Quality
- Unit/integration: `tests/test_store.py` — the budget tests (a seeded
production-shaped store: the task-01 recipe), the cache-invalidation behaviour, the
windowed-rollup correctness (a rollup run twice rolls the new window only, and the
1m/1h values match a whole-table rollup on the same data — the windowing must not
change a single averaged value).
- Integration: `tests/test_api.py` — the N×M history endpoints' response shape is
byte-identical to today (only the cost changes), under the new budget.
- Live: the dev-server watch per task 04 (AGENTS.md invariant 10 — the dev server stays
up while work happens; invariant 9 — the DB stays bounded, on a volume in
deployment).
- Coverage: **>90%** on new/modified code; the 94% gate must hold.
## Completion Criteria
- [ ] the task-01 baseline and the task-04 "after" numbers are both in
`data/logs/NOTES-store-perf.md`, side by side
- [ ] the budget tests are green: the full `/system` history request, `/api/metrics`,
`/api/health` and a 90d `history()` call all inside their pinned budgets on the
seeded store
- [ ] a 25-minute dev-server watch (the task-04 recipe) shows **zero** `store-error`
events and page loads inside the budget — the known-gap-4 failure mode is gone,
not hidden
- [ ] `maintain()`'s write transaction is measurably shorter (the task-03 note) and
the WAL stays bounded across two maintain cycles
- [ ] `uv run pytest` green, `bash .agents/validate.sh` green
- [ ] no behavior change in completed phases: every history payload, rollup value and
event answer is identical to today's shape (the `tests/test_api.py` pins)
@@ -0,0 +1,68 @@
# Task 01 — What the hot paths actually cost
**Phase:** `14_store_performance` · **Source:** `TODO.md:7` — "Pages take forever to load and seem to lock up the database. Fix that."
## Objective
A before/after exists only if the "before" is measured, not remembered. Seed a
production-shaped store, time every hot path of `store.py`/`api.py` against it,
`EXPLAIN QUERY PLAN` the queries that will change, and record the baseline in
`data/logs/NOTES-store-perf.md` — the numbers task 04 compares against.
## Work
1. `tests/test_store.py` — the seeded-store recipe (a fixture the budget tests of this
phase all reuse): an on-disk SQLite file (WAL, like production — the in-memory DB
hides the lock behaviour) seeded to the dev DB's *shape and retention semantics*
(`raw_keep_hours=48`, `minute_keep_days=30`, `hour_keep_days=400` — the
`SampleStore` defaults, i.e. what a 90 d chart actually reads: 1m/1h, never raw
beyond 48 h):
- ~3 hosts × ~30 series per host, 10 s cadence for the 48 h raw window;
- their 1m rollups for the 30 d window and 1h rollups for the 400 d window
(generated by calling `maintain()` — a few hundred MB, the dev DB's order of
magnitude, a temp file, never the dev DB itself: tests stay hermetic per the
AGENTS.md testing policy);
- ~2 000 event rows and the `meta` rows the app writes (`last_maintain` included).
The test asserts the row counts, not the wall time of seeding.
- ASSUMPTION: "production-shaped" means the dev DB's *shape and magnitude* (~610 MB,
per AGENTS.md known gap 4), reproduced in a temp file — the dev DB itself is
never opened by a test (tests stay hermetic, the AGENTS.md testing policy), and
a temp DB of the same shape is the honest substitute.
2. Measure (a script or the test's `--store-perf` marker; record in
`data/logs/NOTES-store-perf.md` with the machine's specs):
- `store.series_names()` (what `/api/metrics` runs per call) — wall time +
`EXPLAIN QUERY PLAN` (expected: a full scan of `sample`);
- `store.stats()` (what the browser-polled `/api/health` runs) — wall time +
EXPLAIN (expected: a full COUNT scan of `sample` + `event`);
- `store.pick_res()` × 3 + `store.history()` for one 90 d window (what one chart
series costs) — wall time + EXPLAIN;
- the full `/api/system/history` request shape (3 hosts × 9 series — the
`tests/test_system_contract.py` table, 27 `_history` calls) through the test app
— total wall time;
- a `maintain()` run on the seeded store — the write transaction's wall time
(the lock-holding window; note the `to_hour` step already scans the *entire*
30 d 1m window on every run — the suspected long write) and the WAL file's
size before/after;
3. Record the baseline table in `data/logs/NOTES-store-perf.md`: path, wall time,
EXPLAIN shape, and the wall time of the poller's `add_many` (a 30-sample insert)
issued **concurrently** with a `stats()`/`series_names()` call — the race known gap
4 describes, made visible as a number (how often does the insert wait, and does it
ever outlive `busy_timeout=5000`? run the race loop ~100× and count the waits).
## Testing & Quality
- Unit: the seeded-store fixture in `tests/test_store.py` (asserts row counts and the
rollup values match a second `maintain()` pass — determinism check).
- No behaviour changes in this task — measurement only.
- Coverage: **>90%** on the fixture code; the 94% gate must hold.
## Completion Criteria
- [ ] `data/logs/NOTES-store-perf.md` exists: the full baseline table (wall times,
EXPLAIN shapes, the concurrent-insert wait count), the machine's specs, and the
seeded DB's row counts
- [ ] the baseline confirms (or refutes — record the refutation) the three suspect
mechanisms: `series_names` full scan, `stats` full scan, `maintain`'s long write
window
- [ ] the fixture is green and deterministic (two runs, same rollup values)
- [ ] `bash .agents/validate.sh` green
@@ -0,0 +1,70 @@
# Task 02 — No full table scan on a per-request path
**Phase:** `14_store_performance` · **Source:** `TODO.md:7` — "Pages take forever to load and seem to lock up the database. Fix that."
## Objective
Three per-request paths pay a whole-table price on a 610 MB database:
`series_names()` (a `SELECT DISTINCT metric` scan, on every `/api/metrics` call),
`stats()` (a `COUNT(*) GROUP BY res` over all of `sample` plus `event`, on every
browser-polled `/api/health`), and the N-hosts × M-metrics history endpoints
(`pick_res()` is three COUNT round-trips per series, so one `/system` chart request
is ~108 serialized round-trips through the single `self._lock`). Make every hot path
bounded: short-TTL caches for the two scanned answers, one grouped query where three
stand, and one grouped round trip where N×M stand. The response shapes are byte-
identical — only the cost moves.
## Work
1. `src/dashboards/store.py` — `series_names()`: an in-memory cache of the distinct
metric set (per `source`/`host` argument tuple, a 60 s TTL, invalidated on
`add`/`add_many`/`delete_series` via a dirty flag — a cache that outlives a write
answers "why is this chart empty" with yesterday's truth). `stats()`: cache the
`sample`/`event` counts the same way (30 s TTL) — the `/api/health` poll (the
browser's `app.js` masthead loop) must never run a full COUNT scan; `last_maintain`
stays read through `get_meta()` as today (the meta table is tiny; do not cache
what a three-row table costs nothing to read).
2. `src/dashboards/store.py` — `pick_res()`: replace the three per-`res` COUNT queries
with **one** `SELECT res, COUNT(*) … WHERE … ts >= ? GROUP BY res` (the three
resolutions in a single index-range pass), same decision logic, same return value.
3. `src/dashboards/api.py` — the four history endpoints' loops (system/disks/llm/ups —
the `[host, metric]` nesting in each) call `store.history()` per pair, which is
`pick_res()` (now 1 query) + the thinning query. Add the batched door in
`store.py`: `pick_res_many(hosts, metrics, span, max_points)` and
`history_many(hosts, metrics, span, max_points)` — one grouped COUNT over the whole
`host IN (…) AND metric IN (…) AND res IN (…) AND ts >= ?` range, and one
per-resolution thinning query (the per-series `LIMIT ?` tail-thinning kept by a
window function or a per-series subquery — the shape of each series' answer must
stay identical: ascending `(ts, value)`, `tail=True` semantics preserved). The
endpoints' response dicts are unchanged (the same series, the same order); the
`tests/test_api.py` pins are the contract.
- ASSUMPTION: one grouped round trip (plus one thinning query per resolution
present) is the target for the N×M endpoints — the alternative (N×M separate
queries, each now cheaper) leaves ~100 serialized round-trips on a 3-host request,
which on a 610 MB file is still the page-load tail; the grouped shape is the fix
the TODO asks for. The per-series thinning keeps `tail=True` (the "the chart must
have 'now'" docstring) exactly as today.
4. `tests/test_store.py` — the pins: `series_names()`/`stats()` answer from cache
within the TTL (a write after the read invalidates `series_names`; `stats` expires
on the clock); `pick_res_many`/`history_many` equal the per-series answers on the
seeded store (the byte-identical-shape test); a cache hit makes a second
`series_names()` call measurably not scan (assert via a query-counter hook or the
wall-time budget of task 04).
## Testing & Quality
- Unit/integration: per the work list, on the task-01 seeded store.
- Regression: every pre-existing `tests/test_store.py` / `tests/test_api.py` test
passes unmodified in intent — the response shapes are the contract.
- Coverage: **>90%** on this task's new/modified code; the 94% gate must hold.
## Completion Criteria
- [ ] no per-request path in `api.py` triggers a full-table scan of `sample` or
`event` (the task-04 budgets are the proof; `EXPLAIN QUERY PLAN` of the grouped
queries shows index-range access, recorded in the notes file)
- [ ] the N×M endpoints' answers are byte-identical to the per-series path on the
seeded store (the equality test)
- [ ] a fresh `series_names()`/`stats()` call costs the full scan (documented — the
first poll after a process start pays it once), a repeated call costs the cache
- [ ] `bash .agents/validate.sh` green
@@ -0,0 +1,78 @@
# Task 03 — A rollup that holds the write lock for seconds, not minutes
**Phase:** `14_store_performance` · **Source:** `TODO.md:7` — "Pages take forever to load and seem to lock up the database. Fix that."
## Objective
`maintain()` is "safe to call every minute" and is idempotent — but on a real-sized
store the `to_hour` step's `SELECT … GROUP BY` scans the **entire 30-day 1m window**
on every run (and `to_minute` the entire 48 h raw window), and that GROUP BY is
inside the one `with self._lock, self._db:` transaction: a multi-second write
transaction that holds the WAL write lock long enough for a poller's `add_many` to
outlive `busy_timeout=5000` and surface as the `store-error: OperationalError:
database is locked` line (AGENTS.md known gap 4). Window each step to *what became
due since the last run* — the idempotent `INSERT OR REPLACE` makes a slightly wider
re-scan safe, so the window is a cost bound, not a correctness device — and tune the
WAL so its 140 MB growth stops being a per-run surprise.
## Work
1. `src/dashboards/store.py` — `maintain()`, both rollup steps:
- read `last_maintain` through `get_meta()` **before** the transaction (the
existing `stats()` note applies: the meta read is outside the non-reentrant
lock);
- `to_minute`: the raw scan bound moves from `ts >= now - raw_keep_seconds` to
`ts >= max(now - raw_keep_seconds, last_maintain - MINUTE - 60)` — steady state
(a run per minute) scans ~one minute of raw rows; a missed interval scans the
catch-up window; a first run on a fresh DB (no `last_maintain` row) scans the
full retention window exactly as today;
- `to_hour`: the same pattern against the 1m rows (`ts >= max(now -
minute_keep_seconds, last_maintain - HOUR - 60)`). The `ts < now - {MINUTE|HOUR}
- 60` incompleteness guard stays — a bucket is still only rolled once its whole
window is in the past.
- the `DELETE … pruned` step is unchanged (the retention window is the correct
bound for a prune; it is `DELETE`s on indexed ranges, not a GROUP BY).
2. `src/dashboards/store.py` — the WAL tuning in `__init__` (after the existing
pragmas): `PRAGMA wal_autocheckpoint = 1000` (the default 1000 pages ≈ 4 MB
auto-checkpoint is already the default — the 140 MB WAL means checkpoints lose to
the write rate; the real lever is the journal-size cap) and
`PRAGMA journal_size_limit = 67108864` (64 MB — the WAL stops growing past it;
checkpoints then truncate it). Keep `journal_mode=WAL` and
`synchronous=NORMAL` (the "a lost sample is fine, corruption is not" comment
stands); keep `busy_timeout=5000` — the fix is a shorter write, not a longer wait.
- ASSUMPTION: 64 MB is the journal-size cap — the dev DB's 140 MB WAL is the
measured "too big" (AGENTS.md known gap 4), 64 MB is the nearest power of two
below it, and the cap is a pragma, not a migration; if the live watch (task 04)
shows checkpoint truncation starving writers, the cap moves (named constant in
`__init__`, one-line change).
3. `tests/test_store.py` — the pins:
- **windowing correctness**: seed a store, run `maintain()` (full window, no
`last_maintain` yet), advance time and add new 1m buckets, run `maintain()`
again — the second run's `to_hour`/`to_minute` scan (assert via the EXPLAIN/row
count hook, or a `sqlite3` trace) touches only the since-`last_maintain` window,
**and** the rolled 1m/1h values are byte-identical to a reference whole-window
rollup on the same data (the windowing must not change a single averaged value —
`INSERT OR REPLACE` idempotency is what makes this safe, prove it);
- **catch-up**: no `maintain` for 10 minutes, then one run — the 10 minutes of
buckets are all rolled (the wider window, not a gap);
- **the lock race**: the task-01 concurrent loop (a poller-style `add_many` during
a `maintain()` run) now completes with **zero** `OperationalError`s over the
100× loop (the baseline's wait count is the before, this is the after).
## Testing & Quality
- Unit: per the work list, on the task-01 seeded store.
- Regression: every pre-existing `tests/test_store.py` test passes unmodified in
intent (the idempotency/retention tests are the load-bearing ones).
- Coverage: **>90%** on this task's new/modified code; the 94% gate must hold.
## Completion Criteria
- [ ] the second `maintain()` run's write transaction is measured shorter than the
task-01 baseline (both numbers in `data/logs/NOTES-store-perf.md`), and the
rollup values are unchanged (the equality test)
- [ ] the 100× concurrent `add_many`-during-`maintain` loop: zero `OperationalError`
(baseline wait count vs after, in the notes file)
- [ ] two `maintain()` cycles on the seeded store: the WAL file size stays bounded by
the 64 MB cap (before/after sizes recorded)
- [ ] `bash .agents/validate.sh` green
@@ -0,0 +1,78 @@
# Task 04 — The budgets, and the dev server that proves them
**Phase:** `14_store_performance` · **Source:** `TODO.md:7` — "Pages take forever to load and seem to lock up the database. Fix that."
## Objective
The "after" numbers, pinned as tests so a regression is red instead of slow: the
task-01 baseline vs the fixed store, side by side in `data/logs/NOTES-store-perf.md`,
plus the live dev-server proof on the real 610 MB DB — page loads inside budget, zero
`store-error` events over a watch window (the known-gap-4 failure mode gone, not
hidden), and the docs updated.
## Work
1. `tests/test_store.py` / `tests/test_api.py` — the budget tests (the task-01 recipe
on the seeded store; budgets are the task-01 baseline's *target* — record the
chosen numbers in the notes file with the reasoning):
- the full `/api/system/history` request (3 hosts × 9 series, 1 h and 90 d):
wall time budget (the 90 d case reads the 1m/1h rollups — the one that used to
be the page-load tail);
- `store.series_names()` / `store.stats()` second call (cache hit): wall time
budget, asserted to be a fraction of the first call's (the scan is gone);
- one 90 d `store.history()` call: wall time budget;
- the budgets are generous (2–3× the measured "after"), so a slower CI machine does
not flake red — the notes file records both the machine's specs and the margin.
2. The live proof (AGENTS.md invariant 10 — the dev server stays up while work
happens, and a fix is screenshotted, not claimed; invariant 9 — the DB is the
volume's business in deployment, the dev DB is the test bench):
- `./scripts/dev-server.sh restart` (its `stop` waits for the old pid to die —
known gap 4's "two servers on one file" cause stays fixed), `status` shows the
sources;
- page loads: `curl -o /dev/null -s -w '%{time_total}'` on `/` and on the
history-heavy pages' API calls (signed in, per phase 11), five runs each,
recorded; the browser's `/llm` or `/system` page with a 90 d window selected is
the worst case — screenshot `data/logs/store-perf-live.png` (the page loaded,
the charts drawn, the masthead dots `ok`);
- the watch: ≥ 25 minutes of `./scripts/dev-server.sh logs` (longer than the
gap-4 baseline's "a few times an hour" rate) with **zero**
`store-error: OperationalError: database is locked` lines; the `store.stats()`
`event` count for `kind='store-error'` over the window is 0 (the `sqlite3`
check on the dev DB — read-only);
- the WAL file's size after the watch (the cap from task 03, in the real DB).
- ASSUMPTION: the watch window is 25 minutes against a "few times an hour"
baseline rate — enough to cover > 0.4 of the baseline's expected occurrences at
the mean rate; if the baseline was at the mean, a zero-count 25-minute window
already moves the needle, and the `sqlite3` event-table check over *the whole
dev DB's* `store-error` rows (not just the window) shows when the last
occurrence predates this phase.
3. Docs: `README.md` — the *Configuration*/*Development* notes gain the two new
`PRAGMA`s' one-line "why" (the journal cap, and that it is a cap not a target);
`AGENTS.md` — known gap 4's entry is updated to record the fix (the windowed
rollups, the journal cap, the budget tests) and the gap's "what remains" line
(checkpoint starvation was never shown to be a cause — the controlled 25 s read
transaction produced none — so the entry's honest wording stays, with the
`database is locked` line now attributed to the rollup's write window). (The
AGENTS.md/README update is the phase's *normal work* per the repo's definition of
done — this instruction is for the phase's executor, not for the conversion.)
## Testing & Quality
- Unit/integration: the budget tests per step 1.
- Live: the step-2 watch, screenshots and log evidence in `data/logs/`.
- Coverage: **>90%** on new/modified code; the 94% gate must hold.
## Completion Criteria
- [ ] `data/logs/NOTES-store-perf.md` has the baseline and the after table side by
side (every task-01 path, wall time before/after, EXPLAIN shapes)
- [ ] the budget tests are green with the recorded margins; `bash .agents/validate.sh`
green
- [ ] the 25-minute watch: zero `store-error` lines in the dev-server logs and zero
new `store-error` rows in the dev DB's `event` table; `data/logs/` holds the
page-load numbers and `store-perf-live.png`
- [ ] the WAL's post-watch size is inside the 64 MB cap on the real DB
- [ ] `README.md`/`AGENTS.md` carry the updated, accurate wording (gap 4: fixed, with
its "what remains" honest)
- [ ] no behavior change in completed phases: the `tests/test_api.py` payload pins all
pass — the answers are the same, only faster
@@ -0,0 +1,44 @@
# Phase 15 — Protected exporter web servers, auth by certificate
**Source:** `TODO.md` L3 — "I want all exporters to be protected web servers. The collector should generate and manage a root certificate that needs to be installed with each exporter. Use the certs as auth to each exporter."
**Story:** n/a
**Context:** `agent/host_agent.py` (the stdlib-only exporter web server; today: a bearer token via `--token`/`--token-file`, refuses an off-loopback bind without one, `ThreadingHTTPServer`, the `resolve_token` idiom), `src/dashboards/config.py` (`Target`/`parse_targets` — token in URL userinfo, `https://` URLs already accepted; **no change needed**), `src/dashboards/collectors.py` (`fetch_snapshot`, the one `UpstreamError`, redacted display URLs, the one shared `httpx.AsyncClient` from the lifespan), `src/dashboards/app.py` (the `PAGES` dict with the settings-page comment, `build_hub`/`build_context`, the lifespan that builds `ctx.auth`), `src/dashboards/api.py` (`AppContext`, the gated-write + event pattern), `src/dashboards/store.py` (`get_meta`/`set_meta`/`delete_meta` — the only durable key/value state, home of `auth.*`), `deploy/install-agent.sh` (the token-file idiom `0640 root:agent-user`, the `ProtectSystem=strict` unit with no `RuntimeDirectory`, the `--status` probe), `tests/test_agent.py` (the in-process agent harness), `tests/test_collectors.py`, `tests/test_deploy_scripts.py`, `tests/test_api.py` (the route walk, the settings-page pin, the orphan-id audit), `src/dashboards/web/pages/settings.html` + `src/dashboards/web/static/page-settings.js` (the "a page, not a dashboard" pattern the new `/certs` admin page follows)
## Objective
Every exporter (host agent) web server is protected end-to-end by certificates the dashboard itself generates and manages: the dashboard holds a root CA in the `meta` table, signs its own client cert and each agent's server cert, the agent serves TLS with a per-path client-cert check (only the dashboard can read it), the dashboard dials `https://` with its client cert, and a new `/certs` admin page lists every cert with its expiry and status and triggers rotation, revocation, and bundle re-delivery. The operator's stated follow-up — firewalling upsd 3051 and talking to the agent only — is the motivation, not part of this phase. The agent's bearer-token mode stays a compatibility mode; the dashboard's own login is a separate auth domain and is untouched.
**Scope (operator-confirmed):** "exporters" = the host agents this repo installs. gufo / vLLM / upsd are third-party services this repo does not deploy; their TLS (e.g. upsd `STARTTLS`) is a separate operator concern.
## Dependencies
- `14_store_performance` (todo) — **shared-file, run-order note**: 14 rewrites the hot paths in `store.py`/`api.py`; 15 only *adds* meta keys and new routes/pages. Apply on top of the file state at the time it runs; never restore a line the other phase owns.
- `11_admin_login` (complete) — the session gate (invariant 12) for the new routes, the gated-write + CSRF + event pattern for the new `POST /api/certs/*`, the `meta` helpers, and the settings-page pattern `/certs` follows.
## Tasks
1. `01_cert_management.md` — `src/dashboards/certs.py`: the dashboard-managed root CA, client cert, per-agent server certs, revocation and reissue, all in `meta`; the `cryptography` dependency (dashboard-side only).
2. `02_agent_tls.md` — `host_agent.py` TLS mode: one installed `--cert-bundle`, the per-path client-cert check, the bundle revocation list, `bundle_rev` in the snapshot; token mode unchanged.
3. `03_collector_tls.md` — the dashboard dials `https://` agent URLs with its client cert, verifies against its root, refuses a revoked agent without dialing; error sentences stay redacted.
4. `04_cert_api.md` — the gated `/api/certs` surface: list, per-host bundle, rotate, revoke, root-rotate; fixed-text events, no keys out of `meta`.
5. `05_certs_admin_page.md` — the `/certs` admin page (the settings pattern): every cert with its expiry and status, issued vs deployed bundle rev, rotate/revoke/root-rotate actions, bundle download.
6. `06_deploy_cert_bundle.md` — `install-agent.sh --cert-bundle` installs `/etc/dashboards-agent.certs` (`0600 root:agent-user`), the unit gains the flag + `RuntimeDirectory`, `--status` probes over TLS; the script pins move with it.
7. `07_live_rollout.md` — the two live agents flip to TLS, `HOST_TARGETS` goes `https://`, the live suite + health table + screenshot prove it; the `driveripper` install procedure is documented.
## Testing & Quality
- Unit: `tests/test_certs.py` (new) — the whole lifecycle against a dict `MetaStore`, exactly how `tests/test_auth.py` does it.
- Integration: `tests/test_agent.py` (the TLS handshake matrix against the in-process agent), `tests/test_collectors.py` (the `https` dial, the revoked refusal with a no-dial spy, the error sentences), `tests/test_api.py` (the `/api/certs` shapes, the gate/CSRF on the new writes, the served `/certs` page pins), `tests/test_deploy_scripts.py` (the new flags/files/permissions).
- Live: task 07 (AGENTS.md invariant 10 — the dev server stays up; screenshots in `data/logs/`).
- Coverage: **>90%** on new/modified code; the 94% gate holds. `uv pip list --outdated` empty (the `cryptography` dependency — invariant 8).
## Completion Criteria
- [ ] a fresh database boots with a `meta` root CA, generated once and idempotent across restarts (pinned in `tests/test_certs.py`)
- [ ] an agent run with a bundle answers `/snapshot` only to a client cert signed by its root; a wrong-CA or certless client is refused; a revoked serial gets the 403; `/health` stays probeable with the CA only
- [ ] with `https://` `HOST_TARGETS`, the dev server's `host:`/`disks:` jobs are `ok`, and a revoked host says "certificate revoked" without any dial (spy-pinned)
- [ ] signed-in `GET /certs` lists every cert with expiry and status; rotate/revoke/root-rotate work through the gated API; the bundle download is byte-identical to the stored bundle
- [ ] the installer places the bundle `0600 root:agent-user`, the unit comes up in TLS mode, `--status` is green over TLS
- [ ] `uv run pytest` green, `bash .agents/validate.sh` green, `uv pip list --outdated` empty
- [ ] live: deskwork + gateway in TLS mode, `https://` dials, live suite green, screenshots in `data/logs/`
- [ ] no behaviour change in completed phases: the token-mode agent is byte-identical, the engine/UPS polling is untouched, the public surface is unchanged (the route walk)
@@ -0,0 +1,43 @@
# Task 01 — The dashboard owns a root CA and every cert it signs
**Phase:** `15_agent_mtls` · **Source:** `TODO.md:3` — "I want all exporters to be protected web servers. The collector should generate and manage a root certificate that needs to be installed with each exporter. Use the certs as auth to each exporter."
## Objective
The certificate lifecycle becomes a pure, testable module: the dashboard generates a root CA once (idempotent across restarts), signs its own client cert and each agent's server cert, and supports reissue and revocation — all state in the `meta` table, exactly how `auth.py` keeps the verifier and the session secret.
## Work
1. `pyproject.toml` — add `cryptography` to `dependencies` (dashboard-side only; the agent stays stdlib-only) and run `uv lock`. Invariant 8: `uv pip list --outdated` empty afterwards.
- ASSUMPTION: `cryptography` is the only new dependency and it stays on the dashboard side; the agent (task 02) gains no third-party code — it loads the PEMs with stdlib `ssl`.
2. `src/dashboards/certs.py` (new) — the whole lifecycle as pure functions over the three-method `MetaStore` protocol (the same protocol `auth.load_auth_state` uses, so this module imports nothing else from the package, and its tests use a dict):
- `ensure_certs(store, *, now: float | None = None)` → a frozen `CertsState` carrying the PEM text (`root_key`, `root_cert`, `client_key`, `client_cert`) plus the root/client `serial` + `not_after`. Reads the `meta` rows first; only generates and persists when missing (first boot); an existing row wins and is **never regenerated on restart** — regenerating on every start would rotate the CA and invalidate every installed agent bundle. `now` is injected (the module never reads a clock itself — the `auth.py` rule).
- meta keys (one concern per key, the `auth.*` precedent): `certs.root.key`, `certs.root.cert`, `certs.client.key`, `certs.client.cert`, `certs.agents` (one JSON object mapping `host name` → `{serial, not_after, status, bundle_rev, bundle, san}` — the bundle text is stored, so what the agent holds is byte-identical to what the API serves).
- Shapes: root CA self-signed, CN `dashboards-root`, **10 years**; client cert signed by the root, CN `dashboards`, **1 year**; agent server certs signed by the root, CN = the host name, **1 year**, `SAN` = the address(es) the dashboard dials for that host.
- ASSUMPTION: 10 y root / 1 y leaf lifetimes. No env seed for the root CA (unlike the admin password, the root key is a secret only the app needs to know, so it is auto-generated on first boot and merely persisted).
- `issue_agent_cert(store, state, name, san: list[str])` / `reissue_agent_cert(store, state, name, san)` — a fresh serial, `bundle_rev` +1 (1 on first issue), `status: "active"`, a fresh `bundle` (below). Idempotent per call — never issued on a read.
- `rotate_root(store, state)` — a fresh root CA + fresh client cert + every existing agent entry reissued (each `bundle_rev` +1): one action re-arms the whole fleet.
- `revoke(store, state, name)` — `status: "revoked"` (the serial stays on the denylist until a reissue gives the host a fresh identity; there is no "unrevoke" — reissue is the fix).
- the **bundle** (the install artifact, the TODO's "root certificate installed with each exporter"): one PEM text = the agent's server cert + the agent's key + the root CA *cert* (never the root key) + a `# BUNDLE-REV <n>` line + a `# REVOKED <serial>` line (one per revoked serial of the *current* root generation — a root rotation resets the list, since every cert is reissued anyway).
- ASSUMPTION: one client cert for the whole dashboard (the agent verifies "signed by our root", not a per-process identity); one agent entry per `HOST_TARGETS` name.
- invariant 6, the `auth.py` rule: no `logging` and no raise with anything interpolated — a malformed stored PEM is a fixed-text `RuntimeError` telling the operator to delete the `certs.*` meta keys (the shape of the auth reset), and nothing else.
3. `src/dashboards/app.py` (+ the `AppContext` in `src/dashboards/api.py`) — the startup path builds `ctx.certs` (an `ensure_certs` read, the one-time bootstrap included) **before** `build_hub`, in both the owned-store (`build_context`) and the injected-store (lifespan) branches, through **one** helper so the two wiring sites cannot drift (the `build_hub` rule in AGENTS.md); the PEMs are written once to a private temp dir (mode `0600`, lives as long as the process) and the *paths* are what `AppContext.certs` carries for task 03's dialing.
- A `certs` whose stored PEM is malformed (a truncated `meta` row) is a fail-closed startup — a dashboard that cannot present its own credential must not start the loops (the `ctx.auth` rule, the one deliberate raise).
## Testing & Quality
- `tests/test_certs.py` (new) — a dict `MetaStore` (the `tests/test_auth.py` idiom):
- the first `ensure_certs` call persists exactly the five `certs.*` keys; the second is byte-identical (no regeneration — pin the `certs.root.cert` value);
- the parsed chain (`cryptography.x509` loaders) verifies: client → root, agent → root; the SAN list appears in the agent cert; the expiries match the assumed lifetimes (injected `now`);
- `reissue_agent_cert` bumps `serial` and `bundle_rev` and sets `status` active; `revoke` flips the status and puts the serial on the bundle's `# REVOKED` line; a reissue after a revoke gives a fresh serial that is *not* on the list;
- `rotate_root` reissues every agent entry (each `bundle_rev` bumped), the bundle carries the new root cert, and no stored value still contains the old root key;
- the bundle contains the agent cert + key + root cert + `# BUNDLE-REV` + the `# REVOKED` lines and **never** the root key (assert via the PEM header and a parsed-key mismatch);
- a malformed stored PEM → the fixed-text `RuntimeError` naming the meta keys, nothing interpolated.
- Coverage: **>90%** on this task's new code.
## Completion Criteria
- [ ] `uv run pytest tests/test_certs.py` green; the full suite green
- [ ] `uv pip list --outdated` empty; `uv run ruff check src tests agent` + `uv run pyright` clean
- [ ] a fresh store boots with the `certs.*` rows present; a second boot changes no byte of `certs.root.cert`
- [ ] no behaviour change in completed work: `auth.*` keys, the public surface, and the poller wiring are untouched
@@ -0,0 +1,37 @@
# Task 02 — The agent serves a protected TLS web server from one installed bundle
**Phase:** `15_agent_mtls` · **Source:** `TODO.md:3` — "I want all exporters to be protected web servers. The collector should generate and manage a root certificate that needs to be installed with each exporter. Use the certs as auth to each exporter."
## Objective
`host_agent.py` gains a TLS mode that makes the web server protected by certificate: one installed bundle holds the server cert + key, the root CA, a bundle revision, and a revocation list; the agent requires a client cert signed by the root on every path except `/health` (mirroring the token-mode `/health` exemption, so an installer probe can reach `/health` with the CA only), and it reports the bundle revision it loaded so the dashboard can tell a stale bundle from a fresh one. The token mode is byte-identical to today.
## Work
1. `agent/host_agent.py` — stdlib only (an `ssl` import; no new third-party dependency):
- a new flag `--cert-bundle <file>` (env `AGENT_CERT_BUNDLE`): one PEM text containing the server cert, the server key, the root CA cert, a `# BUNDLE-REV <n>` line, and `# REVOKED <serial>` lines. `resolve_bundle(path) -> (context, bundle_rev, revoked: frozenset[str], error)` is a pure function (the `resolve_token` idiom): parse the PEM blocks by text, extract the three blocks, write them to a temp dir for `load_cert_chain` (into `/run/dashboards-agent` when present and writable — the unit's `RuntimeDirectory`, task 06 — else `tempfile.mkdtemp`), and build the context.
- The TLS context: `ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER)`, `load_cert_chain`, `load_verify_locations(ca)`, `verify_mode = ssl.CERT_OPTIONAL` — the handshake is optional and the **handler** enforces it: on every path except `/health`, `self.connection.getpeercert()` must carry a verified certificate (subject present), else `403 {"ok": false, "detail": "client certificate required"}`; that cert's serial must not be in the bundle's `# REVOKED` set, else `403 {"ok": false, "detail": "client certificate revoked"}`. The denylist file is re-read on mtime change per request (a small file; a redeployed bundle takes effect without a restart — the reason the `# REVOKED` lines live in the bundle at all).
- The startup gate: off-loopback with neither a token nor a bundle → the existing exit-2 refusal (its message now names the bundle too); a bundle present → the off-loopback bind is allowed (the client cert is the protection); a token plus a bundle → the token check is skipped on the TLS path (the handshake is the auth).
- The snapshot payload gains two **additive** top-level fields in TLS mode: `tls: true` and `bundle_rev: <n>` (absent in token mode — an additive field never changes an existing shape); the `serve()` start line says `tls (client cert required, rev <n>)`.
- `AGENT_VERSION` 1.0.0 → 1.1.0; the docstring's deployment paragraph gains one line for the TLS mode (no flavour text — the AGENTS.md audit rule).
- ASSUMPTION: `CERT_OPTIONAL` + the per-path check (not `CERT_REQUIRED`) is what keeps `/health` probeable with the CA only — the installer's `--status` and the systemd probe (task 06) cannot present a client cert the installer does not hold, and a probe that cannot health-check is a dead install.
- ASSUMPTION: the agent never generates or renews certificates (stdlib cannot mint x509); the bundle is the artifact the dashboard signs and the installer places.
2. `tests/test_agent.py` — the existing in-process harness (`serve()` in a thread on `127.0.0.1`, `urllib`/`ssl` clients), with a test CA + leafs generated by `src/dashboards/certs.py` over a dict store (no second generator):
- a client with a matching client cert → `/snapshot` 200 with `tls: true` and `bundle_rev` equal to the bundle's;
- a client with no cert → the handshake fails or a 403 on a non-`/health` path; `/health` still 200 with a CA-only context (the probe path);
- a client signed by a *different* CA → refused;
- a client whose serial is in the bundle's `# REVOKED` list → 403 "revoked"; rewrite the bundle file without the line (mtime change) → the same client is accepted on the next request (the denylist re-read);
- a token-mode agent (no bundle): the existing tests unchanged and green (byte-identical behaviour — no `tls`/`bundle_rev` fields);
- `resolve_bundle` error paths: a missing file, an empty file, a bundle missing the key block, a bundle missing the `# BUNDLE-REV` line (an error — a revless bundle is not one this app signed; ASSUMPTION: the `# BUNDLE-REV` line is mandatory).
3. `deploy/` is untouched here (task 06); the non-systemd fallback (`tempfile.mkdtemp`) keeps dev and tests working.
## Testing & Quality
- the `tests/test_agent.py` matrix above; the existing token-mode tests stay green unchanged.
- Coverage: **>90%** on this task's new/modified code.
## Completion Criteria
- [ ] a TLS-mode agent on a loopback port answers the full handshake matrix (tests) and reports `bundle_rev`
- [ ] a token-mode agent is byte-identical to today (existing tests green, no new fields present)
- [ ] `uv run pytest` green; `uv run ruff check src tests agent` + `uv run pyright` clean
@@ -0,0 +1,35 @@
# Task 03 — The dashboard dials agents over https with its own client cert
**Phase:** `15_agent_mtls` · **Source:** `TODO.md:3` — "I want all exporters to be protected web servers. The collector should generate and manage a root certificate that needs to be installed with each exporter. Use the certs as auth to each exporter."
## Objective
An `https://` `HOST_TARGETS` entry is now a certificate-authenticated dial: the dashboard presents the client cert task 01 gives it, verifies the agent against its own root CA, refuses a *revoked* agent before dialing (invariant 3's shape — a source it knows is dead says so without a network), and passes the agent's `tls`/`bundle_rev` through the `system` and `disks` payloads so task 05's page can compare the issued rev against the deployed one. Engines and token-mode agents are untouched.
## Work
1. `src/dashboards/config.py` — **no change** (an `https://` URL already parses; a TLS host carries no userinfo, and the userinfo token stays available for token-mode hosts).
2. `src/dashboards/collectors.py` —
- a small frozen `AgentTls` dataclass: `verify: str` (the CA path), `cert: tuple[str, str]` (the client cert + key paths), `status: Callable[[], str]` — the `status` callable returns `"active"`/`"revoked"` from a **fresh** `get_meta("certs.agents")` read (a small-table `SELECT`, cheap at poll cadence, and phase 14's work keeps it off the hot paths; reading it live is what makes a runtime revocation take effect on the next poll without a restart).
- `fetch_snapshot(client, target, *, timeout=None, tls: AgentTls | None = None)`: when `tls` is given **and** `target.url` is `https://`, the request passes `verify=<ca path>` and `cert=(cert path, key path)` to `client.get` (httpx per-request overrides — the shared client dials both plain and TLS upstreams, so a per-client `verify` would poison the engine dials).
- Revocation before any network: `tls is not None` and `tls.status() == "revoked"` → `UpstreamError(f"{target.name}: agent certificate revoked")` **before** `client.get` — the poller's stale/last-good semantics then do the rest (invariant 4).
- A `403` from an agent gets its own honest sentence (the existing `401` branch keeps the token-mode wording): `f"{target.name}: agent refused the client certificate ({detail})"` with the `detail` from the agent's JSON (`client certificate required` / `client certificate revoked`), the redacted `display_url` (invariant 6) like every message in this function.
- `fetch_host_snapshot`/`fetch_disk_snapshot` pass the `tls` through; the per-host `AgentTls` is plumbed from `AppContext.certs` into the `host_jobs`/`disks_jobs` lambdas (a new optional `tls` parameter on both `*_jobs`, `build_hub` takes it — one call site per branch so the production and injected wiring cannot drift, the `build_hub` rule in AGENTS.md). `status` reads from the store the jobs already hold.
3. `src/dashboards/app.py` (+ the `AppContext` in `api.py`) — `AppContext.certs` (task 01) exposes the temp-dir paths (`ca`, `client_cert`, `client_key`); `build_context` and the lifespan's injected branch build the per-host `AgentTls` objects **before** `build_hub`.
4. `src/dashboards/sources/host.py` + `src/dashboards/sources/disks.py` — pass the agent's `tls`/`bundle_rev` snapshot fields through into the per-host payload (additive, the `degraded` idiom) so `/api/system/snapshot` and `/api/disks/snapshot` carry the deployed rev. **No** `severity` for a rev mismatch — a count is not a graded number (the fleet-header rule), the page colours nothing for it.
## Testing & Quality
- `tests/test_collectors.py` — an in-process TLS agent (task 02's harness, in a thread) + `AgentTls` material generated by `certs.ensure_certs` over a dict store:
- an `https` target with `tls` → the 200 body; a missing client-cert file and a wrong-CA failure still surface as `UpstreamError` with the redacted URL;
- a revoked status (dict store with `status: "revoked"`) → the error is raised and a spy on `client.get` (the `tests/test_api.py` spy idiom) proves **no dial happened**;
- an `http` token-mode target: the existing tests unchanged;
- the 403 sentence: a scripted agent 403 (a `tests/conftest.py` fake) → the `detail` appears in the message, the URL redacted;
- the payload passthrough: a snapshot with `tls: true, bundle_rev: 3` → both the `system` and the `disks` payloads carry it.
- Coverage: **>90%** on this task's new/modified code.
## Completion Criteria
- [ ] `uv run pytest` green; the engine/UPS/token-host paths are byte-identical (existing pins hold)
- [ ] a revoked host fails its `host:` and `disks:` jobs with "certificate revoked" and no network dial (spy-pinned)
- [ ] `uv run ruff check src tests agent` + `uv run pyright` clean
@@ -0,0 +1,36 @@
# Task 04 — The gated /api/certs surface: list, bundle, rotate, revoke, root-rotate
**Phase:** `15_agent_mtls` · **Source:** `TODO.md:3` — "I want all exporters to be protected web servers. The collector should generate and manage a root certificate that needs to be installed with each exporter. Use the certs as auth to each exporter."
## Objective
The API half of "the collector manages the certs": a signed-in operator can list every cert with its expiry and status, download the exact bundle an agent should hold, and trigger rotation / revocation / root rotation. The routes dial nothing (invariant 1's shape), are gated like every other write (invariant 12), and write fixed-text events — a key never leaves `meta` (invariant 6).
## Work
1. `src/dashboards/api.py` — the new routes (the middleware gates them by default; the route-walk test covers them for free):
- `GET /api/certs` — `{root: {serial, not_after}, client: {serial, not_after}, agents: [{name, serial, not_after, status, bundle_rev, deployed_bundle_rev, san}]}`. `deployed_bundle_rev` is the `bundle_rev` from the host's last `system` job payload (the same source `alerts.py` reads job states from); a host that never answered → `null`. Reads never issue (no side effects on a read — a `GET` that mints a cert is a surprise).
- `GET /api/certs/<name>/bundle` — the stored bundle text, `Content-Type: application/x-pem-file`, `Content-Disposition: attachment; filename="agent-<name>.bundle"` — byte-identical to what `certs.py` stored (the installer's input, task 06/07).
- `POST /api/certs/<name>/rotate` — a reissue (`certs.reissue_agent_cert`, the SANs taken from the configured `HOST_TARGETS` URL host(s) + the name); a `cert-rotated` event (a fixed message naming the host — no PEM, no key).
- `POST /api/certs/<name>/revoke` — `certs.revoke`; a `cert-revoked` event. The next poll refuses without dialing (task 03) — that is the enforcement; the event is the audit line.
- `POST /api/certs/root/rotate` — `certs.rotate_root`; a `cert-root-rotated` event; the response carries `{"ok": true, "restart_required": true}` — the client cert and CA are startup material, so a new root is fully in force at the next process start (the page, task 05, prints that; the honest constraint, not a hidden one).
- ASSUMPTION: a root rotation requires a dashboard restart to be fully in force (the client cert + CA load at startup); the API says so instead of pretending a hot-swap that does not exist.
- A `<name>` that is not a configured `HOST_TARGETS` name → `404 {"configured": false}` **without** issuing, dialing, or touching the store (invariant 3, the `/api/disks` gate shape, spy-pinned in `tests/test_api.py`).
- Every `POST` is a gated write: session + `Origin`/`Sec-Fetch-Site` + the cookie's `X-CSRF-Token` (the `POST /api/ups/command` test idiom — a tokenless write records nothing on the wire, with the token it does).
- invariant 6: no `meta` value (a key PEM) may appear in an `event` row, a `/api/health` field, or an error `detail`; the events carry the host name + the action in fixed text.
2. `tests/test_api.py` — the signed-in harness (`tests/auth_client.py::sign_in`):
- `GET /api/certs` with zero / one / three configured hosts: the key paths pinned (the `/api/disks` pin idiom — a panel that retires can never shrink the payload);
- an unknown `<name>`: the `404` and the spies on `httpx.AsyncClient.get` and `collectors.fetch_snapshot` (the `tests/test_api.py` spy idiom) prove no network and no store write (a `meta` row count before/after);
- `POST rotate` → `bundle_rev` +1, a `cert-rotated` event row, the `meta` value changed, and the old serial is not on the new bundle's denylist;
- `POST revoke` → the status, the event, and (through task 03's path) the next `fetch_snapshot` for that host refuses without a dial (the spy);
- `POST root/rotate` → a fresh root serial, every agent reissued, `restart_required: true` in the body;
- `GET bundle` → the body byte-identical to the stored `meta` bundle, and a reissue changes the served bytes;
- the unsigned gate: the route-walk test (existing, automatic) plus one explicit unsigned `303`/`401` on `GET /api/certs` and a `POST` refusal (the `tests/test_auth_api.py` idiom).
- Coverage: **>90%** on this task's new/modified code.
## Completion Criteria
- [ ] the routes are signed-in + CSRF-gated (the wire test); the unsigned shape pinned
- [ ] an unknown name 404s with no dial and no store write (spy-pinned)
- [ ] rotate / revoke / root-rotate each leave a fixed-text event row and no key in the `event`/`health` tables (a test greps the stored events for a PEM header and finds none)
- [ ] `uv run pytest` green; `uv run ruff check src tests agent` + `uv run pyright` clean
@@ -0,0 +1,35 @@
# Task 05 — The /certs admin page: every cert, its expiry, rotate, revoke, re-deliver
**Phase:** `15_agent_mtls` · **Source:** `TODO.md:3` — "I want all exporters to be protected web servers. The collector should generate and manage a root certificate that needs to be installed with each exporter. Use the certs as auth to each exporter." (operator addition: an admin page to see generated certs and expirations, trigger rotation, and revoke certs)
## Objective
A signed-in operator sees every cert the dashboard has generated — the root CA, its own client, and one per host — with its expiry and status, can rotate / revoke / root-rotate through task 04's API, and can download the exact bundle an agent should hold. The page follows `settings.html`'s precedent exactly: a page and not a dashboard — in `PAGES` (so it is a route), not in `.tabs`, linked from `.masthead-right`, with no `aria-current` on the tabs and no status dots.
## Work
1. `src/dashboards/web/pages/certs.html` + `src/dashboards/web/static/page-certs.js` (the `page-settings.js` idiom: the same `<head>` + masthead, `Dash.fetchJSON`, errors printed through `textContent`, a `confirm()` for destructive actions — the typed `DANGER` word is reserved for power cuts):
- the **Root CA** block: the CN, the serial, the expiry (a local timestamp **and** the time left, the settings session-display idiom), and the **Rotate root CA** button → `POST /api/certs/root/rotate` with a confirm dialog; the response's `restart_required` prints "restart the dashboard to activate the new root" (the API's own sentence, not an invented one).
- the **Agents** table: one row per configured `HOST_TARGETS` host (the `GET /api/certs` `agents[]`): the name, the serial, the `status` chip (`active` → `ok`, `revoked` → `fail` — a string-to-class map, the page never compares a number), the expiry (local timestamp + time left), the **issued** `bundle_rev` against the **deployed** `bundle_rev` (`n/a` when the agent never reported — a `null` verdict, uncoloured; issued > deployed → a neutral `stale` note, uncoloured — a count is not a graded number, the fleet-header rule), and the `san` list in mono.
- per-row actions: **Rotate** (a confirm → `POST …/rotate`), **Revoke** (a confirm that says it stops the dashboard trusting that host until a reissue and a re-delivered bundle — a fixed sentence from the page, not the API), and **bundle** (a plain `<a href="/api/certs/<name>/bundle">` — a signed-in `GET` needs no CSRF).
- a client-cert line (the serial + the expiry) — the one cert the agents check. It is **display-only**: no revoke button (a revoked client cert is a dashboard that reaches no agent — a footgun the page should not make easy); its rotation happens with the root.
- ASSUMPTION: the client cert is display-only on the page (no revoke control); revoking it is deliberately not offered from the UI.
2. `src/dashboards/app.py` — `PAGES` gains `"/certs": "certs.html"` (the settings-page comment at the `PAGES` dict extends by one line of the same shape); a `Certs` link (a `⚷` glyph is acceptable) in `.masthead-right` of every page including `/settings`, next to `⚙ Account` — **outside** `.tabs` (the settings precedent, `tests/test_api.py::test_every_dashboard_page_links_to_settings` pins that structure; extend the pin, do not break it).
3. `src/dashboards/web/static/dash.css` — reuse existing vocabulary only (`.chip`, the table/`.hero`-family styles present in `settings.html`); **no** new `:root` tokens (the `tests/test_api.py` pin fails a CSS rule that reaches for an undeclared token).
4. `tests/test_api.py` — the served-page pins (the settings-pin idiom):
- the `Certs` masthead link on every dashboard page and on `/settings` (extend `test_every_dashboard_page_links_to_settings` into a two-link pin, or a sibling test);
- the served `/certs` HTML: the root block, the agents table container, the three action labels; the page's own tab structure carries **no** `aria-current` on any of the five (the settings pin, copied);
- the automatic audits cover the rest: every `id` the page keeps is referenced by `page-certs.js` (the orphan-id audit), no `Dash.severityFor` named (the existing pin), the deleted-flavour-text audit, the `no-store`/gate shape from the route walk.
5. the live verification screenshot is task 07 (invariant 10).
## Testing & Quality
- the served-HTML pins above; the JS is verified by the live screenshot (the repo's UI test policy — no JS test runner).
- Coverage: **>90%** on the modified Python (`app.py`'s `PAGES`/route code).
## Completion Criteria
- [ ] signed-in `GET /certs` serves the page; unsigned is a `303` to `/login?next=/certs` (the query included — `auth.safe_next`)
- [ ] the masthead pin green (every page links to both settings and certs)
- [ ] the orphan-id + flavour-text + `severityFor` audits green on the new page
- [ ] `uv run pytest` green; `uv run ruff check src tests agent` + `uv run pyright` clean
@@ -0,0 +1,32 @@
# Task 06 — The installer places the bundle; the unit runs TLS; --status probes over TLS
**Phase:** `15_agent_mtls` · **Source:** `TODO.md:3` — "I want all exporters to be protected web servers. The collector should generate and manage a root certificate that needs to be installed with each exporter. Use the certs as auth to each exporter."
## Objective
`deploy/install-agent.sh` can install the certificate bundle the same way it already installs the token — one file, the right owner and mode, the unit flag, a `--status` that proves the TLS path end-to-end — and `tests/test_deploy_scripts.py` moves its pins so a regression is a red test, not a dead agent.
## Work
1. `deploy/install-agent.sh` —
- a new flag `--cert-bundle <file>` (the operator already fetched it from `GET /api/certs/<name>/bundle` — task 04's surface; the script does not dial the dashboard, per its existing no-dial rule): validate the bundle before touching anything (a `python3` one-liner: three PEM blocks — server cert, key, CA cert — and a `# BUNDLE-REV` line; a bad bundle refuses with the reason, the `resolve_bundle` idiom).
- install to `/etc/dashboards-agent.certs` as `0600 root:$AGENT_USER` (the token-file idiom, with one line of the same comment explaining why the mode matters — the bundle holds the agent's key, so only root and the agent's own service user may read it); on re-run, replace it only if the flag is given (an existing bundle survives an unrelated re-run — the token-reuse idiom).
- the unit: when a bundle is installed, `ExecStart` gains `--cert-bundle /etc/dashboards-agent.certs` (the `--token-file` line stays — task 02's token check is skipped in TLS mode, so the harmless dual mode eases a rollout) and `RuntimeDirectory=dashboards-agent` (the unit's `ProtectSystem=strict` leaves the agent no writable file but this — the extracted PEM blocks live there; `PrivateTmp` already exists, so a non-systemd dev run falls back to `tempfile`).
- the `--status` probe: when the unit runs with a bundle, probe `https://127.0.0.1:$PORT/health` with a `python3 -c` `ssl` context that `load_verify_locations`-es the bundle's CA block and presents **no** client cert (task 02's `CERT_OPTIONAL` + the `/health` exemption is what makes this possible — the installer holds no client cert); a token-mode install keeps the plain `/health` probe. The probe is `python3` (the unit's own interpreter), never a new binary dependency (no `openssl` — one less thing to install on a host).
2. `deploy/agent-sudoers` — **no** new line (the bundle travels through the same root installer invocation as the token file).
3. `tests/test_deploy_scripts.py` — the new pins (the file's existing idiom: read the script, run it unprivileged so it exits before touching anything, and assert on the lines):
- `--cert-bundle` in the usage + the `/etc/dashboards-agent.certs` path with the `0600` and `root:$AGENT_USER` `chown` (assert the `install -m`/`chown` line);
- a `RuntimeDirectory=dashboards-agent` line in the unit when a bundle is installed (assert the conditional that emits it);
- the `--status` probe branch: the `python3` `ssl` one-liner for TLS, the plain probe for token mode;
- the existing pins unchanged (the sudoers-line identity, phase 10's no-tunnel name/port).
## Testing & Quality
- the `tests/test_deploy_scripts.py` pins above; the real-host smoke is task 07 (invariant 10 keeps the dev server up).
- Coverage: the script is shell — the pins are its tests (the file's stated policy: no shell test framework; a subprocess buys nothing more).
## Completion Criteria
- [ ] an unprivileged `--cert-bundle /nonexistent` run refuses with the reason and touches nothing
- [ ] the pins green: the bundle path/mode, the unit flag + `RuntimeDirectory`, the TLS `--status` probe
- [ ] `uv run pytest` green; `uv run ruff check src tests agent` + `uv run pyright` clean
@@ -0,0 +1,29 @@
# Task 07 — The fleet runs TLS: install, flip, verify, document
**Phase:** `15_agent_mtls` · **Source:** `TODO.md:3` — "I want all exporters to be protected web servers. The collector should generate and manage a root certificate that needs to be installed with each exporter. Use the certs as auth to each exporter."
## Objective
The two live agents (deskwork, gateway) present the dashboard-issued server cert and require the dashboard's client cert; the dashboard dials `https://`; the health table, the live suite, and a `/certs` screenshot prove it — and the still-missing `driveripper` install is documented to carry a bundle from day one.
## Work
1. **Fleet (operator root actions, the `deploy/agent-sudoers` grant):** for `deskwork` and `gateway`: `GET /api/certs/<name>/bundle` (a signed-in `curl`), `scp` the bundle to the host, `sudo ./deploy/install-agent.sh --cert-bundle <file>`, `systemctl restart dashboards-agent`, `./deploy/install-agent.sh --status` green over TLS. Record the exact commands + output in `data/logs/NOTES-agent-install.md` (an append to the file's existing per-host record; local evidence, not tracked, the AGENTS.md rule).
2. **The flip:** `.env.dev` — the `HOST_TARGETS` entries become `https://<same address>:9120` (no userinfo — the cert is the auth; the token goes away for a TLS host), `./scripts/dev-server.sh restart`, `./scripts/dev-server.sh status` shows `host:deskwork ok`, `host:gateway ok`, `disks:deskwork ok`, `disks:gateway ok` (the two `driveripper` jobs stay `DOWN` — unchanged, the designed degradation).
3. **Proof (invariant 10):** `uv run pytest -m live --live` — the host/disks tests over the `https` path; a `/certs` screenshot (the table showing both rows `active`, issued rev == deployed rev) **and** a `/system` screenshot (the host rows not dimmed) in `data/logs/`; a `data/logs/NOTES-verify.md`-style note: the `curl` codes (unsigned `/api/certs` refused, signed-in 200), one line for a wrong-CA client's handshake failure (the redacted URL only).
4. **The `driveripper` procedure:** an append to `data/logs/NOTES-agent-install.md` — when the agent is finally installed (the outstanding operator action, the AGENTS.md fleet section), install the bundle in the same invocation (`sudo ./deploy/install-agent.sh --bind … --cert-bundle …`) and point `HOST_TARGETS` at `https://` from day one; a token-only install after this phase is a host the `https`-dialing dashboard cannot reach.
5. **Docs (the AGENTS.md definition of done):** the AGENTS.md fleet section (the transport fact: the agent is now mTLS, the bundle file, the `/certs` page) + the commands section (the bundle-fetch line); a short "Protecting the agent" section in `README.md` (the operator's how-to: fetch the bundle, install, flip the URL — three sentences, no flavour).
## Testing & Quality
- the live suite is read-only (the repo's rule) — no bundle mutations on the hardware; the revocation/rotation proof happens on a dev-box agent, not the fleet's.
- invariant 10: the dev server stays up throughout; screenshots in `data/logs/`.
## Completion Criteria
- [ ] `./scripts/dev-server.sh status`: `host:deskwork ok`, `host:gateway ok`, `disks:deskwork ok`, `disks:gateway ok` over the `https://` dials
- [ ] `uv run pytest -m live --live` green for the host/disks paths
- [ ] the `/certs` + `/system` screenshots in `data/logs/` show `active`, issued rev == deployed rev, the host rows not dimmed
- [ ] the `data/logs/` note records the exact commands + the `curl` codes + the wrong-CA failure line
- [ ] AGENTS.md + `README.md` updated (the definition of done)
- [ ] `bash .agents/validate.sh` green
@@ -0,0 +1,42 @@
# Phase 16 — Pin the agent's scaled smart-pass budget
**Source:** chat 2026-10-19 — the `driveripper` 16-drive host: the flat 20 s smart-pass
budget truncates a 16-device pass (a real read takes ~30–80 s), and because the deadline
resets each pass, the tail drives would have stayed `smart: null` every pass. The fix —
budget = 20 s floor scaled to 5 s × disk count, an explicit override still winning — was
applied to the working tree on 2026-10-19 (operator-confirmed, verified on 16/16 live
drives). This phase pins that already-applied change with tests and documentation.
**Story:** n/a
**Context:** `agent/host_agent.py` (`_smart_pass` — the scaled `budget = max(SMARTCTL_BUDGET_SECONDS, self.smart_timeout_seconds * len(device_names)) when the default is in force, else the explicit override; the `pass budget of {budget:.0f}s exhausted` error line; the module docstring "5 s per device and a pass budget of 20 s scaled to the disk count"), `tests/test_agent.py::test_the_pass_budget_skips_the_rest_with_one_error` (sets `collector.smart_budget_seconds = 0.0` — the override-wins branch), `AGENTS.md` line 185 ("5 s per device / 20 s per pass / 240 s cache") and the `deploy/install-agent.sh` header comment (line 25 area, "bounded 5 s per device and 20 s per pass" if still present there), `README.md` line 94 ("20 s per pass, results cached 4 minutes")
## Objective
The smart pass budget scale-up that is already in the working tree is pinned: a test proves the *default* scales (16 devices → 80 s deadline, one device → 20 s floor, the exhaustion line names the scaled figure), the override-wins branch stays covered, and every doc that still says "20 s per pass" says the floor-scaled truth. No new behaviour — the code change is already deployed on `driveripper` and verified live.
## Dependencies
- `09_disks_smartctl` (complete) — the smart pass, its bounds, and `tests/test_agent.py`'s in-process agent harness
## Tasks
1. `01_pin_the_scaled_budget.md` — the tests that the default budget scales with the disk count and that the error line names the scaled figure
2. `02_docs_and_gate.md` — every "20 s per pass" mention in `AGENTS.md`/`README.md`/`agent/host_agent.py` says the floor-scaled truth; the gate is green
## Testing & Quality
- Unit: `tests/test_agent.py` — the scaled default (a default-attribute collector over 16 device names exhausts only when the 80 s budget runs out; over one device the floor is 20 s; the exhaustion error line carries the scaled figure, not the floor)
- Coverage: **>90%** on the modified code (the `_smart_pass` branch)
- The phase is "Complete" only when the **entire** project test suite passes 100% (every test in the project, not just this phase's)
## Documentation & Consistency
- The final pass reviews `README.md` and `AGENTS.md` in full against the project as it stands after this phase — fixing every inconsistency found, **including pre-existing ones not caused by this phase**.
- Tasks update `README.md`/`AGENTS.md` immediately when their own changes are user-visible (the smart-pass bounds are a documented contract of `host_agent.py`).
## Completion Criteria
- [ ] `uv run pytest tests/test_agent.py` green, including the new scaled-budget tests and the untouched override test
- [ ] `grep -rn "20 s per pass" AGENTS.md README.md agent/ deploy/` returns nothing (the wording is floor-scaled everywhere)
- [ ] `bash .agents/validate.sh` green (pytest --cov --cov-fail-under=94, ruff, pyright, node --check)
- [ ] README.md and AGENTS.md are accurate and consistent with the whole project (verified in the final pass)
- [ ] no behavior change in completed work: the deployed agent on `driveripper` is byte-identical to the repo's `agent/host_agent.py` (md5), the live suite is green
@@ -0,0 +1,26 @@
# Task 01 — Pin the scaled smart-pass budget
**Phase:** `16_agent_smart_pass_budget` · **Story:** n/a
## Objective
The default smart-pass budget (20 s floor, scaled to 5 s × disk count) is proven by tests: the deadline actually moves with the disk count, the exhaustion error line names the *scaled* figure, and the existing override-wins branch (a test sets `smart_budget_seconds = 0.0`) keeps passing untouched. The code change is already in `agent/host_agent.py::_smart_pass` — this task adds the tests, it does not rewrite the code.
## Work
1. `tests/test_agent.py` — add a helper that builds the in-process collector with **default** smart attributes (do not touch `smart_budget_seconds`/`smart_timeout_seconds`) and a `smartctl` fake that records each device's deadline window, in the same style as `test_the_pass_budget_skips_the_rest_with_one_error` (which stays untouched):
- `test_the_pass_budget_scales_with_the_disk_count`: 16 device names → the last device's `smartctl` run happens with `left` (its deadline margin) consistent with a ~80 s budget (assert the deadline the code computes: `max(20, 5 × 16)`; the fake binary is instant, so assert via the recorded `min(left, 5.0)` timeout being 5.0 for a late device and the pass *not* truncating — every device got a `smartctl` run); 1 device name → budget 20 s (assert the recorded timeout is 5.0 and, with the fake made slow enough to exhaust a 20 s but not an 80 s budget in a *separate* case if feasible — if the harness cannot fake wall time, assert the computed `budget` expression via the error-line route instead: force exhaustion with a `time.monotonic` monkeypatch offset).
- `test_the_pass_budget_error_line_names_the_scaled_figure`: force exhaustion on 16 devices (monkeypatch the clock forward past the 80 s deadline between two device runs, as the existing test's `0.0` override does for its 0 s budget) → the error line reads `smart: pass budget of 80s exhausted, N disk(s) skipped` — the scaled figure, not the 20 s floor.
2. `agent/host_agent.py` — **no code change expected.** If a test exposes a real bug in the already-applied change (e.g. the `== SMARTCTL_BUDGET_SECONDS` default check), fix it minimally and record the diff in the task's completion note; do not redesign.
## Testing & Quality
- Unit: the two new tests above + the untouched `test_the_pass_budget_skips_the_rest_with_one_error` (override wins: `0.0` still exhausts immediately on 3 devices with one error line)
- Coverage: **>90%** on `agent/host_agent.py::_smart_pass`'s budget branch (the default-scaled path must now be exercised, not just the override)
## Completion Criteria
- [ ] `uv run pytest tests/test_agent.py -q` green: the two new tests, the untouched override test, and every pre-existing agent test
- [ ] the 80 s case asserts the *scaled* figure in the error line (a test that passed with a 20 s budget would fail on the scaled code — no tautological assertions)
- [ ] `uv run ruff check agent tests` and `uv run pyright` clean
- [ ] no behavior change in completed work: `agent/host_agent.py` unchanged unless a real bug was found (then the fix is minimal and recorded)
@@ -0,0 +1,25 @@
# Task 02 — Docs and gate for the scaled budget
**Phase:** `16_agent_smart_pass_budget` · **Story:** n/a
## Objective
Every place that still says the smart pass is "20 s per pass" says the floor-scaled truth (20 s floor, 5 s × disk count), and the phase gate is green. The agent's module docstring was already updated on 2026-10-19 to the scaled wording — this task verifies it and fixes the two remaining doc lines.
## Work
1. `AGENTS.md` line ~185 (the `host_agent.py` bullet): `5 s per device / 20 s per pass / 240 s cache` → `5 s per device / a pass budget of 20 s scaled to 5 s × disk count / 240 s cache` (keep the bullet's voice; one line).
2. `README.md` line ~94: `20 s per pass, results cached 4 minutes` → `a pass budget of 20 s scaled to the disk count (5 s each), results cached 4 minutes` (one line).
3. `agent/host_agent.py` module docstring — verify it reads `bounded 5 s per device and a pass budget of 20 s scaled to the disk count, cached 240 s`; fix only if it drifted.
4. `bash .agents/validate.sh` — the four checks (pytest --cov --cov-fail-under=94, ruff, pyright, node --check) + repo hygiene, all `uv run`.
## Testing & Quality
- No new logic in this task; the gate run is the check.
- Coverage: **>90%** gate holds (task 01's tests carry the new branch).
## Completion Criteria
- [ ] `grep -rn "20 s per pass" AGENTS.md README.md agent/ deploy/` returns nothing
- [ ] `bash .agents/validate.sh` green
- [ ] the two edited doc lines read the floor-scaled truth and nothing else changed in either file
@@ -0,0 +1,59 @@
# Phase 17 — The installer's broken device filter and the kernel-gate variant
**Source:** chat 2026-10-19 — the `driveripper` install: the template's
`DevicePolicy=strict` + `DeviceAllow` lines produced a **deny-all device filter** on
kernel 7.2 (systemd resolves `DeviceAllow` globs by name against `/proc/devices`, where
this kernel lists NVMe as `blkext` (major 259), so `nvme*` matched nothing and the whole
device namespace was locked). The operator removed the lines by hand; the unit now also
runs `User=root` + `CapabilityBoundingSet=CAP_SYS_RAWIO CAP_SYS_ADMIN` (kernel 7.2's
unprivileged-passthrough gate — `data/logs/NOTES-driveripper-agent.md`). The template
still ships the broken lines, `User=$AGENT_USER`, and an empty bounding set, and re-running
the installer on `driveripper` would **clobber the working unit** (unconditional
`cat > "$UNIT"`). This phase makes the template the single source of the working unit.
**Story:** n/a
**Context:** `deploy/install-agent.sh` (`do_install` — the unconditional unit heredoc with `DevicePolicy=strict` + nine `DeviceAllow` lines, the `PrivateDevices=false` rationale comment block, `User=$AGENT_USER`, `CapabilityBoundingSet=`, the `--bind/--port/--token/--rotate-token/--status/--uninstall` flag set), `tests/test_deploy_scripts.py` (the fake-root install sandbox `test_install_path_installs_into_its_own_paths_and_prints_the_token_once`, the token-file permission test, the `bash -n` + function-naming pins), `AGENTS.md` (the fleet bullet "The deployed unit has since diverged from the template in the three ways the SMART block below records", gap 10's "the template change is its own phase"), `README.md` (the install section, line ~46 `sudo deploy/install-agent.sh --bind <addr> --port 9120`)
## Objective
Re-running `deploy/install-agent.sh` on any host — including `driveripper`, with
`--as-root --caps CAP_SYS_RAWIO,CAP_SYS_ADMIN --force` — produces the working unit: no
`DevicePolicy`/`DeviceAllow` lines (device access falls back to DAC via the `disk` group
+ SELinux, the standard SMART-monitoring arrangement), the kernel-gate variant expressible
by flags, and a re-run that would change an existing unit prints the diff and refuses
without `--force`. The default install (dedicated user, empty bounding set) is otherwise
byte-identical to what the template wrote before this phase.
## Dependencies
- `09_disks_smartctl` (complete) — the smart pass the unit exists to serve
- `15_agent_mtls` (todo) — **shared-file, run-order note**: 15's task 06 adds `--cert-bundle`/`RuntimeDirectory` to the same `do_install`; apply on top of the file state at the time it runs, never restore a line the other phase owns. If 15 lands first, its new lines survive this phase's filter-line removal untouched.
## Tasks
1. `01_drop_the_broken_device_filter.md` — remove `DevicePolicy=strict` + all nine `DeviceAllow` lines and rewrite the rationale comment block
2. `02_root_and_caps_flags.md` — `--as-root` and `--caps CAP1,CAP2` write the kernel-gate unit lines
3. `03_no_silent_clobber.md` — a re-run that would change an existing unit prints the diff and refuses without `--force`
4. `04_tests_docs_gate.md` — the unit pins in `tests/test_deploy_scripts.py`, `AGENTS.md`/`README.md` updated, gate green
## Testing & Quality
- Integration: `tests/test_deploy_scripts.py` — the template unit carries **no** `DevicePolicy`/`DeviceAllow` line; a default-install sandbox unit is byte-identical to the pre-phase template unit minus the filter lines; `--as-root --caps CAP_SYS_RAWIO,CAP_SYS_ADMIN` writes `User=root` and `CapabilityBoundingSet=CAP_SYS_RAWIO CAP_SYS_ADMIN` exactly; the re-run guard refuses (exit 1, diff on stdout) and overwrites with `--force`; `bash -n` and the existing function-naming pin still pass
- Live (task 04, operator-verified): `driveripper`'s deployed unit is reproduced by `sudo deploy/install-agent.sh --bind 10.1.0.126 --port 9120 --as-root --caps CAP_SYS_RAWIO,CAP_SYS_ADMIN --force` — `systemctl diff` empty, `CapEff` still `0000000000220000`, 16/16 drives still live; `deskwork`/`gateway` untouched (their units are not re-run)
- Coverage: the script itself is pinned by text assertions + `bash -n` (house pattern; >90% applies to the new Python test code)
- The phase is "Complete" only when the **entire** project test suite passes 100% (every test in the project, not just this phase's)
## Documentation & Consistency
- The final pass reviews `README.md` and `AGENTS.md` in full against the project as it stands after this phase — fixing every inconsistency found, **including pre-existing ones not caused by this phase**.
- Tasks update `README.md`/`AGENTS.md` immediately when their own changes are user-visible (the new flags are operator-facing; gap 10's "the template change is its own phase" becomes "done in phase 17" with the `driveripper` flag line).
## Completion Criteria
- [ ] `grep -n "DevicePolicy\|DeviceAllow" deploy/install-agent.sh` returns nothing (the `PrivateDevices=false` line stays)
- [ ] `sudo deploy/install-agent.sh --help` names `--as-root` and `--caps` with the kernel-gate sentence
- [ ] a fresh sandbox install (fake root, no flags) produces a unit byte-identical to the pre-phase template unit minus the filter lines (pinned in `tests/test_deploy_scripts.py`)
- [ ] a re-run that would change an existing unit exits 1 with the diff on stdout; with `--force` it overwrites
- [ ] operator-verified on `driveripper`: the flags reproduce the working unit (`systemctl diff` empty, 16/16 drives live after restart)
- [ ] `uv run pytest` green, `bash .agents/validate.sh` green
- [ ] README.md and AGENTS.md are accurate and consistent with the whole project (verified in the final pass)
- [ ] no behavior change in completed work: the default install's unit (minus the filter lines) and every existing flag behave as before; the live suite is green
@@ -0,0 +1,25 @@
# Task 01 — Drop the broken device filter
**Phase:** `17_installer_device_filter` · **Story:** n/a
## Objective
The template unit no longer carries `DevicePolicy=strict` or any `DeviceAllow` line, and the comment block that justified them is rewritten to say why the *absence* is the fix. The `PrivateDevices=false` line stays (explicit, default behaviour, and the rest of the lockdown is unchanged).
## Work
1. `deploy/install-agent.sh` — in the `do_install` unit heredoc: delete the `DevicePolicy=strict` line and all nine `DeviceAllow=…` lines (`/dev/null`, `/dev/zero`, `/dev/full`, `/dev/random`, `/dev/urandom`, `/dev/nvme*`, `/dev/sd*`, `/dev/vd*`, `/dev/sg*`).
2. `deploy/install-agent.sh` — rewrite the comment block above `PrivateDevices=false` (the "The pseudo nodes are not decoration…" paragraph) to the new truth: **no device filter on purpose** — systemd resolves `DeviceAllow` globs by name against `/proc/devices` (fnmatch in `bpf_devices_allow_list_major`), where kernel 7.2 lists NVMe as `blkext` (major 259), so `nvme*` matched nothing and `DevicePolicy=strict` denied the *whole* device namespace (the `driveripper` install went deny-all, root included; `data/logs/NOTES-driveripper-agent.md`). Without the policy, device access falls back to DAC (the `disk` group the script already grants) + SELinux — the standard SMART-monitoring arrangement. Keep the one operational fact that survives: `PrivateDevices=false` is the default (a `true` sandbox with no `DeviceAllow` list is what dies at spawn with `status=208/STDIN` — keep that warning sentence, it is still why the line is written explicitly).
3. `deploy/install-agent.sh` — the script's top-of-file header comment (line ~25: "the unit device-allows the disk nodes") → "the unit applies no device filter (kernel 7.2's `/proc/devices` name matching made the old `nvme*` glob deny-all); device access is DAC via the `disk` group + SELinux".
## Testing & Quality
- Integration: `bash -n deploy/install-agent.sh` clean (the existing pin covers it); a quick sandbox run (the `test_install_path_installs_into_its_own_paths_and_prints_the_token_once` pattern, run manually) shows no `DevicePolicy`/`DeviceAllow` in the written unit — the formal pins are task 04, this task keeps the change small and reviewable.
- Coverage: **>90%** n/a (shell script; the pins are text assertions per the house pattern).
## Completion Criteria
- [ ] `grep -n "DevicePolicy\|DeviceAllow" deploy/install-agent.sh` returns nothing
- [ ] `bash -n deploy/install-agent.sh` clean
- [ ] a manual sandbox install (fake root, `--bind 127.0.0.1 --port 9199`) writes a unit with the filter lines gone, `PrivateDevices=false` present, and every other line byte-identical to the pre-phase template unit
- [ ] the rewritten comment names the `/proc/devices`/`blkext` mechanism in one sentence (an executor reading only the script can understand *why* there is no filter)
@@ -0,0 +1,27 @@
# Task 02 — `--as-root` and `--caps` flags
**Phase:** `17_installer_device_filter` · **Story:** n/a
## Objective
The kernel-gate variant of the unit (what `driveripper` runs) is expressible from the template: `--as-root` writes `User=root` instead of `User=$AGENT_USER`, and `--caps CAP1,CAP2` (comma-separated, e.g. `--caps CAP_SYS_RAWIO,CAP_SYS_ADMIN`) writes `CapabilityBoundingSet=CAP1 CAP2` instead of the empty `CapabilityBoundingSet=` line. Both default off; a flag without its friend is refused with a message.
## Work
1. `deploy/install-agent.sh` — flag parsing: add `--as-root` (sets `AS_ROOT=1`) and `--caps <list>` (sets `CAPS="<list>"`, validates non-empty, rejects an empty list with exit 1) in the existing `case`/flag block, and `--help` text:
- `--as-root run the service as root (for a kernel-gated host: kernel 7.2 gates unprivileged smartctl passthrough behind capabilities, not uid — a root process under the default empty bounding set is capability-less, so this flag needs --caps to mean anything)`
- `--caps LIST CapabilityBoundingSet line for the unit (comma-separated, e.g. CAP_SYS_RAWIO,CAP_SYS_ADMIN); default: empty (capability-less service)`
2. `deploy/install-agent.sh` — validation: `--as-root` without `--caps` → print "note: --as-root with the default empty CapabilityBoundingSet leaves the process capability-less (the kernel gates check capabilities, not uid); pair it with --caps for a kernel-gated host" and **still proceed** (a capability-less root is a legitimate, if unusual, choice — do not hard-refuse; the note is the operator's reminder). `--caps` without `--as-root` → refuse, exit 1 ("--caps needs --as-root: a capability grant on the service user is not the kernel-gate fix").
3. `deploy/install-agent.sh` — the unit heredoc: `User=$AGENT_USER` → `User=$( [ "$AS_ROOT" -eq 1 ] && echo root || echo "$AGENT_USER" )`; the empty `CapabilityBoundingSet=` line → the `CAPS` list converted comma→space when set. Add one comment line above: `# --caps: kernel 7.2 gates unprivileged smartctl passthrough (NVMe admin log: CAP_SYS_ADMIN; SG_IO allow-list: CAP_SYS_RAWIO); a root process needs them named, because an empty bounding set bounds even root to CapEff 0`.
4. `deploy/install-agent.sh` — the `--status` probe is unchanged (it probes the agent over HTTP; the user the unit runs as is invisible to it). The token file stays `0640 root:$AGENT_USER` — with `--as-root` the group read is irrelevant (root reads it), so do **not** change the token-file logic.
## Testing & Quality
- Integration: manual sandbox runs (fake root): default → `User=dashboards-agent` + empty `CapabilityBoundingSet=`; `--as-root --caps CAP_SYS_RAWIO,CAP_SYS_ADMIN` → `User=root` + `CapabilityBoundingSet=CAP_SYS_RAWIO CAP_SYS_ADMIN`; `--caps` alone → exit 1 with the message; `--as-root` alone → proceeds with the note on stdout. (Formal pins: task 04.)
- Coverage: **>90%** n/a (shell script; text pins per the house pattern).
## Completion Criteria
- [ ] `sudo deploy/install-agent.sh --help` names both flags with the kernel-gate sentence (a fresh clone's executor can learn the `driveripper` recipe from `--help` alone)
- [ ] the two sandbox units match the exact lines above (byte-for-byte on the `User=` and `CapabilityBoundingSet=` lines)
- [ ] `bash -n deploy/install-agent.sh` clean; every pre-existing flag (`--bind/--port/--token/--rotate-token/--status/--uninstall`) behaves as before in a sandbox run
@@ -0,0 +1,27 @@
# Task 03 — No silent clobber on re-run
**Phase:** `17_installer_device_filter` · **Story:** n/a
## Objective
A re-run that would change an already-installed unit stops: it prints the diff between the installed unit and the unit the script would write, and exits 1 without touching the file. `--force` proceeds with the overwrite. A re-run whose generated unit is identical to the installed one proceeds silently (idempotent re-runs stay the zero-noise path the script has always had).
## Work
1. `deploy/install-agent.sh` — add a `--force` flag to the flag block (`FORCE=1`) and `--help` text: `--force overwrite an existing unit even if the generated unit differs (prints the diff first)`.
2. `deploy/install-agent.sh` — in `do_install`, immediately before the `cat > "$UNIT"` heredoc write: build the would-be unit into a temp file (the heredoc already exists — render it once to `$(mktemp)`), and if `"$UNIT"` exists and `! cmp -s "$UNIT" "$tmp_unit"`:
- if `FORCE=1`: print the diff (`diff -u "$UNIT" "$tmp_unit"` on stdout) and proceed with the overwrite;
- else: print `refusing to overwrite $UNIT (generated unit differs); run with --force to apply:` + the same diff, and `exit 1` **before** the token-file write, the `systemctl daemon-reload`, and the `systemctl enable --now` — a refused run must leave the host exactly as it was (token file, unit, and running service all untouched; note the token-reuse logic runs earlier in `do_install` and only *reads* the existing token, so ordering the guard before the unit write is enough).
- identical unit → no output, normal proceed.
3. `deploy/install-agent.sh` — `--uninstall` is unchanged (it already removes the unit deliberately; the guard applies to `do_install` only).
## Testing & Quality
- Integration: manual sandbox sequence (fake root): install → modify the installed unit by one line → re-run with the same flags → exit 1, diff on stdout, installed unit byte-identical to the modified version, service still the old one; re-run with `--force` → diff printed, unit overwritten; re-run again unchanged → silent, exit 0. (Formal pins: task 04.)
- Coverage: **>90%** n/a (shell script; text/behaviour pins per the house pattern).
## Completion Criteria
- [ ] the three-way sequence above (refuse → force → silent) holds in the sandbox, with the token file and the running service untouched by a refused run
- [ ] `--force` prints the diff *before* overwriting (an operator sees exactly what changed)
- [ ] `bash -n deploy/install-agent.sh` clean; a fresh install (no existing unit) is unaffected by the guard
@@ -0,0 +1,34 @@
# Task 04 — Pins, docs, gate
**Phase:** `17_installer_device_filter` · **Story:** n/a
## Objective
The template's new shape is pinned in `tests/test_deploy_scripts.py` (no filter lines, the flags' exact lines, the clobber guard's refuse/force/silent sequence), `AGENTS.md` and `README.md` name the `driveripper` recipe, and the gate is green. The operator-verified `driveripper` reproduction (the completion criterion in `00_phase.md`) is the last check.
## Work
1. `tests/test_deploy_scripts.py` — extend the existing sandbox pattern:
- `test_the_template_unit_has_no_device_filter_lines` — read `deploy/install-agent.sh`'s unit heredoc region and assert no line starts with `DevicePolicy=` or `DeviceAllow=`, and `PrivateDevices=false` is still present (the filter's absence is the fix; a regression that re-adds `DeviceAllow=/dev/nvme*` is the deny-all returning).
- `test_a_default_install_unit_is_the_pre_phase_unit_minus_the_filter_lines` — the sandbox-installed unit equals the known pre-phase template unit text minus the ten filter lines (pin the whole unit, so a future template line drift fails here).
- `test_as_root_and_caps_write_the_kernel_gate_unit_lines` — `--as-root --caps CAP_SYS_RAWIO,CAP_SYS_ADMIN` → `User=root` and `CapabilityBoundingSet=CAP_SYS_RAWIO CAP_SYS_ADMIN` exactly; `--caps` alone → exit 1 with the `--as-root` message; `--as-root` alone → proceeds with the capability-less note on stdout.
- `test_a_re_run_that_differs_refuses_without_force_and_applies_with_it` — the refuse/force/silent sequence (exit 1 + diff on stdout + installed unit untouched; `--force` overwrites with the diff printed; an unchanged re-run is silent).
2. `AGENTS.md` — the fleet bullet's "re-running `deploy/install-agent.sh` on driveripper would clobber the working unit, because it rewrites the unit unconditionally (gap 10)" → the post-phase truth: the installer no longer writes the device-filter lines, and the `driveripper` unit is reproducible with `sudo deploy/install-agent.sh --bind 10.1.0.126 --port 9120 --as-root --caps CAP_SYS_RAWIO,CAP_SYS_ADMIN --force` (a re-run that would change an existing unit refuses without `--force`). Update gap 10's "the template change is its own phase" → "done in phase 17" with the flag line.
3. `README.md` — the install section: one line naming the flags' purpose (the kernel-gated-host recipe), keeping the how-to voice.
4. `bash .agents/validate.sh` — all four checks green.
5. **Operator-verified (live, read-only + one restart):** on `driveripper`, `sudo deploy/install-agent.sh --bind 10.1.0.126 --port 9120 --as-root --caps CAP_SYS_RAWIO,CAP_SYS_ADMIN --force` → `systemctl diff dashboards-agent` empty, `systemctl is-active` `active`, `CapEff` still `0000000000220000`, and 16/16 drives still live in the next `/snapshot` (the restart is the only fleet action; `deskwork`/`gateway` are not re-run). Screenshot `data/logs/disks-17-installer-repro.png` per invariant 10.
## Testing & Quality
- Integration: the four new pins above + the pre-existing pins (sudoers, `bash -n`, function naming, token-file permissions) all green.
- Coverage: **>90%** on the new test code; the 94% gate holds.
- Live: the operator-verified reproduction (item 5) — the phase's `driveripper` criterion.
## Completion Criteria
- [ ] the four new pins pass and every pre-existing `test_deploy_scripts.py` pin passes
- [ ] `grep -n "DevicePolicy\|DeviceAllow" deploy/install-agent.sh` returns nothing
- [ ] `AGENTS.md` fleet bullet + gap 10 and `README.md` name the `driveripper` flag recipe
- [ ] the operator-verified reproduction: `systemctl diff` empty, 16/16 drives live, screenshot in `data/logs/`
- [ ] `uv run pytest` green, `bash .agents/validate.sh` green
- [ ] the live suite (`set -a; . ./.env.dev; set +a; uv run pytest -m live --live`) is green
@@ -0,0 +1,36 @@
# Phase 18 — Collapse the nav bar into a hamburger
**Source:** `TODO.md` L3 — "The nav bar needs to collapse into a hamburger like brain of reese does"
**Story:** n/a
**Context:** the seven mastheads that carry the tab nav — `src/dashboards/web/pages/{index,system,disks,llm,ups,settings,certs}.html` (each has `nav class="tabs" aria-label="Dashboards"` with the five dashboard links; `login.html` deliberately has no tabs — it renders only the brand link and the theme toggle, and stays untouched), `src/dashboards/web/static/app.js` (`Dash.initChrome()`, called from the one `DOMContentLoaded` at line 786 — the `themeButton`/`clock`/`nav-status` blocks there are the per-element `if (element)` no-op pattern this phase's binding follows), `src/dashboards/web/static/dash.css` (`.masthead` at line 95 — `position: sticky; z-index: 40; height: var(--header-h)`, `nav.tabs` at line 131, the existing `@media (max-width: 640px)` block at line 286, the `prefers-reduced-motion` block at line 938), and the reference implementation the TODO names — `brain_of_reese` (sibling repo, read-only): `frontend/assets/header.js` lines 326–366 (the phase-46 binding: `#nav-toggle` shown ≤640px, `setNavMenu(open)` funnel, close on link click / Esc with focus return / `matchMedia` resize-back). Pinned test neighbours: `tests/test_api.py::test_the_disks_tab_is_in_the_masthead_of_every_page` (line 3837) and the served-page orphan-`id` audit in `tests/test_auth_api.py` (line 1654)
## Objective
On viewports ≤640px the masthead's five tab links collapse into a dropdown opened by a hamburger button, mirroring brain of reese's phase-46 behaviour: one shared binding in `app.js` (a no-op on any page without the elements — `/login`), the inline nav hidden and the dropdown absolutely positioned under the sticky masthead, and the desktop masthead byte-identical in behaviour above the breakpoint. The `masthead-right` cluster stays visible on phones; only the tab nav collapses.
## Dependencies
- — (none; builds on the shipped masthead, `initChrome`, and the 640px media block)
## Tasks
1. `01_hamburger_markup.md` — the `#nav-toggle` button and the `nav` id, identical across the seven tabbed pages, absent from `/login`
2. `02_dropdown_css.md` — the ≤640px block: toggle visible, inline tabs hidden, `.is-open` dropdown under the masthead, two-row masthead
3. `03_toggle_binding.md` — the `initChrome` binding: toggle + `aria-expanded`, close on link click / Esc / resize-back
4. `04_tests_and_verification.md` — the pins (present on all seven, absent on `/login`, CSS and binding), the orphan-`id` audit, and the narrow-viewport browser verification with screenshots
## Testing & Quality
- Unit/integration: `tests/test_api.py` — the served-page pins (see task 04); the existing masthead/tab pins (`test_the_disks_tab_is_in_the_masthead_of_every_page`, `test_every_page_links_to_both_admin_pages`) and the `tests/test_auth_api.py` orphan-`id` audit pass untouched — the new `id`s are referenced by `aria-controls`, the stylesheet, and `app.js`, so the audit must still clear
- Coverage: **>90%** on new/modified code (the phase is markup + CSS + one small JS binding; the coverage gate applies to the changed Python test files)
- The gate (`bash .agents/validate.sh`) includes `node --check` of `web/static` — the modified `app.js` must pass it
## Documentation & Consistency
- The final pass reviews `README.md` and `AGENTS.md` in full against the project as it stands after this phase — fixing every inconsistency found, **including pre-existing ones not caused by this phase**.
- AGENTS.md's `UI conventions` section gains the masthead-collapsing convention (the 640px breakpoint, the toggle's identity, the close paths, and that `/login` carries no toggle) so a future masthead edit knows the contract. The README's page table is unchanged — the nav is the same five links at every width.
## Completion Criteria
- [ ] at ≤640px: the tab links are hidden, the hamburger is visible, and clicking it opens a dropdown containing the five links; above 640px the desktop masthead is unchanged
- [ ] the dropdown closes on link click, on Esc (focus returns to the button), and on resize back above 640px; `aria-expanded` tracks the state
- [ ] `uv run pytest` green, coverage not down, `ruff` + `pyright` clean
- [ ] `bash .agents/validate.sh` green (including `node --check` of the modified `app.js`)
- [ ] narrow-viewport screenshots (closed, open, and one desktop no-regression shot) in `data/logs/` against the running dev server (invariant 10)
- [ ] no behavior change in completed work: the desktop masthead and every page render identically at widths >640px
@@ -0,0 +1,31 @@
# Task 01 — The hamburger button markup
**Phase:** `18_nav_hamburger` · **Source:** `TODO.md:3` — "The nav bar needs to collapse into a hamburger like brain of reese does"
**Story:** n/a
## Objective
The seven pages that carry the tab nav each gain the hamburger button and the `nav` id, so the (later task's) CSS and binding have stable anchors to work on. The markup is identical on every page, and `/login` — which deliberately renders only the brand link and the theme toggle — stays untouched.
## Work
1. `src/dashboards/web/pages/index.html` · `system.html` · `disks.html` · `llm.html` · `ups.html` · `settings.html` · `certs.html` — in each masthead, immediately before `nav class="tabs"`, add the toggle:
```html
<button class="ghost nav-toggle" id="nav-toggle" type="button"
aria-expanded="false" aria-controls="nav-tabs" aria-label="Toggle navigation">☰</button>
```
and give the nav its id: `<nav class="tabs" id="nav-tabs" aria-label="Dashboards">`. The button sits between `.brand` and `nav.tabs`, inside the same `<header class="masthead">` — one element per page, no script changes in this task.
2. `src/dashboards/web/static/dash.css` — add the one base rule beside the other `button.ghost, a.ghost` family (the masthead section): `.nav-toggle { display: none; }` — the button is invisible on every width until task 02's ≤640px block shows it, so the project stays exactly as it looks today after this task.
3. `src/dashboards/web/pages/login.html` — **no change.** Verify by `grep -c 'id="nav-toggle"'` returning 0 (the pin lands in task 04).
4. `ASSUMPTION:` the TODO named brain of reese's behaviour, not its breakpoint or glyph. This task takes the 640px breakpoint (brain's `(max-width: 640px)` and `dash.css`'s existing 640px block) and the `☰` glyph with `aria-label="Toggle navigation"`; the CSS task owns the visual.
5. `ASSUMPTION:` only the tab nav collapses on phones. The `masthead-right` cluster (`#nav-status` dots, pause, `⚷ Certs`, `⚙ Account`, theme toggle) stays visible; the masthead wraps to two rows at ≤640px (task 02).
## Testing & Quality
- Integration: the served pages still pass the untouched masthead pins (`test_the_disks_tab_is_in_the_masthead_of_every_page`, `test_every_page_links_to_both_admin_pages`) and the orphan-`id` audit — `#nav-toggle` is referenced by the (task 02) stylesheet rule and `aria-controls`, so the audit must not flag it
- Coverage: **>90%** on modified code (HTML only — the gate's `node --check` and pytest suite are the checks)
## Completion Criteria
- [ ] `grep -l 'id="nav-toggle"' src/dashboards/web/pages/*.html` lists exactly the seven tabbed pages, never `login.html`
- [ ] every one of the seven carries the identical button block and the `nav` id (diff of the blocks across pages is empty)
- [ ] `uv run pytest -q` green; `bash .agents/validate.sh` green
- [ ] no behavior change: at every width the pages render exactly as before after this task (the button is `display:none` from the base rule shipped with it)
@@ -0,0 +1,29 @@
# Task 02 — The ≤640px dropdown CSS
**Phase:** `18_nav_hamburger` · **Source:** `TODO.md:3` — "The nav bar needs to collapse into a hamburger like brain of reese does"
**Story:** n/a
## Objective
The CSS that makes the hamburger real at ≤640px: the toggle button visible, the inline tab nav hidden, the `.is-open` nav rendered as a dropdown under the sticky masthead, and the masthead wrapping to two rows so the `masthead-right` cluster does not overflow a phone. Above 640px nothing changes.
## Work
1. `src/dashboards/web/static/dash.css` — the base `.nav-toggle { display: none; }` rule already landed with task 01's markup (the button is invisible on every width until this task's block). Extend the existing `@media (max-width: 640px)` block at line 286 (or add a sibling block — the file's convention is one block per concern; a `/* mobile nav */`-commented block beside the masthead section is the cleaner home):
- `.masthead { height: auto; min-height: var(--header-h); flex-wrap: wrap; }` — the single-row fixed-height masthead cannot hold brand + toggle + the whole `masthead-right` cluster at 360px; two rows, first row brand + toggle, second row the cluster.
- `.nav-toggle { display: inline-flex; align-items: center; }` (it is a `button.ghost`, so the outlined style is inherited — no second palette, the file's rule).
- `nav.tabs { display: none; }` and `nav.tabs.is-open { display: flex; flex-direction: column; gap: 0; position: absolute; top: 100%; left: 0; right: 0; background: var(--bg-raised); border-bottom: 1px solid var(--border); box-shadow: 0 8px 24px rgba(0, 0, 0, .35); padding: .4rem; z-index: 41; }` — the dropdown. The masthead is `position: sticky`, which is a non-static containing block for its absolute descendants, so `top: 100%` pins it directly under the (wrapped, two-row) masthead. `z-index: 41` is one above the masthead's 40 so the dropdown sits on the page content.
- `nav.tabs.is-open a { padding: .6rem .7rem; border-radius: var(--radius-sm); }` — touch-sized rows; the existing `:hover`/`[aria-current="page"]` rules carry over unchanged.
- Optional at most: a 120ms opacity/translateY transition on `nav.tabs.is-open`, **disabled inside the existing `@media (prefers-reduced-motion: reduce)` block at line 938** (add the dropdown selector to that block's list of motion-killers if used). If in doubt, ship without the transition.
2. `ASSUMPTION:` the dropdown is a single full-width column under the masthead (brain's shape, adapted: brain hides the nav in a `#app-nav` container; here the `nav` element itself is the container and gets `id="nav-tabs"` + `.is-open`). No separate menu panel element.
3. `ASSUMPTION:` no `body`-level marker class (brain's `nav-menu-open` exists for its sticky bottom chat cluster, which the dashboards do not have — the dropdown competes with nothing; do not copy a side effect nobody needs).
## Testing & Quality
- Integration: the served-stylesheet pins land in task 04 (base rule present, ≤640px block contains the toggle-visible + tabs-hidden + `.is-open` rules); the untouched `node --check` gate passes (CSS is not checked, but confirm no syntax error by loading the page in the browser — task 04's verification)
- Coverage: **>90%** on modified code (CSS only; the gate's pytest/ruff/pyright must stay green)
## Completion Criteria
- [ ] at a 360px-wide browser window: the tab links are gone from the row, the `☰` button is visible, and toggling `.is-open` (by hand in devtools) shows the full-width dropdown under the masthead with the five links stacked and the `aria-current` tab highlighted
- [ ] at 700px and 1200px the masthead is byte-identical in layout to before (one row, inline tabs, no button)
- [ ] the dropdown does not trap or cover the `masthead-right` cluster, and the page content scrolls under it
- [ ] `bash .agents/validate.sh` green
@@ -0,0 +1,55 @@
# Task 03 — The toggle binding in `initChrome`
**Phase:** `18_nav_hamburger` · **Source:** `TODO.md:3` — "The nav bar needs to collapse into a hamburger like brain of reese does"
**Story:** n/a
## Objective
One shared binding in `app.js` (the file every page loads, including `/login`) that makes the button work on all seven tabbed pages and is a no-op where the elements are absent. The behaviour mirrors brain of reese's phase-46 binding: a `setNavMenu(open)` funnel so no close path can leave the state stuck, `aria-expanded` honest at all times, close on link click, on Esc (focus returns to the button), and on resize back above the breakpoint.
## Work
1. `src/dashboards/web/static/app.js` — inside `initChrome()` (line 620), after the `nav-status` block and before the `return`, add a block in the file's per-element guard style (the `themeButton` pattern at line 626 is the shape: `if (element) { … }`):
```js
const navToggle = $('#nav-toggle');
const appNav = $('#nav-tabs');
if (navToggle && appNav) {
const setNavMenu = (open) => {
appNav.classList.toggle('is-open', open);
navToggle.setAttribute('aria-expanded', open ? 'true' : 'false');
};
navToggle.addEventListener('click', () =>
setNavMenu(!appNav.classList.contains('is-open')));
// A link click navigates — shut the menu (event-delegated: the five
// links are static markup, so one listener on the nav covers all of them).
appNav.addEventListener('click', (event) => {
if (event.target.closest('a')) setNavMenu(false);
});
// Esc closes while open; focus returns to the opener.
document.addEventListener('keydown', (event) => {
if (event.key === 'Escape' && appNav.classList.contains('is-open')) {
setNavMenu(false);
navToggle.focus();
}
});
// Resize back to desktop: the inline nav reappears — drop the open state
// so aria-expanded stays honest (the CSS hides the dropdown above 640px
// regardless, but a stuck `is-open` would re-open on the next shrink).
const mq = window.matchMedia('(max-width: 640px)');
const onMqChange = () => { if (!mq.matches) setNavMenu(false); };
if (mq.addEventListener) mq.addEventListener('change', onMqChange);
}
```
Notes: the breakpoint literal must match task 02's CSS block — if one drifts, the menu opens into a display:none nav. `ASSUMPTION:` that match is pinned by task 04's served-asset test (assert the `640px` literal appears in both the served stylesheet's nav block and the served `app.js`), not by a shared constant — the file's convention is per-file literals (the 640px in CSS, 900px for the clock, and the page scripts' own numbers do not share a constant either).
2. No `body`-level marker class (task 02's assumption carries: nothing on this app competes with the dropdown for taps, so brain's `nav-menu-open` marker is not copied).
3. No change to the login page: `initChrome` already ships to it, and the `if (navToggle && appNav)` guard is the no-op — verify in task 04 that `/login`'s served script is the same shared file (it is: every page loads `/static/app.js`) and the guard makes the binding inert there.
## Testing & Quality
- Integration: the binding's behaviour is verified in the browser (task 04 — the repo has no JS test runner; the pins are structural on the served bytes: the block is present in the served `app.js`, the four close paths named in a comment, `node --check` passes); the untouched Python suite stays green
- Coverage: **>90%** on modified code (one JS function; the Python coverage gate applies to the test files task 04 adds)
## Completion Criteria
- [ ] at 360px: clicking `☰` opens the dropdown and `aria-expanded` reads `true`; clicking again closes it; clicking a tab link closes it as the page navigates
- [ ] with the menu open: Esc closes it and focus is on the `☰` button; resizing to 700px closes it and the next shrink back to 360px leaves it closed (no stale `is-open`)
- [ ] on `/login` the script loads and the guard no-ops (no console error, `node --check` clean)
- [ ] `uv run pytest -q` green; `bash .agents/validate.sh` green (includes `node --check` of the modified `app.js`)
@@ -0,0 +1,32 @@
# Task 04 — Pins and narrow-viewport verification
**Phase:** `18_nav_hamburger` · **Source:** `TODO.md:3` — "The nav bar needs to collapse into a hamburger like brain of reese does"
**Story:** n/a
## Objective
The phase is pinned on the served bytes the way this repo pins every page surface: the button is on all seven tabbed pages and **absent** from `/login`, the CSS and the binding carry the matching 640px literal, and the orphan-`id` audit still clears. Then the designed degradation is proven in a real browser at a phone width — the invariant-10 screenshot pass.
## Work
1. `tests/test_api.py` — new tests, in the style of `test_the_disks_tab_is_in_the_masthead_of_every_page` (line 3837; it iterates the served pages and checks the masthead):
- `test_the_nav_toggle_is_in_the_masthead_of_every_tabbed_page`: for each of the seven served pages (`/`, `/system`, `/disks`, `/llm`, `/ups`, `/settings`, `/certs`) — a `GET` (signed, like the other served-page pins) finds `button#nav-toggle` with `aria-expanded="false"`, `aria-controls="nav-tabs"`, `aria-label="Toggle navigation"`, and `type="button"`, and `nav#nav-tabs` with its five tab links intact; the button precedes the nav in the markup (the dropdown's `top: 100%` anchor needs the nav inside the masthead, which the pin asserts via the `header.masthead` containing both).
- `test_the_login_page_carries_no_nav_and_no_toggle`: `/login` (fetched the way the audit fetches it — unsigned, because a signed-in `GET /login` is a `303`) has no `nav.tabs`, no `#nav-toggle`, and no `#nav-tabs` — the page's contract is brand + theme toggle only, and a stray hamburger there is the bug this test exists to fail.
- `test_the_nav_toggle_css_and_binding_carry_the_same_breakpoint`: the served `dash.css` contains the base `.nav-toggle` `display: none` rule, a `@media (max-width: 640px)` (or the 640px block) with the toggle-visible, `nav.tabs`-hidden, and `.is-open` rules; the served `app.js` contains the binding block (the `is-open` toggle, the `Escape` close, the `matchMedia('(max-width: 640px)')` call) — and the `640px` literal appears in both, so a drift between CSS and JS fails here instead of opening a menu into a `display:none` nav.
2. `tests/test_auth_api.py` — **no change expected**: the served-page orphan-`id` audit (line 1654) must pass untouched, because `#nav-toggle` and `#nav-tabs` are referenced (the stylesheet rule, `aria-controls`, the `app.js` selectors). If the audit flags either id, the fix is in the CSS/JS references, not in the audit.
3. Browser verification (invariant 10): `./scripts/dev-server.sh restart`, sign in with `testpass`, and with a ≤640px viewport (Playwright or a devtools device width — the repo's screenshots land in `data/logs/`):
- `data/logs/nav-hamburger-closed-360.png` — the phone-width masthead: no tab links, the `☰` visible, the `masthead-right` cluster intact.
- `data/logs/nav-hamburger-open-360.png` — the dropdown open: five stacked links, the `aria-current` tab highlighted, the cluster not covered.
- `data/logs/nav-masthead-desktop-1280.png` — the no-regression shot: one-row masthead, inline tabs, no button.
4. `AGENTS.md` — in the `UI conventions` section (where the `/login` masthead contract and the `app.js` ownership are documented), add the collapsing contract in the section's voice: the 640px breakpoint, the toggle's identity and its seven-page scope, the three close paths (link click, Esc with focus return, resize-back), the `aria-expanded` rule, and that `/login` carries no toggle. `README.md` is checked in the phase's final pass (its page table — "power events" etc. — is unchanged by this phase).
## Testing & Quality
- Integration: the three new served-page tests above + the untouched orphan-`id` audit + `test_the_disks_tab_is_in_the_masthead_of_every_page` (the nav's id addition must not break it)
- Coverage: **>90%** on the new test code; the full gate green
## Completion Criteria
- [ ] `uv run pytest tests/test_api.py tests/test_auth_api.py -q` green, including the three new tests and the untouched audit
- [ ] `grep -c 'nav-toggle' data/logs/…` — the three screenshots exist and show the closed / open / desktop states as named
- [ ] `bash .agents/validate.sh` green (pytest --cov --cov-fail-under=94, ruff, pyright, node --check)
- [ ] AGENTS.md's UI conventions name the contract; README.md reviewed and consistent
- [ ] no behavior change in completed work: the desktop pages and the login page are pixel-identical in structure (the pins prove it)
@@ -0,0 +1,36 @@
# Phase 19 — Expandable UPS power-events list
**Source:** `TODO.md` L4 — "The UPS tab should have an expandable, clear list of last incidents with dates/times"
**Story:** n/a
**Context:** `src/dashboards/web/pages/ups.html` lines 72–76 (the `Power events` card: `<header><h3>` + a `ul.events#events` holding a `loading…` placeholder), `src/dashboards/web/static/page-ups.js` (`refreshEvents()` at line 430 — fetches `/api/events?limit=200&source=ups` and hands the rows to `Dash.eventsPager` with the date/time + severity-chip + host row shape; called unconditionally from `refresh()` at line 427, which the page's `poll(refresh, state.pollMs)` loop drives, `state.pollMs` from `config.ups.poll_ms || 5000`), `src/dashboards/web/static/app.js` (`Dash.eventsPager` — 50 rows a page, paging state in a `WeakMap` keyed by the `<ul>`, rows re-written only when their markup changed), `src/dashboards/web/static/dash.css` (`details.charts-box` at lines 714–727 — the summary marker `▸`/`▾`, hidden webkit marker, hover; the `.events` bounded 320px scroll and `.events-pager` strip), and the pattern this phase mirrors — `/system`'s per-host Charts disclosure (`system.html` line 81, `page-system.js` line 315: "a `<details>` renders nothing, so … nothing is built or drawn while the stack is shut") and its pin `test_system_history_charts_are_collapsed_behind_a_toggle`
## Objective
The `/ups` power-events list becomes what the TODO asks for — *expandable*: the card is collapsed by default behind a `<details>` disclosure in the repo's own `charts-box` style, nothing is fetched or drawn while it is shut (the `/system` rule), the first expand fetches and builds, a re-expand paints from the data in hand unless it is stale, and the page's poll loop refreshes the list only while the disclosure is open. The "clear" half already ships and stays: every row carries its date/time, the severity chip, and the host, and the 50-per-page pager bounds the list. No new API surface — the existing `/api/events?source=ups` is the whole backend.
## Dependencies
- `18_nav_hamburger` (todo) — both phases touch `ups.html` (the masthead, the events card); landing 18 first keeps this phase's diff confined to the events section. No shared code between them.
## Tasks
1. `01_events_disclosure.md` — the `ups.html` markup: the card body becomes a collapsed-by-default `<details class="events-box">`
2. `02_events_lazy_loading.md` — `page-ups.js`: no fetch while closed, first-expand fetch, re-expand stale rule, poll-while-open
3. `03_disclosure_css.md` — `dash.css`: the `details.events-box` summary styling in the `charts-box` shape, pager and bounded list inside
4. `04_tests_and_verification.md` — the served-page pins (disclosure present and unopened, the lazy rule, the unchanged row shape) and the closed/open browser verification
## Testing & Quality
- Unit/integration: `tests/test_api.py` — the served-page pins (see task 04), alongside the untouched `Dash.eventsPager` behaviour (a re-paint with the same data keeps the reader's page — the `WeakMap` rule, pinned by the existing event-feed tests)
- Coverage: **>90%** on new/modified code (the phase is markup + JS + CSS; the gate applies to the changed Python test files)
- The phase is "Complete" only when the **entire** project test suite passes 100% (every test in the project, not just this phase's)
## Documentation & Consistency
- The final pass reviews `README.md` and `AGENTS.md` in full against the project as it stands after this phase — fixing every inconsistency found, **including pre-existing ones not caused by this phase**.
- AGENTS.md's `UI conventions` section already documents the event pager ("An event feed is a window, not a printout") and the `/system` Charts disclosure; it gains the `/ups` disclosure's sentence (collapsed by default, lazy-built, poll-while-open, the stale rule) in the section's voice. The README's `/ups` table row ("power events") stays true — the list is on the page, now behind a summary.
## Completion Criteria
- [ ] `/ups` loads with the Power events card collapsed (a summary line, no list visible, no `/api/events` request in the page's first poll) and the expand shows the bounded, paged list with date/time rows
- [ ] closing and re-opening within one poll interval paints the cached rows without a refetch; a stale cache refetches on open; while open, the poll loop keeps the list current
- [ ] `uv run pytest` green, coverage not down, `ruff` + `pyright` clean
- [ ] `bash .agents/validate.sh` green (including `node --check` of the modified `page-ups.js`)
- [ ] closed- and open-state screenshots in `data/logs/` against the running dev server (invariant 10)
- [ ] no behavior change in completed work: every other element on `/ups` (cards, charts, dialog, pager behaviour) is unchanged
@@ -0,0 +1,40 @@
# Task 01 — The disclosure markup
**Phase:** `19_ups_events_expandable` · **Source:** `TODO.md:4` — "The UPS tab should have an expandable, clear list of last incidents with dates/times"
**Story:** n/a
## Objective
The `Power events` card on `/ups` becomes a collapsed-by-default `<details>` disclosure, in the markup shape the repo already owns for `/system`'s Charts stack. The card stays a card; only its body moves inside the disclosure. The `loading…` placeholder stays in the `ul` — it is the honest state between the first expand and the first successful fetch (the pager's output replaces it on arrival; an empty feed then prints the pager's own `empty` sentence, which is a different fact).
## Work
1. `src/dashboards/web/pages/ups.html` — restructure the section at lines 72–76 from
```html
<section class="card" style="margin-top:.85rem">
<header><h3>Power events</h3></header>
<div class="body"><ul class="events" id="events"><li class="muted">loading…</li></ul></div>
</section>
```
to
```html
<section class="card" style="margin-top:.85rem">
<details class="events-box">
<summary>Power events</summary>
<div class="body"><ul class="events" id="events"><li class="muted">loading…</li></ul></div>
</details>
</section>
```
— no `open` attribute (collapsed by default; `ASSUMPTION:` confirmed by the operator — "expandable" means the list is shut until the operator opens it, the `/system` Charts pattern). The `#events` id, the `ul.events` class (the bounded 320px scroll + `.events-pager` strip target it), the `loading…` placeholder, and the pager's contract are all the page-script's, so nothing else in the file moves.
2. `ASSUMPTION:` the summary is the plain word `Power events` (the old `<h3>` text) — no count chip on the summary. A count that has not been fetched would be an invented number (rule 2), and the pager under the list already says `1–50 of N` once data is in hand.
3. `ASSUMPTION:` the disclosure is a plain `details` (no `data-k` attribute) — `/system`'s `data-k="charts"` exists for the per-host JS that rebuilds the stack; here the single page-level `page-ups.js` queries it by class, and the pin in task 04 names the class, not an attribute.
## Testing & Quality
- Integration: the served-page pin lands in task 04 (`details.events-box` present on the served `/ups`, no `open` attribute, the `summary` label, the `#events` ul inside, no `loading…` placeholder); the untouched page renders (the browser verification is task 04)
- Coverage: **>90%** on modified code (HTML only; the gate's suite must stay green)
## Completion Criteria
- [ ] the served `/ups` contains the restructured section exactly as named (a `GET` pin in task 04: `details.events-box`, no `open` attribute, the `summary` label, the `#events` ul with its `loading…` placeholder inside)
- [ ] the page loads in a browser with the card collapsed: a `Power events` summary line, no list visible (task 04's screenshot)
- [ ] `uv run pytest -q` green; `bash .agents/validate.sh` green
- [ ] no behavior change in completed work: the rest of the page (cards, charts, dialog) is byte-identical in markup
@@ -0,0 +1,49 @@
# Task 02 — The lazy loading in `page-ups.js`
**Phase:** `19_ups_events_expandable` · **Source:** `TODO.md:4` — "The UPS tab should have an expandable, clear list of last incidents with dates/times"
**Story:** n/a
## Objective
Nothing is fetched or drawn while the disclosure is shut (the `/system` rule: "nothing is built or drawn while the stack is shut"), the first expand fetches and builds, a re-expand paints from the data in hand unless it is stale, and the page's poll loop refreshes the list only while the disclosure is open. The row shape, the pager, and the `loading…`-then-rows sequence are unchanged.
## Work
1. `src/dashboards/web/static/page-ups.js` — `state` (line 17) gains one key, in the object's existing style: `events: { open: false, painted: false, fetchedAt: 0 }`.
2. Same file, the `refresh()` function (the `await refreshEvents();` at line 427 is its last statement) — the unconditional call becomes the open-gate:
```js
if (state.events.open) await refreshEvents();
```
— the poll loop (the `state.poller = poll(refresh, state.pollMs)` at line 464) keeps driving the list **only while the disclosure is open**; closed, the page pays nothing for a feed nobody can see.
3. Same file, `refreshEvents()` (line 430) — on success, before/after the existing `Dash.eventsPager(…)` call, stamp the state: `state.events.fetchedAt = Date.now(); state.events.painted = true;`. The existing `catch (err) { /* next poll */ }` stays untouched — a failed fetch leaves `painted`/`fetchedAt` where they were, so the next open re-fetches, and a failed poll while open keeps the last good rows (the invariant-4 shape, already the page's behaviour for everything else).
4. Same file, inside the `DOMContentLoaded` handler, before `state.poller = poll(…)`:
```js
const eventsBox = $('.events-box');
if (eventsBox) {
state.events.open = eventsBox.open; // false — task 01 ships no `open` attribute
eventsBox.addEventListener('toggle', () => {
state.events.open = eventsBox.open;
if (!eventsBox.open) return;
// First expand: `painted` is false — fetch. Re-expand: paint from the data
// in hand only while it is fresh; a cache older than one poll interval is
// stale and re-fetches. A fresh, already-painted list needs nothing: the
// `<details>` keeps its children, so the rows (and the pager's page, in its
// WeakMap) are already where the operator left them.
if (!state.events.painted || Date.now() - state.events.fetchedAt > state.pollMs) {
refreshEvents();
}
});
}
```
`ASSUMPTION:` the stale bound is **one poll interval** (`state.pollMs`, from `config.ups.poll_ms || 5000`) — the `/system` Charts rule re-fetches a cached response "from another window"; here the window is the page's own, so one interval is the honest bound. A re-open within the interval paints the cache; a re-open after it re-fetches.
5. `ASSUMPTION:` no `loading…` fiddling — the placeholder from task 01 stays in the `ul` until the first successful fetch's pager output replaces it (a `<details>` does not destroy its children, so the placeholder, the rows, and the pager strip all survive collapse/re-expand untouched).
## Testing & Quality
- Integration: the served-script pins land in task 04 (the `state.events` shape, the `refresh()` open-gate, the `toggle` listener with its stale rule — asserted on the served bytes, the repo's way for JS behaviour since it has no JS test runner); `node --check` of the modified file passes (the gate); the untouched `Dash.eventsPager` tests stay green
- Coverage: **>90%** on modified code (JS; the gate's Python checks apply to task 04's test files)
## Completion Criteria
- [ ] in a browser: the page's first poll makes **no** `/api/events` request (devtools network panel) while the card is collapsed; the first expand makes exactly one; the list then updates on the poll cadence while open and stops when closed
- [ ] close → re-open within one poll interval: no request, the rows are exactly as left (page, scroll); close → wait longer than the interval → open: one request, fresh rows
- [ ] a failed fetch (devtools offline) while open keeps the last good rows and the next poll retries; the next open after a failure re-fetches
- [ ] `uv run pytest -q` green; `bash .agents/validate.sh` green (includes `node --check`)
@@ -0,0 +1,42 @@
# Task 03 — The disclosure CSS
**Phase:** `19_ups_events_expandable` · **Source:** `TODO.md:4` — "The UPS tab should have an expandable, clear list of last incidents with dates/times"
**Story:** n/a
## Objective
The `details.events-box` summary renders in the shape the repo already owns for `/system`'s Charts disclosure (`details.charts-box`, `dash.css` lines 714–727): the `▸`/`▾` marker, the hidden webkit marker, the hover, and the open-state de-emphasis — so a second opinion about the drive's word is never the bug, and a page that ships the summary without the style is the drift this task's pin catches.
## Work
1. `src/dashboards/web/static/dash.css` — beside the existing `details.charts-box` rules (lines 714–727), add the `events-box` rules in the same block's voice (one selector per property, the file's style):
```css
details.events-box { min-width: 0; }
details.events-box > summary {
cursor: pointer; list-style: none;
font-weight: 600; color: var(--text-muted);
padding: .4rem .8rem .4rem 1.4rem;
position: relative;
}
details.events-box > summary::-webkit-details-marker { display: none; }
details.events-box > summary::before {
content: "\25B8"; position: absolute; left: .6rem;
font-size: 9px; color: var(--text-faint);
}
details.events-box[open] > summary::before { content: "\25BE"; }
details.events-box > summary:hover { color: var(--text-muted); }
details.events-box[open] > summary { color: var(--text-muted); margin-bottom: .55rem; }
```
(match the `charts-box` block's actual property set line-for-line — the snippet is the shape, the neighbouring block is the ground truth; do not invent tokens `:root` does not declare — the file's rule, pinned by `tests/test_api.py`).
2. The card's existing `.body` wrapper (task 01's markup) carries the `ul.events` bounded scroll (the existing 320px rule) and the `.events-pager` strip — **no new rules** for those; verify they resolve inside the disclosure (the `.body` is a child of `details`, and the existing selectors are class-based, so they do — confirm in the browser, task 04).
3. `ASSUMPTION:` no open/close animation on the disclosure — `charts-box` has none, and the file's `prefers-reduced-motion` block exists for the things that do move; a summary that snaps is the house style.
## Testing & Quality
- Integration: the served-stylesheet pin lands in task 04 (the `details.events-box` rule family present, the `::before` markers, no token a `:root` does not declare — the existing `test` that fails if a CSS rule reaches for an undeclared token must pass untouched); the `node --check` gate is unaffected (CSS), and the page loads in the browser (task 04)
- Coverage: **>90%** on modified code (CSS only; the gate's suite must stay green)
## Completion Criteria
- [ ] collapsed: the `Power events` summary line shows the `▸` marker in the card header area, muted, with the card's existing padding rhythm
- [ ] open: the marker flips to `▾`, the summary de-emphasises, and the bounded scroll list + pager strip render exactly as the other feeds' (the 320px scroll, the `1–50 of N` strip with its arrows)
- [ ] the page's other cards (the battery charts' `details`-free section, the `#ups-cards` grid) are pixel-identical
- [ ] `bash .agents/validate.sh` green
@@ -0,0 +1,31 @@
# Task 04 — Pins and browser verification
**Phase:** `19_ups_events_expandable` · **Source:** `TODO.md:4` — "The UPS tab should have an expandable, clear list of last incidents with dates/times"
**Story:** n/a
## Objective
The phase is pinned on the served bytes the way this repo pins every page surface: the disclosure is on the served `/ups` and **unopened**, the lazy-load rule is in the served script, and the row shape — the "clear" half of the TODO (date/time, severity chip, host) — is unchanged. Then the designed degradation is proven in a real browser: the closed state, the open state, and the no-request-while-closed fact.
## Work
1. `tests/test_api.py` — new tests, in the file's served-page style (the `self_test_app` harness and the `test_system_history_charts_are_collapsed_behind_a_toggle` pin at the `/system` disclosure are the models):
- `test_the_ups_power_events_list_is_a_collapsed_disclosure`: a signed `GET /ups` finds `details.events-box` with **no** `open` attribute, a `summary` reading `Power events`, and the `ul.events#events` (with its `loading…` placeholder) inside it — and the old always-open shape is gone (no bare `header > h3` directly above the `ul` at the section level).
- `test_the_ups_events_fetch_is_gated_on_the_disclosure`: the served `page-ups.js` carries the lazy rule — the `state.events` shape (`open`, `painted`, `fetchedAt`), the `refresh()` open-gate (`state.events.open` guard on the `refreshEvents()` call), and the `toggle` listener with the stale bound (`state.pollMs`) — asserted on the served script text, the repo's established device for JS behaviour (no JS test runner; the `node --check` in the gate covers syntax).
- `test_the_ups_event_rows_keep_the_date_time_and_severity_shape`: the served `page-ups.js` row template still renders the `<time>` with the date/time `toLocaleString` options, the `sev ${event.severity}` chip, and the host — the "clear list with dates/times" the TODO asks for, pinned so a "cleanup" that drops the timestamp fails here instead of shipping.
2. The existing event-feed pins (`tests/test_api.py`'s `Dash.eventsPager`-behaviour tests, the `EVENT_FEED_LIMIT`/`EVENT_PAGE_SIZE` pairing) pass **untouched** — the phase adds no API surface and no pagination vocabulary.
3. Browser verification (invariant 10): `./scripts/dev-server.sh restart`, sign in with `testpass`, real upsd in the loop (the dev config dials the fleet's `upsd` on 3051 — two real UPSes, no `NUT_SIMULATE`):
- `data/logs/ups-events-closed.png` — the page with the card collapsed: the `Power events` summary line, no list, and the devtools network panel showing **no** `/api/events?source=ups` request on the first poll (screenshot the panel or note the request count in the filename's companion note).
- `data/logs/ups-events-open.png` — the expanded card: the bounded scroll list with date/time rows, the severity chips, and the `1–50 of N` pager strip (or the empty sentence, if the event table is genuinely empty — either is the correct state; the pin is the *shape*, not the data).
4. `AGENTS.md` — in the `UI conventions` section, beside the event-pager paragraph ("An event feed is a window, not a printout") and the `/system` Charts-disclosure paragraph, add the `/ups` sentence in the section's voice: the list is a collapsed-by-default `details` disclosure, nothing is fetched while it is shut, the first expand fetches and builds, a re-expand paints the cache while it is fresher than one poll interval (stale re-fetches), and the poll loop refreshes only while open. `README.md` is checked in the phase's final pass (its `/ups` row — "power events" — stays true; the list is on the page, behind a summary).
## Testing & Quality
- Integration: the three new served-page tests above + the untouched pager/limit pairing tests + `test_system_history_charts_are_collapsed_behind_a_toggle` (the pattern pin, untouched)
- Coverage: **>90%** on the new test code; the full gate green
## Completion Criteria
- [ ] `uv run pytest tests/test_api.py -q` green, including the three new tests and the untouched event-feed pins
- [ ] `data/logs/ups-events-closed.png` and `data/logs/ups-events-open.png` exist and show the states as named (closed: summary only; open: bounded list + pager; the closed shot's note records the zero `/api/events` requests on the first poll)
- [ ] `bash .agents/validate.sh` green (pytest --cov --cov-fail-under=94, ruff, pyright, node --check)
- [ ] AGENTS.md's UI conventions carry the disclosure sentence; README.md reviewed and consistent
- [ ] no behavior change in completed work: the rest of `/ups` (cards, charts, the confirm dialog, the banners) renders exactly as before
@@ -0,0 +1,36 @@
# Phase 20 — Masthead & navigation polish
**Source:** `TODO.md` L4, L5, L6, L10, L11 — "masthead row layout, un-sticky bar, hamburger animation, always-visible admin links"
**Story:** n/a
**Context:** `src/dashboards/web/static/dash.css` (`.masthead` at line 99 — `position: sticky; top: 0; z-index: 40; display: flex; height: var(--header-h)`; `.masthead-right` at line 148 — `margin-inline-start: auto`; `#nav-toggle` at line 190 — the `--nav-toggle-display` token reveal; the ≤640px block at lines 208–280 — masthead wrap, `masthead-right { justify-content: flex-end }`, the `nav.tabs` dropdown at `top: 100%` under the sticky bar, the `env(safe-area-inset-top, 0px)` padding line), `src/dashboards/web/static/app.js` (the `setNavMenu()` funnel at lines 695–722 — four close paths, `matchMedia('(max-width: 640px)')`), `src/dashboards/web/pages/*.html` (the masthead in every page: brand, `#nav-toggle`, `nav.tabs#nav-tabs`, `.masthead-right` holding `#nav-status` + clock + `⚷ Certs` + `⚙ Account` + theme toggle; `certs.html` lacks `⚷ Certs`, `settings.html` lacks `⚙ Account` — the phase-15 "no self-link" rule this phase inverts), `tests/test_api.py` (`test_every_page_links_to_both_admin_pages` at line 999, `test_the_disks_tab_is_in_the_masthead_of_every_page` at line 3913, `test_the_nav_toggle_is_in_the_masthead_of_every_tabbed_page` at line 3957)
## Objective
The masthead becomes a three-row bar at **all widths** — row 1 the brand and the five dashboard tabs, row 2 the per-source status indicators left-justified in their own row, row 3 the `⚷ Certs` / `⚙ Account` / `☾ Dark` controls left-justified in their own row under the indicators — the bar stops being sticky (it scrolls away, maximising data screen space), the hamburger menu slides out and back in (180 ms, the `brain_of_reese` pattern, stilled under `prefers-reduced-motion`), and both admin links stay visible on every page including the two admin pages themselves. The served page markup is unchanged except the two added admin links; the tab structure (five links, one `aria-current`), the `#nav-toggle` token reveal, the safe-area padding line and the four close paths all stay exactly as phase 18 pinned them.
## Dependencies
- `19_ups_events_expandable` (todo) — both phases touch `ups.html` (the masthead, the events card); landing 19 first keeps this phase's diff confined to the masthead.
## Tasks
1. `01_two_row_masthead.md` — the three-row left-justified masthead at all widths (CSS-only reflow; the mobile block re-cut for the new rows)
2. `02_unsticky_masthead.md` — the bar scrolls away: `sticky` → `relative`, the dropdown anchor kept
3. `03_hamburger_slide.md` — the 180 ms slide+fade on the ≤640px dropdown, stilled under `prefers-reduced-motion`
4. `04_admin_links_visible.md` — `⚷ Certs` on `/certs`, `⚙ Account` on `/settings`; the "no self-link" pin inverted to "all eight pages link to both"
## Testing & Quality
- Unit/integration: `tests/test_api.py` — the served-page pins (task 04's rewritten admin-link test over all eight non-login pages, the unchanged tab-structure pins at lines 3913/3957, the CSS pins for the row layout and the un-sticky change), the unchanged `setNavMenu` behaviour
- Coverage: **>90%** on new/modified code (the phase is markup + JS + CSS; the gate applies to the changed Python test files)
- The phase is "Complete" only when the **entire** project test suite passes 100% (every test in the project, not just this phase's)
## Documentation & Consistency
- The final pass reviews `README.md` and `AGENTS.md` in full against the project as it stands after this phase — fixing every inconsistency found, **including pre-existing ones not caused by this phase**. AGENTS.md's masthead paragraph (the phase-18 "specificity ladder" and the `top: 100%`-under-the-sticky-bar wording) and the phase-15 "neither admin page links to itself" sentence (gap 12's "now questioned" item — this phase is the decision) both get rewritten to the new state in the section's voice.
## Completion Criteria
- [ ] at 1280 px and at 360 px, every tabbed page and both admin pages render the three-row masthead: brand + tabs, left-justified indicators row, left-justified controls row under it (screenshots in `data/logs/`, invariant 10)
- [ ] scrolling any page at 1280 px and 360 px leaves the bar behind (no sticky offset), while the mobile dropdown still opens directly under the bar
- [ ] at 360 px the hamburger menu slides out and back in (180 ms); with `prefers-reduced-motion` it snaps
- [ ] `/certs` renders `⚷ Certs` and `/settings` renders `⚙ Account`, before the theme toggle and outside `.tabs`, exactly like the five dashboards
- [ ] `uv run pytest` green, coverage not down, `ruff` + `pyright` clean
- [ ] `bash .agents/validate.sh` green (including `node --check` of the modified `app.js`)
- [ ] no behavior change in completed work: the tab structure, the token reveal, the safe-area padding line, the four close paths, the status dots and the theme toggle are all unchanged
@@ -0,0 +1,27 @@
# Task 01 — The three-row left-justified masthead, at all widths
**Phase:** `20_masthead_nav_polish` · **Source:** `TODO.md:5,6` — "The hosts disks engines and UPS indicators should be left justified in their own row" / "The ⚷ Certs ⚙ Account ☾ Dark buttons should be left justified in their own row under the hosts disks engines UPS indicators"
**Story:** n/a
## Objective
The single-row masthead becomes a three-row bar at **every** width: row 1 keeps the brand, `#nav-toggle` and the five dashboard tabs; row 2 is the `#nav-status` indicator row (the per-source dots), left-justified and on its own; row 3 is the `.masthead-right` cluster (clock, `⚷ Certs`, `⚙ Account`, `☾ Dark` — plus the `⏸ Pause` button until phase 21 removes it), left-justified and on its own row under the indicators. The change is CSS-only: no page markup changes in this task, so the served-HTML pins (tab structure, toggle placement) stay green untouched.
## Work
1. `src/dashboards/web/static/dash.css` — `.masthead` (line 99): `display: flex; flex-direction: column; align-items: stretch; gap: .35rem` (replacing the row layout's `align-items: center; gap: 1.1rem`), `height: var(--header-h)` replaced by `min-height: var(--header-h)` (the ≤640px block already carries its own `min-height` — keep the mobile one, the two must not drift apart; see the comment it carries), `position: sticky; top: 0` **left in place in this task** (task 02 removes the stickiness; keep `z-index: 40`), and a positioned wrapper for row 1: the first flex child is the existing `brand` + `#nav-toggle` + `nav.tabs` trio — since the markup is not touched, use CSS `order`/`flex` only: `#nav-status` and `.masthead-right` each get `flex-basis: 100%` (forcing the row break), `nav.tabs` keeps its current inline position on row 1 (it already flows after the toggle). If `flex-basis: 100%` does not force a row break on the `nav` element in the pinned browser, the fallback is a `display: contents` wrapper-free `flex-wrap: wrap` on `.masthead` with the same `flex-basis: 100%` rules — measure in a browser, do not deduce (the phase-18 rule).
2. `src/dashboards/web/static/dash.css` — `#nav-status` (the `gap` container, line 148 neighbourhood): its own row styling — `flex-basis: 100%; display: flex; align-items: center; gap: .55rem; justify-content: flex-start` (left-justified, no `margin-inline-start: auto`).
3. `src/dashboards/web/static/dash.css` — `.masthead-right` (line 148): `margin-inline-start: auto` removed, `flex-basis: 100%; justify-content: flex-start` (left-justified, the TODO's words). The `⏸ Pause` button stays in this row (phase 21 deletes it; do not pre-empt it).
4. `src/dashboards/web/static/dash.css` — the ≤640px block (lines 208–280): re-cut for the new rows — the whole-bar `flex-wrap`/`min-height` machinery of the old wrap is replaced by the mobile half of the same three-row layout: `#nav-toggle` keeps its `--nav-toggle-display: inline-flex` reveal on row 1 (the token ladder stays exactly as pinned), `nav.tabs` stays hidden except `nav.tabs.is-open` (the dropdown), the `masthead-right > *` no-wrap rule and the `env(safe-area-inset-top, 0px)` top padding line (pinned by `tests/test_api.py`'s rule loop) survive verbatim, and the comment at line 203 (which describes the old 360 px wrap) is rewritten to the new three-row reality. The dropdown's `top: 100%` now lands under the **whole bar** (three rows) on mobile — that is the intended look (a menu under the bar, not wedged between rows); verify in a browser at 360 px.
5. `tests/test_api.py` — extend the served-stylesheet pins for the new layout: `.masthead` has `flex-direction: column` (or the `flex-wrap` fallback, whichever shipped) and no fixed `height: var(--header-h)`, `#nav-status` and `.masthead-right` are both `flex-basis: 100%` + `justify-content: flex-start`, and `.masthead-right` no longer declares `margin-inline-start: auto`. The existing pins (the media-query-vs-`matchMedia` comparison, the safe-area line, the token ladder) must stay green — run them before touching the file.
## Testing & Quality
- Unit/integration: `tests/test_api.py` — the new layout pins above run against the **served** bytes; the unchanged pins (tab structure line 3913, toggle line 3957, the safe-area rule loop) stay green
- Coverage: **>90%** on this task's new/modified code (the gate applies to the changed Python test files)
- Browser verification (invariant 10): screenshots of `/`, `/system`, `/certs`, `/settings` at 1280 px and 360 px in `data/logs/` — the three rows, indicators and controls left-justified, the mobile dropdown under the bar
## Completion Criteria
- [ ] at 1280 px and 360 px, the masthead renders three rows in the order brand+tabs / indicators / controls, the last two left-justified, on every tabbed page and both admin pages (screenshots in `data/logs/`)
- [ ] `tests/test_api.py` green including the new layout pins and every pre-existing masthead pin
- [ ] full test suite green
- [ ] no behavior change in completed work: the tabs, the toggle, the dots, the clock, the links and the theme toggle all behave exactly as before
@@ -0,0 +1,27 @@
# Task 02 — The masthead is not sticky
**Phase:** `20_masthead_nav_polish` · **Source:** `TODO.md:10` — "The navbar is only sticky part way down certain pages on mobile? It shouldn't be sticky at all. We want to maximize the amount of screen space for data viewing."
**Story:** n/a
## Objective
The masthead scrolls away with the page. The "sticky part way down" quirk is the fixed-height sticky bar fighting the wrapped rows on mobile; removing the stickiness removes the quirk and the bar's vertical footprint. The ≤640px dropdown (`nav.tabs.is-open` at `top: 100%`) still opens directly under the bar, because the bar stays a **positioned** ancestor — `position: relative` keeps the anchor without the stick.
## Work
1. `src/dashboards/web/static/dash.css` — `.masthead` (line 99): `position: sticky; top: 0` → `position: relative` (keeps the positioned-ancestor role for the dropdown's `top: 100%` and the `left/right: 0` span; `z-index: 40` stays — the dropdown's `z-index: 41` is "one above the masthead" and that sentence stays true).
2. `src/dashboards/web/static/dash.css` — the dropdown comment (line 266: "`position: sticky`, which is a positioned ancestor") rewritten to `position: relative`; any other comment in the file that calls the bar sticky gets the same fix.
3. `tests/test_api.py` — any pin that asserts the `position: sticky` declaration (grep the test files for `sticky`) is updated to assert `position: relative` **and** the absence of `top: 0` on `.masthead`; if no pin named the declaration, add one to the layout pin loop from task 01 so the stickiness cannot quietly return.
4. `src/dashboards/web/static/dash.css` — check the `sticky; top: 0` at line 796 (a **different** element — the event-feed pager strip's own sticky header): it is not the masthead, do not touch it.
## Testing & Quality
- Unit/integration: `tests/test_api.py` — the `.masthead` pin above, the unchanged dropdown pin (the media query still names `top: 100%` under the bar)
- Coverage: **>90%** on this task's new/modified code
- Browser verification (invariant 10): at 1280 px and 360 px, scroll `/system` and `/` past one viewport — the bar is gone from the top of the viewport; at 360 px open the dropdown — it still renders directly under the bar (screenshot pair in `data/logs/`)
## Completion Criteria
- [ ] scrolling any page at any width leaves the masthead behind (no element pinned to the top edge but the pager strip's own header)
- [ ] the mobile dropdown at 360 px still opens directly under the bar (screenshot in `data/logs/`)
- [ ] `tests/test_api.py` green including the new `.masthead` pin
- [ ] full test suite green
- [ ] no behavior change in completed work: the pager strip's sticky header (line 796) and the dropdown's `z-index` stacking are untouched
@@ -0,0 +1,27 @@
# Task 03 — The hamburger menu slides out and back in
**Phase:** `20_masthead_nav_polish` · **Source:** `TODO.md:11` — "I want a nice animation for the hamburger menu - a slide out and slide back in would be perfect. See ~/Project/Personal/brain-of-reese for an example."
**Story:** n/a
## Objective
The ≤640px dropdown (`nav.tabs.is-open`) opens and closes with an 180 ms slide+fade instead of snapping — the exact pattern the operator pointed at in `brain_of_reese`: closed = `visibility: hidden; opacity: 0; transform: translateY(-8px); pointer-events: none` with a `visibility 0s linear 180ms` delayed transition, open = `visibility: visible; opacity: 1; transform: none; pointer-events: auto` with the `visibility` delay dropped, and the whole thing stilled under `prefers-reduced-motion` with **both states named** (the reference project's phase-46 comment: an open-state transition that is not stilled still animates). `app.js`'s `setNavMenu()` funnel and its four close paths are untouched — the animation is pure CSS over the existing class flip.
## Work
1. Reference (read-only): `/var/home/ducoterra/Projects/Personal/brain_of_reese/frontend/assets/styles.css` lines 6960–6976 (the `.app-nav` closed/open transition pair — note the TODO's path `~/Project/Personal/brain-of-reese` is a typo; the project lives at `~/Projects/Personal/brain_of_reese`) and lines 7228–7235 (the `prefers-reduced-motion` block that names **both** `.app-nav` and `.app-nav.is-open`, with the specificity comment: an open rule that is not named keeps animating).
2. `src/dashboards/web/static/dash.css` — inside the ≤640px block (lines 208–280), the `nav.tabs` rule (the dropdown): add the closed-state properties + transition from the reference (adapted: this nav is `position: absolute; top: 100%`, so the slide is `transform: translateY(-8px)` → `none`, same 180 ms ease; keep the existing `visibility: hidden`/`pointer-events: none` closed-state behaviour if it is already there, otherwise add it — measure the current rule first, do not assume). The `nav.tabs.is-open` state gets the visible counterpart with the `visibility 0s` (undelayed) transition.
3. `src/dashboards/web/static/dash.css` — a `@media (prefers-reduced-motion: reduce)` block (the file's first, or merged with an existing one if there is one) that sets `transition: none` on **both** `nav.tabs` and `nav.tabs.is-open` — both states named, per the reference's phase-46 comment.
4. `tests/test_api.py` — extend the served-CSS pin for the mobile block: the `nav.tabs` rule carries an `opacity`/`transform` transition of 180 ms, the `is-open` state carries the visible counterpart, and the reduced-motion block names both states. The existing media-query-vs-`matchMedia` comparison test stays green untouched (the transition is additive; the `640px` literal still lives in exactly the two pinned places).
## Testing & Quality
- Unit/integration: `tests/test_api.py` — the pins above; the unchanged four-close-path behaviour (the `setNavMenu` funnel tests)
- Coverage: **>90%** on this task's new/modified code
- Browser verification (invariant 10): at 360 px, open and close the menu — a CDP check that the computed `transition` runs (a mid-transition screenshot is not required; assert the closed state's `opacity: 0` + `transform: translateY(-8px)` computed values and that a frame after `.is-open` is added shows `opacity > 0`), and with `prefers-reduced-motion: reduce` emulated the menu snaps (computed `transition: none`) — extend the phase-18 verification script pattern (`data/logs/nav-task04-verify.cjs`) rather than hand-asserting
## Completion Criteria
- [ ] at 360 px the menu slides out (fade + 8 px rise) and back in over 180 ms; all four close paths still work
- [ ] under `prefers-reduced-motion: reduce` the menu snaps in both directions
- [ ] `tests/test_api.py` green including the new transition pins and the pre-existing `640px` comparison pin
- [ ] full test suite green
- [ ] no behavior change in completed work: the token reveal, the safe-area line, the dropdown position under the bar and the close paths are untouched
@@ -0,0 +1,27 @@
# Task 04 — Both admin links stay visible on the admin pages
**Phase:** `20_masthead_nav_polish` · **Source:** `TODO.md:4` — "Clicking the certs tab causes the certs button to disappear, then clicking the account tab causes the accounts button to disappear. These should remain visible."
**Story:** n/a
## Objective
Phase 15's deliberate "neither admin page links to itself" rule is inverted by operator decision (it was already "now questioned" in AGENTS.md gap 12): `certs.html` gains the `⚷ Certs` link and `settings.html` gains the `⚙ Account` link, so the masthead's controls are identical on all eight non-login pages. The links sit before the theme toggle and outside `.tabs` exactly as on the five dashboards; the tab structure (five links, no `aria-current` on the admin pages) and the `/login` page (brand + theme toggle only) are untouched.
## Work
1. `src/dashboards/web/pages/certs.html` — the masthead (lines 14–28): insert `<a class="ghost account" href="/certs" title="Agent certificates">⚷ Certs</a>` between the clock and the `⚙ Account` link (the dashboard order is clock → `⚷ Certs` → `⚙ Account` → theme toggle; `settings.html` already holds that order), and rewrite the HTML comment at lines 14–17, which currently justifies the *missing* link, to the new rule (both links everywhere; a self-link is a no-op the operator can click and not be surprised by — the TODO is the decision).
2. `src/dashboards/web/pages/settings.html` — the masthead (lines 14–30): insert `<a class="ghost account" href="/settings" title="Session and password">⚙ Account</a>` between the `⚷ Certs` link and the theme toggle; the comment at lines 12–14 ("It is reached from the `⚙ Account` link in the masthead of every dashboard") gains the "including this one" sentence.
3. `tests/test_api.py` — rewrite `test_every_page_links_to_both_admin_pages` (line 999): the two loops become **one** loop over all eight non-login pages (`*PAGES, SETTINGS_PAGE, CERTS_PAGE`) asserting both links are present, both before `id="theme-toggle"`, and `"/certs"` is not in the `.tabs` markup; the two `assert … not in … "links to itself"` lines are **deleted** (that was the old rule); add one explicit assertion that `login.html` carries **neither** link (the pre-auth surface renders brand + theme toggle only — that sentence in AGENTS.md stays true).
4. `AGENTS.md` — (Documentation & Consistency pass) the phase-15 "neither admin page links to itself, which is why the pinned two-link audit loops over two different tuples" sentence and gap 12's "now questioned" item are resolved to the new state.
## Testing & Quality
- Unit/integration: `tests/test_api.py` — the rewritten `test_every_page_links_to_both_admin_pages`, the unchanged `test_the_disks_tab_is_in_the_masthead_of_every_page` (line 3913 — the tab structure is untouched by this task)
- Coverage: **>90%** on this task's new/modified code
- Browser verification (invariant 10): signed in, click `⚷ Certs` on `/certs` (stays on `/certs`, no navigation loop) and `⚙ Account` on `/settings` (same) — screenshots of both pages' mastheads in `data/logs/`
## Completion Criteria
- [ ] `/certs` renders `⚷ Certs` and `/settings` renders `⚙ Account`, both before the theme toggle and outside `.tabs`
- [ ] all eight non-login pages render both admin links; `/login` renders neither
- [ ] `tests/test_api.py` green with the rewritten test
- [ ] full test suite green
- [ ] no behavior change in completed work: the tab strip, the `aria-current` absence on the admin pages, the clock and the theme toggle are untouched
@@ -0,0 +1,34 @@
# Phase 21 — Page cleanups: pause, GPU values, certs scroll
**Source:** `TODO.md` L7, L9, L12 — "remove the pause button, GPU data as its own values, /certs table horizontal scroll"
**Story:** n/a
**Context:** `src/dashboards/web/pages/{disks,llm,system,ups}.html` line 25 (the `#pause` button, the only one in the served markup — `/`, `/certs` and `/settings` have none), `src/dashboards/web/static/app.js` (lines 587–615 the `poll()` `_paused` plumbing, lines 773–783 the `#pause` binding plus the `Space` shortcut), `src/dashboards/web/static/dash.css` line 203 (the comment naming the pause button), `src/dashboards/web/static/page-system.js` (lines 150–156 `gpuSub()` — the dense sub-line the TODO quotes, lines 164–188 `renderGpuSlot()`, line 72 the `data-k="gpus"` hero slot), `tests/test_api.py` (line 1425 — the six-heroes pin: "exactly six heroes: CPU, MEM, TEMP, DISK, GPU, NET", asserted over the three captures: discrete GPU, APU, no-GPU), `src/dashboards/web/pages/certs.html` line 81 (`<table class="data" id="agents-table">` — no `.scroll` modifier; `dash.css` gives `table.data` a scroll container only with the `.scroll` class), `src/dashboards/web/static/page-certs.js` (lines 55–56 — the table element references)
## Objective
Three small page-level cleanups, one per workstream: the pause button is gone from every page (the hidden-tab skip stays — a background tab idling its polls is not the operator's pause), the `/system` GPU data is split out of the dense sub-line into its own labelled values and a host with no GPU **omits** the slot instead of printing `n/a`, and the `/certs` agents table scrolls horizontally so the right-hand columns are reachable on a phone. No API surface changes; the pins that name the old behaviour (six heroes incl. an `n/a` GPU slot, the unpinned pause button) are rewritten, not left rotting.
## Dependencies
- `20_masthead_nav_polish` (todo) — the pause button lives in the masthead row task 01.01 reflows, and the `/system` hero row is re-measured under the new bar; landing 20 first keeps 21's diff confined to the page bodies.
## Tasks
1. `01_remove_pause_button.md` — delete the button, the `Space` shortcut, the `setPaused`/`_paused` plumbing and the CSS comment; the hidden-tab skip survives
2. `02_gpu_own_values.md` — the dense `gpuSub()` line becomes four labelled values per card; a GPU-less host omits the slot (five heroes, no `n/a`)
3. `03_certs_table_scroll.md` — `#agents-table` gets the `.scroll` modifier so its right-hand columns are reachable at phone width
## Testing & Quality
- Unit/integration: `tests/test_api.py` — the served-page pins: the pause button absent from the four pages (a return of `#pause` fails a test), the six-heroes pin rewritten (GPU-present host: six, GPU-absent host: five with no `data-k="gpus"` slot), the `#agents-table` `.scroll` pin; `tests/test_system_contract.py` + `tests/card_contract.py` stay green (the API fields are unchanged — only the page's reading of them moves)
- Coverage: **>90%** on new/modified code (the phase is markup + JS + CSS; the gate applies to the changed Python test files)
- The phase is "Complete" only when the **entire** project test suite passes 100% (every test in the project, not just this phase's)
## Documentation & Consistency
- The final pass reviews `README.md` and `AGENTS.md` in full against the project as it stands after this phase — fixing every inconsistency found, **including pre-existing ones not caused by this phase**. AGENTS.md's `/system` paragraph ("a missing number or verdict — gateway's CPU temp, any host's absent GPU — prints `n/a` **uncoloured**") is the one rule this phase inverts for the GPU slot (the CPU-temp `n/a` stays — the TODO quotes the GPU line only); the sentence is split, the GPU half rewritten to the omission rule, and the pause button's mention in the masthead comment dropped.
## Completion Criteria
- [ ] no page serves a `#pause` button or binds the `Space` key; a hidden tab still skips its polls (served-bytes pin + a CDP `visibilitychange` check)
- [ ] `/system`'s GPU slot prints model / VRAM / pool / temp as four labelled values; a GPU-less host (the no-GPU capture, gateway) renders five heroes with no GPU slot and no `n/a` in it (screenshots in `data/logs/`)
- [ ] at 360 px the `/certs` agents table scrolls horizontally to its last column (screenshot in `data/logs/`)
- [ ] `uv run pytest` green, coverage not down, `ruff` + `pyright` clean
- [ ] `bash .agents/validate.sh` green (including `node --check` of the modified page scripts)
- [ ] no behavior change in completed work: the six-hero order on GPU hosts, the hidden-tab skip, the GPU history charts, the `/certs` row contents and the event pager are all unchanged
@@ -0,0 +1,27 @@
# Task 01 — Remove the pause button
**Phase:** `21_page_cleanups` · **Source:** `TODO.md:7` — "Completely remove the pause button"
**Story:** n/a
## Objective
The `⏸ Pause` control is gone from the served markup, the page scripts and the shared poller: no button, no `Space`-key shortcut, no `setPaused`/`_paused` plumbing. The **hidden-tab** behaviour survives — `document.hidden` still skips a poll while the tab is backgrounded and `visibilitychange` still fires one on return; that is idle hygiene, not the operator's pause, and the TODO names the button only.
## Work
1. `src/dashboards/web/pages/disks.html`, `src/dashboards/web/pages/llm.html`, `src/dashboards/web/pages/system.html`, `src/dashboards/web/pages/ups.html` — delete the line-25 `<button class="ghost" id="pause" type="button" aria-pressed="false">⏸ Pause</button>` from each of the four (the only served occurrences — `/`, `/certs`, `/settings`, `/login` have none; a grep of `web/pages` for `pause` must come back empty when done).
2. `src/dashboards/web/static/app.js` — delete the `#pause` binding block (lines 773–783: the button lookup, the `aria-pressed` paint, the click handler, and the `keydown` `Space` handler with its `event.preventDefault()`); in `poll()` (lines 587–615) delete the `_paused` state, `setPaused()`, the `paused` getter and the `state._paused` checks in `tick()`/`schedule()`/`visibilitychange` — the `document.hidden` guard and the `visibilitychange` refire stay, minus the `&& !state._paused` clause; the docstring at line 587 ("pausing when the tab is hidden or the user hit pause") drops the pause half.
3. `src/dashboards/web/static/dash.css` — line 203's comment (which lists "status dots, pause, `⚷ Certs`, `⚙ Account`, theme" as the 360 px wrap's tenants) drops the pause mention.
4. `tests/test_api.py` — add the served-bytes pin: `id="pause"` is **absent** from all eight served pages (a return of the button fails a test) and `⏸` appears nowhere in the served scripts.
## Testing & Quality
- Unit/integration: `tests/test_api.py` — the absence pin above; the existing poller tests (the `Dash.poll` hidden-tab behaviour, if any pin it — grep `hidden` in the test files first) stay green
- Coverage: **>90%** on this task's new/modified code
- Browser verification (invariant 10): `/` and `/system` — no pause control in the masthead, the `Space` key does nothing on the page body; hide the tab (CDP `Emulation.setDocumentHidden` or the window manager) and confirm the poll loop idles, then refire on return (the phase-18 verification script pattern)
## Completion Criteria
- [ ] `grep -ri pause src/dashboards/web/` comes back empty (code, markup and comments all clean)
- [ ] a hidden tab skips polls and a visible tab fires one on return (verified, not deduced)
- [ ] `tests/test_api.py` green including the new absence pin
- [ ] full test suite green
- [ ] no behavior change in completed work: the range chips, the event pagers, the status dots and every other masthead control behave exactly as before
@@ -0,0 +1,29 @@
# Task 02 — GPU data as its own values; GPU-less hosts omit the slot
**Phase:** `21_page_cleanups` · **Source:** `TODO.md:12` — "The GPU data is wrapped poorly - Radeon 8060S Graphics · VRAM 180.2 MiB / 512.0 MiB · pool 79% · 69 °C is too much info for such small text and should be its own values. Boxes without a GPU should not report N/A but instead should omit these values."
**Story:** n/a
## Objective
The `/system` hero card's GPU slot stops stuffing model · VRAM · pool · temp into one small sub-line: each becomes its own labelled value on the card, at a readable size. A host with no GPU (gateway, the no-GPU capture) no longer renders an `n/a` hero slot — the slot is **omitted** and the hero row simply has five. The API is untouched: `gpus[]` keeps every field (`model`, `vram_used_bytes`, `vram_total_bytes`, `mem_used_pct`, `temp_c`), and the per-card history (`gpu.busy_pct@N`, `gpu.temp_c@N`) and the collapsed GPU chart are unchanged.
## Work
1. `src/dashboards/web/static/page-system.js` — replace `gpuSub()` (lines 150–156) with a `gpuValues(gpu)` that renders four labelled cells — **Model** (mono, the `gpu.model` string), **VRAM** (`used / total` via the existing `bytes()` formatter, `n/a` if either side is absent), **Pool** (`mem_used_pct` %), **°C** (`temp_c`) — laid out as a compact `.gpu-values` grid under the busy hero (the hero itself keeps only the `busy_pct` number + its label; colour rules stay: the GPU is uncoloured, phase 06's "a busy GPU is the job" rule — no severity token, no bar). A missing field on a **present** card prints `n/a` in its own cell (rule 2: the field is absent, the card is real); a `null` verdict cell draws faint, the existing `Dash.hero` convention.
2. `src/dashboards/web/static/page-system.js` — `renderGpuSlot()` (lines 164–188): when `gpus` is empty, do **not** render the slot's fallback `n/a` content — render nothing (empty the slot's container and drop its hero styling so the row reflows to five heroes; the `data-k="gpus"` wrapper can stay as an empty, zero-footprint element or be removed — pick one and pin it). The multi-GPU case: one value grid per card, the cards' `GPU i` labels unchanged.
3. `src/dashboards/web/static/dash.css` — the `.gpu-values` grid: a compact 2×2 (or 4×1 on wide) of labelled cells reusing the existing token vocabulary (`.metric`/`.faint`/`.mono` shapes, no new palette — the app's colour rule), sized so a 20-character model string wraps rather than truncates.
4. `tests/test_api.py` — rewrite the six-heroes pin (line 1425): a host whose snapshot carries `gpus` still renders exactly six slots **in the existing order** (CPU, MEM, TEMP, DISK, GPU, NET); a host with no GPU renders exactly five with **no** `data-k="gpus"` slot (or the pinned empty variant) and no `n/a` in the hero row; the `gpu-values` cells (Model / VRAM / Pool / °C) are present in the served script. `tests/test_system_contract.py` and `tests/card_contract.py` stay green untouched (API fields unchanged).
5. `AGENTS.md` — (Documentation & Consistency pass) the `/system` paragraph's "a missing number or verdict — gateway's CPU temp, any host's absent GPU — prints `n/a` **uncoloured**" is split: the CPU-temp half keeps the `n/a` rule, the GPU half becomes the omission rule (the TODO is the decision), and the dense-sub-line description of the GPU slot is rewritten to the four-value layout.
## Testing & Quality
- Unit/integration: `tests/test_api.py` — the rewritten hero pin over the three captures (discrete GPU, APU, no-GPU) **and** over HTTP (the `self_test`-style app harness the contract tests use); `tests/test_system_contract.py` + `tests/card_contract.py` green untouched
- Coverage: **>90%** on this task's new/modified code
- Browser verification (invariant 10): `/system` against the live fleet — deskwork's Strix Halo card shows the four values at readable size (the TODO's quoted string is gone from the sub-line), gateway's card shows five heroes with no GPU slot and no `n/a`; the collapsed GPU charts still draw for the GPU hosts; screenshots in `data/logs/`
## Completion Criteria
- [ ] a GPU host's card shows Model / VRAM / Pool / °C as four labelled values; the dense sub-line string from the TODO is gone
- [ ] a GPU-less host renders five heroes with no GPU slot and no `n/a` in the hero row (served-bytes pin + screenshot)
- [ ] the GPU slot stays uncoloured (no `ok|warn|fail` token in the new markup — the phase-06 rule)
- [ ] `tests/test_api.py` + `tests/test_system_contract.py` + `tests/card_contract.py` green
- [ ] full test suite green
- [ ] no behavior change in completed work: the hero order on GPU hosts, the `gpu.busy_pct@N` history, the GPU chart slot create/destroy and the "More detail" disclosure are untouched
@@ -0,0 +1,27 @@
# Task 03 — The `/certs` agents table scrolls horizontally
**Phase:** `21_page_cleanups` · **Source:** `TODO.md:9` — "In the certs page I can't horizontal scroll in the agents table to see the certs installed on the hosts."
**Story:** n/a
## Objective
The `#agents-table` on `/certs` becomes horizontally reachable at phone width: its right-hand columns (the serial, expiry, `san` list and the Rotate / Revoke / bundle controls) are reachable by scrolling instead of being clipped. The repo's own mechanism is the `.scroll` modifier — `dash.css` gives `table.data` a scroll container **only** with that class (the same rule gap 12 measured on `/`'s fleet table), so this is the class, not a second mechanism. Row contents, the `COLUMNS` contract and every served pin are unchanged.
## Work
1. `src/dashboards/web/pages/certs.html` — line 81: `<table class="data" id="agents-table">` gains the `scroll` modifier (`class="data scroll"`), or is wrapped in `<div class="scroll">` — check which form `dash.css`'s selector actually matches (grep `\.scroll` in the file first; use the one it styles, do not invent a third).
2. `src/dashboards/web/static/dash.css` — if the `.scroll` selector matches `table.data.scroll`, nothing to add; if it matches a `div.scroll > table.data` wrapper, verify the wrapper's max-height/overflow behaviour against the card's padding so the scroll container does not clip the row's own bottom border (measure in a browser, do not deduce). No new token, no new palette.
3. `tests/test_api.py` — the served-HTML pin: `#agents-table` (or its wrapper, whichever shipped) carries the `.scroll` mechanism in the served bytes; the `COLUMNS`-resolution test (the one that reads the served `COLUMNS` table against a live `GET /api/certs` row in both directions) stays green untouched.
4. `src/dashboards/web/static/page-certs.js` — lines 55–56 (`table`/`body` lookups): no change expected — the id and the `tbody` selector are untouched by the wrapper/class; verify the page still renders its rows (a wrapper that breaks `getElementById('agents-table')` would be caught by the existing render path).
## Testing & Quality
- Unit/integration: `tests/test_api.py` — the pin above; the unchanged `COLUMNS` contract test
- Coverage: **>90%** on this task's new/modified code (one class / one wrapper; the gate applies to the changed test file)
- Browser verification (invariant 10): at 360 px on `/certs`, the table's scroll container is reachable — a CDP check that the container's `scrollWidth > clientWidth` and that scrolling to the end exposes the last column (the controls column), plus a screenshot mid-scroll in `data/logs/`; at 1280 px the table renders as before (no scrollbar on a wide viewport with three short rows)
## Completion Criteria
- [ ] at 360 px the `#agents-table` scrolls horizontally and its last column is reachable (screenshot in `data/logs/`)
- [ ] at 1280 px the table is visually unchanged
- [ ] `tests/test_api.py` green including the new pin and the unchanged `COLUMNS` test
- [ ] full test suite green
- [ ] no behavior change in completed work: the row contents, the status chips, the Rotate / Revoke / bundle controls and the `restart_required` outcome line behave exactly as before
@@ -0,0 +1,32 @@
# Phase 22 — /disks hides hosts without smartctl
**Source:** `TODO.md` L13 — "VMs that don't have smartctl enabled disks should not show up. Gateway shouldn't even be a host on the disks page."
**Story:** n/a
**Context:** `src/dashboards/api.py` (`disks_router()` at line 791 — `/api/disks/snapshot` answering one entry per configured host; the `/api/summary` disks block at lines 640–704 — `disks_hosts` built from `job.state.payload or _disks_stub(...)` at line 640, `fleet_disk_summary(disks_hosts)` at line 693; `_disks_stub()` at line 771 — `smartctl_available: None` for a never-answered host), `src/dashboards/sources/disks.py` (`smartctl_available()` at line 142 — the three-way flag: `True` a disk answered, `False` the agent said the binary is not installed, `None` an agent predating the pass or `--no-smart`), `src/dashboards/web/static/page-disks.js` (the per-host cards + the `fleet_disk_summary` header, hidden below two hosts), `tests/test_api.py` (the `/api/disks/*` pins — "one entry per configured host", the `smartctl_available` three-way, "with and without targets configured", the `fleet_disk_summary` key paths)
## Objective
A host whose agent answered `smartctl_available: false` (the binary is not installed — gateway's VM, no smartmontools behind `vda`) is not a host on the `/disks` page at all: it is absent from `/api/disks/snapshot`, from the fleet block and from `/api/summary`'s disks rows, and the fleet header counts only the hosts that are shown. The **three-way** flag itself is unchanged in `sources/disks.py` (the agent still reports it, the `/disks` page's `null`-case why-line still exists for `None` hosts, and an **unreachable** host keeps its dimmed stub row — a dead agent is a different fact from a missing binary, invariant 3). The filter is one API-level decision, so the source, the store and the health records see nothing new.
## Dependencies
- `21_page_cleanups` (todo) — both phases touch the `/disks` page markup (task 21.01 removes the page's `#pause` button); landing 21 first keeps this phase's diff confined to the host list.
## Tasks
1. `01_hide_smartctl_false_hosts.md` — the one API-level visibility gate (explicit `False` only) over `/api/disks/snapshot`, `/api/summary`'s disks block and the fleet summary; the pins rewritten
2. `02_verification.md` — browser + API verification against the live fleet: gateway absent, deskwork/driveripper intact, fleet header consistent, screenshots
## Testing & Quality
- Unit/integration: `tests/test_api.py` — the rewritten `/api/disks/snapshot` pin (one entry per configured host **whose agent can read its drives** — a `smartctl_available: false` payload is absent, a `None` payload is present, an `unreachable` stub is present), the `/api/summary` disks-block pin (fleet counts over the visible hosts, per-host rows for the visible hosts only), the unchanged `smartctl_available` three-way pin in `sources/disks.py`'s own tests (`tests/test_disks.py` or wherever the flag is pinned — grep `smartctl_available` in `tests/` first)
- Coverage: **>90%** on new/modified code
- The phase is "Complete" only when the **entire** project test suite passes 100% (every test in the project, not just this phase's)
## Documentation & Consistency
- The final pass reviews `README.md` and `AGENTS.md` in full against the project as it stands after this phase — fixing every inconsistency found, **including pre-existing ones not caused by this phase**. AGENTS.md's `sources/disks.py` bullet (the three-way flag's `False` arm — "so the page can say *install smartmontools*") and the `/disks` UI paragraph (the `.banner.info` why-line for `false`) are rewritten: the `false` arm's page-side consequence is now *absence*, and the three-way flag keeps its agent-side meaning; the "gateway answers `smartctl_available: false`" fleet fact gains the "and is therefore not a host on the disks page" clause.
## Completion Criteria
- [ ] `/api/disks/snapshot` and `/api/summary`'s disks block carry no entry for a host whose agent answered `smartctl_available: false`; a `None` host and an `unreachable` stub both remain (pinned both ways in `tests/test_api.py`)
- [ ] the `/disks` page on the live fleet shows deskwork + driveripper only, the fleet header counts 2 hosts, and the per-host cards are unchanged (screenshot in `data/logs/`)
- [ ] `uv run pytest` green, coverage not down, `ruff` + `pyright` clean
- [ ] `bash .agents/validate.sh` green
- [ ] no behavior change in completed work: the agent's SMART pass, the `disks:<host>` job's health record, the stored `disk.*` series, the `None`-case why-line and the unreachable stub are all untouched
@@ -0,0 +1,30 @@
# Task 01 — The one visibility gate: `smartctl_available` is `False` only
**Phase:** `22_disks_hide_unconfigured` · **Source:** `TODO.md:13` — "VMs that don't have smartctl enabled disks should not show up. Gateway shouldn't even be a host on the disks page."
**Story:** n/a
## Objective
One API-level helper decides disk-page visibility: a host entry is shown when its payload's `smartctl_available` is **not explicitly `False`** — `True` (a disk answered) and `None` (nobody knows: an agent predating the pass, `--no-smart`, or a never-answered stub) stay, `False` (the agent said the binary is not installed) does not. The gate runs over the assembled per-host list in both endpoints, so `fleet_disk_summary()` (which sums whatever it is handed) sees only the visible hosts and its `hosts_total`/`hosts_live` count exactly the page's rows. `sources/disks.py` keeps the three-way flag verbatim — the agent still reports it and the `None`-case why-line still renders.
- ASSUMPTION: **explicit `False` only** is the hide condition. `None` stays visible (an agent that cannot say is not an agent that said *no* — the three-way flag's whole point), and an `unreachable` stub stays visible (invariant 3's rule: a dead agent must not read as "never configured"; its `smartctl_available` is `None` by `_disks_stub`'s shape anyway). The operator's words — "don't have smartctl **enabled**" / "shouldn't **even be a host**" — are the `false` arm; if a `None` host should also vanish, that is a one-clause change to the helper, not a redesign.
## Work
1. `src/dashboards/api.py` — add one small helper next to `_disks_stub` (line 771): `_disk_visible(host: Mapping) -> bool` — `host.get("smartctl_available") is not False` (the explicit-`False` check, spelled out so a `None` payload passes; a docstring naming the three arms).
2. `src/dashboards/api.py` — `/api/disks/snapshot` (in `disks_router()`, line 791): filter the assembled per-host list through `_disk_visible` before the response is built (the response shape is unchanged — a shorter list, same entry shape).
3. `src/dashboards/api.py` — the `/api/summary` disks block (lines 640–704): `disks_hosts` (line 640) is filtered through the same helper **before** `fleet_disk_summary(disks_hosts)` (line 693) and before the per-host row loop (lines 695–704), so the fleet counts and the rows agree with `/api/disks/snapshot` by construction — one helper, two call sites, no second copy of the predicate.
4. `tests/test_api.py` — rewrite the affected pins: the `/api/disks/snapshot` "one entry per configured host" test gains a `smartctl_available: false` host (a `disks:<host>` job whose payload the test harness holds, the `self_test`-style swap the `nut` job's pin uses) and asserts it is **absent** while a `None` host and an `unreachable` stub are **present**; the `/api/summary` disks-block pin asserts the fleet counts over the visible hosts only (e.g. three configured hosts, one `false` → `hosts_total: 2`); the "without targets configured" leg stays green untouched.
5. `tests/` — locate the `smartctl_available` three-way pin (grep `smartctl_available` in `tests/`; `tests/test_disks.py` or the agent/disks source tests) and confirm it is **untouched** by this change (the flag's computation is unchanged; only the API's reading of it gained a gate).
## Testing & Quality
- Unit/integration: `tests/test_api.py` — the pins above, both directions (a `false` host absent from **both** endpoints; a `None`/`unreachable` host present in both); the unchanged `fleet_disk_summary` unit tests in `tests/` (the function is untouched — it sums what it is handed)
- Coverage: **>90%** on this task's new/modified code
## Completion Criteria
- [ ] `/api/disks/snapshot` and `/api/summary` agree on the visible host set, and neither lists a `smartctl_available: false` host (pinned)
- [ ] a `None` host and an `unreachable` stub both render in both endpoints (pinned)
- [ ] `fleet_disk_summary` counts exactly the visible hosts (pinned)
- [ ] `uv run pytest` green, `ruff` + `pyright` clean
- [ ] full test suite green
- [ ] no behavior change in completed work: `sources/disks.py`, the `disks:<host>` job, the stored `disk.*` series and the health records are byte-identical in behaviour
@@ -0,0 +1,25 @@
# Task 02 — Live verification: gateway gone, the rest intact
**Phase:** `22_disks_hide_unconfigured` · **Source:** `TODO.md:13` — "VMs that don't have smartctl enabled disks should not show up. Gateway shouldn't even be a host on the disks page."
**Story:** n/a
## Objective
The gate is proven on the live fleet, read-only: the dev server (invariant 10: `./scripts/dev-server.sh restart`, sign in with `testpass`) answers `/api/health` with all nine sources `ok`, the `/disks` page shows deskwork and driveripper only, and every number that used to render still renders. No fleet change is made — this task dials the fleet and screenshots it.
## Work
1. `./scripts/dev-server.sh restart` — the dev server up on http://127.0.0.1:8090; `./scripts/dev-server.sh status` — the nine-source table: `host:driveripper`, `host:deskwork`, `host:gateway`, `disks:driveripper`, `disks:deskwork`, `disks:gateway`, `nut`, `llm:gufo`, `llm:vllm` all `ok` (gateway's **agent** still answers — the job is `ok`; only the *page* hides it, which is the point: the `disks:gateway` health record must not be touched by the API filter, and `status` is the proof).
2. Signed-in `GET /api/disks/snapshot`: no `gateway` entry, `deskwork` and `driveripper` present with their `smartctl_available: true` payloads; `GET /api/summary`: the disks block's `hosts_total` is 2, the per-host rows are the two, and the `system` block still lists all three hosts (the filter is `/disks`-scoped only — a `system` host list that lost gateway would be the over-reach the pin must catch).
3. Browser on `/disks` (invariant 10): the fleet header reads the two-host sum; gateway has no card and no why-line; deskwork's `nvme0n1` and driveripper's 16 drives render exactly as before the phase (the live SMART numbers from the fleet record); a screenshot in `data/logs/`.
4. The `None`-arm check: no live host is currently `None` (all three agents post-date the SMART pass), so the `None` arm is proven by the task-01 pins only — state that in the notes rather than fabricating a live case.
## Testing & Quality
- Coverage: n/a (verification task; the gate's coverage is task 01's)
- The read-only rule: no `POST` to the fleet, no `upsd` command, no unit change — a live test that writes is not a test (AGENTS.md testing policy)
## Completion Criteria
- [ ] `./scripts/dev-server.sh status` — nine sources `ok`, including `disks:gateway` (the job is healthy; only the page hides it)
- [ ] `/api/disks/snapshot` has no gateway entry; `/api/summary`'s system block still lists gateway (screenshot or transcript in `data/logs/`)
- [ ] the `/disks` page shows the two live hosts with their pre-phase numbers (screenshot in `data/logs/`)
- [ ] the full suite + `bash .agents/validate.sh` green
@@ -0,0 +1,34 @@
# Phase 23 — LLM rate responsiveness
**Source:** `TODO.md` L8 — "The LLM monitor has a strange delay and a 'ramp up/ramp down' effect that makes it look innacurate. Data reporting should be accurate to as small a delta as possible."
**Story:** n/a
**Context:** `src/dashboards/app.py` line 122 (`build_hub` builds **one** `RateEstimator(span=settings.rate_window)` and hands it to both `host_jobs` (line 124) and `llm_jobs` (line 130)), `src/dashboards/config.py` lines 197–199 (`llm_interval` 2 s, `llm_timeout` 3 s, `rate_window` 10 s), `src/dashboards/collectors.py` line 477 (`llm_jobs(settings, client, rates)` — the estimator is a parameter, so a second one is a wiring change, not a signature redesign), `src/dashboards/rates.py` (line 55 `MIN_WINDOW_SECONDS = 0.5`, line 61 `MAX_WINDOW_RATIO = 2.0`, the phase-13 jump gate — all untouched by this phase), `src/dashboards/web/static/page-llm.js` (line 39 `pollMs: 2000`, the `Δ<window>` labels printed from the payload's `window`), `src/dashboards/sources/llm.py` (the rate cells carry `computed:Δ<series>/Δt` + the honest `Δ<window>`; the window the estimator *used* travels with the value — rule 2), the mechanism the TODO describes (gap 9 / phase 13 in AGENTS.md: a 10 s window over a 2 s poll ramped the hero up over ~5 polls when a slot started and decayed over ~5 polls when it ended — the "ramp up/ramp down" the operator sees; the sub-8× skew family and the stall are **already** closed by phases 13/14 and stay closed: this phase changes the window size, nothing else)
## Objective
The LLM throughput rates are measured over a window as small as the app's own cadence allows — the LLM source gets its **own** `RateEstimator` whose span is the LLM poll interval (2 s), so a slot start reads steady within two polls instead of ramping over ten, while the `host` source keeps the 10 s `RATE_WINDOW_SECONDS` window its 10 s cadence needs (a 2 s window over a 10 s poll would refuse every window — the estimator's floor and ceiling would answer `n/a` forever). The phase-13 protections (the 0.5 s floor, the `2 × span` ceiling, the 8× jump gate with its independent-anchor confirmation) apply to the smaller window unchanged; the `Δ<window>` label prints the smaller window it actually used, so the number always says what it is.
## Dependencies
- — (none) — the phase touches `app.py`/`collectors.py` wiring and the LLM page; it is scheduled after `22_disks_hide_unconfigured` only to keep the fleet-changing phase 24 last.
## Tasks
1. `01_llm_window_per_poll.md` — the second estimator: `build_hub` hands `llm_jobs` a `RateEstimator(span=settings.llm_interval)`, the host keeps the 10 s one
2. `02_window_pins.md` — the pins: LLM rate cells label the 2 s window, host cells keep 10 s, the phase-13 gates still hold over the smaller window
3. `03_live_verification.md` — the live proof: a slot start reads steady within two polls (no 10 s ramp), the `Δ2.0s` label travels with the value, screenshots
## Testing & Quality
- Unit/integration: `tests/test_llm.py` (or wherever the `parse_gufo_metrics`/`parse_vllm_metrics` tests live — grep `rate_window`/`RateEstimator` in `tests/` first) — the window-value pins for both sources, the phase-13 tests (floor refusal, ceiling, jump gate, the stale-anchor tests) run **unchanged** and green; `tests/test_api.py`'s `/api/llm/snapshot` shape pins stay green (the shape is unchanged — only the `window` values move)
- Coverage: **>90%** on new/modified code
- The phase is "Complete" only when the **entire** project test suite passes 100% (every test in the project, not just this phase's)
## Documentation & Consistency
- The final pass reviews `README.md` and `AGENTS.md` in full against the project as it stands after this phase — fixing every inconsistency found, **including pre-existing ones not caused by this phase**. AGENTS.md's `app.py`/`poller.py` bullets (the "one `RateEstimator`" sentence) and `rates.py`'s docstring context gain the two-estimator fact; gap 9's "the estimator no longer turns a slow fetch into 0.2 s arithmetic" history stays — this phase is the window-size decision on top of it, and the `RATE_WINDOW_SECONDS=10` line in `.env.example` stays (it is the host's knob now, and gains a comment saying so if it has none).
## Completion Criteria
- [ ] `/api/llm/snapshot` rate cells carry a `window` ≈ 2 s (the label prints `Δ2.0s` on the page) and `/api/system/snapshot` rate cells keep ≈ 10 s (pinned both ways)
- [ ] the phase-13 tests (floor, ceiling, jump gate, stale anchor) pass unchanged
- [ ] live: a slot start reads steady within two polls and a slot end decays within two — the 10 s ramp is gone (screenshots + the stored 1 m rollup in `data/logs/`)
- [ ] `uv run pytest` green, coverage not down, `ruff` + `pyright` clean
- [ ] `bash .agents/validate.sh` green
- [ ] no behavior change in completed work: the host rates, the estimator's gates, the rollups and the `Δ<window>` labelling are all unchanged in kind
@@ -0,0 +1,28 @@
# Task 01 — The LLM source gets its own 2 s-window estimator
**Phase:** `23_llm_rate_responsiveness` · **Source:** `TODO.md:8` — "The LLM monitor has a strange delay and a 'ramp up/ramp down' effect that makes it look innacurate. Data reporting should be accurate to as small a delta as possible."
**Story:** n/a
## Objective
`build_hub` builds **two** estimators: the host's keeps `RateEstimator(span=settings.rate_window)` (10 s — its 10 s cadence needs the wide window; a 2 s span over a 10 s poll would sit the window-empty fallback against the `2 × span` ceiling and answer `n/a` every poll), and the LLM's is `RateEstimator(span=settings.llm_interval)` (2 s — one poll interval: the smallest window the 2 s cadence can honestly fill). `llm_jobs` (which already takes the estimator as a parameter, `collectors.py` line 477) receives the LLM one; `HostCollector` keeps the 10 s one. No new env var: the LLM window **tracks** `LLM_INTERVAL_SECONDS`, so the one existing knob stays the single source of truth for the LLM cadence-and-window pair.
- ASSUMPTION: **no new env var** — `RATE_WINDOW_SECONDS` (10, `.env.example` line 85) becomes the host's knob and the LLM window is `llm_interval` by construction. A 2 s window is 4× above the 0.5 s `MIN_WINDOW_SECONDS` floor, so the floor never engages on a healthy cadence; the 8× jump gate stays engaged (a real burst from idle still holds one poll for confirmation), and the poll cadence itself is **unchanged** — gufo and vLLM see exactly the same request rate as before (the TODO asks for a smaller *delta*, not more load on the engines).
## Work
1. `src/dashboards/app.py` — `build_hub` (lines 122–130): `rates` becomes `host_rates = RateEstimator(span=settings.rate_window)`; new `llm_rates = RateEstimator(span=settings.llm_interval)`; `host_jobs(..., HostCollector(host_rates), ...)` unchanged in shape; `llm_jobs(settings, client, llm_rates)` takes the new one. The two estimators are keyed on disjoint series names (rule 2's key is the resolved name + label set), so one shared object was never load-bearing — but the docstring/comment at line 122 (if it says "one estimator for all sources") is rewritten to the two-estimator shape.
2. `src/dashboards/collectors.py` — line 477: no signature change expected (the parameter is already there); verify the LLM jobs thread `rates` through to `parse_gufo_metrics`/`parse_vllm_metrics` (they already take it, `sources/llm.py` lines 189/243) — if any LLM path constructs its own estimator, it takes the injected one (one estimator per source, the phase's invariant).
3. `src/dashboards/config.py` — no change expected (the knobs exist); if a comment on `rate_window` (line 199) implies it is the one global knob, it gains the "host's knob; the LLM window is `llm_interval`" clause.
4. `tests/` — the `build_hub`-level test (grep `build_hub` in `tests/` — the `tests/test_api.py` wiring pins) gains one assertion pair: the estimator the LLM jobs receive has `span == settings.llm_interval` and the host collector's has `span == settings.rate_window` (spy the two objects, the way the `agent_tls_map` pin spies its entry).
## Testing & Quality
- Unit/integration: the wiring pin above; the existing LLM parse tests (fixture-driven, `tests/fixtures/`) run with the injected estimator — their assertions on the `window` value, if any hard-code 10.0, are updated to the new 2.0 (grep `10.0`/`window` in `tests/test_llm.py` first); the host parse tests keep 10.0
- Coverage: **>90%** on this task's new/modified code
## Completion Criteria
- [ ] `build_hub` hands the LLM jobs a 2 s-span estimator and the host a 10 s-span one (pinned by the spy)
- [ ] the fixture-driven LLM tests pass with the 2 s window; the host tests keep 10 s
- [ ] `uv run pytest` green, `ruff` + `pyright` clean
- [ ] full test suite green
- [ ] no behavior change in completed work: the host's measured rates, the estimator's gates and the engines' poll cadence are untouched
@@ -0,0 +1,27 @@
# Task 02 — The pins: 2 s LLM window, 10 s host window, gates intact
**Phase:** `23_llm_rate_responsiveness` · **Source:** `TODO.md:8` — "…Data reporting should be accurate to as small a delta as possible."
**Story:** n/a
## Objective
The window decision is pinned on the surface that shows it, in both directions (the repo's testing-policy rule): a LLM rate cell answers its `window` ≈ the LLM interval with the `Δ<window>` label traveling with the value, a host rate cell keeps ≈ 10 s, and the phase-13 gate tests — the 0.5 s floor refusal, the `2 × span` ceiling, the 8× jump hold with its independent-anchor confirmation, and the stale-scrape-body regression — run **unchanged** and green, because they are estimator properties, not window-size properties.
## Work
1. `tests/test_llm.py` (or the file the LLM parse tests live in — grep first) — the two window pins: (a) a gufo metrics scrape driven through `parse_gufo_metrics` with a `RateEstimator(span=2.0)` (the `build_hub` shape, task 01's) answers the rate cell with a `window` in `(0.5, 2.0]` on a healthy 2 s cadence and the cell's `source` string still reads `computed:Δ<series>/Δt`; (b) the same test shape with `span=10.0` on the host's series (or the existing host fixture tests) keeps its `window` ≈ 10 s — the two sources' windows are **different numbers in the same payload shape**, and a pin that asserts only one of them is the gap this task closes.
2. `tests/test_rates.py` (or wherever the phase-13 tests live) — run them unchanged: the floor test (a sub-0.5 s window answers `(None, dt)`), the ceiling test (`2 × span`), the jump-gate test (≥8× holds one poll, publishes on a *different-anchor* confirmation) and the two stale-body regression tests. If a test constructs its estimator with an explicit `span` that now disagrees with the app's shape, it is **not** edited to match — the estimator's constants are app-independent and the tests' own spans are theirs (the app's shape is task 01's pin's job).
3. `tests/test_api.py` — the `/api/llm/snapshot` shape pins stay green (the shape is unchanged; only `window` values move). If a pin hard-codes a `window` number in the LLM payload, it is the one place it is updated — to a range assertion (`0.5 < window <= 2.0 × llm_interval`), not a literal, so a future cadence change does not rot it.
4. `tests/test_config.py` — confirm `rate_window`'s pin (the 10 s default) stays; no new setting was added, so no new pin is needed (the "no new env var" ASSUMPTION is what the absence of a new setting is).
## Testing & Quality
- Unit/integration: the four pin groups above
- Coverage: **>90%** on this task's new/modified code
## Completion Criteria
- [ ] a LLM rate cell's `window` is pinned to the 2 s shape and its label prints `Δ2.0s` (or the honest sub-window when the floor engages)
- [ ] a host rate cell's `window` is pinned to the 10 s shape
- [ ] the phase-13 gate tests pass unchanged
- [ ] `uv run pytest` green, `ruff` + `pyright` clean
- [ ] full test suite green
- [ ] no behavior change in completed work: the payload shapes, the `source` strings and the estimator constants are untouched
@@ -0,0 +1,27 @@
# Task 03 — Live verification: the ramp is gone
**Phase:** `23_llm_rate_responsiveness` · **Source:** `TODO.md:8` — "The LLM monitor has a strange delay and a 'ramp up/ramp down' effect that makes it look innacurate."
**Story:** n/a
## Objective
The smaller window is proven on the live engines, read-only: with the dev server up (invariant 10) and a decode workload running on the engines, the `/llm` hero reaches steady state within **two** polls of the slot starting (pre-phase it ramped over ~five) and decays within two of it ending, every value carrying its `Δ2.0s` label, and the stored 1 m rollups staying in the fleet's measured band (gap 9's pass line: anything above ~300 tok/s is arithmetic, not work). No engine is written to — the workload is whatever the operator's engines are doing, or a deliberately short local decode the operator starts; a live test that loads a shared engine is not a test.
## Work
1. `./scripts/dev-server.sh restart`, sign in with `testpass`; `./scripts/dev-server.sh status` — the nine-source table `ok` (a restart also proves the two-estimator wiring survives a real boot, not just the in-memory test store).
2. Signed-in polling of `/api/llm/snapshot` at 2 s (a short script, the `scripts/watch-store-perf.py` pattern): capture the decode tok/s cell's `value` + `window` for ~60 s while a slot is busy, then again for ~30 s after it frees. The pass shape: within two polls of the transition the value is within 8× of steady (the jump gate's own ratio — a held value is the designed behaviour, and a *confirmation* within the next poll is the honest publish; a value still ramping at poll 5 is the pre-phase defect returning).
3. The `Δ` label check (gap 9's rule, restated): **zero** values printed without their `Δ<window>` label; the label reads the window actually used (`Δ2.0s` on a healthy cadence, the honest sub-window — never a bare number — when the floor or the stall engages).
4. The stored-rollup check (the phase-14 rule): the 1 m rollups of `llm.decode_tokens_per_second` for the window under test top out in the fleet's measured band (gap 9: 119.9 gufo / 171.7 vLLM were the real maxima; the 300 tok/s pass line stands) — a 1 m average set by a ramp would be the arithmetic this phase exists to remove.
5. Screenshots of `/llm` mid-burst and post-free in `data/logs/` (invariant 10), plus the captured series as a TSV beside them (the `data/logs/llm-decode-*.tsv` precedent).
## Testing & Quality
- Coverage: n/a (verification task; the pins are task 02's)
- The read-only rule: no `POST` to the fleet, no `upsd` command, no engine write — observing a workload the engines are already doing is the test
## Completion Criteria
- [ ] a slot start reads steady (within 8× of steady) within two polls, and a slot end decays within two — the 10 s ramp is gone (captured series in `data/logs/`)
- [ ] zero values printed without their `Δ<window>` label, the label reading the window actually used
- [ ] the 1 m rollups stay in the measured band (nothing above the 300 tok/s pass line)
- [ ] screenshots in `data/logs/`
- [ ] the full suite + `bash .agents/validate.sh` green
@@ -0,0 +1,36 @@
# Phase 24 — Dev mTLS: no non-mTLS agent, ever
**Source:** `TODO.md` L3 — "The client should never allow non-mtls communication. deskwork and gateway are still non-mtls according to the dashboard. During dev, the dev server should generate certs in a non-committed directory and then distribute them to clients during registration so we never need non-mtls servers."
**Story:** n/a
**Context:** phase 15 (complete) — the durable CA in `meta` (`certs.py`: five `certs.*` rows, the bundle stored verbatim), the agent's TLS mode (`agent/host_agent.py` 1.1.0: `--cert-bundle`, `CERT_OPTIONAL`, the live `# REVOKED` denylist, `agent.tls`/`agent.bundle_rev` in the snapshot), the collector's certificate dial (`collectors.py` `AgentTls` — an `https://` `HOST_TARGETS` entry dials with this app's client certificate), the bundle download route (`GET /api/certs/<name>/bundle`, session-gated, the stored bytes verbatim), the installer's `--cert-bundle` shape check (`deploy/install-agent.sh` lines 270–331 — the python3 validator runs **before** `need_root`, the bundle installs `0640 root:dashboards-agent` at line 385, the `--status` https probe), and the fleet record (AGENTS.md gap 11: no bundle is installed on any host, `HOST_TARGETS` is `http://` for every host, the installed agents run 1.0.0 so **1.1.0 ships together with the bundle, never the bundle alone**; gap 10: driveripper's unit is reproducible only with the repo script's `--as-root --caps CAP_SYS_RAWIO CAP_SYS_ADMIN --force` line — the Oct-18 sudo-granted snapshot must not be run; deskwork's `ops` has no installer grant, so its install goes through the operator's root). `scripts/dev-server.sh` (the dev server: reads `.env.dev`, nohup `dashboards`, `/api/health` readiness probe, `status` subcommand) and `data/` (gitignored — the non-committed directory the TODO names)
## Objective
The dev loop stops producing `http://` agents. On start, the dev server exports every configured host's stored bundle to non-committed `data/certs/` (the CA is already durable in `meta` — "generate" means *materialise what the first boot minted*, never a second authority), and the installer gains `--fetch-bundle` so a client's **registration** — its install — pulls its bundle from the dashboard and installs it in the same run (the operator no longer curls a file across the network by hand). Then the fleet itself moves: agent 1.1.0 + bundle on deskwork, driveripper and gateway, `HOST_TARGETS` flipped to `https://` host by host (a URL flipped before its bundle is installed turns that host red by construction — the order is the safety), leaving the dashboard's nine sources `ok` with three `agent.tls: true` hosts and no `http://` agent URL anywhere.
## Dependencies
- — (none) — the phase builds on complete phase 15 and phase 17; it is scheduled last on purpose, because it is the fleet-changing one.
## Tasks
1. `01_dev_bundle_export.md` — `scripts/dev-certs-export.py` + `dev-server.sh`: the dev server writes every configured host's bundle to `data/certs/<name>.bundle` (`0600`) on start; an unissued host says so and writes nothing
2. `02_installer_fetch_bundle.md` — `install-agent.sh --fetch-bundle <url> --cert-name <name> --session <token>`: registration pulls the bundle from the dashboard, the shape check runs on the fetched bytes before `need_root`, the token never reaches the token file or the unit
3. `03_fleet_rollout.md` — agent 1.1.0 + bundle on the three hosts (deskwork via the operator's root, driveripper with the gap-10 `--force` line, gateway with its sudoers grant), `HOST_TARGETS` flipped per host, the URL never flipped before the bundle
4. `04_verification.md` — the fleet is mTLS: `agent.tls`/`deployed_bundle_rev` agree for all three hosts, nine sources `ok`, no `http://` agent URL, the 16-driver SMART reads still live, screenshots
## Testing & Quality
- Unit/integration: `tests/test_deploy_scripts.py` — the export helper against a `create_app(...)` hosted on a socket (the sandbox pattern the installer tests already use: fake `install`/`chown` where the installer runs); the `--fetch-bundle` legs (a good bundle fetched, installed and byte-identical; a malformed one refused **before any write**; the token absent from every written artefact)
- Coverage: **>90%** on new/modified code
- The phase is "Complete" only when the **entire** project test suite passes 100% (every test in the project, not just this phase's)
## Documentation & Consistency
- The final pass reviews `README.md` and `AGENTS.md` in full against the project as it stands after this phase — fixing every inconsistency found, **including pre-existing ones not caused by this phase**. The README's *Protecting an agent, host by host* rollout becomes the `--fetch-bundle` recipe (the manual curl is demoted to the fallback), the dev section gains the `data/certs` export, AGENTS.md's fleet record and gap 11 resolve to "all three hosts mTLS since <date>" with the exact installed units, and the "bearer-token transport is the compatibility path, not the designed one" sentence becomes history. **No token, no PEM and no session value go into any tracked file** — the exported bundles live in gitignored `data/`, the fleet's `HOST_TARGETS` URLs carry no userinfo once flipped (invariant 6), and the task transcript names hosts, not credentials.
## Completion Criteria
- [ ] `./scripts/dev-server.sh restart` leaves `data/certs/<name>.bundle` for every configured host that has an issued bundle, `0600`, byte-identical to the stored row, and a one-line note for any unissued host (pinned + verified)
- [ ] `sudo deploy/install-agent.sh --fetch-bundle <url> --cert-name <host> --session <token> ...` installs a byte-identical bundle in the sandbox, refuses a malformed one before `need_root`, and leaves no token in the token file, the unit or the log (pinned)
- [ ] all three hosts run agent 1.1.0 from an installed bundle, `HOST_TARGETS` is `https://` for every host with no userinfo, and `./scripts/dev-server.sh status` reads nine sources `ok` with `agent.tls: true` and `deployed_bundle_rev == bundle_rev` on all three (screenshots of `/certs` in `data/logs/`)
- [ ] driveripper's 16 drives still read live on `/disks` after the `--force` unit rewrite (the gap-10 unit shape is reproduced exactly)
- [ ] `uv run pytest` green, coverage not down, `ruff` + `pyright` clean
- [ ] `bash .agents/validate.sh` green
- [ ] no behavior change in completed work: the token-mode path, the revocation wiring, the bundle route and the agent's token-mode bytes are all unchanged
@@ -0,0 +1,27 @@
# Task 01 — The dev server exports the bundles to `data/certs/`
**Phase:** `24_dev_mtls_rollout` · **Source:** `TODO.md:3` — "During dev, the dev server should generate certs in a non-committed directory and then distribute them to clients during registration…"
**Story:** n/a
## Objective
On start, the dev server materialises the **stored** CA's per-host bundles in non-committed `data/certs/`: one `0600` file per configured host with an issued bundle (`data/certs/<name>.bundle`, byte-identical to the `meta` row the bundle route serves — the CA is durable in `meta`, so "generate" is an export of what the first boot already minted, never a second authority; `data/` is gitignored, so nothing committed ever carries a key). A configured host with **no** issued bundle (the invariant-3 shape: `issued: false`) writes no file and prints one line naming the host and the `/certs` route that issues it. The export is a small standalone script the dev server calls, so the test suite drives it against a `create_app(...)` hosted on a socket without a nohup'd process.
## Work
1. `scripts/dev-certs-export.py` — a standalone helper (stdlib `urllib`, the way `install-agent.sh`'s validator uses `python3`): arguments `--url` (the dev server's base URL), `--password` (the sign-in password — the dev server reads it from the same `.env.dev` it already loads, so the value comes from the environment, never a tracked file), `--names` (the configured host names, space-separated — read from `HOST_TARGETS`' names, the same parsing `parse_targets` does, passed in by the caller rather than re-parsed inside), `--out-dir` (defaults to `<root>/data/certs`). Behaviour: sign in via `POST /api/auth/login` (the public route; the token travels in `Authorization` for the bundle GETs), then per name `GET /api/certs/<name>/bundle` — a `{"configured": true, "issued": false}` answer writes **no file** and prints the one-line note; a configured host that 404s (`{"configured": false}`) is impossible by construction (names come from `HOST_TARGETS`) but is handled as the same no-file line, not a crash; a good bundle is written `<out-dir>/<name>.bundle` at `0600` (created via `os.open`, not `open()`, so the mode is the file's birth mode). The script exits 0 when every name resolved to either a file or the no-issued note, and non-zero on a sign-in failure — a dev server that cannot sign in says so in `dev-server.sh`'s start output, never silently.
2. `scripts/dev-server.sh` — in `start()`, after the readiness probe: call the export (`.venv/bin/python scripts/dev-certs-export.py --url http://127.0.0.1:$PORT --password "$DASH_ADMIN_PASSWORD" --names <the HOST_TARGETS names> --out-dir $ROOT/data/certs`), printing its per-host lines; in `status`, print the `data/certs/` listing (names + sizes) as the "bundles exported" block. The export failure is a **warning line**, not a `start` failure — a bundle export that cannot sign in is a dev-convenience failure, and the server itself is up (the readiness probe already proved it); the one line says what to check.
3. `tests/test_deploy_scripts.py` — the pin: host `create_app(settings with admin_password="testpass", pbkdf2_iterations=1000, one or two configured targets, an in-memory store with an issued bundle for one host and none for the other)` on `127.0.0.1:0` via `uvicorn` (the project's own dependency — `__main__.py`'s pattern), run the script against it, and assert: the issued host's file exists at `0600` and is **byte-identical** to `GET /api/certs/<name>/bundle` from the same app; the unissued host wrote **no file** and the output carries its one-line note; a wrong `--password` exits non-zero and writes nothing.
## Testing & Quality
- Unit/integration: the pin above (the sandbox pattern `tests/test_deploy_scripts.py` already uses for the installer)
- Coverage: **>90%** on this task's new/modified code
- Real-boot check: `./scripts/dev-server.sh restart` on the dev box prints the per-host export lines and `data/certs/` holds the files (the dev box is the test bed — invariant 10)
## Completion Criteria
- [ ] `./scripts/dev-server.sh restart` leaves `data/certs/<name>.bundle` per issued host, `0600`, byte-identical to the stored row; an unissued host is a one-line note and no file (pinned + verified on the dev box)
- [ ] `status` prints the exported-bundles block
- [ ] no bundle, key or PEM is ever written outside `data/` (the gitignored tree) — a grep of the tracked files for the export's output path confirms it lands only under `data/`
- [ ] `uv run pytest` green, `ruff` + `pyright` clean
- [ ] full test suite green
- [ ] no behavior change in completed work: the dev server's start/stop/status semantics, the readiness probe and the CA rows in `meta` are untouched (an export never mints — `certs.read_state` is the request-time read the bundle route uses, and the export dials that route, not `ensure_certs`)

Some files were not shown because too many files have changed in this diff Show More