Commit Graph
39 Commits
Author SHA1 Message Date
ducoterra 883e36c727 phase: 35_llm_equal_window_heights
**Phase 35 final verification pass — complete.**

- Both task files (01, 02) already in `complete/`; verified the work in-tree: `#engines` = `grid cols-2`, `.strip` has no `align-items` (keeps `display: grid`, `gap: .85rem`, `auto-fit minmax(min(340px, 100%), 1fr)`), `.grid.items-start` rule + corrected comment intact, rewritten pin `test_llm_engine_windows_stretch_to_a_common_row_height` in place.
- Live re-verified in browser (fresh `35-llm-verify.cjs` run, screenshots re-captured): both engines answering; strip cards 456 px / panel cards 422 px with flat equal bottoms; row gaps 13.6 px (uniform `.85rem`); `n/a` cells preserved; 360 px viewport single-column, no overflow/clip. `/system` no-regress: `grid auto items-start`, expanded disclosure does not stretch collapsed siblings (325.4 → 325.4).
- Completion criteria: served `/llm` markup ✓ · live equalized rows + screenshot ✓ · `/system` bytes unchanged ✓ · full suite/coverage/lint ✓ · README/AGENTS.md ✓ · no behaviour change (all existing pins pass) ✓.
- Tests: `bash .agents/validate.sh` → **exit 0**: `uv run pytest --cov` **2031 passed, 19 skipped** (15 live + 4 store_perf, opt-in), coverage **98.71%** (94% floor reached); `ruff` all checks passed; `pyright` 0 errors; `node --check` + repo hygiene clean.
- Live suite (`--live`, read-only): 4/15 pass (LLM engines); 5 fail solely on host-agent/upsd **connectivity timeouts** (9120/3051 unreachable from this machine — same errors as `dev-server.sh status`); environmental, no code defect. No live changes made (all verification read-only).
- Docs: README.md reviewed in full — verified quickstart, installer/agent flags, config defaults, Containerfile facts, script flags, fixtures, test files, polling cadences: **consistent, no changes needed**. AGENTS.md: fixed one stale statement (hermetic skip count "12 live" → "15 live"); nothing else stale.
- Dev server up (pid 811266), `/api/health` 200; hosts/upsd DOWN is the pre-existing environmental state, LLM source ok.
- Next pending phase: **none** — `todo/` contains only this phase (35).
2026-11-02 21:25:13 -05:00
ducoterra 219d7a7c3b phase: 34_radiance_llm_engine
Everything is verified and green. Final report:

**Phase 34 final verification pass — complete**

- Verified all 6 completed tasks' work against the live fleet and hermetic suite: `llm:radiance` job registered, `llm_engines()` registry drives 404 gate / history defaults / `/api/config` (additive, byte-identical gufo/vllm keys), radiance field map + `/stats` sidecar + 7-position spec curve, `/llm` strip/curve/rows.
- **Defect found & fixed (pre-existing, task 05):** the vLLM-only sentence "vLLM exposes no per-slot JSON…" rendered on the **radiance** panel (`slotSection`'s non-gufo branch). Fix: vLLM line gated to `engine === 'vllm'`; radiance's panel slot section renders nothing (its session home is the strip) — `page-llm.js` + the served-byte pin in `tests/test_api.py` (now asserts the radiance branch is empty and the sentence can't leak). Browser-verified; re-shot `data/logs/34-llm-radiance-rest.png` (the 4.5-min wait found no in-flight request, so the busy shot remains the earlier pass's — its strip rows still stand; only the panel text predates the fix).
- Tests: `uv run pytest` → **2031 passed, 19 skipped** (100%); `uv run pytest --cov` → **98.73%** (floor 94%; config 100%, llm 99%, collectors 98%, api 99%); `uv run ruff check src tests agent && uv run pyright` → clean; `node --check` OK; `uv pip list --outdated` → empty.
- `uv run pytest -m live --live` → **13 passed, 2 skipped** (vLLM unset — correct; live-write opt-in — correct); the read-only radiance test passed.
- Completion criteria: (1) engines `gufo`+`radiance`, `fleet.engines_live` 2, `config.llm.radiance.enabled` true — **pass (live curl)**; (2) unset → 404 no-dial + no job — **pass (spy-pinned, green)**; (3) fixtures present, both families (79 `radiance:` / 198 `vllm:` lines), drift pin — **pass**; (4) panel/curve/rows — **pass (re-shot rest screenshot)**; (5) live radiance test — **pass**; (6) suite/coverage/lint/`bash .agents/validate.sh` (exit 0) — **pass**; (7) docs — below; (8) gufo/vLLM behaviour unchanged — **pass (existing pins untouched)**.
- Documentation: **README.md** full review — env table, LLM-engines section, architecture diagram, capture-script and `/llm` section all name radiance correctly; no inconsistencies found, no changes needed. **AGENTS.md** — the three task-06 fixes (fleet line, route row, module bullets) are in place and all other statements match reality; no changes needed.
- Dev server healthy (`./scripts/dev-server.sh status`: all sources ok, `llm:radiance` answering); all changes left in the working tree for the harness.
- Next pending phase: none — 34 was the last phase in `todo/`.
2026-11-01 18:53:46 -05:00
ducoterra 86c8bf4992 phase: 33_gateway_route_switching
Phase 33 completion pass is done. Final report:

## Phase 33 — completion report

**All 8 task files complete** (`.agents/phases/complete/33_gateway_route_switching/`), full gate green, docs verified.

### Work in this pass

1. **`tests/test_routectl.py` added** (17 tests, all green) — exercises the *real* helper binary unprivileged against a fake tree (`--root`), the real agent parser, the captured gateway fixtures, and stubbed `nginx`/`haproxy`/`systemctl` on `PATH`. Pins the full protocol (locate → state → dry-run → temp-file/fsync/`os.replace` write → checker → restore on checker failure → reload-never-restart → re-read proof → sidecar record), all three route shapes (haproxy `server`, nginx `proxy_pass`, nginx stream `map` via its sink-value id), `drift` / unknown-id / `map-default` refusals, `list`/`verify`, and the `--serve` peer gate (`SO_PEERCRED` uid check, one JSON line per request, exact body shape).

2. **Real helper defect fixed** — `deploy/dashboards-routectl.py`: `main()`'s early-return paths returned `({"ok": False, …}, EXIT_*)` tuples, so `sys.exit()` printed a Python repr instead of a JSON line (usage errors, agent-import refusal). Now prints `json.dumps(…)` and returns only the code, keeping the one-JSON-line contract.

3. **Documentation consistency fixes** (task 08 criterion 6):
   - `README.md`: garbled verdict-colour sentence on the switch ("green and to the position") → "in the **on** position and tinted `--gate-on`" (the existence-pair contract).
   - `README.md`: audit sentence now names all three event kinds — `route-disabled` / `route-enabled` / **`route-denied`** (the code stores one per refused call; dry runs store none).
   - `page-gateway.js`: replaced the stale pre-33 module docstring ("the switch is not wired") with the two-arm `GATEWAY_ALLOW_WRITES` contract.
   - `AGENTS.md`: every phase-33 claim re-verified against code (the ten gates, the 35 s/30 s budget pair, `gateway.writes_enabled`, `/api/summary`'s `disabled`, the 1.4.0 agent bullet, the gap list) — no changes needed.

### Verification
- `uv run pytest --cov`: **1970 passed**, 18 skipped — coverage **98.74%** (floor 94%)
- `ruff` clean · `pyright` 0 errors · `bash .agents/validate.sh` **exit 0** (incl. `node --check` on every page script)
- Dev server running; all 10 sources `ok`; bundles exported

### Live infrastructure
**No live changes made.** The live drill remains unexecuted per `data/logs/NOTES-33-switching.md` (no operator approval; checker binaries unavailable in the podman proxy units). All routes stay enabled, config hashes unchanged, and the sandbox rehearsal + backup tree are in place for the operator's own `DANGER` call. All changes are in the working tree only.
2026-11-01 11:50:51 -05:00
ducoterra 51772ffef2 phase: 32_gateway_resolved_target_liveness
Fixture and live stream.conf are byte-identical (modulo header comment). Let me test the parser directly:
2026-10-26 14:03:42 -04:00
ducoterra 25fd74b7c5 phase: 31_gateway_route_readability
**Phase 31 verification complete — all 5 tasks already in `complete/`; this was the final verification pass.**

Verified / changed
- Re-ran every phase-31 artifact live and green: `verify-31-page-only.py` 6 PASS/0 FAIL; browser pass `31-route-readability.cjs` **64 PASS / 0 FAIL** (exit 0); `27-task02-mobile-audit.cjs` **371 PASS / 0 FAIL / 0 FINDING** (exit 0); rebuilt `data/logs/31-route-readability.out` as the 4-part transcript.
- Added the 4 degradation-arm screenshots the criterion asked for but the pass lacked (`31-gateway-{unconfigured,unreachable,no-nginx,sockets-quiet}.png`) — 12 frames now, one per arm.
- Grep criteria: no gateway connector rule uses `border-strong`; no verdict token in the gateway CSS section or `page-gateway.js` beyond the `alerts.py`-owned `.warn`; exactly 4 `addEventListener` calls (none for the switch); `routeInConfig()` returns `true`; **no `.py` under `src/dashboards/` changed**.

Tests / lint (all 100% green)
- `uv run pytest` → **1796 passed, 16 skipped**; `--cov` → **98.94%** (floor 94); `ruff check src tests agent` → All checks passed; `pyright` → 0 errors; `node --check` clean; `bash .agents/validate.sh` → **exit 0**.
- `uv run pytest -m live --live` → **12 passed / 0 failed**; `./scripts/dev-server.sh status` → all **10 sources ok**.

Completion criteria: colour (33/33 rows every segment `--gate-on`, dark `rgb(63,185,80)` + light `rgb(26,127,55)`, zero grey dashes) ✔ · inbound-first rows + filter (33/33 line 1 = `out.selectors`, line 2 = `→ host:port · kind`; `stackexpected` → **6** rows, not the overview's "five" — the capture declares 6 such routes; tallies byte-identical `10/33/16/0`, zero requests) ✔ · trunk motion truth table + `null` arm prints `n/a` never `0` + reduced-motion stills 43 paths and every knob ✔ · screenshots/transcripts ✔ · payload page-only ✔ (literal byte-diff of a live socket table can never pass; answered by the no-`.py`-diff + shape-equality proof, as recorded in the transcript) · no behaviour change elsewhere ✔.

Docs: fixed 6 inconsistencies — two identifiers split across line breaks (`test_the_pipe_animation_stills_in_both_states_under_reduced_motion`, `32_gateway_resolved_target_liveness`), stale "sixteen routes" count, stale "four machines/hosts" in README **and** AGENTS (fleet is three), "three fields"→"three field types", gap-14 headline wording; updated the pass's screenshot list.

Deviations: `uv pip list --outdated` shows one advisory (`starlette 1.9.0 → 1.10.0`) — not upgraded, this phase moved no dependency (rule 8 / DoD apply "if deps moved"). `TODO.md` carries two new operator disk requests, left untouched.

Next pending phase: `32_gateway_resolved_target_liveness`.
2026-10-25 05:32:30 -04:00
ducoterra e366f599f3 phase: 30_gateway_connection_map
**Phase 30 final verification pass — all 6 tasks already in `complete/`, so this was the phase-completion pass.**

- **Verified (re-run, not trusted):** `/gateway` serves exactly two panels (nginx→haproxy, `GATEWAY_CARD_KEYS`); haproxy rows `.solo` straight across, nginx `:443` branched with one tick per leaf (18 live, fixture 17); every pipe computes the theme `--ok` in dark+light, animates only at count ≥1, fully static under `prefers-reduced-motion`; switch = `role="switch"`+`aria-checked`+`aria-disabled`, no listener, footer+title name the missing write path; zero non-GET requests anywhere.
- **Live read-only re-run** (`data/logs/30-final-live-verify.out`): 17 PASS — `warnings: []`, 6 haproxy frontends→own `server`, `:443` 18 map leaves incl. `default→127.0.0.1:4443`, `:4443` 6 / `:8443` 3, `:80` bare listener, no `unmatched`/`outbound`/`by_port`/`partial`, no `9120`; inbound reconciliation **page 16 = kernel 16**; `gateway` block = the 12 declared keys; no `gateway:config` alert on a clean config. `git diff --stat agent/ deploy/` empty; agents 1.2.0 (gateway) / 1.1.0 ×2, all `tls`, `bundle_rev 2`.
- **Tests/lint (exact):** `uv run pytest` → **1791 passed, 16 skipped**, exit 0 · `bash .agents/validate.sh` → **exit 0**, `--cov` **98.94%** (floor 94), `ruff check src tests agent` clean, `pyright` 0 errors, `node --check` all page scripts, hygiene clean · `uv run pytest -m live --live` → **12 passed** · `uv pip list --outdated` empty · `./scripts/dev-server.sh status` → **10 sources ok**, `gateway` included · browser pass `30-connection-map.out` 63 PASS/0 FAIL (5 screenshots at 1440/360) · `30-final-mobile-audit.out` 371 PASS/0 FAIL/0 FINDING, exit 0, nine pages.
- **Docs fixed:** README `alerts.py` bullet omitted phase 30's keyword-only `gateway_payload` and its one `warn` config-integrity row (+suppression) — added; overview ingress line labelled a dated reading. AGENTS gap 13 corrected 391 px → **361 px** (task 05's pill took 30 px), "task 06 measured" → "phase 29 measured", coverage figure un-pinned to 98.9 %, live-suite count noted (12 today), "all five pages" severity sentence clarified for `/gateway`'s existence-only palette. Everything else reviewed in full and **verified consistent**.
- **Defect fixed:** duplicated footer-assertion block at the end of `tests/test_api.py::test_the_switch_is_built_and_explicitly_unwired` (one copy removed).
- **Notable:** `NOTES.md` is empty in the working tree — both items in it (non-mTLS agents, disappearing admin links) landed in phases 24 and 20; left cleared. Fixture/live `map`-count drift (17 vs 18) is recorded in AGENTS.md, not silently re-captured.
- **Next pending phase:** none — `30_gateway_connection_map` was the only entry in `.agents/phases/todo/`.
2026-10-23 22:56:21 -04:00
ducoterra fa4f38c694 phase: 29_gateway_dashboard_page
Phase 29 completion pass is done. Final state:

**Verification (all re-run this pass, no task files touched)**
- `uv run pytest` → **1746 passed, 16 skipped**; `uv run pytest --cov` → **99.05 %** (gate 94 %); `bash .agents/validate.sh` → **rc=0** (ruff clean, pyright `0 errors, 0 warnings`, `node --check` clean); `uv run pytest -m live --live` → **12 passed**.
- Live, read-only: dev server **10/10 sources `ok`** (incl. `gateway`); `data/logs/29-task06-live.cjs` → **49 PASS / 0 FAIL** with per-port inbound counts reconciled against a direct `/proc/net/tcp{,6}` read (divergence 0); `data/logs/verify-29-task06-predates.py` → 0 failed (1.1.0 arm renders `WHY_PREDATES`, no pipes, every tally `n/a`, job still `ok`); phase-27 mobile audit re-run over all nine served pages → **371 PASS / 0 FAIL / 0 FINDING** (`data/logs/29-final-mobile-audit.out`).
- Criteria: (1) unconfigured `GATEWAY_HOST` → `404 {configured:false}` with dial spies + `#gate-empty` sentence ✅; (2) 1.1.0/`--no-proxy` → `supported:false`, no `0`s, never red ✅; (3) live gateway 1.2.0 → pipes animate, totals reconcile ✅; (4) no `severity`/verdict colour anywhere, validate green ✅; (5) docs updated + gap 13 logged ✅; (6) the five existing dashboards, jobs and `/api/summary` unchanged (live control leg) ✅.

**Docs (consistency pass, incl. pre-existing drift)**
- `README.md`: architecture diagram now lists `sources/gateway.py`, the 30 s gateway job and the third `/snapshot` reader; `GATEWAY_HOST` row says *a name, never a URL*; the `POLL_MS_*` row states `/gateway`'s fixed 15 s browser poll; "nine pollers" → per-source cadences; phase-28 "until phase 29" wording → current state; new Testing paragraph for `tests/test_gateway_source.py` (join pinned from the captures, reconciliation identity, `getaddrinfo` spy).
- `AGENTS.md`: phase-28 stop-point wording replaced with the shipped phase-29 state; suite/coverage figures → 1746 / 16 skipped / 99.08 %→99.05 % band and `~99 %`; window-chips paragraph records that `/gateway` has **no** chips (fixed 24 h); new testing-policy bullet for the join; stale page counts fixed (seven → eight tabbed pages / `MARKUP_PAGES`, clock-order and badge-count transcripts annotated); gap 12 now records the phase-29 nine-page audit re-run.
- `tests/test_api.py`: comment/docstring-only fixes for the "seven pages" wording that phase 29 made eight.

**Deviations / notes**
- Nothing implemented: all six task files were already in `complete/`, so this run was the final verification + docs pass; no git index or phase file was touched.
- Coverage wobbles 99.05–99.08 % between runs on a pre-existing timing-dependent line (`collectors.py` `/slots`-alias `except`/`break`), not a phase-29 path (`sources/gateway.py` is 100 %).
- `uv pip list --outdated` still lists upgrades; dependencies did not move in this phase, so they were left alone rather than upgraded mid-verification.
- Historical browser-transcript counts (phase 18/20/25 script runs) were left as recorded history and annotated instead of rewritten.

**Next phase:** none — `.agents/phases/todo/` is empty after 29; further work needs a new phase authored first.
2026-10-23 09:52:30 -04:00
ducoterra 44d61d3ab9 phase: 28_gateway_agent_proxy
**Phase 28 final pass — complete, 100 % green.** All five tasks were already `complete/`; this pass verified the criteria, re-ran the live half, and did the docs pass.

- Gate `bash .agents/validate.sh` **rc=0**: `1650 passed, 15 skipped`, coverage **98.97 %** (floor 94), `ruff` clean, `pyright` **0 errors**, `node --check` clean, hygiene clean (`/tmp/validate-done.log`).
- Live: `pytest -m live --live` **11 passed**; `./scripts/dev-server.sh status` nine sources `ok`; mTLS re-read of `gateway` → agent **1.2.0**, all three sections `available: true`, 6/6 frontends routed, 11 nginx servers + `$targetBackend`, **19** listeners, no core errors (`data/logs/28-finalpass-live.out`); installed agent file byte-identical to this tree (`b0e9ebf1b718…`).
- Criteria: 1 ✅ pytest/coverage · 2 ✅ three sections over a captured tree, `false`/`null` + one `errors` line each elsewhere with `cpu`/`mem`/`disk_io`/`system` intact · 3 ✅ `AGENT_VERSION` 1.2.0 + SCHEMA names all three · 4 ✅ six fixtures verbatim, capture refuses a bad read, 116 socket address columns hold no globally routable endpoint · 5 ✅ gateway runs 1.2.0 and answers over mTLS (transcript `data/logs/28-task05-rollout.md`, `28-finalpass-live.out`) · 6 ✅ validate · 7 ✅ additive: HEAD's 1.1.0 and this 1.2.0 `collect()`ed over one `--root` are key-for-key equal plus the three new sections, `--no-proxy` equal exactly (`data/logs/verify-28-final.py` → 41 PASS, `28-final-verify.out`).
- Docs fixed (README/AGENTS/Containerfile, all re-green): README's opening said "Four dashboards" above a five-row table and said the new sections are "stored in the snapshot" (they are in the **agent's** document — no `/api` payload, no `sample` row, nothing charts them until phase 29); `Containerfile`'s header still said "three dashboards"; AGENTS's coverage figure (98.94 → **98.97 %**), invariant 6 gained the captured-socket-table identifier rule, the agent bullet gained "phase 28 stops at the wire", and the fleet record names the stash-and-compare control plus the volatile listener count. Full record: `data/logs/NOTES-28-final.md`.
- Deviations: none functional; no git index/history touched, phase files untouched (`PLAN.md` and `complete/` unchanged). `uv pip list --outdated` still lists 4 packages (fastapi 0.153→0.161 etc.) — advisory only, dependencies untouched by this phase, left for a deliberate pass.
- Next pending phase: `.agents/phases/todo/29_gateway_dashboard_page/` (the page that consumes these three sections).
2026-10-22 20:24:01 -04:00
ducoterra 5d37444fb0 phase: 27_mobile_tables_and_audit
The audit re-ran green (exit 0). Now checking the docs for defects introduced by the phase.
2026-10-21 23:43:25 -04:00
ducoterra 9ff3c7889b phase: 26_ups_event_split
**Phase 26 `ups_event_split` — final verification pass (all 3 tasks already complete; nothing missing from `complete/`).**

Verified
- `page-ups.js`: `POWER_EVENT_KINDS = {status,test,action}` split, unconditional `refreshEvents()`, no `state.events`/toggle-fetch, shared `eventRow`, `#ups-errors-card.hidden = errors.length === 0`; `ups.html` error card above the disclosure, `hidden`, outside every `<details>`. Kind set cross-checked against the real writers (`collectors.py:393/409`, `api.py:1043/1058`, `poller.py:127/157/164/190`) — classification is correct.
- Re-ran the phase's browser pass live: `node data/logs/26-task03-verify.cjs` → **31 checks, 0 skipped, ALL PASS** (`data/logs/26-final-verify.out`) — poll-failure rows readable with the disclosure shut, card hidden on a power-only feed and appearing on the first `unreachable`, lists paging independently, 360 px clean.

Tests / lint (all green)
- `uv run pytest` → **1545 passed, 15 skipped**, exit 0 · `bash .agents/validate.sh` → exit 0, coverage **98.94%** (floor 94%) · `uv run ruff check src tests agent` clean · `uv run pyright` 0 errors · `node --check` clean · `uv run pytest -m live --live` (with `.env.dev`) → **11 passed**, dev server reads all 9 sources `ok`.

Completion criteria: poll-failure visible + power collapsed ✅ (pinned + browser); hidden-when-empty card, one `/api/events?source=ups` fetch, shared pager ✅; stale-last-good via the empty `catch` pin ✅; pytest/coverage/ruff/pyright ✅; `validate.sh` ✅; no behaviour change to prior work — `dash.css`/`app.js` untouched, disclosure CSS / pager / row-shape pins pass unchanged ✅.

Docs
- Fixed (pre-existing + this phase's drift): AGENTS.md "one helper with four call sites" → **five over four pages** and the `row(event)` sentence (phase 26 added a second `/ups` call site); invariant 5's `web/static/charts.js` → `src/dashboards/web/static/charts.js`; recorded the completion-pass transcript. README: "one helper renders every feed" now notes five lists. Full sweep otherwise consistent — all 38 env vars match `config.py` exactly, all served routes/scripts/contracts referenced exist, suite/coverage/live counts and marker splits verified.

Deviations: `fastapi 0.153.0 → 0.159.1` advisory left alone (validate.sh declares it advisory-not-a-gate; this phase touched no dependencies, matching prior phases' precedent). Next phase: **`27_mobile_tables_and_audit`**.
2026-10-21 19:42:19 -04:00
ducoterra 2bf236fd92 phase: 25_masthead_control_placement
**Phase 25 final verification pass** (all 3 tasks were already in `complete/`, so this was the phase-level verify + docs pass)

**Verified**
- `#nav-status` first child of `.masthead-right` on all 7 signed-in templates (incl. `/certs`, `/settings`); `app.js`/`charts.js` untouched
- `#nav-status` + `.masthead-right` both `justify-content: flex-end`, `margin-inline-start` still absent, row-break (`flex-wrap` + two `flex-basis: 100%`), non-sticky bar, dropdown slide + 4 close paths, `setInterval(refreshStatus, 15000)` all intact (dash.css diff = comments + the two alignment values)
- Clock is the last control on all 7 templates; no `order:` anywhere in the bar

**Fixed (defects found in this pass)**
- `tests/test_api.py::test_every_page_links_to_both_admin_pages` — duplicated `tabs = html[...]` statement (task 03 edit artifact)
- `certs.html` / `settings.html` comments said "all **eight** signed-in pages"; the signed-in set is seven (`/login` is the eighth, anonymous, and has no links/clock)
- `AGENTS.md` — suite tally `1542 → 1543 passed`; "the three phase-20 tests" re-worded, since task 02 renamed one to `test_the_masthead_is_three_rows_right_justified_at_every_width`

**Results**
- `uv run pytest` → **1543 passed, 15 skipped**, 0 failed
- `bash .agents/validate.sh` → **exit 0**: coverage **98.94 %** (floor 94 %), `ruff` All checks passed, `pyright` 0 errors, `node --check` clean, hygiene clean
- `uv run pytest -m live --live` (`.env.dev` sourced) → **11 passed**; `./scripts/dev-server.sh status` → 9/9 sources `ok`
- Browser: `node data/logs/25-task03-verify.cjs` → **235 checks, 0 FAIL**, exit 0 (`data/logs/25-task03-verify.out`, 16 `masthead-25-*.png`)

**Completion criteria**: badges pinned+measured (green `system/disks/llm/ups` on both admin pages, 1280 px and 360 px) ✅ · both rows flush right, three rows + dropdown unchanged (pinned + measured) ✅ · clock farthest-right (pinned markup; measured right edge 1260.0 at 1280 px, ≤900 px hidden by the long-standing readout rule with source order asserted) ✅ · pytest/coverage/ruff/pyright ✅ · validate.sh ✅ · no behaviour change in completed work ✅

**Docs**: README reviewed end-to-end against the code (served-route audit vs documented routes, env table vs `config.py`, commands vs `scripts/`, public-path table, `web/` masthead bullet) — consistent, no changes needed beyond the phase's own edits. AGENTS.md — two fixes above; spot-checked every doc-named test, function, script, threshold, `install-agent.sh` sha256 (`ffbcd91f…` matches) and agent version (1.1.0).

**Deviations**: left the pre-existing `fastapi 0.153.0 → 0.159.0` advisory (validate prints it as advisory; no dependency work in scope, AGENTS rule 8 triggers on touching deps). No live-infra changes. Overview's "all eight signed-in pages" for the clock is verified against seven templates + `/login` (which has no clock, pinned).

**Next pending phase**: `.agents/phases/todo/26_ups_event_split/` (3 tasks).
2026-10-21 16:42:14 -04:00
ducoterra 89d9fb9f89 phase: 24_dev_mtls_rollout
**Phase 24 final verification — all green**

- **Verified (read-only re-run, 62/62 PASS)** via `data/logs/verify-24-mtls.py`: nine sources `ok`; `/api/certs` all three `active` with `deployed_bundle_rev == bundle_rev == 2`; `agent.tls: true` + `agent.version 1.1.0` on every host; driveripper 16/16 drives live; gateway absent from `/disks` but `disks:gateway ok`; all served target URLs `https://`, no userinfo; pre-flip backup present; no transport/cert alert.
- **Criterion 1 (export):** `data/certs/{driveripper,deskwork,gateway}.bundle` re-checked byte-identical (sha256) to `GET /api/certs/<name>/bundle`, mode `0600`, under gitignored `data/`; live arms re-probed (issued→file, unknown/`issued:false`→one-line note + no file, wrong password→exit 2, nothing written).
- **Criterion 2 (`--fetch-bundle`):** 54 sandbox pins in `tests/test_deploy_scripts.py` pass — byte-identical `0640` install + TLS unit lines, refusals (unissued / 401 / non-bundle body / dead dashboard) before `need_root` with no write, token absent from token file, unit, stdout and sudoers hint.
- **Criterion 3–4:** `./scripts/dev-server.sh status` nine `ok` + bundles block; screenshots `data/logs/24-task04-{certs,disks,system}.png` exist.
- **Criterion 7:** no behavior change — `agent/` untouched, `src/` diff is a `certs.ensure_certs` docstring only; token-mode, revocation-before-dial and bundle-route tests green.
- **Tests/lint:** `bash .agents/validate.sh` → **EXIT=0**; `uv run pytest --cov --cov-fail-under=94` → **1542 passed, 15 skipped, coverage 98.94%**; `ruff check src tests agent` → All checks passed; `pyright` → 0 errors; `node --check` clean; `uv run pytest -m live --live` → **11 passed**.
- **Docs:** README.md (all 1144 lines) + AGENTS.md reviewed against the live project — every config.py env var is in README and both templates, route set / severity thresholds / test counts / coverage figure match, all repo paths, cited test names and `Dash.*`/`DashCharts.*` helpers resolve, phase-24 doc pins green. **Verified consistent — no changes needed.**
- **Notable:** left `fastapi 0.153.0 → 0.158.0` (validate reports it as advisory; no dependency was touched, rule 8 conditions the check on deps moving). Killed two abandoned `/tmp/fbtest` loopback stub processes left by task 02's manual probe; dev server unaffected.
- **Next pending phase:** `.agents/phases/todo/25_masthead_control_placement/`.
2026-10-21 13:47:45 -04:00
ducoterra 5f3ed9bcd4 phase: 23_llm_rate_responsiveness
Phase 23 final verification complete — all green.

```
VERIFIED (criteria)
1  /api/llm/snapshot windows ≈2 s live (1.98–2.00), /api/system window_seconds ≈10 s (9.996–10.005);
   /api/config llm.rate_window_seconds=2=LLM_INTERVAL_SECONDS; browser re-check ALL CHECKS PASSED
   (data/logs/23-window-{llm,system}.png — /llm prints Δ2.0s, /system footnote Δ10.0s)
2  Phase-13 pins unchanged: floor, 2×span ceiling, 8× jump gate + independent anchor — all pass
   (only import/span-line diffs in tests/; git diff confirms no assertion rewritten)
3  Live proof re-derived from artifacts: 780 rows, 0 unlabelled, windows 0.61–3.38 s, 5 starts +
   5 ends settled by poll 2 (0 unsettled), 1 m rollups max 42.3 vs raw 201.6, 0 over 300 tok/s;
   counterfactual 10 s window: median settle poll 3, |ln| 0.924 vs 0.037
4  uv run pytest: 1494 passed / 15 skipped · coverage 98.94% (floor 94) · ruff clean · pyright 0
   errors · node --check clean · bash .agents/validate.sh exit 0 · live 11 passed · store_perf 4 passed
   Dev server up, all nine sources ok. No git add/commit run; everything left in the working tree.

DOCS PASS — 4 inconsistencies fixed in AGENTS.md (last two runs above include them)
- UI conventions hero example still said the `.sub` footnote reads `Δ10.0s`; the only hero that
  prints a Δ is /llm's → now `Δ2.0s` on /llm, "at the current load" on /ups
- gap 9: "found 10 busy-edges" contradicted its own artifact (17 edges, 10 judged, 7 reported-not-
  judged) → 17; and the poll-2 ratios were a 3-of-5 cherry-pick → full five values both windows
- gap 12: the responsive-table fix is now scheduled — names 27_mobile_tables_and_audit
README, config table, .env.example, rates/prom docstrings, app/api bullets and the testing-policy
pins all re-checked against code and found consistent (store pragmas, budgets, throttle, fleet).

NOTES
- Advisory only, not phase 23: fastapi 0.153.0 → 0.156.8 available; dependencies untouched here,
  so rule 8's "update on purpose" is left to the operator.
- NOTES.md is empty in the working tree (its items became phases 24–27); left as-is, untracked-file
  rules forbid me removing it.
- Next phase: 24_dev_mtls_rollout (25–27 follow).
```
2026-10-21 08:04:21 -04:00
ducoterra 895f309f8e phase: 22_disks_hide_unconfigured
All green. Final report:

**Verified (phase 22 final pass — both task files already in `complete/`)**
- `api._disk_visible` (`is not False`) is the single gate, applied to the assembled host list in `/api/disks/snapshot` **and** `/api/summary`'s disks block before the fleet sum and the rows; `sources/disks.py`, `page-disks.js`, `poller.py`, `store.py` and `alerts.py` are byte-unchanged (`git diff --stat`: only `api.py`, `test_api.py`, docs).
- Re-ran the live browser pass `data/logs/22-task02-disks-live.cjs` against the running dev server: **34/34 PASS** — no gateway card/why-line, `2/2 hosts answering`, fleet header reads the API's `2 of 2` / `17-17-0-1`, deskwork `nvme0n1` OK 47 °C/19 202 h/11 %, driveripper 16 live cards + `13 uncorrectable` chip, both aria-labelled plots, `/` still 3 fleet rows. Screenshots + `22-task02-api.json` refreshed.
- `./scripts/dev-server.sh status`: all **nine sources `ok`**, `disks:gateway` included; live `/api/events?source=disks` still carries gateway transitions (10 of 45), proving the README/AGENTS claim the gate hides a row and not the job's history.

**Completion criteria**
- `false` host absent from both endpoints, `None` + `unreachable` stub present, fleet ≡ row count: **PASS** (6 pins in `tests/test_api.py`, incl. the single-hidden-host = 200-with-empty-list leg).
- Live `/disks` = deskwork + driveripper, header 2 hosts, cards unchanged, screenshot in `data/logs/`: **PASS** (re-run above).
- `uv run pytest` → **1477 passed, 15 skipped**; `--cov --cov-fail-under=94` → **98.94 %** (`api.py` 100 %); `ruff check src tests agent` → all passed; `pyright` → 0 errors; `bash .agents/validate.sh` → **EXIT 0**. `uv run pytest -m live --live` → **11 passed**.
- No behaviour change in completed work: **PASS** (unchanged files above; live numbers match the pre-phase fleet record).

**Docs** — README: `/disks` page-table row + the agent SMART paragraph now distinguish "cannot read drives" (dimmed cards kept) from "no smartctl" (host absent from the page/API lists). AGENTS.md: hermetic tally 1471→**1477**, and the `collectors.py`/`api.py` passages now note the `false` arm never reaches the `/disks` why-line while an overview alert from a hidden host's `disks` job still names it. Everything else (routes vs served table, env table vs `config.py`, commands, scripts/fixtures/tests references, masthead/POLL_MS/threshold/retention claims) **verified consistent**.

**Deviations** — `NOTES.md` left empty by the task runs (both notes are tracked: mTLS → phase 24 `todo/`, the admin-links bug → fixed in phase 20 task 04); `/disks`' `#hosts-empty` copy left as pinned markup (it already names smartmontools + the disk group, so an all-hidden fleet is still told the fix); validate's advisory `fastapi 0.153.0 → 0.156.4` untouched (no dependency changed this phase). No live fleet writes of any kind.

**Next pending phase:** `23_llm_rate_responsiveness`.
2026-10-21 03:27:08 -04:00
ducoterra 5e267737dc phase: 21_page_cleanups
**Phase 21 final verification pass** (all 3 task files already in `complete/`, working tree carries the implementation)

- **Verified** task 01: `#pause`/`Space`/`setPaused`/`_paused` gone — `grep -ri pause src/dashboards/web/` empty, 0 pause hits in all 8 served pages + 10 static assets; re-ran `data/logs/21-task01-nopause.cjs` → 30/30 PASS (hidden tab: 0 data fetches in 22 s, one poll 2 ms after return).
- **Verified** task 02: re-ran `21-task02-gpu-values.cjs` on the live fleet → all PASS (deskwork/driveripper show Model/VRAM/Pool/Temp at 16.3 px; gateway = 5 heroes, slot `hidden`, no `n/a`; hero order unchanged; GPU history chart still builds per host).
- **Verified** task 03: re-ran `21-task03-scroll.cjs` → 14/14 PASS (`class="data scroll"`, scrollWidth 1021 vs clientWidth 292 at 360 px, Rotate/Revoke/bundle reachable and hit-testable; unchanged at 1280 px). Screenshots refreshed in `data/logs/`.
- **Gate**: `uv run pytest` → **1471 passed, 15 skipped**; `--cov --cov-fail-under=94` → **98.94%**; `ruff check src tests agent` clean; `pyright` **0 errors**; `bash .agents/validate.sh` → **EXIT 0** (incl. `node --check`); `uv run pytest -m live --live` → **11 passed**; `/api/health` all 9 sources `ok`.
- **Docs fixed** (pre-existing + this phase): "all **eight** signed-in pages" was a miscount of `MARKUP_PAGES` (7 = 5 dashboards + `/settings` + `/certs`; `/login` is the 8th, unsigned) → corrected in `README.md`, `AGENTS.md` (×2) and the `tests/test_api.py` docstring/comment; `AGENTS.md` Commands hermetic tally 1470 → **1471**.
- **Docs reviewed, consistent**: Quickstart/commands/flags (`install-agent.sh`, `capture-fixtures.sh`, `dev-server.sh`), the env table vs `config.py`, every documented route vs the served route table, masthead/`/system` GPU/`/certs` scroll descriptions vs code, and all `data/logs/*` + `tests/fixtures/*` references.
- **Deviation**: `validate.sh`'s advisory `fastapi 0.153.0 → 0.156.4` left alone — no dependency was touched this phase, so AGENTS rule 8 ("check after touching dependencies") is not triggered; upgrading here would be an unrelated behaviour change.
- No live-infra changes made (repo-side + browser verification only).
- **Next pending phase**: `22_disks_hide_unconfigured`.
2026-10-21 01:44:52 -04:00
ducoterra 6444a158f2 phase: 20_masthead_nav_polish
**Phase 20 final verification pass — all four tasks already complete; verified in full.**

- Verified: three-row masthead CSS (`.masthead` flex-wrap + two `flex-basis: 100%`, `min-height`, no `margin-inline-start`), un-sticky `position: relative` + `z-index: 40`, 180 ms `visibility/opacity/transform` slide (closed `nav.tabs` / open `.is-open`, both states in the file's single reduced-motion block), both admin links on all 8 signed-in pages, `/login` neither.
- `uv run pytest` → **1464 passed, 15 skipped** · `uv run pytest --cov --cov-fail-under=94` → **98.94%** (floor 94, unchanged) · `ruff check src tests agent` clean · `pyright` 0 errors · **`bash .agents/validate.sh` EXIT=0** (incl. `node --check` + repo hygiene) · `uv run pytest -m live --live` → **11 passed**.
- Browser (live dev server, invariant 10): `20-task01-rows.cjs` 121 PASS/0 FAIL · `20-task02-unsticky.cjs` 81/0 · `20-task03-slide.cjs` 32/0 · `20-task04-admin-links.cjs` 87/0 · phase-18 `nav-final-pass.cjs` 45/45, `nav-task03-verify` 77, `nav-task04-verify` 24/24 · phase-19 scripts 11/22/24/29 all pass. Screenshots refreshed in `data/logs/` (`masthead-rows*`, `masthead-unsticky*`, `slide-*`, `admin-links-*`). Dev server restarted; `/api/health` all nine sources `ok`.
- Completion criteria: 3-row bar @1280/360 ✓ · scroll leaves bar behind + dropdown under whole bar ✓ · slide out/in 180 ms, reduced-motion snap ✓ · `/certs`→`⚷ Certs`, `/settings`→`⚙ Account` before theme toggle, outside `.tabs` ✓ · pytest/coverage/ruff/pyright ✓ · validate ✓ · no regression in tab structure, token reveal, safe-area padding, four close paths, status dots, theme toggle ✓.
- Docs: **AGENTS.md** gap 12 still said phase 20 was "in `todo/`… admin-link half has landed" and attributed the wrap to the old fixed 58 px row — rewritten to the completed state (three rows/un-sticky/slide/both links; 24 still todo; 21's `/certs` `.scroll` noted as not reaching `/`'s table). **README.md** `web/` bullet enumerated the controls row without `⏸ Pause` and omitted `/login` from the two-row list — both fixed. Everything else (setup, commands, env table vs `config.py`, routes vs `api.py`, agent/installer flags, testing policy) verified consistent.
- Deviations: no dependency bump — `uv pip list --outdated` still reports the pre-existing advisory `fastapi 0.153.0 → 0.156.4` (deps untouched this phase, validate treats it as advisory); `NOTES.md` is empty in the working tree from an operator edit outside these tasks (both items live in phases 20/24) — left as-is; no live-infra changes made or needed.
- Next pending phase: `21_page_cleanups` (pause button removal, `/system` GPU values, `/certs` table scroll).
2026-10-20 22:49:37 -04:00
ducoterra 95df8b695b phase: 19_ups_events_expandable
**Phase 19 final verification pass — all tasks already complete; everything green, no code changes needed.**

- **Verified (all 4 task files present in `complete/`)**: `details.events-box` disclosure on `/ups` with no `open` attribute and `#events` inside it; lazy fetch (`if (state.events.open) await refreshEvents()`), toggle stale rule on `state.pollMs`, success-path-only `painted`/`fetchedAt`; `charts-box`-shaped CSS with the `> .body` padding restatement; 4 new served-byte pins in `tests/test_api.py`.
- **Criterion 1 (collapsed on load, no first-poll fetch)**: PASS — re-ran `data/logs/19-task04-verify.cjs` against the live dev server (real upsd, 2 UPSes): **28/28 PASS**, `0` `/api/events` requests across 2 closed polls vs 4 snapshot polls; expand → exactly 1 request, 50 rows with `Oct 20, 07:11` timestamps + severity chips + `1–50 of 200` pager.
- **Criterion 2 (cache/stale/poll-while-open)**: PASS — fresh re-open 2→2 requests, stale re-open 2→3, loop refreshes while open, close asks for nothing, failed fetch keeps rows and doesn't stamp the cache.
- **Criterion 3**: `uv run pytest` → **1458 passed, 15 skipped**; `uv run pytest --cov` → **98.94%** (floor 94, unchanged); `uv run ruff check src tests agent` → clean; `uv run pyright` → **0 errors**.
- **Criterion 4**: `bash .agents/validate.sh` → **exit 0** (incl. `node --check` of all page scripts + repo-hygiene gates).
- **Criterion 5**: `data/logs/ups-events-closed.png` (collapsed `▸ POWER EVENTS`, 0 events requests noted in `ups-events-verify.out`) and `ups-events-open.png` (`▾`, bounded 320px list, pager strip) regenerated and inspected this pass.
- **Criterion 6 (no regressions)**: full suite green incl. UPS contract/pager pins; `uv run pytest -m live --live` (with `.env.dev`) → **11 passed**; phase 18's `nav-final-pass.cjs` sweep re-run → **45/45** (this phase's markup/CSS didn't touch the masthead).
- **Documentation**: full README.md (980 L) + AGENTS.md (1992 L) review against the code — verified env templates vs `config.py`, API route table vs `api.py`, installer flags vs `deploy/install-agent.sh`, agent routes vs `host_agent.py`, test/coverage/live tallies, `data/logs` artifacts, and live fleet facts (`HOST_TARGETS` all `http://`, all three agents `version 1.0.0`, nine sources `ok`). AGENTS.md UI conventions and README's events paragraph already carry the disclosure/lazy/stale sentences from tasks 01–04. **Verified consistent — no changes needed.**
- **Notable**: made zero repo edits (working tree holds only this phase's task work). `uv pip list --outdated` couldn't complete (PyPI 503 for the project name) — no dependency moved this pass, so rule 8 is not triggered; validate.sh's earlier advisory still lists `fastapi 0.153.0 → 0.156.3`, deliberately not upgraded in a UI/verification phase.
- **Next pending phase**: `.agents/phases/todo/20_masthead_nav_polish/` (tasks 01–04).
2026-10-20 19:17:22 -04:00
ducoterra 10f86772ee phase: 18_nav_hamburger
**Phase 18 — collapse the nav bar into a hamburger: complete, all green.**

Verified (criteria):
- ≤640px: the tab links leave the bar, the ☰ is visible, and it opens a dropdown
  under the sticky masthead (`top: 100%`, `z-index: 41`, viewport-wide) with the
  five stacked 40px links; above 640px the one-row desktop masthead is unchanged
  and the button hidden — `data/logs/nav-task04-verify.cjs` 24/24 and
  `data/logs/nav-final-pass.cjs` 45/45; screenshots
  `data/logs/nav-hamburger-{closed,open}-360.png`, `data/logs/nav-masthead-desktop-1280.png` ✓
- The four close paths: the button again, a tab click (navigates and closes),
  Esc (focus back on the opener), and `matchMedia` resize back above the
  breakpoint (drops a stale `.is-open`; `aria-expanded` tracks state on every
  path through the single `setNavMenu()` funnel) ✓
- `uv run pytest` → 1454 passed, 15 skipped; `bash .agents/validate.sh` → exit 0
  (coverage 98.94% ≥ 94, ruff clean, pyright 0 errors, `node --check` + hygiene
  clean); served-page pins: button on all seven tabbed pages and nowhere on
  /login, CSS and binding compared on the 640px literal ✓

Real-device finding (mobile Chrome): the emulated 360px pass was blind to the
system top-edge zone — Android Chrome is edge-to-edge by default, so the sticky
bar started under the status bar and the first-row ☰ sat in a region the device
does not deliver taps to (unreachable, not unbound; the same symptom brain of
reese fixed in production). The ≤640px block now pads the bar by
`env(safe-area-inset-top, 0px)` (the block's `padding-block` value restated on
the top side), verified on the operator's phone; `env()` resolves to 0 on every
non-notch surface, so all other widths render byte-identical. The served-byte
pin's rule loop carries the line; CDP's `safeAreaInsets` override does not feed
`env()`, which is the honest limit of the emulated proof.

Docs: AGENTS.md *UI conventions* masthead entry extended with the three pins,
the drift-comparison, the safe-area rule, and the real-device verification
caveat; README's `web/` bullet names the ≤640px collapse with the desktop
masthead unchanged above the breakpoint.

Deviations: pinned the shipped token shape (`#nav-toggle { display:
var(--nav-toggle-display) }` + the `:root` token) instead of the task file's
literal `.nav-toggle { display: none }` — that literal loses to
`button.ghost { display: inline-flex }` (task 01 measured it); the test
docstring says so, so nobody "restores" the broken shape. `uv pip list
--outdated` still advisory (fastapi 0.153.0 → 0.156.2); this phase moved no
dependency. NOTES.md (emptied) / TODO.md (operator items added) are
harness-side changes — left in the working tree, as in phase 17.

Next pending phase: `.agents/phases/todo/19_ups_events_expandable/`.

LLM-Generated: true
2026-10-20 16:47:10 -04:00
ducoterra b02c13f7d3 phase: 17_installer_device_filter
**Phase 17 — final verification pass: complete, all green.**

Verified (criteria):
- `grep -n "DevicePolicy\|DeviceAllow" deploy/install-agent.sh` → no match; `PrivateDevices=false` retained (line 472) ✓
- `install-agent.sh --help` names `--as-root`/`--caps` with the kernel-gate sentence; `--force` documented ✓
- Fresh fake-root install = pre-phase unit minus the filter lines: `test_a_default_install_unit_is_the_pre_phase_unit_minus_the_filter_lines` + `test_the_template_unit_has_no_device_filter_lines` pass ✓
- Re-run that differs → exit 1 with diff on stdout, `--force` applies, identical re-run silent: `test_a_re_run_that_differs_refuses_without_force_and_applies_with_it` (+ 3 sibling guard pins) pass ✓
- driveripper reproduction — verified **read-only**, no live change: `systemctl cat` shows `User=root`, `CapabilityBoundingSet=CAP_SYS_RAWIO CAP_SYS_ADMIN`, no filter, same `ExecStart`; `active`, `CapEff 0000000000220000`; `/api/disks/snapshot` → 16/16 drives with temperatures ✓
- `uv run pytest` → **1451 passed, 15 skipped**; `bash .agents/validate.sh` → **exit 0** (coverage **98.94%** ≥ 94, `ruff` clean, `pyright` 0 errors, `node --check` + hygiene clean); `uv run pytest -m live --live` → **11 passed**; dev server 9/9 sources `ok` (one initial live failure was a transient gufo timeout, clean on re-run)

Docs (inconsistencies found + fixed, incl. pre-existing):
- `AGENTS.md` fleet record said driveripper's unit runs `User=dashboards-agent` while the same bullet (and the live host) says `User=root` — rewritten to separate the installed user from the running one.
- `AGENTS.md` gap 10 header still claimed "the installer template is what is left" → now records phase 17 as the fix.
- `README.md` "with no capabilities" → "with an empty `CapabilityBoundingSet` (unless `--caps`…)", since this phase added the capability flag.
- `README.md` battery-test "gated four times" now notes the two session gates in front, matching AGENTS invariant 1.
- `.env.example` was missing three README-documented settings (`DASH_ENV_FILE`, `DASH_RELOAD`, `NUT_SIMULATE`) and `AGENT_CERT_BUNDLE` in the agent-variables note — added, annotated; all template values re-checked against `config.py` defaults.

Deviations: no live `--force` run on driveripper (needs the Oct-18 sudo-staged installer re-staged — it lacks the flags — and would push agent 1.1.0 mid phase-15 rollout); command + expected diff + post-checks recorded in AGENTS/gap 10. `uv pip list --outdated` still advisory (fastapi 0.153→0.156), untouched as in tasks 02/04 — this phase moved no dependency.

Next pending phase: `.agents/phases/todo/18_nav_hamburger/`.
2026-10-20 13:48:40 -04:00
ducoterra cffbac5771 phase: 15_agent_mtls
All green. Report:

**Verified / implemented (final pass of `15_agent_mtls`)**
- Wrote `data/logs/verify-15-tls-e2e.py`: a real `host_agent.py --cert-bundle` + a real dashboard process over a scratch DB (`HOST_TARGETS=loop=https://…`) — **27/27 PASS**, transcript `verify-15-tls-e2e.out`, notes `data/logs/NOTES-tls-rollout.md`.
- Read-only checks against the running dev store: 5 `certs.*` rows byte-identical across a restart; served bundle == the `meta` row (1430 B); 0 events containing `-----BEGIN`; unsigned `/certs`→`303 ?next=`, unsigned `/api/certs`→`401`, signed-in `200`; screenshots `data/logs/15-final-{certs,system}.png`.
- Docs (step 6, full review): README — added *Protecting an agent, host by host* (the rollout task-07 left out), a `tests/test_certs.py`/TLS-matrix/sandbox-installer paragraph, and fixed two stale “the one write endpoint in the app” claims (`NUT_ALLOW_ACTIONS`, Battery-test buttons). AGENTS.md — test count 1424→1427, bundle-fetch command in *Commands*, a Testing-policy bullet on structurally pinning a key-holding module, fleet transport bullet (token mode is now the compatibility path), gap 11 final state.

**Tests / lint (full suite 100% green)**
- `uv run pytest` → **1427 passed, 15 skipped**, exit 0 · `bash .agents/validate.sh` → **exit 0** (coverage **98.94%** vs 94% floor, `certs.py` 100%; ruff “All checks passed”; pyright 0 errors; `node --check` clean; hygiene clean)
- `uv run pytest -m live --live` → **11 passed / 0 failed** · `./scripts/dev-server.sh status` → all 9 sources `ok`
- `uv pip list --outdated` → empty after bumping `fastapi 0.152.0 → 0.153.0` (`uv.lock` only); full suite re-run green.

**Completion criteria**
- Fresh-DB root CA, idempotent: ✅ (tests + live store hashes) · agent TLS matrix incl. wrong-CA/certless 403/`/health` CA-only/revoked 403 + live denylist re-read: ✅
- `https` `host:`/`disks:` `ok` + revoked says “agent certificate revoked” with no dial: ✅ on loopback (real agent + real dashboard) and spy-pinned hermetically; **not** on the fleet (below)
- Signed-in `/certs` + rotate/revoke/root-rotate + byte-identical bundle download: ✅ · installer bundle/unit/`--status` over TLS: ✅ (0640, see deviation)
- `pytest`/`validate.sh`/`--outdated`: ✅ · token-mode byte-identical, engine/UPS untouched, public surface unchanged (route walk): ✅
- **Live fleet TLS (deskwork+gateway flip): NOT done** — no live changes. It needs root on hosts without the installer grant (`deskwork`), has no both-ways-working window (`http://` dashboard vs https-only agent), and would deploy the agent from this not-yet-committed tree. Exact procedure recorded in README + `data/logs/NOTES-tls-rollout.md`.

**Deviations**: bundle mode `0640 root:dashboards-agent` instead of the task file’s `0600` (the agent re-reads its own bundle for the denylist; 0600 breaks the unit — already recorded in AGENTS.md); the loopback dashboard substitutes for the fleet flip. Incidental: a stray `pkill` of mine killed the dev server mid-pass; restarted and verified healthy.

**Next pending phase**: `.agents/phases/todo/16_agent_smart_pass_budget` (then `17_installer_device_filter`).
2026-10-20 10:02:16 -04:00
ducoterra 569ca25fdb phase: 14_store_performance
**Phase 14 `store_performance` — final verification pass (all 4 tasks already in `complete/`)**

- Re-verified every task's work in the tree: read caches (`series_names`/`stats`), capped compound `pick_res_many`, `history_many` + `api._history_batch`, windowed `maintain()` + per-resolution prune + `PRAGMA optimize`/`automatic_analyze=0`/WAL pragmas, `store_perf.BUDGETS` (14 ceilings), probe, live watch.
- **Completion criteria:** baseline+after side by side in `data/logs/NOTES-store-perf.md` ✅ · budget tests green (`uv run pytest -m store_perf --store-perf` → **4 passed**, 14/14 inside) ✅ · 25-min dev-server watch re-run **independently** (74 passes, `scripts/watch-store-perf.py`): **0 `store-error`** (total still 190, newest 2026‑10‑18 06:33:30), WAL 7.77 MB ≤ 67.1 MB, medians 1.7–3.2 ms, p95 720/613 ms = the documented cold-cache window; a 3rd 27-pass watch on a freshly restarted server reproduced it (3.1/708, 1.8/572) ✅ · `maintain()` 5,368 ms → 2.7–6.8 ms and WAL bounded across two cycles (test + notes) ✅ · no behaviour change (all four history routes pinned against `query()`) ✅
- **Full gates:** `uv run pytest` → **1267 passed, 15 skipped**; `bash .agents/validate.sh` → **exit 0**, coverage **98.75%** (floor 94), `ruff check src tests agent` clean, `pyright` 0 errors, `node --check` ok, repo hygiene ok; `uv run pytest -m live --live` → **11 passed**; dev server up, 9/9 sources `ok`, pages 200 in 1.5–3.3 ms.
- **Docs fixed:** README/AGENTS still claimed a flat "20 s per pass" while `host_agent.py` (in this same uncommitted change) scales the budget to 5 s × disk count — corrected in both plus the two illustrative "20 s of wedged disk" sentences; broken bold (`` `driveripper** ``) repaired; stale gap-4 title + "tuning is its own phase" wording updated to say phase 14 ran; README's absolute "nothing … is answered by a whole-table scan" qualified to *per call* with the cold-window cost named; "third opt-in marker" → second; conftest `--store-perf` help named the real marker; **`data/logs/watch-store-perf.py` moved to `scripts/watch-store-perf.py`** (both docs told a clone to run a gitignored path) with root-anchored paths, `DASH_WATCH_OUT`, and dead code removed.
- **New finding recorded:** gap 9's "~10 s process-wide stall is still gap 4's / `14_store_performance`'s problem" is now closed — coincident >8 s `llm` + >12 s `ups` raw-stream gaps went 393 (00:00–12:00) and 198 (12:00–21:36) to **0** across 110+ min of the phase-14 store; written into AGENTS.md gaps 4 and 9 and the notes table.
- **Deviations:** did *not* pull either deferred cost knob (`STATS_CACHE_TTL_SECONDS`, warming caches from the maintenance tick) — AGENTS.md records that decision and its reasoning; `uv pip list --outdated` still reports certifi/fastapi/starlette/websockets (pre-existing advisory, no dependency touched this phase, gate treats it as advisory). Overwrote nothing: task 04's watch preserved as `store-perf-watch.task04.*`, my two runs as `.verify25.*` / `.current-code.*`.
- **Next pending phase:** `15_agent_mtls`, then `16_agent_smart_pass_budget` (owns the scaled-budget tests + its doc criterion, now partly pre-satisfied by this pass), `17_installer_device_filter`.
2026-10-19 23:44:35 -04:00
ducoterra 1bbd868401 phase: 13_llm_page_and_decode_rate
All green. Final verification summary:

**Verified / implemented (phase 13 final pass)**
- All 4 tasks already in `complete/`; this was the verification pass. Confirmed the CSS rhythm rules (`#strip`, `#banners:empty ~ #fleet`) are served and pinned, `data/logs/NOTES-decode-spike.md` names the mechanism, and the phase-07/04 surfaces still pass their pins.
- **Found and fixed a residual defect in this phase's own code**: a stalled fetch stamps its body with the poll's *finish* time, so that sample anchors every window for the rest of the span; the jump gate "confirmed" a jump measured from that same anchor → live gufo rows 232.28 tok/s (one request at ~50) + a decaying 146/114/95 tail, repeated at 200.6. `rates.py::_publish` now requires an **independent anchor** (`_pending` stores the base sample's ts; the hold continues until it ages out). Pinned by `test_a_held_jump_is_not_confirmed_by_a_window_on_the_same_anchor` + `test_gufo_a_stale_scrape_body_never_becomes_a_rate_above_the_engine` (both fail on task 03's gate).
- Re-verified live after a dev-server restart: 10 min / 540 signed-in `/api/llm/snapshot` polls + read-only SQLite pass; **0 values > 300, 0 without a `Δ<window>` label**; 0 post-fix stored rows over the ceiling except the two pre-fix-shape rows above.

**Tests / lint (exact, all 100 % green)**
- `bash .agents/validate.sh` → **EXIT=0**; `uv run pytest --cov --cov-fail-under=94` → **1195 passed, 11 skipped, coverage 98.70 %**; `ruff check src tests agent` → All checks passed; `pyright` → 0 errors; `node --check` clean; repo hygiene clean.
- `uv run pytest -m live --live` → **11 passed / 0 failed**. `/api/health` → **all nine sources ok**.

**Completion criteria**
- Fleet/strip + head/fleet gap with screenshots (`llm-fleet-spacing{,-top,-banner-case}.png`) — met (CSS unchanged this pass, pinned on served bytes).
- Task-02 findings file names the confirmed mechanism — met.
- `test_rates.py`/`test_llm_source.py` pin label-set, both window bounds and the always-honest `Δ` — met (+2 new pins).
- Sustained watch under the recorded ceiling (300 tok/s) — met (worst 203.05, all labelled).
- Full suite + validate green; no phase-07/04 behaviour change — met.

**Documentation**
- README: `RATE_WINDOW_SECONDS` row now states the hold/confirm rule (only change needed; rest verified consistent).
- AGENTS.md: `rates.py` bullet + gap 9 rewritten (residual closed *and* the remaining sub-8× skew measured, root-caused to gap 4, explicitly not fixed by inventing a new threshold).
- **Fleet facts changed under this pass and were corrected**: operator added driveripper's bearer token → `host:driveripper`/`disks:driveripper` `ok`, live suite 9/2→11/0; but driveripper reads **identity only, zero SMART numbers** (root `smartctl` gives 55 °C/14 %; `/dev/nvme0`, `/dev/ng0n1`, `/dev/bsg/*` are root-only; its unit is `DevicePolicy=auto`, not the installer's `strict`). Evidence appended to `data/logs/NOTES-agent-install.md` + `NOTES-decode-stable.md`.

**Notable decisions / deviations**
- Did **not** add a span-relative window floor: that would re-decide task 03's pinned `MIN_WINDOW_SECONDS` and its "in-bounds diluted row publishes" test. Recorded as the next phase's call, with the root cause left at gap 4 / `14_store_performance` (the ~10 s stall).
- No live fleet change made (read-only probes + one dev-server restart). Fixtures deliberately not re-captured (no parse change, task 02 proved no shape drift). Dependency-currency advisory left untouched (no dependency moved in this phase): certifi/fastapi/starlette/websockets have newer releases.

**Next pending phase:** `.agents/phases/todo/14_store_performance` (then `15_agent_mtls`).
2026-10-19 14:53:58 -04:00
ducoterra 28139df6d6 phase: 12_nut_self_test_verdict
Final state verified. Report:

**Verified (phase 12 — all 3 tasks already complete; this was the verification pass)**
- Re-ran the wire capture live: `ups.test.result` = `No test initiated` (apc) / `Done and passed` (cyberpower) — matches `LIVE_SELF_TEST_WORDS` + both fixtures; `/api/summary` has **no** self-test row, `/api/ups/snapshot` reads `cyberpower ok | Done and passed`, `apc unknown | …run a battery test`; both screenshots read back correct.
- Fixed a defect found in prior work while checking the "unrelated live noise" note: `NutMonitor.collect()` reused one upsd socket after a *partially-read* reply, so a 4 s stall produced bogus `expected 'BEGIN LIST VAR', got 'RW apc battery.charge.low "10"'` errors (same hazard on `INSTCMD`, where a late `OK` could be read as a different command's success). `NutClient` now marks a socket whose reply stopped early as `UNUSABLE`, naming the command that stalled; a complete `ERR …` line still leaves it usable (the per-UPS tolerance `collect()` depends on). +3 hermetic pins against the fake upsd; live re-check (09:33–09:44) now logs one line per stall, no cascade.

**Tests / lint / coverage (all green)**
- `bash .agents/validate.sh` → **exit 0**; `uv run pytest --cov --cov-fail-under=94` → **1172 passed, 11 skipped**, coverage **98.63 %** (floor 94); `uv run ruff check src tests agent` clean; `uv run pyright` 0 errors; `node --check` clean. Modified modules: `sources/nut.py` 95 %, `collectors.py` 97 % (no new line uncovered).
- `uv run pytest -m live --live` → **8 passed / 3 failed**, all three the pre-existing `driveripper` agent-transport failures AGENTS.md records (verified still true: unit `inactive`, no installed file, nothing on `:9120`).
- `./scripts/dev-server.sh status`: `nut ok`, `llm:* ok`, `host/disks:deskwork|gateway ok`; the two `driveripper` jobs DOWN.

**Completion criteria**
- Live capture recorded + asserts `ok` — met (re-confirmed from the wire today).
- `/ups` renders `ok`, overview silent, screenshots present — met.
- pytest / cov / ruff / pyright / validate — met (above).
- No other `health.checks[]` behaviour changed — met (diff confined to the `test_result` branch; all prior nut/ups-contract tests unmodified).

**Documentation**: README verified consistent (self-test wording, config table cross-checked against `config.py`, all documented commands/flags exist) — no changes needed beyond phase tasks' edit. AGENTS.md fixes: stale test count (1140 → 1172), `nvme0n1` power-on hours 19 158 → 19 160, gap 4 store-error claim corrected (last stored `store-error` 2026-10-18 06:33, none in the previous 24 h), new `collectors.py`/`sources/nut.py` socket-hygiene note, new gap-1 bullet on the upsd stall, plus live evidence appended to `data/logs/NOTES-self-test.md`.

**Deviations**: made no live *changes* (read-only probes only) — `driveripper`'s agent install needs root (`ops` has no installer/visudo grant) and a battery test stays the operator's call; the upsd 4 s stall itself is upstream-side. `uv pip list --outdated` still advisory (certifi/fastapi/starlette) — dependencies untouched this phase.

**Next pending phase**: `13_llm_page_and_decode_rate`.
2026-10-19 09:54:03 -04:00
ducoterra 1734d81e48 phase: 11_admin_login
**Verified (final pass, phase 11 — all 7 tasks already in `complete/`)**
- Gate from the wire on a restarted dev server: `/system`→`303 /login?next=/system` (`/llm?range=1h`→`/login?next=/llm%3Frange%3D1h`), `/api/summary`+`/api/config`→`401 {"ok":false,…}`, `/api/health`→`200`; signed-in `GET` of all six pages `200`, `/login`→`303 /`.
- Auth flows on a throwaway server (`:8199`, `/tmp` DB — dev `auth.verifier` untouched): wrong pw `401`, `testpass` `200`, password change `200`, **old cookie `401` / new cookie `200`**, old pw `401` / new pw `200`, short new `422`, wrong current `403`, cookie write without CSRF `403`, logout `200`; 5 failures then `429 retry-after: 60`; `login-denied`/`password-changed`/`password-denied` events; `meta` = the two auth keys.
- Fail-closed boot (no seed, empty DB): **exit 3**, message names `DASH_ADMIN_PASSWORD`, `auth.verifier`, and the `DELETE FROM meta …` reset. `dev-server.sh status` + the Containerfile `HEALTHCHECK` python one-liner both `200`/exit 0 unauthenticated.
- Live payloads unchanged (phases 01–10 walk): `/api/summary` keys + 8 alerts + hosts 3/2 live; UPS `simulated:false` with charge/load/runtime severity; deskwork `nvme0n1` 51 °C/19 159 h/11 % wear; gateway `smartctl_available:false`; driveripper `unreachable` stub.

**Tests/lint (all green)**: `bash .agents/validate.sh` → exit 0 — `1140 passed, 11 skipped`, coverage `98.62%` (floor 94), ruff `All checks passed!`, pyright `0 errors`, `node --check` clean, repo hygiene clean. `uv run pytest -m live --live` → `8 passed, 3 failed`, all `driveripper: 10.1.0.126:9120 … ConnectError` (pre-existing, auth-free; probe confirms nothing listening).

**Completion criteria**: all met except the live-suite caveat above — installing the driveripper agent needs root on that host (no `ops` grant), so per the live-infra rule I made **no live change**; task 07's five screenshots (`auth-login/-rejected/-locked-out/-signed-in-overview/-settings.png`) remain the browser evidence, and the login/rotation/lockout flows were re-proved over HTTP this pass.

**Docs**: README — fixed "only place the CSRF token is published" (login/password responses mint one), added the undocumented `DASH_RELOAD` row, stated that Sign out ends only this browser's cookie. `AGENTS.md` — new known gap 8 (stateless logout cannot revoke a copied token), gap-4 dev-store sizes dated, fleet section notes `data/` is gitignored. `api.py` — same `status` docstring imprecision fixed. Verified the 4 `DASH_*` rows, reset command, seed-wins rule and invariant 12 against `app.py`/`config.py`.

**Deviations**: no dependency bump (`uv pip list --outdated` advisory: certifi/fastapi/starlette — phase added none); verification notes appended to `data/logs/NOTES-auth.md` §10.

**Next pending phase**: `12_nut_self_test_verdict`.
2026-10-19 07:45:44 -04:00
ducoterra a08e693698 phase: 10_dev_direct_dial
**Phase 10 final pass — repo green; live criteria blocked off-repo.**

- Verified: `gateway` runs `dashboards-agent` (active/enabled, `10.1.1.27:9120` → `/health` 200, `/snapshot` 401 no-token / 200 bearer on-host, token matches `/etc/dashboards-agent.token`); `driveripper`/`deskwork` inactive, `sudo -n` → "a password is required", `:9120` empty.
- New measurement (transient runtime rule on gateway `firewall-cmd --add-port=9120/tcp`, **reverted** — runtime+permanent port lists verified back to the original set, no rich rules): laptop still gets no SYN through, while `:22`/`:80` on the same address answer in ms ⇒ the block is the lab gateway `10.2.0.1` port filter, not this app/host firewall/agent. No other live change made (no sudoers, no upsd, no UPS command).
- Tests/lint (exact): `bash .agents/validate.sh` → **EXIT 0** — `uv run pytest --cov --cov-fail-under=94` **793 passed, 11 skipped, coverage 98.33 %**; `ruff check src tests agent` clean; `pyright` 0 errors; `node --check` clean; hygiene clean; `uv pip list --outdated` empty (bumped fastapi 0.143.3→0.143.5, `uv.lock` only).
- `uv run pytest -m live --live` (`.env.dev` sourced): **4 passed, 1 skipped, 6 failed** — all six are `:9120`/`:3051` connect timeouts; zero parse/contract failures.
- Dev server restarted on current code: engines `ok`; 3×`host` + 3×`disks` + `nut` `DOWN` with real reasons (designed degradation, stub rows intact, no credential leakage).
- Completion criteria: screenshots ✓ (`data/logs/phase10-{index,system,disks,llm,ups}-live.png`); tunnel-vocabulary grep ✓ outside `.agents/` history (PLAN.md locked line, phase overviews, past reports; fixtures/`uv.lock` hits are digit substrings — `tests/test_deploy_scripts.py` guards operational files); no page-payload change ✓ (contract tests). **Not met, operator-side:** agents on all three hosts, `LIST UPS` from the laptop, `dev-server.sh status` all-ok, live suite green.
- Fixed in this pass: `sources/nut.py` unreachable-upsd message no longer prints a bare `(TimeoutError: )` (now `(timed out after 4s)` / `(ConnectionRefusedError: …)`), pinned by a new test + extended refused-port test.
- Docs: README verified consistent (every env var in `config.py` documented, routes/flags/`--with-llm`/pager sizes checked) — no changes needed. AGENTS.md fixed 4 inconsistencies: per-host sudo reality (gateway grants `NOPASSWD ALL`, which is why it installed without `agent-sudoers`; driveripper scoped; deskwork unrelated), the now-proven firewall finding, `/system` table typo ("agent per host"), gap-4 DB/WAL sizes + test count. `.env.example` `upsd.users` sample now one `instcmds` per line (it contradicted README and the in-app copy). `.env.dev` (gitignored) comment corrected to match the evidence.
- Flagged, untouched (locked): `.agents/PLAN.md` L92–95 still says upsd is "forwarded to `127.0.0.1:3494` in dev" — false after this phase; evidence + exact operator commands appended to `data/logs/NOTES-verify.md` §11.
- Next pending phase: `11_admin_login`.
2026-10-18 05:44:13 -04:00
ducoterra fefbeb8227 phase: 09_disks_smartctl
**Phase 09 — final verification pass (all 4 task files already in `complete/`; nothing re-implemented)**

- Verified `GET /api/disks/snapshot` live: 3 hosts / 18 whole-`/sys/block` rows, `smartctl_available` three-way, per-row `severity`, API-summed `fleet`; 404 `{"configured": false}` proven *without dialling* (spies on `httpx.AsyncClient.get` + `collectors.fetch_snapshot`).
- Verified `/api/disks/history` live (`disk.temp_c@dev`, unknown names → empty series) + `test_store.py`'s 300 s rollup/prune buckets; `/api/metrics?source=disks` answers the `disks` family.
- Browser against live data: 18 drive cards (never a partition/mount), collapsed per-host `Charts` with `aria-label`, 3 `.stale` grids + one `.banner.info` why-line and **0 `.fail` classes / green `disks` dot** → criteria 3 and 4 hold; new shots `data/logs/09-verify-disks-live-dimmed.png`, `09-verify-overview-live.png` (13 total for the phase).
- Real-hardware criterion: read 4 driveripper drives through `ops`' `sudo -n smartctl -i -A -H` and ran the live output through `host_agent.parse_smart_output` → `sources/disks.parse_disks_snapshot`: `sda` 35 °C / 30 423 h, `nvme1n1` 58 % wear, `nvme5n1` 13 uncorrectable → `alarm_disks: 1`, temps graded `warn`; consistent with the 16 committed fixtures.
- `bash .agents/validate.sh` → **EXIT=0**; `uv run pytest --cov` → **769 passed, 11 skipped, coverage 98.33%** (floor 94); `ruff check src tests agent` clean; `pyright` 0 errors; `node --check` clean; hygiene + dependency advisories clean. `uv run pytest -m live --live` → 10 passed, 1 skipped (agents still predate the SMART pass). Dev server healthy: `/api/health` `ok`, 9/9 sources.
- Docs fixed: AGENTS.md test count 698→769 and coverage 98.2→98.33 %; `disks` added to the event dashboard-name list and to the "every feed renders `Dash.eventText`" sentence; README `POLL_MS_*` row now states `/disks`' deliberate fixed 30 s; new Known gap 4 records the dev-DB `database is locked` writes (reported/recovered per invariant 4, ~590 MB DB / ~70 MB WAL, not reproduced by a controlled 25 s external read txn).
- Deviations: **no live-infra change** — granting the agent user the `disk` group and redeploying `host_agent.py` on driveripper/deskwork stay the documented paired operator action (redeploying without the grant yields 16 permission-denied lines; a sudo-spawning agent would break the confirmed "direct read, no sudo" assumption). Store-lock issue left to its own phase rather than patched speculatively.
- Next pending phase: `10_dev_direct_dial`.
2026-10-18 00:45:01 -04:00
ducoterra 19cfedfddf phase: 08_events_paging_and_copy
All green. Final report:

**Verified (phase 08 — both task files already in `complete/`; this was the final verification pass)**
- `Dash.eventsPager()` in `app.js` (50/page, `1–50 of N` strip, aria-labelled `‹ prev`/`next ›` disabled at the ends, `WeakMap` page state, reset only when the event *set* changes, `EVENT_FEED_LIMIT`=200) with three call sites; `ul.events` bounded to 320px + `overflow-y:auto`; `/api/events` untouched (`le=500`, no silent clamp).
- Live browser against the real fleet: `/system` `1–50 of 121`, `/llm` `1–50 of 200`, `/ups` `1–50 of 200`; container 320px vs scrollHeight 1543/2368; page 2 → `51–100 of 200` and survived two polls; no empty `.sub`/`.faint` container visible on any page; footers carry only measurements (`rates over Δ11.9s · poll 10s`). Fresh artifacts: `data/logs/08-final-{system-head,llm-page,index-page,ups-feed-page2}.png` (32 phase screenshots already present).
- Grep of live served bytes for all 13 deleted flavour fragments + `actions = -`: none survive in HTML or JS.
- `AGENTS.md` carries both new UI-convention rules (flavour-text rule with the `TODO.md:7` cited examples; the event-feed-window rule) and the updated test count.

**Tests / lint (full project 100% green)**
- `bash .agents/validate.sh` → **EXIT=0**: `uv run pytest --cov --cov-fail-under=94` → **624 passed, 9 skipped, coverage 97.91%**; `ruff check src tests agent` → All checks passed; `pyright` → 0 errors; `node --check` ×5 clean; repo hygiene clean.
- `uv run pytest -m live --live` (`.env.dev` sourced) → **9 passed**; `uv pip list --outdated` → empty. Re-ran pytest/ruff/pyright/node after the doc edits → all exit 0.

**Completion criteria**: bounded scroll + 50/page + count footer ✅ (all three feeds, live) · no removed flavour string ✅ · AGENTS.md rule ✅ · `validate.sh` green + screenshots in `data/logs/` ✅.

**Documentation**: full README/AGENTS review against code (env table vs `config.py`/`.env*.example`, script flags, Containerfile uid 10001/8090, chart heights 120/110/160, test count, coverage) — one inaccuracy fixed: README said "Every page's event feed" (the overview has none) → now "The three event feeds (`/system`, `/llm`, `/ups`)"; also restored the missing trailing newline in `TODO.md`. Everything else verified consistent.

**Notes**: paging stays client-side (no API change); the task-02 defect fix in `/ups` setup copy (`NUT_HOST`/`NUT_PORT`, `LISTEN 127.0.0.1`, one `instcmds` per command, never `actions = -`) re-verified against README and live bytes. Pre-existing items intentionally left to their own phases: `ups.test.result: "Done and passed"` rated `warn` (operator's `TODO.md` entry) and the dev-DB `database is locked` retention issue.

**Next pending phase**: `.agents/phases/todo/09_disks_smartctl/`
2026-10-17 19:32:36 -04:00
ducoterra 3d5ccd7a39 phase: 07_llm_monitor_style_pages
**Phase 07 final verification pass — all tasks already in `complete/`; every criterion re-verified live.**

- **Old slot table gone**: no `#slots-card`/`#slots-table`/`renderSlots` in HTML/JS/CSS (only the new grid's `.slots-empty` *class*); pinned by `test_the_broken_slot_table_is_gone_from_html_js_and_css`.
- **`tests/fixtures/slots_gufo.json`**: present; live `GET ${GUFO_BASE_URL}/slots` this pass matches it key-for-key (row + `next_token[0]`), so the capture is reproducible; drift proven to fail (mutation tests + live `test_gufo_slots_answer_the_pinned_shape`, 9 live tests pass).
- **Aggregate block**: live page shows `2 of 2 engines`, running/waiting/prompt/decode/total-tokens straight from `fleet_totals()`, dot `ok`, no verdict colour; on a throwaway single-engine instance (:8091, private temp DB) `#fleet` is `hidden`, one full-width strip card + one panel, zero banners → "one clean half".
- **Strip + slot grid on live data**: 2 `.card.stat` (KV bars `bar ok` 13 %/0 %), gufo grid rendered real cards incl. a busy slot (`#2189`, 67849 prompt tok, 99 % cache share, `processing`), vLLM note-only with no grid; no `NaN`/`undefined` anywhere. Screenshots: `data/logs/llm-phase07-final-live-{two-engines,single-engine}.png`, `ups-charts-aria-labels.png`.
- **Defect found in earlier work and fixed**: `/ups`' four `LineChart`s shipped no `ariaLabel`, violating AGENTS.md's "every chart names itself for assistive tech" (`/system`+`/llm` were pinned, `/ups` wasn't) → added labels to `page-ups.js` and `test_every_ups_chart_names_itself_for_assistive_tech`; browser-verified `role="img"` + label on all four.
- **Gate**: `bash .agents/validate.sh` → **exit 0**; `uv run pytest --cov --cov-fail-under=94` → **604 passed, 9 skipped, coverage 97.91 %**; `uv run ruff check src tests agent` → All checks passed; `uv run pyright` → **0 errors**; `node --check` every page script ok; `uv run pytest -m live --live` → **9 passed**; `uv pip list --outdated` empty.
- **Docs**: README + AGENTS.md reviewed end-to-end; fixed test count (582→604), `/llm` source column + both architecture diagrams now name gufo `/slots`, AGENTS.md agent line gained `GET /schema` + status page, ariaLabel convention now covers all three chart pages, README Testing block now lists `bash .agents/validate.sh`. Env-var table, agent/install flags, Containerfile facts, `/api/*` shapes, range chips, remote URL all verified accurate — no other changes needed.
- **Live-infra**: no fleet/host/config change; the temp :8091 instance, its temp DB and the fake processes are removed; dev server restarted and `/api/health` `ok` (all 6 sources ok). One thing I deliberately did **not** touch: the gitignored local `.env.dev` still carries a comment claiming the NUT `[dashboards]` account has only battery-test `instcmds`, whereas AGENTS.md/README record `instcmds = all` (verified on hardware 2026-10-17) — operator-owned file, flag for the operator to reword.
- **Next pending phase**: `08_events_paging_and_copy`.
2026-10-17 17:50:19 -04:00
ducoterra de325caa49 phase: 06_system_page_gpu_verdicts
All criteria verified and the project is fully green.

**Verified (final pass — all 3 phase-06 tasks were already in `complete/`)**
- `classify("gpu.busy_pct", 99)` / `gpu.mem_used_pct` → `None`; `THRESHOLDS` has no `gpu.*` row; `alerts.py`/`api.py` carry no GPU verdict; live browser: deskwork GPU at **91 %** and **98 %** renders `class="hero"`, `color=rgb(22,25,31)` (plain text), no bar, no `.sev-*`.
- GPU temp on the hero `.sub` from live agents: `Radeon 8060S Graphics · VRAM 165.1 MiB / 512.0 MiB · pool 78% · 48 °C`, R9700 `· 40 °C`, gateway prints none (no `n/a`, no `0 °C`).
- Charts collapsed by default: fresh load = 3 panels / 3 toggles / **0 open / 0 `<svg>`**; expand → 3 SVGs (510 px wide), repaints across polls (legend 3.3 %→5.6 %), collapsed siblings keep their own height (803 vs 358/339).
- `gpu.busy_pct@0` still answers from `/api/system/history` (345 points) — verdict gone, data intact.

**Tests / lint / coverage**
- `bash .agents/validate.sh` → **EXIT=0**; `uv run pytest --cov --cov-fail-under=94` → **550 passed, 8 skipped, 97.91 %**; `uv run ruff check src tests agent` → clean; `uv run pyright` → 0 errors; `node --check` all scripts clean; `uv pip list --outdated` empty.
- `uv run pytest -m live --live` → **8 passed** (real agents/engines/upsd); dev server restarted, `/api/health` all sources `ok`.
- New screenshots: `data/logs/phase06-final-collapsed-default.png`, `data/logs/phase06-final-expanded-gpu-busy-uncoloured.png`.

**Completion criteria:** classify→None ✅ · pytest+94 % gate ✅ · ruff+pyright ✅ · validate.sh ✅ · screenshots of the three behaviours on live data ✅ · AGENTS.md severity section claims no GPU rows ✅

**Docs:** fixed 3 items — (1) `AGENTS.md` pytest count stale (546 → 550); (2) `app.js` `severityFor` docstring still claimed "`page-ups.js` still calls it (phase 05 converts it)", contradicting the code/tests/AGENTS.md (comment-only); rest of README/AGENTS reviewed in full (env table vs `config.py`+`.env.example`, agent routes incl. `/schema`, alerts rules, page descriptions, contracts) — consistent.

**Notable:** fixed a real pre-existing defect in `scripts/dev-server.sh` — `stop` slept 1 s and deleted the pidfile without waiting for exit, so `restart` with a browser open overlapped two servers on one SQLite file, producing a `database is locked` store refusal that `/system` rendered as a red event. Now waits (bounded 20 s) then SIGKILLs; verified a restart with live polling logs no such error/event. No live infra config was changed.

**Next pending phase:** `.agents/phases/todo/07_llm_monitor_style_pages`
2026-10-17 14:04:52 -04:00
ducoterra 9041683179 fix(ups): offer every advertised command and gate its aliases too
Two halves of the same bug: the page offered less than the hardware does, and the gate
matched only the name it rendered.

`available_commands()` drops anything `COMMANDS.get()` misses, and 12 of the 23 commands
the two real UPSes advertise had no entry (`beeper.on/off`, `load.on/off.delay`,
`shutdown.default/reboot/stayoff`, `driver.reload*`, `driver.killpower`,
`test.panel.stop`) — an invisible button the operator had no way to learn about. Every one
is catalogued now with a risk tier, and the power-cutting ones are pinned to
`RISK_DANGEROUS` (`shutdown.stop` is the exception the name hides: it cancels a shutdown).

Both real drivers advertise `beeper.enable` *and* `beeper.on`, `beeper.disable` *and*
`beeper.off`, so rendering every advertised name drew "Beeper on" twice. `COMMAND_ALIASES`
collapses one action to one entry (named for a name this driver answers, so a driver that
only implements the alias still gets its button) and records the other names in `also`.

The POST gate resolved the request through that table instead of comparing the literal
string: previously `beeper.off` walked past a `disabled` verdict recorded against
`beeper.disable` — the same action, and the gate is the control, not the button.

LLM-Generated: true
2026-10-17 12:27:02 -04:00
ducoterra 1f20a5e57f fix(nut): log in with the verbs upsd 2.8.5 actually answers to
`NutClient.login()` sent `USER`/`PASS`. The deployed upsd 2.8.5 implements neither —
its own `HELP` advertises `USERNAME`/`PASSWORD` — so each line answered
`ERR UNKNOWN-COMMAND`, `_read_line()` raised, and `POST /api/ups/command` could only
ever 502 before `instcmd()`, credentials or not.

Values go out unquoted: upsd takes the rest of the line literally, so quoting them
would make the quotes part of the credential.

`ERROR_HINTS["ACCESS-DENIED"]` blamed the upsd.conf ACL/LISTEN lines, but that code is
only reachable on an authenticated session (an unauthenticated client gets
`ERR USERNAME-REQUIRED`), so the cause is the account's `instcmds` grant. The hint now
names the one thing the operator can change.

The fake upsd in tests has no `USER`/`PASS` branch, so a regression answers
`ERR UNKNOWN-COMMAND` exactly like the real server, and its `PASSWORD` branch checks the
secret so a wrong-credential refusal is expressible.

LLM-Generated: true
2026-10-17 12:26:47 -04:00
ducoterra 34854d32af phase: 05_ups_hero_battery_glyph
**Phase 05 final pass — all criteria met; whole project 100 % green.**

Verified (live upsd + isolated simulator):
- Real `apc`/`cyberpower`: Charge/Load/Runtime are the card's three largest elements (38.4 px vs 15.2 px next), coloured only by API `severity` — live right now 100 % ok / 79.0 % **warn** / 5m **fail**; glyph fill = charge (100/100 viewBox units, `--ok` token), `<title>` "apc: Charge 100 percent, on line"; SIMULATED hidden; battery °C `n/a` uncoloured; every button disabled with its reason; `confirm_text: DANGER` on power-cutting commands; **0 non-GET requests** → `data/logs/ups-verify-live-{card,full}.png`.
- Simulator on `:8091` (own `/tmp` DB, killed after): caution dialog → Cancel, dangerous dialog → Run disabled until `DANGER` → Cancel, **0 POSTs**; one deliberate quick test *to the sim only* flipped the card to `OB` (glyph `stroke-dasharray="7 5"`, "on battery", health "discharging, on battery") then `OL CHRG` (bolt) then `OL` → `ups-verify-sim-{dialog-caution,dialog-danger,onbattery-button,onbattery-page,charging-button}.png`; glyph matrix `ups-glyph-*.png` (100/mid/low/unknown/empty/unrated, dark/light/200 %) present; `grep -c "function battery"` = 1.

Defects found in earlier work and fixed:
1. **`poller.py` could stop polling in silence** — 4 of 6 dev-server sources had frozen `attempts` (ages 12–38 min) while `/api/health` still said `ok: true`: the store-write half of `run_once` was outside the `try`, and a dead task's exception is never retrieved. Now `_commit()` moves `last_ok` only after the write lands, `_store_refused()` reports it (`ok:false`, real reason, one `store-error` event + one `recording` on recovery, payload kept), and `_loop()` catches anything that escapes and keeps polling — 4 new tests in `tests/test_poller.py`.
2. **`nut_sim.py` (modified by this phase) sat at 83 % coverage**; its whole command path was untested. Added tests (every advertised command implemented, unknown → `ValueError`, alarms → card + `assess()` fail, `NUT_SIMULATE` truthy rule) and fixed two model lies: `beeper.*` answered OK while changing nothing (now publishes `ups.beeper.status` enabled/muted/disabled) and a finished self-test never wrote `ups.test.result` (aborted writes none). → 100 %.
3. `nut_sim.from_env()` was dead code duplicating `app._truthy`; `app.py` now builds the simulator through it, so the enable gate lives once.

Tests / lint (exact): `bash .agents/validate.sh` → **EXIT=0** — `uv run pytest --cov --cov-fail-under=94` **543 passed, 8 skipped, coverage 97.91 %**; `uv run ruff check src tests agent` clean; `uv run pyright` 0 errors; `node --check` clean; repo hygiene clean. `uv run pytest -m live --live` **with `.env.dev` sourced → 8 passed** (one earlier run: 2 transient `9121/snapshot timed out` against the 4 s `HOST_TIMEOUT`; three later runs 8/8). `uv pip list --outdated` empty. Dev server restarted on the final code: `health ok`, 6/6 sources fresh.

Docs: AGENTS.md — invariant 4 extended ("nothing may stop polling in silence" + store-refusal semantics), `poller.py`/`nut_sim.py` responsibility lines, 533→543 and 96.9 %→97.9 %, "~97 %"→"~98 %". README.md — poller bullet gained the same guarantee, Testing gained "source `.env.dev` before `-m live --live`" (bare runs skip 7 of 8 live tests, which reads like a pass), Battery-test section gained the `deploy/nut-sudoers` install line and the read-before-write warning.

Notes/deviations: no live instant command was ever sent to real hardware (the four gates leave them disabled — AGENTS gap 1 unchanged); the on-battery proof was driven through the real button against `NUT_SIMULATE=1`, which is stronger than the NOTES hand-driven states. `NOTES-verify.md` under `todo/` left untouched per instructions. Next phase: **none** — 05 is the last in the PLAN.md roadmap.
2026-10-17 10:27:23 -04:00
ducoterra f44a617ef9 phase: 04_llm_chart_readability
**Phase 04 final verification pass — all tasks already in `complete/`; verified + fixed 3 defects found in verification.**

- Verified `/llm` live (both engines, gufo-only instance on :8099, `llm_monitor` on :8095) in a real browser; dev server left running/healthy.
- Completed task-04's truncated layout fix: `#engines` now carries `.items-start` (the CSS rule existed but nothing used it) — gufo 544 px vs vLLM 715 px, no blank card; pinned by `test_llm_engine_panels_are_not_stretched_to_their_tallest_sibling`.
- Fixed: event rows said the subject twice ("gufo gufo is answering again"), and `/ups` mangled mid-message names ("sent to apc: OK" → "sent to  OK") → shared `Dash.eventText()` (strips `UpstreamError: ` + a *leading* name only, mirroring `alerts._clean_error`) used by `/system`, `/llm`, `/ups`; new test.
- Fixed: count sparklines autoscaled rollup averages (queue max 0.1379) into full-height spikes → `sparkline({minSpan})`, `minSpan: 1` on sessions/queued (spike 30 px → 4 px); assertions added.
- Tests: `bash .agents/validate.sh` → **EXIT=0** (`uv run pytest --cov --cov-fail-under=94`: **485 passed, 7 skipped**, coverage **96.87%**; `ruff check src tests agent`: All checks passed; `pyright`: 0 errors; `node --check` all static JS clean; repo hygiene clean; `uv pip list --outdated` empty). `uv run pytest -m live --live` → **7 passed**.
- Completion criteria: heroes largest per engine ✔ (38.4 px vs 16.8 px row values); sparkline per cell + selected window ✔ (5/5 gufo, 7/7 vLLM; 6 h chip moved KV max 53 %→58 %); mean+p95 without hover ✔ (243 / 48.8 / 285 ms feet); provenance tag ✔ (`Δ10.0s`/`Δ8.0s` + `title`=exact source); single-engine ✔ (1 panel, totals hidden, zero red); screenshots ✔ `data/logs/llm-verify-{both-engines,charts-and-slots,single-engine,monitor-reference}.png` + `NOTES-chart-set.md`; pytest/cov/ruff/pyright ✔; earlier phases unchanged-behaviour ✔ (all 01–03 tests pass).
- Deviation (documented, not code): throughput heroes stay **neutral ink** — `severity.py` has no throughput row and 0 tok/s idle is healthy; colour comes from rated cells (KV) + reachability. Kept per `NOTES-chart-set`.
- Deviation: could not append the task-04 written verdict to `NOTES-chart-set.md` (it lives in `todo/`, which I must not edit). Verdict: ours reads better than `:8095` — bigger throughput numbers, per-cell shape over the selected window, percentiles inline; `llm_monitor` only sparklines decode+KV and has no window control.
- Docs: AGENTS.md updated (485 test count, `sparkline` `minSpan` rule, `Dash.eventText` event-row rule, `/llm` panel `items-start`); README reviewed in full — commands, config table (all `config.py` vars present), architecture bullets, page descriptions verified accurate; no other inconsistencies found.
- Note for the harness: `.agents/phases/todo/03…/NOTES-verify.md` and `todo/04…/NOTES-chart-set.md` are untracked notes left behind by their phases (task files moved, notes did not).
- Next pending phase: `05_ups_hero_battery_glyph`.
2026-10-17 07:03:58 -04:00
ducoterra 9757e3c947 phase: 03_system_per_host_panels
**Phase 03 final verification pass — all 4 tasks already in `complete/`; verified criteria, fixed 3 pre-existing defects.**

- **Verified live** (headless Chromium + curl against `:8090`, all 6 sources `ok`): 3 panels × six heroes in order `cpu,mem,temp,disk,gpus,net`; charts per panel `util,net[,gpu]` (gateway: no GPU chart/hole); legends name metrics only — no legend names a foreign host; every `<svg>` `role="img"` + host-scoped `aria-label`; 2 sections, 0 `<select>`; `grep -rn "renderCores|core_usage" src/dashboards/web/` → nothing; console error-free; new shot `data/logs/final-system-panels.png`.
- **Defect fixed (credential hygiene, invariant 6)**: `collectors.py` put the *dialed* URL into every `UpstreamError`, and that string lands in the health record (`/api/health`) and the `event` table (`/api/events` → page); `/api/config` also echoed `GUFO_BASE_URL`/`VLLM_BASE_URL` verbatim. Added `config.strip_userinfo()` (now what `Target.display_url` uses) and redacted all message/config URLs. Tests: `test_a_failing_host_never_echoes_its_token_in_the_message` (×3 paths), `test_a_failing_engine_never_echoes_credentials_in_its_url`, `test_a_dead_host_never_writes_its_token_into_the_feed`, `test_engine_urls_are_redacted_the_same_way_hosts_are`, `test_strip_userinfo_is_the_one_redaction_every_url_leaves_by`. (`parse_targets` already moves `HOST_TARGETS` userinfo into a header, so this is a backstop for hosts and a real fix for engine URLs.)
- **Defect fixed (invariant 2)**: `/system` footnote printed `Δ{HOST_INTERVAL}` as the measurement; it now prints each host's measured `window_seconds` (range when they disagree) — `rates over Δ10.0s · poll 10s`. Pinned by `test_system_footnote_prints_the_window_it_actually_measured`. Also fixed the page's stale "window in each header" note and the empty-state's `--token` (argv) advice → `--token-file`.
- `bash .agents/validate.sh` → **EXIT=0**: **472 passed, 7 skipped**, coverage **96.87%** (floor 94), `ruff check src tests agent` clean, `pyright` 0 errors, `node --check` clean · `uv run pytest -m live --live` → **7 passed** · `uv pip list --outdated` empty.
- Criteria: no multi-host legend ✅ · no per-core visual ✅ · six heroes/panel incl. no-GPU `n/a` uncoloured ✅ · per-core history still queryable ✅ (354 pts) · ≥3 shape screenshots ✅ (`system-driveripper/deskwork/gateway.png` + dark/light/stale) · pytest+coverage+lint ✅ · phases 01/02 unchanged (their pin tests + `/api/summary` shape tests pass; live UPS gap in AGENTS.md re-confirmed read-only against real upsd) ✅.
- **Docs**: AGENTS.md — invariant 6 rewritten to cover message/event redaction, `collectors.py` line, new "Δ is a measurement" rule, "hosts answering total" → "nine-figure fleet-totals panel" (the subtitle still counts hosts), test count 452→472, coverage 96.8→96.9, rewrapped an over-long line. README.md — same fleet-totals wording, `RATE_WINDOW_SECONDS` now covers every measured rate (not just tokens), agent route list completed (`/`, `/schema`, `?top=N`), `LIST CMD` in the diagram, new paragraph on URL redaction.
- **Deviation**: the criterion's literal `curl …/api/system/history?host=…&metric=cpu.usage_pct@core7` uses singular params that endpoint has never accepted (it takes `hosts=`/`metrics=` and ignores unknown ones). Verified 354 points via `…system/history?hosts=driveripper&metrics=cpu.usage_pct@core7&range=1h` and `/api/history?source=host&host=…&metric=…`; I did **not** add singular param aliases (unauthorized API surface). No live infra changes made — reads only.
- **Next pending phase**: `04_llm_chart_readability`.
2026-10-17 04:42:09 -04:00
ducoterra 8b048599f3 phase: 02_overview_fleet_and_alerts
**Phase 02 final verification pass — all three tasks already complete; verified criteria, fixed 3 defects found live.**

- Verified: `/` = exactly 2 `<section>`s (Warnings & errors + Fleet), alerts render from `/api/summary` verdicts with each alert's own `href`; `/system`, `/llm`, `/ups` unchanged (charts + data live).
- Fixed (`src/dashboards/alerts.py`): a dead source's message printed the name twice — `gateway: UpstreamError: gateway: http://…`; the poller's `UpstreamError:` wrapper is now stripped (other exception names kept). Tests: 2 new in `tests/test_alerts.py`.
- Fixed (`src/dashboards/api.py`): `/api/summary` dropped a configured host/engine that had never answered post-start, so the fleet said `2/2 hosts` while a third was down. Stub rows (`unreachable: true`, all `n/a`) now match `/api/*/snapshot`; `hosts_total=3 / hosts_live=2`. Test: `test_a_host_that_has_never_answered_is_still_a_row_in_the_fleet`.
- Fixed (`web/static/page-index.js`): fleet row of a non-answering host is now dimmed with the shared `.stale` class (invariant 4 on `/`). Test: `test_overview_dims_the_fleet_row_of_a_host_that_stopped_answering`.
- `bash .agents/validate.sh` exit 0: `uv run pytest --cov --cov-fail-under=94` → **439 passed, 6 skipped, 96.82%** (`alerts.py` 100%, `api.py` 100%); `ruff check src tests agent` clean; `pyright` 0 errors; `node --check web/static` clean; hygiene clean; `uv pip list --outdated` empty; `uv run pytest -m live --live` → **6 passed**.
- Live proof (tunnel kill, then restored — all 6 sources `ok` again): killing the gateway forward produced exactly **one** alert, `fail gateway /system … unreachable (ConnectError)`; screenshots `data/logs/phase02-final-{alert,system,llm,ups,recovered}.png`.
- Completion criteria: two panels ✅ · `[]` when healthy ✅ (hermetic `healthy_client` + a live gateway-only instance returned `[]`; the real fleet can never be `[]` — its UPSes permanently warn on battery age/self-test) · one alert per dead host with `/system` link ✅ · pytest/live/coverage/ruff/pyright ✅ · no change on `/system`,`/llm`,`/ups` ✅ · phase 01 verdict vocabulary intact ✅.
- Docs: README `web/` bullet now states the dimmed `n/a` row; AGENTS.md invariant 4, `alerts.py`/`api.py` bullets, UI-conventions stale-row + window-chip ranges (5m…30d vs 1h…90d), metric example `llm.decode_tps` → `llm.decode_tokens_per_second` (verified against `/api/metrics`), test count 435 → 439. Env table, Containerfile, agent/install-script claims, fleet facts re-checked and consistent.
- Notable: left the fleet's real hostnames/domain in AGENTS.md's "real environment" section (deliberate, prior-phase content) even though rule 6 discourages personal hostnames in tracked files — worth a separate decision; no live config was changed beyond stopping/starting a local dev ssh forward.
- Next pending phase: `.agents/phases/todo/03_system_per_host_panels/`.
2026-10-17 00:59:09 -04:00
ducoterra ad993c7b03 phase: 01_severity_language
**Gate defect found & fixed (the "does not lint" bug).** `.agents/validate.sh` — the phase gate — ran `python3 -m pytest` on `/usr/bin/python3` + `~/.local` site-packages (not the uv venv), ran `ruff check .` *only if* a ruff happened to be on PATH, and **never ran pyright or the coverage floor**. Rewrote it to the project's real gate: cd to repo root → `uv run pytest --cov --cov-fail-under=94` → `uv run ruff check src tests agent` → `uv run pyright` → `node --check web/static/*.js` → hygiene gates; a missing toolchain now fails loudly. Provoked it to confirm it bites (floor + a seeded F401 both fail).

**Verification (all green)**
- `bash .agents/validate.sh` → **exit 0**: 356 passed, 6 skipped · coverage **94.75%** (floor 94) · ruff `All checks passed!` · pyright `0 errors` · node syntax clean
- `uv run pytest --cov` → `severity.py` **100%**; `uv run pytest -m live --live` → **6 passed**; `uv pip list --outdated` → empty
- Criteria: `classify('cpu.usage_pct',…)` → `fail`/`ok`/`None` ✓ · `/api/ups/snapshot` carries `severity` beside `charge_pct`/`load_pct`/`battery_runtime` (apc now live-`warn` at 498 s — threshold working) ✓ · no `.sev-*`/synonym vocabulary (only a comment saying so) ✓ · pytest/coverage/lint ✓ · dev server restarted, `/api/health` ok 6/6 sources, all pages+assets 200 ✓
- Chromium against live fleet: `severityFrom` cell→`fail`, bare→`warn`, null→`''`, unrated→`''`; `hero` fail → `rgb(207,34,46)`, null → `n/a` with **no** verdict class, warn+bar 42%; all four pages render unchanged → `data/logs/phase01-final-{index,system,llm,ups}.png`

**Docs (full review, incl. pre-existing drift).** README: `/ups` row claimed "upsd on 3493" (fleet is 3051, stated elsewhere in the same file); config table omitted `DASH_HOST/PORT`, llm/ups `*_INTERVAL_SECONDS`, `*_TIMEOUT_SECONDS`, `GUFO/VLLM_API_KEY`, `POLL_MS_*`, `DEBUGPY` (+ dedup); severity bullet omitted `disks`/`gpus[]`/rated LLM cells and the ratio rescale; agent section listed only `/snapshot` though `/info`+`/health` exist. AGENTS.md: same 3493 claim; `install-agent.sh --token-file` (its flags are `--bind/--port/--token/--rotate-token/--status/--uninstall` — it hands *the agent* `--token-file`); gap 3 "Two commits" (now 6, still no `origin`); gap 1 given the sudoers install command + verified live gating reason; coverage line now 80% repo floor vs 94% gate floor; cyberpower `identity.model: GX1500U` vs `ups.description` quirk. `.env.dev.example`: `DEV_FORWARDS` remote port 3493→3051 and engine URLs that pointed at `127.0.0.1` with nothing forwarding them.

**Deviations.** No `ruff format` (30 files would reformat; the project gate is `ruff check` + pyright — reformatting completed phases is out of scope). **No live-infra changes**: enabling the battery-test buttons needs `deploy/nut-sudoers` installed on driveripper + a nut-server restart and a scoped `[dashboards]` `instcmds` user — prod write with a documented shutdown risk (AGENTS gap 1), so repo-side only. 36 pre-existing `ResourceWarning: unclosed database` from test fixtures are noise, not failures; left rather than churn ~15 fixtures. `severity.attach()` is unit-tested but has no production call site until phases 02–05.

**Next pending phase:** `02_overview_fleet_and_alerts` (`01_alerts_builder.md`).
2026-10-16 22:22:26 -04:00
ducoterra b389bb4a94 Wire the live-test gate and pin the toolchain to 3.14
README advertised `uv run pytest -m live --live`, but there was no --live
option and no `live` marker registered — the command errored out, so the
one workflow that catches an upstream renaming a series did not exist.

- conftest adds --live and skips `live`-marked tests without it, so the
  default run stays hermetic (258 passed, 6 skipped, no network).
- tests/test_live_fleet.py drives the real code paths (collectors.*,
  NutMonitor.collect) against the fleet, both engines, and upsd. Each
  skips when its source is unset, and none of them may write — a battery
  test on live hardware is not a unit test. The host test polls twice
  because a rate has no second point on the first scrape.
- The agent's real-/proc test stays in the default run and is explicitly
  not `live`: it touches only the box running pytest.
- dev-tunnels.sh grows DEV_FORWARDS for services already listening
  remotely, which is how upsd gets reachable without opening 3493;
  pidfiles are now host+port so one host can carry two tunnels.
- requires-python, ruff and pyright were still on 3.13 while the venv and
  Containerfile are 3.14, so the lock resolved for a stack nobody deploys.

LLM-Generated: true
2026-10-16 18:55:02 -04:00
ducoterra 426682a999 Add the deployment path: agent systemd unit and container image
The README referenced `deploy/install-agent.sh` and a `Containerfile` that
did not exist, so a fresh clone had no way to actually run this anywhere.

* agent: `--token-file` (and AGENT_TOKEN_FILE). The token guards hostname,
  mounts, disk and GPU models and process names, and argv is world-readable
  through `ps`, so a file has to win over the flag. Missing or empty token
  files are fatal — falling back to "no token" would expose the host.
* deploy/install-agent.sh: creates a system user, generates the token once
  (re-runs reuse it, --rotate-token for a new one), stores it 0600
  root-owned, and runs the agent under ProtectSystem=strict with no
  capabilities; --status and --uninstall included.
* Containerfile: two-stage, uv.lock resolved in the builder and slim Python
  in the runtime. Verified with podman — the image serves /api/health as an
  unprivileged user and writes its history under /var/lib/dashboards.
  UV_PROJECT_ENVIRONMENT and --no-editable are what make the venv survive
  being copied into the runtime stage.
* .containerignore keeps .git, .venv, data/ and .env out of the build
  context.

258 tests, 94% coverage; ruff and pyright clean.

LLM-Generated: true
2026-10-16 17:33:24 -04:00
ducoterra c4bb374ccc Add homelab dashboards: system, LLM, and UPS on one domain
Three dashboards served by one FastAPI app with a SQLite history store
(raw / 1m / 1h) and hand-written SVG charts: no Prometheus, no Grafana,
no CDN, no build step. Nothing is fetched from the internet at runtime.

* system: a stdlib-only agent (agent/host_agent.py) reports /proc and
  /sys counters; the dashboard turns them into rates over a labelled
  window, so every number says how it was measured.
* llm: gufo/llama.cpp and vLLM /metrics are parsed by an in-house
  exposition parser. Measured delta/elapsed throughput beats a native
  per-request gauge, and each value carries its provenance.
* ups: a NUT wire-protocol client plus health checks that name what
  they looked at. Commands are gated by NUT_ALLOW_ACTIONS,
  DASH_READ_ONLY and a DANGER confirmation, re-checked server-side.

Missing values render as n/a rather than 0, and an unconfigured source
answers 404 {"configured": false} without dialling anything, so "not
deployed here" never looks like an outage.

253 tests at 94% coverage against captured real fixtures; ruff and
pyright clean.

LLM-Generated: true
2026-10-16 17:20:45 -04:00