services/dashboards
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
37 Commits 1 Branch 0 Tags
10f86772ee2d0aa3ee9208d7a7a3888e04911cb3
Commit Graph
6 Commits
Author SHA1 Message Date
ducoterra cffbac5771 phase: 15_agent_mtls 
All green. Report:

**Verified / implemented (final pass of `15_agent_mtls`)**
- Wrote `data/logs/verify-15-tls-e2e.py`: a real `host_agent.py --cert-bundle` + a real dashboard process over a scratch DB (`HOST_TARGETS=loop=https://…`) — **27/27 PASS**, transcript `verify-15-tls-e2e.out`, notes `data/logs/NOTES-tls-rollout.md`.
- Read-only checks against the running dev store: 5 `certs.*` rows byte-identical across a restart; served bundle == the `meta` row (1430 B); 0 events containing `-----BEGIN`; unsigned `/certs`→`303 ?next=`, unsigned `/api/certs`→`401`, signed-in `200`; screenshots `data/logs/15-final-{certs,system}.png`.
- Docs (step 6, full review): README — added *Protecting an agent, host by host* (the rollout task-07 left out), a `tests/test_certs.py`/TLS-matrix/sandbox-installer paragraph, and fixed two stale “the one write endpoint in the app” claims (`NUT_ALLOW_ACTIONS`, Battery-test buttons). AGENTS.md — test count 1424→1427, bundle-fetch command in *Commands*, a Testing-policy bullet on structurally pinning a key-holding module, fleet transport bullet (token mode is now the compatibility path), gap 11 final state.

**Tests / lint (full suite 100% green)**
- `uv run pytest` → **1427 passed, 15 skipped**, exit 0 · `bash .agents/validate.sh` → **exit 0** (coverage **98.94%** vs 94% floor, `certs.py` 100%; ruff “All checks passed”; pyright 0 errors; `node --check` clean; hygiene clean)
- `uv run pytest -m live --live` → **11 passed / 0 failed** · `./scripts/dev-server.sh status` → all 9 sources `ok`
- `uv pip list --outdated` → empty after bumping `fastapi 0.152.0 → 0.153.0` (`uv.lock` only); full suite re-run green.

**Completion criteria**
- Fresh-DB root CA, idempotent: ✅ (tests + live store hashes) · agent TLS matrix incl. wrong-CA/certless 403/`/health` CA-only/revoked 403 + live denylist re-read: ✅
- `https` `host:`/`disks:` `ok` + revoked says “agent certificate revoked” with no dial: ✅ on loopback (real agent + real dashboard) and spy-pinned hermetically; **not** on the fleet (below)
- Signed-in `/certs` + rotate/revoke/root-rotate + byte-identical bundle download: ✅ · installer bundle/unit/`--status` over TLS: ✅ (0640, see deviation)
- `pytest`/`validate.sh`/`--outdated`: ✅ · token-mode byte-identical, engine/UPS untouched, public surface unchanged (route walk): ✅
- **Live fleet TLS (deskwork+gateway flip): NOT done** — no live changes. It needs root on hosts without the installer grant (`deskwork`), has no both-ways-working window (`http://` dashboard vs https-only agent), and would deploy the agent from this not-yet-committed tree. Exact procedure recorded in README + `data/logs/NOTES-tls-rollout.md`.

**Deviations**: bundle mode `0640 root:dashboards-agent` instead of the task file’s `0600` (the agent re-reads its own bundle for the denylist; 0600 breaks the unit — already recorded in AGENTS.md); the loopback dashboard substitutes for the fleet flip. Incidental: a stray `pkill` of mine killed the dev server mid-pass; restarted and verified healthy.

**Next pending phase**: `.agents/phases/todo/16_agent_smart_pass_budget` (then `17_installer_device_filter`).
 2026-10-20 10:02:16 -04:00
ducoterra 1bbd868401 phase: 13_llm_page_and_decode_rate 
All green. Final verification summary:

**Verified / implemented (phase 13 final pass)**
- All 4 tasks already in `complete/`; this was the verification pass. Confirmed the CSS rhythm rules (`#strip`, `#banners:empty ~ #fleet`) are served and pinned, `data/logs/NOTES-decode-spike.md` names the mechanism, and the phase-07/04 surfaces still pass their pins.
- **Found and fixed a residual defect in this phase's own code**: a stalled fetch stamps its body with the poll's *finish* time, so that sample anchors every window for the rest of the span; the jump gate "confirmed" a jump measured from that same anchor → live gufo rows 232.28 tok/s (one request at ~50) + a decaying 146/114/95 tail, repeated at 200.6. `rates.py::_publish` now requires an **independent anchor** (`_pending` stores the base sample's ts; the hold continues until it ages out). Pinned by `test_a_held_jump_is_not_confirmed_by_a_window_on_the_same_anchor` + `test_gufo_a_stale_scrape_body_never_becomes_a_rate_above_the_engine` (both fail on task 03's gate).
- Re-verified live after a dev-server restart: 10 min / 540 signed-in `/api/llm/snapshot` polls + read-only SQLite pass; **0 values > 300, 0 without a `Δ<window>` label**; 0 post-fix stored rows over the ceiling except the two pre-fix-shape rows above.

**Tests / lint (exact, all 100 % green)**
- `bash .agents/validate.sh` → **EXIT=0**; `uv run pytest --cov --cov-fail-under=94` → **1195 passed, 11 skipped, coverage 98.70 %**; `ruff check src tests agent` → All checks passed; `pyright` → 0 errors; `node --check` clean; repo hygiene clean.
- `uv run pytest -m live --live` → **11 passed / 0 failed**. `/api/health` → **all nine sources ok**.

**Completion criteria**
- Fleet/strip + head/fleet gap with screenshots (`llm-fleet-spacing{,-top,-banner-case}.png`) — met (CSS unchanged this pass, pinned on served bytes).
- Task-02 findings file names the confirmed mechanism — met.
- `test_rates.py`/`test_llm_source.py` pin label-set, both window bounds and the always-honest `Δ` — met (+2 new pins).
- Sustained watch under the recorded ceiling (300 tok/s) — met (worst 203.05, all labelled).
- Full suite + validate green; no phase-07/04 behaviour change — met.

**Documentation**
- README: `RATE_WINDOW_SECONDS` row now states the hold/confirm rule (only change needed; rest verified consistent).
- AGENTS.md: `rates.py` bullet + gap 9 rewritten (residual closed *and* the remaining sub-8× skew measured, root-caused to gap 4, explicitly not fixed by inventing a new threshold).
- **Fleet facts changed under this pass and were corrected**: operator added driveripper's bearer token → `host:driveripper`/`disks:driveripper` `ok`, live suite 9/2→11/0; but driveripper reads **identity only, zero SMART numbers** (root `smartctl` gives 55 °C/14 %; `/dev/nvme0`, `/dev/ng0n1`, `/dev/bsg/*` are root-only; its unit is `DevicePolicy=auto`, not the installer's `strict`). Evidence appended to `data/logs/NOTES-agent-install.md` + `NOTES-decode-stable.md`.

**Notable decisions / deviations**
- Did **not** add a span-relative window floor: that would re-decide task 03's pinned `MIN_WINDOW_SECONDS` and its "in-bounds diluted row publishes" test. Recorded as the next phase's call, with the root cause left at gap 4 / `14_store_performance` (the ~10 s stall).
- No live fleet change made (read-only probes + one dev-server restart). Fixtures deliberately not re-captured (no parse change, task 02 proved no shape drift). Dependency-currency advisory left untouched (no dependency moved in this phase): certifi/fastapi/starlette/websockets have newer releases.

**Next pending phase:** `.agents/phases/todo/14_store_performance` (then `15_agent_mtls`).
 2026-10-19 14:53:58 -04:00
ducoterra 28139df6d6 phase: 12_nut_self_test_verdict 
Final state verified. Report:

**Verified (phase 12 — all 3 tasks already complete; this was the verification pass)**
- Re-ran the wire capture live: `ups.test.result` = `No test initiated` (apc) / `Done and passed` (cyberpower) — matches `LIVE_SELF_TEST_WORDS` + both fixtures; `/api/summary` has **no** self-test row, `/api/ups/snapshot` reads `cyberpower ok | Done and passed`, `apc unknown | …run a battery test`; both screenshots read back correct.
- Fixed a defect found in prior work while checking the "unrelated live noise" note: `NutMonitor.collect()` reused one upsd socket after a *partially-read* reply, so a 4 s stall produced bogus `expected 'BEGIN LIST VAR', got 'RW apc battery.charge.low "10"'` errors (same hazard on `INSTCMD`, where a late `OK` could be read as a different command's success). `NutClient` now marks a socket whose reply stopped early as `UNUSABLE`, naming the command that stalled; a complete `ERR …` line still leaves it usable (the per-UPS tolerance `collect()` depends on). +3 hermetic pins against the fake upsd; live re-check (09:33–09:44) now logs one line per stall, no cascade.

**Tests / lint / coverage (all green)**
- `bash .agents/validate.sh` → **exit 0**; `uv run pytest --cov --cov-fail-under=94` → **1172 passed, 11 skipped**, coverage **98.63 %** (floor 94); `uv run ruff check src tests agent` clean; `uv run pyright` 0 errors; `node --check` clean. Modified modules: `sources/nut.py` 95 %, `collectors.py` 97 % (no new line uncovered).
- `uv run pytest -m live --live` → **8 passed / 3 failed**, all three the pre-existing `driveripper` agent-transport failures AGENTS.md records (verified still true: unit `inactive`, no installed file, nothing on `:9120`).
- `./scripts/dev-server.sh status`: `nut ok`, `llm:* ok`, `host/disks:deskwork|gateway ok`; the two `driveripper` jobs DOWN.

**Completion criteria**
- Live capture recorded + asserts `ok` — met (re-confirmed from the wire today).
- `/ups` renders `ok`, overview silent, screenshots present — met.
- pytest / cov / ruff / pyright / validate — met (above).
- No other `health.checks[]` behaviour changed — met (diff confined to the `test_result` branch; all prior nut/ups-contract tests unmodified).

**Documentation**: README verified consistent (self-test wording, config table cross-checked against `config.py`, all documented commands/flags exist) — no changes needed beyond phase tasks' edit. AGENTS.md fixes: stale test count (1140 → 1172), `nvme0n1` power-on hours 19 158 → 19 160, gap 4 store-error claim corrected (last stored `store-error` 2026-10-18 06:33, none in the previous 24 h), new `collectors.py`/`sources/nut.py` socket-hygiene note, new gap-1 bullet on the upsd stall, plus live evidence appended to `data/logs/NOTES-self-test.md`.

**Deviations**: made no live *changes* (read-only probes only) — `driveripper`'s agent install needs root (`ops` has no installer/visudo grant) and a battery test stays the operator's call; the upsd 4 s stall itself is upstream-side. `uv pip list --outdated` still advisory (certifi/fastapi/starlette) — dependencies untouched this phase.

**Next pending phase**: `13_llm_page_and_decode_rate`.
 2026-10-19 09:54:03 -04:00
ducoterra fefbeb8227 phase: 09_disks_smartctl 
**Phase 09 — final verification pass (all 4 task files already in `complete/`; nothing re-implemented)**

- Verified `GET /api/disks/snapshot` live: 3 hosts / 18 whole-`/sys/block` rows, `smartctl_available` three-way, per-row `severity`, API-summed `fleet`; 404 `{"configured": false}` proven *without dialling* (spies on `httpx.AsyncClient.get` + `collectors.fetch_snapshot`).
- Verified `/api/disks/history` live (`disk.temp_c@dev`, unknown names → empty series) + `test_store.py`'s 300 s rollup/prune buckets; `/api/metrics?source=disks` answers the `disks` family.
- Browser against live data: 18 drive cards (never a partition/mount), collapsed per-host `Charts` with `aria-label`, 3 `.stale` grids + one `.banner.info` why-line and **0 `.fail` classes / green `disks` dot** → criteria 3 and 4 hold; new shots `data/logs/09-verify-disks-live-dimmed.png`, `09-verify-overview-live.png` (13 total for the phase).
- Real-hardware criterion: read 4 driveripper drives through `ops`' `sudo -n smartctl -i -A -H` and ran the live output through `host_agent.parse_smart_output` → `sources/disks.parse_disks_snapshot`: `sda` 35 °C / 30 423 h, `nvme1n1` 58 % wear, `nvme5n1` 13 uncorrectable → `alarm_disks: 1`, temps graded `warn`; consistent with the 16 committed fixtures.
- `bash .agents/validate.sh` → **EXIT=0**; `uv run pytest --cov` → **769 passed, 11 skipped, coverage 98.33%** (floor 94); `ruff check src tests agent` clean; `pyright` 0 errors; `node --check` clean; hygiene + dependency advisories clean. `uv run pytest -m live --live` → 10 passed, 1 skipped (agents still predate the SMART pass). Dev server healthy: `/api/health` `ok`, 9/9 sources.
- Docs fixed: AGENTS.md test count 698→769 and coverage 98.2→98.33 %; `disks` added to the event dashboard-name list and to the "every feed renders `Dash.eventText`" sentence; README `POLL_MS_*` row now states `/disks`' deliberate fixed 30 s; new Known gap 4 records the dev-DB `database is locked` writes (reported/recovered per invariant 4, ~590 MB DB / ~70 MB WAL, not reproduced by a controlled 25 s external read txn).
- Deviations: **no live-infra change** — granting the agent user the `disk` group and redeploying `host_agent.py` on driveripper/deskwork stay the documented paired operator action (redeploying without the grant yields 16 permission-denied lines; a sudo-spawning agent would break the confirmed "direct read, no sudo" assumption). Store-lock issue left to its own phase rather than patched speculatively.
- Next pending phase: `10_dev_direct_dial`.
 2026-10-18 00:45:01 -04:00
ducoterra 9757e3c947 phase: 03_system_per_host_panels 
**Phase 03 final verification pass — all 4 tasks already in `complete/`; verified criteria, fixed 3 pre-existing defects.**

- **Verified live** (headless Chromium + curl against `:8090`, all 6 sources `ok`): 3 panels × six heroes in order `cpu,mem,temp,disk,gpus,net`; charts per panel `util,net[,gpu]` (gateway: no GPU chart/hole); legends name metrics only — no legend names a foreign host; every `<svg>` `role="img"` + host-scoped `aria-label`; 2 sections, 0 `<select>`; `grep -rn "renderCores|core_usage" src/dashboards/web/` → nothing; console error-free; new shot `data/logs/final-system-panels.png`.
- **Defect fixed (credential hygiene, invariant 6)**: `collectors.py` put the *dialed* URL into every `UpstreamError`, and that string lands in the health record (`/api/health`) and the `event` table (`/api/events` → page); `/api/config` also echoed `GUFO_BASE_URL`/`VLLM_BASE_URL` verbatim. Added `config.strip_userinfo()` (now what `Target.display_url` uses) and redacted all message/config URLs. Tests: `test_a_failing_host_never_echoes_its_token_in_the_message` (×3 paths), `test_a_failing_engine_never_echoes_credentials_in_its_url`, `test_a_dead_host_never_writes_its_token_into_the_feed`, `test_engine_urls_are_redacted_the_same_way_hosts_are`, `test_strip_userinfo_is_the_one_redaction_every_url_leaves_by`. (`parse_targets` already moves `HOST_TARGETS` userinfo into a header, so this is a backstop for hosts and a real fix for engine URLs.)
- **Defect fixed (invariant 2)**: `/system` footnote printed `Δ{HOST_INTERVAL}` as the measurement; it now prints each host's measured `window_seconds` (range when they disagree) — `rates over Δ10.0s · poll 10s`. Pinned by `test_system_footnote_prints_the_window_it_actually_measured`. Also fixed the page's stale "window in each header" note and the empty-state's `--token` (argv) advice → `--token-file`.
- `bash .agents/validate.sh` → **EXIT=0**: **472 passed, 7 skipped**, coverage **96.87%** (floor 94), `ruff check src tests agent` clean, `pyright` 0 errors, `node --check` clean · `uv run pytest -m live --live` → **7 passed** · `uv pip list --outdated` empty.
- Criteria: no multi-host legend ✅ · no per-core visual ✅ · six heroes/panel incl. no-GPU `n/a` uncoloured ✅ · per-core history still queryable ✅ (354 pts) · ≥3 shape screenshots ✅ (`system-driveripper/deskwork/gateway.png` + dark/light/stale) · pytest+coverage+lint ✅ · phases 01/02 unchanged (their pin tests + `/api/summary` shape tests pass; live UPS gap in AGENTS.md re-confirmed read-only against real upsd) ✅.
- **Docs**: AGENTS.md — invariant 6 rewritten to cover message/event redaction, `collectors.py` line, new "Δ is a measurement" rule, "hosts answering total" → "nine-figure fleet-totals panel" (the subtitle still counts hosts), test count 452→472, coverage 96.8→96.9, rewrapped an over-long line. README.md — same fleet-totals wording, `RATE_WINDOW_SECONDS` now covers every measured rate (not just tokens), agent route list completed (`/`, `/schema`, `?top=N`), `LIST CMD` in the diagram, new paragraph on URL redaction.
- **Deviation**: the criterion's literal `curl …/api/system/history?host=…&metric=cpu.usage_pct@core7` uses singular params that endpoint has never accepted (it takes `hosts=`/`metrics=` and ignores unknown ones). Verified 354 points via `…system/history?hosts=driveripper&metrics=cpu.usage_pct@core7&range=1h` and `/api/history?source=host&host=…&metric=…`; I did **not** add singular param aliases (unauthorized API surface). No live infra changes made — reads only.
- **Next pending phase**: `04_llm_chart_readability`.
 2026-10-17 04:42:09 -04:00
ducoterra c4bb374ccc Add homelab dashboards: system, LLM, and UPS on one domain 
Three dashboards served by one FastAPI app with a SQLite history store
(raw / 1m / 1h) and hand-written SVG charts: no Prometheus, no Grafana,
no CDN, no build step. Nothing is fetched from the internet at runtime.

* system: a stdlib-only agent (agent/host_agent.py) reports /proc and
  /sys counters; the dashboard turns them into rates over a labelled
  window, so every number says how it was measured.
* llm: gufo/llama.cpp and vLLM /metrics are parsed by an in-house
  exposition parser. Measured delta/elapsed throughput beats a native
  per-request gauge, and each value carries its provenance.
* ups: a NUT wire-protocol client plus health checks that name what
  they looked at. Commands are gated by NUT_ALLOW_ACTIONS,
  DASH_READ_ONLY and a DANGER confirmation, re-checked server-side.

Missing values render as n/a rather than 0, and an unconfigured source
answers 404 {"configured": false} without dialling anything, so "not
deployed here" never looks like an outage.

253 tests at 94% coverage against captured real fixtures; ruff and
pyright clean.

LLM-Generated: true
 2026-10-16 17:20:45 -04:00