services/dashboards
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
9 Commits 1 Branch 0 Tags
9757e3c947dcc90c95de188b277a7837f4bcd7d8
Commit Graph
7 Commits
Author SHA1 Message Date
ducoterra 9757e3c947 phase: 03_system_per_host_panels 
**Phase 03 final verification pass — all 4 tasks already in `complete/`; verified criteria, fixed 3 pre-existing defects.**

- **Verified live** (headless Chromium + curl against `:8090`, all 6 sources `ok`): 3 panels × six heroes in order `cpu,mem,temp,disk,gpus,net`; charts per panel `util,net[,gpu]` (gateway: no GPU chart/hole); legends name metrics only — no legend names a foreign host; every `<svg>` `role="img"` + host-scoped `aria-label`; 2 sections, 0 `<select>`; `grep -rn "renderCores|core_usage" src/dashboards/web/` → nothing; console error-free; new shot `data/logs/final-system-panels.png`.
- **Defect fixed (credential hygiene, invariant 6)**: `collectors.py` put the *dialed* URL into every `UpstreamError`, and that string lands in the health record (`/api/health`) and the `event` table (`/api/events` → page); `/api/config` also echoed `GUFO_BASE_URL`/`VLLM_BASE_URL` verbatim. Added `config.strip_userinfo()` (now what `Target.display_url` uses) and redacted all message/config URLs. Tests: `test_a_failing_host_never_echoes_its_token_in_the_message` (×3 paths), `test_a_failing_engine_never_echoes_credentials_in_its_url`, `test_a_dead_host_never_writes_its_token_into_the_feed`, `test_engine_urls_are_redacted_the_same_way_hosts_are`, `test_strip_userinfo_is_the_one_redaction_every_url_leaves_by`. (`parse_targets` already moves `HOST_TARGETS` userinfo into a header, so this is a backstop for hosts and a real fix for engine URLs.)
- **Defect fixed (invariant 2)**: `/system` footnote printed `Δ{HOST_INTERVAL}` as the measurement; it now prints each host's measured `window_seconds` (range when they disagree) — `rates over Δ10.0s · poll 10s`. Pinned by `test_system_footnote_prints_the_window_it_actually_measured`. Also fixed the page's stale "window in each header" note and the empty-state's `--token` (argv) advice → `--token-file`.
- `bash .agents/validate.sh` → **EXIT=0**: **472 passed, 7 skipped**, coverage **96.87%** (floor 94), `ruff check src tests agent` clean, `pyright` 0 errors, `node --check` clean · `uv run pytest -m live --live` → **7 passed** · `uv pip list --outdated` empty.
- Criteria: no multi-host legend ✅ · no per-core visual ✅ · six heroes/panel incl. no-GPU `n/a` uncoloured ✅ · per-core history still queryable ✅ (354 pts) · ≥3 shape screenshots ✅ (`system-driveripper/deskwork/gateway.png` + dark/light/stale) · pytest+coverage+lint ✅ · phases 01/02 unchanged (their pin tests + `/api/summary` shape tests pass; live UPS gap in AGENTS.md re-confirmed read-only against real upsd) ✅.
- **Docs**: AGENTS.md — invariant 6 rewritten to cover message/event redaction, `collectors.py` line, new "Δ is a measurement" rule, "hosts answering total" → "nine-figure fleet-totals panel" (the subtitle still counts hosts), test count 452→472, coverage 96.8→96.9, rewrapped an over-long line. README.md — same fleet-totals wording, `RATE_WINDOW_SECONDS` now covers every measured rate (not just tokens), agent route list completed (`/`, `/schema`, `?top=N`), `LIST CMD` in the diagram, new paragraph on URL redaction.
- **Deviation**: the criterion's literal `curl …/api/system/history?host=…&metric=cpu.usage_pct@core7` uses singular params that endpoint has never accepted (it takes `hosts=`/`metrics=` and ignores unknown ones). Verified 354 points via `…system/history?hosts=driveripper&metrics=cpu.usage_pct@core7&range=1h` and `/api/history?source=host&host=…&metric=…`; I did **not** add singular param aliases (unauthorized API surface). No live infra changes made — reads only.
- **Next pending phase**: `04_llm_chart_readability`.
 2026-10-17 04:42:09 -04:00
ducoterra 8b048599f3 phase: 02_overview_fleet_and_alerts 
**Phase 02 final verification pass — all three tasks already complete; verified criteria, fixed 3 defects found live.**

- Verified: `/` = exactly 2 `<section>`s (Warnings & errors + Fleet), alerts render from `/api/summary` verdicts with each alert's own `href`; `/system`, `/llm`, `/ups` unchanged (charts + data live).
- Fixed (`src/dashboards/alerts.py`): a dead source's message printed the name twice — `gateway: UpstreamError: gateway: http://…`; the poller's `UpstreamError:` wrapper is now stripped (other exception names kept). Tests: 2 new in `tests/test_alerts.py`.
- Fixed (`src/dashboards/api.py`): `/api/summary` dropped a configured host/engine that had never answered post-start, so the fleet said `2/2 hosts` while a third was down. Stub rows (`unreachable: true`, all `n/a`) now match `/api/*/snapshot`; `hosts_total=3 / hosts_live=2`. Test: `test_a_host_that_has_never_answered_is_still_a_row_in_the_fleet`.
- Fixed (`web/static/page-index.js`): fleet row of a non-answering host is now dimmed with the shared `.stale` class (invariant 4 on `/`). Test: `test_overview_dims_the_fleet_row_of_a_host_that_stopped_answering`.
- `bash .agents/validate.sh` exit 0: `uv run pytest --cov --cov-fail-under=94` → **439 passed, 6 skipped, 96.82%** (`alerts.py` 100%, `api.py` 100%); `ruff check src tests agent` clean; `pyright` 0 errors; `node --check web/static` clean; hygiene clean; `uv pip list --outdated` empty; `uv run pytest -m live --live` → **6 passed**.
- Live proof (tunnel kill, then restored — all 6 sources `ok` again): killing the gateway forward produced exactly **one** alert, `fail gateway /system … unreachable (ConnectError)`; screenshots `data/logs/phase02-final-{alert,system,llm,ups,recovered}.png`.
- Completion criteria: two panels ✅ · `[]` when healthy ✅ (hermetic `healthy_client` + a live gateway-only instance returned `[]`; the real fleet can never be `[]` — its UPSes permanently warn on battery age/self-test) · one alert per dead host with `/system` link ✅ · pytest/live/coverage/ruff/pyright ✅ · no change on `/system`,`/llm`,`/ups` ✅ · phase 01 verdict vocabulary intact ✅.
- Docs: README `web/` bullet now states the dimmed `n/a` row; AGENTS.md invariant 4, `alerts.py`/`api.py` bullets, UI-conventions stale-row + window-chip ranges (5m…30d vs 1h…90d), metric example `llm.decode_tps` → `llm.decode_tokens_per_second` (verified against `/api/metrics`), test count 435 → 439. Env table, Containerfile, agent/install-script claims, fleet facts re-checked and consistent.
- Notable: left the fleet's real hostnames/domain in AGENTS.md's "real environment" section (deliberate, prior-phase content) even though rule 6 discourages personal hostnames in tracked files — worth a separate decision; no live config was changed beyond stopping/starting a local dev ssh forward.
- Next pending phase: `.agents/phases/todo/03_system_per_host_panels/`.
 2026-10-17 00:59:09 -04:00
ducoterra ad993c7b03 phase: 01_severity_language 
**Gate defect found & fixed (the "does not lint" bug).** `.agents/validate.sh` — the phase gate — ran `python3 -m pytest` on `/usr/bin/python3` + `~/.local` site-packages (not the uv venv), ran `ruff check .` *only if* a ruff happened to be on PATH, and **never ran pyright or the coverage floor**. Rewrote it to the project's real gate: cd to repo root → `uv run pytest --cov --cov-fail-under=94` → `uv run ruff check src tests agent` → `uv run pyright` → `node --check web/static/*.js` → hygiene gates; a missing toolchain now fails loudly. Provoked it to confirm it bites (floor + a seeded F401 both fail).

**Verification (all green)**
- `bash .agents/validate.sh` → **exit 0**: 356 passed, 6 skipped · coverage **94.75%** (floor 94) · ruff `All checks passed!` · pyright `0 errors` · node syntax clean
- `uv run pytest --cov` → `severity.py` **100%**; `uv run pytest -m live --live` → **6 passed**; `uv pip list --outdated` → empty
- Criteria: `classify('cpu.usage_pct',…)` → `fail`/`ok`/`None` ✓ · `/api/ups/snapshot` carries `severity` beside `charge_pct`/`load_pct`/`battery_runtime` (apc now live-`warn` at 498 s — threshold working) ✓ · no `.sev-*`/synonym vocabulary (only a comment saying so) ✓ · pytest/coverage/lint ✓ · dev server restarted, `/api/health` ok 6/6 sources, all pages+assets 200 ✓
- Chromium against live fleet: `severityFrom` cell→`fail`, bare→`warn`, null→`''`, unrated→`''`; `hero` fail → `rgb(207,34,46)`, null → `n/a` with **no** verdict class, warn+bar 42%; all four pages render unchanged → `data/logs/phase01-final-{index,system,llm,ups}.png`

**Docs (full review, incl. pre-existing drift).** README: `/ups` row claimed "upsd on 3493" (fleet is 3051, stated elsewhere in the same file); config table omitted `DASH_HOST/PORT`, llm/ups `*_INTERVAL_SECONDS`, `*_TIMEOUT_SECONDS`, `GUFO/VLLM_API_KEY`, `POLL_MS_*`, `DEBUGPY` (+ dedup); severity bullet omitted `disks`/`gpus[]`/rated LLM cells and the ratio rescale; agent section listed only `/snapshot` though `/info`+`/health` exist. AGENTS.md: same 3493 claim; `install-agent.sh --token-file` (its flags are `--bind/--port/--token/--rotate-token/--status/--uninstall` — it hands *the agent* `--token-file`); gap 3 "Two commits" (now 6, still no `origin`); gap 1 given the sudoers install command + verified live gating reason; coverage line now 80% repo floor vs 94% gate floor; cyberpower `identity.model: GX1500U` vs `ups.description` quirk. `.env.dev.example`: `DEV_FORWARDS` remote port 3493→3051 and engine URLs that pointed at `127.0.0.1` with nothing forwarding them.

**Deviations.** No `ruff format` (30 files would reformat; the project gate is `ruff check` + pyright — reformatting completed phases is out of scope). **No live-infra changes**: enabling the battery-test buttons needs `deploy/nut-sudoers` installed on driveripper + a nut-server restart and a scoped `[dashboards]` `instcmds` user — prod write with a documented shutdown risk (AGENTS gap 1), so repo-side only. 36 pre-existing `ResourceWarning: unclosed database` from test fixtures are noise, not failures; left rather than churn ~15 fixtures. `severity.attach()` is unit-tested but has no production call site until phases 02–05.

**Next pending phase:** `02_overview_fleet_and_alerts` (`01_alerts_builder.md`).
 2026-10-16 22:22:26 -04:00
ducoterra 1e454a8f81 Read the real UPSes: upsd lives on 3051, not 3493 
Nothing was listening on 3493, so the dashboard said "connection refused"
and fell back to the simulator while upsd was healthy the whole time. It
binds a nonstandard port: `ss -ltnH` + a NUT handshake found it, and
`upsc -L localhost:3051` lists both batteries. No host config had to
change — dev now forwards 3494 -> 3051 and `/ups` shows real numbers,
including an honest "battery is 5.3 years old" on the APC.

Docs now say what to check before blaming the service (`ss -ltnH`,
`printf 'LIST UPS\n' | nc 127.0.0.1 PORT`, `upsc -L host:port`), and the
remaining UPS gap is rewritten: reads work, instant commands still need a
scoped upsd.users entry — which needs a *read* of that file, because
`tee` writing it blind would drop the users upsmon authenticates as.

Real-hardware quirks recorded while verifying: battery.mfr.date is the
date that means something (battery.date is often an unset placeholder),
CyberPower reports `battery.mfr.date: CPS`, and neither driver reports a
battery temperature — all three already render as unknown rather than a
invented number.

LLM-Generated: true
 2026-10-16 19:00:44 -04:00
ducoterra 87042f8855 Add AGENTS.md and align the docs with the code 
The brief asked for an AGENTS.md capturing the overall plan and it never
landed, while two files already pointed readers at it. It now holds the
invariants (read-only by construction, never invent a number,
unconfigured is not broken, no CDN, no guessed metric names), the locked
decisions with their reasons, the verified fleet facts, the commands, and
the four known gaps.

Also corrected the docs that had drifted:
- upsd config lives in /etc/ups on driveripper, not /etc/nut, and the
  real blocker is that upsd binds nothing on 3493 — with the sudo rules
  still needed written down in deploy/nut-sudoers.
- .env.example told operators to set `LISTEN 0.0.0.0` and `actions = -`;
  both are worse than needed. Loopback + ssh forward, and per-command
  instcmds instead of granting load.off.
- nut.py claimed the command path is gated three times and that the
  confirmation asks for the UPS name; it is gated four times and asks
  for DANGER.
- .gitignore described a tracked .agents/ roadmap that does not exist.

LLM-Generated: true
 2026-10-16 18:54:35 -04:00
ducoterra 426682a999 Add the deployment path: agent systemd unit and container image 
The README referenced `deploy/install-agent.sh` and a `Containerfile` that
did not exist, so a fresh clone had no way to actually run this anywhere.

* agent: `--token-file` (and AGENT_TOKEN_FILE). The token guards hostname,
  mounts, disk and GPU models and process names, and argv is world-readable
  through `ps`, so a file has to win over the flag. Missing or empty token
  files are fatal — falling back to "no token" would expose the host.
* deploy/install-agent.sh: creates a system user, generates the token once
  (re-runs reuse it, --rotate-token for a new one), stores it 0600
  root-owned, and runs the agent under ProtectSystem=strict with no
  capabilities; --status and --uninstall included.
* Containerfile: two-stage, uv.lock resolved in the builder and slim Python
  in the runtime. Verified with podman — the image serves /api/health as an
  unprivileged user and writes its history under /var/lib/dashboards.
  UV_PROJECT_ENVIRONMENT and --no-editable are what make the venv survive
  being copied into the runtime stage.
* .containerignore keeps .git, .venv, data/ and .env out of the build
  context.

258 tests, 94% coverage; ruff and pyright clean.

LLM-Generated: true
 2026-10-16 17:33:24 -04:00
ducoterra c4bb374ccc Add homelab dashboards: system, LLM, and UPS on one domain 
Three dashboards served by one FastAPI app with a SQLite history store
(raw / 1m / 1h) and hand-written SVG charts: no Prometheus, no Grafana,
no CDN, no build step. Nothing is fetched from the internet at runtime.

* system: a stdlib-only agent (agent/host_agent.py) reports /proc and
  /sys counters; the dashboard turns them into rates over a labelled
  window, so every number says how it was measured.
* llm: gufo/llama.cpp and vLLM /metrics are parsed by an in-house
  exposition parser. Measured delta/elapsed throughput beats a native
  per-request gauge, and each value carries its provenance.
* ups: a NUT wire-protocol client plus health checks that name what
  they looked at. Commands are gated by NUT_ALLOW_ACTIONS,
  DASH_READ_ONLY and a DANGER confirmation, re-checked server-side.

Missing values render as n/a rather than 0, and an unconfigured source
answers 404 {"configured": false} without dialling anything, so "not
deployed here" never looks like an outage.

253 tests at 94% coverage against captured real fixtures; ruff and
pyright clean.

LLM-Generated: true
 2026-10-16 17:20:45 -04:00