services/dashboards
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
13 Commits 1 Branch 0 Tags
1f20a5e57f96fb6c93abf0595dc9a5ccc68afdeb
Commit Graph
9 Commits
Author SHA1 Message Date
ducoterra 34854d32af phase: 05_ups_hero_battery_glyph 
**Phase 05 final pass — all criteria met; whole project 100 % green.**

Verified (live upsd + isolated simulator):
- Real `apc`/`cyberpower`: Charge/Load/Runtime are the card's three largest elements (38.4 px vs 15.2 px next), coloured only by API `severity` — live right now 100 % ok / 79.0 % **warn** / 5m **fail**; glyph fill = charge (100/100 viewBox units, `--ok` token), `<title>` "apc: Charge 100 percent, on line"; SIMULATED hidden; battery °C `n/a` uncoloured; every button disabled with its reason; `confirm_text: DANGER` on power-cutting commands; **0 non-GET requests** → `data/logs/ups-verify-live-{card,full}.png`.
- Simulator on `:8091` (own `/tmp` DB, killed after): caution dialog → Cancel, dangerous dialog → Run disabled until `DANGER` → Cancel, **0 POSTs**; one deliberate quick test *to the sim only* flipped the card to `OB` (glyph `stroke-dasharray="7 5"`, "on battery", health "discharging, on battery") then `OL CHRG` (bolt) then `OL` → `ups-verify-sim-{dialog-caution,dialog-danger,onbattery-button,onbattery-page,charging-button}.png`; glyph matrix `ups-glyph-*.png` (100/mid/low/unknown/empty/unrated, dark/light/200 %) present; `grep -c "function battery"` = 1.

Defects found in earlier work and fixed:
1. **`poller.py` could stop polling in silence** — 4 of 6 dev-server sources had frozen `attempts` (ages 12–38 min) while `/api/health` still said `ok: true`: the store-write half of `run_once` was outside the `try`, and a dead task's exception is never retrieved. Now `_commit()` moves `last_ok` only after the write lands, `_store_refused()` reports it (`ok:false`, real reason, one `store-error` event + one `recording` on recovery, payload kept), and `_loop()` catches anything that escapes and keeps polling — 4 new tests in `tests/test_poller.py`.
2. **`nut_sim.py` (modified by this phase) sat at 83 % coverage**; its whole command path was untested. Added tests (every advertised command implemented, unknown → `ValueError`, alarms → card + `assess()` fail, `NUT_SIMULATE` truthy rule) and fixed two model lies: `beeper.*` answered OK while changing nothing (now publishes `ups.beeper.status` enabled/muted/disabled) and a finished self-test never wrote `ups.test.result` (aborted writes none). → 100 %.
3. `nut_sim.from_env()` was dead code duplicating `app._truthy`; `app.py` now builds the simulator through it, so the enable gate lives once.

Tests / lint (exact): `bash .agents/validate.sh` → **EXIT=0** — `uv run pytest --cov --cov-fail-under=94` **543 passed, 8 skipped, coverage 97.91 %**; `uv run ruff check src tests agent` clean; `uv run pyright` 0 errors; `node --check` clean; repo hygiene clean. `uv run pytest -m live --live` **with `.env.dev` sourced → 8 passed** (one earlier run: 2 transient `9121/snapshot timed out` against the 4 s `HOST_TIMEOUT`; three later runs 8/8). `uv pip list --outdated` empty. Dev server restarted on the final code: `health ok`, 6/6 sources fresh.

Docs: AGENTS.md — invariant 4 extended ("nothing may stop polling in silence" + store-refusal semantics), `poller.py`/`nut_sim.py` responsibility lines, 533→543 and 96.9 %→97.9 %, "~97 %"→"~98 %". README.md — poller bullet gained the same guarantee, Testing gained "source `.env.dev` before `-m live --live`" (bare runs skip 7 of 8 live tests, which reads like a pass), Battery-test section gained the `deploy/nut-sudoers` install line and the read-before-write warning.

Notes/deviations: no live instant command was ever sent to real hardware (the four gates leave them disabled — AGENTS gap 1 unchanged); the on-battery proof was driven through the real button against `NUT_SIMULATE=1`, which is stronger than the NOTES hand-driven states. `NOTES-verify.md` under `todo/` left untouched per instructions. Next phase: **none** — 05 is the last in the PLAN.md roadmap.
 2026-10-17 10:27:23 -04:00
ducoterra f44a617ef9 phase: 04_llm_chart_readability 
**Phase 04 final verification pass — all tasks already in `complete/`; verified + fixed 3 defects found in verification.**

- Verified `/llm` live (both engines, gufo-only instance on :8099, `llm_monitor` on :8095) in a real browser; dev server left running/healthy.
- Completed task-04's truncated layout fix: `#engines` now carries `.items-start` (the CSS rule existed but nothing used it) — gufo 544 px vs vLLM 715 px, no blank card; pinned by `test_llm_engine_panels_are_not_stretched_to_their_tallest_sibling`.
- Fixed: event rows said the subject twice ("gufo gufo is answering again"), and `/ups` mangled mid-message names ("sent to apc: OK" → "sent to  OK") → shared `Dash.eventText()` (strips `UpstreamError: ` + a *leading* name only, mirroring `alerts._clean_error`) used by `/system`, `/llm`, `/ups`; new test.
- Fixed: count sparklines autoscaled rollup averages (queue max 0.1379) into full-height spikes → `sparkline({minSpan})`, `minSpan: 1` on sessions/queued (spike 30 px → 4 px); assertions added.
- Tests: `bash .agents/validate.sh` → **EXIT=0** (`uv run pytest --cov --cov-fail-under=94`: **485 passed, 7 skipped**, coverage **96.87%**; `ruff check src tests agent`: All checks passed; `pyright`: 0 errors; `node --check` all static JS clean; repo hygiene clean; `uv pip list --outdated` empty). `uv run pytest -m live --live` → **7 passed**.
- Completion criteria: heroes largest per engine ✔ (38.4 px vs 16.8 px row values); sparkline per cell + selected window ✔ (5/5 gufo, 7/7 vLLM; 6 h chip moved KV max 53 %→58 %); mean+p95 without hover ✔ (243 / 48.8 / 285 ms feet); provenance tag ✔ (`Δ10.0s`/`Δ8.0s` + `title`=exact source); single-engine ✔ (1 panel, totals hidden, zero red); screenshots ✔ `data/logs/llm-verify-{both-engines,charts-and-slots,single-engine,monitor-reference}.png` + `NOTES-chart-set.md`; pytest/cov/ruff/pyright ✔; earlier phases unchanged-behaviour ✔ (all 01–03 tests pass).
- Deviation (documented, not code): throughput heroes stay **neutral ink** — `severity.py` has no throughput row and 0 tok/s idle is healthy; colour comes from rated cells (KV) + reachability. Kept per `NOTES-chart-set`.
- Deviation: could not append the task-04 written verdict to `NOTES-chart-set.md` (it lives in `todo/`, which I must not edit). Verdict: ours reads better than `:8095` — bigger throughput numbers, per-cell shape over the selected window, percentiles inline; `llm_monitor` only sparklines decode+KV and has no window control.
- Docs: AGENTS.md updated (485 test count, `sparkline` `minSpan` rule, `Dash.eventText` event-row rule, `/llm` panel `items-start`); README reviewed in full — commands, config table (all `config.py` vars present), architecture bullets, page descriptions verified accurate; no other inconsistencies found.
- Note for the harness: `.agents/phases/todo/03…/NOTES-verify.md` and `todo/04…/NOTES-chart-set.md` are untracked notes left behind by their phases (task files moved, notes did not).
- Next pending phase: `05_ups_hero_battery_glyph`.
 2026-10-17 07:03:58 -04:00
ducoterra 9757e3c947 phase: 03_system_per_host_panels 
**Phase 03 final verification pass — all 4 tasks already in `complete/`; verified criteria, fixed 3 pre-existing defects.**

- **Verified live** (headless Chromium + curl against `:8090`, all 6 sources `ok`): 3 panels × six heroes in order `cpu,mem,temp,disk,gpus,net`; charts per panel `util,net[,gpu]` (gateway: no GPU chart/hole); legends name metrics only — no legend names a foreign host; every `<svg>` `role="img"` + host-scoped `aria-label`; 2 sections, 0 `<select>`; `grep -rn "renderCores|core_usage" src/dashboards/web/` → nothing; console error-free; new shot `data/logs/final-system-panels.png`.
- **Defect fixed (credential hygiene, invariant 6)**: `collectors.py` put the *dialed* URL into every `UpstreamError`, and that string lands in the health record (`/api/health`) and the `event` table (`/api/events` → page); `/api/config` also echoed `GUFO_BASE_URL`/`VLLM_BASE_URL` verbatim. Added `config.strip_userinfo()` (now what `Target.display_url` uses) and redacted all message/config URLs. Tests: `test_a_failing_host_never_echoes_its_token_in_the_message` (×3 paths), `test_a_failing_engine_never_echoes_credentials_in_its_url`, `test_a_dead_host_never_writes_its_token_into_the_feed`, `test_engine_urls_are_redacted_the_same_way_hosts_are`, `test_strip_userinfo_is_the_one_redaction_every_url_leaves_by`. (`parse_targets` already moves `HOST_TARGETS` userinfo into a header, so this is a backstop for hosts and a real fix for engine URLs.)
- **Defect fixed (invariant 2)**: `/system` footnote printed `Δ{HOST_INTERVAL}` as the measurement; it now prints each host's measured `window_seconds` (range when they disagree) — `rates over Δ10.0s · poll 10s`. Pinned by `test_system_footnote_prints_the_window_it_actually_measured`. Also fixed the page's stale "window in each header" note and the empty-state's `--token` (argv) advice → `--token-file`.
- `bash .agents/validate.sh` → **EXIT=0**: **472 passed, 7 skipped**, coverage **96.87%** (floor 94), `ruff check src tests agent` clean, `pyright` 0 errors, `node --check` clean · `uv run pytest -m live --live` → **7 passed** · `uv pip list --outdated` empty.
- Criteria: no multi-host legend ✅ · no per-core visual ✅ · six heroes/panel incl. no-GPU `n/a` uncoloured ✅ · per-core history still queryable ✅ (354 pts) · ≥3 shape screenshots ✅ (`system-driveripper/deskwork/gateway.png` + dark/light/stale) · pytest+coverage+lint ✅ · phases 01/02 unchanged (their pin tests + `/api/summary` shape tests pass; live UPS gap in AGENTS.md re-confirmed read-only against real upsd) ✅.
- **Docs**: AGENTS.md — invariant 6 rewritten to cover message/event redaction, `collectors.py` line, new "Δ is a measurement" rule, "hosts answering total" → "nine-figure fleet-totals panel" (the subtitle still counts hosts), test count 452→472, coverage 96.8→96.9, rewrapped an over-long line. README.md — same fleet-totals wording, `RATE_WINDOW_SECONDS` now covers every measured rate (not just tokens), agent route list completed (`/`, `/schema`, `?top=N`), `LIST CMD` in the diagram, new paragraph on URL redaction.
- **Deviation**: the criterion's literal `curl …/api/system/history?host=…&metric=cpu.usage_pct@core7` uses singular params that endpoint has never accepted (it takes `hosts=`/`metrics=` and ignores unknown ones). Verified 354 points via `…system/history?hosts=driveripper&metrics=cpu.usage_pct@core7&range=1h` and `/api/history?source=host&host=…&metric=…`; I did **not** add singular param aliases (unauthorized API surface). No live infra changes made — reads only.
- **Next pending phase**: `04_llm_chart_readability`.
 2026-10-17 04:42:09 -04:00
ducoterra 8b048599f3 phase: 02_overview_fleet_and_alerts 
**Phase 02 final verification pass — all three tasks already complete; verified criteria, fixed 3 defects found live.**

- Verified: `/` = exactly 2 `<section>`s (Warnings & errors + Fleet), alerts render from `/api/summary` verdicts with each alert's own `href`; `/system`, `/llm`, `/ups` unchanged (charts + data live).
- Fixed (`src/dashboards/alerts.py`): a dead source's message printed the name twice — `gateway: UpstreamError: gateway: http://…`; the poller's `UpstreamError:` wrapper is now stripped (other exception names kept). Tests: 2 new in `tests/test_alerts.py`.
- Fixed (`src/dashboards/api.py`): `/api/summary` dropped a configured host/engine that had never answered post-start, so the fleet said `2/2 hosts` while a third was down. Stub rows (`unreachable: true`, all `n/a`) now match `/api/*/snapshot`; `hosts_total=3 / hosts_live=2`. Test: `test_a_host_that_has_never_answered_is_still_a_row_in_the_fleet`.
- Fixed (`web/static/page-index.js`): fleet row of a non-answering host is now dimmed with the shared `.stale` class (invariant 4 on `/`). Test: `test_overview_dims_the_fleet_row_of_a_host_that_stopped_answering`.
- `bash .agents/validate.sh` exit 0: `uv run pytest --cov --cov-fail-under=94` → **439 passed, 6 skipped, 96.82%** (`alerts.py` 100%, `api.py` 100%); `ruff check src tests agent` clean; `pyright` 0 errors; `node --check web/static` clean; hygiene clean; `uv pip list --outdated` empty; `uv run pytest -m live --live` → **6 passed**.
- Live proof (tunnel kill, then restored — all 6 sources `ok` again): killing the gateway forward produced exactly **one** alert, `fail gateway /system … unreachable (ConnectError)`; screenshots `data/logs/phase02-final-{alert,system,llm,ups,recovered}.png`.
- Completion criteria: two panels ✅ · `[]` when healthy ✅ (hermetic `healthy_client` + a live gateway-only instance returned `[]`; the real fleet can never be `[]` — its UPSes permanently warn on battery age/self-test) · one alert per dead host with `/system` link ✅ · pytest/live/coverage/ruff/pyright ✅ · no change on `/system`,`/llm`,`/ups` ✅ · phase 01 verdict vocabulary intact ✅.
- Docs: README `web/` bullet now states the dimmed `n/a` row; AGENTS.md invariant 4, `alerts.py`/`api.py` bullets, UI-conventions stale-row + window-chip ranges (5m…30d vs 1h…90d), metric example `llm.decode_tps` → `llm.decode_tokens_per_second` (verified against `/api/metrics`), test count 435 → 439. Env table, Containerfile, agent/install-script claims, fleet facts re-checked and consistent.
- Notable: left the fleet's real hostnames/domain in AGENTS.md's "real environment" section (deliberate, prior-phase content) even though rule 6 discourages personal hostnames in tracked files — worth a separate decision; no live config was changed beyond stopping/starting a local dev ssh forward.
- Next pending phase: `.agents/phases/todo/03_system_per_host_panels/`.
 2026-10-17 00:59:09 -04:00
ducoterra ad993c7b03 phase: 01_severity_language 
**Gate defect found & fixed (the "does not lint" bug).** `.agents/validate.sh` — the phase gate — ran `python3 -m pytest` on `/usr/bin/python3` + `~/.local` site-packages (not the uv venv), ran `ruff check .` *only if* a ruff happened to be on PATH, and **never ran pyright or the coverage floor**. Rewrote it to the project's real gate: cd to repo root → `uv run pytest --cov --cov-fail-under=94` → `uv run ruff check src tests agent` → `uv run pyright` → `node --check web/static/*.js` → hygiene gates; a missing toolchain now fails loudly. Provoked it to confirm it bites (floor + a seeded F401 both fail).

**Verification (all green)**
- `bash .agents/validate.sh` → **exit 0**: 356 passed, 6 skipped · coverage **94.75%** (floor 94) · ruff `All checks passed!` · pyright `0 errors` · node syntax clean
- `uv run pytest --cov` → `severity.py` **100%**; `uv run pytest -m live --live` → **6 passed**; `uv pip list --outdated` → empty
- Criteria: `classify('cpu.usage_pct',…)` → `fail`/`ok`/`None` ✓ · `/api/ups/snapshot` carries `severity` beside `charge_pct`/`load_pct`/`battery_runtime` (apc now live-`warn` at 498 s — threshold working) ✓ · no `.sev-*`/synonym vocabulary (only a comment saying so) ✓ · pytest/coverage/lint ✓ · dev server restarted, `/api/health` ok 6/6 sources, all pages+assets 200 ✓
- Chromium against live fleet: `severityFrom` cell→`fail`, bare→`warn`, null→`''`, unrated→`''`; `hero` fail → `rgb(207,34,46)`, null → `n/a` with **no** verdict class, warn+bar 42%; all four pages render unchanged → `data/logs/phase01-final-{index,system,llm,ups}.png`

**Docs (full review, incl. pre-existing drift).** README: `/ups` row claimed "upsd on 3493" (fleet is 3051, stated elsewhere in the same file); config table omitted `DASH_HOST/PORT`, llm/ups `*_INTERVAL_SECONDS`, `*_TIMEOUT_SECONDS`, `GUFO/VLLM_API_KEY`, `POLL_MS_*`, `DEBUGPY` (+ dedup); severity bullet omitted `disks`/`gpus[]`/rated LLM cells and the ratio rescale; agent section listed only `/snapshot` though `/info`+`/health` exist. AGENTS.md: same 3493 claim; `install-agent.sh --token-file` (its flags are `--bind/--port/--token/--rotate-token/--status/--uninstall` — it hands *the agent* `--token-file`); gap 3 "Two commits" (now 6, still no `origin`); gap 1 given the sudoers install command + verified live gating reason; coverage line now 80% repo floor vs 94% gate floor; cyberpower `identity.model: GX1500U` vs `ups.description` quirk. `.env.dev.example`: `DEV_FORWARDS` remote port 3493→3051 and engine URLs that pointed at `127.0.0.1` with nothing forwarding them.

**Deviations.** No `ruff format` (30 files would reformat; the project gate is `ruff check` + pyright — reformatting completed phases is out of scope). **No live-infra changes**: enabling the battery-test buttons needs `deploy/nut-sudoers` installed on driveripper + a nut-server restart and a scoped `[dashboards]` `instcmds` user — prod write with a documented shutdown risk (AGENTS gap 1), so repo-side only. 36 pre-existing `ResourceWarning: unclosed database` from test fixtures are noise, not failures; left rather than churn ~15 fixtures. `severity.attach()` is unit-tested but has no production call site until phases 02–05.

**Next pending phase:** `02_overview_fleet_and_alerts` (`01_alerts_builder.md`).
 2026-10-16 22:22:26 -04:00
ducoterra 1e454a8f81 Read the real UPSes: upsd lives on 3051, not 3493 
Nothing was listening on 3493, so the dashboard said "connection refused"
and fell back to the simulator while upsd was healthy the whole time. It
binds a nonstandard port: `ss -ltnH` + a NUT handshake found it, and
`upsc -L localhost:3051` lists both batteries. No host config had to
change — dev now forwards 3494 -> 3051 and `/ups` shows real numbers,
including an honest "battery is 5.3 years old" on the APC.

Docs now say what to check before blaming the service (`ss -ltnH`,
`printf 'LIST UPS\n' | nc 127.0.0.1 PORT`, `upsc -L host:port`), and the
remaining UPS gap is rewritten: reads work, instant commands still need a
scoped upsd.users entry — which needs a *read* of that file, because
`tee` writing it blind would drop the users upsmon authenticates as.

Real-hardware quirks recorded while verifying: battery.mfr.date is the
date that means something (battery.date is often an unset placeholder),
CyberPower reports `battery.mfr.date: CPS`, and neither driver reports a
battery temperature — all three already render as unknown rather than a
invented number.

LLM-Generated: true
 2026-10-16 19:00:44 -04:00
ducoterra 87042f8855 Add AGENTS.md and align the docs with the code 
The brief asked for an AGENTS.md capturing the overall plan and it never
landed, while two files already pointed readers at it. It now holds the
invariants (read-only by construction, never invent a number,
unconfigured is not broken, no CDN, no guessed metric names), the locked
decisions with their reasons, the verified fleet facts, the commands, and
the four known gaps.

Also corrected the docs that had drifted:
- upsd config lives in /etc/ups on driveripper, not /etc/nut, and the
  real blocker is that upsd binds nothing on 3493 — with the sudo rules
  still needed written down in deploy/nut-sudoers.
- .env.example told operators to set `LISTEN 0.0.0.0` and `actions = -`;
  both are worse than needed. Loopback + ssh forward, and per-command
  instcmds instead of granting load.off.
- nut.py claimed the command path is gated three times and that the
  confirmation asks for the UPS name; it is gated four times and asks
  for DANGER.
- .gitignore described a tracked .agents/ roadmap that does not exist.

LLM-Generated: true
 2026-10-16 18:54:35 -04:00
ducoterra 426682a999 Add the deployment path: agent systemd unit and container image 
The README referenced `deploy/install-agent.sh` and a `Containerfile` that
did not exist, so a fresh clone had no way to actually run this anywhere.

* agent: `--token-file` (and AGENT_TOKEN_FILE). The token guards hostname,
  mounts, disk and GPU models and process names, and argv is world-readable
  through `ps`, so a file has to win over the flag. Missing or empty token
  files are fatal — falling back to "no token" would expose the host.
* deploy/install-agent.sh: creates a system user, generates the token once
  (re-runs reuse it, --rotate-token for a new one), stores it 0600
  root-owned, and runs the agent under ProtectSystem=strict with no
  capabilities; --status and --uninstall included.
* Containerfile: two-stage, uv.lock resolved in the builder and slim Python
  in the runtime. Verified with podman — the image serves /api/health as an
  unprivileged user and writes its history under /var/lib/dashboards.
  UV_PROJECT_ENVIRONMENT and --no-editable are what make the venv survive
  being copied into the runtime stage.
* .containerignore keeps .git, .venv, data/ and .env out of the build
  context.

258 tests, 94% coverage; ruff and pyright clean.

LLM-Generated: true
 2026-10-16 17:33:24 -04:00
ducoterra c4bb374ccc Add homelab dashboards: system, LLM, and UPS on one domain 
Three dashboards served by one FastAPI app with a SQLite history store
(raw / 1m / 1h) and hand-written SVG charts: no Prometheus, no Grafana,
no CDN, no build step. Nothing is fetched from the internet at runtime.

* system: a stdlib-only agent (agent/host_agent.py) reports /proc and
  /sys counters; the dashboard turns them into rates over a labelled
  window, so every number says how it was measured.
* llm: gufo/llama.cpp and vLLM /metrics are parsed by an in-house
  exposition parser. Measured delta/elapsed throughput beats a native
  per-request gauge, and each value carries its provenance.
* ups: a NUT wire-protocol client plus health checks that name what
  they looked at. Commands are gated by NUT_ALLOW_ACTIONS,
  DASH_READ_ONLY and a DANGER confirmation, re-checked server-side.

Missing values render as n/a rather than 0, and an unconfigured source
answers 404 {"configured": false} without dialling anything, so "not
deployed here" never looks like an outage.

253 tests at 94% coverage against captured real fixtures; ruff and
pyright clean.

LLM-Generated: true
 2026-10-16 17:20:45 -04:00