ducoterra
ad993c7b03
**Gate defect found & fixed (the "does not lint" bug).** `.agents/validate.sh` — the phase gate — ran `python3 -m pytest` on `/usr/bin/python3` + `~/.local` site-packages (not the uv venv), ran `ruff check .` *only if* a ruff happened to be on PATH, and **never ran pyright or the coverage floor**. Rewrote it to the project's real gate: cd to repo root → `uv run pytest --cov --cov-fail-under=94` → `uv run ruff check src tests agent` → `uv run pyright` → `node --check web/static/*.js` → hygiene gates; a missing toolchain now fails loudly. Provoked it to confirm it bites (floor + a seeded F401 both fail).
**Verification (all green)**
- `bash .agents/validate.sh` → **exit 0**: 356 passed, 6 skipped · coverage **94.75%** (floor 94) · ruff `All checks passed!` · pyright `0 errors` · node syntax clean
- `uv run pytest --cov` → `severity.py` **100%**; `uv run pytest -m live --live` → **6 passed**; `uv pip list --outdated` → empty
- Criteria: `classify('cpu.usage_pct',…)` → `fail`/`ok`/`None` ✓ · `/api/ups/snapshot` carries `severity` beside `charge_pct`/`load_pct`/`battery_runtime` (apc now live-`warn` at 498 s — threshold working) ✓ · no `.sev-*`/synonym vocabulary (only a comment saying so) ✓ · pytest/coverage/lint ✓ · dev server restarted, `/api/health` ok 6/6 sources, all pages+assets 200 ✓
- Chromium against live fleet: `severityFrom` cell→`fail`, bare→`warn`, null→`''`, unrated→`''`; `hero` fail → `rgb(207,34,46)`, null → `n/a` with **no** verdict class, warn+bar 42%; all four pages render unchanged → `data/logs/phase01-final-{index,system,llm,ups}.png`
**Docs (full review, incl. pre-existing drift).** README: `/ups` row claimed "upsd on 3493" (fleet is 3051, stated elsewhere in the same file); config table omitted `DASH_HOST/PORT`, llm/ups `*_INTERVAL_SECONDS`, `*_TIMEOUT_SECONDS`, `GUFO/VLLM_API_KEY`, `POLL_MS_*`, `DEBUGPY` (+ dedup); severity bullet omitted `disks`/`gpus[]`/rated LLM cells and the ratio rescale; agent section listed only `/snapshot` though `/info`+`/health` exist. AGENTS.md: same 3493 claim; `install-agent.sh --token-file` (its flags are `--bind/--port/--token/--rotate-token/--status/--uninstall` — it hands *the agent* `--token-file`); gap 3 "Two commits" (now 6, still no `origin`); gap 1 given the sudoers install command + verified live gating reason; coverage line now 80% repo floor vs 94% gate floor; cyberpower `identity.model: GX1500U` vs `ups.description` quirk. `.env.dev.example`: `DEV_FORWARDS` remote port 3493→3051 and engine URLs that pointed at `127.0.0.1` with nothing forwarding them.
**Deviations.** No `ruff format` (30 files would reformat; the project gate is `ruff check` + pyright — reformatting completed phases is out of scope). **No live-infra changes**: enabling the battery-test buttons needs `deploy/nut-sudoers` installed on driveripper + a nut-server restart and a scoped `[dashboards]` `instcmds` user — prod write with a documented shutdown risk (AGENTS gap 1), so repo-side only. 36 pre-existing `ResourceWarning: unclosed database` from test fixtures are noise, not failures; left rather than churn ~15 fixtures. `severity.attach()` is unit-tested but has no production call site until phases 02–05.
**Next pending phase:** `02_overview_fleet_and_alerts` (`01_alerts_builder.md`).
289 lines
19 KiB
Markdown
289 lines
19 KiB
Markdown
# AGENTS.md — homelab dashboards
|
||
|
||
The plan, the rules, and the environment facts an agent needs to work on this repo without
|
||
re-deriving them. `README.md` is the operator-facing how-to; this file is the why, the
|
||
invariants, and the state of the fleet.
|
||
|
||
## Working protocol
|
||
|
||
This project runs the `.agents/phases/` phased-execution pipeline.
|
||
|
||
1. Always read `.agents/PLAN.md` first to understand the project context and goals.
|
||
2. Follow the phased execution protocol in `.agents/phases/`.
|
||
3. Never modify `.agents/PLAN.md` or any files in `.agents/phases/complete/`.
|
||
4. If you need to update any file in `.agents/phases/todo/`, you must ask the user for permission first.
|
||
5. Strictly adhere to the **LOCKED DECISIONS** listed in `.agents/PLAN.md`.
|
||
|
||
Everything below this section is the project's own content: what the app is, the rules that
|
||
keep its numbers honest, and the state of the fleet it watches.
|
||
|
||
## What this is
|
||
|
||
Three dashboards for one homelab, behind one domain, in one process, backed by one SQLite
|
||
file — plus an overview page that says which of the three you need:
|
||
|
||
| Route | Question it answers | Source |
|
||
| --------- | ------------------------------------------------------------------------------ | ------------------------------------------------- |
|
||
| `/` | Is anything wrong, and where do I look first? | all three, in one request |
|
||
| `/system` | What are my machines doing — CPU, memory, disk, GPU, network, and the history | `agent/host_agent.py`, our own stdlib agent/host |
|
||
| `/llm` | Tokens/sec, active sessions, queue depth, KV pressure, latency percentiles | gufo (llama.cpp family) + vLLM `/metrics` |
|
||
| `/ups` | Battery health, runtime left, power events, battery-test buttons | NUT `upsd` (our own wire-protocol client; 3051 here, not 3493) |
|
||
|
||
History is a first-class feature, not a sparkline afterthought: every number worth graphing
|
||
is stored, rolled up (raw → 1m → 1h), and served from `/api/*/history`.
|
||
|
||
## Non-negotiable rules
|
||
|
||
These are the things that make the numbers trustworthy. Breaking one is a bug even if the
|
||
page still renders.
|
||
|
||
1. **Read-only by construction, with exactly one exception.** Nothing in this app writes to
|
||
an upstream except `POST /api/ups/command`. That endpoint is gated four times
|
||
(`NUT_ALLOW_ACTIONS`, not `DASH_READ_ONLY`, the upsd.users account actually having the
|
||
command, and a UI confirmation — `DANGER` for anything that can cut power) and the server
|
||
re-checks every gate on every request. Hiding a button is not a safety control.
|
||
Commands that would cut output are refused outright while the UPS is already on battery.
|
||
2. **Never invent a number.** A value the upstream does not publish renders as `n/a`, never
|
||
as `0`. Every derived value carries the `source` string it came from and `derived: true`
|
||
so the UI can print *how* it was obtained (`computed:Δtokens/Δt`, `Δ10.0s`). Aggregating
|
||
two different series into one rate is the failure mode this rule exists to catch — the
|
||
rate estimator is keyed by the *resolved* upstream series name for exactly that reason.
|
||
3. **Unconfigured is not broken.** A source with no env config answers
|
||
`404 {"configured": false}` **without dialling**, and the page says "not configured".
|
||
A red panel for a server the operator never installed destroys the signal from the
|
||
panels that matter.
|
||
4. **One dead source must not blink another.** Each source is its own asyncio task with its
|
||
own timeout and health record (`poller.py`). A failed poll keeps the last good payload and
|
||
marks it stale in the UI instead of zeroing it. Transitions write events; repeated failure
|
||
writes one line, not 500.
|
||
5. **No CDN, no build step, no internet at runtime.** Every asset is served by this app;
|
||
charts are hand-rolled SVG (`web/static/charts.js`). The engines live on a network with
|
||
no outbound access — a CDN reference is a request that never comes back.
|
||
6. **Never leak a credential.** Target URLs may carry a userinfo token; `/api/config`
|
||
exposes them only through `Target.display_url`, which strips userinfo. No token,
|
||
password, or personal hostname goes into tracked files, fixtures, or the browser.
|
||
7. **Verify against the deployed upstream; never guess a metric name.** Field maps in
|
||
`sources/llm.py` and `sources/nut.py` were captured from the running engines and NUT
|
||
2.8.5, and are pinned by fixtures in `tests/fixtures/`. Re-capture with
|
||
`scripts/capture-fixtures.sh` (and `scripts/capture-node-exporter.sh` when comparing
|
||
exporters) rather than trusting a README or a memory of an upstream.
|
||
8. **Keep dependencies current on purpose.** LLM-written version floors are usually stale.
|
||
Check after touching dependencies: `uv pip list --outdated` must come back empty.
|
||
Python is 3.14 (`requires-python >=3.14`, ruff `py314`, pyright `3.14`, Containerfile
|
||
`python3.14`) — do not lower these to make something else work.
|
||
9. **SQLite owns the history and must stay bounded.** Writes go through `store.py`; new
|
||
metrics need a rollup path and a retention story, or a year of charts becomes a disk
|
||
emergency. Put the DB on a volume in deployment.
|
||
10. **The dev server stays up while work happens** (`./scripts/dev-server.sh restart`,
|
||
http://127.0.0.1:8090), and UI changes are verified in a real browser with a screenshot
|
||
before they are called done. `data/logs/*.png` is where those land.
|
||
|
||
## Architecture
|
||
|
||
```
|
||
agent/host_agent.py (each host) gufo / vLLM upsd (NUT)
|
||
│ GET /snapshot │ GET /metrics │ LIST UPS / LIST VAR / LIST CMD
|
||
▼ ▼ ▼
|
||
collectors.py ── fetch, one UpstreamError type, no writes except run_ups_command ──┐
|
||
│ parse + Δcounter/Δt │
|
||
▼ ▼
|
||
sources/host.py · sources/llm.py · sources/nut.py poller.py (task per source)
|
||
│ Result(payload, samples, events) │
|
||
▼ ▼
|
||
store.py ── SQLite (WAL): sample(source, host, metric, res, ts) + event ──► api.py ──► web/
|
||
```
|
||
|
||
Responsibilities, one line each:
|
||
|
||
- `agent/host_agent.py` — stdlib-only (3.10+), reads `/proc`, `/sys/block`, `/sys/class/drm`,
|
||
`statvfs`; serves `GET /snapshot` + `GET /info` + `GET /health`. Refuses to bind
|
||
off-loopback without a token (exit 2). Counters stay counters: the dashboard computes
|
||
rates so the agent never remembers who polled it last.
|
||
- `config.py` — env only, no config file format, no CLI flags. `.env` is a *seed*; a real
|
||
env var always wins. Defaults are safe except the sources, which default to unset.
|
||
- `poller.py` — job loop, health records, stale/last-good semantics, transition events.
|
||
- `collectors.py` — the fetch half. `NutMonitor` owns per-UPS transition bookkeeping.
|
||
- `sources/host.py` — snapshot → payload + samples; reboot detection; GPU series per card
|
||
(`gpu.busy_pct@<index>`).
|
||
- `sources/llm.py` — gufo + vLLM field maps (`GUFO_SERIES`, `VLLM_SERIES`,
|
||
`VLLM_HISTOGRAMS`), measured Δ rates, prefill speed as
|
||
`mean(prompt tokens)/mean(prefill time)`, session list from gufo `/slots`.
|
||
- `sources/nut.py` — upsd client, `UpsReading`, `VAR_MAP`, `assess()` (explicit checks, not
|
||
a mystery score), `COMMANDS` catalog with risk tiers, `available_commands()` filtering,
|
||
kelvin-vs-celsius normalisation.
|
||
- `sources/nut_sim.py` — a **labelled** simulator so the UPS page is developable without a
|
||
battery. It must never be the default, and every payload it produces says `simulated`.
|
||
- `prom.py` — Prometheus text-exposition parser (in-house; only for engines we don't own).
|
||
- `rates.py` — Δ/window estimator, keyed by resolved series name.
|
||
- `severity.py` — the only place thresholds live: `THRESHOLDS`/`INVERTED` per metric,
|
||
`classify()` → `ok`/`warn`/`fail`/`None`, plus `attach()`/`with_severity()` to put the verdict
|
||
beside a value. No measurement, no verdict: `None` in → `None` out, never a green pass for
|
||
an unknown. The verdicts travel **additively** in payloads — `cpu.severity`,
|
||
`memory.severity`, `disks.severity`, `gpus[].severity`, each UPS card's `severity`
|
||
(charge/load/runtime), `severity` inside rated LLM cells (KV pressure, classified on a
|
||
percent scale because the engines publish a 0-1 ratio), and `severity` per host/fleet/UPS
|
||
row in `/api/summary`. Pages render the class; they never compare a number to a threshold.
|
||
- `store.py` — `sample(source,host,metric,res,ts)` + `event` + `meta`, rollups, pruning.
|
||
- `api.py` — GETs everywhere plus the one POST. `/api/config` is redaction boundary.
|
||
- `app.py` — assembly; `create_app()` takes injectable settings/store so tests build an app
|
||
with an in-memory DB and no targets.
|
||
- `web/` — four static pages, one stylesheet, `charts.js`. No framework. `app.js` owns the
|
||
colour vocabulary: `Dash.severityFrom(cellOrSeverity)` turns a payload verdict into an
|
||
`ok|warn|fail` class (`''` for anything unrated, so unknown never looks green) and
|
||
`Dash.hero(el, {label, value, unit, digits, severity, sub, bar})` renders/patches a
|
||
`.hero` big number (plus an optional `.bar`) from that class — it never compares a number
|
||
to a threshold. `Dash.severityFor(value, warnAt, failAt)` is the documented fallback for
|
||
values the API does not rate; converting its remaining call sites is phases 02–05's job.
|
||
|
||
Metric naming: dotted (`cpu.usage_pct`, `mem.used_pct`, `disk.used_pct@/srv`,
|
||
`gpu.busy_pct@0`, `llm.decode_tps`, `ups.charge_pct`), `@stream` for per-device series.
|
||
That scheme is the history API's key — treat it as a public interface.
|
||
|
||
## The fleet (real environment, verified 2026-10)
|
||
|
||
| Host | ssh alias | What it gives |
|
||
| ----------------- | ------------------ | -------------------------------------------------------------------- |
|
||
| `driveripper` | `driveripper-ops` | Threadripper 24c/48t + Radeon AI PRO R9700, vLLM `:8000`, **upsd**, the dashboards |
|
||
| `deskwork` | `deskwork-ops` | Strix Halo (Radeon 8060S iGPU), gufo `:8731` |
|
||
| `gateway` | `gateway-ops` | router, no GPU |
|
||
|
||
- Agents run from `~/lab-agent/host_agent.py` on each host, forwarded to `127.0.0.1:9121/9122/9123`
|
||
by `scripts/dev-tunnels.sh` (`DEV_TUNNELS`). Nothing listens on the hosts in dev.
|
||
- Engines are reached directly: gufo `deskwork.reeselink.com:8731`, vLLM `driveripper.reeselink.com:8000`.
|
||
- NUT 2.8.5 on driveripper: **configs live in `/etc/ups/`, not `/etc/nut/`**, and **upsd is
|
||
on port 3051, not 3493**. Nothing listens on 3493 at all, so a default client reports
|
||
*Connection refused* and it looks like a dead service — `ss -ltnH` plus `printf 'LIST UPS\n'
|
||
| nc 127.0.0.1 3051` (answers `BEGIN LIST UPS`) is how that was pinned down. Two UPSes are
|
||
exported over USB and read live through `DEV_FORWARDS="3494:driveripper-ops:3051"`:
|
||
`apc` = APC Back-UPS RS 1500MS, `cyberpower` = CyberPower PR1500LCDRT2U (that is
|
||
`ups.description`, the operator's label — the driver's own `identity.model` answers
|
||
`GX1500U`, so the description is what identifies the box).
|
||
Real-hardware quirks the parsers already handle: `battery.mfr.date` is the useful date
|
||
(`battery.date` is often an unset placeholder like `2001-09-25`), CyberPower reports
|
||
`battery.mfr.date: CPS` (not a date → `unknown`, never a fake age), neither driver reports
|
||
a battery temperature, and `ups.test.result: No test initiated` is the factory state on both.
|
||
- Host access is via ssh keys as `ops`. Passwordless sudo exists but is command-scoped; the
|
||
set we rely on is documented in `deploy/nut-sudoers` (additions are listed there for the
|
||
work that is still blocked).
|
||
|
||
## Commands
|
||
|
||
```bash
|
||
uv sync --python 3.14 # deps + editable install
|
||
cp .env.example .env # production-shaped config
|
||
cp .env.dev.example .env.dev # dev-only: ssh tunnels + live targets
|
||
|
||
./scripts/dev-tunnels.sh start # agents (9121-9123) + DEV_FORWARDS (upsd 3494)
|
||
./scripts/dev-server.sh restart # keep http://127.0.0.1:8090 up
|
||
./scripts/dev-server.sh status # per-source ok/DOWN
|
||
./scripts/dev-server.sh logs
|
||
|
||
uv run pytest # hermetic: 356 tests, no network
|
||
uv run pytest --cov # coverage gate (94.75%; floor 94%)
|
||
uv run pytest -m live --live # opt-in: real fleet / engines / upsd
|
||
uv run ruff check src tests agent && uv run pyright
|
||
./scripts/capture-fixtures.sh # re-record ground truth
|
||
|
||
bash .agents/validate.sh # the phase gate: the four checks above (pytest
|
||
# --cov --cov-fail-under=94, ruff, pyright, plus a
|
||
# node --check of web/static) + repo hygiene.
|
||
# Always `uv run` — a bare `python3`/`ruff` on PATH
|
||
# tests an interpreter nobody deploys.
|
||
```
|
||
|
||
Deployment: `Containerfile` (two-stage, `uv sync --frozen`, runs as uid 10001 on `0.0.0.0:8090`,
|
||
SQLite under `/var/lib/dashboards` — put a volume there) and `deploy/install-agent.sh`
|
||
per monitored host (systemd unit, own user, `--bind`/`--port`/`--token`/`--rotate-token`/
|
||
`--status`/`--uninstall`; it stores the token in a root-owned 0600 file and hands the agent
|
||
`--token-file`, never argv, and runs under `ProtectSystem=strict`).
|
||
The quadlet unit + env template + nginx/SNI live in the deployment repo, like every other
|
||
service here.
|
||
|
||
## Testing policy
|
||
|
||
**Real captures for shape, fakes for behaviour, `live` for drift.**
|
||
|
||
- Anything that must match an upstream — `/proc` layouts, vLLM series names, NUT framing —
|
||
is asserted against a verbatim capture in `tests/fixtures/`. If the schema drifts, a
|
||
fixture test fails before a dashboard does.
|
||
- Anything that needs a failure mode — upsd refusing a command, an engine 500ing, a host
|
||
rebooting mid-poll, a counter reset — is a fake in `tests/conftest.py`.
|
||
- `live` (`tests/test_live_fleet.py`) re-runs the production code paths against the real
|
||
fleet, engines, and upsd. Skipped unless `--live`; each test skips when its source is
|
||
unset, so partial fleet availability is normal. **Live tests must not write** — a battery
|
||
test on hardware something is plugged into is not a test.
|
||
- The agent's real-`/proc` test (`tests/test_agent.py`) touches only the box running pytest,
|
||
so it stays in the default run and is *not* marked `live`.
|
||
- Coverage gate: 80% is the repo floor, the phase gate in `.agents/validate.sh` enforces
|
||
94% (`--cov-fail-under`), and ~95% is where we are; keep it there. New/modified Python
|
||
still has to clear >90% on its own. API tests use `create_app()` with an in-memory DB,
|
||
never a real target.
|
||
|
||
## UI conventions
|
||
|
||
Hand-rolled SVG (`charts.js`), one stylesheet, dark/light toggle, `Δ<window>` labels on
|
||
measured rates, `n/a` for the unknown, `SIMULATED` badges on modelled data, per-source status
|
||
dots in the header, window chips (1h…90d) on history pages. A panel that cannot get data says
|
||
why in the panel — the operator should never have to open devtools to learn that a source is
|
||
unconfigured. Escape anything upstream-provided through `DashCharts.escapeHtml`.
|
||
|
||
A big number is `Dash.hero()` on `.hero` / `.hero-bar` (`dash.css`): 2.4rem tabular numerals,
|
||
muted unit, mono `.sub` footnote (`Δ10.0s`, "at current load"), coloured by the payload's
|
||
`severity` through the existing `.ok` / `.warn` / `.fail` classes — there is no `.sev-*`
|
||
family and no second palette, and `tests/test_api.py` fails if either appears or if a CSS
|
||
rule reaches for a token that `:root` does not declare.
|
||
|
||
## Known gaps / next work
|
||
|
||
1. **UPS instant commands are read-only in practice.** Reads work against real upsd (see
|
||
above), but every button is gated off with "NUT_USER / NUT_PASSWORD must name a user with
|
||
`actions` in upsd.users" because no such user exists yet. Creating one safely needs a
|
||
*read* of `/etc/ups/upsd.users` first — `sudo tee` can write the file but overwriting it
|
||
blind would delete the users upsmon already authenticates as, which can take the box down
|
||
at the wrong moment. `deploy/nut-sudoers` is the installable ruleset that makes that
|
||
read (and only a scoped write + reload) possible — it carries the
|
||
`ops ALL=(root) NOPASSWD: NUT_READ, NUT_WRITE` grant and is installed with
|
||
`sudo install -m 0440 -o root -g root deploy/nut-sudoers /etc/sudoers.d/dashboards-nut`,
|
||
but it is **repo-side only so far**: the live UPS cards still answer "NUT_USER /
|
||
NUT_PASSWORD must name a user with `actions` in upsd.users" (verified through
|
||
`/api/ups/snapshot`). Once it is installed: read `upsd.users`, add `[dashboards]` with
|
||
`instcmds` limited to the battery tests, put the credentials in `.env.dev`, and the
|
||
buttons come alive. Never grant `actions = -`, and never write the file blind.
|
||
2. **GPU coverage is AMD only.** `host_agent.gpus()` reads amdgpu DRM sysfs
|
||
(`gpu_busy_percent`, `mem_info_vram_*`, hwmon temps, clocks). That covers the current
|
||
fleet (R9700 + Strix Halo iGPU; gateway has none). An NVIDIA host would report no GPU at
|
||
all — needs a `nvidia-smi`/NVML or DCGM path, and a fixture from a real card before the
|
||
field map is written.
|
||
3. **No remote yet.** No `origin` is configured (`git remote -v` is empty) — the user is
|
||
providing it, so every phase so far is local-only history.
|
||
4. **UPS history depth.** Events and the numeric series are stored; long-window battery
|
||
health (capacity fade over months) has the rollups but no dedicated view.
|
||
|
||
## Locked decisions (and why)
|
||
|
||
- **SQLite, not Prometheus/Grafana.** Handful of boxes, 1–10s resolution, one file the
|
||
container owns. Rollups keep a year of charts at a few hundred MB.
|
||
- **Our own stdlib host agent, not node_exporter.** One JSON snapshot with exactly the
|
||
fields the page needs (GPU model strings, per-mount usage, temperature labels) versus
|
||
~1500 series and a re-labeling layer; no dependency to install, `ProtectSystem=strict`
|
||
friendly, and read-only by construction. `scripts/capture-node-exporter.sh` remains to
|
||
compare ground truth if that is ever revisited.
|
||
- **Our own NUT client, not `pynut`/`upsc`.** The protocol is ~10 line commands, the
|
||
third-party libs are subprocess-based or stalled, and we need instant commands with
|
||
explicit gating.
|
||
- **In-house Prometheus parser.** Only two upstreams use the format; keeps `prometheus_client`
|
||
out of the dependency set.
|
||
- **Port 8090**, inherited from the `llm_monitor` prototype this started from: 8000 is vLLM
|
||
and 8731 is gufo in this fleet.
|
||
- **`.env` is a seed, never an override** — container/systemd env must win.
|
||
|
||
## Definition of done (per change)
|
||
|
||
The first three are exactly what `bash .agents/validate.sh` (the phase gate) checks, so no
|
||
phase may be called done on a subset of them.
|
||
|
||
`uv run pytest` green · coverage not down · `ruff` + `pyright` clean · `uv pip list --outdated`
|
||
empty if deps moved · dev server restarted and `/api/health` sane · UI change screenshotted
|
||
in a browser against **live** data · fixtures re-captured if an upstream contract changed ·
|
||
this file and `README.md` updated when a rule, command, gap, or fleet fact changed.
|