dashboards/AGENTS.md

AGENTS.md — homelab dashboards

Orientation for an agent starting work in this repo: what the app is, the rules that keep its numbers honest, where things live, the fleet it watches, and the commands. README.md is the operator-facing how-to (quickstart, agent install, signing in, configuration, deployment) — read it for those; this file does not repeat it. This file is orientation, not a changelog: phase history lives in .agents/phases/ and data/logs/. When a phase closes, update the sections below only if a rule, invariant, command, or fleet fact changed — never append the phase's narrative.

Working protocol

1. Read .agents/PLAN.md first; follow the phased execution protocol in .agents/phases/.
2. Never modify .agents/PLAN.md or files in .agents/phases/complete/; ask the user before updating anything in .agents/phases/todo/.
3. Strictly adhere to the LOCKED DECISIONS in .agents/PLAN.md and the section at the bottom of this file.

What this is

Five dashboards for one homelab, behind one domain, in one process, backed by one SQLite file — plus an overview page:

Route	Question it answers	Source
/	Is anything wrong, and where do I look first?	all five, in one request
/system	CPU, memory, disk, GPU, network, and the history	agent/host_agent.py, our own stdlib agent per host
/disks	Are any physical disks failing, and which — health, temp, wear	the same agents, smartctl -i -A -H per whole drive
/llm	Tokens/s, active sessions, queue depth, KV pressure, latency	gufo /metrics + /slots, vLLM /metrics
/ups	Battery health, runtime left, power events, test buttons	NUT upsd (our own wire client; port 3051, not 3493)
/gateway	Which domains on which port, and where each goes	the same agent's haproxy + nginx sections joined to /proc/net/tcp{,6} on the one GATEWAY_HOST host

History is a first-class feature: every number worth graphing is stored, rolled up (raw → 1m → 1h) and served from /api/*/history.

None of it is public. One admin signs in at /login (the only page an anonymous visitor reaches); /settings (⚙ Account) and /certs (⚷ Certs) live behind it. Every page and every /api route requires the session except the public entries in rule 12.

Non-negotiable rules

Breaking one is a bug even if the page still renders.

1. Read-only by construction, with exactly one exception. Nothing writes to an upstream except POST /api/ups/command, gated four times (env allow, not DASH_READ_ONLY, the upsd account actually having the command, a UI confirmation — typed DANGER for anything that can cut power) and re-checked on every request; power-cutting commands are refused outright while on battery. Hiding a button is not a safety control.
2. Never invent a number. A value the upstream does not publish renders as n/a, never as 0. Every derived value carries its source string and derived: true; the rate estimator is keyed by the resolved series name plus its label set and refuses a window too short or too stale. Aggregating two different series into one rate is the failure mode this rule exists to catch.
3. Unconfigured is not broken. A source with no env config answers 404 {"configured": false} without dialling, and the page says "not configured". A red panel for a server the operator never installed destroys the signal.
4. One dead source must not blink another — and nothing may stop polling in silence. Each source is its own asyncio task with its own timeout and health record. A failed poll keeps the last good payload and marks it stale instead of zeroing it; a host that never answered still gets its row (unreachable: true, dimmed, n/a), because dropping it would turn a dead agent into "never configured". The task is the source's only poller, so no exception may end it: a store that refuses an otherwise good poll is a reported failure (ok: false, the real reason, last_ok frozen, one event then recording on recovery), and anything that escapes the poll body is logged and the loop continues. A frozen source that looks healthy is the one failure mode worse than an outage.
5. No CDN, no build step, no internet at runtime. Every asset is served by this app; charts are hand-rolled SVG (src/dashboards/web/static/charts.js). The engines live on a network with no outbound access — a CDN reference is a request that never comes back.
6. Never leak a credential. Target URLs may carry a userinfo token (parse_targets moves it into Authorization, so Target.url ends up clean), but nothing that leaves the process may: URLs in UpstreamError messages, health records, event rows (/api/events serves them to the browser) and /api/config are redacted. The admin's credential, the private keys, and session tokens are the same case: auth.py records nothing and answers False/None to every rejection, certs.py has no logging import, and both modules' raise sites and auth.py's one logging call carry fixed text with nothing interpolated — pinned structurally by tests/test_auth.py and tests/test_certs.py. No token, password, or personal hostname goes into tracked files or fixtures; captured socket tables and resolver answers substitute globally routable addresses for documentation space (RFC 5737 / RFC 3849), idempotently.
7. Verify against the deployed upstream; never guess a metric name. Field maps in sources/llm.py and sources/nut.py were captured from the running engines and pinned by tests/fixtures/. Re-capture with scripts/capture-fixtures.sh (and scripts/capture-gateway-config.sh for the gateway's proxy tree, socket tables and resolved names) rather than trusting a README or a memory of an upstream.
8. Keep dependencies current on purpose. After touching dependencies, uv pip list --outdated must come back empty. Python is 3.14 everywhere (requires-python, ruff py314, pyright, Containerfile) — do not lower it to make something else work.
9. SQLite owns the history and must stay bounded. Writes go through store.py; new metrics need a rollup path and a retention story, or a year of charts becomes a disk emergency. Put the DB on a volume in deployment.
10. The dev server stays up while work happens (./scripts/dev-server.sh restart, http://127.0.0.1:8090), and UI changes are verified in a real browser with a screenshot (data/logs/*.png) before they are called done.
11. The dashboard never starts open, and the stored password beats the environment. The admin verifier and the session-signing key are durable meta rows (auth.verifier, auth.session_secret) and the only place either secret lives; DASH_ADMIN_PASSWORD is a seed that hashes into the row on the first boot of a database and is ignored thereafter. A server with neither a stored verifier nor the seed refuses to start (a RuntimeError out of the lifespan, non-zero exit). testpass is the local-testing password (dev config and the hermetic suite), never a production fallback.
12. The gate is default-deny and its public surface is four entries (/login, /api/auth/login, /api/health, /static/). A page GET gets 303 to /login?next=… (query string included, checked by auth.safe_next); everything else the one JSON refusal; every answer carries no-store; a write additionally needs a matching Origin/Sec-Fetch-Site and — when it came in on the cookie — this session's X-CSRF-Token. A process whose lifespan never produced ctx.auth answers 503 for everything, public paths included. tests/test_auth_api.py walks the served route table so a route that forgets the gate fails a test instead of shipping — the middleware guards by default. Nothing may be added to /api/health without re-deciding that: it is public, and a new field there is a field the whole internet reads.

Architecture

agent/host_agent.py (each host)      gufo / vLLM            upsd (NUT)
        │ GET /snapshot                    │ GET /metrics           │ LIST UPS / VAR / CMD
        ▼                                  ▼                        ▼
collectors.py ── fetch, one UpstreamError type, no writes except run_ups_command
        │  parse + Δcounter/Δt
        ▼
sources/host.py · disks.py · llm.py · nut.py · gateway.py          poller.py (task per source)
        │        Result(payload, samples, events)
        ▼
store.py ── SQLite (WAL): sample(source, host, metric, res, ts) + event ──► api.py ──► web/

One line each — the load-bearing behaviour is in the bullet, the rest is in the module:

• agent/host_agent.py — stdlib-only (3.10+), one per host; serves GET /snapshot, /info, /schema, /health. Reads /proc, /sys/block, amdgpu DRM for GPUs, statvfs; smartctl -i -A -H per whole /sys/block disk with a scaled pass budget (20 s floor, 5 s per disk) whose refresh runs on a background thread — never inline on a later poll. Refuses to bind off-loopback without a token or a certificate bundle (exit 2). TLS mode (≥ 1.1.0): the bundle is the auth (the bearer token is the compatibility path), the # REVOKED denylist in the bundle is re-read when the bundle changes, and a serial on it is refused on every path including /health. The proxy add-on (≥ 1.2.0): the haproxy, nginx (plus its one-level includes) and sockets sections; the resolver section (1.3.0) looks up only the names the two config files declare, through libc on the box that dials, cached and single-flight. All four sections are additive, available is three-way (true/false/null) with exactly one error line per non-true arm — the missing one saying not found, the pinned wording the dashboard reads to answer "is it installed?" — and --no-proxy / --no-resolve remove their keys in silence. Nothing in a section can fail /snapshot or /health. Counters stay counters: the dashboard computes rates.
• config.py — env only, no config file format, no CLI flags. .env is a seed; a real env var always wins. Safe defaults except the sources, which default to unset. The gateway source is enabled only when GATEWAY_HOST names a HOST_TARGETS entry.
• poller.py — job loop, health records, stale/last-good semantics, transition events. disks:<host> is a source of its own (300 s / 30 s) and gateway is one job (30 s / 10 s) that exists only when configured, because the page is about the single ingress; it is handed no RateEstimator (a count is not a rate). The success path records (payload → samples → events) and then moves last_ok, so a refused SQLite write ages the source and alerts instead of leaving it current with charts that silently stop drawing. Events are stored under the dashboard name (system/disks/gateway/ llm/ups), not the sample source. The job list is built once (app.build_hub()) because it is wired twice (production and the injected-store test path).
• collectors.py — the fetch half. AgentTls makes an https:// HOST_TARGETS entry a certificate-authenticated dial (per-dial client, this app's client cert verified against its own root; the revocation answer is a fresh read per poll, so a /certs revoke bites on the next poll with no restart, before any socket opens). Failure messages name the redacted URL. Agent errors become degraded except the add-on lines (ADDON_ERROR_PREFIXES: smart, haproxy, nginx, sockets, resolver) — a host without smartctl or a proxy config is unconfigured, not broken (rule 3); those lines stay verbatim for the page that owns the add-on. A new agent add-on section adds its prefix here in the same commit.
• sources/host.py — snapshot → payload + samples; reboot detection; GPU series per card. Carries the agent's tls/bundle_rev passthrough (present only when published — rule 2).
• sources/disks.py — physical drives only (never a partition, never a mount), the agent's smart block verbatim, smartctl_available three-way on purpose, per-row severity (temp_c, percentage_used, the drive's own word for health); sector counters are alarm chips, never graded. Nothing here is a rate.
• sources/gateway.py — the snapshot projected onto the connection topology. The join is two attributions and nothing else: a socket whose local port names a configured listener is inbound; a socket whose remote (address, port) equals a target written as an address in a config is outbound; everything else is ignored. A name target is credited only through an address the gateway's own resolver section published — the module never calls a resolver itself (spy-pinned). Pipes are null, never 0, when unmeasured. warnings() judges the two config files against themselves (seven kinds, each {id, kind, subject, message}) and never joins a warning to a row. No severity anywhere in the payload (a busy proxy is the job).
• sources/llm.py — gufo + vLLM field maps, measured Δ rates diffed per label set, prefill speed, the session list from gufo /slots. A refused estimator window is still an answer: the cell keeps its source and an honest Δ<window> beside a null.
• sources/nut.py — the NUT wire client, assess() (explicit checks, not a mystery score), SELF_TEST_OUTCOMES, the COMMANDS catalog with risk tiers and available_commands() filtering, kelvin/celsius and charger-status normalisation.
• sources/nut_sim.py — a labelled simulator; never the default, every payload says simulated; its modelled battery date ages relative to today.
• prom.py — in-house Prometheus text parser (only for engines we don't own); values_by_series() is the per-label-set view counter deltas need.
• rates.py — the Δ/window estimator: keyed by series name + label set, window bounded both sides (floor; 2 × span ceiling), a rate ≥ 8× the previous one held one poll and confirmed only by an independent anchor. app.build_hub builds one estimator per measuring cadence — host at RATE_WINDOW_SECONDS (10 s), LLM at LLM_INTERVAL_SECONDS (2 s) — because one estimator cannot be right about two cadences.
• severity.py — the only place thresholds live. classify() → ok/warn/fail/None; verdicts travel additively in payloads; pages render the class and never compare a number to a threshold; unknown in → unknown out, never a green pass. There is no gpu.* row and no gateway.* row: a busy GPU or a busy proxy is the job at every level.
• alerts.py — the pure "what is wrong right now" projection /api/summary serves: reachability, severity breaches, UPS checks. An unconfigured source never alerts; a dead source gets exactly one line; one alert per (source, metric), at most 12. It has no thresholds of its own.
• store.py — sample + event + meta, rollups, pruning. meta is the only durable key/value state, reached through get_meta/set_meta/delete_meta only. No per-request path may pay a whole-table scan: series_names() and stats() answer from short-TTL caches. maintain() is windowed (only what became due since the last pass, snapped down to a bucket boundary) and its prune is one DELETE per resolution; it ends with PRAGMA optimize (auto-analyze is off, so no poller's commit pays for it).
• auth.py — every auth decision as a pure function, stdlib only: PBKDF2 verifier (self-describing, so the cost can rise without a migration), HMAC-signed stateless session cookie, CSRF bound to the session, per-address login throttle, safe_next(), and the verifier bootstrap (env seeds once; the stored row wins; no row and no seed raises). The primitives never raise and record nothing — the reason an input was rejected is the credential (rule 6).
• certs.py — the fleet's certificate authority over the same three-method MetaStore protocol: root CA, dashboard client cert, and per-host agent certs, all in meta; the bundle is the install artifact (agent cert + key + root certificate, never the root key, + # BUNDLE-REV + the denylist). A stored row always wins — the root is generated on the first boot of a database and never on a later one. ensure_certs is the startup read (it mints; nothing else may call it); read_state is the request-time read (same validation, never a write). cryptography is the app's only new dependency, used here alone; the agent stays stdlib.
• api.py — GETs everywhere plus the one POST that dials an upstream. Every source's configured gate runs before anything that could reach the network (spy-pinned: no client, no fetch). /api/config is a redaction boundary. All five history routes go through one door (history_many). /api/ups/command stays the app's only upstream write (pinned by enumerating every served route's verbs).
• app.py — assembly and the session gate middleware (invariant 12), answering in this order: no AuthState → 503 for everything; no session and no such route → the router's 404; public → through (the origin check still applies to a public write); no session → 303 for a page / JSON for the rest; a write → origin, then CSRF. build_hub is called in both the owned-store and injected-store branches; the lifespan builds ctx.auth before the first poll, and a RuntimeError there leaves the process non-zero.
• web/ — the six dashboards plus /login, /settings, /certs; one stylesheet, charts.js, no framework. Colour comes only from the payload severity verdict (/gateway colours nothing by verdict — its only pair is existence); n/a for the unknown, SIMULATED badges, per-source status dots, Δ<window> labels on measured rates. The served-byte pins in tests/test_api.py are the UI contract — when a UI rule is in doubt, read that test.

Metric naming: dotted (cpu.usage_pct, mem.used_pct, disk.used_pct@/srv, gpu.busy_pct@0, ups.charge_pct), @stream for per-device series. sample.source is the dashboard family — host, llm, ups, disks (the SMART series) and gateway (the connection gauges) — so GET /api/metrics?source=… answers "what does this page have history for". Treat the scheme as a public interface of the history API.

The fleet (real environment)

Host	ssh	What it gives
driveripper	driveripper-ops	Threadripper 24c/48t + Radeon AI PRO R9700, vLLM :8000, upsd, the dashboards. 16 whole disks; the agent unit runs User=root with CapabilityBoundingSet=CAP_SYS_RAWIO CAP_SYS_ADMIN (kernel 7.2's unprivileged-passthrough gate; an empty bounding set is capability-less, CapEff 0)
deskwork	deskwork-ops	Strix Halo (Radeon 8060S iGPU), gufo :8731
gateway	gateway-ops	the router, the fleet's ingress: haproxy (/etc/haproxy/haproxy.cfg, six frontends → six backends) + nginx (/etc/nginx/nginx.conf plus conf.d/ vhosts and stream.d/'s map on :443); runs agent 1.3.0

• Transport is direct dial; the fleet is mTLS. HOST_TARGETS are https:// with no userinfo — the certificate bundle is the credential (the token in a URL is the compatibility path for a host nobody has bundled). All three hosts run the installed dashboards-agent systemd service, installed by deploy/install-agent.sh under the sudo grant. Re-stage /home/ops/agent-install/ first — the grant names a path, not a checkout — and a re-run whose rendered unit differs from the installed one prints the diff and refuses without --force. deskwork/driveripper still run 1.1.0: the 1.2.0/1.3.0 sections are purely additive (a 1.1.0 snapshot has none of the keys, which sources/gateway.py answers as supported: false), so they move at their next re-stage. A TLS agent serves https only — rolling a host back means removing the bundle lines too, not just the URL.
• NUT 2.8.5 on driveripper: configs in /etc/ups/ (not /etc/nut/), upsd on port 3051 (not 3493 — nothing listens on 3493). NUT 2.8 has no ACL directive; per-source restriction is a firewall rule, never an upsd.conf line. Two UPSes: apc (APC Back-UPS RS 1500MS) and cyberpower (CyberPower PR1500LCDRT2U — that is ups.description; the driver's own identity.model answers GX1500U). Real driver quirks the parsers already handle: CyberPower's battery.mfr.date is CPS (not a date → unknown, never a fake age); neither driver reports a battery temperature or battery.charger.status; ups.test.result is No test initiated (APC) vs Done and passed (CyberPower) — both words re-read from the wire and pinned in tests/test_nut.py.
• SMART: smartctl 7.5 on deskwork and driveripper; the agent shells it as itself (never sudo). This binary annotates the attribute table's RAW_VALUE column (35 (Min/Max 34/44), 43894h+43m+32.519s), so the parser reads that column, not the end of the line. gateway has no smartmontools (a VM) → smartctl_available: false → no /disks card at all (the visibility gate, api._disk_visible), while the disks:gateway job still polls and its events still reach the page.
• Ground truth: ./scripts/dev-server.sh status (per-source ok/DOWN, unauthenticated) and uv run pytest -m live --live (opt-in, read-only). The per-host install record is in data/logs/ (gitignored) — re-run the probes rather than trusting a note.

Commands

uv sync --python 3.14                      # deps + editable install
set -a; . ./.env.dev; set +a              # dev dials the fleet the prod way, no tunnels
./scripts/dev-server.sh restart           # keep http://127.0.0.1:8090 up; sign in with testpass
./scripts/dev-server.sh status            # per-source ok/DOWN (unauthenticated)
./scripts/dev-server.sh bundles           # the bundles the restart exported to data/certs/
./scripts/dev-server.sh logs
uv run pytest                             # hermetic: no network (skips 12 live + 4 store_perf)
uv run pytest --cov                       # coverage gate (floor 94%)
uv run pytest -m live --live              # opt-in: real fleet / engines / upsd — read-only
uv run pytest -m store_perf --store-perf  # opt-in: store hot-path timings vs store_perf.BUDGETS
uv run ruff check src tests agent && uv run pyright
bash .agents/validate.sh                  # the gate: the four checks above + repo hygiene
./scripts/capture-fixtures.sh             # re-record ground truth (UPS + gufo /slots + smartctl)
./scripts/capture-gateway-config.sh       # the gateway's haproxy/nginx tree, /proc/net/tcp{,6},
                                         # and the recorded resolver answers (via $GATEWAY_SSH)

Deployment: Containerfile (two-stage, uv sync --frozen, runs as uid 10001 on 0.0.0.0:8090, SQLite under /var/lib/dashboards — put a volume there) and deploy/install-agent.sh per monitored host. The README's Deployment section is the operator's how-to for both.

Testing policy

Real captures for shape, fakes for behaviour, live for drift.

• Anything that must match an upstream (/proc layouts, vLLM series names, NUT framing, smartctl output, the gateway's configs) is asserted against a verbatim capture in tests/fixtures/. If the schema drifts, a fixture test fails before a dashboard does.
• Failure modes (a dead upsd, an engine 500ing, a host rebooting mid-poll, a counter reset) are fakes in tests/conftest.py.
• live (tests/test_live_fleet.py) re-runs the production code paths against the real fleet. Skipped unless --live; each test skips when its source is unset. Live tests must not write — a battery test on hardware something is plugged into is not a test. An https:// agent is dialled through fetch_snapshot with the stored authority, never a hand-rolled client.get.
• A verdict the operator sees is pinned on the surface that shows it, in both directions — an absence-only assertion is satisfied by deleting the check.
• API tests use create_app() with an in-memory DB and the testpass seed at pbkdf2_iterations=1000 (the real 600 000 cost is pinned separately in tests/test_auth.py and tests/test_config.py, so the cheap cost cannot become the default there).
• tests/test_auth_api.py is the perimeter: it walks the served route table and demands a session for every path outside the four public entries, and proves the UPS CSRF refusal on the wire — 403 after instcmd would be a power cut with an error message attached.
• A cost claim is pinned as a shape and a control, never as a wall clock (timings live behind store_perf and in the probe scripts, with budgets in a table).
• Coverage: 80 % is the repo floor, bash .agents/validate.sh enforces 94 %; new/modified Python must still clear > 90 % on its own.

UI conventions (short)

• Hand-rolled SVG (charts.js), one stylesheet, dark/light toggle, no framework, no CDN, no build step.
• Colour comes only from the payload severity verdict, through Dash.severityFrom(); a missing number or verdict prints n/a uncoloured — never 0, never green. A value the API should rate gets a row in severity.py, never a bound in JS.
• Δ<window> labels on measured rates print the window the estimator actually used (the payload's window_seconds), never the poll interval; SIMULATED badges on modelled data; per-source status dots in the header.
• Flavour text is not helpful information: copy survives only where the operator would act differently without it — a why-this-panel-is-empty sentence, a provenance tag, a measured Δ label, a unit. No pipeline narration.
• A wide table scrolls sideways inside its own card at ≤ 640 px; card headers wrap their facts; disclosures scroll what they reveal; nothing is painted past a card edge and eaten. The masthead is three rows, the two control rows flush right, and the bar is not sticky.
• An event feed is a window (50 per page, Dash.eventsPager), not a printout; /ups splits power events (behind a collapsed disclosure) from connection errors (a card of their own, hidden at rest).
• /gateway is two panels (one per config file), trunks per bound port with their configured routes branching off; its only colour pair is existence (--gate-on/--gate-off) — no verdict colour anywhere on the page; its filter box is an eye, not a query (it hides rows whose own payload text does not contain what was typed; no DNS, no re-ranking).
• Every chart names itself for assistive tech (role="img" + aria-label); every chart on every page has one, pinned in tests/test_api.py.

Known gaps (the current state, one line each)

1. The UPS gates are the control: the upsd account has instcmds = all, and NUT has no call for a user's privileges — enabled: true cannot mean "may send", so the risk tiers and typed DANGER are load-bearing, not decoration.
2. One admin, one factor: no TOTP/WebAuthn, no per-user accounts; a logout ends only this browser's cookie, and the only revocation is a password change (which rotates the session secret). The login throttle is per-process and keys on the socket peer address (behind nginx every login shares one bucket; a restart clears it).
3. GPU coverage is AMD only — an NVIDIA host would report no GPU at all.
4. UPS history depth: events and numeric series are stored; long-window capacity fade has the rollups but no dedicated view.
5. The store's cold-cache tail (a ~1.3 s whole-table re-derivation once per cache window) is a known cost, not a bug; the database is locked cascade is fixed by the windowed maintain().
6. driveripper's drives read live only because the unit runs as root with the bounded gate capabilities (kernel 7.2's unprivileged-passthrough gate); the installer template reproduces it with --as-root --caps CAP_SYS_RAWIO,CAP_SYS_ADMIN --force.
7. The fleet trusts this deployment's CA — a second deployment with its own certs.* rows cannot dial these agents until it carries the same root or re-issues and re-installs every bundle.
8. The /gateway route switches are built but not wired — a route write path is phase 33's decision and amends invariant 1; until then the knobs stay aria-disabled.

Locked decisions (and why)

• SQLite, not Prometheus/Grafana. Handful of boxes, 1–10 s resolution, one file the container owns; rollups keep a year of charts at a few hundred MB.
• Our own stdlib host agent, not node_exporter. One JSON snapshot with exactly the fields the page needs versus ~1500 series and a re-labeling layer; no dependency to install, ProtectSystem=strict friendly.
• Our own NUT client, not pynut/upsc. The protocol is ~10-line commands; the third-party libs are subprocess-based or stalled; we need instant commands with explicit gating.
• In-house Prometheus parser. Only two upstreams use the format; keeps prometheus_client out of the dependency set.
• cryptography for X.509, dashboard-side only. Minting a CA is not something to hand-roll; the agent stays stdlib (it loads the PEMs with ssl), which is why the bundle is plain text and the agent runs on a host's Python 3.10.
• Port 8090, inherited from the llm_monitor prototype: 8000 is vLLM and 8731 is gufo in this fleet.
• .env is a seed, never an override — container/systemd env must win.

Definition of done (per change)

uv run pytest green · coverage not down · ruff + pyright clean · uv pip list --outdated empty if deps moved · dev server restarted and /api/health sane · UI change screenshotted in a browser against live data (in data/logs/) · fixtures re-captured if an upstream contract changed · this file and README.md updated when a rule, command, or fleet fact changed.