services/dashboards
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

phase: 10_dev_direct_dial

Browse Source 
**Phase 10 final pass — repo green; live criteria blocked off-repo.**

- Verified: `gateway` runs `dashboards-agent` (active/enabled, `10.1.1.27:9120` → `/health` 200, `/snapshot` 401 no-token / 200 bearer on-host, token matches `/etc/dashboards-agent.token`); `driveripper`/`deskwork` inactive, `sudo -n` → "a password is required", `:9120` empty.
- New measurement (transient runtime rule on gateway `firewall-cmd --add-port=9120/tcp`, **reverted** — runtime+permanent port lists verified back to the original set, no rich rules): laptop still gets no SYN through, while `:22`/`:80` on the same address answer in ms ⇒ the block is the lab gateway `10.2.0.1` port filter, not this app/host firewall/agent. No other live change made (no sudoers, no upsd, no UPS command).
- Tests/lint (exact): `bash .agents/validate.sh` → **EXIT 0** — `uv run pytest --cov --cov-fail-under=94` **793 passed, 11 skipped, coverage 98.33 %**; `ruff check src tests agent` clean; `pyright` 0 errors; `node --check` clean; hygiene clean; `uv pip list --outdated` empty (bumped fastapi 0.143.3→0.143.5, `uv.lock` only).
- `uv run pytest -m live --live` (`.env.dev` sourced): **4 passed, 1 skipped, 6 failed** — all six are `:9120`/`:3051` connect timeouts; zero parse/contract failures.
- Dev server restarted on current code: engines `ok`; 3×`host` + 3×`disks` + `nut` `DOWN` with real reasons (designed degradation, stub rows intact, no credential leakage).
- Completion criteria: screenshots ✓ (`data/logs/phase10-{index,system,disks,llm,ups}-live.png`); tunnel-vocabulary grep ✓ outside `.agents/` history (PLAN.md locked line, phase overviews, past reports; fixtures/`uv.lock` hits are digit substrings — `tests/test_deploy_scripts.py` guards operational files); no page-payload change ✓ (contract tests). **Not met, operator-side:** agents on all three hosts, `LIST UPS` from the laptop, `dev-server.sh status` all-ok, live suite green.
- Fixed in this pass: `sources/nut.py` unreachable-upsd message no longer prints a bare `(TimeoutError: )` (now `(timed out after 4s)` / `(ConnectionRefusedError: …)`), pinned by a new test + extended refused-port test.
- Docs: README verified consistent (every env var in `config.py` documented, routes/flags/`--with-llm`/pager sizes checked) — no changes needed. AGENTS.md fixed 4 inconsistencies: per-host sudo reality (gateway grants `NOPASSWD ALL`, which is why it installed without `agent-sudoers`; driveripper scoped; deskwork unrelated), the now-proven firewall finding, `/system` table typo ("agent per host"), gap-4 DB/WAL sizes + test count. `.env.example` `upsd.users` sample now one `instcmds` per line (it contradicted README and the in-app copy). `.env.dev` (gitignored) comment corrected to match the evidence.
- Flagged, untouched (locked): `.agents/PLAN.md` L92–95 still says upsd is "forwarded to `127.0.0.1:3494` in dev" — false after this phase; evidence + exact operator commands appended to `data/logs/NOTES-verify.md` §11.
- Next pending phase: `11_admin_login`.
This commit is contained in:
ducoterra
2026-10-18 05:44:13 -04:00
parent fefbeb8227
commit a08e693698
55 changed files with 1259 additions and 299 deletions
Download Patch File Download Diff File Expand all files Collapse all files
AGENTS.md
+75 -20
View File
@@ -25,7 +25,7 @@ file — plus an overview page that says which of the four you need:
| Route | Question it answers | Source |
| --------- | ------------------------------------------------------------------------------ | ------------------------------------------------- |
| `/` | Is anything wrong, and where do I look first? | all four, in one request |
| `/system` | What are my machines doing — CPU, memory, disk, GPU, network, and the history | `agent/host_agent.py`, our own stdlib agent/host |
| `/system` | What are my machines doing — CPU, memory, disk, GPU, network, and the history | `agent/host_agent.py`, our own stdlib agent per host |
| `/disks` | Are any of my physical disks failing, and which one — health, temp, hours, wear | the same agents, `smartctl -i -A -H` per whole drive (never a partition) |
| `/llm` | Tokens/sec, active sessions, queue depth, KV pressure, latency percentiles | gufo `/metrics` + `/slots`, vLLM `/metrics` |
| `/ups` | Battery health, runtime left, power events, battery-test buttons | NUT `upsd` (our own wire-protocol client; 3051 here, not 3493) |
@@ -311,14 +311,55 @@ That scheme is the history API's key — treat it as a public interface.
| `deskwork` | `deskwork-ops` | Strix Halo (Radeon 8060S iGPU), gufo `:8731` |
| `gateway` | `gateway-ops` | router, no GPU |
- Agents run from `~/lab-agent/host_agent.py` on each host, forwarded to `127.0.0.1:9121/9122/9123`
by `scripts/dev-tunnels.sh` (`DEV_TUNNELS`). Nothing listens on the hosts in dev.
- **Transport is direct dial: dev is the prod topology, repo-wide (phase 10 task 05).** The
designed shape is one `dashboards-agent` **systemd service per host**, installed by `ops`
through the `deploy/agent-sudoers` grant with `deploy/install-agent.sh`, bound to
the address the dashboard dials, its bearer token carried in the `HOST_TARGETS` URL
userinfo; driveripper's upsd serves the LAN on 3051 with the host and laptop ranges allowed
by the **firewall**, never by a `upsd.conf` line. The old ssh-tunnel script is deleted and
neither `.env.dev` nor `.env.dev.example` declares a forwarded port — dev dials the same
addresses the deployed dashboard does, and `scripts/capture-fixtures.sh` reads
`HOST_TARGETS` from `.env.dev` instead of naming hosts of its own
(`tests/test_deploy_scripts.py` fails the day any tunnel name or port returns). Current state: `gateway` runs its installed service, bound to the address
`gateway.reeselink.com` resolves to, and answers `/health` + `/snapshot` **on the host**
(re-verified 2026-10-18, and the token in `.env.dev` matches `/etc/dashboards-agent.token`),
but the laptop cannot reach its `:9120` — and that is now proven to be nobody's firewall here:
opening `9120/tcp` in that host's own firewalld (a runtime rule, added and reverted in the
phase-10 final pass; runtime and permanent port lists are back to their original set) changed
nothing, because the SYN never arrives — `ss -tan sport = :9120` on the host shows only the
`LISTEN` — while the same laptop reaches `:22` and `:80` on the same address in milliseconds.
The drop is the port-selective inter-VLAN filter on the lab gateway `10.2.0.1`, the same one
that eats 3051/3493. On `driveripper` and `deskwork` the ad-hoc `~/lab-agent/host_agent.py` agents
are gone (stopped with the old tunnel sessions, nothing replaced them, `:9120` is empty on
both) because their `deploy/agent-sudoers` grant is not installed yet, so the installer
cannot run there. Every `system`/`disks` source is therefore `DOWN` with its real reason
and renders as an `unreachable` stub row — the designed degradation, recorded rather than
hidden — which is also why 6 of the `live` tests fail on transport while both engines pass.
The per-host record, the exact root commands and the one firewall line each host still
needs are in `data/logs/NOTES-agent-install.md`, how far the NUT wire reaches from each
network is in `data/logs/NOTES-upsd-acl.md`, and the end-to-end direct-dial verification
(health table, no-tunnel-port proof, live-suite tally, screenshots, and the probes that put
the block on the lab gateway's port allow-list rather than in this app) is in
`data/logs/NOTES-verify.md`.
- Engines are reached directly: gufo `deskwork.reeselink.com:8731`, vLLM `driveripper.reeselink.com:8000`.
- NUT 2.8.5 on driveripper: **configs live in `/etc/ups/`, not `/etc/nut/`**, and **upsd is
on port 3051, not 3493**. Nothing listens on 3493 at all, so a default client reports
*Connection refused* and it looks like a dead service — `ss -ltnH` plus `printf 'LIST UPS\n'
| nc 127.0.0.1 3051` (answers `BEGIN LIST UPS`) is how that was pinned down. Two UPSes are
exported over USB and read live through `DEV_FORWARDS="3494:driveripper-ops:3051"`:
| nc 127.0.0.1 3051` (answers `BEGIN LIST UPS`) is how that was pinned down.
**NUT 2.8 has no `ACL` directive** (this Fedora build is `--without-wrap`; upsd logs
`ACL in upsd.conf is no longer supported - switch to LISTEN` once per line and then ignores
it), so per-source restriction is a **firewall** rule and never an `upsd.conf` line — and
2.8 does not refuse non-local clients the way older docs suggest: `LISTEN 0.0.0.0 3051` +
`LISTEN :: 3051` with `3051/tcp` in the `FedoraServer` zone answers any client routed to it
(reads are unauthenticated; `SET`/`INSTCMD` still need an `upsd.users` account). The host
range `10.1.0.0/21` does reach it today (verified from `deskwork`); the laptop's
`10.2.0.0/22` does **not** — the lab gateway `10.2.0.1` forwards 22/1234/8000-8004 and
silently drops 3051/3493 in both families, which no sudo on a fleet host can change, so the
laptop cannot yet dial `NUT_PORT=3051` (probe evidence and the operator's one rule:
`data/logs/NOTES-upsd-acl.md`).
Two UPSes are exported over USB and dialled directly as `NUT_HOST` + `NUT_PORT=3051` (from the
laptop that is still the blocked path above; from the host itself, and from the deployed
dashboard, it reads live):
`apc` = APC Back-UPS RS 1500MS, `cyberpower` = CyberPower PR1500LCDRT2U (that is
`ups.description`, the operator's label — the driver's own `identity.model` answers
`GX1500U`, so the description is what identifies the box).
@@ -331,19 +372,28 @@ That scheme is the history API's key — treat it as a public interface.
Both UPSes are recorded verbatim (`LIST UPS`/`LIST VAR`/`LIST CMD`, serials scrubbed) as
`tests/fixtures/nut_<ups>.vars` by `./scripts/capture-fixtures.sh`, and the wire quotes a
field only when it must — `VAR apc ups.status "OL"`, not three quoted fields.
- Host access is via ssh keys as `ops`. Passwordless sudo exists but is command-scoped; the
set we rely on is documented in `deploy/nut-sudoers` (additions are listed there for the
work that is still blocked).
- Host access is via ssh keys as `ops`, and **passwordless sudo is different on every host** —
that is what decides how much of the fleet an install can finish unattended. `gateway` grants
`(ALL) NOPASSWD: ALL`, which is why its agent installed in phase 10 with no
`deploy/agent-sudoers` line present; `driveripper` grants a command-scoped set (`smartctl`,
`upsc`, `udevadm`, the `/etc/ups` reads, and the `tee /etc/ups/upsd.users|upsd.conf` +
`systemctl restart|reload nut-server` pair documented in `deploy/nut-sudoers`); `deskwork`
grants an unrelated set (`btrfs scrub`, `snapper`, `setfacl /srv/*`, `osbuild-auto.sh`).
Neither `driveripper` nor `deskwork` can run the installer or `visudo`, so those two installs
— and the `firewall-cmd` line each still needs — stay with the operator.
- **SMART is captured from real hardware; the live agents still cannot read it.** `smartctl`
7.5 is installed on deskwork and driveripper. The `ops` account (which runs the dev agents)
is in no `disk` group; driveripper grants it `(root) NOPASSWD: /usr/sbin/smartctl*` and
deskwork grants nothing, so the agent's own unprivileged `smartctl -i -A -H` — it shells the
binary and never calls sudo — answers *Permission denied* (exit 2) on every host and `smart`
stays `null` end to end: the designed degradation, not a bug. The three deployed agents also
predate the SMART pass, so the `disks` source today yields one row per physical disk (16 on
driveripper, 1 on deskwork, `vda` on gateway) with `smart: null`, `smart_errors: []` and
`smartctl_available: null` — "nobody told us either way", which is why that flag is
three-way and not a boolean.
stays `null` end to end: the designed degradation, not a bug. The agents deployed on
driveripper and deskwork also predate the SMART pass, so the `disks` source today yields one row
per physical disk (16 on driveripper, 1 on deskwork, `vda` on gateway) with `smart: null`,
`smart_errors: []` and `smartctl_available: null` — "nobody told us either way", which is why
that flag is three-way and not a boolean. `gateway`'s agent is the one exception as of phase 10:
it is the installed systemd service, which does know about the SMART pass and answers
`smartctl_available: false` there because smartmontools is not installed — the *install it*
half of the three-way flag, from a host the dashboard cannot yet reach (its firewall).
**Real data is in the repo now:** `tests/fixtures/smartctl_driveripper_<dev>.txt` is all 16
drives (8 NVMe, 8 spinning) read through that sudo rule, and `tests/test_agent.py` parses
every one — which is what found the parser's one real bug: this binary annotates the
@@ -376,14 +426,14 @@ That scheme is the history API's key — treat it as a public interface.
```bash
uv sync --python 3.14 # deps + editable install
cp .env.example .env # production-shaped config
cp .env.dev.example .env.dev # dev-only: ssh tunnels + live targets
cp .env.dev.example .env.dev # dev-only: the real hosts, tokens in URL userinfo
./scripts/dev-tunnels.sh start # agents (9121-9123) + DEV_FORWARDS (upsd 3494)
set -a; . ./.env.dev; set +a # dev dials the fleet the prod way, no tunnels
./scripts/dev-server.sh restart # keep http://127.0.0.1:8090 up
./scripts/dev-server.sh status # per-source ok/DOWN
./scripts/dev-server.sh logs
uv run pytest # hermetic: 769 passed + 11 live-skipped, no network
uv run pytest # hermetic: 793 passed + 11 live-skipped, no network
uv run pytest --cov # coverage gate (98.33%; floor 94%)
uv run pytest -m live --live # opt-in: real fleet / engines / upsd
uv run ruff check src tests agent && uv run pyright
@@ -402,8 +452,13 @@ bash .agents/validate.sh # the phase gate: the four checks abov
Deployment: `Containerfile` (two-stage, `uv sync --frozen`, runs as uid 10001 on `0.0.0.0:8090`,
SQLite under `/var/lib/dashboards` — put a volume there) and `deploy/install-agent.sh`
per monitored host (systemd unit, own user, `--bind`/`--port`/`--token`/`--rotate-token`/
`--status`/`--uninstall`; it stores the token in a root-owned 0600 file and hands the agent
`--token-file`, never argv, and runs under `ProtectSystem=strict`).
`--status`/`--uninstall`; it stores the token `0640 root:dashboards-agent` — the unit hands the
agent the *path* and the agent opens it as its own user, so that user must be able to read it and
no other account may — never puts a token on argv, and runs under `ProtectSystem=strict` with
`DevicePolicy=strict` reopening only the disk nodes plus the `/dev/null` family, because strict
without `/dev/null` kills the unit at spawn (`status=208/STDIN`). No root shell? The one sudoers
line that lets `ops` run the installer is `deploy/agent-sudoers`; run the script without root and
it prints that line with its own resolved path.)
The quadlet unit + env template + nginx/SNI live in the deployment repo, like every other
service here. The source itself is `git@git.reeseapps.com:services/dashboards.git` (`origin`,
branch `main`); there is no CI on it, so `bash .agents/validate.sh` is the only gate.
@@ -823,8 +878,8 @@ memory plot; the `cpu.usage_pct@coreN` series is still stored and still queryabl
field map is written.
3. **UPS history depth.** Events and the numeric series are stored; long-window battery
health (capacity fade over months) has the rollups but no dedicated view.
4. **The dev DB is big enough to refuse a write now and then.** The dev store (~590 MB, WAL
grown to ~70 MB) logs `store-error: OperationalError: database is locked` a few times an
4. **The dev DB is big enough to refuse a write now and then.** The dev store (~610 MB, WAL
grown past 140 MB) logs `store-error: OperationalError: database is locked` a few times an
hour across `system`/`llm`/`ups`; the poller handles it exactly as invariant 4 requires —
the source goes red with the real reason, ages, and writes `recording` when the next poll
lands — so nothing freezes, but those polls' samples are lost. The one cause this repo can