phase: 10_dev_direct_dial
**Phase 10 final pass — repo green; live criteria blocked off-repo.**
- Verified: `gateway` runs `dashboards-agent` (active/enabled, `10.1.1.27:9120` → `/health` 200, `/snapshot` 401 no-token / 200 bearer on-host, token matches `/etc/dashboards-agent.token`); `driveripper`/`deskwork` inactive, `sudo -n` → "a password is required", `:9120` empty.
- New measurement (transient runtime rule on gateway `firewall-cmd --add-port=9120/tcp`, **reverted** — runtime+permanent port lists verified back to the original set, no rich rules): laptop still gets no SYN through, while `:22`/`:80` on the same address answer in ms ⇒ the block is the lab gateway `10.2.0.1` port filter, not this app/host firewall/agent. No other live change made (no sudoers, no upsd, no UPS command).
- Tests/lint (exact): `bash .agents/validate.sh` → **EXIT 0** — `uv run pytest --cov --cov-fail-under=94` **793 passed, 11 skipped, coverage 98.33 %**; `ruff check src tests agent` clean; `pyright` 0 errors; `node --check` clean; hygiene clean; `uv pip list --outdated` empty (bumped fastapi 0.143.3→0.143.5, `uv.lock` only).
- `uv run pytest -m live --live` (`.env.dev` sourced): **4 passed, 1 skipped, 6 failed** — all six are `:9120`/`:3051` connect timeouts; zero parse/contract failures.
- Dev server restarted on current code: engines `ok`; 3×`host` + 3×`disks` + `nut` `DOWN` with real reasons (designed degradation, stub rows intact, no credential leakage).
- Completion criteria: screenshots ✓ (`data/logs/phase10-{index,system,disks,llm,ups}-live.png`); tunnel-vocabulary grep ✓ outside `.agents/` history (PLAN.md locked line, phase overviews, past reports; fixtures/`uv.lock` hits are digit substrings — `tests/test_deploy_scripts.py` guards operational files); no page-payload change ✓ (contract tests). **Not met, operator-side:** agents on all three hosts, `LIST UPS` from the laptop, `dev-server.sh status` all-ok, live suite green.
- Fixed in this pass: `sources/nut.py` unreachable-upsd message no longer prints a bare `(TimeoutError: )` (now `(timed out after 4s)` / `(ConnectionRefusedError: …)`), pinned by a new test + extended refused-port test.
- Docs: README verified consistent (every env var in `config.py` documented, routes/flags/`--with-llm`/pager sizes checked) — no changes needed. AGENTS.md fixed 4 inconsistencies: per-host sudo reality (gateway grants `NOPASSWD ALL`, which is why it installed without `agent-sudoers`; driveripper scoped; deskwork unrelated), the now-proven firewall finding, `/system` table typo ("agent per host"), gap-4 DB/WAL sizes + test count. `.env.example` `upsd.users` sample now one `instcmds` per line (it contradicted README and the in-app copy). `.env.dev` (gitignored) comment corrected to match the evidence.
- Flagged, untouched (locked): `.agents/PLAN.md` L92–95 still says upsd is "forwarded to `127.0.0.1:3494` in dev" — false after this phase; evidence + exact operator commands appended to `data/logs/NOTES-verify.md` §11.
- Next pending phase: `11_admin_login`.
This commit is contained in:
@@ -27,29 +27,26 @@ uv run uvicorn dashboards.app:app --port 8090 # equivalent
|
||||
uv run pytest # unit tests, no network needed
|
||||
```
|
||||
|
||||
Developing against the real fleet from a laptop (agents stay unexposed — ssh forwards
|
||||
them to localhost):
|
||||
Developing against the real fleet from a laptop — dev dials exactly the addresses the deployed
|
||||
dashboard dials, so there is nothing to tunnel and no port to forward:
|
||||
|
||||
```bash
|
||||
cp .env.dev.example .env.dev # your ssh aliases + which ports
|
||||
python3 - <<'PY' # push the agent to each host (once)
|
||||
import shlex, subprocess, pathlib
|
||||
for host in ("driveripper-ops", "deskwork-ops", "gateway-ops"):
|
||||
subprocess.run(["ssh", host, "mkdir -p ~/lab-agent"], check=True)
|
||||
subprocess.run(["scp", "agent/host_agent.py", f"{host}:~/lab-agent/"], check=True)
|
||||
PY
|
||||
./scripts/dev-tunnels.sh start # 9121/9122/9123 -> each host's agent,
|
||||
# + DEV_FORWARDS -> upsd on 127.0.0.1:3494
|
||||
# (the remote port is upsd's real one — 3051
|
||||
# on driveripper, not the 3493 default)
|
||||
cp .env.dev.example .env.dev # your host addresses + agent tokens (chmod 0600)
|
||||
sudo deploy/install-agent.sh --bind <addr> --port 9120
|
||||
# once per monitored host: the agent as a systemd
|
||||
# service, printing its bearer token exactly once
|
||||
set -a; . ./.env.dev; set +a
|
||||
uv run dashboards # foreground
|
||||
./scripts/dev-server.sh restart # or backgrounded, with a pidfile + logs
|
||||
```
|
||||
|
||||
`DEV_TUNNELS` starts the agent remotely and forwards it; `DEV_FORWARDS` forwards a service
|
||||
that already listens there (upsd) — that is how the UPS page reads a real battery without
|
||||
3493 being reachable from anywhere but this laptop.
|
||||
Each `HOST_TARGETS` entry carries that token as URL userinfo
|
||||
(`driveripper=http://<token>@<addr>:9120`); `parse_targets` moves it into an `Authorization`
|
||||
header, so `Target.url`, `/api/config` and every error message stay credential-free. No root
|
||||
shell on a host? `deploy/agent-sudoers` is the one-line grant that lets `ops` run the installer,
|
||||
and running the installer without root prints that exact line. upsd is dialled at
|
||||
`NUT_HOST`/`NUT_PORT` on the port it really answers on (3051 on driveripper, not the stock
|
||||
3493) — see the NUT section below for who may connect.
|
||||
|
||||
## The host agent
|
||||
|
||||
@@ -68,6 +65,10 @@ deploy/install-agent.sh --status # unit state and /he
|
||||
sudo deploy/install-agent.sh --uninstall
|
||||
```
|
||||
|
||||
No root shell on the monitored host? `deploy/agent-sudoers` is the single sudoers line that
|
||||
lets the fleet's unprivileged `ops` account run the installer; run it without root and it
|
||||
prints that exact line with the script's own resolved path instead of saying "run as root".
|
||||
|
||||
It is read-only by construction: there is no code path that writes, and the HTTP server
|
||||
only answers `GET`. Counters are reported as counters — the dashboard computes rates over
|
||||
a labelled window, so the agent never has to remember who polled it last.
|
||||
@@ -92,8 +93,11 @@ Binding off-loopback without a token is refused (exit 2) rather than warned abou
|
||||
`--token-file` over `--token`: argv is world-readable through `ps`, and this token guards
|
||||
hostname, mounts, disk and GPU models, and process names. `deploy/install-agent.sh` creates
|
||||
a system user, generates a token once (re-runs reuse it; `--rotate-token` for a new one),
|
||||
stores it `0600` root-owned, and runs the agent under `ProtectSystem=strict` with no
|
||||
capabilities — enough to read `/proc`, `/sys` and `statvfs()` and nothing else.
|
||||
stores it `0640 root:dashboards-agent` — the unit hands the agent the *path*, and the agent
|
||||
opens it as its own user, so the file must be readable by that user and by nobody else —
|
||||
and runs the agent under `ProtectSystem=strict`, `DevicePolicy=strict` with only the disk
|
||||
nodes and the `/dev/null` family reopened (SMART needs the former; without `/dev/null` in
|
||||
the allow list systemd cannot start the service at all), with no capabilities.
|
||||
|
||||
## Configuration
|
||||
|
||||
@@ -400,8 +404,14 @@ files: `sudo install -m 0440 -o root -g root deploy/nut-sudoers /etc/sudoers.d/d
|
||||
Read `upsd.users` before you write it — `upsmon` authenticates from the same file, so
|
||||
overwriting it blind can take the box down at the wrong moment.
|
||||
|
||||
Bind upsd to loopback (`LISTEN 127.0.0.1` in `upsd.conf`) and reach it with an ssh forward
|
||||
(`DEV_FORWARDS`) or a tailnet address. `NUT_SIMULATE=1` stands in for upsd during
|
||||
Bind upsd to loopback (`LISTEN 127.0.0.1 3051` in `upsd.conf`) when the dashboard runs on the
|
||||
same box, and to a real interface plus a source-scoped firewall rule when it runs elsewhere —
|
||||
and do not try to scope it by client inside `upsd.conf` instead: **NUT 2.8
|
||||
removed the `ACL`/`ACCEPT`/`REJECT` directives** (a build without TCP Wrappers logs
|
||||
`ACL in upsd.conf is no longer supported - switch to LISTEN` once per line and ignores the
|
||||
line), so `LISTEN` plus a host firewall rule is the only access control `upsd.conf` offers,
|
||||
and an unauthenticated `LIST VAR` from any routed source is the normal state of a NUT server.
|
||||
`NUT_SIMULATE=1` stands in for upsd during
|
||||
development; everything it produces is badged `SIMULATED`, and it is never the default.
|
||||
|
||||
## Deployment
|
||||
|
||||
Reference in New Issue
Block a user