diff --git a/.env.dev.example b/.env.dev.example index dbe93f3..1e97b33 100644 --- a/.env.dev.example +++ b/.env.dev.example @@ -12,6 +12,12 @@ export DEV_TUNNELS="9121:driveripper-ops:9120 9122:deskwork-ops:9120 9123:gateway-ops:9120" export DEV_FORWARDS="3494:driveripper-ops:3493" +# Do not assume upsd is on 3493 — nonstandard ports are common and a refused connection then +# reads as "upsd is broken" while the service is fine. Ask the host what actually speaks NUT: +# ss -ltn # what is listening +# printf 'LIST UPS\n' | nc 127.0.0.1 PORT # the one that answers "BEGIN LIST UPS" +# upsc -L localhost:PORT # names + descriptions of every UPS + # Point the dashboard at those tunnels. export HOST_TARGETS="driveripper=http://127.0.0.1:9121 deskwork=http://127.0.0.1:9122 gateway=http://127.0.0.1:9123" export GUFO_BASE_URL=http://127.0.0.1:8731 diff --git a/.env.example b/.env.example index 3f0f6a3..fde6b9b 100644 --- a/.env.example +++ b/.env.example @@ -37,9 +37,13 @@ RATE_WINDOW_SECONDS=10 # --------------------------------------------------------------------------- # # UPS dashboard: the NUT data server (upsd) -# upsd must be reachable from this process. Keep it off the public interface: bind it to -# loopback (`LISTEN 127.0.0.1` in upsd.conf) and reach it with an ssh forward or a tailnet -# address, or bind a LAN address only if a firewall rule limits it to this host. +# upsd must be reachable from this process. Confirm the port before assuming 3493 — a wrong +# one looks exactly like a dead service: +# ss -ltnH # what the NUT host is listening on +# printf 'LIST UPS\n' | nc 127.0.0.1 PORT # NUT answers: BEGIN LIST UPS +# Keep it off the public interface: bind it to loopback (`LISTEN 127.0.0.1` in upsd.conf) and +# reach it with an ssh forward or a tailnet address, or bind a LAN address only if a firewall +# rule limits it to this host. # # Reads need no user in stock NUT. For the battery-test buttons add one, scoped as narrowly # as the buttons need — `instcmds` beats `actions = -`, which grants load.off and shutdown.* diff --git a/AGENTS.md b/AGENTS.md index 1ffbb7d..ab58f43 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -123,12 +123,16 @@ That scheme is the history API's key — treat it as a public interface. - Agents run from `~/lab-agent/host_agent.py` on each host, forwarded to `127.0.0.1:9121/9122/9123` by `scripts/dev-tunnels.sh` (`DEV_TUNNELS`). Nothing listens on the hosts in dev. - Engines are reached directly: gufo `deskwork.reeselink.com:8731`, vLLM `driveripper.reeselink.com:8000`. -- NUT 2.8.5 on driveripper: **configs live in `/etc/ups/`, not `/etc/nut/`**. `nut-server` - is active and `nut-driver@apc` + `nut-driver@cyberpower` are both running, so `ups.conf` - already exports two driver instances; a CyberPower PR1500LCDRT2U is attached on USB - (`0764:0601`). **`upsd` does not bind 3493**, so reads fail with *Connection refused* and - the dashboard falls back to the labelled simulator. A restart of `nut-server` does not fix - it. See *Known gaps* — this needs root logs and a config read. +- NUT 2.8.5 on driveripper: **configs live in `/etc/ups/`, not `/etc/nut/`**, and **upsd is + on port 3051, not 3493**. Nothing listens on 3493 at all, so a default client reports + *Connection refused* and it looks like a dead service — `ss -ltnH` plus `printf 'LIST UPS\n' + | nc 127.0.0.1 3051` (answers `BEGIN LIST UPS`) is how that was pinned down. Two UPSes are + exported over USB and read live through `DEV_FORWARDS="3494:driveripper-ops:3051"`: + `apc` = APC Back-UPS RS 1500MS, `cyberpower` = CyberPower PR1500LCDRT2U. + Real-hardware quirks the parsers already handle: `battery.mfr.date` is the useful date + (`battery.date` is often an unset placeholder like `2001-09-25`), CyberPower reports + `battery.mfr.date: CPS` (not a date → `unknown`, never a fake age), neither driver reports + a battery temperature, and `ups.test.result: No test initiated` is the factory state. - Host access is via ssh keys as `ops`. Passwordless sudo exists but is command-scoped; the set we rely on is documented in `deploy/nut-sudoers` (additions are listed there for the work that is still blocked). @@ -186,12 +190,14 @@ unconfigured. Escape anything upstream-provided through `DashCharts.escapeHtml`. ## Known gaps / next work -1. **Real UPS data (blocking issue, host-side).** `upsd` on driveripper runs but binds - nothing. Needed: root journal for `nut-server` (`journalctl -u nut-server`), a read of - `/etc/ups/{upsd.conf,upsd.users,ups.conf}`, then `LISTEN 127.0.0.1` (plus a `dashboards` - user with scoped `instcmds`). Then: `DEV_FORWARDS="3494:driveripper-ops:3493"`, - `NUT_HOST=127.0.0.1 NUT_PORT=3494`, unset `NUT_SIMULATE`, and re-verify `/ups` against a - real battery. Until then every UPS number is modelled, and the page says so. +1. **UPS instant commands are read-only in practice.** Reads work against real upsd (see + above), but every button is gated off with "NUT_USER / NUT_PASSWORD must name a user with + `actions` in upsd.users" because no such user exists yet. Creating one safely needs a + *read* of `/etc/ups/upsd.users` first — `sudo tee` can write the file but overwriting it + blind would delete the users upsmon already authenticates as, which can take the box down + at the wrong moment. `deploy/nut-sudoers` lists the read rules; once they are in, add + `[dashboards]` with `instcmds` limited to the battery tests, put the credentials in + `.env.dev`, and the buttons come alive. Never grant `actions = -`. 2. **GPU coverage is AMD only.** `host_agent.gpus()` reads amdgpu DRM sysfs (`gpu_busy_percent`, `mem_info_vram_*`, hwmon temps, clocks). That covers the current fleet (R9700 + Strix Halo iGPU; gateway has none). An NVIDIA host would report no GPU at diff --git a/README.md b/README.md index 0bd458a..b7472c1 100644 --- a/README.md +++ b/README.md @@ -87,7 +87,7 @@ documentation and is never loaded, and `scripts/dev-server.sh` exports `.env.dev | `HOST_INTERVAL_SECONDS` | poll interval for hosts (default 10) | | `GUFO_BASE_URL` / `VLLM_BASE_URL` | LLM engines; unset means *not deployed here*, and its half of the page disappears | | `RATE_WINDOW_SECONDS` | the Δ window behind every measured token rate (default 10) | -| `NUT_HOST` / `NUT_PORT` | upsd address. Reads work unauthenticated in stock NUT | +| `NUT_HOST` / `NUT_PORT` | upsd address — **whatever port it really listens on**, not always 3493. Reads are unauthenticated in stock NUT | | `NUT_USER` / `NUT_PASSWORD` | only needed for instant commands (battery tests) | | `NUT_ALLOW_ACTIONS` | `1` enables the one write endpoint in the app | | `NUT_SIMULATE` | `1` replaces upsd with a **labelled** simulator (development only) |