Read the real UPSes: upsd lives on 3051, not 3493
Nothing was listening on 3493, so the dashboard said "connection refused" and fell back to the simulator while upsd was healthy the whole time. It binds a nonstandard port: `ss -ltnH` + a NUT handshake found it, and `upsc -L localhost:3051` lists both batteries. No host config had to change — dev now forwards 3494 -> 3051 and `/ups` shows real numbers, including an honest "battery is 5.3 years old" on the APC. Docs now say what to check before blaming the service (`ss -ltnH`, `printf 'LIST UPS\n' | nc 127.0.0.1 PORT`, `upsc -L host:port`), and the remaining UPS gap is rewritten: reads work, instant commands still need a scoped upsd.users entry — which needs a *read* of that file, because `tee` writing it blind would drop the users upsmon authenticates as. Real-hardware quirks recorded while verifying: battery.mfr.date is the date that means something (battery.date is often an unset placeholder), CyberPower reports `battery.mfr.date: CPS`, and neither driver reports a battery temperature — all three already render as unknown rather than a invented number. LLM-Generated: true
This commit is contained in:
@@ -12,6 +12,12 @@
|
|||||||
export DEV_TUNNELS="9121:driveripper-ops:9120 9122:deskwork-ops:9120 9123:gateway-ops:9120"
|
export DEV_TUNNELS="9121:driveripper-ops:9120 9122:deskwork-ops:9120 9123:gateway-ops:9120"
|
||||||
export DEV_FORWARDS="3494:driveripper-ops:3493"
|
export DEV_FORWARDS="3494:driveripper-ops:3493"
|
||||||
|
|
||||||
|
# Do not assume upsd is on 3493 — nonstandard ports are common and a refused connection then
|
||||||
|
# reads as "upsd is broken" while the service is fine. Ask the host what actually speaks NUT:
|
||||||
|
# ss -ltn # what is listening
|
||||||
|
# printf 'LIST UPS\n' | nc 127.0.0.1 PORT # the one that answers "BEGIN LIST UPS"
|
||||||
|
# upsc -L localhost:PORT # names + descriptions of every UPS
|
||||||
|
|
||||||
# Point the dashboard at those tunnels.
|
# Point the dashboard at those tunnels.
|
||||||
export HOST_TARGETS="driveripper=http://127.0.0.1:9121 deskwork=http://127.0.0.1:9122 gateway=http://127.0.0.1:9123"
|
export HOST_TARGETS="driveripper=http://127.0.0.1:9121 deskwork=http://127.0.0.1:9122 gateway=http://127.0.0.1:9123"
|
||||||
export GUFO_BASE_URL=http://127.0.0.1:8731
|
export GUFO_BASE_URL=http://127.0.0.1:8731
|
||||||
|
|||||||
+7
-3
@@ -37,9 +37,13 @@ RATE_WINDOW_SECONDS=10
|
|||||||
|
|
||||||
# --------------------------------------------------------------------------- #
|
# --------------------------------------------------------------------------- #
|
||||||
# UPS dashboard: the NUT data server (upsd)
|
# UPS dashboard: the NUT data server (upsd)
|
||||||
# upsd must be reachable from this process. Keep it off the public interface: bind it to
|
# upsd must be reachable from this process. Confirm the port before assuming 3493 — a wrong
|
||||||
# loopback (`LISTEN 127.0.0.1` in upsd.conf) and reach it with an ssh forward or a tailnet
|
# one looks exactly like a dead service:
|
||||||
# address, or bind a LAN address only if a firewall rule limits it to this host.
|
# ss -ltnH # what the NUT host is listening on
|
||||||
|
# printf 'LIST UPS\n' | nc 127.0.0.1 PORT # NUT answers: BEGIN LIST UPS
|
||||||
|
# Keep it off the public interface: bind it to loopback (`LISTEN 127.0.0.1` in upsd.conf) and
|
||||||
|
# reach it with an ssh forward or a tailnet address, or bind a LAN address only if a firewall
|
||||||
|
# rule limits it to this host.
|
||||||
#
|
#
|
||||||
# Reads need no user in stock NUT. For the battery-test buttons add one, scoped as narrowly
|
# Reads need no user in stock NUT. For the battery-test buttons add one, scoped as narrowly
|
||||||
# as the buttons need — `instcmds` beats `actions = -`, which grants load.off and shutdown.*
|
# as the buttons need — `instcmds` beats `actions = -`, which grants load.off and shutdown.*
|
||||||
|
|||||||
@@ -123,12 +123,16 @@ That scheme is the history API's key — treat it as a public interface.
|
|||||||
- Agents run from `~/lab-agent/host_agent.py` on each host, forwarded to `127.0.0.1:9121/9122/9123`
|
- Agents run from `~/lab-agent/host_agent.py` on each host, forwarded to `127.0.0.1:9121/9122/9123`
|
||||||
by `scripts/dev-tunnels.sh` (`DEV_TUNNELS`). Nothing listens on the hosts in dev.
|
by `scripts/dev-tunnels.sh` (`DEV_TUNNELS`). Nothing listens on the hosts in dev.
|
||||||
- Engines are reached directly: gufo `deskwork.reeselink.com:8731`, vLLM `driveripper.reeselink.com:8000`.
|
- Engines are reached directly: gufo `deskwork.reeselink.com:8731`, vLLM `driveripper.reeselink.com:8000`.
|
||||||
- NUT 2.8.5 on driveripper: **configs live in `/etc/ups/`, not `/etc/nut/`**. `nut-server`
|
- NUT 2.8.5 on driveripper: **configs live in `/etc/ups/`, not `/etc/nut/`**, and **upsd is
|
||||||
is active and `nut-driver@apc` + `nut-driver@cyberpower` are both running, so `ups.conf`
|
on port 3051, not 3493**. Nothing listens on 3493 at all, so a default client reports
|
||||||
already exports two driver instances; a CyberPower PR1500LCDRT2U is attached on USB
|
*Connection refused* and it looks like a dead service — `ss -ltnH` plus `printf 'LIST UPS\n'
|
||||||
(`0764:0601`). **`upsd` does not bind 3493**, so reads fail with *Connection refused* and
|
| nc 127.0.0.1 3051` (answers `BEGIN LIST UPS`) is how that was pinned down. Two UPSes are
|
||||||
the dashboard falls back to the labelled simulator. A restart of `nut-server` does not fix
|
exported over USB and read live through `DEV_FORWARDS="3494:driveripper-ops:3051"`:
|
||||||
it. See *Known gaps* — this needs root logs and a config read.
|
`apc` = APC Back-UPS RS 1500MS, `cyberpower` = CyberPower PR1500LCDRT2U.
|
||||||
|
Real-hardware quirks the parsers already handle: `battery.mfr.date` is the useful date
|
||||||
|
(`battery.date` is often an unset placeholder like `2001-09-25`), CyberPower reports
|
||||||
|
`battery.mfr.date: CPS` (not a date → `unknown`, never a fake age), neither driver reports
|
||||||
|
a battery temperature, and `ups.test.result: No test initiated` is the factory state.
|
||||||
- Host access is via ssh keys as `ops`. Passwordless sudo exists but is command-scoped; the
|
- Host access is via ssh keys as `ops`. Passwordless sudo exists but is command-scoped; the
|
||||||
set we rely on is documented in `deploy/nut-sudoers` (additions are listed there for the
|
set we rely on is documented in `deploy/nut-sudoers` (additions are listed there for the
|
||||||
work that is still blocked).
|
work that is still blocked).
|
||||||
@@ -186,12 +190,14 @@ unconfigured. Escape anything upstream-provided through `DashCharts.escapeHtml`.
|
|||||||
|
|
||||||
## Known gaps / next work
|
## Known gaps / next work
|
||||||
|
|
||||||
1. **Real UPS data (blocking issue, host-side).** `upsd` on driveripper runs but binds
|
1. **UPS instant commands are read-only in practice.** Reads work against real upsd (see
|
||||||
nothing. Needed: root journal for `nut-server` (`journalctl -u nut-server`), a read of
|
above), but every button is gated off with "NUT_USER / NUT_PASSWORD must name a user with
|
||||||
`/etc/ups/{upsd.conf,upsd.users,ups.conf}`, then `LISTEN 127.0.0.1` (plus a `dashboards`
|
`actions` in upsd.users" because no such user exists yet. Creating one safely needs a
|
||||||
user with scoped `instcmds`). Then: `DEV_FORWARDS="3494:driveripper-ops:3493"`,
|
*read* of `/etc/ups/upsd.users` first — `sudo tee` can write the file but overwriting it
|
||||||
`NUT_HOST=127.0.0.1 NUT_PORT=3494`, unset `NUT_SIMULATE`, and re-verify `/ups` against a
|
blind would delete the users upsmon already authenticates as, which can take the box down
|
||||||
real battery. Until then every UPS number is modelled, and the page says so.
|
at the wrong moment. `deploy/nut-sudoers` lists the read rules; once they are in, add
|
||||||
|
`[dashboards]` with `instcmds` limited to the battery tests, put the credentials in
|
||||||
|
`.env.dev`, and the buttons come alive. Never grant `actions = -`.
|
||||||
2. **GPU coverage is AMD only.** `host_agent.gpus()` reads amdgpu DRM sysfs
|
2. **GPU coverage is AMD only.** `host_agent.gpus()` reads amdgpu DRM sysfs
|
||||||
(`gpu_busy_percent`, `mem_info_vram_*`, hwmon temps, clocks). That covers the current
|
(`gpu_busy_percent`, `mem_info_vram_*`, hwmon temps, clocks). That covers the current
|
||||||
fleet (R9700 + Strix Halo iGPU; gateway has none). An NVIDIA host would report no GPU at
|
fleet (R9700 + Strix Halo iGPU; gateway has none). An NVIDIA host would report no GPU at
|
||||||
|
|||||||
@@ -87,7 +87,7 @@ documentation and is never loaded, and `scripts/dev-server.sh` exports `.env.dev
|
|||||||
| `HOST_INTERVAL_SECONDS` | poll interval for hosts (default 10) |
|
| `HOST_INTERVAL_SECONDS` | poll interval for hosts (default 10) |
|
||||||
| `GUFO_BASE_URL` / `VLLM_BASE_URL` | LLM engines; unset means *not deployed here*, and its half of the page disappears |
|
| `GUFO_BASE_URL` / `VLLM_BASE_URL` | LLM engines; unset means *not deployed here*, and its half of the page disappears |
|
||||||
| `RATE_WINDOW_SECONDS` | the Δ window behind every measured token rate (default 10) |
|
| `RATE_WINDOW_SECONDS` | the Δ window behind every measured token rate (default 10) |
|
||||||
| `NUT_HOST` / `NUT_PORT` | upsd address. Reads work unauthenticated in stock NUT |
|
| `NUT_HOST` / `NUT_PORT` | upsd address — **whatever port it really listens on**, not always 3493. Reads are unauthenticated in stock NUT |
|
||||||
| `NUT_USER` / `NUT_PASSWORD` | only needed for instant commands (battery tests) |
|
| `NUT_USER` / `NUT_PASSWORD` | only needed for instant commands (battery tests) |
|
||||||
| `NUT_ALLOW_ACTIONS` | `1` enables the one write endpoint in the app |
|
| `NUT_ALLOW_ACTIONS` | `1` enables the one write endpoint in the app |
|
||||||
| `NUT_SIMULATE` | `1` replaces upsd with a **labelled** simulator (development only) |
|
| `NUT_SIMULATE` | `1` replaces upsd with a **labelled** simulator (development only) |
|
||||||
|
|||||||
Reference in New Issue
Block a user