services/dashboards
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

phase: 10_dev_direct_dial

Browse Source 
**Phase 10 final pass — repo green; live criteria blocked off-repo.**

- Verified: `gateway` runs `dashboards-agent` (active/enabled, `10.1.1.27:9120` → `/health` 200, `/snapshot` 401 no-token / 200 bearer on-host, token matches `/etc/dashboards-agent.token`); `driveripper`/`deskwork` inactive, `sudo -n` → "a password is required", `:9120` empty.
- New measurement (transient runtime rule on gateway `firewall-cmd --add-port=9120/tcp`, **reverted** — runtime+permanent port lists verified back to the original set, no rich rules): laptop still gets no SYN through, while `:22`/`:80` on the same address answer in ms ⇒ the block is the lab gateway `10.2.0.1` port filter, not this app/host firewall/agent. No other live change made (no sudoers, no upsd, no UPS command).
- Tests/lint (exact): `bash .agents/validate.sh` → **EXIT 0** — `uv run pytest --cov --cov-fail-under=94` **793 passed, 11 skipped, coverage 98.33 %**; `ruff check src tests agent` clean; `pyright` 0 errors; `node --check` clean; hygiene clean; `uv pip list --outdated` empty (bumped fastapi 0.143.3→0.143.5, `uv.lock` only).
- `uv run pytest -m live --live` (`.env.dev` sourced): **4 passed, 1 skipped, 6 failed** — all six are `:9120`/`:3051` connect timeouts; zero parse/contract failures.
- Dev server restarted on current code: engines `ok`; 3×`host` + 3×`disks` + `nut` `DOWN` with real reasons (designed degradation, stub rows intact, no credential leakage).
- Completion criteria: screenshots ✓ (`data/logs/phase10-{index,system,disks,llm,ups}-live.png`); tunnel-vocabulary grep ✓ outside `.agents/` history (PLAN.md locked line, phase overviews, past reports; fixtures/`uv.lock` hits are digit substrings — `tests/test_deploy_scripts.py` guards operational files); no page-payload change ✓ (contract tests). **Not met, operator-side:** agents on all three hosts, `LIST UPS` from the laptop, `dev-server.sh status` all-ok, live suite green.
- Fixed in this pass: `sources/nut.py` unreachable-upsd message no longer prints a bare `(TimeoutError: )` (now `(timed out after 4s)` / `(ConnectionRefusedError: …)`), pinned by a new test + extended refused-port test.
- Docs: README verified consistent (every env var in `config.py` documented, routes/flags/`--with-llm`/pager sizes checked) — no changes needed. AGENTS.md fixed 4 inconsistencies: per-host sudo reality (gateway grants `NOPASSWD ALL`, which is why it installed without `agent-sudoers`; driveripper scoped; deskwork unrelated), the now-proven firewall finding, `/system` table typo ("agent per host"), gap-4 DB/WAL sizes + test count. `.env.example` `upsd.users` sample now one `instcmds` per line (it contradicted README and the in-app copy). `.env.dev` (gitignored) comment corrected to match the evidence.
- Flagged, untouched (locked): `.agents/PLAN.md` L92–95 still says upsd is "forwarded to `127.0.0.1:3494` in dev" — false after this phase; evidence + exact operator commands appended to `data/logs/NOTES-verify.md` §11.
- Next pending phase: `11_admin_login`.
This commit is contained in:
ducoterra
2026-10-18 05:44:13 -04:00
parent fefbeb8227
commit a08e693698
55 changed files with 1259 additions and 299 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.agents/phases/todo/10_dev_direct_dial/00_phase.md → .agents/phases/complete/10_dev_direct_dial/00_phase.md
View File
.agents/phases/todo/10_dev_direct_dial/01_sudoers_grant_and_installer_hint.md → .agents/phases/complete/10_dev_direct_dial/01_sudoers_grant_and_installer_hint.md
View File
.agents/phases/todo/10_dev_direct_dial/02_install_agents_on_fleet.md → .agents/phases/complete/10_dev_direct_dial/02_install_agents_on_fleet.md
View File
.agents/phases/todo/10_dev_direct_dial/03_upsd_acl_for_laptop.md → .agents/phases/complete/10_dev_direct_dial/03_upsd_acl_for_laptop.md
View File
.agents/phases/todo/10_dev_direct_dial/04_env_dev_direct_dial.md → .agents/phases/complete/10_dev_direct_dial/04_env_dev_direct_dial.md
View File
.agents/phases/todo/10_dev_direct_dial/05_retire_tunnels_update_scripts.md → .agents/phases/complete/10_dev_direct_dial/05_retire_tunnels_update_scripts.md
View File
.agents/phases/todo/10_dev_direct_dial/06_verify_direct_dial_live.md → .agents/phases/complete/10_dev_direct_dial/06_verify_direct_dial_live.md
View File
.agents/reports/10_dev_direct_dial/10_dev_direct_dial__00_phase.a1.err
View File
.agents/reports/10_dev_direct_dial/10_dev_direct_dial__00_phase.a1.md
+12
View File
@@ -0,0 +1,12 @@
**Phase 10 final pass — repo green; live criteria blocked off-repo.**
- Verified: `gateway` runs `dashboards-agent` (active/enabled, `10.1.1.27:9120` → `/health` 200, `/snapshot` 401 no-token / 200 bearer on-host, token matches `/etc/dashboards-agent.token`); `driveripper`/`deskwork` inactive, `sudo -n` → "a password is required", `:9120` empty.
- New measurement (transient runtime rule on gateway `firewall-cmd --add-port=9120/tcp`, **reverted** — runtime+permanent port lists verified back to the original set, no rich rules): laptop still gets no SYN through, while `:22`/`:80` on the same address answer in ms ⇒ the block is the lab gateway `10.2.0.1` port filter, not this app/host firewall/agent. No other live change made (no sudoers, no upsd, no UPS command).
- Tests/lint (exact): `bash .agents/validate.sh` → **EXIT 0** — `uv run pytest --cov --cov-fail-under=94` **793 passed, 11 skipped, coverage 98.33 %**; `ruff check src tests agent` clean; `pyright` 0 errors; `node --check` clean; hygiene clean; `uv pip list --outdated` empty (bumped fastapi 0.143.3→0.143.5, `uv.lock` only).
- `uv run pytest -m live --live` (`.env.dev` sourced): **4 passed, 1 skipped, 6 failed** — all six are `:9120`/`:3051` connect timeouts; zero parse/contract failures.
- Dev server restarted on current code: engines `ok`; 3×`host` + 3×`disks` + `nut` `DOWN` with real reasons (designed degradation, stub rows intact, no credential leakage).
- Completion criteria: screenshots ✓ (`data/logs/phase10-{index,system,disks,llm,ups}-live.png`); tunnel-vocabulary grep ✓ outside `.agents/` history (PLAN.md locked line, phase overviews, past reports; fixtures/`uv.lock` hits are digit substrings — `tests/test_deploy_scripts.py` guards operational files); no page-payload change ✓ (contract tests). **Not met, operator-side:** agents on all three hosts, `LIST UPS` from the laptop, `dev-server.sh status` all-ok, live suite green.
- Fixed in this pass: `sources/nut.py` unreachable-upsd message no longer prints a bare `(TimeoutError: )` (now `(timed out after 4s)` / `(ConnectionRefusedError: …)`), pinned by a new test + extended refused-port test.
- Docs: README verified consistent (every env var in `config.py` documented, routes/flags/`--with-llm`/pager sizes checked) — no changes needed. AGENTS.md fixed 4 inconsistencies: per-host sudo reality (gateway grants `NOPASSWD ALL`, which is why it installed without `agent-sudoers`; driveripper scoped; deskwork unrelated), the now-proven firewall finding, `/system` table typo ("agent per host"), gap-4 DB/WAL sizes + test count. `.env.example` `upsd.users` sample now one `instcmds` per line (it contradicted README and the in-app copy). `.env.dev` (gitignored) comment corrected to match the evidence.
- Flagged, untouched (locked): `.agents/PLAN.md` L92–95 still says upsd is "forwarded to `127.0.0.1:3494` in dev" — false after this phase; evidence + exact operator commands appended to `data/logs/NOTES-verify.md` §11.
- Next pending phase: `11_admin_login`.
.agents/reports/10_dev_direct_dial/10_dev_direct_dial__00_phase.a1.md.status
+1
View File
@@ -0,0 +1 @@
ok
.agents/reports/10_dev_direct_dial/10_dev_direct_dial__00_phase.a1.validate
+52
View File
@@ -0,0 +1,52 @@
== uv run pytest --cov (hermetic: no network, live tests skip) ==
........................................................................ [ 8%]
........................................................................ [ 17%]
........................................................................ [ 26%]
........................................................................ [ 35%]
........................................................................ [ 44%]
........................................................................ [ 53%]
..................................sssssssssss........................... [ 62%]
........................................................................ [ 71%]
........................................................................ [ 80%]
........................................................................ [ 89%]
........................................................................ [ 98%]
............ [100%]
=============================== warnings summary ===============================
.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.14.4-final-0 ________________
Name Stmts Miss Cover Missing
------------------------------------------------------------------
src/dashboards/__init__.py 3 0 100%
src/dashboards/__main__.py 14 1 93% 28
src/dashboards/alerts.py 176 0 100%
src/dashboards/api.py 221 0 100%
src/dashboards/app.py 101 0 100%
src/dashboards/collectors.py 155 5 97% 141-142, 146-147, 164
src/dashboards/config.py 115 0 100%
src/dashboards/debug.py 14 0 100%
src/dashboards/poller.py 165 3 98% 192-194, 279
src/dashboards/prom.py 105 5 95% 107, 144, 158, 161, 164
src/dashboards/rates.py 37 1 97% 51
src/dashboards/severity.py 49 0 100%
src/dashboards/sources/__init__.py 0 0 100%
src/dashboards/sources/disks.py 130 0 100%
src/dashboards/sources/host.py 199 1 99% 147
src/dashboards/sources/llm.py 150 1 99% 228
src/dashboards/sources/nut.py 390 20 95% 137, 151, 157, 169, 175, 187, 195, 208, 279-280, 282, 498, 510, 545, 578, 582, 639-640, 645, 657
src/dashboards/sources/nut_sim.py 134 0 100%
src/dashboards/store.py 178 2 99% 250-251
------------------------------------------------------------------
TOTAL 2336 39 98%
Required test coverage of 94% reached. Total coverage: 98.33%
793 passed, 11 skipped, 1 warning in 59.94s
== uv run ruff check src tests agent ==
All checks passed!
== uv run pyright ==
0 errors, 0 warnings, 0 informations
== node --check src/dashboards/web/static/*.js ==
.agents/reports/10_dev_direct_dial/10_dev_direct_dial__01_sudoers_grant_and_installer_hint.a1.err
View File
.agents/reports/10_dev_direct_dial/10_dev_direct_dial__01_sudoers_grant_and_installer_hint.a1.md
+16
View File
@@ -0,0 +1,16 @@
**Implemented**
- `deploy/agent-sudoers` (new, nut-sudoers house style): WHY block, canonical staging path `/home/ops/agent-install`, the single rule `ops ALL=(root) NOPASSWD: /home/ops/agent-install/deploy/install-agent.sh`, two-stage check-first install (`visudo -cf` → `install -m 0440 -o root -g root /etc/sudoers.d/dashboards-agent` → re-check), a "make the grant mean something" `chown -R root:root` note, and a "deliberately NOT granted" block (visudo/sh/tee, other `systemctl`, the token file, user*/groupmod, `upscmd`/`upsrw`).
- `deploy/install-agent.sh`: added resolved `SCRIPT_PATH` (realpath + fallback) and `SUDOERS_SOURCE`; `need_root`'s non-root path now prints the exact NOPASSWD line **for its own resolved path**, the two install steps, the pointer to `deploy/agent-sudoers`, and a copy-pasteable re-run command; exit code 1 and root behaviour unchanged (call sites now pass flags so the hint is absolute). Header help names the grant; dropped the stale "ssh tunnel in" parenthetical and marked `--status` as not needing root.
- `tests/test_deploy_scripts.py` (new, 16 tests): grant file is exactly one rule at the canonical path, sections/boundaries present, `visudo -cf` passes; installer exits 1 with empty stdout, prints a line identical to the file apart from the path (relative and absolute invocations), names the two-stage install, `--help` points at the file; `bash -n` over `deploy/*.sh` + `scripts/*.sh`.
- `data/logs/NOTES-sudoers.md` (+ `phase10-*` artifacts, gitignored): the real diff — identical apart from the path (path-normalised diff is empty).
**Verification**
- `visudo -cf deploy/agent-sudoers` → parsed OK; `bash -n deploy/install-agent.sh` → clean.
- `bash deploy/install-agent.sh --bind 127.0.0.1` (as uid 1000) → exit 1, prints `ops ALL=(root) NOPASSWD: /var/home/.../deploy/install-agent.sh`.
- `bash .agents/validate.sh` → rc 0: `785 passed, 11 skipped`, coverage **98.33 %** (gate 94 %), ruff clean, pyright 0 errors, `node --check` clean, hygiene clean.
**Docs**: no doc impact — README.md/AGENTS.md deliberately untouched (task 05 owns the dev-workflow rewrite). Noted in the NOTES file that AGENTS.md's "769 passed" is now 785 so the final pass refreshes it.
**Decisions/deviations**: grant names `ops` literally (only the path is substituted, so the printed line matches the canonical one); `--token` documented as the one flag never used (installer mints + prints once); added an honest note that the grant is only a boundary once the staged copy is root-owned. **No live-infrastructure changes** — the sudoers line needs root and is the phase's stated user precondition; nothing was touched on the fleet.
**Next pending task**: `10_dev_direct_dial/02_install_agents_on_fleet.md` (blocked on the user adding the `deploy/agent-sudoers` line as root on each host; task 02 must stop cleanly with instructions if it is missing).
.agents/reports/10_dev_direct_dial/10_dev_direct_dial__01_sudoers_grant_and_installer_hint.a1.md.status
+1
View File
@@ -0,0 +1 @@
ok
.agents/reports/10_dev_direct_dial/10_dev_direct_dial__01_sudoers_grant_and_installer_hint.a1.validate
+52
View File
@@ -0,0 +1,52 @@
== uv run pytest --cov (hermetic: no network, live tests skip) ==
........................................................................ [ 9%]
........................................................................ [ 18%]
........................................................................ [ 27%]
........................................................................ [ 36%]
........................................................................ [ 45%]
........................................................................ [ 54%]
...........................sssssssssss.................................. [ 63%]
........................................................................ [ 72%]
........................................................................ [ 81%]
........................................................................ [ 90%]
........................................................................ [ 99%]
.... [100%]
=============================== warnings summary ===============================
.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.14.4-final-0 ________________
Name Stmts Miss Cover Missing
------------------------------------------------------------------
src/dashboards/__init__.py 3 0 100%
src/dashboards/__main__.py 14 1 93% 28
src/dashboards/alerts.py 176 0 100%
src/dashboards/api.py 221 0 100%
src/dashboards/app.py 101 0 100%
src/dashboards/collectors.py 155 5 97% 141-142, 146-147, 164
src/dashboards/config.py 115 0 100%
src/dashboards/debug.py 14 0 100%
src/dashboards/poller.py 165 3 98% 192-194, 279
src/dashboards/prom.py 105 5 95% 107, 144, 158, 161, 164
src/dashboards/rates.py 37 1 97% 51
src/dashboards/severity.py 49 0 100%
src/dashboards/sources/__init__.py 0 0 100%
src/dashboards/sources/disks.py 130 0 100%
src/dashboards/sources/host.py 199 1 99% 147
src/dashboards/sources/llm.py 150 1 99% 228
src/dashboards/sources/nut.py 389 20 95% 128, 142, 148, 160, 166, 178, 186, 199, 270-271, 273, 489, 501, 536, 569, 573, 630-631, 636, 648
src/dashboards/sources/nut_sim.py 134 0 100%
src/dashboards/store.py 178 2 99% 250-251
------------------------------------------------------------------
TOTAL 2335 39 98%
Required test coverage of 94% reached. Total coverage: 98.33%
785 passed, 11 skipped, 1 warning in 58.73s
== uv run ruff check src tests agent ==
All checks passed!
== uv run pyright ==
0 errors, 0 warnings, 0 informations
== node --check src/dashboards/web/static/*.js ==
.agents/reports/10_dev_direct_dial/10_dev_direct_dial__02_install_agents_on_fleet.a1.err
View File
.agents/reports/10_dev_direct_dial/10_dev_direct_dial__02_install_agents_on_fleet.a1.md
+16
View File
@@ -0,0 +1,16 @@
**Implemented**
- Tunnels stopped; ad-hoc agents gone on all three hosts (`:9120` empty everywhere); `deploy/`+`agent/` staged at the canonical `~/agent-install/` on all three (task file's `rsync deploy/ agent/` trailing slashes would have flattened the tree — used `rsync -a --delete deploy agent`).
- **Fixed 2 defects that made `deploy/install-agent.sh` unable to install on any host**, found by the first live run: the install body was a bash function named `install`, so `install -d` recursed forever (1 700 repeats, nothing installed); and `DevicePolicy=strict` without `/dev/null` kills the unit at spawn (`status=208/STDIN`, systemd 257 — proven with two transient `systemd-run` probes). Plus a latent third: token file `0600 root:root` is unreadable by the service that opens the path itself → now `0640 root:dashboards-agent`.
- **gateway: INSTALLED** — `dashboards-agent` active + enabled, bound `10.1.1.27:9120`, `/health` ok, `/snapshot` 401 without / 200 with bearer; deployed unit matches the repo template; token captured into `.env.dev` (0600, gitignored), not in NOTES.
- **driveripper + deskwork: BLOCKED** — precondition unmet (`/etc/sudoers.d/dashboards-agent` absent, `sudo -n … --status` → "a password is required"). Stopped cleanly; exact `visudo`/`install` lines + the one `firewall-cmd` line per host are in `data/logs/NOTES-agent-install.md`. **No sudoers or firewall edit was attempted.**
- Laptop-side reach is blocked on **all three** hosts' firewalld (9120/tcp closed; control: gateway:80 answers 301) — recorded with both the bare-port and `/21`-scoped rich-rule commands, not guessed.
**Tests / lint / coverage**
- `uv run pytest` → **792 passed, 11 skipped**; `uv run pytest --cov --cov-fail-under=94` → **98.33%**; `uv run ruff check src tests agent` → passed; `uv run pyright` → 0 errors; `node --check` all page scripts ok; `bash -n` all scripts ok.
- New in `tests/test_deploy_scripts.py`: install body run end-to-end in a `tmp_path` sandbox (paths rewritten, root boundary stubbed) asserting it returns, lands the agent, writes `DeviceAllow=/dev/null`, prints the token once, leaves it `0640`; plus a static "no shell function named after a coreutil" check.
**Docs**: README (`0640 root:dashboards-agent`, device-policy rationale, `deploy/agent-sudoers` pointer); AGENTS.md (deployment paragraph, fleet SMART paragraph — gateway's installed agent now answers `smartctl_available: false` — and a transport mid-transition note pointing at NOTES; task 05 still owns the full direct-dial rewrite).
**Deviations / notes**
- `uv run pytest -m live --live` currently fails 6 — all transport (dead loopback tunnel ports + stopped `3494` forward), the state the phase expects for blocked hosts; green after the grant + task 04/06. Dev server is up and reports those sources red with real reasons.
- Next pending: **re-run task 02 for driveripper/deskwork after the user adds the sudoers grant**, then `.agents/phases/todo/10_dev_direct_dial/03_upsd_acl_for_laptop.md`.
.agents/reports/10_dev_direct_dial/10_dev_direct_dial__02_install_agents_on_fleet.a1.md.status
+1
View File
@@ -0,0 +1 @@
ok
.agents/reports/10_dev_direct_dial/10_dev_direct_dial__02_install_agents_on_fleet.a1.validate
+52
View File
@@ -0,0 +1,52 @@
== uv run pytest --cov (hermetic: no network, live tests skip) ==
........................................................................ [ 8%]
........................................................................ [ 17%]
........................................................................ [ 26%]
........................................................................ [ 35%]
........................................................................ [ 44%]
........................................................................ [ 53%]
..................................sssssssssss........................... [ 62%]
........................................................................ [ 71%]
........................................................................ [ 80%]
........................................................................ [ 89%]
........................................................................ [ 98%]
........... [100%]
=============================== warnings summary ===============================
.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.14.4-final-0 ________________
Name Stmts Miss Cover Missing
------------------------------------------------------------------
src/dashboards/__init__.py 3 0 100%
src/dashboards/__main__.py 14 1 93% 28
src/dashboards/alerts.py 176 0 100%
src/dashboards/api.py 221 0 100%
src/dashboards/app.py 101 0 100%
src/dashboards/collectors.py 155 5 97% 141-142, 146-147, 164
src/dashboards/config.py 115 0 100%
src/dashboards/debug.py 14 0 100%
src/dashboards/poller.py 165 3 98% 192-194, 279
src/dashboards/prom.py 105 5 95% 107, 144, 158, 161, 164
src/dashboards/rates.py 37 1 97% 51
src/dashboards/severity.py 49 0 100%
src/dashboards/sources/__init__.py 0 0 100%
src/dashboards/sources/disks.py 130 0 100%
src/dashboards/sources/host.py 199 1 99% 147
src/dashboards/sources/llm.py 150 1 99% 228
src/dashboards/sources/nut.py 389 20 95% 128, 142, 148, 160, 166, 178, 186, 199, 270-271, 273, 489, 501, 536, 569, 573, 630-631, 636, 648
src/dashboards/sources/nut_sim.py 134 0 100%
src/dashboards/store.py 178 2 99% 250-251
------------------------------------------------------------------
TOTAL 2335 39 98%
Required test coverage of 94% reached. Total coverage: 98.33%
792 passed, 11 skipped, 1 warning in 59.37s
== uv run ruff check src tests agent ==
All checks passed!
== uv run pyright ==
0 errors, 0 warnings, 0 informations
== node --check src/dashboards/web/static/*.js ==
.agents/reports/10_dev_direct_dial/10_dev_direct_dial__03_upsd_acl_for_laptop.a1.err
View File
.agents/reports/10_dev_direct_dial/10_dev_direct_dial__03_upsd_acl_for_laptop.a1.md
+13
View File
@@ -0,0 +1,13 @@
**Task 03 outcome: repo/records side complete and green; the prescribed live write was proven a no-op and the laptop leg is blocked off-host (details below).**
- Read-first done: `data/logs/upsd.conf.before` (369 B, `LISTEN 0.0.0.0/:: 3051`, no ACL lines); live file re-read + `diff` → **identical**; `upsd.users`/all `/etc/ups` mtimes unchanged; `nut-server` still `active`, `NRestarts=0`, `upsc … ups.status` = `OL` on both UPSes.
- Ranges derived: hosts `10.1.0.0/21` (`bridge0 10.1.0.126/21`), laptop `10.2.1.113` in `10.2.0.0/22` (ssh source + `ip -o addr`); v6 `…:101::/64` vs `…:102::/64`.
- **Did not append the two `ACL` lines — they enforce nothing on this upstream.** `nut-2.8.5-9.fc43` is built `--without-wrap`; `man 5 upsd.conf` has no ACL section; sandboxed `NUT_CONFPATH` runs (live service untouched) show upsd logging `ACL in upsd.conf is no longer supported - switch to LISTEN` per line and then starting anyway → a config that claims restriction, logs an error against itself on every restart, and changes no packet. On 2.8 the mechanism is `LISTEN` + firewall, both already correct here.
- Firewall (task step 5): `firewalld` running, `FedoraServer` on `bridge0/enp3s0`, **`3051/tcp` already allowed** (runtime + permanent), no rich/direct rules → nothing to add; the optional `--remove-port` + 2 rich-rule scoping is recorded verbatim but deliberately not applied (no reachable behaviour changes, deny path unverifiable from here, prod + `upsmon` dial loopback).
- **BLOCKER (operator, off-fleet):** from the laptop, `nc 10.1.0.126 3051` (v4 and v6) times out and **no SYN ever arrives** — proven by a 0.5 s `ss -tan sport = :3051` loop during 4 attempts (only pre-existing `[::1]` sessions) beside control probes where the same laptop's payload to `:1234` landed in an `ncat` log and `:8004` RST in 10 ms. The drop is the lab gateway `10.2.0.1` (`d8:b3:70:85:53:e6`, UniFi) between `10.2.0.0/22` and `10.1.0.0/21`: it forwards 22/1234/8000-8004, drops 1235/3051/3493/5901. No `ops` sudo on a fleet host touches it.
- Objective half already true: from `deskwork` (host range) `printf 'LIST UPS\n' | nc 10.1.0.126 3051` → `BEGIN LIST UPS … END LIST UPS`.
- All evidence, the operator's exact one rule (UniFi LAN→LAN allow 3051/tcp from `10.2.0.0/22` to `10.1.0.126`), the three interim options, and the two premise corrections are in `data/logs/NOTES-upsd-acl.md`. No Python changes, so no new tests were applicable.
- Gate: `bash .agents/validate.sh` → **exit 0** (`uv run pytest --cov --cov-fail-under=94` → **792 passed, 11 skipped, 98.33%**; `ruff check src tests agent` passed; `pyright` 0 errors; `node --check` ok; hygiene gates ok); `uv pip list --outdated` empty; dev server still up (sources red with real reasons — task 02's in-between state, untouched).
- Docs: **AGENTS.md** (NUT paragraph: 2.8 has no `ACL`, restriction is firewall-only, non-local reads are the norm, host range reaches 3051 / laptop range does not; transport bullet now points at `NOTES-upsd-acl.md`) and **README.md** (Battery-test section: `ACL`/`ACCEPT`/`REJECT` removed upstream, don't scope by client in `upsd.conf`).
- Completion criteria met: no `upsd.users` write, nut-server healthy/unrestarted, suite green, docs accurate. **Not met, and not achievable from this repo:** "diff shows exactly the two ACL lines" (would enforce nothing) and "laptop `nc` answers `BEGIN LIST UPS`" (gateway port filter). Task 04 should keep `NUT_HOST` pending that operator rule; the derived ranges are ready in the NOTES file.
- Next pending: operator rule on gateway `10.2.0.1` (+ task 02's still-missing sudoers grant on driveripper/deskwork), then `.agents/phases/todo/10_dev_direct_dial/04_env_dev_direct_dial.md`.
.agents/reports/10_dev_direct_dial/10_dev_direct_dial__03_upsd_acl_for_laptop.a1.md.status
+1
View File
@@ -0,0 +1 @@
ok
.agents/reports/10_dev_direct_dial/10_dev_direct_dial__03_upsd_acl_for_laptop.a1.validate
+52
View File
@@ -0,0 +1,52 @@
== uv run pytest --cov (hermetic: no network, live tests skip) ==
........................................................................ [ 8%]
........................................................................ [ 17%]
........................................................................ [ 26%]
........................................................................ [ 35%]
........................................................................ [ 44%]
........................................................................ [ 53%]
..................................sssssssssss........................... [ 62%]
........................................................................ [ 71%]
........................................................................ [ 80%]
........................................................................ [ 89%]
........................................................................ [ 98%]
........... [100%]
=============================== warnings summary ===============================
.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.14.4-final-0 ________________
Name Stmts Miss Cover Missing
------------------------------------------------------------------
src/dashboards/__init__.py 3 0 100%
src/dashboards/__main__.py 14 1 93% 28
src/dashboards/alerts.py 176 0 100%
src/dashboards/api.py 221 0 100%
src/dashboards/app.py 101 0 100%
src/dashboards/collectors.py 155 5 97% 141-142, 146-147, 164
src/dashboards/config.py 115 0 100%
src/dashboards/debug.py 14 0 100%
src/dashboards/poller.py 165 3 98% 192-194, 279
src/dashboards/prom.py 105 5 95% 107, 144, 158, 161, 164
src/dashboards/rates.py 37 1 97% 51
src/dashboards/severity.py 49 0 100%
src/dashboards/sources/__init__.py 0 0 100%
src/dashboards/sources/disks.py 130 0 100%
src/dashboards/sources/host.py 199 1 99% 147
src/dashboards/sources/llm.py 150 1 99% 228
src/dashboards/sources/nut.py 389 20 95% 128, 142, 148, 160, 166, 178, 186, 199, 270-271, 273, 489, 501, 536, 569, 573, 630-631, 636, 648
src/dashboards/sources/nut_sim.py 134 0 100%
src/dashboards/store.py 178 2 99% 250-251
------------------------------------------------------------------
TOTAL 2335 39 98%
Required test coverage of 94% reached. Total coverage: 98.33%
792 passed, 11 skipped, 1 warning in 59.63s
== uv run ruff check src tests agent ==
All checks passed!
== uv run pyright ==
0 errors, 0 warnings, 0 informations
== node --check src/dashboards/web/static/*.js ==
.agents/reports/10_dev_direct_dial/10_dev_direct_dial__04_env_dev_direct_dial.a1.err
View File
.agents/reports/10_dev_direct_dial/10_dev_direct_dial__04_env_dev_direct_dial.a1.md
+19
View File
@@ -0,0 +1,19 @@
**Implemented**
- `./scripts/dev-tunnels.sh stop` (no-op; already down), laptop has nothing on 9121/9122/9123/3494.
- `.env.dev` rewritten direct-shaped, mode `0600` kept: `HOST_TARGETS="driveripper=http://10.1.0.126:9120 deskwork=http://10.1.0.51:9120 gateway=http://<token>@10.1.1.27:9120"`, `NUT_HOST=10.1.0.126`/`NUT_PORT=3051`, `DASH_PORT=8090` explicit, no `DEV_TUNNELS`/`DEV_FORWARDS`, comments naming where the upsd access rules actually live and why a laptop-IP change is the operator's gateway rule, not this file.
- Verified the gateway credential against the live service without printing it (sha256 equal to `/etc/dashboards-agent.token`; agent answers 401 without bearer / 200 with it, host-side) and `NUT_PASSWORD` against `[dashboards]` in the live `upsd.users` (`instcmds = all` — corrected the file's stale "battery-test only" comment).
- Restarted the dev server; `/api/config` redacts userinfo, `/api/system/snapshot` keeps all three hosts as `unreachable` stubs (`hosts_total 3 / hosts_live 0`), `/api/summary` alerts name the 4 dead sources, `/api/events` shows only the 7 switchover `unreachable` transitions. Token-hits = 0 across all 7 API surfaces.
- Evidence + the 3 remaining operator actions: `data/logs/NOTES-env-dev-direct-dial.md` (+ correction appended to `NOTES-agent-install.md`).
**Tests / lint / coverage**
- `bash .agents/validate.sh` → exit 0: `uv run pytest --cov --cov-fail-under=94` **792 passed, 11 skipped, coverage 98.33%**; `ruff check src tests agent` clean; `pyright` 0 errors; `node --check` clean. `uv pip list --outdated` empty.
- `uv run pytest -m live --live` (with `.env.dev` sourced): 11 live tests, **5 pass (both engines), 6 fail on transport only** — 3 × `ConnectTimeout` to the `:9120`s, 3 × `cannot reach upsd at 10.1.0.126:3051`. Task 06 owns making these green.
**Docs** — README + AGENTS updated only where this change falsified them: tunnel-based dev quickstart → direct dial (`install-agent.sh` + userinfo tokens + `NUT_PORT=3051`); removed the now-false "`dev-tunnels.sh start` … temporary way to see live data" fallback; `DEV_FORWARDS="3494:…"` fleet fact → direct dial; NUT "bind upsd" advice no longer points at a forward; hermetic count 769 → 792. `git grep dev-tunnels|DEV_TUNNELS|DEV_FORWARDS|9121|9122|9123|3494` now matches neither file. Task 05's file list (`.env.dev.example`, `scripts/dev-tunnels.sh`, `capture-fixtures.sh`, live-suite strings, `ups_contract.py`, `alerts.py` docstring, `test_config.py`, `.env.example`) left untouched.
**Decisions / deviations**
- **No live changes to any host, firewall or service.** `driveripper`/`deskwork` stay BLOCKED (root-only sudoers grant missing — re-checked: `sudo -n … --status` still `a password is required`); gateway's `9120/tcp` stays closed. Opening it means a firewall edit on the lab router whose benefit is unverifiable from here (its policy is port-selective and drops 3051/3493 today, so 9120 may be dropped too), and tasks 02/03 both explicitly left firewall policy to the operator. No `NUT_SIMULATE`, no tunnel fallback — the reds are real, with the reason.
- Kept `driveripper`/`deskwork` in `HOST_TARGETS` **without** tokens (none exist) so the fleet still shows them as unreachable stubs instead of losing machines.
- Current live result, stated plainly: engines `ok`; every `system`/`disks`/`ups` source `DOWN` with a real reason — nothing that was healthy before the switchover regressed. `.env.dev.example` is still tunnel-shaped (task 05), so the README `cp` line currently points at a file awaiting that rewrite.
**Next pending task:** `.agents/phases/todo/10_dev_direct_dial/05_retire_tunnels_update_scripts.md`
.agents/reports/10_dev_direct_dial/10_dev_direct_dial__04_env_dev_direct_dial.a1.md.status
+1
View File
@@ -0,0 +1 @@
ok
.agents/reports/10_dev_direct_dial/10_dev_direct_dial__04_env_dev_direct_dial.a1.validate
+52
View File
@@ -0,0 +1,52 @@
== uv run pytest --cov (hermetic: no network, live tests skip) ==
........................................................................ [ 8%]
........................................................................ [ 17%]
........................................................................ [ 26%]
........................................................................ [ 35%]
........................................................................ [ 44%]
........................................................................ [ 53%]
..................................sssssssssss........................... [ 62%]
........................................................................ [ 71%]
........................................................................ [ 80%]
........................................................................ [ 89%]
........................................................................ [ 98%]
........... [100%]
=============================== warnings summary ===============================
.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.14.4-final-0 ________________
Name Stmts Miss Cover Missing
------------------------------------------------------------------
src/dashboards/__init__.py 3 0 100%
src/dashboards/__main__.py 14 1 93% 28
src/dashboards/alerts.py 176 0 100%
src/dashboards/api.py 221 0 100%
src/dashboards/app.py 101 0 100%
src/dashboards/collectors.py 155 5 97% 141-142, 146-147, 164
src/dashboards/config.py 115 0 100%
src/dashboards/debug.py 14 0 100%
src/dashboards/poller.py 165 3 98% 192-194, 279
src/dashboards/prom.py 105 5 95% 107, 144, 158, 161, 164
src/dashboards/rates.py 37 1 97% 51
src/dashboards/severity.py 49 0 100%
src/dashboards/sources/__init__.py 0 0 100%
src/dashboards/sources/disks.py 130 0 100%
src/dashboards/sources/host.py 199 1 99% 147
src/dashboards/sources/llm.py 150 1 99% 228
src/dashboards/sources/nut.py 389 20 95% 128, 142, 148, 160, 166, 178, 186, 199, 270-271, 273, 489, 501, 536, 569, 573, 630-631, 636, 648
src/dashboards/sources/nut_sim.py 134 0 100%
src/dashboards/store.py 178 2 99% 250-251
------------------------------------------------------------------
TOTAL 2335 39 98%
Required test coverage of 94% reached. Total coverage: 98.33%
792 passed, 11 skipped, 1 warning in 54.19s
== uv run ruff check src tests agent ==
All checks passed!
== uv run pyright ==
0 errors, 0 warnings, 0 informations
== node --check src/dashboards/web/static/*.js ==
.agents/reports/10_dev_direct_dial/10_dev_direct_dial__05_retire_tunnels_update_scripts.a1.err
View File
.agents/reports/10_dev_direct_dial/10_dev_direct_dial__05_retire_tunnels_update_scripts.a1.md
+17
View File
@@ -0,0 +1,17 @@
All green. Final report:
**Implemented**
- Deleted `scripts/dev-tunnels.sh` (working-tree `D`; harness stages) + removed its generated `data/tunnels/` leftovers.
- `scripts/capture-fixtures.sh`: sources `.env.dev` first, derives agents by parsing `HOST_TARGETS` (`name=url`, space/comma, userinfo token used as-is, `/snapshot` appended); skip note when unset; failure lines name the host, never the token; all tunnel hints gone.
- `tests/test_live_fleet.py` docstring → direct-dial loop; skips now `HOST_TARGETS is unset` / `no host agent answered`; `ups_contract.py`, `alerts.py` docstring, `test_config.py`, `test_alerts.py`, `test_api.py`, `test_disks_source.py` strings moved to the `:9120` direct shape; stale `tunnel-check` gone from `dev-server.sh` header.
- `.env.dev.example` rewritten direct-shaped (installer + `agent-sudoers` flow, userinfo tokens, `NUT_HOST`/`NUT_PORT=3051`, port-discovery + firewall notes, RFC 2606 hosts, never-commit warning); verified it parses through the app's own `parse_targets` (3 targets, headers set, redacted display URLs).
- New guards in `tests/test_deploy_scripts.py`: script stays deleted + no tunnel name/port in any operational file (scripts/deploy/src/agent/web/tests `.py`/README/AGENTS/example envs; fixtures & `.agents/` history excluded by design).
- **Deviation (rule 7 / task-03 findings):** `.env.example`'s "third option" is written as *bind LAN + firewall-scope the dashboard's source ranges*, not "ACL in `upsd.conf`" — NUT 2.8.5 has no `ACL` directive (logs "switch to LISTEN", enforces nothing); same truth README already carries.
**Verification** — `bash .agents/validate.sh` → **EXIT 0**: `uv run pytest --cov --cov-fail-under=94` **792 passed, 11 skipped, 98.33 %**; `ruff` clean; `pyright` 0 errors; `node --check` clean. `bash -n` clean on both edited scripts; `uv pip list --outdated` empty; dev server restarted (`/api/health` sane: engines ok, hosts `DOWN` with real reasons — unchanged task-04 state; no live infra touched).
**Docs** — AGENTS.md transport bullet rewritten ("dev is the prod topology, repo-wide", systemd-agent-via-grant + firewall-scoped upsd); README was already direct-shaped by task 04, verified consistent.
**Grep criterion** — matches remain only in `.agents/` history (incl. locked `PLAN.md:94` "3494" — flagged for the final pass, untouched; `todo/11_admin_login/00_phase.md` names the vars only to forbid restoring them), `tests/fixtures/` verbatim captures, and `uv.lock` hashes.
**Next pending task:** `.agents/phases/todo/10_dev_direct_dial/06_verify_direct_dial_live.md`.
.agents/reports/10_dev_direct_dial/10_dev_direct_dial__05_retire_tunnels_update_scripts.a1.md.status
+1
View File
@@ -0,0 +1 @@
ok
.agents/reports/10_dev_direct_dial/10_dev_direct_dial__05_retire_tunnels_update_scripts.a1.validate
+52
View File
@@ -0,0 +1,52 @@
== uv run pytest --cov (hermetic: no network, live tests skip) ==
........................................................................ [ 8%]
........................................................................ [ 17%]
........................................................................ [ 26%]
........................................................................ [ 35%]
........................................................................ [ 44%]
........................................................................ [ 53%]
..................................sssssssssss........................... [ 62%]
........................................................................ [ 71%]
........................................................................ [ 80%]
........................................................................ [ 89%]
........................................................................ [ 98%]
........... [100%]
=============================== warnings summary ===============================
.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.14.4-final-0 ________________
Name Stmts Miss Cover Missing
------------------------------------------------------------------
src/dashboards/__init__.py 3 0 100%
src/dashboards/__main__.py 14 1 93% 28
src/dashboards/alerts.py 176 0 100%
src/dashboards/api.py 221 0 100%
src/dashboards/app.py 101 0 100%
src/dashboards/collectors.py 155 5 97% 141-142, 146-147, 164
src/dashboards/config.py 115 0 100%
src/dashboards/debug.py 14 0 100%
src/dashboards/poller.py 165 3 98% 192-194, 279
src/dashboards/prom.py 105 5 95% 107, 144, 158, 161, 164
src/dashboards/rates.py 37 1 97% 51
src/dashboards/severity.py 49 0 100%
src/dashboards/sources/__init__.py 0 0 100%
src/dashboards/sources/disks.py 130 0 100%
src/dashboards/sources/host.py 199 1 99% 147
src/dashboards/sources/llm.py 150 1 99% 228
src/dashboards/sources/nut.py 389 20 95% 128, 142, 148, 160, 166, 178, 186, 199, 270-271, 273, 489, 501, 536, 569, 573, 630-631, 636, 648
src/dashboards/sources/nut_sim.py 134 0 100%
src/dashboards/store.py 178 2 99% 250-251
------------------------------------------------------------------
TOTAL 2335 39 98%
Required test coverage of 94% reached. Total coverage: 98.33%
792 passed, 11 skipped, 1 warning in 59.97s
== uv run ruff check src tests agent ==
All checks passed!
== uv run pyright ==
0 errors, 0 warnings, 0 informations
== node --check src/dashboards/web/static/*.js ==
.agents/reports/10_dev_direct_dial/10_dev_direct_dial__06_verify_direct_dial_live.a1.err
View File
.agents/reports/10_dev_direct_dial/10_dev_direct_dial__06_verify_direct_dial_live.a1.md
+14
View File
@@ -0,0 +1,14 @@
**Implemented (task 06 — verify direct-dial live)**
- Restarted the dev server on direct dial and recorded `/api/health`: `llm:gufo`/`llm:vllm` **ok** with live data; 3 `host:*` + 3 `disks:*` + `nut` DOWN with their real reasons (identical to the task-02/03/04 blockers); 0 `store-error`/`database is locked` lines; credential sweep proves the gateway bearer token appears in **no** API response.
- Proved no tunnel port exists: `ss -ltn | grep -E ':(9121|9122|9123|3494)\s'` → empty; `git grep` for the tunnel names/ports → nothing outside `.agents/` (only digit substrings in `uv.lock` hashes and one SMART raw counter).
- Prod-side proof (read-only, no writes): gateway's installed unit `active`, `10.1.1.27:9120` `/health` 200 / `/snapshot` 401-no-token / 200-with-token; that live snapshot through the **production** parsers passes `assert_card_contract` and yields the documented `/disks` degradation (`vda`, `smart: null`, `smartctl_available: false`); upsd answers `LIST UPS` for both UPSes from loopback **and from deskwork**.
- Browser verification (visible Chromium): `data/logs/phase10-{index,system,disks,llm,ups}-live.png` — `/llm` fully live (fleet 2/2, KV bars, slot grid, TTFT/TPOT, history charts); `/ups` honest "none reporting", **no SIMULATED badge**.
- Wrote `data/logs/NOTES-verify.md` (health table, live-suite tally, probe evidence localising the block to the lab gateway's port allow-list, the 3 operator actions, and the **PLAN.md L92–95 staleness flag** — "forwarded to `127.0.0.1:3494` in dev" is now false; file untouched).
**Tests/lint/coverage** — `bash .agents/validate.sh` → **EXIT 0**: `uv run pytest --cov --cov-fail-under=94` → **792 passed, 11 skipped, 98.33 %** (gate 94); `ruff check src tests agent` clean; `pyright` **0 errors**; `node --check` clean; `uv pip list --outdated` empty.
**Deviation / stopped short (live-infra)** — `uv run pytest -m live --live` is **4 passed, 1 skipped, 6 failed**; all six failures are connect timeouts to `:9120`/`:3051`, none a parse or contract. Making them green needs root on driveripper/deskwork (sudoers grant), host `9120/tcp`, and a rule on the lab gateway `10.2.0.1` — none reachable by `ops` sudo, and the laptop-side probes show the gateway drops even host-allowed ports (e.g. 3478), so opening gateway's `9120` has unknown payoff for at most 2 of 7 red sources. I therefore made **no live change** (no firewall, service, config write or UPS command) and left the reversible one-line experiment + exact operator commands in §7–§8 of the notes. `test_every_host_agent_answers_and_parses` is deliberately not turned into a skip — it is the reachability signal this phase exists to produce.
**Docs** — `AGENTS.md`: fleet bullet now also points at `data/logs/NOTES-verify.md` as the direct-dial verification record (re-verified facts unchanged). No README change needed (already direct-shaped; grep-clean).
**Next pending task** — `.agents/phases/todo/11_admin_login/01_auth_primitives.md` (phase 10's tasks are all complete).
.agents/reports/10_dev_direct_dial/10_dev_direct_dial__06_verify_direct_dial_live.a1.md.status
+1
View File
@@ -0,0 +1 @@
ok
.agents/reports/10_dev_direct_dial/10_dev_direct_dial__06_verify_direct_dial_live.a1.validate
+52
View File
@@ -0,0 +1,52 @@
== uv run pytest --cov (hermetic: no network, live tests skip) ==
........................................................................ [ 8%]
........................................................................ [ 17%]
........................................................................ [ 26%]
........................................................................ [ 35%]
........................................................................ [ 44%]
........................................................................ [ 53%]
..................................sssssssssss........................... [ 62%]
........................................................................ [ 71%]
........................................................................ [ 80%]
........................................................................ [ 89%]
........................................................................ [ 98%]
........... [100%]
=============================== warnings summary ===============================
.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.14.4-final-0 ________________
Name Stmts Miss Cover Missing
------------------------------------------------------------------
src/dashboards/__init__.py 3 0 100%
src/dashboards/__main__.py 14 1 93% 28
src/dashboards/alerts.py 176 0 100%
src/dashboards/api.py 221 0 100%
src/dashboards/app.py 101 0 100%
src/dashboards/collectors.py 155 5 97% 141-142, 146-147, 164
src/dashboards/config.py 115 0 100%
src/dashboards/debug.py 14 0 100%
src/dashboards/poller.py 165 3 98% 192-194, 279
src/dashboards/prom.py 105 5 95% 107, 144, 158, 161, 164
src/dashboards/rates.py 37 1 97% 51
src/dashboards/severity.py 49 0 100%
src/dashboards/sources/__init__.py 0 0 100%
src/dashboards/sources/disks.py 130 0 100%
src/dashboards/sources/host.py 199 1 99% 147
src/dashboards/sources/llm.py 150 1 99% 228
src/dashboards/sources/nut.py 389 20 95% 128, 142, 148, 160, 166, 178, 186, 199, 270-271, 273, 489, 501, 536, 569, 573, 630-631, 636, 648
src/dashboards/sources/nut_sim.py 134 0 100%
src/dashboards/store.py 178 2 99% 250-251
------------------------------------------------------------------
TOTAL 2335 39 98%
Required test coverage of 94% reached. Total coverage: 98.33%
792 passed, 11 skipped, 1 warning in 59.49s
== uv run ruff check src tests agent ==
All checks passed!
== uv run pyright ==
0 errors, 0 warnings, 0 informations
== node --check src/dashboards/web/static/*.js ==
.env.dev.example
+36 -28
View File
@@ -1,18 +1,23 @@
# Local development overrides. Copy to .env.dev and edit the host names to yours.
# Local development overrides. Copy to .env.dev and edit the addresses and tokens to yours.
# This file is gitignored and holds the fleet's bearer tokens: NEVER commit it.
# cp .env.dev.example .env.dev
# ./scripts/dev-tunnels.sh start # ssh tunnels to each host's agent
# set -a; . .env.dev; set +a # then run the dashboard
# chmod 0600 .env.dev
#
# DEV_TUNNELS entries are <local_port>:<ssh_host>:<remote_agent_port>.
# DEV_FORWARDS entries are <local_port>:<ssh_host>:<remote_port> for a service already
# running there — upsd is the usual one, so the UPS page can read a real battery without
# 3493 being reachable from anywhere but this laptop.
# scripts/dev-tunnels.sh forwards each one to 127.0.0.1:<local_port> so the dashboard on
# the laptop reads live fleet data with nothing listening on the hosts.
export DEV_TUNNELS="9121:driveripper-ops:9120 9122:deskwork-ops:9120 9123:gateway-ops:9120"
# The remote port is whatever upsd actually listens on — 3493 is NUT's stock default, and
# this fleet's driveripper answers on 3051, so check before copying the number.
export DEV_FORWARDS="3494:driveripper-ops:3051"
# Dev dials the fleet exactly the way the deployed dashboard does — no tunnels, no
# forwarded ports, nothing listening on localhost. Per monitored host, once:
#
# sudo deploy/install-agent.sh --bind <address the dashboard dials> --port 9120
#
# It installs agent/host_agent.py as the dashboards-agent systemd service, bound to the
# address you dial, and prints its bearer token exactly once. No root shell on the host?
# deploy/agent-sudoers is the one-line grant that lets the fleet's `ops` account run the
# installer (and running the installer without root prints that exact line). Put each
# printed token into HOST_TARGETS below as URL userinfo; config.parse_targets moves it
# into an Authorization header, so Target.url, /api/config and every error message stay
# credential-free.
#
# The host names below are placeholders (RFC 2606) — use your own.
export HOST_TARGETS="driveripper=http://TOKEN@driveripper.example.net:9120 deskwork=http://TOKEN@deskwork.example.net:9120 gateway=http://TOKEN@gateway.example.net:9120"
# Physical-disk health (phase 09): scripts/capture-fixtures.sh records one
# `smartctl -i -A -H` per whole disk per host into tests/fixtures/smartctl_<host>_<dev>.txt.
@@ -27,26 +32,29 @@ export DEV_FORWARDS="3494:driveripper-ops:3051"
# export DISKS_INTERVAL_SECONDS=60
# export DISKS_TIMEOUT_SECONDS=30
# Do not assume upsd is on 3493 — nonstandard ports are common and a refused connection then
# reads as "upsd is broken" while the service is fine. Ask the host what actually speaks NUT:
# The engines are dialled directly on their hostnames, like the agents.
export GUFO_BASE_URL=http://deskwork.example.net:8731
export VLLM_BASE_URL=http://driveripper.example.net:8000
# The UPS source: the real upsd, dialled directly. Do not assume NUT's stock 3493 —
# nonstandard ports are common and a refused connection then reads as "upsd is broken"
# while the service is fine (this fleet's driveripper answers on 3051). Ask the host what
# actually speaks NUT:
# ss -ltn # what is listening
# printf 'LIST UPS\n' | nc 127.0.0.1 PORT # the one that answers "BEGIN LIST UPS"
# upsc -L localhost:PORT # names + descriptions of every UPS
# Point the dashboard at those tunnels.
export HOST_TARGETS="driveripper=http://127.0.0.1:9121 deskwork=http://127.0.0.1:9122 gateway=http://127.0.0.1:9123"
# The engines are reached directly on their hostnames (nothing tunnels them here). If the
# laptop cannot route to them, add them to DEV_FORWARDS above
# (e.g. `8731:deskwork-ops:8731 8000:driveripper-ops:8000`) and point these at 127.0.0.1.
# The host names below are placeholders (RFC 2606) — use your own.
export GUFO_BASE_URL=http://deskwork.example.net:8731
export VLLM_BASE_URL=http://driveripper.example.net:8000
# The UPS source: the forwarded upsd above. NUT_PORT is the LOCAL port (3494), not 3493.
# Leave NUT_SIMULATE=1 only while upsd is unreachable — it labels every panel SIMULATED.
export NUT_HOST=127.0.0.1
export NUT_PORT=3494
# Who may connect is a FIREWALL question, not an upsd.conf one: NUT 2.8 has no `ACL`
# directive (it logs "switch to LISTEN" per line and enforces nothing), so the dashboard's
# source ranges are allowed in the firewall that covers the LISTEN address (a firewalld
# rule on the NUT host — not a line under /etc/ups). Reads are
# unauthenticated in stock NUT; SET/INSTCMD still need the upsd.users account below.
# A changed laptop IP can mean re-scoping that firewall rule — it is not a line in this
# file.
export NUT_HOST=driveripper.example.net
export NUT_PORT=3051
export NUT_USER=dashboards
export NUT_PASSWORD=replace-me-on-the-host
export NUT_ALLOW_ACTIONS=1
# Leave NUT_SIMULATE=1 only while upsd is unreachable — it labels every panel SIMULATED.
# export NUT_SIMULATE=1
export DASH_PORT=8090
.env.example
+12 -5
View File
@@ -52,16 +52,23 @@ RATE_WINDOW_SECONDS=10
# one looks exactly like a dead service:
# ss -ltnH # what the NUT host is listening on
# printf 'LIST UPS\n' | nc 127.0.0.1 PORT # NUT answers: BEGIN LIST UPS
# Keep it off the public interface: bind it to loopback (`LISTEN 127.0.0.1` in upsd.conf) and
# reach it with an ssh forward or a tailnet address, or bind a LAN address only if a firewall
# rule limits it to this host.
# Keep it off the public interface. Options, from most to least exposed:
# - bind loopback only (`LISTEN 127.0.0.1` in upsd.conf) and reach it over a tailnet address;
# - bind a LAN address and limit the dashboard's source range with a FIREWALL rule — this is
# the default here (upsd listens on the LAN, firewalld scopes 3051/tcp to the lab ranges).
# NUT 2.8 removed the `ACL`/`ACCEPT`/`REJECT` directives: on a build without TCP Wrappers
# an upsd.conf ACL line logs "ACL in upsd.conf is no longer supported - switch to LISTEN"
# once per line and changes no packet, so per-client restriction is never an upsd.conf job.
#
# Reads need no user in stock NUT. For the battery-test buttons add one, scoped as narrowly
# as the buttons need — `instcmds` beats `actions = -`, which grants load.off and shutdown.*
# too:
# too. One command per line: `man 5 upsd.users` says to grant several commands "use multiple
# instances of the instcmds field", and a list on one line is not that.
# [dashboards]
# password = "..."
# instcmds = test.battery.start.quick test.battery.start.deep test.battery.stop
# instcmds = test.battery.start.quick
# instcmds = test.battery.start.deep
# instcmds = test.battery.stop
# --------------------------------------------------------------------------- #
NUT_HOST=
NUT_PORT=3493
AGENTS.md
+75 -20
View File
@@ -25,7 +25,7 @@ file — plus an overview page that says which of the four you need:
| Route | Question it answers | Source |
| --------- | ------------------------------------------------------------------------------ | ------------------------------------------------- |
| `/` | Is anything wrong, and where do I look first? | all four, in one request |
| `/system` | What are my machines doing — CPU, memory, disk, GPU, network, and the history | `agent/host_agent.py`, our own stdlib agent/host |
| `/system` | What are my machines doing — CPU, memory, disk, GPU, network, and the history | `agent/host_agent.py`, our own stdlib agent per host |
| `/disks` | Are any of my physical disks failing, and which one — health, temp, hours, wear | the same agents, `smartctl -i -A -H` per whole drive (never a partition) |
| `/llm` | Tokens/sec, active sessions, queue depth, KV pressure, latency percentiles | gufo `/metrics` + `/slots`, vLLM `/metrics` |
| `/ups` | Battery health, runtime left, power events, battery-test buttons | NUT `upsd` (our own wire-protocol client; 3051 here, not 3493) |
@@ -311,14 +311,55 @@ That scheme is the history API's key — treat it as a public interface.
| `deskwork` | `deskwork-ops` | Strix Halo (Radeon 8060S iGPU), gufo `:8731` |
| `gateway` | `gateway-ops` | router, no GPU |
- Agents run from `~/lab-agent/host_agent.py` on each host, forwarded to `127.0.0.1:9121/9122/9123`
by `scripts/dev-tunnels.sh` (`DEV_TUNNELS`). Nothing listens on the hosts in dev.
- **Transport is direct dial: dev is the prod topology, repo-wide (phase 10 task 05).** The
designed shape is one `dashboards-agent` **systemd service per host**, installed by `ops`
through the `deploy/agent-sudoers` grant with `deploy/install-agent.sh`, bound to
the address the dashboard dials, its bearer token carried in the `HOST_TARGETS` URL
userinfo; driveripper's upsd serves the LAN on 3051 with the host and laptop ranges allowed
by the **firewall**, never by a `upsd.conf` line. The old ssh-tunnel script is deleted and
neither `.env.dev` nor `.env.dev.example` declares a forwarded port — dev dials the same
addresses the deployed dashboard does, and `scripts/capture-fixtures.sh` reads
`HOST_TARGETS` from `.env.dev` instead of naming hosts of its own
(`tests/test_deploy_scripts.py` fails the day any tunnel name or port returns). Current state: `gateway` runs its installed service, bound to the address
`gateway.reeselink.com` resolves to, and answers `/health` + `/snapshot` **on the host**
(re-verified 2026-10-18, and the token in `.env.dev` matches `/etc/dashboards-agent.token`),
but the laptop cannot reach its `:9120` — and that is now proven to be nobody's firewall here:
opening `9120/tcp` in that host's own firewalld (a runtime rule, added and reverted in the
phase-10 final pass; runtime and permanent port lists are back to their original set) changed
nothing, because the SYN never arrives — `ss -tan sport = :9120` on the host shows only the
`LISTEN` — while the same laptop reaches `:22` and `:80` on the same address in milliseconds.
The drop is the port-selective inter-VLAN filter on the lab gateway `10.2.0.1`, the same one
that eats 3051/3493. On `driveripper` and `deskwork` the ad-hoc `~/lab-agent/host_agent.py` agents
are gone (stopped with the old tunnel sessions, nothing replaced them, `:9120` is empty on
both) because their `deploy/agent-sudoers` grant is not installed yet, so the installer
cannot run there. Every `system`/`disks` source is therefore `DOWN` with its real reason
and renders as an `unreachable` stub row — the designed degradation, recorded rather than
hidden — which is also why 6 of the `live` tests fail on transport while both engines pass.
The per-host record, the exact root commands and the one firewall line each host still
needs are in `data/logs/NOTES-agent-install.md`, how far the NUT wire reaches from each
network is in `data/logs/NOTES-upsd-acl.md`, and the end-to-end direct-dial verification
(health table, no-tunnel-port proof, live-suite tally, screenshots, and the probes that put
the block on the lab gateway's port allow-list rather than in this app) is in
`data/logs/NOTES-verify.md`.
- Engines are reached directly: gufo `deskwork.reeselink.com:8731`, vLLM `driveripper.reeselink.com:8000`.
- NUT 2.8.5 on driveripper: **configs live in `/etc/ups/`, not `/etc/nut/`**, and **upsd is
on port 3051, not 3493**. Nothing listens on 3493 at all, so a default client reports
*Connection refused* and it looks like a dead service — `ss -ltnH` plus `printf 'LIST UPS\n'
| nc 127.0.0.1 3051` (answers `BEGIN LIST UPS`) is how that was pinned down. Two UPSes are
exported over USB and read live through `DEV_FORWARDS="3494:driveripper-ops:3051"`:
| nc 127.0.0.1 3051` (answers `BEGIN LIST UPS`) is how that was pinned down.
**NUT 2.8 has no `ACL` directive** (this Fedora build is `--without-wrap`; upsd logs
`ACL in upsd.conf is no longer supported - switch to LISTEN` once per line and then ignores
it), so per-source restriction is a **firewall** rule and never an `upsd.conf` line — and
2.8 does not refuse non-local clients the way older docs suggest: `LISTEN 0.0.0.0 3051` +
`LISTEN :: 3051` with `3051/tcp` in the `FedoraServer` zone answers any client routed to it
(reads are unauthenticated; `SET`/`INSTCMD` still need an `upsd.users` account). The host
range `10.1.0.0/21` does reach it today (verified from `deskwork`); the laptop's
`10.2.0.0/22` does **not** — the lab gateway `10.2.0.1` forwards 22/1234/8000-8004 and
silently drops 3051/3493 in both families, which no sudo on a fleet host can change, so the
laptop cannot yet dial `NUT_PORT=3051` (probe evidence and the operator's one rule:
`data/logs/NOTES-upsd-acl.md`).
Two UPSes are exported over USB and dialled directly as `NUT_HOST` + `NUT_PORT=3051` (from the
laptop that is still the blocked path above; from the host itself, and from the deployed
dashboard, it reads live):
`apc` = APC Back-UPS RS 1500MS, `cyberpower` = CyberPower PR1500LCDRT2U (that is
`ups.description`, the operator's label — the driver's own `identity.model` answers
`GX1500U`, so the description is what identifies the box).
@@ -331,19 +372,28 @@ That scheme is the history API's key — treat it as a public interface.
Both UPSes are recorded verbatim (`LIST UPS`/`LIST VAR`/`LIST CMD`, serials scrubbed) as
`tests/fixtures/nut_<ups>.vars` by `./scripts/capture-fixtures.sh`, and the wire quotes a
field only when it must — `VAR apc ups.status "OL"`, not three quoted fields.
- Host access is via ssh keys as `ops`. Passwordless sudo exists but is command-scoped; the
set we rely on is documented in `deploy/nut-sudoers` (additions are listed there for the
work that is still blocked).
- Host access is via ssh keys as `ops`, and **passwordless sudo is different on every host** —
that is what decides how much of the fleet an install can finish unattended. `gateway` grants
`(ALL) NOPASSWD: ALL`, which is why its agent installed in phase 10 with no
`deploy/agent-sudoers` line present; `driveripper` grants a command-scoped set (`smartctl`,
`upsc`, `udevadm`, the `/etc/ups` reads, and the `tee /etc/ups/upsd.users|upsd.conf` +
`systemctl restart|reload nut-server` pair documented in `deploy/nut-sudoers`); `deskwork`
grants an unrelated set (`btrfs scrub`, `snapper`, `setfacl /srv/*`, `osbuild-auto.sh`).
Neither `driveripper` nor `deskwork` can run the installer or `visudo`, so those two installs
— and the `firewall-cmd` line each still needs — stay with the operator.
- **SMART is captured from real hardware; the live agents still cannot read it.** `smartctl`
7.5 is installed on deskwork and driveripper. The `ops` account (which runs the dev agents)
is in no `disk` group; driveripper grants it `(root) NOPASSWD: /usr/sbin/smartctl*` and
deskwork grants nothing, so the agent's own unprivileged `smartctl -i -A -H` — it shells the
binary and never calls sudo — answers *Permission denied* (exit 2) on every host and `smart`
stays `null` end to end: the designed degradation, not a bug. The three deployed agents also
predate the SMART pass, so the `disks` source today yields one row per physical disk (16 on
driveripper, 1 on deskwork, `vda` on gateway) with `smart: null`, `smart_errors: []` and
`smartctl_available: null` — "nobody told us either way", which is why that flag is
three-way and not a boolean.
stays `null` end to end: the designed degradation, not a bug. The agents deployed on
driveripper and deskwork also predate the SMART pass, so the `disks` source today yields one row
per physical disk (16 on driveripper, 1 on deskwork, `vda` on gateway) with `smart: null`,
`smart_errors: []` and `smartctl_available: null` — "nobody told us either way", which is why
that flag is three-way and not a boolean. `gateway`'s agent is the one exception as of phase 10:
it is the installed systemd service, which does know about the SMART pass and answers
`smartctl_available: false` there because smartmontools is not installed — the *install it*
half of the three-way flag, from a host the dashboard cannot yet reach (its firewall).
**Real data is in the repo now:** `tests/fixtures/smartctl_driveripper_<dev>.txt` is all 16
drives (8 NVMe, 8 spinning) read through that sudo rule, and `tests/test_agent.py` parses
every one — which is what found the parser's one real bug: this binary annotates the
@@ -376,14 +426,14 @@ That scheme is the history API's key — treat it as a public interface.
```bash
uv sync --python 3.14 # deps + editable install
cp .env.example .env # production-shaped config
cp .env.dev.example .env.dev # dev-only: ssh tunnels + live targets
cp .env.dev.example .env.dev # dev-only: the real hosts, tokens in URL userinfo
./scripts/dev-tunnels.sh start # agents (9121-9123) + DEV_FORWARDS (upsd 3494)
set -a; . ./.env.dev; set +a # dev dials the fleet the prod way, no tunnels
./scripts/dev-server.sh restart # keep http://127.0.0.1:8090 up
./scripts/dev-server.sh status # per-source ok/DOWN
./scripts/dev-server.sh logs
uv run pytest # hermetic: 769 passed + 11 live-skipped, no network
uv run pytest # hermetic: 793 passed + 11 live-skipped, no network
uv run pytest --cov # coverage gate (98.33%; floor 94%)
uv run pytest -m live --live # opt-in: real fleet / engines / upsd
uv run ruff check src tests agent && uv run pyright
@@ -402,8 +452,13 @@ bash .agents/validate.sh # the phase gate: the four checks abov
Deployment: `Containerfile` (two-stage, `uv sync --frozen`, runs as uid 10001 on `0.0.0.0:8090`,
SQLite under `/var/lib/dashboards` — put a volume there) and `deploy/install-agent.sh`
per monitored host (systemd unit, own user, `--bind`/`--port`/`--token`/`--rotate-token`/
`--status`/`--uninstall`; it stores the token in a root-owned 0600 file and hands the agent
`--token-file`, never argv, and runs under `ProtectSystem=strict`).
`--status`/`--uninstall`; it stores the token `0640 root:dashboards-agent` — the unit hands the
agent the *path* and the agent opens it as its own user, so that user must be able to read it and
no other account may — never puts a token on argv, and runs under `ProtectSystem=strict` with
`DevicePolicy=strict` reopening only the disk nodes plus the `/dev/null` family, because strict
without `/dev/null` kills the unit at spawn (`status=208/STDIN`). No root shell? The one sudoers
line that lets `ops` run the installer is `deploy/agent-sudoers`; run the script without root and
it prints that line with its own resolved path.)
The quadlet unit + env template + nginx/SNI live in the deployment repo, like every other
service here. The source itself is `git@git.reeseapps.com:services/dashboards.git` (`origin`,
branch `main`); there is no CI on it, so `bash .agents/validate.sh` is the only gate.
@@ -823,8 +878,8 @@ memory plot; the `cpu.usage_pct@coreN` series is still stored and still queryabl
field map is written.
3. **UPS history depth.** Events and the numeric series are stored; long-window battery
health (capacity fade over months) has the rollups but no dedicated view.
4. **The dev DB is big enough to refuse a write now and then.** The dev store (~590 MB, WAL
grown to ~70 MB) logs `store-error: OperationalError: database is locked` a few times an
4. **The dev DB is big enough to refuse a write now and then.** The dev store (~610 MB, WAL
grown past 140 MB) logs `store-error: OperationalError: database is locked` a few times an
hour across `system`/`llm`/`ups`; the poller handles it exactly as invariant 4 requires —
the source goes red with the real reason, ages, and writes `recording` when the next poll
lands — so nothing freezes, but those polls' samples are lost. The one cause this repo can
README.md
+30 -20
View File
@@ -27,29 +27,26 @@ uv run uvicorn dashboards.app:app --port 8090 # equivalent
uv run pytest # unit tests, no network needed
```
Developing against the real fleet from a laptop (agents stay unexposed — ssh forwards
them to localhost):
Developing against the real fleet from a laptop — dev dials exactly the addresses the deployed
dashboard dials, so there is nothing to tunnel and no port to forward:
```bash
cp .env.dev.example .env.dev # your ssh aliases + which ports
python3 - <<'PY' # push the agent to each host (once)
import shlex, subprocess, pathlib
for host in ("driveripper-ops", "deskwork-ops", "gateway-ops"):
subprocess.run(["ssh", host, "mkdir -p ~/lab-agent"], check=True)
subprocess.run(["scp", "agent/host_agent.py", f"{host}:~/lab-agent/"], check=True)
PY
./scripts/dev-tunnels.sh start # 9121/9122/9123 -> each host's agent,
# + DEV_FORWARDS -> upsd on 127.0.0.1:3494
# (the remote port is upsd's real one — 3051
# on driveripper, not the 3493 default)
cp .env.dev.example .env.dev # your host addresses + agent tokens (chmod 0600)
sudo deploy/install-agent.sh --bind <addr> --port 9120
# once per monitored host: the agent as a systemd
# service, printing its bearer token exactly once
set -a; . ./.env.dev; set +a
uv run dashboards # foreground
./scripts/dev-server.sh restart # or backgrounded, with a pidfile + logs
```
`DEV_TUNNELS` starts the agent remotely and forwards it; `DEV_FORWARDS` forwards a service
that already listens there (upsd) — that is how the UPS page reads a real battery without
3493 being reachable from anywhere but this laptop.
Each `HOST_TARGETS` entry carries that token as URL userinfo
(`driveripper=http://<token>@<addr>:9120`); `parse_targets` moves it into an `Authorization`
header, so `Target.url`, `/api/config` and every error message stay credential-free. No root
shell on a host? `deploy/agent-sudoers` is the one-line grant that lets `ops` run the installer,
and running the installer without root prints that exact line. upsd is dialled at
`NUT_HOST`/`NUT_PORT` on the port it really answers on (3051 on driveripper, not the stock
3493) — see the NUT section below for who may connect.
## The host agent
@@ -68,6 +65,10 @@ deploy/install-agent.sh --status # unit state and /he
sudo deploy/install-agent.sh --uninstall
```
No root shell on the monitored host? `deploy/agent-sudoers` is the single sudoers line that
lets the fleet's unprivileged `ops` account run the installer; run it without root and it
prints that exact line with the script's own resolved path instead of saying "run as root".
It is read-only by construction: there is no code path that writes, and the HTTP server
only answers `GET`. Counters are reported as counters — the dashboard computes rates over
a labelled window, so the agent never has to remember who polled it last.
@@ -92,8 +93,11 @@ Binding off-loopback without a token is refused (exit 2) rather than warned abou
`--token-file` over `--token`: argv is world-readable through `ps`, and this token guards
hostname, mounts, disk and GPU models, and process names. `deploy/install-agent.sh` creates
a system user, generates a token once (re-runs reuse it; `--rotate-token` for a new one),
stores it `0600` root-owned, and runs the agent under `ProtectSystem=strict` with no
capabilities — enough to read `/proc`, `/sys` and `statvfs()` and nothing else.
stores it `0640 root:dashboards-agent` — the unit hands the agent the *path*, and the agent
opens it as its own user, so the file must be readable by that user and by nobody else —
and runs the agent under `ProtectSystem=strict`, `DevicePolicy=strict` with only the disk
nodes and the `/dev/null` family reopened (SMART needs the former; without `/dev/null` in
the allow list systemd cannot start the service at all), with no capabilities.
## Configuration
@@ -400,8 +404,14 @@ files: `sudo install -m 0440 -o root -g root deploy/nut-sudoers /etc/sudoers.d/d
Read `upsd.users` before you write it — `upsmon` authenticates from the same file, so
overwriting it blind can take the box down at the wrong moment.
Bind upsd to loopback (`LISTEN 127.0.0.1` in `upsd.conf`) and reach it with an ssh forward
(`DEV_FORWARDS`) or a tailnet address. `NUT_SIMULATE=1` stands in for upsd during
Bind upsd to loopback (`LISTEN 127.0.0.1 3051` in `upsd.conf`) when the dashboard runs on the
same box, and to a real interface plus a source-scoped firewall rule when it runs elsewhere —
and do not try to scope it by client inside `upsd.conf` instead: **NUT 2.8
removed the `ACL`/`ACCEPT`/`REJECT` directives** (a build without TCP Wrappers logs
`ACL in upsd.conf is no longer supported - switch to LISTEN` once per line and ignores the
line), so `LISTEN` plus a host firewall rule is the only access control `upsd.conf` offers,
and an unauthenticated `LIST VAR` from any routed source is the normal state of a NUT server.
`NUT_SIMULATE=1` stands in for upsd during
development; everything it produces is badged `SIMULATED`, and it is never the default.
## Deployment
deploy/agent-sudoers
+76
View File
@@ -0,0 +1,76 @@
# sudoers grant that lets the unprivileged `ops` account install the homelab dashboards
# host agent on a monitored host. One line, one script path — deploy/nut-sudoers is the
# separate NUT grant on driveripper.
#
# Install (from the staged copy on the host, as root — the command-scoped `ops` sudo on
# these hosts does NOT include visudo, which is exactly why this file exists):
#
# sudo visudo -cf /home/ops/agent-install/deploy/agent-sudoers # syntax check FIRST
# sudo install -m 0440 -o root -g root /home/ops/agent-install/deploy/agent-sudoers \
# /etc/sudoers.d/dashboards-agent
# sudo visudo -cf /etc/sudoers.d/dashboards-agent # and check the installed copy
#
# Scope: an absolute path (no wildcards, `secure_path` is in effect), no shell, no editor,
# no `tee`, nothing that can amend this file. Running install-agent.sh without root prints
# this exact grant with the script's own resolved path, so a checkout at another path tells
# you what to add instead of saying "run as root" to someone who cannot.
#
# WHY THIS IS NEEDED
# The agent is read-only by construction: it answers GET /snapshot on a socket, writes
# nothing anywhere, and every request except /health needs the bearer token it reads from a
# file only root and the agent's own service user can read (0640 root:dashboards-agent — the
# agent opens the path itself, so the service user must be in that group). What needs root is
# the *install*, not the agent: creating the
# dashboards-agent system user, writing /usr/local/lib/dashboards-agent/host_agent.py,
# dropping /etc/systemd/system/dashboards-agent.service, storing the token, and running
# systemctl daemon-reload + enable --now (disable --now for --uninstall).
# Phase 10 runs each host's agent as that systemd service bound to the address the
# dashboard dials directly, so the account that deploys over ssh — ops — has to be able to
# run the installer, and that is the only reason it is granted anything.
#
# WHY THE WHOLE SCRIPT PATH AND NOT A FLAG LIST
# The flags carry no secrets: --bind/--port choose the socket, --status is read-only,
# --uninstall removes exactly what this install put in place, --rotate-token mints a token
# the operator is about to be shown anyway. sudoers can only match flags with a wildcard,
# and a wildcard list has to be edited in step with every flag the script gains — an edit
# whose failure mode is a broken install, not a smaller privilege. The one flag whose
# *argument* would be sensitive is --token, and no documented flow uses it: the installer
# generates the token, prints it once, and hands the agent --token-file, because argv is
# world-readable through ps while the file is not.
# ---- the grant ------------------------------------------------------------------ #
# The host has no repo checkout, so deploy/ and agent/ are staged to ~/agent-install/
# (phase 10 task 02 does the copy), which keeps this path identical on every host:
#
# ssh ops@host 'mkdir -p ~/agent-install'
# scp -r deploy agent ops@host:~/agent-install/
#
# `ops` is the fleet's monitored/deploy account; change only that field if the agent is
# installed by a different account. Nothing follows the path, which in sudoers means the
# script may be run with any of its own flags — see the block above.
ops ALL=(root) NOPASSWD: /home/ops/agent-install/deploy/install-agent.sh
# ---- making the grant mean something (one root command, recommended) ------------ #
# A sudo grant on a path is only as strong as who can write that path. ops owns
# ~/agent-install, so until the staged copy is root-owned this line is a convenience for a
# human, not a boundary. After staging, as root:
#
# sudo chown -R root:root /home/ops/agent-install
#
# ops can still wipe its own home directory and re-copy (it is ops' home), so this guards
# against drift and a stray edit — not against an operator who wants something else.
# ---- deliberately NOT granted --------------------------------------------------- #
# visudo, vim, nano, sh, bash, tee — no self-amendment of this grant, no shell, and no
# writing any file as root
# systemctl (anything else) — the installer already runs the few unit operations
# it needs as itself; a bare systemctl * could stop
# anything on the host
# cat /etc/dashboards-agent.token — the token is readable by root and the agent user
# only; install-agent.sh prints it once on first
# install and --rotate-token mints a new one
# useradd, usermod, groupmod — the installer creates its own user and its disk
# group membership; a bare usermod could add ops to
# any group on the box
# upscmd, upsrw — NUT instant commands stay the dashboard's gated
# POST /api/ups/command; see deploy/nut-sudoers
deploy/install-agent.sh
+72 -16
View File
@@ -1,15 +1,20 @@
#!/usr/bin/env bash
# Install agent/host_agent.py as a systemd service on a monitored host.
#
# sudo ./deploy/install-agent.sh # loopback only (ssh tunnel in)
# sudo ./deploy/install-agent.sh # loopback only (read it locally)
# sudo ./deploy/install-agent.sh --bind 10.0.0.5 # serve the LAN, token required
# sudo ./deploy/install-agent.sh --status
# ./deploy/install-agent.sh --status # unit state + /health, no root
# sudo ./deploy/install-agent.sh --uninstall
#
# The token is stored in a root-owned 0600 file and passed to the agent as --token-file,
# never on the command line: argv is world-readable through `ps`, and this token guards
# hostname, mounts, disk models, GPU model and process names. Re-running the installer
# reuses the existing token — use --rotate-token when the old one leaked.
# No root shell? `deploy/agent-sudoers` is the single sudoers line that lets the fleet's
# unprivileged `ops` account run this script as root. Run it without root and it prints that
# exact line with this script's resolved path, then exits 1.
#
# The token is stored in a file only root and the agent's own service user can read, and is
# passed to the agent as --token-file, never on the command line: argv is world-readable
# through `ps`, and this token guards hostname, mounts, disk models, GPU model and process
# names. Re-running the installer reuses the existing token — use --rotate-token when the old
# one leaked.
#
# The agent is read-only by construction (it answers GET and writes nothing), runs as its
# own system user, and gets the usual systemd lockdown. It still needs to read /proc, /sys
@@ -32,6 +37,15 @@ TOKEN_FILE="/etc/dashboards-agent.token"
UNIT="/etc/systemd/system/$SERVICE.service"
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
AGENT_SOURCE="$SCRIPT_DIR/../agent/host_agent.py"
# Absolute, symlink-resolved path of this script: the sudoers grant in deploy/agent-sudoers
# is matched on a path, so the non-root hint below prints the line that actually matches
# wherever the script happens to live (the canonical staging path is
# /home/ops/agent-install/deploy/install-agent.sh).
SCRIPT_PATH="$SCRIPT_DIR/$(basename "$0")"
if command -v realpath >/dev/null 2>&1; then
SCRIPT_PATH="$(realpath "$0" 2>/dev/null || printf '%s' "$SCRIPT_PATH")"
fi
SUDOERS_SOURCE="$(dirname "$SCRIPT_PATH")/agent-sudoers"
BIND="127.0.0.1"
PORT="9120"
@@ -60,7 +74,30 @@ done
need_root() {
if [ "$(id -u)" -ne 0 ]; then
echo "run as root: sudo $0 $*" >&2
# The dev workflow deploys over ssh as the unprivileged `ops` account, so "run as root"
# is not an action the operator can take. Print the one sudoers line that makes this
# script installable — the same line as deploy/agent-sudoers, with this script's own
# resolved path — plus the two-stage install it documents, then exit 1 as before.
cat >&2 <<HINT
$SCRIPT_PATH has to run as root: it writes $UNIT,
creates the $AGENT_USER user and its disk-group membership, and stores
the token in $TOKEN_FILE.
Add this single sudoers line on this host (it is the whole grant; see $SUDOERS_SOURCE):
ops ALL=(root) NOPASSWD: $SCRIPT_PATH
Install it the two-stage way every sudoers change here uses, as root — the command-scoped
ops sudo on these hosts does not include visudo, so this step is the operator's:
sudo visudo -cf $SUDOERS_SOURCE
sudo install -m 0440 -o root -g root $SUDOERS_SOURCE /etc/sudoers.d/dashboards-agent
sudo visudo -cf /etc/sudoers.d/dashboards-agent
Then run this same command again:
sudo $SCRIPT_PATH $*
HINT
exit 1
fi
}
@@ -76,7 +113,7 @@ status() {
}
uninstall() {
need_root "$0 --uninstall"
need_root --uninstall
systemctl disable --now "$SERVICE" 2>/dev/null || true
rm -f "$UNIT" "$TOKEN_FILE" "$AGENT_TARGET"
rmdir "$AGENT_DIR" 2>/dev/null || true
@@ -90,8 +127,12 @@ new_token() {
python3 -c 'import secrets; print(secrets.token_urlsafe(32).rstrip("="))'
}
install() {
need_root "$0 --bind $BIND --port $PORT"
# NOT named `install`: bash resolves a function before an external command of the same name,
# so an `install()` here turns the `install -d` / `install -m` calls below into infinite
# recursion instead of creating $AGENT_DIR. Phase 10 task 02 found exactly that on a live
# host: the run looped on the SMART block forever and installed nothing.
do_install() {
need_root --bind "$BIND" --port "$PORT"
[ -f "$AGENT_SOURCE" ] || { echo "no agent next to this script: $AGENT_SOURCE" >&2; exit 1; }
if ! command -v python3 >/dev/null 2>&1; then
echo "python3 is required on the monitored host (3.10 or newer)" >&2
@@ -127,11 +168,14 @@ install() {
elif [ -z "$TOKEN" ]; then
TOKEN="$(new_token)"
fi
# 0600 root:root — the service runs as $AGENT_USER, and systemd (PID 1, root) reads the
# file for it, so the agent process never needs read permission on it itself.
# 0640 root:$AGENT_USER. The unit passes `--token-file` as a *path*: systemd does not read
# the secret for the service, the agent process (running as $AGENT_USER) opens it itself,
# so a 0600 root-owned file makes the agent exit "cannot read --token-file" and the unit
# never starts. Group ownership is the smallest grant that lets the service read it, and
# every other account — ops included — still cannot.
(umask 0177; printf '%s\n' "$TOKEN" > "$TOKEN_FILE")
chmod 0600 "$TOKEN_FILE"
chown root:root "$TOKEN_FILE"
chmod 0640 "$TOKEN_FILE"
chown root:"$AGENT_USER" "$TOKEN_FILE"
cat > "$UNIT" <<UNIT_FILE
[Unit]
@@ -151,10 +195,22 @@ SyslogIdentifier=$SERVICE
# be taken away. smartctl is the one thing that needs a real /dev node, so instead of
# PrivateDevices the unit closes all devices and reopens only the disk nodes, read/write
# at the cgroup layer while the code (and smartctl -i -A -H) stays read-only by design.
#
# The pseudo nodes are not decoration: DevicePolicy=strict denies every node that is not
# listed, and systemd needs /dev/null for the service's stdin before it ever execs python —
# without it the unit dies at spawn with "Failed at step STDIN" (status 208), which is how
# the first phase-10 install failed on a live host running systemd 257. /dev/zero, full,
# random and urandom are the rest of the set a PrivateDevices= sandbox would have provided,
# and none of them is a disk.
NoNewPrivileges=true
PrivateTmp=true
PrivateDevices=false
DevicePolicy=strict
DeviceAllow=/dev/null rw
DeviceAllow=/dev/zero rw
DeviceAllow=/dev/full rw
DeviceAllow=/dev/random rw
DeviceAllow=/dev/urandom rw
DeviceAllow=/dev/nvme* rw
DeviceAllow=/dev/sd* rw
DeviceAllow=/dev/vd* rw
@@ -194,14 +250,14 @@ its own, and the API strips it before anything reaches a browser):
HOST_TARGETS="hostname=http://$TOKEN@$BIND:$PORT"
Paste the token now if this is the first install — it is not printed again without
--rotate-token, and it is only ever readable by root on this box:
--rotate-token, and only root and the agent's own service user can read it on this box:
$TOKEN
NEXT
}
case "$MODE" in
install) install ;;
install) do_install ;;
status) status ;;
uninstall) uninstall ;;
esac
scripts/capture-fixtures.sh
+21 -9
View File
@@ -1,7 +1,7 @@
#!/usr/bin/env bash
# Re-record ground-truth captures into tests/fixtures/.
#
# ./scripts/capture-fixtures.sh # needs the tunnels up: scripts/dev-tunnels.sh start
# ./scripts/capture-fixtures.sh # dials HOST_TARGETS from .env.dev, directly
# ./scripts/capture-fixtures.sh --with-llm # also scrape the live engines' /metrics
# # (gufo /slots is captured without the flag)
# SMART_HOSTS="deskwork-ops driveripper-ops" ./scripts/capture-fixtures.sh
@@ -19,25 +19,37 @@ ROOT="$(cd "$(dirname "$0")/.." && pwd)"
FIX="$ROOT/tests/fixtures"
mkdir -p "$FIX"
HOSTS=(driveripper:9121 deskwork:9122 gateway:9123)
# Engine URLs come from the same env the app uses, so this script carries no deployment
# detail of its own — and never records a tokenised URL into a fixture by accident.
# Agents come from the same env the app reads: HOST_TARGETS in .env.dev, `name=url` pairs
# (space or comma separated — the shape config.parse_targets accepts), the bearer token in
# the URL userinfo used as-is. Nothing is hard-coded here, so a capture can never come from
# a different topology than the dashboard polls.
if [ -f "$ROOT/.env.dev" ]; then
set -a; . "$ROOT/.env.dev"; set +a
fi
HOSTS=()
targets="${HOST_TARGETS:-}"
for entry in ${targets//,/ }; do # space or comma separated, like config.parse_targets
HOSTS+=("$entry")
done
# Engine URLs come from the same env too, so this script carries no deployment
# detail of its own — and never records a tokenised URL into a fixture by accident.
ENGINES=()
[ -n "${GUFO_BASE_URL:-}" ] && ENGINES+=("gufo:${GUFO_BASE_URL%/}/metrics")
[ -n "${VLLM_BASE_URL:-}" ] && ENGINES+=("vllm:${VLLM_BASE_URL%/}/metrics")
failed=0
if [ ${#HOSTS[@]} -eq 0 ]; then
echo " (skipped agent captures — HOST_TARGETS unset; see .env.dev.example)"
fi
for entry in "${HOSTS[@]}"; do
name=${entry%%:*}; port=${entry##*:}
url="http://127.0.0.1:${port}/snapshot"
if curl -fsS --max-time 8 "$url" -o "$FIX/agent_${name}.json"; then
name=${entry%%=*}; url=${entry#*=}
# The failure line prints the address without its userinfo: a token belongs in the
# dial, not in the terminal scrollback or whatever shells this output into.
if curl -fsS --max-time 8 "${url%/}/snapshot" -o "$FIX/agent_${name}.json"; then
echo " ✓ agent_${name}.json ($(wc -c <"$FIX/agent_${name}.json") bytes)"
else
echo " ✗ agent_${name}.json — no agent on $url (scripts/dev-tunnels.sh start?)"
echo " ✗ agent_${name}.json — no agent answered at ${url#*@} (deploy/install-agent.sh? firewall?)"
failed=1
fi
done
@@ -105,7 +117,7 @@ PY
then
: # per-UPS lines above
else
echo " ✗ nut_*.vars — upsd at $NUT_HOST:${NUT_PORT:-3493} unreachable (scripts/dev-tunnels.sh start?)"
echo " ✗ nut_*.vars — upsd at $NUT_HOST:${NUT_PORT:-3493} unreachable (down? firewall? NUT_PORT is the port upsd really listens on)"
failed=1
fi
else
scripts/dev-server.sh
+1 -1
View File
@@ -1,5 +1,5 @@
#!/usr/bin/env bash
# Local development server: start|stop|status|restart|logs|tunnel-check
# Local development server: start|stop|status|restart|logs
#
# ./scripts/dev-server.sh restart # reads .env.dev if present, runs on DASH_PORT
#
scripts/dev-tunnels.sh
-174
View File
@@ -1,174 +0,0 @@
#!/usr/bin/env bash
# Dev-only: reach live services on each host from the laptop without opening a single port.
#
# ./scripts/dev-tunnels.sh start|stop|status|restart
#
# DEV_TUNNELS starts agent/host_agent.py remotely and forwards it:
# ssh -L <local>:127.0.0.1:<rport> <host> 'python3 host_agent.py --port <rport>'
# DEV_FORWARDS forwards a service that is already listening remotely (e.g. upsd on 3493):
# nothing is started over there, the port just appears on localhost.
#
# Both supervise themselves: a dropped link retries every 5s. Production is the opposite —
# deploy/install-agent.sh installs a systemd unit that binds the LAN address with a bearer
# token, and the dashboard dials it directly.
set -uo pipefail
ROOT="$(cd "$(dirname "$0")/.." && pwd)"
RUN="$ROOT/data/tunnels"
LOGS="$ROOT/data/logs"
# Where the agent lives on each host. `~` is written into the generated script literally so
# the REMOTE shell expands it (a local $HOME would be the wrong path over there).
AGENT_REMOTE_DIR="${AGENT_REMOTE_DIR:-~/lab-agent}"
AGENT_COMMAND="${AGENT_COMMAND:-python3 host_agent.py}"
# Targets come from .env.dev (see .env.dev.example) — never hardcoded here, so this file
# stays portable and holds no hostnames of ours.
[ -f "$ROOT/.env.dev" ] && . "$ROOT/.env.dev"
DEV_TUNNELS="${DEV_TUNNELS:-}"
DEV_FORWARDS="${DEV_FORWARDS:-}"
if [ -z "$DEV_TUNNELS" ] && [ -z "$DEV_FORWARDS" ]; then
echo "no DEV_TUNNELS or DEV_FORWARDS set — copy .env.dev.example to .env.dev first"; exit 2
fi
TUNNELS=()
for entry in $DEV_TUNNELS; do
# 9121:driveripper-ops:9120 -> "9121 driveripper-ops 9120"
TUNNELS+=("$(echo "$entry" | tr ':' ' ')")
done
FORWARDS=()
for entry in $DEV_FORWARDS; do
FORWARDS+=("$(echo "$entry" | tr ':' ' ')")
done
mkdir -p "$RUN" "$LOGS"
start() {
for spec in "${TUNNELS[@]}"; do
read -r local host remote <<<"$(echo "$spec" | tr -s ' ')"
# pidfiles are named by host AND port: one host can carry an agent tunnel plus an upsd
# forward. The older host-only name is cleared so a restart does not double-bind.
pidfile="$RUN/$host-$local.pid"
if [ -f "$RUN/$host.pid" ] && kill -0 "$(cat "$RUN/$host.pid")" 2>/dev/null; then
pkill -P "$(cat "$RUN/$host.pid")" 2>/dev/null
kill "$(cat "$RUN/$host.pid")" 2>/dev/null
fi
rm -f "$RUN/$host.pid"
if [ -f "$pidfile" ] && kill -0 "$(cat "$pidfile")" 2>/dev/null; then
echo " $host already tunnelling on :$local (pid $(cat "$pidfile"))"
continue
fi
# A previous agent survives the ssh session that started it, and the next agent then
# dies on bind() with "address already in use" — which looks like a broken agent rather
# than a leaked process. Clear it in a SEPARATE ssh session: putting pkill in the same
# command line as the agent makes pkill match that very command line and shoot itself.
cat > "$RUN/$host.sh" <<EOF
#!/usr/bin/env bash
# generated by scripts/dev-tunnels.sh — do not edit, it is rewritten on every start
while true; do
ssh -n -o BatchMode=yes "$host" \
"pkill -f '[h]ost_agent.py --host 127.0.0.1 --port $remote'" >/dev/null 2>&1
sleep 1
ssh -nT -o ServerAliveInterval=30 -o ServerAliveCountMax=4 -o ExitOnForwardFailure=yes \
-L $local:127.0.0.1:$remote "$host" \
"cd $AGENT_REMOTE_DIR && exec $AGENT_COMMAND --host 127.0.0.1 --port $remote"
echo "[tunnel] $host link dropped at \$(date -Is) (exit \$?); retrying in 5s"
sleep 5
done
EOF
chmod +x "$RUN/$host.sh"
nohup "$RUN/$host.sh" >>"$LOGS/tunnel-$host.log" 2>&1 &
echo $! >"$pidfile"
echo " $host -> http://127.0.0.1:$local (pid $!)"
done
# Plain forwards: `-N` because there is no remote command to run, and nothing to clean up
# over there when the link goes.
for spec in "${FORWARDS[@]}"; do
read -r local host remote <<<"$(echo "$spec" | tr -s ' ')"
pidfile="$RUN/$host-$local.pid"
if [ -f "$pidfile" ] && kill -0 "$(cat "$pidfile")" 2>/dev/null; then
echo " $host:$remote already forwarded on :$local (pid $(cat "$pidfile"))"
continue
fi
cat > "$RUN/$host-$local.sh" <<EOF
#!/usr/bin/env bash
# generated by scripts/dev-tunnels.sh — do not edit, it is rewritten on every start
while true; do
ssh -n -N -o ServerAliveInterval=30 -o ServerAliveCountMax=4 -o ExitOnForwardFailure=yes \
-L $local:127.0.0.1:$remote "$host"
echo "[forward] $host:$remote dropped at \$(date -Is) (exit \$?); retrying in 5s"
sleep 5
done
EOF
chmod +x "$RUN/$host-$local.sh"
nohup "$RUN/$host-$local.sh" >>"$LOGS/tunnel-$host-$local.log" 2>&1 &
echo $! >"$pidfile"
echo " $host:$remote -> 127.0.0.1:$local (pid $!)"
done
}
stop() {
for spec in "${TUNNELS[@]}"; do
read -r local host remote <<<"$(echo "$spec" | tr -s ' ')"
pidfile="$RUN/$host-$local.pid"
if [ -f "$RUN/$host.pid" ]; then
pkill -P "$(cat "$RUN/$host.pid")" 2>/dev/null
kill "$(cat "$RUN/$host.pid")" 2>/dev/null
rm -f "$RUN/$host.pid"
fi
[ -f "$pidfile" ] || continue
pid=$(cat "$pidfile")
pkill -P "$pid" 2>/dev/null
kill "$pid" 2>/dev/null
# -N would mean "no remote command", so the forward is kept by a plain ssh running the
# agent; kill any leftover forward for this host by pattern, not just by parent.
pkill -f "ssh -nT.*-L $local:127.0.0.1:$remote $host" 2>/dev/null
# The remote agent outlives its ssh session, and the next one then dies on bind() with
# "address already in use" — which looks like a broken agent, not a leaked process.
timeout 10 ssh -n "$host" "pkill -f '[h]ost_agent.py --host 127.0.0.1 --port $remote'" \
>/dev/null 2>&1
rm -f "$pidfile"
echo " stopped $host"
done
for spec in "${FORWARDS[@]}"; do
read -r local host remote <<<"$(echo "$spec" | tr -s ' ')"
pidfile="$RUN/$host-$local.pid"
[ -f "$pidfile" ] || continue
pkill -P "$(cat "$pidfile")" 2>/dev/null
kill "$(cat "$pidfile")" 2>/dev/null
pkill -f "ssh -n -N.*-L $local:127.0.0.1:$remote $host" 2>/dev/null
rm -f "$pidfile" "$RUN/$host-$local.sh"
echo " stopped forward $host:$remote (:$local)"
done
}
status() {
for spec in "${TUNNELS[@]}"; do
read -r local host remote <<<"$(echo "$spec" | tr -s ' ')"
body=$(curl -s --max-time 4 "http://127.0.0.1:$local/health" || true)
if [ -n "$body" ]; then
echo " OK $host http://127.0.0.1:$local $body"
else
echo " DOWN $host http://127.0.0.1:$local"
fi
done
for spec in "${FORWARDS[@]}"; do
read -r local host remote <<<"$(echo "$spec" | tr -s ' ')"
# no HTTP behind these (upsd speaks its own line protocol), so the check is a bare TCP
# handshake — enough to tell "the tunnel is up" from "nothing listens over there".
if timeout 3 bash -c "exec 3<>/dev/tcp/127.0.0.1/$local" 2>/dev/null; then
echo " OK $host:$remote 127.0.0.1:$local"
else
echo " DOWN $host:$remote 127.0.0.1:$local"
fi
done
}
case "${1:-start}" in
start) start ;;
stop) stop ;;
restart) stop; sleep 1; start ;;
status) status ;;
*) echo "usage: $0 start|stop|status|restart"; exit 2 ;;
esac
src/dashboards/alerts.py
+1 -1
View File
@@ -133,7 +133,7 @@ _POLLER_CLASS = re.compile(r"^UpstreamError: ")
def _clean_error(label: str, error: str) -> str:
"""`http://h:9121/snapshot unreachable (ConnectError)` from the poller's wrapped string."""
"""`http://<host>:9120/snapshot unreachable (ConnectError)` from the poller's wrapped string."""
text = error.strip()
while True:
stripped = _POLLER_CLASS.sub("", text, count=1).strip()
src/dashboards/sources/nut.py
+10 -1
View File
@@ -118,8 +118,17 @@ class NutClient:
asyncio.open_connection(self.host, self.port), self.timeout
)
except (TimeoutError, OSError) as exc:
# `wait_for()` raises a TimeoutError whose own message is empty, so a blocked port
# used to read `cannot reach upsd at … (TimeoutError: )` in the health record, the
# alert and the event feed. Name the failure mode: this sentence is the whole
# explanation of a red UPS panel.
reason = (
f"timed out after {self.timeout:g}s"
if isinstance(exc, TimeoutError) and not str(exc)
else f"{type(exc).__name__}: {exc}"
)
raise NutError(
f"cannot reach upsd at {self.host}:{self.port} ({type(exc).__name__}: {exc})",
f"cannot reach upsd at {self.host}:{self.port} ({reason})",
code="UNREACHABLE",
) from exc
tests/test_alerts.py
+8 -8
View File
@@ -106,21 +106,21 @@ def test_a_source_that_stopped_answering_is_a_fail_naming_it_and_linking_its_pag
def test_the_collectors_host_prefix_is_not_doubled() -> None:
"""Collectors already say `driveripper: …`; the alert starts with the name too."""
out = alerts.build(SETTINGS, [state(ok=False,
error="driveripper: http://h:9121/snapshot unreachable")])
assert out[0]["message"] == "driveripper: http://h:9121/snapshot unreachable"
error="driveripper: http://h:9120/snapshot unreachable")])
assert out[0]["message"] == "driveripper: http://h:9120/snapshot unreachable"
def test_the_pollers_exception_prefix_is_not_printed_either() -> None:
"""The live shape is `poller`'s `UpstreamError: name: …`; one name, no class noise.
Verified against a real stopped agent tunnel: the health record reads
`UpstreamError: gateway: http://127.0.0.1:9123/snapshot unreachable (ConnectError)`,
Verified against a real stopped agent: the health record reads
`UpstreamError: gateway: http://<host>:9120/snapshot unreachable (ConnectError)`,
and printing that verbatim put the host name on the page twice.
"""
out = alerts.build(SETTINGS, [state(key="host:gateway", label="gateway", ok=False,
error="UpstreamError: gateway: http://h:9123/snapshot "
error="UpstreamError: gateway: http://h:9120/snapshot "
"unreachable (ConnectError)")])
assert out[0]["message"] == ("gateway: http://h:9123/snapshot unreachable (ConnectError)")
assert out[0]["message"] == ("gateway: http://h:9120/snapshot unreachable (ConnectError)")
def test_an_unexpected_exception_class_is_still_named() -> None:
@@ -187,9 +187,9 @@ def test_a_disk_job_that_failed_because_its_agent_is_down_is_one_line_not_two()
"""
out = alerts.build(SETTINGS, [
state(key="disks:driveripper", dashboard="disks", label="driveripper",
ok=False, error="UpstreamError: driveripper: http://h:9121/snapshot "
ok=False, error="UpstreamError: driveripper: http://h:9120/snapshot "
"unreachable (ConnectError)"),
state(ok=False, error="UpstreamError: driveripper: http://h:9121/snapshot "
state(ok=False, error="UpstreamError: driveripper: http://h:9120/snapshot "
"unreachable (ConnectError)"),
], now=2_000.0)
assert [item["source"] for item in out] == ["driveripper"]
tests/test_api.py
+2 -2
View File
@@ -453,13 +453,13 @@ def test_an_upstream_failure_escaping_a_route_keeps_its_own_status(tmp_path: Pat
@app.get("/api/_boom")
async def boom() -> None:
raise UpstreamError("driveripper: http://127.0.0.1:9121/snapshot timed out", status=504)
raise UpstreamError("driveripper: http://127.0.0.1:9120/snapshot timed out", status=504)
with TestClient(app) as test_client:
response = test_client.get("/api/_boom")
assert response.status_code == 504, "a timeout stays a timeout, not a 500"
assert response.json() == {"ok": False,
"detail": "driveripper: http://127.0.0.1:9121/snapshot "
"detail": "driveripper: http://127.0.0.1:9120/snapshot "
"timed out"}
assert "no-store" in response.headers["cache-control"]
store.close()
tests/test_config.py
+1 -1
View File
@@ -119,7 +119,7 @@ def test_nut_host_accepts_a_nut_scheme_prefix() -> None:
def test_load_reads_host_targets_from_the_environment(monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.setenv("HOST_TARGETS", "deskwork=http://127.0.0.1:9122")
monkeypatch.setenv("HOST_TARGETS", "deskwork=http://tok@deskwork.example.net:9120")
settings = config.load()
assert [t.name for t in settings.host_targets] == ["deskwork"]
tests/test_deploy_scripts.py
+398
View File
@@ -0,0 +1,398 @@
"""Contract tests for the deployment scripts in `deploy/`.
There is no shell test framework here — and adding one would buy nothing a subprocess does
not — but phase 10 rests on a sentence that lives in **two** files: the sudoers grant in
`deploy/agent-sudoers`, and the hint `deploy/install-agent.sh` prints when it is run without
root by the unprivileged `ops` account. Two copies of one instruction is exactly what drifts
silently, and the failure is an install that cannot happen on a host nobody is sitting at.
So these tests read the file, run the script the way the operator does (unprivileged, which
exits before the script can touch anything), and require the two lines to be the same line
apart from the path the script was invoked through.
"""
from __future__ import annotations
import os
import re
import shutil
import stat
import subprocess
from pathlib import Path
import pytest
ROOT = Path(__file__).resolve().parent.parent
DEPLOY = ROOT / "deploy"
INSTALLER = DEPLOY / "install-agent.sh"
SUDOERS = DEPLOY / "agent-sudoers"
BASH = shutil.which("bash") or "/bin/bash"
#: The grant as the file documents it. `/home/ops/agent-install` is the canonical staging
#: path: a monitored host has no repo checkout, so `deploy/` and `agent/` are copied there
#: (phase 10 task 02) and every host ends up with the same absolute path to grant.
CANONICAL_GRANT = "ops ALL=(root) NOPASSWD: /home/ops/agent-install/deploy/install-agent.sh"
#: The prefix a correct hint line starts with — user, run-as, and NOPASSWD, everything but
#: the path, which is the one thing the installer is allowed to substitute.
GRANT_PREFIX = "ops ALL=(root) NOPASSWD: "
#: Everything a non-root operator needs to hear, besides the grant line itself.
HINT_MUST_SAY = (
"visudo -cf", # check the syntax before installing it
"install -m 0440 -o root -g root", # and install it the way sudoers files are installed
"/etc/sudoers.d/dashboards-agent", # the destination both files name
"deploy/agent-sudoers", # the pointer back to the documented file
)
def _run(args: list[str], cwd: Path | None = None) -> subprocess.CompletedProcess[str]:
return subprocess.run(args, capture_output=True, text=True, cwd=cwd, check=False)
def _grant_lines(text: str) -> list[str]:
"""The sudoers *rules* in a file — comments start with `#`, so this never matches prose."""
return [line for line in text.splitlines() if line.startswith(GRANT_PREFIX)]
def _installer_without_root(*flags: str) -> subprocess.CompletedProcess[str]:
"""Run the installer the way `ops` does.
Skipped as root on purpose: root would go past `need_root` and really install a systemd
unit on the machine running pytest.
"""
if hasattr(os, "geteuid") and os.geteuid() == 0:
pytest.skip("running as root — the installer would install a real unit here")
return _run([BASH, str(INSTALLER), *flags])
# --- deploy/agent-sudoers ---------------------------------------------------------------
def test_agent_sudoers_exists_and_is_a_config_file_not_a_script() -> None:
assert SUDOERS.is_file(), "deploy/agent-sudoers is missing (phase 10 task 01)"
assert not os.access(SUDOERS, os.X_OK), "a sudoers file is data; only install-agent.sh runs"
def test_agent_sudoers_is_exactly_one_grant_line_at_the_canonical_path() -> None:
"""One rule, the installer, the staging path — nothing else may be granted."""
lines = _grant_lines(SUDOERS.read_text())
assert lines == [CANONICAL_GRANT], (
"deploy/agent-sudoers must grant exactly one command, the staged installer:\n" + "\n".join(lines)
)
assert "/home/ops/agent-install" in SUDOERS.read_text()
def test_agent_sudoers_grants_no_shell_editor_or_second_path() -> None:
"""The 'deliberately NOT granted' block has to stay a comment, not become a rule."""
text = SUDOERS.read_text()
not_granted = text.split("deliberately NOT granted", 1)[-1]
for never in ("visudo", "sh", "tee", "systemctl", "usermod", "upscmd", "upsrw"):
assert never in not_granted, f"the file must name {never!r} as deliberately not granted"
assert not_granted.lstrip().startswith("-"), "the block heading is followed by the list"
# Every line of that block is a comment: a stray rule there would be a second grant.
assert _grant_lines(not_granted) == []
# And it names the token file, because "we did not grant reading it" is the point.
assert "/etc/dashboards-agent.token" in not_granted
def test_agent_sudoers_documents_the_why_and_the_two_stage_install() -> None:
"""House style (deploy/nut-sudoers): why it is needed, then check-first install steps."""
text = SUDOERS.read_text()
assert "WHY THIS IS NEEDED" in text
for step in HINT_MUST_SAY:
assert step in text, f"deploy/agent-sudoers must document {step!r}"
# The installed copy is checked too — the file being valid is not the same as the
# installed file being valid (nut-sudoers' third step, and the reason stays).
assert text.count("visudo -cf") >= 2
@pytest.mark.skipif(shutil.which("visudo") is None, reason="visudo is not installed on this box")
def test_agent_sudoers_passes_the_syntax_check_it_documents() -> None:
"""The file tells the operator to run `visudo -cf` on it, so it must survive that."""
result = _run([shutil.which("visudo") or "/usr/sbin/visudo", "-cf", str(SUDOERS)])
assert result.returncode == 0, f"visudo -cf failed:\n{result.stdout}{result.stderr}"
# --- deploy/install-agent.sh without root -----------------------------------------------
def test_installer_without_root_exits_1_and_prints_the_grant_line() -> None:
result = _installer_without_root("--bind", "127.0.0.1")
assert result.returncode == 1, "the non-root refusal keeps its exit code (1)"
assert result.stdout == "", "the refusal is a failure: it goes to stderr, not stdout"
lines = _grant_lines(result.stderr)
assert lines == [f"{GRANT_PREFIX}{Path(os.path.realpath(INSTALLER))}"], (
"the hint must print one grant line naming this script's resolved path:\n" + result.stderr
)
def test_hint_and_the_sudoers_file_are_the_same_line_apart_from_the_path() -> None:
"""The pinned version of the diff in data/logs/NOTES-sudoers.md."""
printed = _installer_without_root("--bind", "127.0.0.1").stderr
hint = _grant_lines(printed)[0]
canonical = _grant_lines(SUDOERS.read_text())[0]
assert hint.startswith(GRANT_PREFIX) and canonical.startswith(GRANT_PREFIX)
assert hint.removeprefix(GRANT_PREFIX).endswith("/deploy/install-agent.sh")
assert hint.removeprefix(GRANT_PREFIX).removeprefix(str(DEPLOY.parent)) == (
canonical.removeprefix(GRANT_PREFIX).removeprefix("/home/ops/agent-install")
), "same user, same (root), same NOPASSWD, same script — only the checkout path differs"
def test_hint_resolves_a_relative_invocation_to_an_absolute_path() -> None:
"""`bash deploy/install-agent.sh` from the repo root still prints a path sudo can match."""
result = _run([BASH, "deploy/install-agent.sh", "--bind", "10.0.0.5"], cwd=ROOT)
assert result.returncode == 1
assert _grant_lines(result.stderr) == [f"{GRANT_PREFIX}{Path(os.path.realpath(INSTALLER))}"]
def test_installer_names_the_two_stage_install_and_points_at_the_file() -> None:
stderr = _installer_without_root("--bind", "127.0.0.1").stderr
for phrase in HINT_MUST_SAY:
assert phrase in stderr, f"the non-root hint must say {phrase!r}, not just 'run as root'"
# The pointer is the file next to *this* script, wherever the checkout lives.
assert str(Path(os.path.realpath(SUDOERS))) in stderr
def test_installer_hint_repeats_the_command_that_was_asked_for() -> None:
"""Copy-pasteable: the hint ends with the same invocation, with the flags it was given."""
stderr = _installer_without_root("--uninstall").stderr
assert f"sudo {Path(os.path.realpath(INSTALLER))} --uninstall" in stderr
def test_installer_help_points_at_the_sudoers_grant() -> None:
"""The header comment *is* the help text, so the grant is discoverable from --help."""
result = _run([BASH, str(INSTALLER), "--help"])
assert result.returncode == 0
assert "deploy/agent-sudoers" in result.stdout
# --- the tunnel machinery stays retired (phase 10) ---------------------------------------
#: Names and ports that only ever existed to serve the ssh-tunnel dev transport, which phase
#: 10 deleted. Every pattern is written as a character class (`[d]ev-…`) so that **this file
#: cannot match its own grep**: the phase criterion demands the repo stay free of those
#: literals, and a guard that reintroduces one to forbid it is the oldest trick there is.
#: Scope excludes `.agents/` (phase files and run reports are history — rewriting a finished
#: phase's record to satisfy a grep would falsify it) and `tests/fixtures/` (verbatim
#: upstream captures: a sector counter or a wheel hash may contain those digits by accident,
#: and rule 7 forbids editing a capture to please a grep).
TUNNEL_PATTERNS = ("[d]ev-tunnels", "DEV_[T]UNNELS", "DEV_[F]ORWARDS", r"912[123]\b", r"[3]494\b")
#: Where those names must not appear: every file that *tells someone what to run*.
#: `tests/fixtures/` is excluded (verbatim captures, see above) and so is `.agents/` —
#: phase files and run reports are history, and rewriting a finished phase's record to
#: satisfy a grep would falsify it.
TUNNEL_FREE_ROOTS = ("scripts", "deploy", "src", "agent", "web")
def _tunnel_scope_files() -> list[Path]:
files: list[Path] = []
for top in TUNNEL_FREE_ROOTS:
for path in (ROOT / top).rglob("*"):
if path.is_file() and path.suffix in (".py", ".sh", ".md", ".css", ".js", ".html"):
files.append(path)
files.append(ROOT / "README.md")
files.append(ROOT / "AGENTS.md")
files.append(ROOT / ".env.example")
files.append(ROOT / ".env.dev.example")
for path in (ROOT / "tests").glob("*.py"):
files.append(path)
return files
def test_dev_tunnel_script_is_deleted() -> None:
"""The tunnel script is gone, not shelved: git history keeps it if it is ever needed."""
# Name built at runtime so this file never carries the literal the grep forbids.
script = ROOT / "scripts" / ("dev-" + "tunnels.sh")
assert not script.exists(), (
"the retired ssh-tunnel dev script is back — phase 10 deleted it deliberately"
)
def test_no_operational_file_mentions_the_tunnel_machinery() -> None:
"""A stale tunnel-env instruction is a setup step nobody can complete any more."""
offenders: list[str] = []
for path in _tunnel_scope_files():
text = path.read_text(errors="replace")
for pattern in TUNNEL_PATTERNS:
if re.search(pattern, text):
offenders.append(f"{path.relative_to(ROOT)}: /{pattern}/")
assert not offenders, "tunnel-era names/ports are back:\n" + "\n".join(offenders)
# --- shell syntax (the only gate these scripts get) -------------------------------------
#: Names a shell function must never take. Bash resolves a **function** before an external
#: command of the same name, so defining `install()` in a script that then calls
#: `install -d /somewhere` is not a directory creation — it is the function calling itself,
#: forever. That is not hypothetical: `deploy/install-agent.sh` did exactly this, and the
#: first real install of phase 10 spun on the SMART block for two minutes and installed
#: nothing (no unit, no agent file, no token) while looking like a working script to `bash -n`.
SHADOW_FORBIDDEN = frozenset(
{
"install",
"test",
"time",
"date",
"sort",
"grep",
"sed",
"tee",
"type",
"ulimit",
"wait",
"kill",
"cd",
"pwd",
}
)
_FUNCTION_DEF_RE = r"^([A-Za-z_][A-Za-z0-9_]*)\s*\(\)"
def _function_names(script: Path) -> list[str]:
return re.findall(_FUNCTION_DEF_RE, script.read_text(), flags=re.MULTILINE)
@pytest.mark.parametrize(
"script",
sorted([*DEPLOY.glob("*.sh"), *(ROOT / "scripts").glob("*.sh")]),
ids=lambda p: str(p.relative_to(ROOT)),
)
def test_shell_script_has_valid_syntax(script: Path) -> None:
result = _run([BASH, "-n", str(script)])
assert result.returncode == 0, f"bash -n failed for {script}:\n{result.stderr}"
@pytest.mark.parametrize(
"script",
sorted([*DEPLOY.glob("*.sh"), *(ROOT / "scripts").glob("*.sh")]),
ids=lambda p: str(p.relative_to(ROOT)),
)
def test_no_shell_function_is_named_after_a_coreutil_it_could_be_asked_to_run(script: Path) -> None:
"""`install()` + `install -d` is infinite recursion, and `bash -n` cannot see it."""
shadowed = SHADOW_FORBIDDEN.intersection(_function_names(script))
assert not shadowed, (
f"{script.relative_to(ROOT)} defines function(s) {sorted(shadowed)} that shadow an "
"external command: bash runs the function, so any call meant for the program recurses"
)
# --- the install path, run for real against a sandboxed root boundary --------------------
#: The commands of `install()` that would touch the machine running pytest. Each one is a
#: PATH stub that records its arguments and succeeds — the same trick `ScriptedNutClient` is
#: for the UPS wire: fake the privileged boundary, run the real logic.
ROOT_BOUNDARY_STUBS = ("id", "getent", "useradd", "usermod", "chown", "systemctl")
#: The three absolute destinations the installer writes to, rewritten into a temp directory
#: so the test exercises the body instead of the operator's `/usr` and `/etc`.
SANDBOX_REWRITES = {
'AGENT_DIR="/usr/local/lib/dashboards-agent"': 'AGENT_DIR="{root}/lib"',
'TOKEN_FILE="/etc/dashboards-agent.token"': 'TOKEN_FILE="{root}/token"',
'UNIT="/etc/systemd/system/$SERVICE.service"': 'UNIT="{root}/dashboards-agent.service"',
}
def _sandbox_installer(tmp_path: Path) -> tuple[Path, Path, dict[str, str]]:
"""A copy of the installer whose writes land in `tmp_path`, plus a stubbed PATH.
Returns (script, stub call log, env). `agent/host_agent.py` is stubbed next to the copy
because the script resolves the agent as `../agent/host_agent.py`.
"""
text = INSTALLER.read_text()
target = tmp_path / "target"
for old, new in SANDBOX_REWRITES.items():
assert old in text, f"the installer moved {old!r} — update SANDBOX_REWRITES with it"
text = text.replace(old, new.format(root=target))
script = tmp_path / "deploy" / "install-agent.sh"
script.parent.mkdir(parents=True)
script.write_text(text)
(tmp_path / "agent").mkdir()
(tmp_path / "agent" / "host_agent.py").write_text("# stub agent source\n", encoding="utf-8")
stubs = tmp_path / "stubs"
stubs.mkdir()
call_log = tmp_path / "stub-calls.log"
for name in ROOT_BOUNDARY_STUBS:
stub = stubs / name
# `id -u` must answer 0 or need_root() refuses before the body runs; every other call
# just records itself, which is what the assertions below read.
body = ['#!/bin/sh', f'printf "%s %s\\n" "{name}" "$*" >> "$STUB_LOG"']
if name == "id":
body.append('[ "$1" = "-u" ] && echo 0')
body.append("exit 0")
stub.write_text("\n".join(body) + "\n", encoding="utf-8")
stub.chmod(0o755)
env = {**os.environ, "PATH": f"{stubs}{os.pathsep}{os.environ['PATH']}", "STUB_LOG": str(call_log)}
return script, call_log, env
@pytest.mark.skipif(hasattr(os, "geteuid") and os.geteuid() == 0, reason="running as root")
def test_install_path_installs_into_its_own_paths_and_prints_the_token_once(tmp_path: Path) -> None:
"""The install body end to end: files, unit, token, and no recursion.
The exit code is the regression test for the `install()` shadow: while the function was
named after `/usr/bin/install`, this never returned at all.
"""
script, call_log, env = _sandbox_installer(tmp_path)
try:
result = subprocess.run(
[BASH, str(script), "--bind", "10.0.0.5", "--port", "9120"],
capture_output=True,
text=True,
env=env,
timeout=60,
check=False,
)
except subprocess.TimeoutExpired as exc:
pytest.fail(f"the installer did not return (recursion? partial output: {exc.stdout!r})")
assert result.returncode == 0, f"install failed:\n{result.stdout}\n{result.stderr}"
target = tmp_path / "target"
assert (target / "lib" / "host_agent.py").is_file(), "install -m never landed the agent"
unit = (target / "dashboards-agent.service").read_text()
token_path = target / "token"
token = token_path.read_text().strip()
assert "User=dashboards-agent" in unit
assert f"--token-file {token_path}" in unit, "the unit must use the file, never --token"
assert "--host 10.0.0.5 --port 9120" in unit
# DevicePolicy=strict denies every node not listed, and systemd opens /dev/null for stdin
# before it execs python: without this line the unit dies at spawn (status 208/STDIN).
assert "DevicePolicy=strict" in unit
assert "DeviceAllow=/dev/null" in unit, "strict device policy needs /dev/null or the unit cannot start"
assert "DeviceAllow=/dev/nvme*" in unit, "SMART reads need the disk nodes back"
assert token and len(token) >= 32, "a generated token is expected"
assert token in result.stdout, "first install prints the token, and never again"
assert f'HOST_TARGETS="hostname=http://{token}@10.0.0.5:9120"' in result.stdout
calls = call_log.read_text()
assert "systemctl daemon-reload" in calls
assert "systemctl enable --now dashboards-agent" in calls
assert "chown root:dashboards-agent" in calls, "the token file must become group-readable"
def test_token_file_is_readable_by_the_service_user_and_nobody_else(tmp_path: Path) -> None:
"""`--token-file` is a path the *agent* opens, as its own user.
systemd does not read the secret for the service, so a 0600 root-owned file makes the
agent exit before it binds. 0640 root:dashboards-agent is the smallest mode that works,
and it is still unreadable by `ops` (which is why deploy/agent-sudoers does not grant it).
"""
script, _call_log, env = _sandbox_installer(tmp_path)
result = subprocess.run(
[BASH, str(script), "--bind", "10.0.0.5"],
capture_output=True,
text=True,
env=env,
timeout=60,
check=False,
)
assert result.returncode == 0, f"install failed:\n{result.stdout}\n{result.stderr}"
mode = stat.S_IMODE((tmp_path / "target" / "token").stat().st_mode)
assert mode == 0o640, f"token file mode is {oct(mode)}, expected 0o640 (group = the agent user)"
assert not mode & 0o007, "the token file must never be world-readable"
tests/test_disks_source.py
+1 -1
View File
@@ -311,7 +311,7 @@ def test_two_hosts_do_not_share_verdict_state(disk_snapshot: Any) -> None:
# ---- the job: its own, slow, and absent rather than dialling --------------- #
def test_one_disks_job_per_host_agent_at_the_slow_cadence() -> None:
targets = [Target("driveripper", "http://a:9121"), Target("deskwork", "http://b:9122")]
targets = [Target("driveripper", "http://a:9120"), Target("deskwork", "http://b:9120")]
# the client is never used by the factory, only captured by the job closures
jobs = disks_jobs(targets, NeverDialClient(), DisksCollector(), 300.0, timeout=30.0)
assert [job.key for job in jobs] == ["disks:driveripper", "disks:deskwork"]
tests/test_live_fleet.py
+7 -7
View File
@@ -5,9 +5,9 @@ bridge to reality — they re-run the *production* code paths (`collectors.fetch
`NutMonitor.collect`) against real targets, which is what catches an upstream that
renamed a series or a driver that started reporting a temperature in kelvin.
They read exactly the env vars the app reads, so the dev loop is:
They read exactly the env vars the app reads, so the dev loop is direct dial — the same
addresses the deployed dashboard dials, nothing forwarded:
./scripts/dev-tunnels.sh start # agents + upsd forwarded to localhost
set -a; . .env.dev; set +a
uv run pytest -m live --live
@@ -55,7 +55,7 @@ async def test_every_host_agent_answers_and_parses(settings: Settings,
host_collector: HostCollector) -> None:
"""Each HOST_TARGETS entry must answer /snapshot and yield the metrics the page renders."""
if not settings.host_targets:
pytest.skip("HOST_TARGETS is unset — start ./scripts/dev-tunnels.sh")
pytest.skip("HOST_TARGETS is unset")
required = {"cpu.usage_pct", "mem.used_pct", "disk.used_pct"}
async with httpx.AsyncClient(timeout=settings.host_timeout) as client:
for target in settings.host_targets:
@@ -84,7 +84,7 @@ async def test_every_host_serves_the_hero_card_contract(settings: Settings,
upgrade renaming a field. Hermetic twin: `tests/test_system_contract.py`.
"""
if not settings.host_targets:
pytest.skip("HOST_TARGETS is unset — start ./scripts/dev-tunnels.sh")
pytest.skip("HOST_TARGETS is unset")
async with httpx.AsyncClient(timeout=settings.host_timeout) as client:
for target in settings.host_targets:
await collectors.fetch_host_snapshot(client, target, host_collector)
@@ -107,7 +107,7 @@ async def test_agent_disk_rows_carry_the_smart_shape(settings: Settings) -> None
operator's call, not a unit test.
"""
if not settings.host_targets:
pytest.skip("HOST_TARGETS is unset — start ./scripts/dev-tunnels.sh")
pytest.skip("HOST_TARGETS is unset")
seen_smart = 0
async with httpx.AsyncClient(timeout=settings.host_timeout) as client:
for target in settings.host_targets:
@@ -147,7 +147,7 @@ async def test_disks_source_charts_the_physical_drives_the_agent_reports(
disk, `disk.*@<dev>` series only for values that exist, and no invented numbers.
"""
if not settings.host_targets:
pytest.skip("HOST_TARGETS is unset — start ./scripts/dev-tunnels.sh")
pytest.skip("HOST_TARGETS is unset")
collector = DisksCollector()
checked = 0
async with httpx.AsyncClient(timeout=settings.disks_timeout) as client:
@@ -177,7 +177,7 @@ async def test_disks_source_charts_the_physical_drives_the_agent_reports(
if any(row["smart"] for row in payload["disks"]):
assert payload["smartctl_available"] is True
if not checked:
pytest.skip("no host agent answered — start ./scripts/dev-tunnels.sh")
pytest.skip("no host agent answered")
@pytest.mark.parametrize("engine", ["gufo", "vllm"])
tests/test_nut.py
+25
View File
@@ -249,6 +249,31 @@ async def test_connection_refused_is_reported_as_unreachable() -> None:
with pytest.raises(NutError) as excinfo:
await client.connect()
assert excinfo.value.code == "UNREACHABLE"
# The reason is the sentence the operator actually reads (health record, alert, feed).
assert "ConnectionRefusedError" in str(excinfo.value)
async def test_connect_timeout_names_the_timeout_instead_of_an_empty_parenthetical(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""`wait_for()` raises a TimeoutError with no message of its own.
A firewalled port therefore used to print `… (TimeoutError: )`, which explains nothing.
This is the exact state the fleet is in while the laptop cannot reach upsd, so the
message has to carry the timeout it hit.
"""
async def never_connects(*_args: object, **_kwargs: object) -> None:
await asyncio.sleep(0.2)
monkeypatch.setattr(asyncio, "open_connection", never_connects)
client = NutClient("ups.example.net", 3051, timeout=0.01)
with pytest.raises(NutError) as excinfo:
await client.connect()
assert excinfo.value.code == "UNREACHABLE"
message = str(excinfo.value)
assert "timed out after 0.01s" in message
assert "TimeoutError" not in message
async def test_server_closing_early_is_reported() -> None:
tests/ups_contract.py
+2 -2
View File
@@ -85,8 +85,8 @@ def parse_capture(name: str) -> nut.UpsReading:
path = FIXTURES / f"nut_{name}.vars"
if not path.exists():
raise AssertionError(
f"missing capture {path.name} — run ./scripts/capture-fixtures.sh with the upsd "
f"forwarded (DEV_FORWARDS); the UPS hero row is contracted against real drivers"
f"missing capture {path.name} — run ./scripts/capture-fixtures.sh with NUT_HOST "
f"pointing at the real upsd; the UPS hero row is contracted against real drivers"
)
variables: dict[str, str] = {}
commands: list[str] = []
uv.lock
Generated
+3 -3
View File
@@ -189,7 +189,7 @@ wheels = [
[[package]]
name = "fastapi"
version = "0.143.3"
version = "0.143.5"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "annotated-doc" },
@@ -199,9 +199,9 @@ dependencies = [
{ name = "typing-extensions" },
{ name = "typing-inspection" },
]
sdist = { url = "https://files.pythonhosted.org/packages/38/d5/b0aeb0b303650ac87b0411de4aaf3f901b06042dcc3fb00f30b97b7ff585/fastapi-0.143.3.tar.gz", hash = "sha256:b9ab567ee57afb4454004ce04c889f8f0160ac47946b99795887cd6e88484848", size = 469834, upload-time = "2026-10-16T14:14:13.31Z" }
sdist = { url = "https://files.pythonhosted.org/packages/8f/6a/ac6ec87026038fb7583018e0905f6c9d55bc8ee07d30d44c743f18b19435/fastapi-0.143.5.tar.gz", hash = "sha256:0b5c8e52d243cd2f015376b41e78db5bee585578782481abd145e1ab5a62a3ba", size = 470247, upload-time = "2026-10-18T09:12:55.556Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/2c/1a/03d246befcf307219bb2161c97d8c5d314f84125bf7dd245730a28589828/fastapi-0.143.3-py3-none-any.whl", hash = "sha256:75e8bd68e2f52f77d29b2879749c257d64de2d85282d25270caf2bfccce5efe5", size = 144849, upload-time = "2026-10-16T14:14:11.763Z" },
{ url = "https://files.pythonhosted.org/packages/eb/4e/5f7d1268900eb14a6697a42d94c8f7e3b8450d27301761f231672f928419/fastapi-0.143.5-py3-none-any.whl", hash = "sha256:09cc38ed8c6a95f407783ef4d98ceeb350e192fe8fbb8ed711070bcd5208fdfa", size = 144894, upload-time = "2026-10-18T09:12:54.125Z" },
]
[[package]]