From a08e693698268023c2ef25093c912e61b71746c4 Mon Sep 17 00:00:00 2001 From: ducoterra Date: Sun, 18 Oct 2026 05:44:13 -0400 Subject: [PATCH] phase: 10_dev_direct_dial MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit **Phase 10 final pass — repo green; live criteria blocked off-repo.** - Verified: `gateway` runs `dashboards-agent` (active/enabled, `10.1.1.27:9120` → `/health` 200, `/snapshot` 401 no-token / 200 bearer on-host, token matches `/etc/dashboards-agent.token`); `driveripper`/`deskwork` inactive, `sudo -n` → "a password is required", `:9120` empty. - New measurement (transient runtime rule on gateway `firewall-cmd --add-port=9120/tcp`, **reverted** — runtime+permanent port lists verified back to the original set, no rich rules): laptop still gets no SYN through, while `:22`/`:80` on the same address answer in ms ⇒ the block is the lab gateway `10.2.0.1` port filter, not this app/host firewall/agent. No other live change made (no sudoers, no upsd, no UPS command). - Tests/lint (exact): `bash .agents/validate.sh` → **EXIT 0** — `uv run pytest --cov --cov-fail-under=94` **793 passed, 11 skipped, coverage 98.33 %**; `ruff check src tests agent` clean; `pyright` 0 errors; `node --check` clean; hygiene clean; `uv pip list --outdated` empty (bumped fastapi 0.143.3→0.143.5, `uv.lock` only). - `uv run pytest -m live --live` (`.env.dev` sourced): **4 passed, 1 skipped, 6 failed** — all six are `:9120`/`:3051` connect timeouts; zero parse/contract failures. - Dev server restarted on current code: engines `ok`; 3×`host` + 3×`disks` + `nut` `DOWN` with real reasons (designed degradation, stub rows intact, no credential leakage). - Completion criteria: screenshots ✓ (`data/logs/phase10-{index,system,disks,llm,ups}-live.png`); tunnel-vocabulary grep ✓ outside `.agents/` history (PLAN.md locked line, phase overviews, past reports; fixtures/`uv.lock` hits are digit substrings — `tests/test_deploy_scripts.py` guards operational files); no page-payload change ✓ (contract tests). **Not met, operator-side:** agents on all three hosts, `LIST UPS` from the laptop, `dev-server.sh status` all-ok, live suite green. - Fixed in this pass: `sources/nut.py` unreachable-upsd message no longer prints a bare `(TimeoutError: )` (now `(timed out after 4s)` / `(ConnectionRefusedError: …)`), pinned by a new test + extended refused-port test. - Docs: README verified consistent (every env var in `config.py` documented, routes/flags/`--with-llm`/pager sizes checked) — no changes needed. AGENTS.md fixed 4 inconsistencies: per-host sudo reality (gateway grants `NOPASSWD ALL`, which is why it installed without `agent-sudoers`; driveripper scoped; deskwork unrelated), the now-proven firewall finding, `/system` table typo ("agent per host"), gap-4 DB/WAL sizes + test count. `.env.example` `upsd.users` sample now one `instcmds` per line (it contradicted README and the in-app copy). `.env.dev` (gitignored) comment corrected to match the evidence. - Flagged, untouched (locked): `.agents/PLAN.md` L92–95 still says upsd is "forwarded to `127.0.0.1:3494` in dev" — false after this phase; evidence + exact operator commands appended to `data/logs/NOTES-verify.md` §11. - Next pending phase: `11_admin_login`. --- .../10_dev_direct_dial/00_phase.md | 0 .../01_sudoers_grant_and_installer_hint.md | 0 .../02_install_agents_on_fleet.md | 0 .../03_upsd_acl_for_laptop.md | 0 .../04_env_dev_direct_dial.md | 0 .../05_retire_tunnels_update_scripts.md | 0 .../06_verify_direct_dial_live.md | 0 .../10_dev_direct_dial__00_phase.a1.err | 0 .../10_dev_direct_dial__00_phase.a1.md | 12 + .../10_dev_direct_dial__00_phase.a1.md.status | 1 + .../10_dev_direct_dial__00_phase.a1.validate | 52 +++ ...01_sudoers_grant_and_installer_hint.a1.err | 0 ..._01_sudoers_grant_and_installer_hint.a1.md | 16 + ...oers_grant_and_installer_hint.a1.md.status | 1 + ...doers_grant_and_installer_hint.a1.validate | 52 +++ ...ct_dial__02_install_agents_on_fleet.a1.err | 0 ...ect_dial__02_install_agents_on_fleet.a1.md | 16 + ...l__02_install_agents_on_fleet.a1.md.status | 1 + ...al__02_install_agents_on_fleet.a1.validate | 52 +++ ...direct_dial__03_upsd_acl_for_laptop.a1.err | 0 ..._direct_dial__03_upsd_acl_for_laptop.a1.md | 13 + ..._dial__03_upsd_acl_for_laptop.a1.md.status | 1 + ...t_dial__03_upsd_acl_for_laptop.a1.validate | 52 +++ ...direct_dial__04_env_dev_direct_dial.a1.err | 0 ..._direct_dial__04_env_dev_direct_dial.a1.md | 19 + ..._dial__04_env_dev_direct_dial.a1.md.status | 1 + ...t_dial__04_env_dev_direct_dial.a1.validate | 52 +++ ...l__05_retire_tunnels_update_scripts.a1.err | 0 ...al__05_retire_tunnels_update_scripts.a1.md | 17 + ...retire_tunnels_update_scripts.a1.md.status | 1 + ..._retire_tunnels_update_scripts.a1.validate | 52 +++ ...ct_dial__06_verify_direct_dial_live.a1.err | 0 ...ect_dial__06_verify_direct_dial_live.a1.md | 14 + ...l__06_verify_direct_dial_live.a1.md.status | 1 + ...al__06_verify_direct_dial_live.a1.validate | 52 +++ .env.dev.example | 64 +-- .env.example | 17 +- AGENTS.md | 95 ++++- README.md | 50 ++- deploy/agent-sudoers | 76 ++++ deploy/install-agent.sh | 88 +++- scripts/capture-fixtures.sh | 30 +- scripts/dev-server.sh | 2 +- scripts/dev-tunnels.sh | 174 -------- src/dashboards/alerts.py | 2 +- src/dashboards/sources/nut.py | 11 +- tests/test_alerts.py | 16 +- tests/test_api.py | 4 +- tests/test_config.py | 2 +- tests/test_deploy_scripts.py | 398 ++++++++++++++++++ tests/test_disks_source.py | 2 +- tests/test_live_fleet.py | 14 +- tests/test_nut.py | 25 ++ tests/ups_contract.py | 4 +- uv.lock | 6 +- 55 files changed, 1259 insertions(+), 299 deletions(-) rename .agents/phases/{todo => complete}/10_dev_direct_dial/00_phase.md (100%) rename .agents/phases/{todo => complete}/10_dev_direct_dial/01_sudoers_grant_and_installer_hint.md (100%) rename .agents/phases/{todo => complete}/10_dev_direct_dial/02_install_agents_on_fleet.md (100%) rename .agents/phases/{todo => complete}/10_dev_direct_dial/03_upsd_acl_for_laptop.md (100%) rename .agents/phases/{todo => complete}/10_dev_direct_dial/04_env_dev_direct_dial.md (100%) rename .agents/phases/{todo => complete}/10_dev_direct_dial/05_retire_tunnels_update_scripts.md (100%) rename .agents/phases/{todo => complete}/10_dev_direct_dial/06_verify_direct_dial_live.md (100%) create mode 100644 .agents/reports/10_dev_direct_dial/10_dev_direct_dial__00_phase.a1.err create mode 100644 .agents/reports/10_dev_direct_dial/10_dev_direct_dial__00_phase.a1.md create mode 100644 .agents/reports/10_dev_direct_dial/10_dev_direct_dial__00_phase.a1.md.status create mode 100644 .agents/reports/10_dev_direct_dial/10_dev_direct_dial__00_phase.a1.validate create mode 100644 .agents/reports/10_dev_direct_dial/10_dev_direct_dial__01_sudoers_grant_and_installer_hint.a1.err create mode 100644 .agents/reports/10_dev_direct_dial/10_dev_direct_dial__01_sudoers_grant_and_installer_hint.a1.md create mode 100644 .agents/reports/10_dev_direct_dial/10_dev_direct_dial__01_sudoers_grant_and_installer_hint.a1.md.status create mode 100644 .agents/reports/10_dev_direct_dial/10_dev_direct_dial__01_sudoers_grant_and_installer_hint.a1.validate create mode 100644 .agents/reports/10_dev_direct_dial/10_dev_direct_dial__02_install_agents_on_fleet.a1.err create mode 100644 .agents/reports/10_dev_direct_dial/10_dev_direct_dial__02_install_agents_on_fleet.a1.md create mode 100644 .agents/reports/10_dev_direct_dial/10_dev_direct_dial__02_install_agents_on_fleet.a1.md.status create mode 100644 .agents/reports/10_dev_direct_dial/10_dev_direct_dial__02_install_agents_on_fleet.a1.validate create mode 100644 .agents/reports/10_dev_direct_dial/10_dev_direct_dial__03_upsd_acl_for_laptop.a1.err create mode 100644 .agents/reports/10_dev_direct_dial/10_dev_direct_dial__03_upsd_acl_for_laptop.a1.md create mode 100644 .agents/reports/10_dev_direct_dial/10_dev_direct_dial__03_upsd_acl_for_laptop.a1.md.status create mode 100644 .agents/reports/10_dev_direct_dial/10_dev_direct_dial__03_upsd_acl_for_laptop.a1.validate create mode 100644 .agents/reports/10_dev_direct_dial/10_dev_direct_dial__04_env_dev_direct_dial.a1.err create mode 100644 .agents/reports/10_dev_direct_dial/10_dev_direct_dial__04_env_dev_direct_dial.a1.md create mode 100644 .agents/reports/10_dev_direct_dial/10_dev_direct_dial__04_env_dev_direct_dial.a1.md.status create mode 100644 .agents/reports/10_dev_direct_dial/10_dev_direct_dial__04_env_dev_direct_dial.a1.validate create mode 100644 .agents/reports/10_dev_direct_dial/10_dev_direct_dial__05_retire_tunnels_update_scripts.a1.err create mode 100644 .agents/reports/10_dev_direct_dial/10_dev_direct_dial__05_retire_tunnels_update_scripts.a1.md create mode 100644 .agents/reports/10_dev_direct_dial/10_dev_direct_dial__05_retire_tunnels_update_scripts.a1.md.status create mode 100644 .agents/reports/10_dev_direct_dial/10_dev_direct_dial__05_retire_tunnels_update_scripts.a1.validate create mode 100644 .agents/reports/10_dev_direct_dial/10_dev_direct_dial__06_verify_direct_dial_live.a1.err create mode 100644 .agents/reports/10_dev_direct_dial/10_dev_direct_dial__06_verify_direct_dial_live.a1.md create mode 100644 .agents/reports/10_dev_direct_dial/10_dev_direct_dial__06_verify_direct_dial_live.a1.md.status create mode 100644 .agents/reports/10_dev_direct_dial/10_dev_direct_dial__06_verify_direct_dial_live.a1.validate create mode 100644 deploy/agent-sudoers delete mode 100755 scripts/dev-tunnels.sh create mode 100644 tests/test_deploy_scripts.py diff --git a/.agents/phases/todo/10_dev_direct_dial/00_phase.md b/.agents/phases/complete/10_dev_direct_dial/00_phase.md similarity index 100% rename from .agents/phases/todo/10_dev_direct_dial/00_phase.md rename to .agents/phases/complete/10_dev_direct_dial/00_phase.md diff --git a/.agents/phases/todo/10_dev_direct_dial/01_sudoers_grant_and_installer_hint.md b/.agents/phases/complete/10_dev_direct_dial/01_sudoers_grant_and_installer_hint.md similarity index 100% rename from .agents/phases/todo/10_dev_direct_dial/01_sudoers_grant_and_installer_hint.md rename to .agents/phases/complete/10_dev_direct_dial/01_sudoers_grant_and_installer_hint.md diff --git a/.agents/phases/todo/10_dev_direct_dial/02_install_agents_on_fleet.md b/.agents/phases/complete/10_dev_direct_dial/02_install_agents_on_fleet.md similarity index 100% rename from .agents/phases/todo/10_dev_direct_dial/02_install_agents_on_fleet.md rename to .agents/phases/complete/10_dev_direct_dial/02_install_agents_on_fleet.md diff --git a/.agents/phases/todo/10_dev_direct_dial/03_upsd_acl_for_laptop.md b/.agents/phases/complete/10_dev_direct_dial/03_upsd_acl_for_laptop.md similarity index 100% rename from .agents/phases/todo/10_dev_direct_dial/03_upsd_acl_for_laptop.md rename to .agents/phases/complete/10_dev_direct_dial/03_upsd_acl_for_laptop.md diff --git a/.agents/phases/todo/10_dev_direct_dial/04_env_dev_direct_dial.md b/.agents/phases/complete/10_dev_direct_dial/04_env_dev_direct_dial.md similarity index 100% rename from .agents/phases/todo/10_dev_direct_dial/04_env_dev_direct_dial.md rename to .agents/phases/complete/10_dev_direct_dial/04_env_dev_direct_dial.md diff --git a/.agents/phases/todo/10_dev_direct_dial/05_retire_tunnels_update_scripts.md b/.agents/phases/complete/10_dev_direct_dial/05_retire_tunnels_update_scripts.md similarity index 100% rename from .agents/phases/todo/10_dev_direct_dial/05_retire_tunnels_update_scripts.md rename to .agents/phases/complete/10_dev_direct_dial/05_retire_tunnels_update_scripts.md diff --git a/.agents/phases/todo/10_dev_direct_dial/06_verify_direct_dial_live.md b/.agents/phases/complete/10_dev_direct_dial/06_verify_direct_dial_live.md similarity index 100% rename from .agents/phases/todo/10_dev_direct_dial/06_verify_direct_dial_live.md rename to .agents/phases/complete/10_dev_direct_dial/06_verify_direct_dial_live.md diff --git a/.agents/reports/10_dev_direct_dial/10_dev_direct_dial__00_phase.a1.err b/.agents/reports/10_dev_direct_dial/10_dev_direct_dial__00_phase.a1.err new file mode 100644 index 0000000..e69de29 diff --git a/.agents/reports/10_dev_direct_dial/10_dev_direct_dial__00_phase.a1.md b/.agents/reports/10_dev_direct_dial/10_dev_direct_dial__00_phase.a1.md new file mode 100644 index 0000000..f000a24 --- /dev/null +++ b/.agents/reports/10_dev_direct_dial/10_dev_direct_dial__00_phase.a1.md @@ -0,0 +1,12 @@ +**Phase 10 final pass — repo green; live criteria blocked off-repo.** + +- Verified: `gateway` runs `dashboards-agent` (active/enabled, `10.1.1.27:9120` → `/health` 200, `/snapshot` 401 no-token / 200 bearer on-host, token matches `/etc/dashboards-agent.token`); `driveripper`/`deskwork` inactive, `sudo -n` → "a password is required", `:9120` empty. +- New measurement (transient runtime rule on gateway `firewall-cmd --add-port=9120/tcp`, **reverted** — runtime+permanent port lists verified back to the original set, no rich rules): laptop still gets no SYN through, while `:22`/`:80` on the same address answer in ms ⇒ the block is the lab gateway `10.2.0.1` port filter, not this app/host firewall/agent. No other live change made (no sudoers, no upsd, no UPS command). +- Tests/lint (exact): `bash .agents/validate.sh` → **EXIT 0** — `uv run pytest --cov --cov-fail-under=94` **793 passed, 11 skipped, coverage 98.33 %**; `ruff check src tests agent` clean; `pyright` 0 errors; `node --check` clean; hygiene clean; `uv pip list --outdated` empty (bumped fastapi 0.143.3→0.143.5, `uv.lock` only). +- `uv run pytest -m live --live` (`.env.dev` sourced): **4 passed, 1 skipped, 6 failed** — all six are `:9120`/`:3051` connect timeouts; zero parse/contract failures. +- Dev server restarted on current code: engines `ok`; 3×`host` + 3×`disks` + `nut` `DOWN` with real reasons (designed degradation, stub rows intact, no credential leakage). +- Completion criteria: screenshots ✓ (`data/logs/phase10-{index,system,disks,llm,ups}-live.png`); tunnel-vocabulary grep ✓ outside `.agents/` history (PLAN.md locked line, phase overviews, past reports; fixtures/`uv.lock` hits are digit substrings — `tests/test_deploy_scripts.py` guards operational files); no page-payload change ✓ (contract tests). **Not met, operator-side:** agents on all three hosts, `LIST UPS` from the laptop, `dev-server.sh status` all-ok, live suite green. +- Fixed in this pass: `sources/nut.py` unreachable-upsd message no longer prints a bare `(TimeoutError: )` (now `(timed out after 4s)` / `(ConnectionRefusedError: …)`), pinned by a new test + extended refused-port test. +- Docs: README verified consistent (every env var in `config.py` documented, routes/flags/`--with-llm`/pager sizes checked) — no changes needed. AGENTS.md fixed 4 inconsistencies: per-host sudo reality (gateway grants `NOPASSWD ALL`, which is why it installed without `agent-sudoers`; driveripper scoped; deskwork unrelated), the now-proven firewall finding, `/system` table typo ("agent per host"), gap-4 DB/WAL sizes + test count. `.env.example` `upsd.users` sample now one `instcmds` per line (it contradicted README and the in-app copy). `.env.dev` (gitignored) comment corrected to match the evidence. +- Flagged, untouched (locked): `.agents/PLAN.md` L92–95 still says upsd is "forwarded to `127.0.0.1:3494` in dev" — false after this phase; evidence + exact operator commands appended to `data/logs/NOTES-verify.md` §11. +- Next pending phase: `11_admin_login`. diff --git a/.agents/reports/10_dev_direct_dial/10_dev_direct_dial__00_phase.a1.md.status b/.agents/reports/10_dev_direct_dial/10_dev_direct_dial__00_phase.a1.md.status new file mode 100644 index 0000000..9766475 --- /dev/null +++ b/.agents/reports/10_dev_direct_dial/10_dev_direct_dial__00_phase.a1.md.status @@ -0,0 +1 @@ +ok diff --git a/.agents/reports/10_dev_direct_dial/10_dev_direct_dial__00_phase.a1.validate b/.agents/reports/10_dev_direct_dial/10_dev_direct_dial__00_phase.a1.validate new file mode 100644 index 0000000..a2cd931 --- /dev/null +++ b/.agents/reports/10_dev_direct_dial/10_dev_direct_dial__00_phase.a1.validate @@ -0,0 +1,52 @@ +== uv run pytest --cov (hermetic: no network, live tests skip) == +........................................................................ [ 8%] +........................................................................ [ 17%] +........................................................................ [ 26%] +........................................................................ [ 35%] +........................................................................ [ 44%] +........................................................................ [ 53%] +..................................sssssssssss........................... [ 62%] +........................................................................ [ 71%] +........................................................................ [ 80%] +........................................................................ [ 89%] +........................................................................ [ 98%] +............ [100%] +=============================== warnings summary =============================== +.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1 + /var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead. + from starlette.testclient import TestClient as TestClient # noqa + +-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html +================================ tests coverage ================================ +_______________ coverage: platform linux, python 3.14.4-final-0 ________________ + +Name Stmts Miss Cover Missing +------------------------------------------------------------------ +src/dashboards/__init__.py 3 0 100% +src/dashboards/__main__.py 14 1 93% 28 +src/dashboards/alerts.py 176 0 100% +src/dashboards/api.py 221 0 100% +src/dashboards/app.py 101 0 100% +src/dashboards/collectors.py 155 5 97% 141-142, 146-147, 164 +src/dashboards/config.py 115 0 100% +src/dashboards/debug.py 14 0 100% +src/dashboards/poller.py 165 3 98% 192-194, 279 +src/dashboards/prom.py 105 5 95% 107, 144, 158, 161, 164 +src/dashboards/rates.py 37 1 97% 51 +src/dashboards/severity.py 49 0 100% +src/dashboards/sources/__init__.py 0 0 100% +src/dashboards/sources/disks.py 130 0 100% +src/dashboards/sources/host.py 199 1 99% 147 +src/dashboards/sources/llm.py 150 1 99% 228 +src/dashboards/sources/nut.py 390 20 95% 137, 151, 157, 169, 175, 187, 195, 208, 279-280, 282, 498, 510, 545, 578, 582, 639-640, 645, 657 +src/dashboards/sources/nut_sim.py 134 0 100% +src/dashboards/store.py 178 2 99% 250-251 +------------------------------------------------------------------ +TOTAL 2336 39 98% +Required test coverage of 94% reached. Total coverage: 98.33% +793 passed, 11 skipped, 1 warning in 59.94s +== uv run ruff check src tests agent == +All checks passed! +== uv run pyright == +0 errors, 0 warnings, 0 informations +== node --check src/dashboards/web/static/*.js == diff --git a/.agents/reports/10_dev_direct_dial/10_dev_direct_dial__01_sudoers_grant_and_installer_hint.a1.err b/.agents/reports/10_dev_direct_dial/10_dev_direct_dial__01_sudoers_grant_and_installer_hint.a1.err new file mode 100644 index 0000000..e69de29 diff --git a/.agents/reports/10_dev_direct_dial/10_dev_direct_dial__01_sudoers_grant_and_installer_hint.a1.md b/.agents/reports/10_dev_direct_dial/10_dev_direct_dial__01_sudoers_grant_and_installer_hint.a1.md new file mode 100644 index 0000000..bbae89c --- /dev/null +++ b/.agents/reports/10_dev_direct_dial/10_dev_direct_dial__01_sudoers_grant_and_installer_hint.a1.md @@ -0,0 +1,16 @@ +**Implemented** +- `deploy/agent-sudoers` (new, nut-sudoers house style): WHY block, canonical staging path `/home/ops/agent-install`, the single rule `ops ALL=(root) NOPASSWD: /home/ops/agent-install/deploy/install-agent.sh`, two-stage check-first install (`visudo -cf` → `install -m 0440 -o root -g root /etc/sudoers.d/dashboards-agent` → re-check), a "make the grant mean something" `chown -R root:root` note, and a "deliberately NOT granted" block (visudo/sh/tee, other `systemctl`, the token file, user*/groupmod, `upscmd`/`upsrw`). +- `deploy/install-agent.sh`: added resolved `SCRIPT_PATH` (realpath + fallback) and `SUDOERS_SOURCE`; `need_root`'s non-root path now prints the exact NOPASSWD line **for its own resolved path**, the two install steps, the pointer to `deploy/agent-sudoers`, and a copy-pasteable re-run command; exit code 1 and root behaviour unchanged (call sites now pass flags so the hint is absolute). Header help names the grant; dropped the stale "ssh tunnel in" parenthetical and marked `--status` as not needing root. +- `tests/test_deploy_scripts.py` (new, 16 tests): grant file is exactly one rule at the canonical path, sections/boundaries present, `visudo -cf` passes; installer exits 1 with empty stdout, prints a line identical to the file apart from the path (relative and absolute invocations), names the two-stage install, `--help` points at the file; `bash -n` over `deploy/*.sh` + `scripts/*.sh`. +- `data/logs/NOTES-sudoers.md` (+ `phase10-*` artifacts, gitignored): the real diff — identical apart from the path (path-normalised diff is empty). + +**Verification** +- `visudo -cf deploy/agent-sudoers` → parsed OK; `bash -n deploy/install-agent.sh` → clean. +- `bash deploy/install-agent.sh --bind 127.0.0.1` (as uid 1000) → exit 1, prints `ops ALL=(root) NOPASSWD: /var/home/.../deploy/install-agent.sh`. +- `bash .agents/validate.sh` → rc 0: `785 passed, 11 skipped`, coverage **98.33 %** (gate 94 %), ruff clean, pyright 0 errors, `node --check` clean, hygiene clean. + +**Docs**: no doc impact — README.md/AGENTS.md deliberately untouched (task 05 owns the dev-workflow rewrite). Noted in the NOTES file that AGENTS.md's "769 passed" is now 785 so the final pass refreshes it. + +**Decisions/deviations**: grant names `ops` literally (only the path is substituted, so the printed line matches the canonical one); `--token` documented as the one flag never used (installer mints + prints once); added an honest note that the grant is only a boundary once the staged copy is root-owned. **No live-infrastructure changes** — the sudoers line needs root and is the phase's stated user precondition; nothing was touched on the fleet. + +**Next pending task**: `10_dev_direct_dial/02_install_agents_on_fleet.md` (blocked on the user adding the `deploy/agent-sudoers` line as root on each host; task 02 must stop cleanly with instructions if it is missing). diff --git a/.agents/reports/10_dev_direct_dial/10_dev_direct_dial__01_sudoers_grant_and_installer_hint.a1.md.status b/.agents/reports/10_dev_direct_dial/10_dev_direct_dial__01_sudoers_grant_and_installer_hint.a1.md.status new file mode 100644 index 0000000..9766475 --- /dev/null +++ b/.agents/reports/10_dev_direct_dial/10_dev_direct_dial__01_sudoers_grant_and_installer_hint.a1.md.status @@ -0,0 +1 @@ +ok diff --git a/.agents/reports/10_dev_direct_dial/10_dev_direct_dial__01_sudoers_grant_and_installer_hint.a1.validate b/.agents/reports/10_dev_direct_dial/10_dev_direct_dial__01_sudoers_grant_and_installer_hint.a1.validate new file mode 100644 index 0000000..53d3c6b --- /dev/null +++ b/.agents/reports/10_dev_direct_dial/10_dev_direct_dial__01_sudoers_grant_and_installer_hint.a1.validate @@ -0,0 +1,52 @@ +== uv run pytest --cov (hermetic: no network, live tests skip) == +........................................................................ [ 9%] +........................................................................ [ 18%] +........................................................................ [ 27%] +........................................................................ [ 36%] +........................................................................ [ 45%] +........................................................................ [ 54%] +...........................sssssssssss.................................. [ 63%] +........................................................................ [ 72%] +........................................................................ [ 81%] +........................................................................ [ 90%] +........................................................................ [ 99%] +.... [100%] +=============================== warnings summary =============================== +.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1 + /var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead. + from starlette.testclient import TestClient as TestClient # noqa + +-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html +================================ tests coverage ================================ +_______________ coverage: platform linux, python 3.14.4-final-0 ________________ + +Name Stmts Miss Cover Missing +------------------------------------------------------------------ +src/dashboards/__init__.py 3 0 100% +src/dashboards/__main__.py 14 1 93% 28 +src/dashboards/alerts.py 176 0 100% +src/dashboards/api.py 221 0 100% +src/dashboards/app.py 101 0 100% +src/dashboards/collectors.py 155 5 97% 141-142, 146-147, 164 +src/dashboards/config.py 115 0 100% +src/dashboards/debug.py 14 0 100% +src/dashboards/poller.py 165 3 98% 192-194, 279 +src/dashboards/prom.py 105 5 95% 107, 144, 158, 161, 164 +src/dashboards/rates.py 37 1 97% 51 +src/dashboards/severity.py 49 0 100% +src/dashboards/sources/__init__.py 0 0 100% +src/dashboards/sources/disks.py 130 0 100% +src/dashboards/sources/host.py 199 1 99% 147 +src/dashboards/sources/llm.py 150 1 99% 228 +src/dashboards/sources/nut.py 389 20 95% 128, 142, 148, 160, 166, 178, 186, 199, 270-271, 273, 489, 501, 536, 569, 573, 630-631, 636, 648 +src/dashboards/sources/nut_sim.py 134 0 100% +src/dashboards/store.py 178 2 99% 250-251 +------------------------------------------------------------------ +TOTAL 2335 39 98% +Required test coverage of 94% reached. Total coverage: 98.33% +785 passed, 11 skipped, 1 warning in 58.73s +== uv run ruff check src tests agent == +All checks passed! +== uv run pyright == +0 errors, 0 warnings, 0 informations +== node --check src/dashboards/web/static/*.js == diff --git a/.agents/reports/10_dev_direct_dial/10_dev_direct_dial__02_install_agents_on_fleet.a1.err b/.agents/reports/10_dev_direct_dial/10_dev_direct_dial__02_install_agents_on_fleet.a1.err new file mode 100644 index 0000000..e69de29 diff --git a/.agents/reports/10_dev_direct_dial/10_dev_direct_dial__02_install_agents_on_fleet.a1.md b/.agents/reports/10_dev_direct_dial/10_dev_direct_dial__02_install_agents_on_fleet.a1.md new file mode 100644 index 0000000..9ae8b8e --- /dev/null +++ b/.agents/reports/10_dev_direct_dial/10_dev_direct_dial__02_install_agents_on_fleet.a1.md @@ -0,0 +1,16 @@ +**Implemented** +- Tunnels stopped; ad-hoc agents gone on all three hosts (`:9120` empty everywhere); `deploy/`+`agent/` staged at the canonical `~/agent-install/` on all three (task file's `rsync deploy/ agent/` trailing slashes would have flattened the tree — used `rsync -a --delete deploy agent`). +- **Fixed 2 defects that made `deploy/install-agent.sh` unable to install on any host**, found by the first live run: the install body was a bash function named `install`, so `install -d` recursed forever (1 700 repeats, nothing installed); and `DevicePolicy=strict` without `/dev/null` kills the unit at spawn (`status=208/STDIN`, systemd 257 — proven with two transient `systemd-run` probes). Plus a latent third: token file `0600 root:root` is unreadable by the service that opens the path itself → now `0640 root:dashboards-agent`. +- **gateway: INSTALLED** — `dashboards-agent` active + enabled, bound `10.1.1.27:9120`, `/health` ok, `/snapshot` 401 without / 200 with bearer; deployed unit matches the repo template; token captured into `.env.dev` (0600, gitignored), not in NOTES. +- **driveripper + deskwork: BLOCKED** — precondition unmet (`/etc/sudoers.d/dashboards-agent` absent, `sudo -n … --status` → "a password is required"). Stopped cleanly; exact `visudo`/`install` lines + the one `firewall-cmd` line per host are in `data/logs/NOTES-agent-install.md`. **No sudoers or firewall edit was attempted.** +- Laptop-side reach is blocked on **all three** hosts' firewalld (9120/tcp closed; control: gateway:80 answers 301) — recorded with both the bare-port and `/21`-scoped rich-rule commands, not guessed. + +**Tests / lint / coverage** +- `uv run pytest` → **792 passed, 11 skipped**; `uv run pytest --cov --cov-fail-under=94` → **98.33%**; `uv run ruff check src tests agent` → passed; `uv run pyright` → 0 errors; `node --check` all page scripts ok; `bash -n` all scripts ok. +- New in `tests/test_deploy_scripts.py`: install body run end-to-end in a `tmp_path` sandbox (paths rewritten, root boundary stubbed) asserting it returns, lands the agent, writes `DeviceAllow=/dev/null`, prints the token once, leaves it `0640`; plus a static "no shell function named after a coreutil" check. + +**Docs**: README (`0640 root:dashboards-agent`, device-policy rationale, `deploy/agent-sudoers` pointer); AGENTS.md (deployment paragraph, fleet SMART paragraph — gateway's installed agent now answers `smartctl_available: false` — and a transport mid-transition note pointing at NOTES; task 05 still owns the full direct-dial rewrite). + +**Deviations / notes** +- `uv run pytest -m live --live` currently fails 6 — all transport (dead loopback tunnel ports + stopped `3494` forward), the state the phase expects for blocked hosts; green after the grant + task 04/06. Dev server is up and reports those sources red with real reasons. +- Next pending: **re-run task 02 for driveripper/deskwork after the user adds the sudoers grant**, then `.agents/phases/todo/10_dev_direct_dial/03_upsd_acl_for_laptop.md`. diff --git a/.agents/reports/10_dev_direct_dial/10_dev_direct_dial__02_install_agents_on_fleet.a1.md.status b/.agents/reports/10_dev_direct_dial/10_dev_direct_dial__02_install_agents_on_fleet.a1.md.status new file mode 100644 index 0000000..9766475 --- /dev/null +++ b/.agents/reports/10_dev_direct_dial/10_dev_direct_dial__02_install_agents_on_fleet.a1.md.status @@ -0,0 +1 @@ +ok diff --git a/.agents/reports/10_dev_direct_dial/10_dev_direct_dial__02_install_agents_on_fleet.a1.validate b/.agents/reports/10_dev_direct_dial/10_dev_direct_dial__02_install_agents_on_fleet.a1.validate new file mode 100644 index 0000000..97bd910 --- /dev/null +++ b/.agents/reports/10_dev_direct_dial/10_dev_direct_dial__02_install_agents_on_fleet.a1.validate @@ -0,0 +1,52 @@ +== uv run pytest --cov (hermetic: no network, live tests skip) == +........................................................................ [ 8%] +........................................................................ [ 17%] +........................................................................ [ 26%] +........................................................................ [ 35%] +........................................................................ [ 44%] +........................................................................ [ 53%] +..................................sssssssssss........................... [ 62%] +........................................................................ [ 71%] +........................................................................ [ 80%] +........................................................................ [ 89%] +........................................................................ [ 98%] +........... [100%] +=============================== warnings summary =============================== +.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1 + /var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead. + from starlette.testclient import TestClient as TestClient # noqa + +-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html +================================ tests coverage ================================ +_______________ coverage: platform linux, python 3.14.4-final-0 ________________ + +Name Stmts Miss Cover Missing +------------------------------------------------------------------ +src/dashboards/__init__.py 3 0 100% +src/dashboards/__main__.py 14 1 93% 28 +src/dashboards/alerts.py 176 0 100% +src/dashboards/api.py 221 0 100% +src/dashboards/app.py 101 0 100% +src/dashboards/collectors.py 155 5 97% 141-142, 146-147, 164 +src/dashboards/config.py 115 0 100% +src/dashboards/debug.py 14 0 100% +src/dashboards/poller.py 165 3 98% 192-194, 279 +src/dashboards/prom.py 105 5 95% 107, 144, 158, 161, 164 +src/dashboards/rates.py 37 1 97% 51 +src/dashboards/severity.py 49 0 100% +src/dashboards/sources/__init__.py 0 0 100% +src/dashboards/sources/disks.py 130 0 100% +src/dashboards/sources/host.py 199 1 99% 147 +src/dashboards/sources/llm.py 150 1 99% 228 +src/dashboards/sources/nut.py 389 20 95% 128, 142, 148, 160, 166, 178, 186, 199, 270-271, 273, 489, 501, 536, 569, 573, 630-631, 636, 648 +src/dashboards/sources/nut_sim.py 134 0 100% +src/dashboards/store.py 178 2 99% 250-251 +------------------------------------------------------------------ +TOTAL 2335 39 98% +Required test coverage of 94% reached. Total coverage: 98.33% +792 passed, 11 skipped, 1 warning in 59.37s +== uv run ruff check src tests agent == +All checks passed! +== uv run pyright == +0 errors, 0 warnings, 0 informations +== node --check src/dashboards/web/static/*.js == diff --git a/.agents/reports/10_dev_direct_dial/10_dev_direct_dial__03_upsd_acl_for_laptop.a1.err b/.agents/reports/10_dev_direct_dial/10_dev_direct_dial__03_upsd_acl_for_laptop.a1.err new file mode 100644 index 0000000..e69de29 diff --git a/.agents/reports/10_dev_direct_dial/10_dev_direct_dial__03_upsd_acl_for_laptop.a1.md b/.agents/reports/10_dev_direct_dial/10_dev_direct_dial__03_upsd_acl_for_laptop.a1.md new file mode 100644 index 0000000..0c3e242 --- /dev/null +++ b/.agents/reports/10_dev_direct_dial/10_dev_direct_dial__03_upsd_acl_for_laptop.a1.md @@ -0,0 +1,13 @@ +**Task 03 outcome: repo/records side complete and green; the prescribed live write was proven a no-op and the laptop leg is blocked off-host (details below).** + +- Read-first done: `data/logs/upsd.conf.before` (369 B, `LISTEN 0.0.0.0/:: 3051`, no ACL lines); live file re-read + `diff` → **identical**; `upsd.users`/all `/etc/ups` mtimes unchanged; `nut-server` still `active`, `NRestarts=0`, `upsc … ups.status` = `OL` on both UPSes. +- Ranges derived: hosts `10.1.0.0/21` (`bridge0 10.1.0.126/21`), laptop `10.2.1.113` in `10.2.0.0/22` (ssh source + `ip -o addr`); v6 `…:101::/64` vs `…:102::/64`. +- **Did not append the two `ACL` lines — they enforce nothing on this upstream.** `nut-2.8.5-9.fc43` is built `--without-wrap`; `man 5 upsd.conf` has no ACL section; sandboxed `NUT_CONFPATH` runs (live service untouched) show upsd logging `ACL in upsd.conf is no longer supported - switch to LISTEN` per line and then starting anyway → a config that claims restriction, logs an error against itself on every restart, and changes no packet. On 2.8 the mechanism is `LISTEN` + firewall, both already correct here. +- Firewall (task step 5): `firewalld` running, `FedoraServer` on `bridge0/enp3s0`, **`3051/tcp` already allowed** (runtime + permanent), no rich/direct rules → nothing to add; the optional `--remove-port` + 2 rich-rule scoping is recorded verbatim but deliberately not applied (no reachable behaviour changes, deny path unverifiable from here, prod + `upsmon` dial loopback). +- **BLOCKER (operator, off-fleet):** from the laptop, `nc 10.1.0.126 3051` (v4 and v6) times out and **no SYN ever arrives** — proven by a 0.5 s `ss -tan sport = :3051` loop during 4 attempts (only pre-existing `[::1]` sessions) beside control probes where the same laptop's payload to `:1234` landed in an `ncat` log and `:8004` RST in 10 ms. The drop is the lab gateway `10.2.0.1` (`d8:b3:70:85:53:e6`, UniFi) between `10.2.0.0/22` and `10.1.0.0/21`: it forwards 22/1234/8000-8004, drops 1235/3051/3493/5901. No `ops` sudo on a fleet host touches it. +- Objective half already true: from `deskwork` (host range) `printf 'LIST UPS\n' | nc 10.1.0.126 3051` → `BEGIN LIST UPS … END LIST UPS`. +- All evidence, the operator's exact one rule (UniFi LAN→LAN allow 3051/tcp from `10.2.0.0/22` to `10.1.0.126`), the three interim options, and the two premise corrections are in `data/logs/NOTES-upsd-acl.md`. No Python changes, so no new tests were applicable. +- Gate: `bash .agents/validate.sh` → **exit 0** (`uv run pytest --cov --cov-fail-under=94` → **792 passed, 11 skipped, 98.33%**; `ruff check src tests agent` passed; `pyright` 0 errors; `node --check` ok; hygiene gates ok); `uv pip list --outdated` empty; dev server still up (sources red with real reasons — task 02's in-between state, untouched). +- Docs: **AGENTS.md** (NUT paragraph: 2.8 has no `ACL`, restriction is firewall-only, non-local reads are the norm, host range reaches 3051 / laptop range does not; transport bullet now points at `NOTES-upsd-acl.md`) and **README.md** (Battery-test section: `ACL`/`ACCEPT`/`REJECT` removed upstream, don't scope by client in `upsd.conf`). +- Completion criteria met: no `upsd.users` write, nut-server healthy/unrestarted, suite green, docs accurate. **Not met, and not achievable from this repo:** "diff shows exactly the two ACL lines" (would enforce nothing) and "laptop `nc` answers `BEGIN LIST UPS`" (gateway port filter). Task 04 should keep `NUT_HOST` pending that operator rule; the derived ranges are ready in the NOTES file. +- Next pending: operator rule on gateway `10.2.0.1` (+ task 02's still-missing sudoers grant on driveripper/deskwork), then `.agents/phases/todo/10_dev_direct_dial/04_env_dev_direct_dial.md`. diff --git a/.agents/reports/10_dev_direct_dial/10_dev_direct_dial__03_upsd_acl_for_laptop.a1.md.status b/.agents/reports/10_dev_direct_dial/10_dev_direct_dial__03_upsd_acl_for_laptop.a1.md.status new file mode 100644 index 0000000..9766475 --- /dev/null +++ b/.agents/reports/10_dev_direct_dial/10_dev_direct_dial__03_upsd_acl_for_laptop.a1.md.status @@ -0,0 +1 @@ +ok diff --git a/.agents/reports/10_dev_direct_dial/10_dev_direct_dial__03_upsd_acl_for_laptop.a1.validate b/.agents/reports/10_dev_direct_dial/10_dev_direct_dial__03_upsd_acl_for_laptop.a1.validate new file mode 100644 index 0000000..c17eb16 --- /dev/null +++ b/.agents/reports/10_dev_direct_dial/10_dev_direct_dial__03_upsd_acl_for_laptop.a1.validate @@ -0,0 +1,52 @@ +== uv run pytest --cov (hermetic: no network, live tests skip) == +........................................................................ [ 8%] +........................................................................ [ 17%] +........................................................................ [ 26%] +........................................................................ [ 35%] +........................................................................ [ 44%] +........................................................................ [ 53%] +..................................sssssssssss........................... [ 62%] +........................................................................ [ 71%] +........................................................................ [ 80%] +........................................................................ [ 89%] +........................................................................ [ 98%] +........... [100%] +=============================== warnings summary =============================== +.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1 + /var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead. + from starlette.testclient import TestClient as TestClient # noqa + +-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html +================================ tests coverage ================================ +_______________ coverage: platform linux, python 3.14.4-final-0 ________________ + +Name Stmts Miss Cover Missing +------------------------------------------------------------------ +src/dashboards/__init__.py 3 0 100% +src/dashboards/__main__.py 14 1 93% 28 +src/dashboards/alerts.py 176 0 100% +src/dashboards/api.py 221 0 100% +src/dashboards/app.py 101 0 100% +src/dashboards/collectors.py 155 5 97% 141-142, 146-147, 164 +src/dashboards/config.py 115 0 100% +src/dashboards/debug.py 14 0 100% +src/dashboards/poller.py 165 3 98% 192-194, 279 +src/dashboards/prom.py 105 5 95% 107, 144, 158, 161, 164 +src/dashboards/rates.py 37 1 97% 51 +src/dashboards/severity.py 49 0 100% +src/dashboards/sources/__init__.py 0 0 100% +src/dashboards/sources/disks.py 130 0 100% +src/dashboards/sources/host.py 199 1 99% 147 +src/dashboards/sources/llm.py 150 1 99% 228 +src/dashboards/sources/nut.py 389 20 95% 128, 142, 148, 160, 166, 178, 186, 199, 270-271, 273, 489, 501, 536, 569, 573, 630-631, 636, 648 +src/dashboards/sources/nut_sim.py 134 0 100% +src/dashboards/store.py 178 2 99% 250-251 +------------------------------------------------------------------ +TOTAL 2335 39 98% +Required test coverage of 94% reached. Total coverage: 98.33% +792 passed, 11 skipped, 1 warning in 59.63s +== uv run ruff check src tests agent == +All checks passed! +== uv run pyright == +0 errors, 0 warnings, 0 informations +== node --check src/dashboards/web/static/*.js == diff --git a/.agents/reports/10_dev_direct_dial/10_dev_direct_dial__04_env_dev_direct_dial.a1.err b/.agents/reports/10_dev_direct_dial/10_dev_direct_dial__04_env_dev_direct_dial.a1.err new file mode 100644 index 0000000..e69de29 diff --git a/.agents/reports/10_dev_direct_dial/10_dev_direct_dial__04_env_dev_direct_dial.a1.md b/.agents/reports/10_dev_direct_dial/10_dev_direct_dial__04_env_dev_direct_dial.a1.md new file mode 100644 index 0000000..c440ad4 --- /dev/null +++ b/.agents/reports/10_dev_direct_dial/10_dev_direct_dial__04_env_dev_direct_dial.a1.md @@ -0,0 +1,19 @@ +**Implemented** +- `./scripts/dev-tunnels.sh stop` (no-op; already down), laptop has nothing on 9121/9122/9123/3494. +- `.env.dev` rewritten direct-shaped, mode `0600` kept: `HOST_TARGETS="driveripper=http://10.1.0.126:9120 deskwork=http://10.1.0.51:9120 gateway=http://@10.1.1.27:9120"`, `NUT_HOST=10.1.0.126`/`NUT_PORT=3051`, `DASH_PORT=8090` explicit, no `DEV_TUNNELS`/`DEV_FORWARDS`, comments naming where the upsd access rules actually live and why a laptop-IP change is the operator's gateway rule, not this file. +- Verified the gateway credential against the live service without printing it (sha256 equal to `/etc/dashboards-agent.token`; agent answers 401 without bearer / 200 with it, host-side) and `NUT_PASSWORD` against `[dashboards]` in the live `upsd.users` (`instcmds = all` — corrected the file's stale "battery-test only" comment). +- Restarted the dev server; `/api/config` redacts userinfo, `/api/system/snapshot` keeps all three hosts as `unreachable` stubs (`hosts_total 3 / hosts_live 0`), `/api/summary` alerts name the 4 dead sources, `/api/events` shows only the 7 switchover `unreachable` transitions. Token-hits = 0 across all 7 API surfaces. +- Evidence + the 3 remaining operator actions: `data/logs/NOTES-env-dev-direct-dial.md` (+ correction appended to `NOTES-agent-install.md`). + +**Tests / lint / coverage** +- `bash .agents/validate.sh` → exit 0: `uv run pytest --cov --cov-fail-under=94` **792 passed, 11 skipped, coverage 98.33%**; `ruff check src tests agent` clean; `pyright` 0 errors; `node --check` clean. `uv pip list --outdated` empty. +- `uv run pytest -m live --live` (with `.env.dev` sourced): 11 live tests, **5 pass (both engines), 6 fail on transport only** — 3 × `ConnectTimeout` to the `:9120`s, 3 × `cannot reach upsd at 10.1.0.126:3051`. Task 06 owns making these green. + +**Docs** — README + AGENTS updated only where this change falsified them: tunnel-based dev quickstart → direct dial (`install-agent.sh` + userinfo tokens + `NUT_PORT=3051`); removed the now-false "`dev-tunnels.sh start` … temporary way to see live data" fallback; `DEV_FORWARDS="3494:…"` fleet fact → direct dial; NUT "bind upsd" advice no longer points at a forward; hermetic count 769 → 792. `git grep dev-tunnels|DEV_TUNNELS|DEV_FORWARDS|9121|9122|9123|3494` now matches neither file. Task 05's file list (`.env.dev.example`, `scripts/dev-tunnels.sh`, `capture-fixtures.sh`, live-suite strings, `ups_contract.py`, `alerts.py` docstring, `test_config.py`, `.env.example`) left untouched. + +**Decisions / deviations** +- **No live changes to any host, firewall or service.** `driveripper`/`deskwork` stay BLOCKED (root-only sudoers grant missing — re-checked: `sudo -n … --status` still `a password is required`); gateway's `9120/tcp` stays closed. Opening it means a firewall edit on the lab router whose benefit is unverifiable from here (its policy is port-selective and drops 3051/3493 today, so 9120 may be dropped too), and tasks 02/03 both explicitly left firewall policy to the operator. No `NUT_SIMULATE`, no tunnel fallback — the reds are real, with the reason. +- Kept `driveripper`/`deskwork` in `HOST_TARGETS` **without** tokens (none exist) so the fleet still shows them as unreachable stubs instead of losing machines. +- Current live result, stated plainly: engines `ok`; every `system`/`disks`/`ups` source `DOWN` with a real reason — nothing that was healthy before the switchover regressed. `.env.dev.example` is still tunnel-shaped (task 05), so the README `cp` line currently points at a file awaiting that rewrite. + +**Next pending task:** `.agents/phases/todo/10_dev_direct_dial/05_retire_tunnels_update_scripts.md` diff --git a/.agents/reports/10_dev_direct_dial/10_dev_direct_dial__04_env_dev_direct_dial.a1.md.status b/.agents/reports/10_dev_direct_dial/10_dev_direct_dial__04_env_dev_direct_dial.a1.md.status new file mode 100644 index 0000000..9766475 --- /dev/null +++ b/.agents/reports/10_dev_direct_dial/10_dev_direct_dial__04_env_dev_direct_dial.a1.md.status @@ -0,0 +1 @@ +ok diff --git a/.agents/reports/10_dev_direct_dial/10_dev_direct_dial__04_env_dev_direct_dial.a1.validate b/.agents/reports/10_dev_direct_dial/10_dev_direct_dial__04_env_dev_direct_dial.a1.validate new file mode 100644 index 0000000..59038dc --- /dev/null +++ b/.agents/reports/10_dev_direct_dial/10_dev_direct_dial__04_env_dev_direct_dial.a1.validate @@ -0,0 +1,52 @@ +== uv run pytest --cov (hermetic: no network, live tests skip) == +........................................................................ [ 8%] +........................................................................ [ 17%] +........................................................................ [ 26%] +........................................................................ [ 35%] +........................................................................ [ 44%] +........................................................................ [ 53%] +..................................sssssssssss........................... [ 62%] +........................................................................ [ 71%] +........................................................................ [ 80%] +........................................................................ [ 89%] +........................................................................ [ 98%] +........... [100%] +=============================== warnings summary =============================== +.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1 + /var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead. + from starlette.testclient import TestClient as TestClient # noqa + +-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html +================================ tests coverage ================================ +_______________ coverage: platform linux, python 3.14.4-final-0 ________________ + +Name Stmts Miss Cover Missing +------------------------------------------------------------------ +src/dashboards/__init__.py 3 0 100% +src/dashboards/__main__.py 14 1 93% 28 +src/dashboards/alerts.py 176 0 100% +src/dashboards/api.py 221 0 100% +src/dashboards/app.py 101 0 100% +src/dashboards/collectors.py 155 5 97% 141-142, 146-147, 164 +src/dashboards/config.py 115 0 100% +src/dashboards/debug.py 14 0 100% +src/dashboards/poller.py 165 3 98% 192-194, 279 +src/dashboards/prom.py 105 5 95% 107, 144, 158, 161, 164 +src/dashboards/rates.py 37 1 97% 51 +src/dashboards/severity.py 49 0 100% +src/dashboards/sources/__init__.py 0 0 100% +src/dashboards/sources/disks.py 130 0 100% +src/dashboards/sources/host.py 199 1 99% 147 +src/dashboards/sources/llm.py 150 1 99% 228 +src/dashboards/sources/nut.py 389 20 95% 128, 142, 148, 160, 166, 178, 186, 199, 270-271, 273, 489, 501, 536, 569, 573, 630-631, 636, 648 +src/dashboards/sources/nut_sim.py 134 0 100% +src/dashboards/store.py 178 2 99% 250-251 +------------------------------------------------------------------ +TOTAL 2335 39 98% +Required test coverage of 94% reached. Total coverage: 98.33% +792 passed, 11 skipped, 1 warning in 54.19s +== uv run ruff check src tests agent == +All checks passed! +== uv run pyright == +0 errors, 0 warnings, 0 informations +== node --check src/dashboards/web/static/*.js == diff --git a/.agents/reports/10_dev_direct_dial/10_dev_direct_dial__05_retire_tunnels_update_scripts.a1.err b/.agents/reports/10_dev_direct_dial/10_dev_direct_dial__05_retire_tunnels_update_scripts.a1.err new file mode 100644 index 0000000..e69de29 diff --git a/.agents/reports/10_dev_direct_dial/10_dev_direct_dial__05_retire_tunnels_update_scripts.a1.md b/.agents/reports/10_dev_direct_dial/10_dev_direct_dial__05_retire_tunnels_update_scripts.a1.md new file mode 100644 index 0000000..68dbc4c --- /dev/null +++ b/.agents/reports/10_dev_direct_dial/10_dev_direct_dial__05_retire_tunnels_update_scripts.a1.md @@ -0,0 +1,17 @@ +All green. Final report: + +**Implemented** +- Deleted `scripts/dev-tunnels.sh` (working-tree `D`; harness stages) + removed its generated `data/tunnels/` leftovers. +- `scripts/capture-fixtures.sh`: sources `.env.dev` first, derives agents by parsing `HOST_TARGETS` (`name=url`, space/comma, userinfo token used as-is, `/snapshot` appended); skip note when unset; failure lines name the host, never the token; all tunnel hints gone. +- `tests/test_live_fleet.py` docstring → direct-dial loop; skips now `HOST_TARGETS is unset` / `no host agent answered`; `ups_contract.py`, `alerts.py` docstring, `test_config.py`, `test_alerts.py`, `test_api.py`, `test_disks_source.py` strings moved to the `:9120` direct shape; stale `tunnel-check` gone from `dev-server.sh` header. +- `.env.dev.example` rewritten direct-shaped (installer + `agent-sudoers` flow, userinfo tokens, `NUT_HOST`/`NUT_PORT=3051`, port-discovery + firewall notes, RFC 2606 hosts, never-commit warning); verified it parses through the app's own `parse_targets` (3 targets, headers set, redacted display URLs). +- New guards in `tests/test_deploy_scripts.py`: script stays deleted + no tunnel name/port in any operational file (scripts/deploy/src/agent/web/tests `.py`/README/AGENTS/example envs; fixtures & `.agents/` history excluded by design). +- **Deviation (rule 7 / task-03 findings):** `.env.example`'s "third option" is written as *bind LAN + firewall-scope the dashboard's source ranges*, not "ACL in `upsd.conf`" — NUT 2.8.5 has no `ACL` directive (logs "switch to LISTEN", enforces nothing); same truth README already carries. + +**Verification** — `bash .agents/validate.sh` → **EXIT 0**: `uv run pytest --cov --cov-fail-under=94` **792 passed, 11 skipped, 98.33 %**; `ruff` clean; `pyright` 0 errors; `node --check` clean. `bash -n` clean on both edited scripts; `uv pip list --outdated` empty; dev server restarted (`/api/health` sane: engines ok, hosts `DOWN` with real reasons — unchanged task-04 state; no live infra touched). + +**Docs** — AGENTS.md transport bullet rewritten ("dev is the prod topology, repo-wide", systemd-agent-via-grant + firewall-scoped upsd); README was already direct-shaped by task 04, verified consistent. + +**Grep criterion** — matches remain only in `.agents/` history (incl. locked `PLAN.md:94` "3494" — flagged for the final pass, untouched; `todo/11_admin_login/00_phase.md` names the vars only to forbid restoring them), `tests/fixtures/` verbatim captures, and `uv.lock` hashes. + +**Next pending task:** `.agents/phases/todo/10_dev_direct_dial/06_verify_direct_dial_live.md`. diff --git a/.agents/reports/10_dev_direct_dial/10_dev_direct_dial__05_retire_tunnels_update_scripts.a1.md.status b/.agents/reports/10_dev_direct_dial/10_dev_direct_dial__05_retire_tunnels_update_scripts.a1.md.status new file mode 100644 index 0000000..9766475 --- /dev/null +++ b/.agents/reports/10_dev_direct_dial/10_dev_direct_dial__05_retire_tunnels_update_scripts.a1.md.status @@ -0,0 +1 @@ +ok diff --git a/.agents/reports/10_dev_direct_dial/10_dev_direct_dial__05_retire_tunnels_update_scripts.a1.validate b/.agents/reports/10_dev_direct_dial/10_dev_direct_dial__05_retire_tunnels_update_scripts.a1.validate new file mode 100644 index 0000000..9983a57 --- /dev/null +++ b/.agents/reports/10_dev_direct_dial/10_dev_direct_dial__05_retire_tunnels_update_scripts.a1.validate @@ -0,0 +1,52 @@ +== uv run pytest --cov (hermetic: no network, live tests skip) == +........................................................................ [ 8%] +........................................................................ [ 17%] +........................................................................ [ 26%] +........................................................................ [ 35%] +........................................................................ [ 44%] +........................................................................ [ 53%] +..................................sssssssssss........................... [ 62%] +........................................................................ [ 71%] +........................................................................ [ 80%] +........................................................................ [ 89%] +........................................................................ [ 98%] +........... [100%] +=============================== warnings summary =============================== +.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1 + /var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead. + from starlette.testclient import TestClient as TestClient # noqa + +-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html +================================ tests coverage ================================ +_______________ coverage: platform linux, python 3.14.4-final-0 ________________ + +Name Stmts Miss Cover Missing +------------------------------------------------------------------ +src/dashboards/__init__.py 3 0 100% +src/dashboards/__main__.py 14 1 93% 28 +src/dashboards/alerts.py 176 0 100% +src/dashboards/api.py 221 0 100% +src/dashboards/app.py 101 0 100% +src/dashboards/collectors.py 155 5 97% 141-142, 146-147, 164 +src/dashboards/config.py 115 0 100% +src/dashboards/debug.py 14 0 100% +src/dashboards/poller.py 165 3 98% 192-194, 279 +src/dashboards/prom.py 105 5 95% 107, 144, 158, 161, 164 +src/dashboards/rates.py 37 1 97% 51 +src/dashboards/severity.py 49 0 100% +src/dashboards/sources/__init__.py 0 0 100% +src/dashboards/sources/disks.py 130 0 100% +src/dashboards/sources/host.py 199 1 99% 147 +src/dashboards/sources/llm.py 150 1 99% 228 +src/dashboards/sources/nut.py 389 20 95% 128, 142, 148, 160, 166, 178, 186, 199, 270-271, 273, 489, 501, 536, 569, 573, 630-631, 636, 648 +src/dashboards/sources/nut_sim.py 134 0 100% +src/dashboards/store.py 178 2 99% 250-251 +------------------------------------------------------------------ +TOTAL 2335 39 98% +Required test coverage of 94% reached. Total coverage: 98.33% +792 passed, 11 skipped, 1 warning in 59.97s +== uv run ruff check src tests agent == +All checks passed! +== uv run pyright == +0 errors, 0 warnings, 0 informations +== node --check src/dashboards/web/static/*.js == diff --git a/.agents/reports/10_dev_direct_dial/10_dev_direct_dial__06_verify_direct_dial_live.a1.err b/.agents/reports/10_dev_direct_dial/10_dev_direct_dial__06_verify_direct_dial_live.a1.err new file mode 100644 index 0000000..e69de29 diff --git a/.agents/reports/10_dev_direct_dial/10_dev_direct_dial__06_verify_direct_dial_live.a1.md b/.agents/reports/10_dev_direct_dial/10_dev_direct_dial__06_verify_direct_dial_live.a1.md new file mode 100644 index 0000000..43fd058 --- /dev/null +++ b/.agents/reports/10_dev_direct_dial/10_dev_direct_dial__06_verify_direct_dial_live.a1.md @@ -0,0 +1,14 @@ +**Implemented (task 06 — verify direct-dial live)** +- Restarted the dev server on direct dial and recorded `/api/health`: `llm:gufo`/`llm:vllm` **ok** with live data; 3 `host:*` + 3 `disks:*` + `nut` DOWN with their real reasons (identical to the task-02/03/04 blockers); 0 `store-error`/`database is locked` lines; credential sweep proves the gateway bearer token appears in **no** API response. +- Proved no tunnel port exists: `ss -ltn | grep -E ':(9121|9122|9123|3494)\s'` → empty; `git grep` for the tunnel names/ports → nothing outside `.agents/` (only digit substrings in `uv.lock` hashes and one SMART raw counter). +- Prod-side proof (read-only, no writes): gateway's installed unit `active`, `10.1.1.27:9120` `/health` 200 / `/snapshot` 401-no-token / 200-with-token; that live snapshot through the **production** parsers passes `assert_card_contract` and yields the documented `/disks` degradation (`vda`, `smart: null`, `smartctl_available: false`); upsd answers `LIST UPS` for both UPSes from loopback **and from deskwork**. +- Browser verification (visible Chromium): `data/logs/phase10-{index,system,disks,llm,ups}-live.png` — `/llm` fully live (fleet 2/2, KV bars, slot grid, TTFT/TPOT, history charts); `/ups` honest "none reporting", **no SIMULATED badge**. +- Wrote `data/logs/NOTES-verify.md` (health table, live-suite tally, probe evidence localising the block to the lab gateway's port allow-list, the 3 operator actions, and the **PLAN.md L92–95 staleness flag** — "forwarded to `127.0.0.1:3494` in dev" is now false; file untouched). + +**Tests/lint/coverage** — `bash .agents/validate.sh` → **EXIT 0**: `uv run pytest --cov --cov-fail-under=94` → **792 passed, 11 skipped, 98.33 %** (gate 94); `ruff check src tests agent` clean; `pyright` **0 errors**; `node --check` clean; `uv pip list --outdated` empty. + +**Deviation / stopped short (live-infra)** — `uv run pytest -m live --live` is **4 passed, 1 skipped, 6 failed**; all six failures are connect timeouts to `:9120`/`:3051`, none a parse or contract. Making them green needs root on driveripper/deskwork (sudoers grant), host `9120/tcp`, and a rule on the lab gateway `10.2.0.1` — none reachable by `ops` sudo, and the laptop-side probes show the gateway drops even host-allowed ports (e.g. 3478), so opening gateway's `9120` has unknown payoff for at most 2 of 7 red sources. I therefore made **no live change** (no firewall, service, config write or UPS command) and left the reversible one-line experiment + exact operator commands in §7–§8 of the notes. `test_every_host_agent_answers_and_parses` is deliberately not turned into a skip — it is the reachability signal this phase exists to produce. + +**Docs** — `AGENTS.md`: fleet bullet now also points at `data/logs/NOTES-verify.md` as the direct-dial verification record (re-verified facts unchanged). No README change needed (already direct-shaped; grep-clean). + +**Next pending task** — `.agents/phases/todo/11_admin_login/01_auth_primitives.md` (phase 10's tasks are all complete). diff --git a/.agents/reports/10_dev_direct_dial/10_dev_direct_dial__06_verify_direct_dial_live.a1.md.status b/.agents/reports/10_dev_direct_dial/10_dev_direct_dial__06_verify_direct_dial_live.a1.md.status new file mode 100644 index 0000000..9766475 --- /dev/null +++ b/.agents/reports/10_dev_direct_dial/10_dev_direct_dial__06_verify_direct_dial_live.a1.md.status @@ -0,0 +1 @@ +ok diff --git a/.agents/reports/10_dev_direct_dial/10_dev_direct_dial__06_verify_direct_dial_live.a1.validate b/.agents/reports/10_dev_direct_dial/10_dev_direct_dial__06_verify_direct_dial_live.a1.validate new file mode 100644 index 0000000..8135055 --- /dev/null +++ b/.agents/reports/10_dev_direct_dial/10_dev_direct_dial__06_verify_direct_dial_live.a1.validate @@ -0,0 +1,52 @@ +== uv run pytest --cov (hermetic: no network, live tests skip) == +........................................................................ [ 8%] +........................................................................ [ 17%] +........................................................................ [ 26%] +........................................................................ [ 35%] +........................................................................ [ 44%] +........................................................................ [ 53%] +..................................sssssssssss........................... [ 62%] +........................................................................ [ 71%] +........................................................................ [ 80%] +........................................................................ [ 89%] +........................................................................ [ 98%] +........... [100%] +=============================== warnings summary =============================== +.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1 + /var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead. + from starlette.testclient import TestClient as TestClient # noqa + +-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html +================================ tests coverage ================================ +_______________ coverage: platform linux, python 3.14.4-final-0 ________________ + +Name Stmts Miss Cover Missing +------------------------------------------------------------------ +src/dashboards/__init__.py 3 0 100% +src/dashboards/__main__.py 14 1 93% 28 +src/dashboards/alerts.py 176 0 100% +src/dashboards/api.py 221 0 100% +src/dashboards/app.py 101 0 100% +src/dashboards/collectors.py 155 5 97% 141-142, 146-147, 164 +src/dashboards/config.py 115 0 100% +src/dashboards/debug.py 14 0 100% +src/dashboards/poller.py 165 3 98% 192-194, 279 +src/dashboards/prom.py 105 5 95% 107, 144, 158, 161, 164 +src/dashboards/rates.py 37 1 97% 51 +src/dashboards/severity.py 49 0 100% +src/dashboards/sources/__init__.py 0 0 100% +src/dashboards/sources/disks.py 130 0 100% +src/dashboards/sources/host.py 199 1 99% 147 +src/dashboards/sources/llm.py 150 1 99% 228 +src/dashboards/sources/nut.py 389 20 95% 128, 142, 148, 160, 166, 178, 186, 199, 270-271, 273, 489, 501, 536, 569, 573, 630-631, 636, 648 +src/dashboards/sources/nut_sim.py 134 0 100% +src/dashboards/store.py 178 2 99% 250-251 +------------------------------------------------------------------ +TOTAL 2335 39 98% +Required test coverage of 94% reached. Total coverage: 98.33% +792 passed, 11 skipped, 1 warning in 59.49s +== uv run ruff check src tests agent == +All checks passed! +== uv run pyright == +0 errors, 0 warnings, 0 informations +== node --check src/dashboards/web/static/*.js == diff --git a/.env.dev.example b/.env.dev.example index 4c64549..ced58ad 100644 --- a/.env.dev.example +++ b/.env.dev.example @@ -1,18 +1,23 @@ -# Local development overrides. Copy to .env.dev and edit the host names to yours. +# Local development overrides. Copy to .env.dev and edit the addresses and tokens to yours. +# This file is gitignored and holds the fleet's bearer tokens: NEVER commit it. # cp .env.dev.example .env.dev -# ./scripts/dev-tunnels.sh start # ssh tunnels to each host's agent -# set -a; . .env.dev; set +a # then run the dashboard +# chmod 0600 .env.dev # -# DEV_TUNNELS entries are ::. -# DEV_FORWARDS entries are :: for a service already -# running there — upsd is the usual one, so the UPS page can read a real battery without -# 3493 being reachable from anywhere but this laptop. -# scripts/dev-tunnels.sh forwards each one to 127.0.0.1: so the dashboard on -# the laptop reads live fleet data with nothing listening on the hosts. -export DEV_TUNNELS="9121:driveripper-ops:9120 9122:deskwork-ops:9120 9123:gateway-ops:9120" -# The remote port is whatever upsd actually listens on — 3493 is NUT's stock default, and -# this fleet's driveripper answers on 3051, so check before copying the number. -export DEV_FORWARDS="3494:driveripper-ops:3051" +# Dev dials the fleet exactly the way the deployed dashboard does — no tunnels, no +# forwarded ports, nothing listening on localhost. Per monitored host, once: +# +# sudo deploy/install-agent.sh --bind
--port 9120 +# +# It installs agent/host_agent.py as the dashboards-agent systemd service, bound to the +# address you dial, and prints its bearer token exactly once. No root shell on the host? +# deploy/agent-sudoers is the one-line grant that lets the fleet's `ops` account run the +# installer (and running the installer without root prints that exact line). Put each +# printed token into HOST_TARGETS below as URL userinfo; config.parse_targets moves it +# into an Authorization header, so Target.url, /api/config and every error message stay +# credential-free. +# +# The host names below are placeholders (RFC 2606) — use your own. +export HOST_TARGETS="driveripper=http://TOKEN@driveripper.example.net:9120 deskwork=http://TOKEN@deskwork.example.net:9120 gateway=http://TOKEN@gateway.example.net:9120" # Physical-disk health (phase 09): scripts/capture-fixtures.sh records one # `smartctl -i -A -H` per whole disk per host into tests/fixtures/smartctl__.txt. @@ -27,26 +32,29 @@ export DEV_FORWARDS="3494:driveripper-ops:3051" # export DISKS_INTERVAL_SECONDS=60 # export DISKS_TIMEOUT_SECONDS=30 -# Do not assume upsd is on 3493 — nonstandard ports are common and a refused connection then -# reads as "upsd is broken" while the service is fine. Ask the host what actually speaks NUT: +# The engines are dialled directly on their hostnames, like the agents. +export GUFO_BASE_URL=http://deskwork.example.net:8731 +export VLLM_BASE_URL=http://driveripper.example.net:8000 + +# The UPS source: the real upsd, dialled directly. Do not assume NUT's stock 3493 — +# nonstandard ports are common and a refused connection then reads as "upsd is broken" +# while the service is fine (this fleet's driveripper answers on 3051). Ask the host what +# actually speaks NUT: # ss -ltn # what is listening # printf 'LIST UPS\n' | nc 127.0.0.1 PORT # the one that answers "BEGIN LIST UPS" # upsc -L localhost:PORT # names + descriptions of every UPS - -# Point the dashboard at those tunnels. -export HOST_TARGETS="driveripper=http://127.0.0.1:9121 deskwork=http://127.0.0.1:9122 gateway=http://127.0.0.1:9123" -# The engines are reached directly on their hostnames (nothing tunnels them here). If the -# laptop cannot route to them, add them to DEV_FORWARDS above -# (e.g. `8731:deskwork-ops:8731 8000:driveripper-ops:8000`) and point these at 127.0.0.1. -# The host names below are placeholders (RFC 2606) — use your own. -export GUFO_BASE_URL=http://deskwork.example.net:8731 -export VLLM_BASE_URL=http://driveripper.example.net:8000 -# The UPS source: the forwarded upsd above. NUT_PORT is the LOCAL port (3494), not 3493. -# Leave NUT_SIMULATE=1 only while upsd is unreachable — it labels every panel SIMULATED. -export NUT_HOST=127.0.0.1 -export NUT_PORT=3494 +# Who may connect is a FIREWALL question, not an upsd.conf one: NUT 2.8 has no `ACL` +# directive (it logs "switch to LISTEN" per line and enforces nothing), so the dashboard's +# source ranges are allowed in the firewall that covers the LISTEN address (a firewalld +# rule on the NUT host — not a line under /etc/ups). Reads are +# unauthenticated in stock NUT; SET/INSTCMD still need the upsd.users account below. +# A changed laptop IP can mean re-scoping that firewall rule — it is not a line in this +# file. +export NUT_HOST=driveripper.example.net +export NUT_PORT=3051 export NUT_USER=dashboards export NUT_PASSWORD=replace-me-on-the-host export NUT_ALLOW_ACTIONS=1 +# Leave NUT_SIMULATE=1 only while upsd is unreachable — it labels every panel SIMULATED. # export NUT_SIMULATE=1 export DASH_PORT=8090 diff --git a/.env.example b/.env.example index 9afb5e7..8444f92 100644 --- a/.env.example +++ b/.env.example @@ -52,16 +52,23 @@ RATE_WINDOW_SECONDS=10 # one looks exactly like a dead service: # ss -ltnH # what the NUT host is listening on # printf 'LIST UPS\n' | nc 127.0.0.1 PORT # NUT answers: BEGIN LIST UPS -# Keep it off the public interface: bind it to loopback (`LISTEN 127.0.0.1` in upsd.conf) and -# reach it with an ssh forward or a tailnet address, or bind a LAN address only if a firewall -# rule limits it to this host. +# Keep it off the public interface. Options, from most to least exposed: +# - bind loopback only (`LISTEN 127.0.0.1` in upsd.conf) and reach it over a tailnet address; +# - bind a LAN address and limit the dashboard's source range with a FIREWALL rule — this is +# the default here (upsd listens on the LAN, firewalld scopes 3051/tcp to the lab ranges). +# NUT 2.8 removed the `ACL`/`ACCEPT`/`REJECT` directives: on a build without TCP Wrappers +# an upsd.conf ACL line logs "ACL in upsd.conf is no longer supported - switch to LISTEN" +# once per line and changes no packet, so per-client restriction is never an upsd.conf job. # # Reads need no user in stock NUT. For the battery-test buttons add one, scoped as narrowly # as the buttons need — `instcmds` beats `actions = -`, which grants load.off and shutdown.* -# too: +# too. One command per line: `man 5 upsd.users` says to grant several commands "use multiple +# instances of the instcmds field", and a list on one line is not that. # [dashboards] # password = "..." -# instcmds = test.battery.start.quick test.battery.start.deep test.battery.stop +# instcmds = test.battery.start.quick +# instcmds = test.battery.start.deep +# instcmds = test.battery.stop # --------------------------------------------------------------------------- # NUT_HOST= NUT_PORT=3493 diff --git a/AGENTS.md b/AGENTS.md index af33a49..a7f98cf 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -25,7 +25,7 @@ file — plus an overview page that says which of the four you need: | Route | Question it answers | Source | | --------- | ------------------------------------------------------------------------------ | ------------------------------------------------- | | `/` | Is anything wrong, and where do I look first? | all four, in one request | -| `/system` | What are my machines doing — CPU, memory, disk, GPU, network, and the history | `agent/host_agent.py`, our own stdlib agent/host | +| `/system` | What are my machines doing — CPU, memory, disk, GPU, network, and the history | `agent/host_agent.py`, our own stdlib agent per host | | `/disks` | Are any of my physical disks failing, and which one — health, temp, hours, wear | the same agents, `smartctl -i -A -H` per whole drive (never a partition) | | `/llm` | Tokens/sec, active sessions, queue depth, KV pressure, latency percentiles | gufo `/metrics` + `/slots`, vLLM `/metrics` | | `/ups` | Battery health, runtime left, power events, battery-test buttons | NUT `upsd` (our own wire-protocol client; 3051 here, not 3493) | @@ -311,14 +311,55 @@ That scheme is the history API's key — treat it as a public interface. | `deskwork` | `deskwork-ops` | Strix Halo (Radeon 8060S iGPU), gufo `:8731` | | `gateway` | `gateway-ops` | router, no GPU | -- Agents run from `~/lab-agent/host_agent.py` on each host, forwarded to `127.0.0.1:9121/9122/9123` - by `scripts/dev-tunnels.sh` (`DEV_TUNNELS`). Nothing listens on the hosts in dev. +- **Transport is direct dial: dev is the prod topology, repo-wide (phase 10 task 05).** The + designed shape is one `dashboards-agent` **systemd service per host**, installed by `ops` + through the `deploy/agent-sudoers` grant with `deploy/install-agent.sh`, bound to + the address the dashboard dials, its bearer token carried in the `HOST_TARGETS` URL + userinfo; driveripper's upsd serves the LAN on 3051 with the host and laptop ranges allowed + by the **firewall**, never by a `upsd.conf` line. The old ssh-tunnel script is deleted and + neither `.env.dev` nor `.env.dev.example` declares a forwarded port — dev dials the same + addresses the deployed dashboard does, and `scripts/capture-fixtures.sh` reads + `HOST_TARGETS` from `.env.dev` instead of naming hosts of its own + (`tests/test_deploy_scripts.py` fails the day any tunnel name or port returns). Current state: `gateway` runs its installed service, bound to the address + `gateway.reeselink.com` resolves to, and answers `/health` + `/snapshot` **on the host** + (re-verified 2026-10-18, and the token in `.env.dev` matches `/etc/dashboards-agent.token`), + but the laptop cannot reach its `:9120` — and that is now proven to be nobody's firewall here: + opening `9120/tcp` in that host's own firewalld (a runtime rule, added and reverted in the + phase-10 final pass; runtime and permanent port lists are back to their original set) changed + nothing, because the SYN never arrives — `ss -tan sport = :9120` on the host shows only the + `LISTEN` — while the same laptop reaches `:22` and `:80` on the same address in milliseconds. + The drop is the port-selective inter-VLAN filter on the lab gateway `10.2.0.1`, the same one + that eats 3051/3493. On `driveripper` and `deskwork` the ad-hoc `~/lab-agent/host_agent.py` agents + are gone (stopped with the old tunnel sessions, nothing replaced them, `:9120` is empty on + both) because their `deploy/agent-sudoers` grant is not installed yet, so the installer + cannot run there. Every `system`/`disks` source is therefore `DOWN` with its real reason + and renders as an `unreachable` stub row — the designed degradation, recorded rather than + hidden — which is also why 6 of the `live` tests fail on transport while both engines pass. + The per-host record, the exact root commands and the one firewall line each host still + needs are in `data/logs/NOTES-agent-install.md`, how far the NUT wire reaches from each + network is in `data/logs/NOTES-upsd-acl.md`, and the end-to-end direct-dial verification + (health table, no-tunnel-port proof, live-suite tally, screenshots, and the probes that put + the block on the lab gateway's port allow-list rather than in this app) is in + `data/logs/NOTES-verify.md`. - Engines are reached directly: gufo `deskwork.reeselink.com:8731`, vLLM `driveripper.reeselink.com:8000`. - NUT 2.8.5 on driveripper: **configs live in `/etc/ups/`, not `/etc/nut/`**, and **upsd is on port 3051, not 3493**. Nothing listens on 3493 at all, so a default client reports *Connection refused* and it looks like a dead service — `ss -ltnH` plus `printf 'LIST UPS\n' - | nc 127.0.0.1 3051` (answers `BEGIN LIST UPS`) is how that was pinned down. Two UPSes are - exported over USB and read live through `DEV_FORWARDS="3494:driveripper-ops:3051"`: + | nc 127.0.0.1 3051` (answers `BEGIN LIST UPS`) is how that was pinned down. + **NUT 2.8 has no `ACL` directive** (this Fedora build is `--without-wrap`; upsd logs + `ACL in upsd.conf is no longer supported - switch to LISTEN` once per line and then ignores + it), so per-source restriction is a **firewall** rule and never an `upsd.conf` line — and + 2.8 does not refuse non-local clients the way older docs suggest: `LISTEN 0.0.0.0 3051` + + `LISTEN :: 3051` with `3051/tcp` in the `FedoraServer` zone answers any client routed to it + (reads are unauthenticated; `SET`/`INSTCMD` still need an `upsd.users` account). The host + range `10.1.0.0/21` does reach it today (verified from `deskwork`); the laptop's + `10.2.0.0/22` does **not** — the lab gateway `10.2.0.1` forwards 22/1234/8000-8004 and + silently drops 3051/3493 in both families, which no sudo on a fleet host can change, so the + laptop cannot yet dial `NUT_PORT=3051` (probe evidence and the operator's one rule: + `data/logs/NOTES-upsd-acl.md`). + Two UPSes are exported over USB and dialled directly as `NUT_HOST` + `NUT_PORT=3051` (from the + laptop that is still the blocked path above; from the host itself, and from the deployed + dashboard, it reads live): `apc` = APC Back-UPS RS 1500MS, `cyberpower` = CyberPower PR1500LCDRT2U (that is `ups.description`, the operator's label — the driver's own `identity.model` answers `GX1500U`, so the description is what identifies the box). @@ -331,19 +372,28 @@ That scheme is the history API's key — treat it as a public interface. Both UPSes are recorded verbatim (`LIST UPS`/`LIST VAR`/`LIST CMD`, serials scrubbed) as `tests/fixtures/nut_.vars` by `./scripts/capture-fixtures.sh`, and the wire quotes a field only when it must — `VAR apc ups.status "OL"`, not three quoted fields. -- Host access is via ssh keys as `ops`. Passwordless sudo exists but is command-scoped; the - set we rely on is documented in `deploy/nut-sudoers` (additions are listed there for the - work that is still blocked). +- Host access is via ssh keys as `ops`, and **passwordless sudo is different on every host** — + that is what decides how much of the fleet an install can finish unattended. `gateway` grants + `(ALL) NOPASSWD: ALL`, which is why its agent installed in phase 10 with no + `deploy/agent-sudoers` line present; `driveripper` grants a command-scoped set (`smartctl`, + `upsc`, `udevadm`, the `/etc/ups` reads, and the `tee /etc/ups/upsd.users|upsd.conf` + + `systemctl restart|reload nut-server` pair documented in `deploy/nut-sudoers`); `deskwork` + grants an unrelated set (`btrfs scrub`, `snapper`, `setfacl /srv/*`, `osbuild-auto.sh`). + Neither `driveripper` nor `deskwork` can run the installer or `visudo`, so those two installs + — and the `firewall-cmd` line each still needs — stay with the operator. - **SMART is captured from real hardware; the live agents still cannot read it.** `smartctl` 7.5 is installed on deskwork and driveripper. The `ops` account (which runs the dev agents) is in no `disk` group; driveripper grants it `(root) NOPASSWD: /usr/sbin/smartctl*` and deskwork grants nothing, so the agent's own unprivileged `smartctl -i -A -H` — it shells the binary and never calls sudo — answers *Permission denied* (exit 2) on every host and `smart` - stays `null` end to end: the designed degradation, not a bug. The three deployed agents also - predate the SMART pass, so the `disks` source today yields one row per physical disk (16 on - driveripper, 1 on deskwork, `vda` on gateway) with `smart: null`, `smart_errors: []` and - `smartctl_available: null` — "nobody told us either way", which is why that flag is - three-way and not a boolean. + stays `null` end to end: the designed degradation, not a bug. The agents deployed on + driveripper and deskwork also predate the SMART pass, so the `disks` source today yields one row + per physical disk (16 on driveripper, 1 on deskwork, `vda` on gateway) with `smart: null`, + `smart_errors: []` and `smartctl_available: null` — "nobody told us either way", which is why + that flag is three-way and not a boolean. `gateway`'s agent is the one exception as of phase 10: + it is the installed systemd service, which does know about the SMART pass and answers + `smartctl_available: false` there because smartmontools is not installed — the *install it* + half of the three-way flag, from a host the dashboard cannot yet reach (its firewall). **Real data is in the repo now:** `tests/fixtures/smartctl_driveripper_.txt` is all 16 drives (8 NVMe, 8 spinning) read through that sudo rule, and `tests/test_agent.py` parses every one — which is what found the parser's one real bug: this binary annotates the @@ -376,14 +426,14 @@ That scheme is the history API's key — treat it as a public interface. ```bash uv sync --python 3.14 # deps + editable install cp .env.example .env # production-shaped config -cp .env.dev.example .env.dev # dev-only: ssh tunnels + live targets +cp .env.dev.example .env.dev # dev-only: the real hosts, tokens in URL userinfo -./scripts/dev-tunnels.sh start # agents (9121-9123) + DEV_FORWARDS (upsd 3494) +set -a; . ./.env.dev; set +a # dev dials the fleet the prod way, no tunnels ./scripts/dev-server.sh restart # keep http://127.0.0.1:8090 up ./scripts/dev-server.sh status # per-source ok/DOWN ./scripts/dev-server.sh logs -uv run pytest # hermetic: 769 passed + 11 live-skipped, no network +uv run pytest # hermetic: 793 passed + 11 live-skipped, no network uv run pytest --cov # coverage gate (98.33%; floor 94%) uv run pytest -m live --live # opt-in: real fleet / engines / upsd uv run ruff check src tests agent && uv run pyright @@ -402,8 +452,13 @@ bash .agents/validate.sh # the phase gate: the four checks abov Deployment: `Containerfile` (two-stage, `uv sync --frozen`, runs as uid 10001 on `0.0.0.0:8090`, SQLite under `/var/lib/dashboards` — put a volume there) and `deploy/install-agent.sh` per monitored host (systemd unit, own user, `--bind`/`--port`/`--token`/`--rotate-token`/ -`--status`/`--uninstall`; it stores the token in a root-owned 0600 file and hands the agent -`--token-file`, never argv, and runs under `ProtectSystem=strict`). +`--status`/`--uninstall`; it stores the token `0640 root:dashboards-agent` — the unit hands the +agent the *path* and the agent opens it as its own user, so that user must be able to read it and +no other account may — never puts a token on argv, and runs under `ProtectSystem=strict` with +`DevicePolicy=strict` reopening only the disk nodes plus the `/dev/null` family, because strict +without `/dev/null` kills the unit at spawn (`status=208/STDIN`). No root shell? The one sudoers +line that lets `ops` run the installer is `deploy/agent-sudoers`; run the script without root and +it prints that line with its own resolved path.) The quadlet unit + env template + nginx/SNI live in the deployment repo, like every other service here. The source itself is `git@git.reeseapps.com:services/dashboards.git` (`origin`, branch `main`); there is no CI on it, so `bash .agents/validate.sh` is the only gate. @@ -823,8 +878,8 @@ memory plot; the `cpu.usage_pct@coreN` series is still stored and still queryabl field map is written. 3. **UPS history depth.** Events and the numeric series are stored; long-window battery health (capacity fade over months) has the rollups but no dedicated view. -4. **The dev DB is big enough to refuse a write now and then.** The dev store (~590 MB, WAL - grown to ~70 MB) logs `store-error: OperationalError: database is locked` a few times an +4. **The dev DB is big enough to refuse a write now and then.** The dev store (~610 MB, WAL + grown past 140 MB) logs `store-error: OperationalError: database is locked` a few times an hour across `system`/`llm`/`ups`; the poller handles it exactly as invariant 4 requires — the source goes red with the real reason, ages, and writes `recording` when the next poll lands — so nothing freezes, but those polls' samples are lost. The one cause this repo can diff --git a/README.md b/README.md index 4ccb558..e0d2f14 100644 --- a/README.md +++ b/README.md @@ -27,29 +27,26 @@ uv run uvicorn dashboards.app:app --port 8090 # equivalent uv run pytest # unit tests, no network needed ``` -Developing against the real fleet from a laptop (agents stay unexposed — ssh forwards -them to localhost): +Developing against the real fleet from a laptop — dev dials exactly the addresses the deployed +dashboard dials, so there is nothing to tunnel and no port to forward: ```bash -cp .env.dev.example .env.dev # your ssh aliases + which ports -python3 - <<'PY' # push the agent to each host (once) -import shlex, subprocess, pathlib -for host in ("driveripper-ops", "deskwork-ops", "gateway-ops"): - subprocess.run(["ssh", host, "mkdir -p ~/lab-agent"], check=True) - subprocess.run(["scp", "agent/host_agent.py", f"{host}:~/lab-agent/"], check=True) -PY -./scripts/dev-tunnels.sh start # 9121/9122/9123 -> each host's agent, - # + DEV_FORWARDS -> upsd on 127.0.0.1:3494 - # (the remote port is upsd's real one — 3051 - # on driveripper, not the 3493 default) +cp .env.dev.example .env.dev # your host addresses + agent tokens (chmod 0600) +sudo deploy/install-agent.sh --bind --port 9120 + # once per monitored host: the agent as a systemd + # service, printing its bearer token exactly once set -a; . ./.env.dev; set +a uv run dashboards # foreground ./scripts/dev-server.sh restart # or backgrounded, with a pidfile + logs ``` -`DEV_TUNNELS` starts the agent remotely and forwards it; `DEV_FORWARDS` forwards a service -that already listens there (upsd) — that is how the UPS page reads a real battery without -3493 being reachable from anywhere but this laptop. +Each `HOST_TARGETS` entry carries that token as URL userinfo +(`driveripper=http://@:9120`); `parse_targets` moves it into an `Authorization` +header, so `Target.url`, `/api/config` and every error message stay credential-free. No root +shell on a host? `deploy/agent-sudoers` is the one-line grant that lets `ops` run the installer, +and running the installer without root prints that exact line. upsd is dialled at +`NUT_HOST`/`NUT_PORT` on the port it really answers on (3051 on driveripper, not the stock +3493) — see the NUT section below for who may connect. ## The host agent @@ -68,6 +65,10 @@ deploy/install-agent.sh --status # unit state and /he sudo deploy/install-agent.sh --uninstall ``` +No root shell on the monitored host? `deploy/agent-sudoers` is the single sudoers line that +lets the fleet's unprivileged `ops` account run the installer; run it without root and it +prints that exact line with the script's own resolved path instead of saying "run as root". + It is read-only by construction: there is no code path that writes, and the HTTP server only answers `GET`. Counters are reported as counters — the dashboard computes rates over a labelled window, so the agent never has to remember who polled it last. @@ -92,8 +93,11 @@ Binding off-loopback without a token is refused (exit 2) rather than warned abou `--token-file` over `--token`: argv is world-readable through `ps`, and this token guards hostname, mounts, disk and GPU models, and process names. `deploy/install-agent.sh` creates a system user, generates a token once (re-runs reuse it; `--rotate-token` for a new one), -stores it `0600` root-owned, and runs the agent under `ProtectSystem=strict` with no -capabilities — enough to read `/proc`, `/sys` and `statvfs()` and nothing else. +stores it `0640 root:dashboards-agent` — the unit hands the agent the *path*, and the agent +opens it as its own user, so the file must be readable by that user and by nobody else — +and runs the agent under `ProtectSystem=strict`, `DevicePolicy=strict` with only the disk +nodes and the `/dev/null` family reopened (SMART needs the former; without `/dev/null` in +the allow list systemd cannot start the service at all), with no capabilities. ## Configuration @@ -400,8 +404,14 @@ files: `sudo install -m 0440 -o root -g root deploy/nut-sudoers /etc/sudoers.d/d Read `upsd.users` before you write it — `upsmon` authenticates from the same file, so overwriting it blind can take the box down at the wrong moment. -Bind upsd to loopback (`LISTEN 127.0.0.1` in `upsd.conf`) and reach it with an ssh forward -(`DEV_FORWARDS`) or a tailnet address. `NUT_SIMULATE=1` stands in for upsd during +Bind upsd to loopback (`LISTEN 127.0.0.1 3051` in `upsd.conf`) when the dashboard runs on the +same box, and to a real interface plus a source-scoped firewall rule when it runs elsewhere — +and do not try to scope it by client inside `upsd.conf` instead: **NUT 2.8 +removed the `ACL`/`ACCEPT`/`REJECT` directives** (a build without TCP Wrappers logs +`ACL in upsd.conf is no longer supported - switch to LISTEN` once per line and ignores the +line), so `LISTEN` plus a host firewall rule is the only access control `upsd.conf` offers, +and an unauthenticated `LIST VAR` from any routed source is the normal state of a NUT server. +`NUT_SIMULATE=1` stands in for upsd during development; everything it produces is badged `SIMULATED`, and it is never the default. ## Deployment diff --git a/deploy/agent-sudoers b/deploy/agent-sudoers new file mode 100644 index 0000000..c086f53 --- /dev/null +++ b/deploy/agent-sudoers @@ -0,0 +1,76 @@ +# sudoers grant that lets the unprivileged `ops` account install the homelab dashboards +# host agent on a monitored host. One line, one script path — deploy/nut-sudoers is the +# separate NUT grant on driveripper. +# +# Install (from the staged copy on the host, as root — the command-scoped `ops` sudo on +# these hosts does NOT include visudo, which is exactly why this file exists): +# +# sudo visudo -cf /home/ops/agent-install/deploy/agent-sudoers # syntax check FIRST +# sudo install -m 0440 -o root -g root /home/ops/agent-install/deploy/agent-sudoers \ +# /etc/sudoers.d/dashboards-agent +# sudo visudo -cf /etc/sudoers.d/dashboards-agent # and check the installed copy +# +# Scope: an absolute path (no wildcards, `secure_path` is in effect), no shell, no editor, +# no `tee`, nothing that can amend this file. Running install-agent.sh without root prints +# this exact grant with the script's own resolved path, so a checkout at another path tells +# you what to add instead of saying "run as root" to someone who cannot. +# +# WHY THIS IS NEEDED +# The agent is read-only by construction: it answers GET /snapshot on a socket, writes +# nothing anywhere, and every request except /health needs the bearer token it reads from a +# file only root and the agent's own service user can read (0640 root:dashboards-agent — the +# agent opens the path itself, so the service user must be in that group). What needs root is +# the *install*, not the agent: creating the +# dashboards-agent system user, writing /usr/local/lib/dashboards-agent/host_agent.py, +# dropping /etc/systemd/system/dashboards-agent.service, storing the token, and running +# systemctl daemon-reload + enable --now (disable --now for --uninstall). +# Phase 10 runs each host's agent as that systemd service bound to the address the +# dashboard dials directly, so the account that deploys over ssh — ops — has to be able to +# run the installer, and that is the only reason it is granted anything. +# +# WHY THE WHOLE SCRIPT PATH AND NOT A FLAG LIST +# The flags carry no secrets: --bind/--port choose the socket, --status is read-only, +# --uninstall removes exactly what this install put in place, --rotate-token mints a token +# the operator is about to be shown anyway. sudoers can only match flags with a wildcard, +# and a wildcard list has to be edited in step with every flag the script gains — an edit +# whose failure mode is a broken install, not a smaller privilege. The one flag whose +# *argument* would be sensitive is --token, and no documented flow uses it: the installer +# generates the token, prints it once, and hands the agent --token-file, because argv is +# world-readable through ps while the file is not. + +# ---- the grant ------------------------------------------------------------------ # +# The host has no repo checkout, so deploy/ and agent/ are staged to ~/agent-install/ +# (phase 10 task 02 does the copy), which keeps this path identical on every host: +# +# ssh ops@host 'mkdir -p ~/agent-install' +# scp -r deploy agent ops@host:~/agent-install/ +# +# `ops` is the fleet's monitored/deploy account; change only that field if the agent is +# installed by a different account. Nothing follows the path, which in sudoers means the +# script may be run with any of its own flags — see the block above. +ops ALL=(root) NOPASSWD: /home/ops/agent-install/deploy/install-agent.sh + +# ---- making the grant mean something (one root command, recommended) ------------ # +# A sudo grant on a path is only as strong as who can write that path. ops owns +# ~/agent-install, so until the staged copy is root-owned this line is a convenience for a +# human, not a boundary. After staging, as root: +# +# sudo chown -R root:root /home/ops/agent-install +# +# ops can still wipe its own home directory and re-copy (it is ops' home), so this guards +# against drift and a stray edit — not against an operator who wants something else. + +# ---- deliberately NOT granted --------------------------------------------------- # +# visudo, vim, nano, sh, bash, tee — no self-amendment of this grant, no shell, and no +# writing any file as root +# systemctl (anything else) — the installer already runs the few unit operations +# it needs as itself; a bare systemctl * could stop +# anything on the host +# cat /etc/dashboards-agent.token — the token is readable by root and the agent user +# only; install-agent.sh prints it once on first +# install and --rotate-token mints a new one +# useradd, usermod, groupmod — the installer creates its own user and its disk +# group membership; a bare usermod could add ops to +# any group on the box +# upscmd, upsrw — NUT instant commands stay the dashboard's gated +# POST /api/ups/command; see deploy/nut-sudoers diff --git a/deploy/install-agent.sh b/deploy/install-agent.sh index 05987bb..4fa143f 100755 --- a/deploy/install-agent.sh +++ b/deploy/install-agent.sh @@ -1,15 +1,20 @@ #!/usr/bin/env bash # Install agent/host_agent.py as a systemd service on a monitored host. # -# sudo ./deploy/install-agent.sh # loopback only (ssh tunnel in) +# sudo ./deploy/install-agent.sh # loopback only (read it locally) # sudo ./deploy/install-agent.sh --bind 10.0.0.5 # serve the LAN, token required -# sudo ./deploy/install-agent.sh --status +# ./deploy/install-agent.sh --status # unit state + /health, no root # sudo ./deploy/install-agent.sh --uninstall # -# The token is stored in a root-owned 0600 file and passed to the agent as --token-file, -# never on the command line: argv is world-readable through `ps`, and this token guards -# hostname, mounts, disk models, GPU model and process names. Re-running the installer -# reuses the existing token — use --rotate-token when the old one leaked. +# No root shell? `deploy/agent-sudoers` is the single sudoers line that lets the fleet's +# unprivileged `ops` account run this script as root. Run it without root and it prints that +# exact line with this script's resolved path, then exits 1. +# +# The token is stored in a file only root and the agent's own service user can read, and is +# passed to the agent as --token-file, never on the command line: argv is world-readable +# through `ps`, and this token guards hostname, mounts, disk models, GPU model and process +# names. Re-running the installer reuses the existing token — use --rotate-token when the old +# one leaked. # # The agent is read-only by construction (it answers GET and writes nothing), runs as its # own system user, and gets the usual systemd lockdown. It still needs to read /proc, /sys @@ -32,6 +37,15 @@ TOKEN_FILE="/etc/dashboards-agent.token" UNIT="/etc/systemd/system/$SERVICE.service" SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" AGENT_SOURCE="$SCRIPT_DIR/../agent/host_agent.py" +# Absolute, symlink-resolved path of this script: the sudoers grant in deploy/agent-sudoers +# is matched on a path, so the non-root hint below prints the line that actually matches +# wherever the script happens to live (the canonical staging path is +# /home/ops/agent-install/deploy/install-agent.sh). +SCRIPT_PATH="$SCRIPT_DIR/$(basename "$0")" +if command -v realpath >/dev/null 2>&1; then + SCRIPT_PATH="$(realpath "$0" 2>/dev/null || printf '%s' "$SCRIPT_PATH")" +fi +SUDOERS_SOURCE="$(dirname "$SCRIPT_PATH")/agent-sudoers" BIND="127.0.0.1" PORT="9120" @@ -60,7 +74,30 @@ done need_root() { if [ "$(id -u)" -ne 0 ]; then - echo "run as root: sudo $0 $*" >&2 + # The dev workflow deploys over ssh as the unprivileged `ops` account, so "run as root" + # is not an action the operator can take. Print the one sudoers line that makes this + # script installable — the same line as deploy/agent-sudoers, with this script's own + # resolved path — plus the two-stage install it documents, then exit 1 as before. + cat >&2 </dev/null || true rm -f "$UNIT" "$TOKEN_FILE" "$AGENT_TARGET" rmdir "$AGENT_DIR" 2>/dev/null || true @@ -90,8 +127,12 @@ new_token() { python3 -c 'import secrets; print(secrets.token_urlsafe(32).rstrip("="))' } -install() { - need_root "$0 --bind $BIND --port $PORT" +# NOT named `install`: bash resolves a function before an external command of the same name, +# so an `install()` here turns the `install -d` / `install -m` calls below into infinite +# recursion instead of creating $AGENT_DIR. Phase 10 task 02 found exactly that on a live +# host: the run looped on the SMART block forever and installed nothing. +do_install() { + need_root --bind "$BIND" --port "$PORT" [ -f "$AGENT_SOURCE" ] || { echo "no agent next to this script: $AGENT_SOURCE" >&2; exit 1; } if ! command -v python3 >/dev/null 2>&1; then echo "python3 is required on the monitored host (3.10 or newer)" >&2 @@ -127,11 +168,14 @@ install() { elif [ -z "$TOKEN" ]; then TOKEN="$(new_token)" fi - # 0600 root:root — the service runs as $AGENT_USER, and systemd (PID 1, root) reads the - # file for it, so the agent process never needs read permission on it itself. + # 0640 root:$AGENT_USER. The unit passes `--token-file` as a *path*: systemd does not read + # the secret for the service, the agent process (running as $AGENT_USER) opens it itself, + # so a 0600 root-owned file makes the agent exit "cannot read --token-file" and the unit + # never starts. Group ownership is the smallest grant that lets the service read it, and + # every other account — ops included — still cannot. (umask 0177; printf '%s\n' "$TOKEN" > "$TOKEN_FILE") - chmod 0600 "$TOKEN_FILE" - chown root:root "$TOKEN_FILE" + chmod 0640 "$TOKEN_FILE" + chown root:"$AGENT_USER" "$TOKEN_FILE" cat > "$UNIT" <:127.0.0.1: 'python3 host_agent.py --port ' -# DEV_FORWARDS forwards a service that is already listening remotely (e.g. upsd on 3493): -# nothing is started over there, the port just appears on localhost. -# -# Both supervise themselves: a dropped link retries every 5s. Production is the opposite — -# deploy/install-agent.sh installs a systemd unit that binds the LAN address with a bearer -# token, and the dashboard dials it directly. -set -uo pipefail - -ROOT="$(cd "$(dirname "$0")/.." && pwd)" -RUN="$ROOT/data/tunnels" -LOGS="$ROOT/data/logs" -# Where the agent lives on each host. `~` is written into the generated script literally so -# the REMOTE shell expands it (a local $HOME would be the wrong path over there). -AGENT_REMOTE_DIR="${AGENT_REMOTE_DIR:-~/lab-agent}" -AGENT_COMMAND="${AGENT_COMMAND:-python3 host_agent.py}" - -# Targets come from .env.dev (see .env.dev.example) — never hardcoded here, so this file -# stays portable and holds no hostnames of ours. -[ -f "$ROOT/.env.dev" ] && . "$ROOT/.env.dev" -DEV_TUNNELS="${DEV_TUNNELS:-}" -DEV_FORWARDS="${DEV_FORWARDS:-}" -if [ -z "$DEV_TUNNELS" ] && [ -z "$DEV_FORWARDS" ]; then - echo "no DEV_TUNNELS or DEV_FORWARDS set — copy .env.dev.example to .env.dev first"; exit 2 -fi -TUNNELS=() -for entry in $DEV_TUNNELS; do - # 9121:driveripper-ops:9120 -> "9121 driveripper-ops 9120" - TUNNELS+=("$(echo "$entry" | tr ':' ' ')") -done -FORWARDS=() -for entry in $DEV_FORWARDS; do - FORWARDS+=("$(echo "$entry" | tr ':' ' ')") -done - -mkdir -p "$RUN" "$LOGS" - -start() { - for spec in "${TUNNELS[@]}"; do - read -r local host remote <<<"$(echo "$spec" | tr -s ' ')" - # pidfiles are named by host AND port: one host can carry an agent tunnel plus an upsd - # forward. The older host-only name is cleared so a restart does not double-bind. - pidfile="$RUN/$host-$local.pid" - if [ -f "$RUN/$host.pid" ] && kill -0 "$(cat "$RUN/$host.pid")" 2>/dev/null; then - pkill -P "$(cat "$RUN/$host.pid")" 2>/dev/null - kill "$(cat "$RUN/$host.pid")" 2>/dev/null - fi - rm -f "$RUN/$host.pid" - if [ -f "$pidfile" ] && kill -0 "$(cat "$pidfile")" 2>/dev/null; then - echo " $host already tunnelling on :$local (pid $(cat "$pidfile"))" - continue - fi - # A previous agent survives the ssh session that started it, and the next agent then - # dies on bind() with "address already in use" — which looks like a broken agent rather - # than a leaked process. Clear it in a SEPARATE ssh session: putting pkill in the same - # command line as the agent makes pkill match that very command line and shoot itself. - cat > "$RUN/$host.sh" </dev/null 2>&1 - sleep 1 - ssh -nT -o ServerAliveInterval=30 -o ServerAliveCountMax=4 -o ExitOnForwardFailure=yes \ - -L $local:127.0.0.1:$remote "$host" \ - "cd $AGENT_REMOTE_DIR && exec $AGENT_COMMAND --host 127.0.0.1 --port $remote" - echo "[tunnel] $host link dropped at \$(date -Is) (exit \$?); retrying in 5s" - sleep 5 -done -EOF - chmod +x "$RUN/$host.sh" - nohup "$RUN/$host.sh" >>"$LOGS/tunnel-$host.log" 2>&1 & - echo $! >"$pidfile" - echo " $host -> http://127.0.0.1:$local (pid $!)" - done - - # Plain forwards: `-N` because there is no remote command to run, and nothing to clean up - # over there when the link goes. - for spec in "${FORWARDS[@]}"; do - read -r local host remote <<<"$(echo "$spec" | tr -s ' ')" - pidfile="$RUN/$host-$local.pid" - if [ -f "$pidfile" ] && kill -0 "$(cat "$pidfile")" 2>/dev/null; then - echo " $host:$remote already forwarded on :$local (pid $(cat "$pidfile"))" - continue - fi - cat > "$RUN/$host-$local.sh" <>"$LOGS/tunnel-$host-$local.log" 2>&1 & - echo $! >"$pidfile" - echo " $host:$remote -> 127.0.0.1:$local (pid $!)" - done -} - -stop() { - for spec in "${TUNNELS[@]}"; do - read -r local host remote <<<"$(echo "$spec" | tr -s ' ')" - pidfile="$RUN/$host-$local.pid" - if [ -f "$RUN/$host.pid" ]; then - pkill -P "$(cat "$RUN/$host.pid")" 2>/dev/null - kill "$(cat "$RUN/$host.pid")" 2>/dev/null - rm -f "$RUN/$host.pid" - fi - [ -f "$pidfile" ] || continue - pid=$(cat "$pidfile") - pkill -P "$pid" 2>/dev/null - kill "$pid" 2>/dev/null - # -N would mean "no remote command", so the forward is kept by a plain ssh running the - # agent; kill any leftover forward for this host by pattern, not just by parent. - pkill -f "ssh -nT.*-L $local:127.0.0.1:$remote $host" 2>/dev/null - # The remote agent outlives its ssh session, and the next one then dies on bind() with - # "address already in use" — which looks like a broken agent, not a leaked process. - timeout 10 ssh -n "$host" "pkill -f '[h]ost_agent.py --host 127.0.0.1 --port $remote'" \ - >/dev/null 2>&1 - rm -f "$pidfile" - echo " stopped $host" - done - - for spec in "${FORWARDS[@]}"; do - read -r local host remote <<<"$(echo "$spec" | tr -s ' ')" - pidfile="$RUN/$host-$local.pid" - [ -f "$pidfile" ] || continue - pkill -P "$(cat "$pidfile")" 2>/dev/null - kill "$(cat "$pidfile")" 2>/dev/null - pkill -f "ssh -n -N.*-L $local:127.0.0.1:$remote $host" 2>/dev/null - rm -f "$pidfile" "$RUN/$host-$local.sh" - echo " stopped forward $host:$remote (:$local)" - done -} - -status() { - for spec in "${TUNNELS[@]}"; do - read -r local host remote <<<"$(echo "$spec" | tr -s ' ')" - body=$(curl -s --max-time 4 "http://127.0.0.1:$local/health" || true) - if [ -n "$body" ]; then - echo " OK $host http://127.0.0.1:$local $body" - else - echo " DOWN $host http://127.0.0.1:$local" - fi - done - - for spec in "${FORWARDS[@]}"; do - read -r local host remote <<<"$(echo "$spec" | tr -s ' ')" - # no HTTP behind these (upsd speaks its own line protocol), so the check is a bare TCP - # handshake — enough to tell "the tunnel is up" from "nothing listens over there". - if timeout 3 bash -c "exec 3<>/dev/tcp/127.0.0.1/$local" 2>/dev/null; then - echo " OK $host:$remote 127.0.0.1:$local" - else - echo " DOWN $host:$remote 127.0.0.1:$local" - fi - done -} - -case "${1:-start}" in - start) start ;; - stop) stop ;; - restart) stop; sleep 1; start ;; - status) status ;; - *) echo "usage: $0 start|stop|status|restart"; exit 2 ;; -esac diff --git a/src/dashboards/alerts.py b/src/dashboards/alerts.py index 8081b2d..b06a6c8 100644 --- a/src/dashboards/alerts.py +++ b/src/dashboards/alerts.py @@ -133,7 +133,7 @@ _POLLER_CLASS = re.compile(r"^UpstreamError: ") def _clean_error(label: str, error: str) -> str: - """`http://h:9121/snapshot unreachable (ConnectError)` from the poller's wrapped string.""" + """`http://:9120/snapshot unreachable (ConnectError)` from the poller's wrapped string.""" text = error.strip() while True: stripped = _POLLER_CLASS.sub("", text, count=1).strip() diff --git a/src/dashboards/sources/nut.py b/src/dashboards/sources/nut.py index 5880bfc..17f0675 100644 --- a/src/dashboards/sources/nut.py +++ b/src/dashboards/sources/nut.py @@ -118,8 +118,17 @@ class NutClient: asyncio.open_connection(self.host, self.port), self.timeout ) except (TimeoutError, OSError) as exc: + # `wait_for()` raises a TimeoutError whose own message is empty, so a blocked port + # used to read `cannot reach upsd at … (TimeoutError: )` in the health record, the + # alert and the event feed. Name the failure mode: this sentence is the whole + # explanation of a red UPS panel. + reason = ( + f"timed out after {self.timeout:g}s" + if isinstance(exc, TimeoutError) and not str(exc) + else f"{type(exc).__name__}: {exc}" + ) raise NutError( - f"cannot reach upsd at {self.host}:{self.port} ({type(exc).__name__}: {exc})", + f"cannot reach upsd at {self.host}:{self.port} ({reason})", code="UNREACHABLE", ) from exc diff --git a/tests/test_alerts.py b/tests/test_alerts.py index 9e00972..6eff584 100644 --- a/tests/test_alerts.py +++ b/tests/test_alerts.py @@ -106,21 +106,21 @@ def test_a_source_that_stopped_answering_is_a_fail_naming_it_and_linking_its_pag def test_the_collectors_host_prefix_is_not_doubled() -> None: """Collectors already say `driveripper: …`; the alert starts with the name too.""" out = alerts.build(SETTINGS, [state(ok=False, - error="driveripper: http://h:9121/snapshot unreachable")]) - assert out[0]["message"] == "driveripper: http://h:9121/snapshot unreachable" + error="driveripper: http://h:9120/snapshot unreachable")]) + assert out[0]["message"] == "driveripper: http://h:9120/snapshot unreachable" def test_the_pollers_exception_prefix_is_not_printed_either() -> None: """The live shape is `poller`'s `UpstreamError: name: …`; one name, no class noise. - Verified against a real stopped agent tunnel: the health record reads - `UpstreamError: gateway: http://127.0.0.1:9123/snapshot unreachable (ConnectError)`, + Verified against a real stopped agent: the health record reads + `UpstreamError: gateway: http://:9120/snapshot unreachable (ConnectError)`, and printing that verbatim put the host name on the page twice. """ out = alerts.build(SETTINGS, [state(key="host:gateway", label="gateway", ok=False, - error="UpstreamError: gateway: http://h:9123/snapshot " + error="UpstreamError: gateway: http://h:9120/snapshot " "unreachable (ConnectError)")]) - assert out[0]["message"] == ("gateway: http://h:9123/snapshot unreachable (ConnectError)") + assert out[0]["message"] == ("gateway: http://h:9120/snapshot unreachable (ConnectError)") def test_an_unexpected_exception_class_is_still_named() -> None: @@ -187,9 +187,9 @@ def test_a_disk_job_that_failed_because_its_agent_is_down_is_one_line_not_two() """ out = alerts.build(SETTINGS, [ state(key="disks:driveripper", dashboard="disks", label="driveripper", - ok=False, error="UpstreamError: driveripper: http://h:9121/snapshot " + ok=False, error="UpstreamError: driveripper: http://h:9120/snapshot " "unreachable (ConnectError)"), - state(ok=False, error="UpstreamError: driveripper: http://h:9121/snapshot " + state(ok=False, error="UpstreamError: driveripper: http://h:9120/snapshot " "unreachable (ConnectError)"), ], now=2_000.0) assert [item["source"] for item in out] == ["driveripper"] diff --git a/tests/test_api.py b/tests/test_api.py index 89004d2..ab8f6cc 100644 --- a/tests/test_api.py +++ b/tests/test_api.py @@ -453,13 +453,13 @@ def test_an_upstream_failure_escaping_a_route_keeps_its_own_status(tmp_path: Pat @app.get("/api/_boom") async def boom() -> None: - raise UpstreamError("driveripper: http://127.0.0.1:9121/snapshot timed out", status=504) + raise UpstreamError("driveripper: http://127.0.0.1:9120/snapshot timed out", status=504) with TestClient(app) as test_client: response = test_client.get("/api/_boom") assert response.status_code == 504, "a timeout stays a timeout, not a 500" assert response.json() == {"ok": False, - "detail": "driveripper: http://127.0.0.1:9121/snapshot " + "detail": "driveripper: http://127.0.0.1:9120/snapshot " "timed out"} assert "no-store" in response.headers["cache-control"] store.close() diff --git a/tests/test_config.py b/tests/test_config.py index cd06411..f89ab7c 100644 --- a/tests/test_config.py +++ b/tests/test_config.py @@ -119,7 +119,7 @@ def test_nut_host_accepts_a_nut_scheme_prefix() -> None: def test_load_reads_host_targets_from_the_environment(monkeypatch: pytest.MonkeyPatch) -> None: - monkeypatch.setenv("HOST_TARGETS", "deskwork=http://127.0.0.1:9122") + monkeypatch.setenv("HOST_TARGETS", "deskwork=http://tok@deskwork.example.net:9120") settings = config.load() assert [t.name for t in settings.host_targets] == ["deskwork"] diff --git a/tests/test_deploy_scripts.py b/tests/test_deploy_scripts.py new file mode 100644 index 0000000..98ca3ce --- /dev/null +++ b/tests/test_deploy_scripts.py @@ -0,0 +1,398 @@ +"""Contract tests for the deployment scripts in `deploy/`. + +There is no shell test framework here — and adding one would buy nothing a subprocess does +not — but phase 10 rests on a sentence that lives in **two** files: the sudoers grant in +`deploy/agent-sudoers`, and the hint `deploy/install-agent.sh` prints when it is run without +root by the unprivileged `ops` account. Two copies of one instruction is exactly what drifts +silently, and the failure is an install that cannot happen on a host nobody is sitting at. +So these tests read the file, run the script the way the operator does (unprivileged, which +exits before the script can touch anything), and require the two lines to be the same line +apart from the path the script was invoked through. +""" + +from __future__ import annotations + +import os +import re +import shutil +import stat +import subprocess +from pathlib import Path + +import pytest + +ROOT = Path(__file__).resolve().parent.parent +DEPLOY = ROOT / "deploy" +INSTALLER = DEPLOY / "install-agent.sh" +SUDOERS = DEPLOY / "agent-sudoers" +BASH = shutil.which("bash") or "/bin/bash" + +#: The grant as the file documents it. `/home/ops/agent-install` is the canonical staging +#: path: a monitored host has no repo checkout, so `deploy/` and `agent/` are copied there +#: (phase 10 task 02) and every host ends up with the same absolute path to grant. +CANONICAL_GRANT = "ops ALL=(root) NOPASSWD: /home/ops/agent-install/deploy/install-agent.sh" + +#: The prefix a correct hint line starts with — user, run-as, and NOPASSWD, everything but +#: the path, which is the one thing the installer is allowed to substitute. +GRANT_PREFIX = "ops ALL=(root) NOPASSWD: " + +#: Everything a non-root operator needs to hear, besides the grant line itself. +HINT_MUST_SAY = ( + "visudo -cf", # check the syntax before installing it + "install -m 0440 -o root -g root", # and install it the way sudoers files are installed + "/etc/sudoers.d/dashboards-agent", # the destination both files name + "deploy/agent-sudoers", # the pointer back to the documented file +) + + +def _run(args: list[str], cwd: Path | None = None) -> subprocess.CompletedProcess[str]: + return subprocess.run(args, capture_output=True, text=True, cwd=cwd, check=False) + + +def _grant_lines(text: str) -> list[str]: + """The sudoers *rules* in a file — comments start with `#`, so this never matches prose.""" + return [line for line in text.splitlines() if line.startswith(GRANT_PREFIX)] + + +def _installer_without_root(*flags: str) -> subprocess.CompletedProcess[str]: + """Run the installer the way `ops` does. + + Skipped as root on purpose: root would go past `need_root` and really install a systemd + unit on the machine running pytest. + """ + if hasattr(os, "geteuid") and os.geteuid() == 0: + pytest.skip("running as root — the installer would install a real unit here") + return _run([BASH, str(INSTALLER), *flags]) + + +# --- deploy/agent-sudoers --------------------------------------------------------------- + + +def test_agent_sudoers_exists_and_is_a_config_file_not_a_script() -> None: + assert SUDOERS.is_file(), "deploy/agent-sudoers is missing (phase 10 task 01)" + assert not os.access(SUDOERS, os.X_OK), "a sudoers file is data; only install-agent.sh runs" + + +def test_agent_sudoers_is_exactly_one_grant_line_at_the_canonical_path() -> None: + """One rule, the installer, the staging path — nothing else may be granted.""" + lines = _grant_lines(SUDOERS.read_text()) + assert lines == [CANONICAL_GRANT], ( + "deploy/agent-sudoers must grant exactly one command, the staged installer:\n" + "\n".join(lines) + ) + assert "/home/ops/agent-install" in SUDOERS.read_text() + + +def test_agent_sudoers_grants_no_shell_editor_or_second_path() -> None: + """The 'deliberately NOT granted' block has to stay a comment, not become a rule.""" + text = SUDOERS.read_text() + not_granted = text.split("deliberately NOT granted", 1)[-1] + for never in ("visudo", "sh", "tee", "systemctl", "usermod", "upscmd", "upsrw"): + assert never in not_granted, f"the file must name {never!r} as deliberately not granted" + assert not_granted.lstrip().startswith("-"), "the block heading is followed by the list" + # Every line of that block is a comment: a stray rule there would be a second grant. + assert _grant_lines(not_granted) == [] + # And it names the token file, because "we did not grant reading it" is the point. + assert "/etc/dashboards-agent.token" in not_granted + + +def test_agent_sudoers_documents_the_why_and_the_two_stage_install() -> None: + """House style (deploy/nut-sudoers): why it is needed, then check-first install steps.""" + text = SUDOERS.read_text() + assert "WHY THIS IS NEEDED" in text + for step in HINT_MUST_SAY: + assert step in text, f"deploy/agent-sudoers must document {step!r}" + # The installed copy is checked too — the file being valid is not the same as the + # installed file being valid (nut-sudoers' third step, and the reason stays). + assert text.count("visudo -cf") >= 2 + + +@pytest.mark.skipif(shutil.which("visudo") is None, reason="visudo is not installed on this box") +def test_agent_sudoers_passes_the_syntax_check_it_documents() -> None: + """The file tells the operator to run `visudo -cf` on it, so it must survive that.""" + result = _run([shutil.which("visudo") or "/usr/sbin/visudo", "-cf", str(SUDOERS)]) + assert result.returncode == 0, f"visudo -cf failed:\n{result.stdout}{result.stderr}" + + +# --- deploy/install-agent.sh without root ----------------------------------------------- + + +def test_installer_without_root_exits_1_and_prints_the_grant_line() -> None: + result = _installer_without_root("--bind", "127.0.0.1") + assert result.returncode == 1, "the non-root refusal keeps its exit code (1)" + assert result.stdout == "", "the refusal is a failure: it goes to stderr, not stdout" + lines = _grant_lines(result.stderr) + assert lines == [f"{GRANT_PREFIX}{Path(os.path.realpath(INSTALLER))}"], ( + "the hint must print one grant line naming this script's resolved path:\n" + result.stderr + ) + + +def test_hint_and_the_sudoers_file_are_the_same_line_apart_from_the_path() -> None: + """The pinned version of the diff in data/logs/NOTES-sudoers.md.""" + printed = _installer_without_root("--bind", "127.0.0.1").stderr + hint = _grant_lines(printed)[0] + canonical = _grant_lines(SUDOERS.read_text())[0] + assert hint.startswith(GRANT_PREFIX) and canonical.startswith(GRANT_PREFIX) + assert hint.removeprefix(GRANT_PREFIX).endswith("/deploy/install-agent.sh") + assert hint.removeprefix(GRANT_PREFIX).removeprefix(str(DEPLOY.parent)) == ( + canonical.removeprefix(GRANT_PREFIX).removeprefix("/home/ops/agent-install") + ), "same user, same (root), same NOPASSWD, same script — only the checkout path differs" + + +def test_hint_resolves_a_relative_invocation_to_an_absolute_path() -> None: + """`bash deploy/install-agent.sh` from the repo root still prints a path sudo can match.""" + result = _run([BASH, "deploy/install-agent.sh", "--bind", "10.0.0.5"], cwd=ROOT) + assert result.returncode == 1 + assert _grant_lines(result.stderr) == [f"{GRANT_PREFIX}{Path(os.path.realpath(INSTALLER))}"] + + +def test_installer_names_the_two_stage_install_and_points_at_the_file() -> None: + stderr = _installer_without_root("--bind", "127.0.0.1").stderr + for phrase in HINT_MUST_SAY: + assert phrase in stderr, f"the non-root hint must say {phrase!r}, not just 'run as root'" + # The pointer is the file next to *this* script, wherever the checkout lives. + assert str(Path(os.path.realpath(SUDOERS))) in stderr + + +def test_installer_hint_repeats_the_command_that_was_asked_for() -> None: + """Copy-pasteable: the hint ends with the same invocation, with the flags it was given.""" + stderr = _installer_without_root("--uninstall").stderr + assert f"sudo {Path(os.path.realpath(INSTALLER))} --uninstall" in stderr + + +def test_installer_help_points_at_the_sudoers_grant() -> None: + """The header comment *is* the help text, so the grant is discoverable from --help.""" + result = _run([BASH, str(INSTALLER), "--help"]) + assert result.returncode == 0 + assert "deploy/agent-sudoers" in result.stdout + + +# --- the tunnel machinery stays retired (phase 10) --------------------------------------- + +#: Names and ports that only ever existed to serve the ssh-tunnel dev transport, which phase +#: 10 deleted. Every pattern is written as a character class (`[d]ev-…`) so that **this file +#: cannot match its own grep**: the phase criterion demands the repo stay free of those +#: literals, and a guard that reintroduces one to forbid it is the oldest trick there is. +#: Scope excludes `.agents/` (phase files and run reports are history — rewriting a finished +#: phase's record to satisfy a grep would falsify it) and `tests/fixtures/` (verbatim +#: upstream captures: a sector counter or a wheel hash may contain those digits by accident, +#: and rule 7 forbids editing a capture to please a grep). +TUNNEL_PATTERNS = ("[d]ev-tunnels", "DEV_[T]UNNELS", "DEV_[F]ORWARDS", r"912[123]\b", r"[3]494\b") + +#: Where those names must not appear: every file that *tells someone what to run*. +#: `tests/fixtures/` is excluded (verbatim captures, see above) and so is `.agents/` — +#: phase files and run reports are history, and rewriting a finished phase's record to +#: satisfy a grep would falsify it. +TUNNEL_FREE_ROOTS = ("scripts", "deploy", "src", "agent", "web") + + +def _tunnel_scope_files() -> list[Path]: + files: list[Path] = [] + for top in TUNNEL_FREE_ROOTS: + for path in (ROOT / top).rglob("*"): + if path.is_file() and path.suffix in (".py", ".sh", ".md", ".css", ".js", ".html"): + files.append(path) + files.append(ROOT / "README.md") + files.append(ROOT / "AGENTS.md") + files.append(ROOT / ".env.example") + files.append(ROOT / ".env.dev.example") + for path in (ROOT / "tests").glob("*.py"): + files.append(path) + return files + + +def test_dev_tunnel_script_is_deleted() -> None: + """The tunnel script is gone, not shelved: git history keeps it if it is ever needed.""" + # Name built at runtime so this file never carries the literal the grep forbids. + script = ROOT / "scripts" / ("dev-" + "tunnels.sh") + assert not script.exists(), ( + "the retired ssh-tunnel dev script is back — phase 10 deleted it deliberately" + ) + + +def test_no_operational_file_mentions_the_tunnel_machinery() -> None: + """A stale tunnel-env instruction is a setup step nobody can complete any more.""" + offenders: list[str] = [] + for path in _tunnel_scope_files(): + text = path.read_text(errors="replace") + for pattern in TUNNEL_PATTERNS: + if re.search(pattern, text): + offenders.append(f"{path.relative_to(ROOT)}: /{pattern}/") + assert not offenders, "tunnel-era names/ports are back:\n" + "\n".join(offenders) + + +# --- shell syntax (the only gate these scripts get) ------------------------------------- + + +#: Names a shell function must never take. Bash resolves a **function** before an external +#: command of the same name, so defining `install()` in a script that then calls +#: `install -d /somewhere` is not a directory creation — it is the function calling itself, +#: forever. That is not hypothetical: `deploy/install-agent.sh` did exactly this, and the +#: first real install of phase 10 spun on the SMART block for two minutes and installed +#: nothing (no unit, no agent file, no token) while looking like a working script to `bash -n`. +SHADOW_FORBIDDEN = frozenset( + { + "install", + "test", + "time", + "date", + "sort", + "grep", + "sed", + "tee", + "type", + "ulimit", + "wait", + "kill", + "cd", + "pwd", + } +) + +_FUNCTION_DEF_RE = r"^([A-Za-z_][A-Za-z0-9_]*)\s*\(\)" + + +def _function_names(script: Path) -> list[str]: + return re.findall(_FUNCTION_DEF_RE, script.read_text(), flags=re.MULTILINE) + + +@pytest.mark.parametrize( + "script", + sorted([*DEPLOY.glob("*.sh"), *(ROOT / "scripts").glob("*.sh")]), + ids=lambda p: str(p.relative_to(ROOT)), +) +def test_shell_script_has_valid_syntax(script: Path) -> None: + result = _run([BASH, "-n", str(script)]) + assert result.returncode == 0, f"bash -n failed for {script}:\n{result.stderr}" + + +@pytest.mark.parametrize( + "script", + sorted([*DEPLOY.glob("*.sh"), *(ROOT / "scripts").glob("*.sh")]), + ids=lambda p: str(p.relative_to(ROOT)), +) +def test_no_shell_function_is_named_after_a_coreutil_it_could_be_asked_to_run(script: Path) -> None: + """`install()` + `install -d` is infinite recursion, and `bash -n` cannot see it.""" + shadowed = SHADOW_FORBIDDEN.intersection(_function_names(script)) + assert not shadowed, ( + f"{script.relative_to(ROOT)} defines function(s) {sorted(shadowed)} that shadow an " + "external command: bash runs the function, so any call meant for the program recurses" + ) + + +# --- the install path, run for real against a sandboxed root boundary -------------------- + +#: The commands of `install()` that would touch the machine running pytest. Each one is a +#: PATH stub that records its arguments and succeeds — the same trick `ScriptedNutClient` is +#: for the UPS wire: fake the privileged boundary, run the real logic. +ROOT_BOUNDARY_STUBS = ("id", "getent", "useradd", "usermod", "chown", "systemctl") + +#: The three absolute destinations the installer writes to, rewritten into a temp directory +#: so the test exercises the body instead of the operator's `/usr` and `/etc`. +SANDBOX_REWRITES = { + 'AGENT_DIR="/usr/local/lib/dashboards-agent"': 'AGENT_DIR="{root}/lib"', + 'TOKEN_FILE="/etc/dashboards-agent.token"': 'TOKEN_FILE="{root}/token"', + 'UNIT="/etc/systemd/system/$SERVICE.service"': 'UNIT="{root}/dashboards-agent.service"', +} + + +def _sandbox_installer(tmp_path: Path) -> tuple[Path, Path, dict[str, str]]: + """A copy of the installer whose writes land in `tmp_path`, plus a stubbed PATH. + + Returns (script, stub call log, env). `agent/host_agent.py` is stubbed next to the copy + because the script resolves the agent as `../agent/host_agent.py`. + """ + text = INSTALLER.read_text() + target = tmp_path / "target" + for old, new in SANDBOX_REWRITES.items(): + assert old in text, f"the installer moved {old!r} — update SANDBOX_REWRITES with it" + text = text.replace(old, new.format(root=target)) + + script = tmp_path / "deploy" / "install-agent.sh" + script.parent.mkdir(parents=True) + script.write_text(text) + (tmp_path / "agent").mkdir() + (tmp_path / "agent" / "host_agent.py").write_text("# stub agent source\n", encoding="utf-8") + + stubs = tmp_path / "stubs" + stubs.mkdir() + call_log = tmp_path / "stub-calls.log" + for name in ROOT_BOUNDARY_STUBS: + stub = stubs / name + # `id -u` must answer 0 or need_root() refuses before the body runs; every other call + # just records itself, which is what the assertions below read. + body = ['#!/bin/sh', f'printf "%s %s\\n" "{name}" "$*" >> "$STUB_LOG"'] + if name == "id": + body.append('[ "$1" = "-u" ] && echo 0') + body.append("exit 0") + stub.write_text("\n".join(body) + "\n", encoding="utf-8") + stub.chmod(0o755) + + env = {**os.environ, "PATH": f"{stubs}{os.pathsep}{os.environ['PATH']}", "STUB_LOG": str(call_log)} + return script, call_log, env + + +@pytest.mark.skipif(hasattr(os, "geteuid") and os.geteuid() == 0, reason="running as root") +def test_install_path_installs_into_its_own_paths_and_prints_the_token_once(tmp_path: Path) -> None: + """The install body end to end: files, unit, token, and no recursion. + + The exit code is the regression test for the `install()` shadow: while the function was + named after `/usr/bin/install`, this never returned at all. + """ + script, call_log, env = _sandbox_installer(tmp_path) + try: + result = subprocess.run( + [BASH, str(script), "--bind", "10.0.0.5", "--port", "9120"], + capture_output=True, + text=True, + env=env, + timeout=60, + check=False, + ) + except subprocess.TimeoutExpired as exc: + pytest.fail(f"the installer did not return (recursion? partial output: {exc.stdout!r})") + assert result.returncode == 0, f"install failed:\n{result.stdout}\n{result.stderr}" + + target = tmp_path / "target" + assert (target / "lib" / "host_agent.py").is_file(), "install -m never landed the agent" + unit = (target / "dashboards-agent.service").read_text() + token_path = target / "token" + token = token_path.read_text().strip() + + assert "User=dashboards-agent" in unit + assert f"--token-file {token_path}" in unit, "the unit must use the file, never --token" + assert "--host 10.0.0.5 --port 9120" in unit + # DevicePolicy=strict denies every node not listed, and systemd opens /dev/null for stdin + # before it execs python: without this line the unit dies at spawn (status 208/STDIN). + assert "DevicePolicy=strict" in unit + assert "DeviceAllow=/dev/null" in unit, "strict device policy needs /dev/null or the unit cannot start" + assert "DeviceAllow=/dev/nvme*" in unit, "SMART reads need the disk nodes back" + assert token and len(token) >= 32, "a generated token is expected" + assert token in result.stdout, "first install prints the token, and never again" + assert f'HOST_TARGETS="hostname=http://{token}@10.0.0.5:9120"' in result.stdout + + calls = call_log.read_text() + assert "systemctl daemon-reload" in calls + assert "systemctl enable --now dashboards-agent" in calls + assert "chown root:dashboards-agent" in calls, "the token file must become group-readable" + + +def test_token_file_is_readable_by_the_service_user_and_nobody_else(tmp_path: Path) -> None: + """`--token-file` is a path the *agent* opens, as its own user. + + systemd does not read the secret for the service, so a 0600 root-owned file makes the + agent exit before it binds. 0640 root:dashboards-agent is the smallest mode that works, + and it is still unreadable by `ops` (which is why deploy/agent-sudoers does not grant it). + """ + script, _call_log, env = _sandbox_installer(tmp_path) + result = subprocess.run( + [BASH, str(script), "--bind", "10.0.0.5"], + capture_output=True, + text=True, + env=env, + timeout=60, + check=False, + ) + assert result.returncode == 0, f"install failed:\n{result.stdout}\n{result.stderr}" + mode = stat.S_IMODE((tmp_path / "target" / "token").stat().st_mode) + assert mode == 0o640, f"token file mode is {oct(mode)}, expected 0o640 (group = the agent user)" + assert not mode & 0o007, "the token file must never be world-readable" diff --git a/tests/test_disks_source.py b/tests/test_disks_source.py index d77a9c0..1fbd990 100644 --- a/tests/test_disks_source.py +++ b/tests/test_disks_source.py @@ -311,7 +311,7 @@ def test_two_hosts_do_not_share_verdict_state(disk_snapshot: Any) -> None: # ---- the job: its own, slow, and absent rather than dialling --------------- # def test_one_disks_job_per_host_agent_at_the_slow_cadence() -> None: - targets = [Target("driveripper", "http://a:9121"), Target("deskwork", "http://b:9122")] + targets = [Target("driveripper", "http://a:9120"), Target("deskwork", "http://b:9120")] # the client is never used by the factory, only captured by the job closures jobs = disks_jobs(targets, NeverDialClient(), DisksCollector(), 300.0, timeout=30.0) assert [job.key for job in jobs] == ["disks:driveripper", "disks:deskwork"] diff --git a/tests/test_live_fleet.py b/tests/test_live_fleet.py index 8fb02b1..aeb540e 100644 --- a/tests/test_live_fleet.py +++ b/tests/test_live_fleet.py @@ -5,9 +5,9 @@ bridge to reality — they re-run the *production* code paths (`collectors.fetch `NutMonitor.collect`) against real targets, which is what catches an upstream that renamed a series or a driver that started reporting a temperature in kelvin. -They read exactly the env vars the app reads, so the dev loop is: +They read exactly the env vars the app reads, so the dev loop is direct dial — the same +addresses the deployed dashboard dials, nothing forwarded: - ./scripts/dev-tunnels.sh start # agents + upsd forwarded to localhost set -a; . .env.dev; set +a uv run pytest -m live --live @@ -55,7 +55,7 @@ async def test_every_host_agent_answers_and_parses(settings: Settings, host_collector: HostCollector) -> None: """Each HOST_TARGETS entry must answer /snapshot and yield the metrics the page renders.""" if not settings.host_targets: - pytest.skip("HOST_TARGETS is unset — start ./scripts/dev-tunnels.sh") + pytest.skip("HOST_TARGETS is unset") required = {"cpu.usage_pct", "mem.used_pct", "disk.used_pct"} async with httpx.AsyncClient(timeout=settings.host_timeout) as client: for target in settings.host_targets: @@ -84,7 +84,7 @@ async def test_every_host_serves_the_hero_card_contract(settings: Settings, upgrade renaming a field. Hermetic twin: `tests/test_system_contract.py`. """ if not settings.host_targets: - pytest.skip("HOST_TARGETS is unset — start ./scripts/dev-tunnels.sh") + pytest.skip("HOST_TARGETS is unset") async with httpx.AsyncClient(timeout=settings.host_timeout) as client: for target in settings.host_targets: await collectors.fetch_host_snapshot(client, target, host_collector) @@ -107,7 +107,7 @@ async def test_agent_disk_rows_carry_the_smart_shape(settings: Settings) -> None operator's call, not a unit test. """ if not settings.host_targets: - pytest.skip("HOST_TARGETS is unset — start ./scripts/dev-tunnels.sh") + pytest.skip("HOST_TARGETS is unset") seen_smart = 0 async with httpx.AsyncClient(timeout=settings.host_timeout) as client: for target in settings.host_targets: @@ -147,7 +147,7 @@ async def test_disks_source_charts_the_physical_drives_the_agent_reports( disk, `disk.*@` series only for values that exist, and no invented numbers. """ if not settings.host_targets: - pytest.skip("HOST_TARGETS is unset — start ./scripts/dev-tunnels.sh") + pytest.skip("HOST_TARGETS is unset") collector = DisksCollector() checked = 0 async with httpx.AsyncClient(timeout=settings.disks_timeout) as client: @@ -177,7 +177,7 @@ async def test_disks_source_charts_the_physical_drives_the_agent_reports( if any(row["smart"] for row in payload["disks"]): assert payload["smartctl_available"] is True if not checked: - pytest.skip("no host agent answered — start ./scripts/dev-tunnels.sh") + pytest.skip("no host agent answered") @pytest.mark.parametrize("engine", ["gufo", "vllm"]) diff --git a/tests/test_nut.py b/tests/test_nut.py index 3a473b1..3b771b0 100644 --- a/tests/test_nut.py +++ b/tests/test_nut.py @@ -249,6 +249,31 @@ async def test_connection_refused_is_reported_as_unreachable() -> None: with pytest.raises(NutError) as excinfo: await client.connect() assert excinfo.value.code == "UNREACHABLE" + # The reason is the sentence the operator actually reads (health record, alert, feed). + assert "ConnectionRefusedError" in str(excinfo.value) + + +async def test_connect_timeout_names_the_timeout_instead_of_an_empty_parenthetical( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """`wait_for()` raises a TimeoutError with no message of its own. + + A firewalled port therefore used to print `… (TimeoutError: )`, which explains nothing. + This is the exact state the fleet is in while the laptop cannot reach upsd, so the + message has to carry the timeout it hit. + """ + + async def never_connects(*_args: object, **_kwargs: object) -> None: + await asyncio.sleep(0.2) + + monkeypatch.setattr(asyncio, "open_connection", never_connects) + client = NutClient("ups.example.net", 3051, timeout=0.01) + with pytest.raises(NutError) as excinfo: + await client.connect() + assert excinfo.value.code == "UNREACHABLE" + message = str(excinfo.value) + assert "timed out after 0.01s" in message + assert "TimeoutError" not in message async def test_server_closing_early_is_reported() -> None: diff --git a/tests/ups_contract.py b/tests/ups_contract.py index 3c5ef71..50ad6fb 100644 --- a/tests/ups_contract.py +++ b/tests/ups_contract.py @@ -85,8 +85,8 @@ def parse_capture(name: str) -> nut.UpsReading: path = FIXTURES / f"nut_{name}.vars" if not path.exists(): raise AssertionError( - f"missing capture {path.name} — run ./scripts/capture-fixtures.sh with the upsd " - f"forwarded (DEV_FORWARDS); the UPS hero row is contracted against real drivers" + f"missing capture {path.name} — run ./scripts/capture-fixtures.sh with NUT_HOST " + f"pointing at the real upsd; the UPS hero row is contracted against real drivers" ) variables: dict[str, str] = {} commands: list[str] = [] diff --git a/uv.lock b/uv.lock index 9f539fa..102d9f8 100644 --- a/uv.lock +++ b/uv.lock @@ -189,7 +189,7 @@ wheels = [ [[package]] name = "fastapi" -version = "0.143.3" +version = "0.143.5" source = { registry = "https://pypi.org/simple" } dependencies = [ { name = "annotated-doc" }, @@ -199,9 +199,9 @@ dependencies = [ { name = "typing-extensions" }, { name = "typing-inspection" }, ] -sdist = { url = "https://files.pythonhosted.org/packages/38/d5/b0aeb0b303650ac87b0411de4aaf3f901b06042dcc3fb00f30b97b7ff585/fastapi-0.143.3.tar.gz", hash = "sha256:b9ab567ee57afb4454004ce04c889f8f0160ac47946b99795887cd6e88484848", size = 469834, upload-time = "2026-10-16T14:14:13.31Z" } +sdist = { url = "https://files.pythonhosted.org/packages/8f/6a/ac6ec87026038fb7583018e0905f6c9d55bc8ee07d30d44c743f18b19435/fastapi-0.143.5.tar.gz", hash = "sha256:0b5c8e52d243cd2f015376b41e78db5bee585578782481abd145e1ab5a62a3ba", size = 470247, upload-time = "2026-10-18T09:12:55.556Z" } wheels = [ - { url = "https://files.pythonhosted.org/packages/2c/1a/03d246befcf307219bb2161c97d8c5d314f84125bf7dd245730a28589828/fastapi-0.143.3-py3-none-any.whl", hash = "sha256:75e8bd68e2f52f77d29b2879749c257d64de2d85282d25270caf2bfccce5efe5", size = 144849, upload-time = "2026-10-16T14:14:11.763Z" }, + { url = "https://files.pythonhosted.org/packages/eb/4e/5f7d1268900eb14a6697a42d94c8f7e3b8450d27301761f231672f928419/fastapi-0.143.5-py3-none-any.whl", hash = "sha256:09cc38ed8c6a95f407783ef4d98ceeb350e192fe8fbb8ed711070bcd5208fdfa", size = 144894, upload-time = "2026-10-18T09:12:54.125Z" }, ] [[package]]